21.22 Removing Sparse Terms

We are often not interested in infrequent terms in our documents. Such ``sparse’’ terms can be removed from the document term matrix quite easily using tm::removeSparseTerms():

dim(dtm)

## [1]   46 6508

dtms <- removeSparseTerms(dtm, 0.1)
dim(dtms)

## [1] 46  6

This has removed most terms!

inspect(dtms)

## <<DocumentTermMatrix (documents: 46, terms: 6)>>
## Non-/sparse entries: 257/19
## Sparsity           : 7%
## Maximal term length: 7
## Weighting          : term frequency (tf)
## Sample             :
##                               Terms
## Docs                           data graham inform time use william
##   adm02.txt                     189      1     12   45  31       1
##   his03.txt                     176     17      1    7   7      25
##   ijdwm2012.txt                 110      2     65    4  65       2
##   karisk.txt                     83      1     24    6  55       3
##   kddmodel.txt                  127      1     23    1  44       3
##   pakdd07.txt                   161      1      6    2  26       3
##   RJournal_2009-2_Williams.txt  122      2      8    9  62      10
##   seal98.txt                    171      2     18   13  31       2
##   story.txt                     205     11      4   18  38      35
##   templates.txt                  67      1     55   25  34       6

We can see the effect by looking at the terms we have left:

freq <- colSums(as.matrix(dtms))
freq

##    data  graham  inform    time     use william 
##    3101     108     467     483    1366     236

table(freq)

## freq
##  108  236  467  483 1366 3101 
##    1    1    1    1    1    1

---

If you find this curated material useful then you can consider a donation to support it's ongoing availability and give you access to the PDF version of this book. The material has been scoped up by Generative AI without permission or any kind of recompense so do consider a donation if you can afford it. Unlike Generative AI your access to this materials is freely given. Desktop Survival Guides include Data Science, GNU/Linux, and MLHub. Books available on Amazon include Data Mining with Rattle and Essentials of Data Science. Togaware has a 30 year tradition of making popular open source software which includes sold privacy preserving productivity apps, rattle, wajig, and mlhub. Hosted by Togaware, a pioneer of free and open source software since 1984. Copyright © 1995-2022 Graham.Williams@togaware.com Creative Commons Attribution-ShareAlike 4.0