21.2 Corpus as a Data Set
The primary package for text mining, (Feinerer and Hornik 2024), provides a framework within which we perform our text mining. A collection of other standard R packages add value to the data processing and visualizations for text mining.
The basic concept is that of a corpus. This is a collection of texts, usually stored electronically, and from which we perform our analysis. A corpus might be a collection of news articles from Reuters or the published works of Shakespeare. Within each corpus we will have separate documents, which might be articles, stories, or book volumes. Each document is treated as a separate entity or record.
Documents which we wish to analyse come in many different formats. Quite a few formats are supported by (Feinerer and Hornik 2024), the package we will illustrate text mining with in this module. The supported formats include text, PDF, Microsoft Word, and XML.
A number of open source tools are also available to convert most
document formats to text files. For our corpus used initially in this
module, a collection of PDF documents were converted to text using
pdftotext from the xpdf application which is
available for GNU/Linux and MS/Windows and others. On GNU/Linux we can
convert a folder of PDF documents to text with:
The -enc ASCII7 ensures the text is converted to ASCII since
otherwise we may end up with binary characters in our text documents.
We can also convert Word documents to text using anitword,
which is another application available for GNU/Linux.
References
Your donation will support ongoing availability and give you access to the PDF version of this book. Desktop Survival Guides include Data Science, GNU/Linux, and MLHub. Books available on Amazon include Data Mining with Rattle and Essentials of Data Science. Popular open source software includes rattle, wajig, and mlhub. Hosted by Togaware, a pioneer of free and open source software since 1984. Copyright © 1995-2022 Graham.Williams@togaware.com Creative Commons Attribution-ShareAlike 4.0
