---
title: "Digital Humanities Use Case: Replication of analyses from *Text Analysis with R for Students of Literature*"
output:
  rmarkdown::html_document:
    theme: null
    css: mystyle.css
    toc: yes
vignette: >
  %\VignetteIndexEntry{Literature}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}
---

<!--
%\VignetteEngine{knitr::rmarkdown}
%\VignetteIndexEntry{Quickstart}
-->

In this vignette we show how the **quanteda** package can be used to replicate the analysis from Matthew Jockers' book *Text Analysis with R for Students of Literature* (London: Springer, 2014).  Most of the Jockers book consists of loading, transforming, and analyzing quantities derived from text and data from text.  Because **quanteda** has built in most of the code to perform these data transformations and analyses, it makes it possible to replicate the results from the book with far less code.

In what follows, each section corresponds to a chapter in the book.

# 1 R Basics

Our closest equivalent is simply:
```{r eval=FALSE}
install.packages("quanteda", dependencies = TRUE)
```

But if you are reading this vignette, than chances are that you have already completed this step.

# 2 First Foray

Moby Dick: Descriptive analysis

## 2.1 Loading the first text file

The code below scans and splits the text of Moby Dick from Project Gutenberg, as implemented in the text.  The command `textfile()` loads almost any file, including those found on the Internet (beginning with a URL, such as "http" or "https").
```{r eval=TRUE}
require(quanteda)

# read the text as a single file
# alternative:
# mobydickText <- texts(textfile("http://www.gutenberg.org/cache/epub/2701/pg2701.txt"))
summary(mobydickText)
```

The `textfile()` loads the text and places inside a structured, intermediate object known as a `corpusSource` object.  We see this by outputting it to the global environment, as above.

We can access the text from a `corpusSource` object (and also, as we will see, a `corpus` class object), using the `texts()` method.  Here we will display just the first 75 characters, to prevent a massive dump of the text of the entire novel.  We do this using the `substring()` function, which shows the 1st through the 75th characters of the texts of our new object `mobydicktf`.  Because we have not assigned the return from this command to any object, it invokes a print method for character objects, and is displayed on the screen.

```{r}
substring(mobydickText, 1, 75)
```

## 2.2 Separate content from metadata
The Gutenburg edition of the text contains some metadata before and after the text of the novel. The code below uses the `regexec` and `substring` functions to separate this from the text.
```{r}
# extract the header information
endMetadataIndex <- regexec("CHAPTER 1. Loomings.", mobydickText)[[1]]
metadata.v <- substring(mobydickText, 1, endMetadataIndex - 1)
```

To trim the extra text at the end of the Gutenburg version of the text, we can use the keyword-in-context (`kwic`) function to view the contexts around the word 'orphan', which we know should occur at the end of the book.

```{r}
# verify that "orphan" is the end of the novel
kwic(mobydickText, "orphan")

# extract the novel -- a better way
novel.v <- substring(mobydickText, endMetadataIndex, 
                     regexec("End of Project Gutenberg's Moby Dick.", mobydickText)[[1]]-1)
```


## 2.3 Reprocessing the content

We begin processing the text by converting to lower case. `quanteda`'s `toLower` function works like the built-in `tolower`, with an extra option to preserve upper-case acronyms when detected.

```{r}
# lowercase
novel.lower.v <- toLower(novel.v)
```

`quanteda`'s `tokenize` function splits the text into words, with many options available for which characters should be preserved, and which should be used to define word boundaries. The default behaviour works similarly to splitting on the regular expression for word boundary (`\W`), but does not treat apostrophes as word boundaries. This means that *'s* and *'t* are not treated as whole words from possessive forms and contractions. 

```{r}
# tokenize
moby.word.v <- tokenize(novel.lower.v, removePunct = TRUE, simplify = TRUE)
length(moby.word.v)
total.length <- length(moby.word.v)
str(moby.word.v)
moby.word.v[1:10]
moby.word.v[99986] 

moby.word.v[c(4,5,6)]

head(which(moby.word.v=="whale"))
```

## 2.4 Beginning the analysis

The code below uses the tokenized text to the occurrence of the word *whale*. To include the possessive form *whale's*, we may sum the counts of both forms, count the keyword-in-context matches by regular expression or glob[^1].  `quanteda`'s tokenize function separates punctuation into tokens by default. To match the counts in the book, we can choose to remove the punctuation.

[^1] A *glob* is a simple wildcard matching pattern common on Unix systems -- asterisks match zero or more characters.

```{r}
moby.word.v <- tokenize(novel.lower.v, simplify = TRUE)
# count of the word 'whale'
length(moby.word.v[which(moby.word.v == "whale")])

# total occurrences of 'whale' including possessive
length(moby.word.v[which(moby.word.v == "whale")]) + length(moby.word.v[which(moby.word.v == "whale's")])
# same thing using kwic()
nrow(kwic(novel.lower.v, "whale"))
nrow(kwic(novel.lower.v, "whale*")) # includes words like 'whalemen'
(total.whale.hits <- nrow(kwic(novel.lower.v, "^whale('s){0,1}$", valuetype = 'regex')))
```

What fraction of the total words in the novel are 'whale'?
```{r}
total.whale.hits / ntoken(novel.lower.v, removePunct=TRUE)  
```


Calculating the size of the vocabulary -- includes possessive forms.
```{r}
# total unique words
length(unique(moby.word.v))
ntype(toLower(novel.v), removePunct = TRUE)
```


To quickly sort the word types by their frequency, we can use the `dfm` command to create a matrix of counts of each word type -- a document-frequency matrix. In this case there is only one document, the entire book.
```{r eval=TRUE}
# ten most frequent words
mobyDfm <- dfm(novel.lower.v)
mobyDfm[, "whale"]

topfeatures(mobyDfm)
plot(topfeatures(mobyDfm, 100), log = "y", cex = .6, ylab = "Term frequency")
```


# 3 Accessing and Comparing Word Frequency Data

## 3.1 Accessing Word Data

We can query the document-frequency matrix to retrieve word frequencies, as with a normal matrix:
```{r eval=TRUE}
# frequencies of 'he' and 'she' - these are matrixes, not numerics
mobyDfm[, c("he", "she", "him", "her")]
mobyDfm[, "her"]
mobyDfm[, "him"]/mobyDfm[, "her"]
mobyDfm[, "he"]/mobyDfm[, "she"]
```

## 3.2 Recycling

```{r}
mobyDfmPct <- weight(mobyDfm, "relFreq") * 100
mobyDfmPct[, "the"]

plot(topfeatures(mobyDfmPct), type="b",
     xlab="Top Ten Words", ylab="Percentage of Full Text", xaxt ="n")
axis(1, 1:10, labels = names(topfeatures(mobyDfmPct)))
```


# 4 Token Distribution Analysis

## 4.1 Dispersion plots

A dispersion plot allows us to visualize the occurrences of particular terms throughout the text. The object returned by the `kwic` function can be plotted to display a dispersion plot.
```{r eval=TRUE, fig.width=8, fig.height=1.5}
# using words from tokenized corpus for dispersion
plot(kwic(novel.v, "whale"))
```

You can also pass multiple kwic objects to `plot` to compare the dispersion of different terms:
```{r eval=TRUE, fig.width=8, fig.height=2.5}
plot(
     kwic(novel.v, "whale"),
     kwic(novel.v, "Ahab"),
     kwic(novel.v, "Pequod")
)
```

## 4.2 Searching with `grep`

```{r eval = FALSE}
# identify the chapter break locations
(chap.positions.v <- kwic(novel.v, "CHAPTER \\d", valuetype = "regex")$position)
```


## Identifying chapter breaks

Splitting the text into chapters means that we will have a collection of documents, which makes this a good time to make a `corpus` object to hold the texts. Initially, we make a single-document corpus, and then use the `segment` function to split this by the string which specifies the chapter breaks. 

```{r}
head(kwic(novel.v, 'chapter'))
chaptersVec <-unlist(segment(novel.v, what='other', delimiter="CHAPTER\\s\\d", perl=TRUE))
chaptersLowerVec <- toLower(chaptersVec)
chaptersCorp <- corpus(chaptersVec)
```

With the corpus split into chapters, we can use the `dfm` command to create a matrix of counts of each word in each chapter -- a document-frequency matrix.

## Fig 4.4 barplots of whale and ahab
```{r eval=TRUE}
chapDfm <- dfm(chaptersCorp)
barplot(as.numeric(chapDfm[, 'whale']))
barplot(as.numeric(chapDfm[, 'ahab']))
```


The above plots are raw frequency plots. For relative frequency plots, (word count divided by the length of the chapter) we can weight the document-frequency matrix. To obtain expected word frequency per 100 words, we multiply by 100. To get a feel for what the resulting weighted dfm (document feature matrix) looks like, you can inspect it with the `head` function, which prints the first few rows and columns.

## Relative frequency barplots of whale and ahab
```{r eval=TRUE}
relDfm <- weight(chapDfm, type='relFreq') * 100
head(relDfm)
barplot(as.numeric(relDfm[, 'whale']))
barplot(as.numeric(relDfm[, 'ahab']))
```

# 5 Correlation

## 5.2 Correlation Analysis

The `dfm` function constructs a matrix which contains zeroes (rather than NAs) for words that do not occur in a chapter, so there's no need to manually convert NAs. We can compute the individual correlation or the correlation for a matrix of the two columns.

```{r}
wf <- as.numeric(relDfm[,'whale'])
af <- as.numeric(relDfm[,'ahab'])
cor(wf, af)

waDfm <- cbind(relDfm[,'whale'], relDfm[,'ahab'])
cor(as.matrix(waDfm))
```

With the ahab frequency and whale frequency vectors extracted from the dfm, it is easy to calculate the significance of the correlation.

##  5.4 Random Sampling
```{r}
samples <- replicate(1000, cor(sample(af), sample(wf)))

h <- hist(samples, breaks=100, col="grey",
xlab="Correlation Coefficient",
main="Histogram of Random Correlation Coefficients\n
with Normal Curve",
plot=T)
xfit <- seq(min(samples),max(samples),length=1000)
yfit <- dnorm(xfit,mean=mean(samples),sd=sd(samples))
yfit <- yfit*diff(h$mids[1:2])*length(samples)
lines(xfit, yfit, col="black", lwd=2)

cor.test(wf, af)

```


# 6 Measures of Lexical Variety

## 6.2 Mean word frequency
The mean word frequency for a particular chapter can be calculated simply with the dfm. Each row is a document (chapter), so, for example, the mean word frequency of the first chapter is the sum of the first row of the matrix, divided by the number of word types in the first chapter. To get the number of word types in the first chapter only, we can either exclude words in that row which have a frequency of zero, or use the `ntype` function on the first document in the corpus to achieve the same result.

```{r}
firstChap <- as.matrix(chapDfm[1,])
numWords <- length(firstChap[firstChap > 0])
sum(chapDfm[1,])/numWords
sum(chapDfm[1,])/ntype(chaptersCorp[1], removePunct=TRUE)
```

## 6.3 Extracting Word Usage Means

The `rowMeans` matrix function, which operates on a dfm, allows us to retreieve the means for all of the chapters.
```{r}
chapMeans <- Matrix::rowMeans(chapDfm)
plot(chapMeans, type="h")
```

