---
title: "Less Volume, More Creativity -- Getting Started with the mosaic Package"
author: "Randall Pruim"
date: '`r Sys.Date()`'
vignette: |
  %\VignetteIndexEntry{Less Volume, More Creativity} 
  %\VignetteEngine{knitr::rmarkdown} 
  \usepackage[utf8]{inputenc}
---

<style type="text/css">
span.boxed {
  border:5px solid gray;
  border-radius:10px;
  padding: 5px;
}
span.invboxed {
  border:5px solid gray;
  padding: 5px;
  border-radius:10px;
  color: white;
}
table, td, th
{
border:0px;
}
</style>

```{r, include=FALSE}
require(lubridate)
require(dplyr)
require(mosaic)
require(mosaicData)
trellis.par.set(theme=col.mosaic())
require(knitr)
opts_chunk$set(
  size='tiny', 
  tidy=FALSE,
  fig.width=6, 
  fig.height=3,
  fig.align="center",
  out.width="70%"
)
options(width=90)
```

## Project MOSAIC and the `mosaic` package

NSF-funded project to develop a new way to introduce mathematics, statistics, computation 
and modeling to students in colleges and universities.

 * more information at [mosaic-web.org](http://mosaic-web.org)
 
 * the `mosaic` package is available via 
     * [CRAN](http://cran.r-project.org/package=mosaic)
     * [github](https://github.com/ProjectMOSAIC/mosaic) 
         * updates more frequently than CRAN
         * use `devtools::install_github(ProjectMOSAIC/mosaic)` to install 
         directly from github
         * add the optional `ref="beta"` to install from the beta (developmental) 
         branch
         * add `build_vignettes=TRUE` if your system is equipped to build vignettes.
     
### A note about this document  

This document was originally created as an R presentation to be used as slides
accompanying various presentations.  It has been converted into a more traditional
document for use as a vignette in the `mosaic` package.

The examples below use the `mosaic` and `mosaicData` packages.  

```{r}
require(mosaic)
require(mosaicData)
```

## Less Volume, More Creativity

Many of the guiding principles of the `mosaic` package reflect the 
"Less Volume, More Creativity" mantra of Mike McCarthy who had a large
poster with those words placed in the "war room" (where assistant coaches
decide on the game plan for the upcoming opponent) as a constant reminder
not to add too much complexity to the game plan.


<table><tr align="top"><td width="20%" align="top">
<img src="images/MikeMcCarthy.jpg" align="top" width="200"></td>
<td align="top"> A lot of times you end up putting in a lot more volume, because you are teaching fundamentals  and you are teaching concepts that you need to put in, but you may not necessarily use because they are building blocks for other concepts and variations that will come off of that ... In the offseason you have a chance to take a step back and tailor it more specifically towards your  team and towards your players." 
<br><br>
Mike McCarthy, Head Coach, Green Bay Packers 
</td>
</tr>
</table>

                                          
### SIBKIS: See It Big, Keep It Simple

Here is another elegant phrasing of a similar principle.
<table>
<tr>
<td>
Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. 
<br><br>
--- Antoine de Saint-Exupery (writer, poet, pioneering aviator)
</td>
<td width="20%"> 
<img src="images/SaintExupery.jpg">
</tr>
</table>

## Less Volume, More Creativity in R

One key to successfully introducing R is finding a set of commands that is

 * **small**: fewer is better
 * **coherent**:  commands should be as similar as possible
 * **powerful**: can do what needs doing 
 
It is not enough to use R, it must be used elegantly.

Two examples of this principle:
 * the **mosaic** package
 * the **dplyr** package (Hadley Wickham)

### Minimal R

**Goal:** a minimal set of R commands for Intro Stats

**Result:** Minimal R Vignette (`vignette("MinimalR")`)

Much of the work on the `mosaic` package has been motivated
by 

* The Less Volume, More Creativity approach
* The Minimal R goal

### A few little details

If you (or your students) are just getting started with R, it is good to 
keep the following in mind:

#### R is case sensitive
 * many students are *not* case sensitive

#### Arrows and Tab
 * up/down arrows scroll through history
 * TAB completion can simplify typing

#### If all else fails, try ESC
 * If you see a + prompt, it means R is waiting for more input
 * If this is unintentional, you probably have a typo
 * ESC will get you pack to the command prompt

## The Most Important Template
&nbsp;

The following template is important because we can do so much with it.

&nbsp;

<center>
<h2><strong><span class="invboxed">goal</span> ( <span class="invboxed">yyy</span> ~ <span class="invboxed">xxx</span> , data = <span class="invboxed">mydata</span> )</strong> 
</h2></center>

&nbsp;

It is useful to name the components of the template:

&nbsp;

<center>
<h2><strong><span class="boxed">goal</span> ( <span class="boxed">&nbsp;y&nbsp;</span> ~ <span class="boxed">&nbsp;x&nbsp;</span> , data = <span class="boxed">mydata</span> )</strong> 
</h2></center>

 
&nbsp;
We're hiding a bit of complexity in the template, and there will be times 
that we will want to gussy things up a bit.  We'll indicate that by adding
`...` to the end of the template.  Just don't let `...` become a distractor
early on.

&nbsp;

<center>
<h2><strong><span class="boxed">goal</span> ( <span class="boxed">&nbsp;y&nbsp;</span> ~ <span class="boxed">&nbsp;x&nbsp;</span> , data = <span class="boxed">mydata</span> , ...)</strong> 
</h2></center>
&nbsp;

### Other versions
Here are some variations on the template.
```{r, eval = FALSE}
# simpler version
goal( ~ x, data = mydata )          
# fancier version
goal( y ~ x | z , data = mydata )   
# unified version
goal( formula , data = mydata )     
```

## 2 Questions

Using the template generally requires answering two questions.  (These questions
are useful in the context of nearly all computer tools, just substitute "the computer" in for R in the questions.)

&nbsp;
<center>
<h2><strong><span class="boxed">goal</span> ( <span class="boxed">&nbsp;y&nbsp;</span> ~ <span class="boxed">&nbsp;x&nbsp;</span> , data = <span class="boxed">mydata</span> )</strong> 
</h2></center>
 
&nbsp;


###  What do you want R to do?  (goal)
 * This determines the R function to use 


###  What must R know to do that?  
 * This determines the inputs to the function
 * Must identify the variables and data frame

## How do we make this plot? (Questions)
```{r, echo=FALSE, out.width="60%", out.height="35%"}
xyplot( births ~ date, data=Births78) 
```

### What is the Goal?

 * 

### What does R need to know?

 * 
 * 

### How do we make this plot? (Answers)

#### What is the Goal?

* a scatter plot (`xyplot()`)

#### What does R need to know?

 * which variable goes where (`births ~ date`)
 * which data set (`data=Births78`)
    * use `?Births78` for documentation

#### Putting it all together

```{r, echo=TRUE, out.width="60%"}
xyplot( births ~ date, data=Births78) 
```

### Your turn: How do you make this plot?


```{r, echo=FALSE}
bwplot( age ~ substance, data=HELPrct, xlab="substance" )
```

Some things you will need to know:

1. Command: <code>bwplot()</code>

2. The data: <code>HELPrct</code>
 * Variables: <code>age</code>, <code>substance</code>
 * use `?HELPrct` for info about data

#### Answer

```{r, echo=TRUE}
bwplot( age ~ substance, data=HELPrct)
```

### Your turn: How about this one?

```{r, echo=FALSE}
bwplot( substance ~ age, data=HELPrct)
```

Some things you will need to know:

1. Command: <code>bwplot()</code>

2. The data: <code>HELPrct</code>
 * Variables: <code>age</code>, <code>substance</code>
 * use `?HELPrct` for info about data

#### Answer

```{r, echo=TRUE}
bwplot( substance ~ age, data=HELPrct )
```

Note that we have reversed which variable is mapped to the x-axis and which
to the y-axis by reversing their order in the formula.

### Graphical Summaries: One Variable
```{r, }
histogram( ~ age, data=HELPrct) 
```

Note: When there is **one variable** it is on 
the **right side** of the formula.

### Graphical Summaries: Overview

#### One Variable
```{r, eval=FALSE, tidy=FALSE}
  histogram( ~age, data=HELPrct ) 
densityplot( ~age, data=HELPrct ) 
     bwplot( ~age, data=HELPrct ) 
     qqmath( ~age, data=HELPrct ) 
freqpolygon( ~age, data=HELPrct ) 
   bargraph( ~sex, data=HELPrct )
```

#### Two Variables

```{r, eval=FALSE, tidy=FALSE}
xyplot(  i1 ~ age,       data=HELPrct ) 
bwplot( age ~ substance, data=HELPrct ) 
bwplot( substance ~ age, data=HELPrct ) 
```

**Note:** `i1` is the average number of drinks (standard units) 
consumed per day in the past 30 days (measured at baseline)


### The Graphics Template

#### One variable
<center>
<h3><strong><span class="boxed">plotname</span> ( 
~ <span class="boxed">&nbsp;x&nbsp;</span> , data = <span class="boxed">mydata</span> , ...)</strong> 
</h3></center>

 * `histogram()`,  `qqmath()`,  `densityplot()`, `freqpolygon()`, `bargraph()` 

&nbsp;


#### Two Variables
<center>
<h3><strong><span class="boxed">plotname</span> ( <span class="boxed">&nbsp;y&nbsp;</span> ~ <span class="boxed">&nbsp;x&nbsp;</span> , data = <span class="boxed">mydata</span> , ...)</strong> 
</h3></center>


 * `xyplot()`, `bwplot()`

### Your turn
Create a plot of your own choosing with one of these data sets
```{r, eval=FALSE}
names(KidsFeet)    # 4th graders' feet
?KidsFeet
```
```{r, eval=FALSE}
names(Utilities)   # utility bill data
?Utilities
```
```{r, eval=FALSE}
require(NHANES)    # load package
names(NHANES)      # body shape, etc. 
?NHANES
```


### groups and panels

 * Add `groups = `<span class="invboxed">group</span> to overlay.
 
 * Use `y ~ x | z` to create multipanel plots.

Here is an example.

```{r, tidy=FALSE}
densityplot( ~ age | sex, data=HELPrct,  
               groups=substance,  
               auto.key=TRUE)   
```

Beginners can create plots with 3 or 4 variables easily and quickly using
this template.

Bells & Whistles
================

The `lattice` graphics system includes lots of bells and whistles including

 * titles
 * axis labels
 * colors
 * sizes
 * transparency
 * etc, etc.
 
I used to introduce these too early.  My current approach: 

 * Let the students ask or 
 * Let the data analysis drive 

#### An example with some bells and whistles

```{r, out.width="85%", tidy=FALSE}
require(lubridate)
xyplot( births ~ date, data=Births78,  
  groups=wday(date, label=TRUE, abbr=TRUE), 
  type='l',
  auto.key=list(columns=4, lines=TRUE, points=FALSE),
  par.settings=list(
    superpose.line=list( lty=1 ) ))
```
```{r, eval = FALSE, include = FALSE}
xyplot( births ~ date, data=Births78,  
        groups=wday(date, label=TRUE, abbr=TRUE), type='l',
        auto.key=list(columns=4),
        superpose.symbol=list(
            pch=16, cex=1.2, alpha=.8)))
```

**Notes** 

 * `wday()` is in the `lubridate` package
 * This version of the plot reveals a clear weekend (and holiday) pattern.  Typically, I like to have students conjecture about the "double wave" pattern and see if we can build plots to test their conjectures.

## Numerical Summaries

The `mosaic` package provides functions that make it simple to create
numerical summaries using the same template used for graphing (and later for 
describing linear models).

### Numerical Summaries: One Variable

Big idea: 

 * Replace plot name with summary name
 * Nothing else changes
 
```{r, fig.show='hold'}
histogram( ~ age, data=HELPrct )  # width=5 (or 10) might be good here
     mean( ~ age, data=HELPrct )
```

#### Other summaries
The mosaic package includes formula aware versions of
`mean()`,
`sd()`,
`var()`,
`min()`,
`max()`,
`sum()`,
`IQR()`, ...

Also provides `favstats()` to compute our favorites.
```{r}
favstats( ~ age, data=HELPrct )
```

`favstats()` quickly becomes a go-to function in our courses.

### Tallying 

```{r}
tally( ~ sex, data=HELPrct)
tally( ~ substance, data=HELPrct)
```

### Numerical Summaries: Two Variables

There are three ways to think about this.  All do the same thing.
```{r, eval = FALSE}
sd(   age ~ substance, data=HELPrct )
sd( ~ age | substance, data=HELPrct )
sd( ~ age, groups=substance, data=HELPrct )
```
```{r, echo=FALSE}
sd( ~ age, groups=substance, data=HELPrct )
```

This makes it possible to easily convert three different types of plots
into the (same) corresponding numerical summary.

### Numerical Summaries: Tables

2-way tables are just tallies of 2 variables.

```{r}
tally( sex ~ substance, data=HELPrct )
tally( ~ sex + substance, data=HELPrct )
```

Other output formats are available
```{r}
tally( sex ~ substance,   data=HELPrct, format="proportion" )
tally( substance ~ sex,   data=HELPrct, format="proportion", margins=TRUE )
tally( ~ sex + substance, data=HELPrct, format="proportion", margins=TRUE )
tally( sex ~ substance,   data=HELPrct, format="percent" )
```

#### More examples
```{r, echo=FALSE}
HELPrct <- transform(HELPrct, sex=factor(sex, labels=c('F','M')),
                     substance = factor(substance, labels=c('A', 'C', 'H')))
```
```{r, size='small'}
mean( age ~ substance | sex, data=HELPrct )
mean( age ~ substance | sex, data=HELPrct, .format="table" )
```

```{r, echo=FALSE}
rm(HELPrct)
data(HELPrct)
```
 * I've abbreviated some labels to make things fit better.  You can do this using `mutate()` (in the `dplyr` package) or `transform()`.
 * This also works for `median()`, `min()`, `max()`, `sd()`, `var()`, `favstats()`, etc.

### One Template to Rule a Lot

This master template can be used to do a large portion of what needs 
doing in an Intro Stats course.

 * single and multiple variable graphical summaries
 * single and multiple variable numerical summaries
 * linear models
 
```{r, eval=FALSE}
  mean( age ~ sex, data=HELPrct )
bwplot( age ~ sex, data=HELPrct ) 
    lm( age ~ sex, data=HELPrct )
```

```{r, echo=FALSE}
  mean( age ~ sex, data=HELPrct )
    coef(lm( age ~ sex, data=HELPrct ))
```

It can be learned early and practiced often so that students become 
secure in their ability to use these functions.

## Some other things
The `mosaic` package includes some other things, too

 * data sets (they have now been moved to separate `mosaicData` and `NHANES` packages)
 * xtras: `xchisq.test()`, `xpnorm()`, `xqqmath()`
   * these functions add a bit of extra output to the similarly named functions that don't have a leading `x`
 * `mplot()` 
   * `mplot(HELPrct)` interactive plot creation
   * replacements for `plot()` in some situations
 * simplified `histogram()` controls (e.g., `width`)
 * simplified ways to add onto lattice plots (`add = TRUE` works in many situations)

### Examples

```{r}
xpnorm( 700, mean=500, sd=100)
```

```{r}
xpnorm( c(300, 700), mean=500, sd=100)
```

```{r, echo=FALSE}
phs <- cbind(c(104,189),c(10933,10845))
colnames(phs) <- c("heart attack","no heart attack")
rownames(phs) <- c("aspirin","placebo")
```

```{r}
xchisq.test(phs)
```

## Modeling 
Modeling is really the starting point for the `mosaic` 
design.

 * linear models (`lm()` and `glm()`) defined the template
 * `lattice` graphics use the template (so we chose `lattice`)
 * we added functionality so numerical summaries can be done with the same template
 * additional things added to make modeling easier for beginners

### Models as Functions
```{r}
model <- lm(width ~ length * sex, 
            data=KidsFeet)
Width <- makeFun(model)
Width( length=25, sex="B")
Width( length=25, sex="G")
```

Once models have been converted into functions, we can easily add them
to out plots using `plotFun()`.

```{r, include=FALSE}
trellis.par.set(
  superpose.symbol=list(col=c('navy','red'), pch=16), 
  superpose.line=list(lty=1, col=c('navy','red'))
)
```
                                                       
```{r, fig.keep='last'}
xyplot( width ~ length, data=KidsFeet, 
        groups=sex, auto.key=TRUE )
plotFun( Width(length, sex="B") ~ length, 
         col=1, add=TRUE)
plotFun( Width(length, sex="G") ~ length, 
         col=2, add=TRUE)
```

```{r, include=FALSE}
trellis.par.set(theme=col.mosaic())
```

## Resampling -- You can do() it!

If you want to teach using randomization tests and bootstrap intervals,
the `mosaic` package provides some functions to simplify creating the
random distirubtions involved.

### An example: The Lady Tasting Tea

 * Often used on first day of class
 
 * Story
 
   * woman claims she can tell whether milk
   has been poured into tea or vice versa.
   
   * Question: How do we test this claim?
   
```{r, echo=FALSE }
require(mosaic)
trellis.par.set(theme=col.mosaic())
require(knitr)
opts_chunk$set(size='small', cache=TRUE)
options(width=90)
set.seed(12345)
```


We use `rflip()` to simulate flipping coins
```{r}
rflip()
```

**Note:** We do this with students who do not (yet) know what a binomial
distribution is, so we want to avoid using `rbinom()` at this point.

Rather than flip each coin separately, we can flip multiple coins at once.

```{r} 
rflip(10)
```

 * easier to consider `heads` = correct; `tails` = incorrect than to compare with a given pattern
   * this switch bothers me more than it bothers my students

### Now let's do that a lot of times 

`rflip(10)` simulates 1 lady tasting 10 cups 1 time.

We can do that many times to see how guessing ladies do:
```{r}
do(2) * rflip(10)
```

 * `do()` is clever about what it remembers (in many common situations)
 * 2 isn't many -- we'll do many next -- but it is a good idea to take a look at a small example before generating a lot of random data.

Now let's simulate 5000 guessing ladies

```{r, ladies5000}
Ladies <- do(5000) * rflip(10)
head(Ladies, 1)
histogram( ~ heads, data=Ladies, width=1 )
```

**Q.** How often does guessing score 9 or 10?

Here are 3 ways to find out

```{r}
tally( ~(heads >= 9), data=Ladies)
tally( ~(heads >= 9), data=Ladies, format="prop")
 prop( ~(heads >= 9), data=Ladies)
```

### A general approach to randomization

The Lady Tasting Tea illustrates a 3-step process that can be reused in many situations:

 1. Do it for your data
 2. Do it for "random" data
 3. Do it lots of times for "random" data
 
 * definition of "random" is important, but can often be handled by the `mosaic` functions `shuffle()` or `resample()`

### Example: Do mean ages differ by sex?
```{r}
diffmean(age ~ sex, data=HELPrct)
do(1) * 
  diffmean(age ~ shuffle(sex), data=HELPrct)
Null <- do(5000) * 
  diffmean(age ~ shuffle(sex), data=HELPrct)
```

```{r}
prop( ~(abs(diffmean) > 0.7841), data=Null )
histogram(~ diffmean, data=Null, v=-.7841) 
```

### Example:  Bootstrap CI for difference in means

```{r}
Bootstrap <- do(5000) * 
  diffmean(age~sex, data= resample(HELPrct))

histogram( ~diffmean, data=Bootstrap, 
                      v=-.7841, glwd=4 )
```

```{r}
cdata(~diffmean, data=Bootstrap, p=.95)
confint(Bootstrap, method="quantile")
confint(Bootstrap)  # default uses bootstrap st. err.
```

### Randomization and linear models

```{r, size="tiny"}
do(1) * lm(width ~ length, data=KidsFeet)
do(3) * lm( width ~ shuffle(length), data=KidsFeet)
```


```{r, size="tiny"}
do(1) * 
  lm(width ~ length + sex, data=KidsFeet)
do(3) * 
  lm( width ~ length + shuffle(sex), data=KidsFeet)
```


```{r}
Null <- do(5000) * 
  lm( width ~ length + shuffle(sex), 
                       data=KidsFeet)
histogram( ~ sexG, data=Null, 
           v=-0.2325, glwd=4)
```

```{r}
histogram(~sexG, data=Null, 
           v=-0.2325, glwd=4)
prop(~ (sexG <= -0.2325), data=Null)
```

