---
author: "Sean Laverty"
title: "Class 5 notes"
date: "`r Sys.Date()`"
format: html
output-dir: out
editor: source
---

# Data Visualization and Exploration

## Unleashed

Read in one or more of the data sets and start using commands from class to explore.

## Cautious first steps

The dataset `warpbreaks` is hopefully familiar by now. Here we don't even have to read anything in, but might was well leave that as a reminder. It doesn't take up much space.^[I have a suspicion that `tab-` is a useful code chunk label for referencing, but I never really care about showing this so the labeling is secondary for me. I'm almost 100% sure I'm wrong. Some things never change.]
```{r}
#| label: wool-tab
#dat <- read.delim("./data/cards.csv", header = TRUE, sep = ',')
tab <- table(warpbreaks$wool)
tab2 <- table(warpbreaks[, c("wool", "tension")])
tab2 <- as.data.frame(tab2)
```
There is a difference between the objects created in the last two lines above. They contain exactly the same information, and the first is probably easier to understand. But the structure of the second, _where all combinations are repeated as rows_ is a _*much*_ better habit to build. You will thank yourself later.^[You may also bounce to the fancier tools in a couple of weeks, but, oh well, at least you'll have some perspective. "Oh, that was terrible," is ok sometimes.] 

Sometimes it is useful to return some numerical summary for one variable across some combination of categories of related variables^[Things could go off the rails here in terms of a philosophical thing that I am too tired for.]. Below we will calculate the mean of a numerical measurement among members of collections of potentially related groups. You could list as many things as you wanted, by name^[As a weird aside, it is quite tricky to do this one by column numbers without some clumsy side-stepping.]. The order just forces the order of columns, so you should pay close attention if columns have similar looking entries and one could be confused for another.

Below we'll call the first one `agg` to pay homage to the   `aggregate()` function. Then, `agg2` could represent the 2D table, or more simply the second in a quick series of calculations where you want to take small steps.  Trying to write the second of these as a data frame results in no change.

```{r}
#| label: aggregate
agg <- aggregate(warpbreaks$breaks, by = list(warpbreaks$wool), mean)
agg2 <- aggregate(warpbreaks$breaks, by = list(warpbreaks$wool, warpbreaks$tension), mean)
## no effect of the line below
## already a data frame
agg2.dat <- as.data.frame(agg2)
```

In reality the approach below is what I should have been doing all these years and I am rapidly trying to incorporate it into my work. That said, however you get started, you'll be okay.

```{r}
#| label: aggregate-notation
agg2b <- aggregate(breaks ~ wool + tension,  warpbreaks, mean)
```

To make this idea a little more useful, we can start to build little summary tables that are nicely formatted.

You can do this mostly by copy-and-paste. There's really no harm in that. There are fancier tools and ways to come, but it's really only a matter of what a couple of characters are and where you put them in the code. This is speedy enough and is quite transparent.  We calculate all relevant cominations of statistics of interest, there could be more and we'd just repeat versions of the last two lines in sequence^[This is a time to be creative. There was a time where I was very excited about _Big Data_, but it's always the case that despite best interests, people always seem to have _small data_ and quirks of them all are worth the individual attention.]

```{r}
#| label: merge-notations
agg2b.mn <- aggregate(breaks ~ wool + tension,  warpbreaks, mean)
agg2b.sd <- aggregate(breaks ~ wool + tension,  warpbreaks, sd)
agg2b.mn <- merge(agg2b.mn, agg2b.sd, by = c("wool", "tension"))
```

You can simply repeat the last two of those lines for as many applied functions as you want. You could get quite creative^[Homework.]. If you even happened to have a larger object with the same classifiers, or even some combination^[`... all.x = <T/F>...`, `'...all.y=<T/F>...` or some combination], and you can merge and maintain about anything.  It's really quite remarkable. When you'are all done with that, you just print the table to a file. There usually seem to be nothing harmful about setting `row.names = TRUE`, sometimes ugly rownames accumulate through an inefficient cycle of `merge()` and after `aggregate()` or after loadging the results of another dataset. Sometimes it's nice to just say no.

```{r}
#| label: file-output
write.table(agg2b.mn, "warpbreaks-summary.csv", sep = ",", col.names = TRUE, row.names = FALSE)
```

Suppose out of habit I'd defined  `agg2` as above,  I could get to the same point with the following code. Does this really matter? No. But its often nice to have a bunch of flexible ways to do the same thing. It makes you versatile. I grew up watching the original MacGyver^[It had its faults, but generally it did a lot of good.]. Remember that `agg2` was the version with the obscure variable names, to be clear, `Group.1`, `Group.2`, and `x`.  The reminder here is that `merge()` is flexible. 

It's always nice to be able to rename things.  In the event that to-be-merged variable names match, they'll be dotted `.x` and `.y` accordingly, so it's nice so to have a way to rename things regardless of how you approach certain steps. You can also rename any subset of variables `[a:b]` or even with reordering^[Eek.] some subset (e.g., `c(1, 9, 4)` chosen by name or number). Please be careful, it might not give an error, but it might not make much sense either.   

```{r}
agg2.mn <- merge(agg2, agg2b.sd, by.x = c("Group.1", "Group.2"), by.y = c("wool", "tension"))
names(agg2.mn) <- c("wool", "tension", "mean", "sd")
```

I thought I could avoid `iris` but I couldn't. In some circumstance for a quick calculation it was relatively helpful to compute and nicely organize and summary statistic of a few variables at a time across one category. Bebow the category is referenced as a name, but it could just as easily be done with the dollar sign (`iris$Species`) or the number (ugh, `iris[5, ]`, I like the wordy variables first). This feels _off_ to me, just by looking at it. I like keeping columns with different types of entries organized in groups - categorical or text-based first and numerical values after.

```{r}
head(iris)
aggregate(iris[ , 1:4], by = list(iris[, "Species"]), var)
```

# GROUP PROJECT TIME

To start wtih graphing, here is a reminder of using a `par()` argument for changing margins.  This can be useful if you have long axis tick mark labels. As always, `c(b, l, t, r)`.

```{r}
par(mar = c(8.1, 4.1, 0.8, 0.8))
# plot(...)
```

Here goes. Experiment with and interpret the results of each of the following.  You may have to execute them one at a time and either run as  `(tab <- table(warpbreaks$wool))`, the parenthesis are a clever way to force printing after assignment, without having to call the result by name in the next line. But, this always confuses me in Quarto because it only prints the last text-based object of a code block (unless it has graphs).

```{r}
#| label: 🏷️warpbreaks-stuff
tab <- table(warpbreaks$wool)
tab
tab2 <- table(warpbreaks[, c("tension", "wool")])
tab2 <- table(warpbreaks[, c("breaks", "wool")])
tab2 <- table(warpbreaks[, c("tension", "wool")])
tab2 <- table(warpbreaks[, c("wool", "tension")])
```

Those tables are each nice, but we can often make them more useful by turning the combinations into rows in a list instead of cells in a more complicated interaction.  To me this is a _Pro-Tip_. In particular you can use this to generate as a piece of data to be merged with something else. _Hint: it is very nice for checking combinations in dates from field notes to see if you didn't lose or skip a page when digitizing._

```{r}
tab2 <- as.data.frame(tab2)
```

Since someone asked, if merging objects throughout a file, make sure they match. If you've modified naming conventions you can use `by.x` and `by.y` to specify whatever variable names you need for matching. If one has things the other doesn't, you can use `all.x = TRUE` (same for `all.y`) or `all` (both) to control how many `NA` entries might clutter your screen. Experiment with this merging two different, but related, datasets that contain one or more common identifiers.  The `dim()` command is useful to learn about the size of the objects you are creating through merging.

```{r}
agg2 <- merge(agg2, agg2b.sd, by.x = c("Group.1", "Group.2"), by.y = c("wool", "tension"))
```

We have a bunch of similar calculations laying around and have to be careful about the names.  Weve used the `.` ('dot') and you could use camelCapping, but you can't use `-` ('dash') because it is interpreted as subtraction, and probably shouldn't use the `_` ('underscore') because it is ugly. Once again we can rename some or all of the names of a data frame at any time.

```{r}
names(agg2.mn) <- c("wool", "tension", "mean", "sd")
```

Suppose that was to polish off a dataset for printing. We can use either line below to print. If we happen to have row names that we want, or do not, here might be our chance to make changes.  In practice, I can only think of a time or two that this really mattered and it was usually only a matter of the difference of a couple of `sed ...` commands in the Terminal or Command Prompt.
```{r}
write.table(agg2.mn, "warpbreaks-summary.csv", sep = ",", col.names = TRUE, row.names = FALSE)
write.table(agg2.mn, "warpbreaks-summary.csv", sep = ",", col.names = TRUE, row.names = TRUE)
```

## Iris^[The dig was at myself. I'd made a bet that I could get by without saying the name `iris`.]

I'd hoped to avoid using this data set, `iris`, but some times, some things are just to perfect. There are reasons everybody uses this data set, for one its nice to stop and thing about flowers every so often.  Below we verify the structure of `iris` then conmite a variety of summary tables. These could all be saved or merged. If you were really fancy you could store a bunch of functions in a vector, put that in a loop, and proably automate that.

```{r}
head(iris)
aggregate(iris[ , 1:4], by = list(iris[, "Species"]), median)
aggregate(iris[ , 1:4], by = list(iris[, "Species"]), sd)
aggregate(iris[ , 1:4], by = list(iris[, "Species"]), length)
aggregate(iris[ , 1:4], by = list(iris[, "Species"]), var)
```

## Convenience
In a question of convenience, it was asked how to more easily see, and store, the result of a calculation.  I guess I should have said write `name <- ...; name`. But again outside of Quarto, using `( ... <- ...)` works well.  Since earlier we turned one of these tables into a data frame, I tried that here, and it didn't matter because the original output was about as simplified as it could be.

```{r}
agg <- aggregate(warpbreaks$breaks, by = list(warpbreaks$wool), mean)
(agg <- aggregate(warpbreaks$breaks, by = list(warpbreaks$wool), mean))
## no effect of the line below
## already a data frame
(agg2.dat <- as.data.frame(agg2))
```

## The Mouse-Elephant curve

The primary challenge here is that the two numeric variables span a massive range. It would be great here to graph with a log scale.  Time could be spent using `text()` with text positions calculated from rows in the data to annotate key points (i.e., animal names) of interest.  Just for a reminder, the first line won't work because the separation character needs to be specified.  There are a variety of `read.*()` functions, so find your favorite.

While on the topic, there was a question about pasting strings, possibly for annotation.  The goal was neat, but too specific for the moment.  Just to toss out a suggestion for now, I would read `?plotmath`, if that documentation exists, and experiment with every possible permutation of `paste()`, `expression()`, `bquote()`, and possibly `parse()`, or look for a suggested response to a related question on <https://stackexchange.com> and pick the one with the most of those words.


```{r}
#mouse <- read.delim("./data/mouse.csv")
mouse <- read.delim("./data/mouse.csv", sep = ',')
paste(mouse$Animal, mouse$Mass)
```



## Diabetes data

The `diabetes` data from <https://github.com/datasets/diagnosed-diabetes-prevalence> can be approached below. The file is in the corresponding class folder.

```{r}
diabetes <- read.delim("./data/diabetes.csv", sep = ',')
```

After using `names()`, `head()`, `tail()`, `summary()`, or even on a whim `plot()`, it's time to make tables. After that initial exploration, we know that columns are `"State"` and `"Year"`.  We can make a table of the combinations of these that occur within the dataset. The results are rather interesting. Perhaps we should have known, but sometimes you forget, and sometimes there are some hidden assumptions within the structure of a dataset that you can easily overlook.

```{r}
table(diabetes[, c("State", "Year")])
diabetes2 <- as.data.frame(table(diabetes[, c("State", "Year")]))
```
The outputs from the above lines (only the first one has been printed) are both equally useful in different ways.  I don't hate either. The one thing that they seemed to show is that at some point, one Alaskan City Council member (or the math professor that downloaded the file made a typo).

In all of this, it became clear that there was that typo.  We can find it.
```{r}
diabetes[diabetes$State == "\\Alaska", ]
```

Remember that the indexing of a data frame is `[rows, columns]`, the rows and columns can be concatenated numbers, variable names, or sequences of numbers.  Mostly I find myself doing logic to extract subsets of rows, and _*carefully_* making slight adjustments to the order of columns at times. Below we inspect some incorectly-written entries of `"\\Alaska"`, quite honestly I wondered if I accidentally added that while downloading the file.

```{r}
diabetes[diabetes$State == "\\Alaska", "State"]
```

We can simultaneously overwrite those entries with the corrected encoding. I'm sure there are a lot of fancier tools, but I've learned a lot of hard lessons, faster this way and avoided negative consequequences of sloppy work. After overwriting those entries, we can test and see that nothing matches the original entry.

```{r}
diabetes[diabetes$State == "\\Alaska", "State"] <- "Alaska"
diabetes[diabetes$State == "\\Alaska", ]
```

With that fixed, we'll show the head of the data where `State` is California and `Year` is 2004. There was no specific reason for these choices, any other combination of place and time would have been sufficient.  We will first look at all columns, then just the `"Number"` column.

```{r}
head(diabetes[diabetes$State == "California" & diabetes$Year == 2004, ])
diabetes[diabetes$State == "California" & diabetes$Year == 2004, "Number"]
```

With that worth of thing fixed up, we can get on with the exploration.  We might try the aggregate by `State` and `Year` to find the mean `Number` of cases. Trying other functions for `mean`, including `length` which lets you verify totals (that's mostly what made me see the data entry error earlier). 

Below we'll just show the head as well, because otherwise this gets pretty long. If you were to save this for future use, you would want to be _very_ careful to remember to remove the `head()`, otherwise you woud _*only save*_ those first few rows.  Notice that I've gone back to old habits and we've lost the original variable names in the `aggregation()` step.

```{r}
head(aggregate(diabetes$Number, by = list(diabetes$State, diabetes$Year), mean))
```


## NIH funding data

This contains projected cuts across the States under projected NIH funding cuts^[I'll have to check the source.].

```{r}
dat <- read.delim("./data/nih-funding-state.csv", sep = ',')
head(dat)
```

It might be interesting to explore the connections between amounts and think of calculating measures like relative amounts.
```{r}
dat[, 3] - dat[, 2]
dat[, 3]/dat[, 2]
plot(dat[, 2], dat[, 3]/dat[, 2])
```

## Miscellaneous graphing

After just a couple mistakes (not shown, e.g., `col.text`) this works to customize axes. The goal would almost never be to do this, but instead this was meant to show which argument highlighted which feature. Feel free to slap either of these on a graph.
`axis(1, col = "red", col.ticks = "yellow")``, or
``axis(1, col = "red", col.ticks = "yellow", col.axis = "blue")``.

## Corruption Index

None of this code works. Some of it generates output, some of it warnings, some of it errors. This is one of my least favorite things. We all have those. The goal was to reshape "wide" data into "long" data where we had columns for `"Country"`, `"Year"`, and `"CPI"`. It's somewhat offputting that the `X` is prepended to the year as a column name. Variable names can end, but not start, with numbers.

```{r}
# dat2 <- read.delim("./data/cpi.csv")
# head(dat2)
# # oops sep = ','
# dat2 <- read.delim("./data/cpi.csv", sep = ',')
# head(dat2)
# 
# reshape(dat2, varying = 2:ncol(dat2))
# reshape(dat2, varying = 2:ncol(dat2), direction = "long")
# ...
# redacted
# ...
```


## Olympic athlete data

This excellent data comes from GitHub^[<https://github.com/datasets/rio2016>.].  Below we will make a table of values of each `sport` entry.  First is a `plot()` of a `table()`, then a `barplot()`, then finally the same barplot with rotated axes. These are all fine but could easily be made better.

```{r}
#| label: fig-bar-line
#| fig.width: 5
#| fig.height: 5
athletes <- read.delim("./data/athletes.csv", sep = ",")
head(athletes)
tab <- table(athletes$sport)
par(mfrow = c(3, 1))
barplot(tab)
plot(tab)
plot(tab, las = 3)
```
The table entries can be ordered (in either order), and the table replotted in a potentially nicer order. This would almost certainly break apart any sort of regional or alphabetical ordering (unless, "Which state is closest to the front of the alphabet?"). This rather quickly makes an excellent summary.  It could be touched up a bit, two orientations and constructions are shown in @fig-bar-line2.


```{r}
#| label: fig-bar-line2
#| #| fig.width: 5
#| fig.height: 5
tab <- tab[order(tab)]
par(mfrow = c(1, 2))
barplot(tab, las = 1, horiz = TRUE)

plot(tab, las = 3)
```

We've earlier looked at tasks like subsetting data, to create a data frame, you could estract the rows matching a condition by the following. This leads into a very interesting related task. Read ahead.

To save all columns of data for which the entries of `athletes$nationality` match `"KOR"` leave the second position empty. It would be nice to save this in a scheme similar to what you're already doing, for me `dat` leads to `dat.kor` (if you felt like being annoying you could capitalize to `dat.KOR`, but if you capitalize that, why stop ? Just scream all of the time.).

```{r}
dat.kor <- athletes[athletes$nationality == "KOR", ]
```

If you wanted to make a plot you could pull that subset in the plot.
```{r}
#| eval: false
plot(output ~ input, athletes, subset = nationality == "KOR")
```

Either of those could be done with multiple logic conditions.  By specifiying a column or more using any of those methods, you could extract any subset of columns at the step creating `dat.kor`.

On the topic of selecting certain subsets, there are some clever approaches below. We did this a while back, but remember `tab`?

```{r}
tab <- tab[order(tab)]; tab
```

Suppose we wanted the top ten sports by "abundance".  Looking at our table, in particular its dimensions, this is the entries `19:28` (in front of people you should always check your math `19:28` is of length `r length(19:28)`). 

```{r}
tab[19:28]
names(tab[19:28])
```

In the second line we pull off the names (you can verify its output).  Next we will search for which rows in the `sport` column have an entry matching a sport in at list.  We can also do this in reverse, identifying sports not in the, by bracketing the logical test with a `!()`.  

Here we aren't going to store data frames, but will show the dimensions of those matching various characteristics with respect to row and column selections.  The "list search" operator searches its input against a each elemnt of a provided list.

```{r}
dim(athletes[athletes$sport %in% names(tab[19:28]), ])
dim(athletes[!(athletes$sport %in% names(tab[19:28])), ])
```

For proof-of-concept, we can take subsets of columns as numbers or variable names.
```{r}
dim(athletes[!(athletes$sport %in% names(tab[19:28])), 1:5])
```

```{r}
head(athletes[!(athletes$sport %in% names(tab[19:28])), 1:5])
```

```{r}
head(athletes[!(athletes$sport %in% names(tab[19:28])), ])
```

There was interest in calculating age using the participant's given birth year (and the date of the Olympics). You could convert both to julian dates and subtract.

## Hepatitis

We read in the hepatitis data^[<https://github.com/datasets/hepatitis>.] and demontrate the construction of a multidimensional table. Applying `as.data.frame()` to these results is usually very useful in small datasets, often to check for the presence of missing or unusual combinations of variables.

```{r}
dat <- read.delim("./data/hepatitis.csv", sep = ",")
head(dat)
table(dat[, c("steroid", "antivirals", "fatigue")])
```

That's all for now.
