MPA 5830 - Module 01

Spring 2020

1 Reading data

One of the golden rules to follow when reading data is to make your life easy by not having to memorize or discover through frantic trial-and-error runs where a particular file is located. So how do we make our life easy?

1. Put all code in a sub-folder called code and all data in a sub-folder called data
2. When reading or saving data files, make sure to use the {here} library. In the examples that follow you will see how {here} is used to (a) first specify where the data file is located, and (b) then to list the filename.

Both (1) and (2) have been done for us. Look inside the data folder and you will see a number of data files. Look inside the code folder and you will see files called Module01.Rmd, Module02.Rmd, and so on. These *.Rmd files are in the RMarkdown format, and each file corresponds to a module. Now on to reading data …

Data come in several formats but I will walk you through the formats you are most likely to encounter – MS Excel, CSV, TXT, fixed-width, and then in any one of these commercial software formats: SAS, Stata, or SPSS.

1.1 CSV data files

We start by reading a simple comma-separated variable format file and then a tab-delimited variable format file. A CSV file has each column separated by a comma while a tab-delimited file has every column separated by a tab – , versus \t

read.csv(
  here("data", "ImportDataCSV.csv"),
  sep = ",",
  header = TRUE
  ) -> df.csv 

read.csv(
  here("data", "ImportDataTAB.txt"),
  sep = "\t",
  header = TRUE
  ) -> df.tab

Note … * here("data") specifies where the files are located but this is an incomplete command because it does not specify the name of the file to be read. Instead, here("data", "ImportDataCSV.csv") is the complete command. * The sep = "," switch says the individual variables are separated by a comma * header = TRUE switch indicates that the first row in the file that is being read in has the column names
* The tab-delimited file needs sep = "\t" because the columns are separated by a tab

If both files were read then your RStudio Environment should show objects called df.csv and df.tab. If you don’t see these, check to see if both files are indeed in the data folder or not.

1.2 MS Excel files

Microsoft Excel files can be read via the readxl package, and you see two versions below – one for the older *.xls format and the other for the newer *.xlsx format.

As a rule, I would recommend against storing data in Excel formats since Excel tends to do some funny things. If you must store and share data, try to use the CSV format.

library(readxl)
read_excel(
  here("data", "ImportDataXLS.xls")
  ) -> df.xls 

read_excel(
  here("data", "ImportDataXLSX.xlsx")
  ) -> df.xlsx

1.3 SPSS, Stata, SAS files

Some agencies and other sources tend to use SAS, Stata, or SPSS for storing data and for carrying out various data analyses. This is a legacy issue that is changing but a little too slowly for most of us who do not use these commercial software packages as the centerpiece of our work. But, even if you do not use these packages, you should know how to read in data created in their native formats. As it turns out, SPSS, Stata, SAS files can be read via the haven package, and with relative ease.

library(haven)

read_stata(
  here("data", "ImportDataStata.dta")
  ) -> df.stata

read_sas(
  here("data", "ImportDataSAS.sas7bdat")
  ) -> df.sas

read_sav(
  here("data", "ImportDataSPSS.sav")
  ) -> df.spss

Again, if you look at the Environment you should see these three data files as well.

1.4 Fixed-width files

It is also common to encounter fixed-width files where the raw data are stored without any gaps between successive columns. This is also a legacy format from the early days of computers and punch cards, and one of the most efficient ways of storing large amounts of data. These files come with documentation that will give you the necessary details about where each column starts and ends, etc. See here for some examples of layouts from the Census Bureau.

read.fwf(
  here("data", "fwfdata.txt"),
  widths = c(4, 9, 2, 4),
  header = FALSE,
  col.names = c("Name", "Month", "Day", "Year")
  ) -> df.fw 

Notice we need widths = c(4,9,2,4) to indicate how many slots each column takes and then col.names = c("Name", "Month", "Day", "Year") to label the columns since the data file does not have variable names. if you mess up with widths = you end up with garbage because R does not know where any column starts or ends so be careful.

2 Reading Files from the Web

One of the benefits of software like R is its ability to read data files over the web without requiring you to manually download the file and save a physical copy to be read in. Specifically, it is possible to list the full web-path for a file and read it in. This ability is invaluable when the data tend to be periodically updated by the source (for example, by the Census Bureau, Bureau of Labor, Bureau of Economic Analysis, etc.). Here are a few examples.

read.table(
  "http://data.princeton.edu/wws509/datasets/effort.dat"
  ) -> fpe 

fpe # Show me what fpe contains 

##                setting effort change
## Bolivia             46      0      1
## Brazil              74      0     10
## Chile               89     16     29
## Colombia            77     16     25
## CostaRica           84     21     29
## Cuba                89     15     40
## DominicanRep        68     14     21
## Ecuador             70      6      0
## ElSalvador          60     13     13
## Guatemala           55      9      4
## Haiti               35      3      0
## Honduras            51      7      7
## Jamaica             87     23     21
## Mexico              83      4      9
## Nicaragua           68      0      7
## Panama              84     19     22
## Paraguay            74      3      6
## Peru                73      0      2
## TrinidadTobago      84     15     29
## Venezuela           91      7     11

read.table(
  "https://stats.idre.ucla.edu/stat/data/test.txt",
  header = TRUE
  ) -> test.txt 

read_csv(
  "https://stats.idre.ucla.edu/stat/data/test.csv"
  ) -> test.csv

read_sav(
  "https://stats.idre.ucla.edu/stat/data/hsb2.sav"
  ) -> hsb2.spss 

If you look at fpe you will notice that there are three columns but the rows each have a unique name. These are row names that are remembered by R but will not show up with a column name.

3 Reading compressed files

Large files may sit in compressed archives (.zip, .gz, etc.) on the web and R has a neat way of allowing you to download the file, unarchive it, and then read it in. Why is this useful? Because if these files tend to be periodically updated, R’s ability lets you use the same piece of R code to download/unzip/read in the updated file. The tedious way would be to manually download the file, unzip it, place the file in the appropriate data folder, and then read it in.

temp = tempfile()

download.file("ftp://ftp.cdc.gov/pub/Health_Statistics/NCHS/Datasets/NVSS/bridgepop/2016/pcen_v2016_y1016.sas7bdat.zip", temp)

oursasdata = haven::read_sas(unz(temp, "pcen_v2016_y1016.sas7bdat"))

unlink(temp)

4 Saving R data files

You can save your data in a format that R will recognize, giving it the RData or rdata extension

save(
  oursasdata,
  file = here("data", "oursasdata.RData")
  )

Check the data directory to confirm that both files are present

5 Minimal example of data processing

Working with the hsb2 data: 200 students from the High School and Beyond study. The variables in this file are:

• female = (0/1)
• race = (1=hispanic 2=asian 3=african-amer 4=white)
• ses = socioeconomic status (1=low 2=middle 3=high)
• schtyp = type of school (1=public 2=private)
• prog = type of program (1=general 2=academic 3=vocational)
• read = standardized reading score
• write = standardized writing score
• math = standardized math score
• science = standardized science score
• socst = standardized social studies score

read.table(
  'https://stats.idre.ucla.edu/stat/data/hsb2.csv',
  header = TRUE,
  sep = ","
  ) -> hsb2

What are the variable names (i.e., the column headings) in this file? The names() command will tell you that.

names(hsb2)

##  [1] "id"      "female"  "race"    "ses"     "schtyp"  "prog"    "read"   
##  [8] "write"   "math"    "science" "socst"

Similarly, the str() command will show you the details of each variable

str(hsb2)

## 'data.frame':    200 obs. of  11 variables:
##  $ id     : int  70 121 86 141 172 113 50 11 84 48 ...
##  $ female : int  0 1 0 0 0 0 0 0 0 0 ...
##  $ race   : int  4 4 4 4 4 4 3 1 4 3 ...
##  $ ses    : int  1 2 3 3 2 2 2 2 2 2 ...
##  $ schtyp : int  1 1 1 1 1 1 1 1 1 1 ...
##  $ prog   : int  1 3 1 3 2 2 1 2 1 2 ...
##  $ read   : int  57 68 44 63 47 44 50 34 63 57 ...
##  $ write  : int  52 59 33 44 52 52 59 46 57 55 ...
##  $ math   : int  41 53 54 47 57 51 42 45 54 52 ...
##  $ science: int  47 63 58 53 53 63 53 39 58 50 ...
##  $ socst  : int  57 61 31 56 61 61 61 36 51 51 ...

and the summary() command will give you some summary information about each variable.

summary(hsb2)

##        id             female           race           ses            schtyp    
##  Min.   :  1.00   Min.   :0.000   Min.   :1.00   Min.   :1.000   Min.   :1.00  
##  1st Qu.: 50.75   1st Qu.:0.000   1st Qu.:3.00   1st Qu.:2.000   1st Qu.:1.00  
##  Median :100.50   Median :1.000   Median :4.00   Median :2.000   Median :1.00  
##  Mean   :100.50   Mean   :0.545   Mean   :3.43   Mean   :2.055   Mean   :1.16  
##  3rd Qu.:150.25   3rd Qu.:1.000   3rd Qu.:4.00   3rd Qu.:3.000   3rd Qu.:1.00  
##  Max.   :200.00   Max.   :1.000   Max.   :4.00   Max.   :3.000   Max.   :2.00  
##       prog            read           write            math      
##  Min.   :1.000   Min.   :28.00   Min.   :31.00   Min.   :33.00  
##  1st Qu.:2.000   1st Qu.:44.00   1st Qu.:45.75   1st Qu.:45.00  
##  Median :2.000   Median :50.00   Median :54.00   Median :52.00  
##  Mean   :2.025   Mean   :52.23   Mean   :52.77   Mean   :52.65  
##  3rd Qu.:2.250   3rd Qu.:60.00   3rd Qu.:60.00   3rd Qu.:59.00  
##  Max.   :3.000   Max.   :76.00   Max.   :67.00   Max.   :75.00  
##     science          socst      
##  Min.   :26.00   Min.   :26.00  
##  1st Qu.:44.00   1st Qu.:46.00  
##  Median :53.00   Median :52.00  
##  Mean   :51.85   Mean   :52.41  
##  3rd Qu.:58.00   3rd Qu.:61.00  
##  Max.   :74.00   Max.   :71.00

There are no value labels for the various qualitative/categorical variables (female, race, ses, schtyp, and prog) but these are easily created as shown below.

factor(hsb2$female,
       levels = c(0, 1),
       labels = c("Male", "Female")
       ) -> hsb2$female.f 

factor(hsb2$race,
       levels = c(1:4),
       labels = c("Hispanic", "Asian", "African American", "White")
       ) -> hsb2$race.f

factor(hsb2$ses,
       levels = c(1:3),
       labels = c("Low", "Middle", "High")
       ) -> hsb2$ses.f

factor(hsb2$schtyp,
       levels = c(1:2),
       labels = c("Public", "Private")
       ) -> hsb2$schtyp.f

factor(hsb2$prog,
       levels = c(1:3),
       labels = c("General", "Academic", "Vocational")
       ) -> hsb2$prog.f

Let us now take a quick look at the contents of each column in hsb2

summary(hsb2)

##        id             female           race           ses            schtyp    
##  Min.   :  1.00   Min.   :0.000   Min.   :1.00   Min.   :1.000   Min.   :1.00  
##  1st Qu.: 50.75   1st Qu.:0.000   1st Qu.:3.00   1st Qu.:2.000   1st Qu.:1.00  
##  Median :100.50   Median :1.000   Median :4.00   Median :2.000   Median :1.00  
##  Mean   :100.50   Mean   :0.545   Mean   :3.43   Mean   :2.055   Mean   :1.16  
##  3rd Qu.:150.25   3rd Qu.:1.000   3rd Qu.:4.00   3rd Qu.:3.000   3rd Qu.:1.00  
##  Max.   :200.00   Max.   :1.000   Max.   :4.00   Max.   :3.000   Max.   :2.00  
##       prog            read           write            math      
##  Min.   :1.000   Min.   :28.00   Min.   :31.00   Min.   :33.00  
##  1st Qu.:2.000   1st Qu.:44.00   1st Qu.:45.75   1st Qu.:45.00  
##  Median :2.000   Median :50.00   Median :54.00   Median :52.00  
##  Mean   :2.025   Mean   :52.23   Mean   :52.77   Mean   :52.65  
##  3rd Qu.:2.250   3rd Qu.:60.00   3rd Qu.:60.00   3rd Qu.:59.00  
##  Max.   :3.000   Max.   :76.00   Max.   :67.00   Max.   :75.00  
##     science          socst         female.f                race.f   
##  Min.   :26.00   Min.   :26.00   Male  : 91   Hispanic        : 24  
##  1st Qu.:44.00   1st Qu.:46.00   Female:109   Asian           : 11  
##  Median :53.00   Median :52.00                African American: 20  
##  Mean   :51.85   Mean   :52.41                White           :145  
##  3rd Qu.:58.00   3rd Qu.:61.00                                      
##  Max.   :74.00   Max.   :71.00                                      
##     ses.f       schtyp.f          prog.f   
##  Low   :47   Public :168   General   : 45  
##  Middle:95   Private: 32   Academic  :105  
##  High  :58                 Vocational: 50  
##                                            
##                                            
## 

Note: We did not overwrite the original columns, and this is important because in case we mess up with this modification, we still have the original columns to work with. Also pay attention to how the modifications were made, for example, …

1. take column female that has values 0 and 1
2. a value of 0 should be labeled Male and a value of 1 should be labeled Female
3. save the new column as female.f

Further, there are are four values for race, 3 for ses, 2 for schtyp, and 3 for prog, so the mapping has to reflect this.

This is just a quick run through with creating value labels; we will cover this in greater detail in a later module.

save your work!!

Having created new columns we can now save the resulting data for later use.

save(hsb2,
     file = here("data", "hsb2.RData")
  )

Let us test if this R Markdown file will Knit to html.

6 Data in packages

Almost all R packages come bundled with data-sets, too many of them to walk you through but

• see here for standard ones
• here are some more
• and some more

To load data from a package, if you know the data-set’s name, run

library(HistData)

data("Galton")

names(Galton)

## [1] "parent" "child"

or you can run

data("GaltonFamilies", package = "HistData")

names(GaltonFamilies)

## [1] "family"          "father"          "mother"          "midparentHeight"
## [5] "children"        "childNum"        "gender"          "childHeight"

7 More on saving data and workspaces

You can save your data via

• save(dataname, file = here("filepath", "filename.RData"))

or

• save(dataname, file = here("filepath", "filename.rdata"))

data(mtcars)

save(mtcars,
     file = here("data", "mtcars.RData")
  )

Check the data folder and you should see mtcars.RData.

Now, say we wanted to load a saved R data file. How might we do that? This is done below but as you run the code, note that the rm(list = ls()) command is basically wiping the Global Environment clean by removing everything in it. This is the same as clicking the ‘broom" icon beside ’Import Dataset v’

rm(list = ls()) # To clear the Environment of everything in it 

load(
  here("data", "mtcars.RData")
  )

You can also save multiple data files, individual objects, and even everything you have done in a work session. These are covered in the textbook.

If you are not in a project and they try to close RStudio after some code has been run, you will be prompted to save (or not) the workspace and you should say “no” by default unless you want to save the workspace.

8 RStudio webinars

The fantastic team at RStudio runs free webinar that are often very helpful so be sure to signup with your email. Here are some video recordings of webinars that are relevant to what we have covered so far.

• Programming Part 1 (Writing code in RStudio)
• Programming Part 2 (Debugging code in RStudio)
• Managing Change Part 1 (Projects in RStudio)
• Importing Data into R
• Whats new with readxl
• Getting your data into R
• Getting Started with R Markdown

---

9 Exercises for practice

These are some exercises you can use to practice and build your R skills. They are not for grade.

9.1 Ex. 1: Creating and knitting a new RMarkdown file

Open a fresh session by launching RStudio and then running File -> Open Project...

Give it a title, your name as the author, and then save it with in code with the following name: m1ex1.Rmd

Add this level 1 heading The Starwars Data and then insert your first code chunk as shown below but change eval = FALSE to eval = TRUE. The R code will not be executed if eval = FALSE.

library(dplyr)
data(starwars)
names(starwars)

Add this level 2 heading Character Heights and Weights and then your second code chunk

plot(starwars$height, starwars$mass)

Now knit this file to html

9.2 Ex. 2: Lorem Ipsum paragraphs and graphs

Go to this website and generate five Lorem Ipsum placeholder text paragraphs

• para 1: must have level 1 heading
• para 2: must have level 2 heading
• para 3: must have level 3 heading
• para 4: must have level 4 heading
• para 5: must have level 5 heading

Using the starwars data, create five code chunks, one after each paragraph, making sure to set eval = TRUE

• Each code chunk will have the same R code (see below)

plot(starwars$height, starwars$mass)

Now knit this file to html

9.3 Ex. 3: Reading in three data files

Create a new RMarkdown file that is blank after the initial setup code chunk

Insert a code chunk that reads in both these files found on the web

• http://www.stata.com/data/jwooldridge/eacsap/mroz.dta
• http://calcnet.mth.cmich.edu/org/spss/V16_materials/DataSets_v16/airline_passengers.sav In a follow-up code chunk, run the summary() command on each data-set

In a separate code chunk, read in this dataset after you download it and save the unzipped file in your data folder.

• The variable gender has the following codes: Zero = unknown; 1 = male; 2 = female
• Use this coding scheme to convert gender into a factor with these value labels

In a follow-up chunk run the following commands on this data-set

• names()
• str()
• summary()

In a final chunk, run the commands necessary to save each of the three data-sets as separate RData files. Make sure you save them in your data folder.

Now knit the complete Rmd file to html

9.3.1 Ex. 4: Welcome to Kaggle & Mass Shootings

Go to this page on Kaggle and read the description of the data-set on mass shootings in the United States that occurred during the 1966-2017 period. once you have read the overview of the data, click the “Data” tab and download the file called Mass Shootings Dataset.csv. Be careful; there are several versions so the one you want is the very last one and not any that have a version number attached, such as “Mass Shootings Dataset Ver 2.csv” for example.

Now read this file into R, perhaps naming it shootings and run the summary() command on it. Note the number of observations and the number of variables in the data-set.

9.3.2 Ex. 5: Animal Shelters

Go to this page on Kaggle and download the file called aac_shelter_outcomes.zip, unzip it, and AFTER reading the data overview, read in the file and generate a list of variable names with an appropriate command.