EP624: Statistical Programming and Reproducible Research

• Install any latest distribution of GNU/Linux on your computer. I use Manjaro, but feel free to install any.
  • If you have a Mac, you can continue using OSX, and install the required applications on OSX.

ssh or Secure Shell is a protocol for securely transferring data over networks.

Use these commands to create your ssh keys.

# If you do not already have ssh keys, create them as below
mkdir ~/.ssh
ssh-keygen -t rsa #No need to use a password, when asked. Just press enter for all the questions (accept defaults).

If you have just accepted the defaults, these commands would have created two files in .ssh directory: ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub.

id_rsa is your private key which much be kept safe and is never shared with anyone.

id_rsa.pub is the matching public key which is shared with any machine with which you want to have secure data transfers.

In your github.com account, go to settings, click on “SSH and GPG keys”, and add a new ssh key. In the box that opens, in Title add your name, and the name of your machine, and in the box below, paste the contents of the public key that was created in the previous step (~/.ssh/id_rsa.pub).

Once that is done, following commands will configure Emacs for the requirements of this course.

git clone git@github.com:vikasrawal/prelude-ep624.git
#git clone https://github.com/vikasrawal/prelude-ep624.git # Use may use this if you do not have a github account with your public ssh key added to it.

mv .emacs.d .emacs.d.old # Renames if you already have a .emacs.d directory
mv prelude-ep624 .emacs.d
cd .emacs.d
git checkout ep624
cd

Git is a version control system. Created originally by Linus Torvaldes, it is the most widely used version control system for collaboration and tracking by software developers. Since most of our work – the org files, the R scripts, the LaTeX files – would be in plain text files, we can use git not only to track evolution of our programs but also of our research papers, dissertations and books.

Apart from locally hosting your repositories on your own machine, you can also keep your repositories on platforms like Github and Gitlab. If you want to keep your work-in-progress research on these platforms, you would need rights to create private repositories. While Gitlab provides it by default, you can enable it on Github by claiming educational discount.

I will be using git and Github to run this course. Which means that you will need to know basic git commands to submit your assignments and get my feedback. The idea is to use this to familiarise you with basic git commands.

I will create a separate repository for each student on Github, and invite you as a collaborator. You will clone it on your computer using the commands mentioned below, create a separate sub-directory for each assignment and do all your work in these sub-directories. Once you have completed some work and want to share it with me, you will commit it and push it to the github repository. You commit anything to the repository in two steps. You first add it to the staging area and then you commit it.

 1: ## Basic git workflow for this course
 2: ## The following command, after modifying the name of your repository,
 3: ## clones your repository. A directory -- 2020_yourname -- will be
 4: ## created in the directory from where the command is given
 5: ## (your home directory in this case). All future work for the course
 6: ## should be done in 2020_yourname directory.
 7: 
 8: cd ~
 9: git clone --recurse-submodules git@github.com:ep624/2020_yourname.git
10: 
11: cd ~/2020_yourname
12: rm -Rf resources
13: git clone git@github.com:ep624/resources.git
14: 
15: cd ~/2020_yourname/resources
16: git pull origin master
17: 
18: 
19: ## create a separate subdirectory in 2020_yourname for each of your assignments.
20: cd ~/2020_yourname
21: mkdir midterm
22: 
23: ## To add a file to the staging area (a new file, or a file that has been modified)
24: git add mainfile.org
25: git add figure1.png ## the repository accepts png files as git lfs objects.
26: 
27: ## To commit the file
28: git commit -m "A message describing this commit"
29: 
30: ## To push it to github repository
31: git push
32: 
33: ## To pull from github the work of other collaborators (my comments/edits, for example)
34: ## and merge it into your local repository
35: git pull
36: 

A separate team has been created on Github for your batch. You should accept the invite and join the team.

It would be useful for you to learn how to

• install a Linux system.
• use package manager of your Linux distribution to install software.
• customise Emacs.

While we have given some instructions here, and provided some ready-made configuration files for you to deploy, you should learn what these are doing, how to modify them, and how to extend them.

Emacs is an powerful cross-platform editor which can be used for all kinds of tasks that you do on your computer. Although Emacs is different from every other editor you are likely to have used, learning to use Emacs would be a huge asset for your computing abilities. A good way to learn Emacs, to ask, for each activity you do on your computer, can I do this using Emacs? You will be amazed at how many times the answer is: yes, and more efficiently than in your current application.

This section does not provide a detailed guide to the use of Emacs. In this guide, I will just provide a minimal set of basic commands in emacs to get you started. This is a minimal but sufficient set to be able to work. I expect that you would learn more commands as you start using emacs.

In emacs, a buffer is like a tab in an application. It is normal to have several buffers open at the same time. Each file opens in emacs as a buffer. Buffers could also have processes like R running in them. Emacs displays any messages for you in a separate buffer.

Most commands in emacs are given using the Control (ctrl) or the Meta (often mapped to alt) keys. Control key is usually referred to as C- and the Meta key as M-. So a command C-c means pressing Control and c together. Command M-x means pressing Meta and x together. Everything is case-sensitive. So M-X would mean, pressing Meta, Shift and x together. C-c M-x l would mean pressing C-c, release, then M-x, release, and then l.

Table 1 gives the commands that are the most important. This is a minimal set, commands that you should aim to learn as soon as possible. There are many more, which you will learn as you start using emacs.

All commands have a verbose version that can be used by pressing M-x and writing the command. For example, M-x find-file to open a file. All major commands are also mapped to a shortcut. For example, instead of typing M-x find-file to open a file, you can say C-x C-f. I remember shortcuts for commands that I use most frequently. For others, I use the verbose versions. Over time, one learns more shortcuts and starts using them instead of the verbose versions.

• A Tutorial Introduction to GNU Emacs
• A. J Rossini and Richard M Heiberger and Rodney A Sparapani and Martin Mächler and Kurt Hornik (2004), “Emacs Speaks Statistics: A Multiplatform, Multipackage Development Environment for Statistical Analysis”, Journal of Computational and Graphical Statistics, 13(1), 247-261
• Heiberger, Richard M (2001), “Emacs Speaks Statistcs: One Interface, Many Programs”, in K. Hornik and F. Leisch (eds.), Proceedings of the 2nd International Workshop on Distributed Statistical Computing, March 15-17, Vienna, Austria
• Rossini, A.J., Heiberger, R.M., Hornik, K., Maechler, M., Sparapani, R.A., Eglen, S.J., Spinu, V., Henry, L. (2016), ESS: Emacs Speaks Statistics
• Rossini, Anthony, Martin Maechler, Kurt Hornik, Richard M. Heiberger, (2001) “Emacs Speaks Statistics: A Universal Interface for Statistical Analysis”, University of Washington Biostatistics Available at: http://works.bepress.com/anthony_rossini/3/

• R is a case sensitive language. Do not mix upper case and lower case characters.
• You should have a separate directory for each project/article you work on.
• By default, the work you do is saved in this working directory in a file called .RData
• All R commands are followed by (). Arguments that the command needs are given inside the parentheses. Empty parentheses have to be used even if a command needs no arguments. For example,
  • q will not work as a command
  • q() is a command
  • q(save="yes") is a command with an argument that asks the R environment to be saved before quitting.
• -> or <- is used to create an object using output of a command.
  • c(1:5)->col1 creates an object col1 containing the output of the command c(1:5).
  • col1<-c(1:5) also does the same thing.
• In this guide, all commands are formatted using a mono-spaced font to distinguish them. Such lines are numbered. But line numbers are not a part of the command. In the boxes, text starting with a # is used to write comments for the reader. These comments are not part of the commands. You do not need to type them into R as any line starting with a # is treated as a comment in R and not executed. When you write your own programs, use lines starting with # to write your own comments that may be necessary to make your program intelligible to your collaborators.

 1: #+CAPTION: Some basic commands in R
 2: # Text after # is a comment to explain what the command does
 3: q() # Quit R; prompts for saving the workspace.
 4: ?command # Open help for 'command'; q to quit help.
 5: ??keyword # List commands with 'keyword' in the title
 6: help.search("keyword") # List commands with 'keyword'
 7:                        #  anywhere in the docs
 8: setwd("c:/project1")  # On a Windows machine, sets the working directory to c:/project1
 9: setwd("~/proj2")  # On a Linux or OSX machine, sets the working directory to ~/proj2
10: getwd() # Reports the present working directory

Showing statistical information in an appropriate graphical form can help reveal the underlying patterns and trends. This is the purpose of using statistical graphs. Graphs are not made to add colour to your report. Spreadsheet software generated chartjunk has increasingly been passed off as statistical graphs. tufte2001visual remarked, “graphical decoration, which prospers in technical publications as well as in commercial and media graphics, comes cheaper than the hard work required to produce intriguing numbers and secure evidence”.

The ggplot2 package provides an extremely powerful system for making statistical graphs. The ggplot2 package is based on Leland Wilkinson’s The Grammer of Graphics wilkinson2006grammar. It can be used for making a large variety of statistical graphs as well as maps. Although it is somewhat complex, mastering ggplot2 would give you a big advantage in making interesting statistical graphs.

This guide would give you a basic idea of the ggplot2 syntax. You should look at the ggplot2 website for more examples. You are also advised to read wickham2016ggplot2 and chang2012r. These are expected to be used as ready reckoners.

In ggplot2, graphs are made in steps. You start from a basic specification of the graph, and then add/modify the details.

Many commands in R, and in several add-on packages, can be used for working with and creating statistical tables. Among the most popular are the apply family of functions and plyr, dplyr and reshape2 packages. The data.tables package may be considered a swiss knife for working with and producing statistical tables. Of all the tools, this is one of the most flexible, versatile and efficient.

• data.table does not work directly on data frames but needs you to create a data frame like object called “data.table”.

• You can convert an existing data frame to a data.table using

  as.data.table(data1)->data1

• Or you can use fread("delimited_file_name")->data1 to read a delimited file and directly create a data table.
  • fread is fast and efficient.
  • It also automatically detects separating character and variable names.
• All commands that work on a data frame can also be used on a “data.table”. So nothing is lost by keeping your data as a data table rather than as a data frame.
• The basic syntax of data.table is as follows: DT[i,j,by]. This subsets DT using i, calculates j, groups using “by”
  • i (subsetting rows)
    • a set of logical expressions to subset DT.
    • columns can be referred to directly by their names.
    • data2.dt[sector==1] subsets rows for which variable sector==1.
  • j (selecting columns, summarising)
    • [,col] selects a column and returns it as a vector.
    • [,.col] selects a column and returns the output as datatable.
    • [,.(col1, col2)] or [,list(col1, col2)] return two columns.
    • [,.(colname1=col1, colname2=col2)] simultaneously renames columns.
    • [,.(c1mn=mean(col1),c2med=med(col2))] calculates mean and median.
  • by (separately doing the calculations for groups)

    • Following code computes mean and median separately for each combination of col3 and col4.

      [,.(c1mn=mean(col1),c2med=med(col2)),
        .(col3,col4)]
      

• In a data.table, you can define some variables as “keys”. These can be used for quickly sorting, subsetting and merging.
  • setkey(DT, sector) defines sector as a key variable in DT.

  • If sector in DT is defined as a key variable, then the following command will subset rows for which the key variable takes values “Industry” or “Agriculture”.

    DT[c("Industry","Agriculture")]
    

  • In the usual DT[i,j,by] type commands, you can replace by with keyby as shown below to simultaneously define keys and sort the output.

    [,.(c1mn=mean(col1),c2med=med(col2)),
     keyby=.(col3,col4)]
    

• Output of a data.table is also a data.table. This means, that a sequence of commands can be chained as:

  # Three commands run in a sequence
  
  DT[i1,j1,by1]->DT2
  DT2[i2,j2,by2]->DT3
  DT3[i3,j2,by3]
  
  # Can be combined into one line
  # as follows
  
  DT[i1,j1,by1][i2,j2,by2][i3,j2,by3]
  
  

• It is also possible to use DT[i,j] to do a transformation on DT itself rather than create a new data.table.

  • For example, to create a new column called newcol in DT, one can use j as

    newcol:=columnname1*columnname2.

    Note that := is used to say that the transformation is to be done on DT itself.

  • A limitation of this usage is that only one field can be transformed at a time. Transforming multiple fields would require chaining the data.table commands.
• Sometimes you may wish to rotate a cross-table created using data table. That is, you may wish to create columns out of one or more disaggregating (keyby) variables that have been used. This can be achieved in the following way.

  • In the command below, the first data.table transformation computes mean by sector, sex and religion. The second data.table transformation creates all combinations of sector, sex and religion, and ensures that DT2 has rows for all combinations (including null rows). The third data.table transformation rotates the table, with religion and sex being identified as row disaggregators and the remaining keyby variable (sector) as the column disaggregator.

    DT[,.(col1mean=mean(columnname1)),
       keyby=.(sector,sex,religion)]->DT1
    
    DT1[CJ(unique(sector),
           unique(sex),
           unique(religion))]->DT2
    
    DT2[,as.list(col1mean)
       ,by=.(sex,religion)]
    

  • Like all data.table commands, these can be chained together into one line of code.
• data.table provides a very convenient syntax for merging data.tables.
  • DT1[DT2] joins matching data from DT2 with DT1; the matches are done using their keys (presuming the keys are defined).
  • DT1[DT2,on=c(var1=“var2”)] joins matching data from DT2 with DT1; matches are done using var1 of DT1 and var2 of DT2.

• Chang, Winston (2013), R Graphics Cookbook, OReilly
• Crawley, Michael J (2015), Statistics: An Introduction using R, Wiley
• Fox, John and Weisberg, Sanford (2011), An R Companion to Applied Regression, Sage Publications, Thousand Oaks, CA.
• Hatekar, Neeraj (2010), Principles of Econometrics: An Introduction (Using R), Sage
• Hilbe, Joseph (2013), Methods of Statistical Model Estimation, Chapman & Hall/CRC Press.
• Lumley, Thomas, Complex Surveys: A Guide to Analysis Using R, John Wiley & Sons
• Maindonald

, John and Braun, John (2003), Data Analysis and Graphics Using R: An Example-based Approach, Cambridge University Press

• Michael J. Crawley (2013), The R Book, Second Edition, John Wiley & Sons
• Murrell, Paul (2011), R Graphics, Chapman & Hall/CRC Press.
• R Core Team (2015), An Introduction to R, R Foundation for Statistical Computing,Vienna.
• R Core Team (2015), R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing,Vienna.
• Spector, Phil (2008), Data Manipulation with R, Springer
• Wikham, Hadley, ggplot2: Elegant Graphics for Data Analysis, UseR Series, Springer

• A sample biblatex database to get you started can be taken from here. For now, you may just download it, save it and add to it. Please note that the source file on github keeps getting updated. Eventually, you may want to manage your own biblatex database using something like git.
• There is also another Emacs mode – ebib – that can be used for maintaining your biblatex database. To start ebib, use M-x ebib

• Dominik, Carsten and others (2016), The Org Manual, Free Software Foundation
• Gentleman, Robert and Duncan Temple Lang, “Statistical Analyses and Reproducible Research”, Journal of Computational and Graphical Statistics, 16 (1), pp. 1-23
• Kitchin, John (2014), The org-ref Manual
• Knuth, Donald E. (1992), Literate Programming, Center for the Study of Language and Information, Stanford.
• Rawal, Vikas (2014), Using Emacs, Org-mode and R for Research Writing in Social Sciences: A Toolkit for Writing Reproducible Research Papers and Monographs, http://archive.indianstatistics.org/tools/orgpapers.pdf; a revised but incomplete version may be seen at https://github.com/vikasrawal/orgpaper/blob/master/orgpapers.org
• Rossini, Anthony and Friedrich Leisch. (2003) “Literate Statistical Practice”, University of Washington Biostatistics Working Paper Series , Available at: http://works.bepress.com/anthony_rossini/6/
• Schulte, Eric and Dan Davison (2011), “Active Documents with Org-Mode”, Computing in Science & Engineering.
• Schulte, Eric, Dan Davison, Thomas Dye and Carsten Dominik (2016), “A Multi-Language Computing Environment for Literate Programming and Reproducible Research.” Journal of Statistical Software, 46 (3), 1 – 24.