A Data Carpentry Workshop

BELSPO

Avenue Louise/Louizalaan 231, 1050 Brussels

 

Nov 2-3, 2015

9:00 - 17:00

Instructors: Leszek Tarkowski, Frederik Coppens

Helpers: Dima Fishman, Christof De Bo

Workshop Information

Data Carpentry

In many domains of research the rapid generation of large amounts of data is fundamentally changing how research is done. The deluge of data presents great opportunities, but also many challenges in managing, analyzing and sharing data. Data Carpentry is designed to teach basic concepts, skills and tools for working more effectively with data. The workshop is aimed at researchers in the life sciences at all career stages and is designed for learners with little to no prior knowledge of programming, shell scripting, or command line tools.

Etherpad

For communication we will be using this Etherpad

Workshop learning objectives & programme

Workshop Learning objectives & programme

Venue

Where: Avenue Louise/Louizalaan 231, 1050 Brussels. Get directions with OpenStreetMap or Google Maps.

Contact

Contact: Please mail leszek.tarkowski@gmail.com for more information.

The participants will need to bring their own laptops configured before the workshop according to the setup instructions.

We especially encourage to register those who may be less familiar with the above topics. There is no prerequisite as to what computing skills and knowledge is required.

Acknowledgements & Support

The workshop is supported by the ELIXIR programme. Data Carpentry is a sibling organization of Software Carpentry and is designed to teach basic concepts, skills and tools for working more effectively with data. The structure and objectives of the curriculum as well as the teaching style are informed by Software Carpentry.


Schedule

Day 1

09:00 Welcome and setup help
09:30 Caveats of working with data in spreadsheets
10:00 Introduction to Open Refine
10:30 Coffee
11:00 Working with Open Refine
12:00 Lunch break
13:00 Introduction to RStudio
13:20 Introduction to data manipulation with R - part 1
15:00 Coffee
15:30 Introduction to data manipulation with R - part 2
17:00 Wrap-up

Day 2

09:00 Data visualisation with R - part 1
10:30 Coffee break
11:00 Data visualisation with R - part 2
12:00 Lunch break
13:00 Introduction to data manipulation with SQL - part 1
15:00 Coffee break
15:30 Introduction to data manipluation with SQL - part 2
16:30 Wrap-up and feedback
17:00 Close

Etherpad: http://pad.software-carpentry.org/2015-11-02-DC-Brussels.
We will use this Etherpad for chatting, taking notes, and sharing URLs and bits of code.


Setup

To participate in a Data Carpentry workshop, you will need working copies of the software described below. Please make sure to install everything (or at least to download the installers) before the start of your workshop.

Overview

Editor

When you're writing scripts or text, it's nice to have a text editor that is optimized for writing code, with features like automatic color-coding of key words. The default text editor on Mac OS X and Linux is usually set to Vim, which is not famous for being intuitive. if you accidentally find yourself stuck in it, try typing the escape key, followed by ':wq!' (colon, lower-case 'q', exclamation mark), then hitting Return to return to the shell. (This will exit Vim and save the changes you made in the file.)

The Bash Shell

Bash is a commonly-used command line interface, a so-called shell. Using a shell gives you the power to do more tasks more quickly by freeing you from having to click everything.

R

R is a programming language that specializes in statistical computing. It is a powerful tool for exploratory data analysis. To interact with R, we will use RStudio, an interactive development environment (IDE).

SQL

SQL is a specialized programming language used with databases. We use a simple database manager called SQLite, either directly or through a browser plugin.

Spreadsheet

We'll be using a spreadsheet program. If you already have a spreadsheet program installed, like LibreOffice, Excel or OpenOffice, you can use whatever you already have. If you don't have a spreadsheet program, please download and install LibreOffice.

Windows

Editor

Notepad++ is a popular free code editor for Windows. Be aware that you must add its installation directory to your system path in order to launch it from the command line (or have other tools like Git launch it for you). The instructions to modify your path are available online here. Please ask your instructor to help you do this.

Git Bash

Install Git (version control) and a Bash shell for Windows from the msysGit project's homepage. This will provide you with Bash in the Git Bash program.

R

Install R by downloading and running this .exe file from CRAN. Also, please install the RStudio IDE.

SQLite

Install the Firefox SQLite browser plugin described below.

Software Carpentry Installer

Other tools used in Data Carpentry have been packaged up by Software Carpentry in an installer. This installer requires an active internet connection.

  • Download the installer.
  • Double click on the file to run it.
  • A command window will open and then close again. The installer downloads and installs a couple of command line programs that we'll need for course.
  • If you saw it open and then close again, you're set!

Mac OS X

Bash

The default shell in all versions of Mac OS X is bash, so no need to install anything. You access bash from the Terminal (found in /Applications/Utilities). You may want to keep Terminal in your dock for this workshop.

Editor

We recommend Text Wrangler or Sublime Text. Alternatively, you can use nano, which should be pre-installed.

R

Install R by downloading and running this .pkg file from CRAN. Also, please install the RStudio IDE.

SQLite

sqlite3 comes pre-installed on Mac OS X. Also install the Firefox SQLite browser plugin described below.

Linux

Bash

The default shell is usually bash, but if your machine is set up differently you can run it by opening a terminal and typing bash. There is no need to install anything.

Editor

Kate is one option for Linux users. Alternatively, you can use nano, which should be pre-installed.

R

You can download the binary files for your distribution from CRAN. Or you can use your package manager, e.g. for Debian/Ubuntu run apt-get install r-base. Also, please install the RStudio IDE.

SQLite

sqlite3 comes pre-installed on Linux. Also install the Firefox SQLite browser plugin described below.

 

Extras - everyone needs to install these

Firefox SQLite Plugin

Instead of using sqlite3 from the command line, you will use this plugin for Firefox instead. If you don't already have firefox, install it first. To install the sqlite plugin:

  • Start Firefox.
  • Go to the plugin homepage.
  • Click the "Add Now" button.
  • Click "Install Now" on the dialog that appears after the download completes.
  • Restart Firefox when prompted.
  • To check that it was installed correctly, select "SQLite Manager" from the "Tools" menu.

OpenRefine

OpenRefine (formerly Google Refine) is a powerful tool for exploring and working with messy data. You will need a browser (Firefox or Chrome is recommended) to work with Open Refine. Once you install Open Refine, you should be able to launch it double clicking like most of the programs. Open Refine will open in the browser (but you don't have to be online)

Spreadsheets

Spreadsheets are useful for data entry and data organization, and some subsetting and sorting of the data as well as getting an overview of the data. To interact with spreadsheets, we can use LibreOffice, Microsoft Excel, Gnumeric, OpenOffice.org, or other programs. Commands may differ a bit between programs, but general ideas for thinking about spreadsheets is the same.

For this lesson, if you don't have a spreadsheet program already, you can use LibreOffice. It's a free, open source spreadsheet program.

Windows

  • Download the Installer
    Install LibreOffice by going to the installation page. The version for Windows should automatically be selected. Click Download Version 4.4.2. You will go to a page that asks about a donation, but you don't need to make one. Your download should begin automatically.
  • Install LibreOffice
    Once the installer is downloaded, double click on it and it should install.

Mac OS X

  • Download the Installer
    Install LibreOffice by going to the installation page. The version for Mac OS X should automatically be selected. Click Download Version 4.4.2. You will go to a page that asks about a donation, but you don't need to make one. Your download should begin automatically.
  • Install LibreOffice
    The file LibreOffice_4.4.2_MacOS_x86-64 should have been downloaded. Double click on this file, and LibreOffice will be installed.

Linux

  • Download the Installer
    Install LibreOffice by going to the installation page. The version for Linux should automatically be selected. Click Download Version 4.4.2. You will go to a page that asks about a donation, but you don't need to make one. Your download should begin automatically.
  • Install LibreOffice
    Once the installer is downloaded, double click on it and it shou\ ld install.