Data frames and tibbles

The table that every statistical analysis starts from

Understand tidy data (units in rows, variables in columns), create data frames and tibbles, inspect them, index them with [rows, cols], $ and [[, add columns and subset rows.

Level 1 · Foundations ⏱ ~60 min Prerequisites: Vectors, Factors, Lists

Learning objectives

By the end of this lesson you will be able to:

identify the statistical unit, the variables and the observations in a dataset, and recognise tidy data;
create a data frame with data.frame() and a tibble with tibble(), and explain how they differ;
inspect an unknown dataset with str(), glimpse(), head(), summary(), nrow(), ncol() and names();
extract cells, rows and columns with [rows, cols], $ and [[;
add a column and keep only the rows that satisfy a condition.

library(tibble)
library(dplyr)

1. Why do we need data frames?

Imagine a field biologist who weighs a penguin, measures its bill and flipper, and writes down the species, the island and the sex. She repeats this for 344 birds. What she ends up with is a table: one line per bird, one column per measurement.

Some columns contain numbers (body mass), others contain labels (species). A matrix cannot hold this, because all its cells must have the same type. A data frame can: it is R’s object for a table in which every column is a vector of its own type, and all columns have the same length.

Almost every statistical method in this course — from a mean to a random forest — takes a data frame as input. Getting comfortable with them is the first real step of data analysis.

2. Units, variables, observations: tidy data

Before touching R, let us fix the vocabulary that statistics uses for a table.

The statistical unit is the “thing” we collect information about: a penguin, a patient, a country in a given year, a car.
A variable is a characteristic measured on each unit: body mass, species, blood pressure. Its values vary from unit to unit.
An observation is the set of values measured on one unit.

A dataset is tidy when it follows three simple rules:

each variable is a column;
each observation (unit) is a row;
each value is a cell.

Why statistics cares about tidy data

When you compute “the mean body mass”, you average a column. When you count “how many penguins”, you count rows. When you fit a model body_mass ~ species, R looks for two columns. Statistical software assumes the tidy layout; if your table is not tidy, the first job is to reshape it (see Tidy data and reshaping).

Here is a table that is not tidy. It stores the same kind of measurement (mass) in two columns, one per year:

untidy <- tibble(
  penguin   = c("A1", "A2", "A3"),
  mass_2008 = c(3750, 3800, 3250),
  mass_2009 = c(3900, 3850, 3400)
)
untidy

#> # A tibble: 3 × 3
#>   penguin mass_2008 mass_2009
#>   <chr>       <dbl>     <dbl>
#> 1 A1           3750      3900
#> 2 A2           3800      3850
#> 3 A3           3250      3400

Is the unit “a penguin” or “a penguin in a year”? If the question is how does mass change over time, the unit is the penguin-year, and the tidy version has three columns: penguin, year, mass. Deciding what the unit is comes before any code.

3. Creating a data frame

Base R: `data.frame()`

df <- data.frame(
  id    = c(10, 20, 30, 40),
  item  = c("book", "pen", "textbook", "pencil_case"),
  store = c(TRUE, FALSE, TRUE, FALSE),
  price = c(2.5, 8, 10, 7)
)
df

#>   id        item store price
#> 1 10        book  TRUE   2.5
#> 2 20         pen FALSE   8.0
#> 3 30    textbook  TRUE  10.0
#> 4 40 pencil_case FALSE   7.0

Each argument becomes a column: id and price are numeric, item is character, store is logical. Since R 4.0, text columns stay character (the old default stringsAsFactors = TRUE is gone); you convert to a factor only when you want one.

Tidyverse: `tibble()`

A tibble is a modern data frame from the tibble package. It is a data frame (every function that accepts a data frame accepts a tibble), with friendlier behaviour.

df_tbl <- tibble(
  id    = c(10, 20, 30, 40),
  item  = c("book", "pen", "textbook", "pencil_case"),
  store = c(TRUE, FALSE, TRUE, FALSE),
  price = c(2.5, 8, 10, 7)
)
df_tbl

#> # A tibble: 4 × 4
#>      id item        store price
#>   <dbl> <chr>       <lgl> <dbl>
#> 1    10 book        TRUE    2.5
#> 2    20 pen         FALSE   8  
#> 3    30 textbook    TRUE   10  
#> 4    40 pencil_case FALSE   7

	`data.frame()`	`tibble()`
Printing	prints all rows	prints the first 10 rows and the column types (`<dbl>`, `<chr>`, …)
`df[, 1]` (one column)	returns a vector	returns a tibble (one column)
Row names	allowed (e.g. `mtcars`)	not used; move them into a column
Partial matching of `$`	`df$pri` returns the column `price` (with a warning)	warning and `NULL`

You can convert in both directions with as_tibble() and as.data.frame().

Going deeper: a data frame is a list

Under the hood a data frame is a list of vectors of equal length, with the class "data.frame". This is why length(df) returns the number of columns (4), and why df[["price"]] works exactly like for a list.

is.list(df)

#> [1] TRUE

length(df)

#> [1] 4

4. Inspecting a dataset

Base R (≥ 4.5) ships the penguins dataset: measurements of 344 penguins from three islands in Antarctica. When you meet a new dataset, run these functions first.

nrow(penguins)

#> [1] 344

ncol(penguins)

#> [1] 8

names(penguins)

#> [1] "species"     "island"      "bill_len"    "bill_dep"    "flipper_len"
#> [6] "body_mass"   "sex"         "year"

str(penguins)

#> 'data.frame':    344 obs. of  8 variables:
#>  $ species    : Factor w/ 3 levels "Adelie","Chinstrap",..: 1 1 1 1 1 1 1 1 1 1 ...
#>  $ island     : Factor w/ 3 levels "Biscoe","Dream",..: 3 3 3 3 3 3 3 3 3 3 ...
#>  $ bill_len   : num  39.1 39.5 40.3 NA 36.7 39.3 38.9 39.2 34.1 42 ...
#>  $ bill_dep   : num  18.7 17.4 18 NA 19.3 20.6 17.8 19.6 18.1 20.2 ...
#>  $ flipper_len: int  181 186 195 NA 193 190 181 195 193 190 ...
#>  $ body_mass  : int  3750 3800 3250 NA 3450 3650 3625 4675 3475 4250 ...
#>  $ sex        : Factor w/ 2 levels "female","male": 2 1 1 NA 1 2 1 2 NA NA ...
#>  $ year       : int  2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 ...

str() (“structure”) shows, for each column, its type and its first values. glimpse() from dplyr shows the same information in a tidier layout:

glimpse(penguins)

#> Rows: 344
#> Columns: 8
#> $ species     <fct> Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, Ad…
#> $ island      <fct> Torgersen, Torgersen, Torgersen, Torgersen, Torgersen, Tor…
#> $ bill_len    <dbl> 39.1, 39.5, 40.3, NA, 36.7, 39.3, 38.9, 39.2, 34.1, 42.0, …
#> $ bill_dep    <dbl> 18.7, 17.4, 18.0, NA, 19.3, 20.6, 17.8, 19.6, 18.1, 20.2, …
#> $ flipper_len <int> 181, 186, 195, NA, 193, 190, 181, 195, 193, 190, 186, 180,…
#> $ body_mass   <int> 3750, 3800, 3250, NA, 3450, 3650, 3625, 4675, 3475, 4250, …
#> $ sex         <fct> male, female, female, NA, female, male, female, male, NA, …
#> $ year        <int> 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007…

head(penguins, n = 3)

#>   species    island bill_len bill_dep flipper_len body_mass    sex year
#> 1  Adelie Torgersen     39.1     18.7         181      3750   male 2007
#> 2  Adelie Torgersen     39.5     17.4         186      3800 female 2007
#> 3  Adelie Torgersen     40.3     18.0         195      3250 female 2007

summary(penguins)

#>       species          island       bill_len       bill_dep     flipper_len 
#>  Adelie   :152   Biscoe   :168   Min.   :32.1   Min.   :13.1   Min.   :172  
#>  Chinstrap: 68   Dream    :124   1st Qu.:39.2   1st Qu.:15.6   1st Qu.:190  
#>  Gentoo   :124   Torgersen: 52   Median :44.5   Median :17.3   Median :197  
#>                                  Mean   :43.9   Mean   :17.2   Mean   :201  
#>                                  3rd Qu.:48.5   3rd Qu.:18.7   3rd Qu.:213  
#>                                  Max.   :59.6   Max.   :21.5   Max.   :231  
#>                                  NAs    :2      NAs    :2      NAs    :2    
#>    body_mass        sex           year     
#>  Min.   :2700   female:165   Min.   :2007  
#>  1st Qu.:3550   male  :168   1st Qu.:2007  
#>  Median :4050   NAs   : 11   Median :2008  
#>  Mean   :4202                Mean   :2008  
#>  3rd Qu.:4750                3rd Qu.:2009  
#>  Max.   :6300                Max.   :2009  
#>  NAs    :2

How to read this output

For numeric columns, summary() gives minimum, quartiles, mean and maximum — the first look at a distribution.
For factor columns (species, island, sex) it gives the count of each level.
NA's counts the missing values: 11 penguins have no recorded sex, and 2 have no body mass. Remember this: missing values will come back in almost every lesson.
year is stored as an integer, so it gets a mean (2008), which is not very meaningful. Knowing the type of variable you are dealing with matters more than what R computes.

Tibbles print nicely, so it is common to convert a base data frame once at the start:

as_tibble(penguins)

#> # A tibble: 344 × 8
#>    species island    bill_len bill_dep flipper_len body_mass sex     year
#>    <fct>   <fct>        <dbl>    <dbl>       <int>     <int> <fct>  <int>
#>  1 Adelie  Torgersen     39.1     18.7         181      3750 male    2007
#>  2 Adelie  Torgersen     39.5     17.4         186      3800 female  2007
#>  3 Adelie  Torgersen     40.3     18           195      3250 female  2007
#>  4 Adelie  Torgersen     NA       NA            NA        NA <NA>    2007
#>  5 Adelie  Torgersen     36.7     19.3         193      3450 female  2007
#>  6 Adelie  Torgersen     39.3     20.6         190      3650 male    2007
#>  7 Adelie  Torgersen     38.9     17.8         181      3625 female  2007
#>  8 Adelie  Torgersen     39.2     19.6         195      4675 male    2007
#>  9 Adelie  Torgersen     34.1     18.1         193      3475 <NA>    2007
#> 10 Adelie  Torgersen     42       20.2         190      4250 <NA>    2007
#> # ℹ 334 more rows

Other built-in datasets are listed by data(); each has a help page (?mtcars, ?airquality). Some, like mtcars, store the unit name in row names. Tibbles prefer it as a proper variable:

mtcars |>
  rownames_to_column(var = "model") |>
  as_tibble()

#> # A tibble: 32 × 12
#>    model         mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
#>    <chr>       <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#>  1 Mazda RX4    21       6 160     110  3.9  2.62  16.46     0     1     4     4
#>  2 Mazda RX4 …  21       6 160     110  3.9  2.875 17.02     0     1     4     4
#>  3 Datsun 710   22.8     4 108      93  3.85 2.32  18.61     1     1     4     1
#>  4 Hornet 4 D…  21.4     6 258     110  3.08 3.215 19.44     1     0     3     1
#>  5 Hornet Spo…  18.7     8 360     175  3.15 3.44  17.02     0     0     3     2
#>  6 Valiant      18.1     6 225     105  2.76 3.46  20.22     1     0     3     1
#>  7 Duster 360   14.3     8 360     245  3.21 3.57  15.84     0     0     3     4
#>  8 Merc 240D    24.4     4 146.7    62  3.69 3.19  20        1     0     4     2
#>  9 Merc 230     22.8     4 140.8    95  3.92 3.15  22.9      1     0     4     2
#> 10 Merc 280     19.2     6 167.6   123  3.92 3.44  18.3      1     0     4     4
#> # ℹ 22 more rows

5. Indexing: `[rows, cols]`

A data frame has two dimensions, so square brackets take two indices separated by a comma: df[rows, cols]. Leaving one side empty means “all”.

# One cell: row 1, column 2
df[1, 2]

#> [1] "book"

# Rows 1 to 2, all columns
df[1:2, ]

#>   id item store price
#> 1 10 book  TRUE   2.5
#> 2 20  pen FALSE   8.0

# Column 1, all rows
df[, 1]

#> [1] 10 20 30 40

# Columns chosen by name
df[, c("id", "store")]

#>   id store
#> 1 10  TRUE
#> 2 20 FALSE
#> 3 30  TRUE
#> 4 40 FALSE

# One cell by row number and column name
df[2, "price"]

#> [1] 8

You can index with numbers (positions), names or logical vectors, exactly as with vectors. Names are safer: if a column is added or moved, df[, "price"] still works, df[, 4] may not.

Extracting one column as a vector: `$` and `[[`

df$id         # convenient when typing interactively

#> [1] 10 20 30 40

df[["id"]]    # works with a name stored in a variable

#> [1] 10 20 30 40

col <- "price"
df[[col]]

#> [1]  2.5  8.0 10.0  7.0

Single brackets with one index, df["id"], return a data frame with one column; double brackets df[["id"]] return the vector inside. This is the same “train carriage vs its content” difference you met with lists.

6. Adding columns

A new column must have one value per row.

df$quantity <- c(10, 35, 40, 5)
df

#>   id        item store price quantity
#> 1 10        book  TRUE   2.5       10
#> 2 20         pen FALSE   8.0       35
#> 3 30    textbook  TRUE  10.0       40
#> 4 40 pencil_case FALSE   7.0        5

If the length does not match, base R stops with an error:

df$quantity <- c(10, 35, 40)

#> Error in `$<-.data.frame`:
#> ! sostituzione con 3 righe, dati con 4

New columns are often computed from existing ones. In base R:

df$revenue <- df$price * df$quantity
df

#>   id        item store price quantity revenue
#> 1 10        book  TRUE   2.5       10      25
#> 2 20         pen FALSE   8.0       35     280
#> 3 30    textbook  TRUE  10.0       40     400
#> 4 40 pencil_case FALSE   7.0        5      35

The tidyverse way is mutate(), which you will use constantly from the next lesson:

df_tbl |>
  mutate(quantity = c(10, 35, 40, 5),
         revenue  = price * quantity)

#> # A tibble: 4 × 6
#>      id item        store price quantity revenue
#>   <dbl> <chr>       <lgl> <dbl>    <dbl>   <dbl>
#> 1    10 book        TRUE    2.5       10      25
#> 2    20 pen         FALSE   8         35     280
#> 3    30 textbook    TRUE   10         40     400
#> 4    40 pencil_case FALSE   7          5      35

The symbol |> is the pipe: read it as “and then”. It passes the table on its left as the first argument of the function on its right.

7. Keeping some rows: subsetting

Put a logical condition in the row position: rows where it is TRUE are kept.

df[df$price > 5, ]

#>   id        item store price quantity revenue
#> 2 20         pen FALSE     8       35     280
#> 3 30    textbook  TRUE    10       40     400
#> 4 40 pencil_case FALSE     7        5      35

df[df$price > 5 & df$store, ]

#>   id     item store price quantity revenue
#> 3 30 textbook  TRUE    10       40     400

subset() saves some typing, and also lets you choose columns:

subset(df, price > 5, select = c(item, price))

#>          item price
#> 2         pen     8
#> 3    textbook    10
#> 4 pencil_case     7

The dplyr verb filter() does the same job and fits into pipelines:

df |>
  filter(price > 5)

#>   id        item store price quantity revenue
#> 1 20         pen FALSE     8       35     280
#> 2 30    textbook  TRUE    10       40     400
#> 3 40 pencil_case FALSE     7        5      35

Missing values in conditions

Watch what happens on real data with missing values:

penguins[penguins$body_mass > 6000, ]

#>      species island bill_len bill_dep flipper_len body_mass  sex year
#> NA      <NA>   <NA>       NA       NA          NA        NA <NA>   NA
#> 170   Gentoo Biscoe     49.2     15.2         221      6300 male 2007
#> 186   Gentoo Biscoe     59.6     17.0         230      6050 male 2007
#> NA.1    <NA>   <NA>       NA       NA          NA        NA <NA>   NA

The rows full of NA come from the two penguins whose mass is missing: NA > 6000 is NA, and indexing with NA returns a row of NAs. Two ways out:

penguins[which(penguins$body_mass > 6000), ]

#>     species island bill_len bill_dep flipper_len body_mass  sex year
#> 170  Gentoo Biscoe     49.2     15.2         221      6300 male 2007
#> 186  Gentoo Biscoe     59.6     17.0         230      6050 male 2007

penguins |> filter(body_mass > 6000)

#>   species island bill_len bill_dep flipper_len body_mass  sex year
#> 1  Gentoo Biscoe     49.2     15.2         221      6300 male 2007
#> 2  Gentoo Biscoe     59.6     17.0         230      6050 male 2007

which() turns the logical vector into the positions of the TRUEs, and filter() drops rows where the condition is NA. This is one reason why filter() is the recommended tool for real data.

Common pitfalls

Forgetting the comma. df[1:2] selects columns 1–2 (list-style), df[1:2, ] selects rows 1–2.
Expecting a vector from a tibble. tbl[, 1] returns a one-column tibble; use tbl[[1]] or tbl$name to get the vector.
Conditions with NA. df[df$x > 5, ] returns rows of NA when x is missing; use which() or filter().
Using row positions as identifiers. After sorting or filtering, row 3 is a different unit. Keep an id column.
Columns of the wrong type. Numbers read as text ("3,5") or codes stored as numbers (sex = 1/2) — always check str() or glimpse() first.

8. Check your understanding

Q1. In a hospital dataset each row is one visit, and the same patient can appear several times. What is the statistical unit?

The visit, not the patient. If your question is about patients (e.g. “proportion of patients with diabetes”) you first need one row per patient — for example by summarising visits per patient (see Grouped summaries).

Q2. What do df[2, ], df[, 2] and df[[2]] return for a base data frame?

df[2, ] is the second row (a one-row data frame); df[, 2] is the second column as a vector; df[[2]] is also the second column as a vector. For a tibble, df[, 2] would be a one-column tibble.

Q3. Why does length(penguins) return 8 and not 344?

Because a data frame is a list of columns: length() counts the elements of the list, i.e. the columns. Use nrow() for the number of units.

9. Exercises

Interactive (run in your browser)

★ Exercise 1 — first look. How many penguins and how many variables are in penguins? Which columns are factors? Fill in the blanks.

★ Exercise 2 — indexing. Using [rows, cols], extract (a) the body mass of the 10th penguin; (b) the columns species and bill_len for the first 5 penguins.

★★ Exercise 3 — predict, then debug. A classmate wants all Gentoo penguins heavier than 5500 g, but gets rows full of NA. Fix the code in two different ways (base R and filter()), and count how many penguins satisfy the condition.

heavy <- penguins[which(penguins$species == "Gentoo" &
                          penguins$body_mass > 5500), ]
nrow(heavy)

heavy2 <- penguins |> filter(species == "Gentoo", body_mass > 5500)
nrow(heavy2)

Both give the same rows. In filter(), conditions separated by commas are combined with “and”.

Practice on your own computer

★★ Exercise 4 — build a tidy table. Five students took a quiz. Create a tibble with columns student (names of your choice), score (out of 30) and passed (TRUE if score >= 18, computed from score, not typed by hand). Then add a column grade equal to score / 30 * 100 and print only the students who passed.

Solution

quiz <- tibble(
  student = c("Ada", "Ben", "Chiara", "Dev", "Eli"),
  score   = c(25, 14, 18, 29, 11)
)
quiz$passed <- quiz$score >= 18
quiz$grade  <- quiz$score / 30 * 100
quiz[quiz$passed, ]

The unit is the student; score, passed and grade are variables.

★★ Exercise 5 — what is the unit? For each dataset, name the statistical unit and two variables: (a) airquality; (b) ToothGrowth; (c) mtcars. Use ?name and head() to find out.