Strings, factors and dates

Clean text, tame categories and compute with calendar dates

Essential stringr functions and a first look at regular expressions, a forcats recap for cleaning categorical variables, and lubridate for parsing dates, extracting their parts and computing differences — ending with a full cleaning example.

Level 1 · Foundations ⏱ ~75 min Prerequisites: Factors, Data manipulation with dplyr

NoteLearning objectives

By the end of this lesson you will be able to:

  • clean text with str_trim(), str_squish(), str_to_lower(), str_to_title(), str_replace() and str_sub();
  • detect and extract patterns with str_detect() and str_split(), using five basic regular expression patterns;
  • build text from data with str_glue();
  • fix inconsistent categories with forcats (fct_recode(), fct_collapse(), fct_lump_n(), fct_infreq());
  • parse dates with ymd() / dmy(), extract year, month and weekday, compute differences and round dates with floor_date().
library(dplyr)
library(tibble)
library(stringr)
library(forcats)
library(lubridate)

1. Why clean strings and dates?

Data typed by people are messy. The same city appears as "Rome", "rome" and "Rome " (with a trailing space). Prices are stored as text like "€12.50". Dates arrive as "05/03/2024" — is that 5 March or 3 May?

For R, "Rome" and "Rome " are different values: a frequency table will count them separately, a join will fail to match them, and a model will treat them as different categories. Cleaning text and dates is therefore not cosmetic: it decides which units are counted together, and so it changes the statistics.

All stringr functions start with str_, take the string(s) as the first argument and are vectorised: they work on a whole column at once.

2. Cleaning text with stringr

x <- c("  Rome", "rome ", "ROME", "Milan  centre")

str_trim(x)          # remove spaces at both ends
#> [1] "Rome"          "rome"          "ROME"          "Milan  centre"
str_squish(x)        # also collapse internal repeated spaces
#> [1] "Rome"         "rome"         "ROME"         "Milan centre"
str_to_lower(x)      # lower case
#> [1] "  rome"        "rome "         "rome"          "milan  centre"
str_to_title(str_squish(x))   # Title Case
#> [1] "Rome"         "Rome"         "Rome"         "Milan Centre"
str_length(x)        # number of characters (spaces count!)
#> [1]  6  5  4 13

str_length() shows why " Rome" and "Rome" do not match: they have a different number of characters.

Replacing and extracting parts

prices <- c("€12.50", "€7.00", "€30.10")

str_replace(prices, "€", "")            # replace the first match
#> [1] "12.50" "7.00"  "30.10"
str_remove(prices, "€") |> as.numeric() # shortcut + convert to number
#> [1] 12.5  7.0 30.1
codes <- c("IT-2024-001", "FR-2023-017", "ES-2024-112")
str_sub(codes, 1, 2)     # characters 1 to 2
#> [1] "IT" "FR" "ES"
str_sub(codes, -3)       # the last three characters
#> [1] "001" "017" "112"

str_replace() changes only the first match in each string; str_replace_all() changes all of them. Negative positions in str_sub() count from the end.

Splitting and combining

str_split("Adelie,Chinstrap,Gentoo", ",")
#> [[1]]
#> [1] "Adelie"    "Chinstrap" "Gentoo"
str_split_i(codes, "-", 2)     # the 2nd piece of each string
#> [1] "2024" "2023" "2024"

str_split() returns a list (each string can produce a different number of pieces); str_split_i() returns just the i-th piece as a vector. For splitting a column into several columns of a data frame use separate_wider_delim() from Tidy data and reshaping.

str_glue() builds text by inserting values inside { }:

penguins |>
  summarise(m = mean(body_mass, na.rm = TRUE), .by = species) |>
  mutate(label = str_glue("{species}: {round(m)} g"))
#>     species    m             label
#> 1    Adelie 3701    Adelie: 3701 g
#> 2    Gentoo 5076    Gentoo: 5076 g
#> 3 Chinstrap 3733 Chinstrap: 3733 g

This is handy for plot labels and automated reports.

3. A first look at regular expressions

str_detect(string, pattern) returns TRUE/FALSE for each string. The pattern is a regular expression (regex): a small language for describing text. Five patterns cover most everyday needs:

fruits <- c("apple", "banana", "kiwi 2", "Cherry", "pear.", "blueberry")
Pattern Meaning Matches in fruits
"^b" starts with b banana, blueberry
"y$" ends with y Cherry, blueberry
"\\d" any digit kiwi 2
"[aeiou]{2}" two vowels in a row pear., blueberry
"\\." a literal dot pear. 
str_detect(fruits, "^b")
#> [1] FALSE  TRUE FALSE FALSE FALSE  TRUE
fruits[str_detect(fruits, "y$")]
#> [1] "Cherry"    "blueberry"
str_subset(fruits, "\\d")          # shortcut for x[str_detect(x, p)]
#> [1] "kiwi 2"
str_subset(fruits, "[aeiou]{2}")
#> [1] "pear."     "blueberry"
str_subset(fruits, "\\.")
#> [1] "pear."
str_subset(fruits, "an|err")      # | means "or"
#> [1] "banana"    "Cherry"    "blueberry"

Two details you will need often:

  • . means any character; to match a real dot write "\\." (the backslash “escapes” the special meaning, and in an R string it must itself be doubled).
  • Matching is case sensitive: "^c" does not match "Cherry". Lower-case the text first, or use regex("^c", ignore_case = TRUE).
str_extract(c("kiwi 2", "pack of 12 eggs", "none"), "\\d+")
#> [1] "2"  "12" NA

+ means “one or more of the previous”, so "\\d+" extracts a whole number. Strings without a match give NA.

Pattern Meaning
. any character
\\d, \\s, \\w digit, white space, “word” character (letter, digit, _)
[abc], [^abc], [a-z] one of / none of / a range
*, +, ?, {n,m} 0 or more, 1 or more, 0 or 1, between n and m
^, $ start, end of the string
(...) a group (used by str_match() and names_pattern)

The vertical bar means “or”: "cat|dog" matches either word.

stringr uses the ICU regex engine. The R4DS chapter “Regular expressions” is a gentle and complete introduction.

4. Categorical variables: a forcats recap

Once a text column is clean, it is usually a categorical variable and should become a factor. The forcats functions all start with fct_:

city <- c("Rome", "Milan", "Milano", "Roma", "Turin", "Rome", "Naples")

fct_recode(city, Milan = "Milano", Rome = "Roma") |> fct_count()
#> # A tibble: 4 × 2
#>   f          n
#>   <fct>  <int>
#> 1 Milan      2
#> 2 Naples     1
#> 3 Rome       3
#> 4 Turin      1

fct_recode(new = "old") renames individual levels. To merge several levels into groups, use fct_collapse():

fct_collapse(city,
  North = c("Milan", "Milano", "Turin"),
  Centre = c("Rome", "Roma"),
  South = "Naples"
) |> fct_count()
#> # A tibble: 3 × 2
#>   f          n
#>   <fct>  <int>
#> 1 North      3
#> 2 South      1
#> 3 Centre     3

Two functions are especially useful before tables and bar charts:

models <- word(rownames(mtcars), 1)   # first word = brand
fct_infreq(models) |> levels() |> head()        # order levels by frequency
#> [1] "Merc"   "Fiat"   "Hornet" "Mazda"  "Toyota" "AMC"
fct_lump_n(models, n = 3) |> fct_count(sort = TRUE)
#> # A tibble: 6 × 2
#>   f          n
#>   <fct>  <int>
#> 1 Other     17
#> 2 Merc       7
#> 3 Fiat       2
#> 4 Hornet     2
#> 5 Mazda      2
#> 6 Toyota     2

fct_infreq() orders levels from most to least frequent; fct_lump_n(n = 3) keeps the three most frequent levels (plus ties) and merges the rest into "Other". Missing values can be made an explicit level with fct_na_value_to_level().

5. Dates with lubridate

Parsing: from text to Date

Base R’s as.Date() expects the ISO format "YYYY-MM-DD". With other formats it can silently go wrong:

as.Date("05/03/2024")
#> [1] "0005-03-20"

R read this as the year 5! lubridate’s parsing functions are named after the order of the components: ymd(), dmy(), mdy(), … They accept any common separator.

ymd("2024-03-05")
#> [1] "2024-03-05"
dmy(c("05/03/2024", "5-3-2024", "05.03.2024"))
#> [1] "2024-03-05" "2024-03-05" "2024-03-05"
mdy("March 5, 2024")
#> [1] "2024-03-05"

Choose the function from how the data were recorded: European data are usually dmy, US data mdy. Strings that cannot be parsed become NA, with a warning.

Extracting components

d <- ymd(c("2024-03-05", "2024-07-14", "2024-12-25"))

year(d)
#> [1] 2024 2024 2024
month(d)
#> [1]  3  7 12
month(d, label = TRUE)                  # an ordered factor
#> [1] mar lug dic
#> 12 Levels: gen < feb < mar < apr < mag < giu < lug < ago < set < ... < dic
wday(d, label = TRUE, week_start = 1)   # weekday, weeks starting on Monday
#> [1] mar dom mer
#> Levels: lun < mar < mer < gio < ven < sab < dom
yday(d)                                 # day of the year
#> [1]  65 196 360

month(label = TRUE) and wday(label = TRUE) return ordered factors, so tables and plots show months in calendar order rather than alphabetically.

Differences and rounding

start <- ymd("2024-03-05")
end   <- ymd("2024-07-17")

end - start                         # a difftime, in days
#> Time difference of 134 days
as.numeric(end - start)             # just the number
#> [1] 134
time_length(interval(start, end), "months")
#> [1] 4.387
start + days(30)                    # arithmetic with periods
#> [1] "2024-04-04"

floor_date() rounds dates down to a unit — the key tool for summarising by week or month:

floor_date(d, "month")
#> [1] "2024-03-01" "2024-07-01" "2024-12-01"
floor_date(d, "week", week_start = 1)
#> [1] "2024-03-04" "2024-07-08" "2024-12-23"

Applied: building dates in airquality

airquality stores the date in two columns, Month and Day, of the year 1973. make_date() combines them:

aq <- airquality |>
  as_tibble() |>
  mutate(date = make_date(1973, Month, Day),
         weekday = wday(date, label = TRUE, week_start = 1))

aq |>
  summarise(mean_temp = mean(Temp), n = n(), .by = weekday) |>
  arrange(weekday)
#> # A tibble: 7 × 3
#>   weekday mean_temp     n
#>   <ord>       <dbl> <int>
#> 1 lun         77.90    21
#> 2 mar         79.14    22
#> 3 mer         77.23    22
#> 4 gio         78.59    22
#> 5 ven         77.18    22
#> 6 sab         77.64    22
#> 7 dom         77.5     22
TipHow to read this output
  • Because weekday is an ordered factor, arrange() sorts it from Monday to Sunday.
  • The mean temperatures are very similar across weekdays (between about 77 and 79 °F): no surprise, since the weather does not know which day of the week it is. A weekday effect would be expected for traffic pollution, not for temperature.

6. Applied mini-example: cleaning a messy table

A small survey was typed in by hand:

messy <- tibble(
  name   = c("  ada LOVELACE", "Ben  Smith ", "chiara ROSSI",
             "Dev patel", "eli  MARTIN"),
  city   = c("rome", "Rome ", "MILAN", "milano", "Rome"),
  phone  = c("+39 333 123 4567", "333-987-6543", "(333) 555 0000",
             "3331112222", NA),
  signup = c("05/03/2024", "5-4-2024", "17.06.2024", "01/08/2024",
             "30/09/2024"),
  spent  = c("€12.50", "€ 7", "30,00 €", "15.2", "€0")
)
messy
#> # A tibble: 5 × 5
#>   name             city     phone            signup     spent  
#>   <chr>            <chr>    <chr>            <chr>      <chr>  
#> 1 "  ada LOVELACE" "rome"   +39 333 123 4567 05/03/2024 €12.50 
#> 2 "Ben  Smith "    "Rome "  333-987-6543     5-4-2024   € 7    
#> 3 "chiara ROSSI"   "MILAN"  (333) 555 0000   17.06.2024 30,00 €
#> 4 "Dev patel"      "milano" 3331112222       01/08/2024 15.2   
#> 5 "eli  MARTIN"    "Rome"   <NA>             30/09/2024 €0

Every column has a problem: inconsistent case and spaces, two spellings of Milan, phone numbers in different formats, dates with different separators, amounts with currency symbols and a decimal comma. One pipeline fixes them all:

clean <- messy |>
  mutate(
    name   = name |> str_squish() |> str_to_title(),
    city   = city |> str_trim() |> str_to_title() |>
      fct_recode(Milan = "Milano"),
    phone  = phone |> str_remove_all("[^0-9]") |> str_sub(-10),
    signup = dmy(signup),
    spent  = spent |> str_replace(",", ".") |>
      str_remove_all("[^0-9.]") |> as.numeric()
  )
clean
#> # A tibble: 5 × 5
#>   name         city  phone      signup     spent
#>   <chr>        <fct> <chr>      <date>     <dbl>
#> 1 Ada Lovelace Rome  3331234567 2024-03-05  12.5
#> 2 Ben Smith    Rome  3339876543 2024-04-05   7  
#> 3 Chiara Rossi Milan 3335550000 2024-06-17  30  
#> 4 Dev Patel    Milan 3331112222 2024-08-01  15.2
#> 5 Eli Martin   Rome  <NA>       2024-09-30   0
TipHow to read this output
  • name: extra spaces removed and Title Case applied.
  • city: a factor with 2 levels instead of 5 different spellings.
  • phone: "[^0-9]" means “any character that is not a digit”; removing them and keeping the last 10 digits drops the country code +39. The missing phone stays NA — cleaning must not invent data.
  • signup: a real Date, so we can compute with it.
  • spent: a number, so we can summarise it.

Now the data can be analysed:

clean |>
  summarise(
    customers   = n(),
    total_spent = sum(spent),
    first_signup = min(signup),
    .by = city
  )
#> # A tibble: 2 × 4
#>   city  customers total_spent first_signup
#>   <fct>     <int>       <dbl> <date>      
#> 1 Rome          3        19.5 2024-03-05  
#> 2 Milan         2        45.2 2024-06-17
clean |>
  mutate(signup_month = month(signup, label = TRUE)) |>
  count(signup_month)
#> # A tibble: 5 × 2
#>   signup_month     n
#>   <ord>        <int>
#> 1 mar              1
#> 2 apr              1
#> 3 giu              1
#> 4 ago              1
#> 5 set              1

On the raw messy table, the first summary would have shown five “cities” and could not have computed a total.

WarningCommon pitfalls
  1. Invisible spaces. "Rome " ≠ "Rome". Use str_trim()/str_squish() on every text column that will be grouped or joined.
  2. Forgetting to escape special characters in regex: "." matches anything, "\\." matches a dot; "$" means “end of string”, so to match a dollar sign write "\\$".
  3. Parsing dates with the wrong order. "05/03/2024" is 5 March with dmy() and 3 May with mdy(). Both “work”; only one is right. Check with a date whose day is above 12.
  4. as.Date() on non-ISO text can silently produce absurd dates (year 5). Check range() of every new date column.
  5. Cleaning that invents information: filling a missing phone or guessing an ambiguous date. Leave genuine unknowns as NA.

7. Check your understanding

Whether each string starts with an upper-case letter. ^ anchors the pattern at the start; [A-Z] is one character between A and Z.

month(label = TRUE) returns an ordered factor, so the table lists months in calendar order (Jan, Feb, …). format() returns plain text, which is sorted alphabetically (Apr, Aug, Dec, …).

Lower-case the text with str_to_lower(), then fct_collapse(x, male = c("m", "male"), female = c("f", "female")).

8. Exercises

Interactive (run in your browser)

★ Exercise 1 — detect. models contains the 32 car names of mtcars. How many are Mercedes (they start with "Merc")? How many names contain at least one digit?

“Starts with” is ^; “a digit” is \\d. sum() of a logical vector counts the TRUEs.

sum(str_detect(models, "^Merc"))   # 7
str_subset(models, "\\d")          # 17 names, from "Mazda RX4" to "Volvo 142E"

★★ Exercise 2 — brands. Extract the brand (the first word) of each car with str_split_i(), and show the three most frequent brands, lumping the others into "Other".

Split on a space " " and take piece 1.

brand <- str_split_i(models, " ", 1)
fct_lump_n(brand, n = 3) |> fct_count(sort = TRUE)

Merc is by far the most frequent brand (7 cars). Several brands are tied with 2 cars, and fct_lump_n() keeps all the tied levels, so you may see more than three named levels before "Other".

★★ Exercise 3 — debug the dates. The code below should compute how many days each exam came after the start of the course, but the results are absurd. Find the problem and fix it with lubridate.

The dates are day/month/year. Replace as.Date() with dmy() in both places.

course_start <- dmy("04/03/2024")

exams |>
  mutate(date = dmy(date),
         days_after = as.numeric(date - course_start))

as.Date() read "17/06/2024" as the year 17, so all differences were meaningless. With dmy() the exams come 105, 120 and 133 days after the course started.

★★★ Exercise 4 — weekly ozone. Build a date column for airquality with make_date(), then compute the mean Ozone per week (use floor_date() with weeks starting on Monday). Which week had the highest mean ozone?

make_date(1973, Month, Day) and the unit "week".

airquality |>
  mutate(date = make_date(1973, Month, Day),
         week = floor_date(date, "week", week_start = 1)) |>
  summarise(mean_ozone = mean(Ozone, na.rm = TRUE),
            n_obs = sum(!is.na(Ozone)),
            .by = week) |>
  arrange(desc(mean_ozone))

The top of the list is the week starting 25 June 1973 — but look at n_obs: its mean is based on a single day, because most June ozone values are missing. Among weeks with at least five observations, the highest means are in August (weeks starting 27 and 6 August). A mean based on one value is not a reliable summary of a week; always report n next to it.

Practice on your own computer

★★ Exercise 5 — your own messy table. Create a tibble with 6 rows and the columns sex (values like "M", " f", "Female", "male "), birth (dates written as "1999-03-12", "2001/11/05", …) and height (text like "1,72 m", "1.80m"). Clean it so that sex is a factor with two levels, birth is a Date, height is numeric in centimetres, and add the age in whole years on 1 January 2025.

people <- tibble(
  sex    = c("M", " f", "Female", "male ", "F", "m"),
  birth  = c("1999-03-12", "2001/11/05", "2000-01-30",
             "1998.07.21", "2002-12-01", "1997/05/17"),
  height = c("1,72 m", "1.80m", "1,65 m", "1.91 m", "1.58m", "1,77m")
)

people |>
  mutate(
    sex = sex |> str_trim() |> str_to_lower() |>
      fct_collapse(male = c("m", "male"), female = c("f", "female")),
    birth  = ymd(birth),
    height = height |> str_replace(",", ".") |>
      str_remove_all("[^0-9.]") |> as.numeric() * 100,
    age = floor(time_length(interval(birth, ymd("2025-01-01")), "years"))
  )

Check the result with glimpse(): sex should be <fct>, birth <date>, height and age <dbl>.

Summary

Task Function(s)
Remove spaces str_trim(), str_squish()
Change case str_to_lower(), str_to_upper(), str_to_title()
Find patterns str_detect(), str_subset(), str_extract()
Replace / remove str_replace(), str_replace_all(), str_remove_all()
Pieces of strings str_sub(), str_split(), str_split_i()
Build text str_glue()
Fix categories fct_recode(), fct_collapse(), fct_lump_n(), fct_infreq()
Parse dates ymd(), dmy(), mdy(), make_date()
Date parts year(), month(label = TRUE), wday(label = TRUE), yday()
Date arithmetic d2 - d1, time_length(interval(d1, d2), "years"), days(), floor_date()
  • Text values that look the same may not be equal: trim, squish and fix the case before grouping or joining.
  • Five regex patterns (^, $, \\d, [...], |) cover most cleaning tasks; escape special characters with \\.
  • Parse dates with the function that matches the order in which they were recorded, then check their range.
  • Cleaning decides which units are counted together: do it explicitly, in code, and never invent values.
Back to top