library(brolgar)

When we first get a longitudinal dataset, you need to understand some of its structure. This vignette demonstrates part of the process of understanding your new longitudinal data.

Setting up your data

To use brolgar with your work, you should convert your longitudinal data into a time series tsibble using the tsibble package. To do so, you need to identify the unique identifying key, and time index. For example:

wages <- as_tsibble(wages,
                    key = id,
                    index = xp,
                    regular = FALSE)

To learn more about longitudinal data as time series, see the vignette: Longitudinal Data Structures.

Basic summaries of the data

When you first get a dataset, you need to get an overall sense of what is in the data.

How many observations are there?

We can kind the number of keys using n_keys():

n_keys(wages)
#> [1] 888

Note that this is a single number, in this case, we have 888 observations.

However, we might want to know how many observations we have for each individual. If we want the number of observations in each variable, then we can use n_obs() with features().

wages %>%
  features(ln_wages, n_obs)
#> # A tibble: 888 × 2
#>       id n_obs
#>    <int> <int>
#>  1    31     8
#>  2    36    10
#>  3    53     8
#>  4   122    10
#>  5   134    12
#>  6   145     9
#>  7   155    11
#>  8   173     6
#>  9   206     3
#> 10   207    11
#> # ℹ 878 more rows

A plot of this can help provide better understanding of the distribution of observations.

library(ggplot2)
wages %>%
  features(ln_wages, n_obs) %>%
  ggplot(aes(x = n_obs)) + 
  geom_bar()

`add_n_obs()`

You can add information about the number of observations for each key with add_n_obs():

wages %>% add_n_obs()
#> # A tsibble: 6,402 x 10 [!]
#> # Key:       id [888]
#>       id    xp n_obs ln_wages   ged xp_since_ged black hispanic high_grade
#>    <int> <dbl> <int>    <dbl> <int>        <dbl> <int>    <int>      <int>
#>  1    31 0.015     8     1.49     1        0.015     0        1          8
#>  2    31 0.715     8     1.43     1        0.715     0        1          8
#>  3    31 1.73      8     1.47     1        1.73      0        1          8
#>  4    31 2.77      8     1.75     1        2.77      0        1          8
#>  5    31 3.93      8     1.93     1        3.93      0        1          8
#>  6    31 4.95      8     1.71     1        4.95      0        1          8
#>  7    31 5.96      8     2.09     1        5.96      0        1          8
#>  8    31 6.98      8     2.13     1        6.98      0        1          8
#>  9    36 0.315    10     1.98     1        0.315     0        0          9
#> 10    36 0.983    10     1.80     1        0.983     0        0          9
#> # ℹ 6,392 more rows
#> # ℹ 1 more variable: unemploy_rate <dbl>

Which you can then use to filter() observations:

library(dplyr)
wages %>% 
  add_n_obs() %>%
  filter(n_obs > 3)
#> # A tsibble: 6,145 x 10 [!]
#> # Key:       id [764]
#>       id    xp n_obs ln_wages   ged xp_since_ged black hispanic high_grade
#>    <int> <dbl> <int>    <dbl> <int>        <dbl> <int>    <int>      <int>
#>  1    31 0.015     8     1.49     1        0.015     0        1          8
#>  2    31 0.715     8     1.43     1        0.715     0        1          8
#>  3    31 1.73      8     1.47     1        1.73      0        1          8
#>  4    31 2.77      8     1.75     1        2.77      0        1          8
#>  5    31 3.93      8     1.93     1        3.93      0        1          8
#>  6    31 4.95      8     1.71     1        4.95      0        1          8
#>  7    31 5.96      8     2.09     1        5.96      0        1          8
#>  8    31 6.98      8     2.13     1        6.98      0        1          8
#>  9    36 0.315    10     1.98     1        0.315     0        0          9
#> 10    36 0.983    10     1.80     1        0.983     0        0          9
#> # ℹ 6,135 more rows
#> # ℹ 1 more variable: unemploy_rate <dbl>

We can also look at the distance between experience, to understand what the distribution of experience is

wages_xp_range <- wages %>% 
  features(xp,
           feat_ranges)

ggplot(wages_xp_range,
       aes(x = range_diff)) + 
  geom_histogram()

We can then explore the range of experience to see what the most common experience is

wages_xp_range %>% 
  count(range_diff) %>% 
  mutate(prop = n / sum(n)) 
#> # A tibble: 829 × 3
#>    range_diff     n    prop
#>         <dbl> <int>   <dbl>
#>  1     0         38 0.0428 
#>  2     0.0150     1 0.00113
#>  3     0.068      1 0.00113
#>  4     0.137      1 0.00113
#>  5     0.153      1 0.00113
#>  6     0.185      1 0.00113
#>  7     0.22       1 0.00113
#>  8     0.225      1 0.00113
#>  9     0.231      1 0.00113
#> 10     0.26       1 0.00113
#> # ℹ 819 more rows

Efficiently exploring longitudinal data

To avoid staring at a plate of spaghetti, you can look at a random subset of the data. Brolgar provides some intuitive functions to help with this.

`sample_n_keys()`

In dplyr, you can use sample_n() to sample n observations. Similarly, with brolgar, you can take a random sample of n keys using sample_n_keys():

set.seed(2019-7-15-1300)
wages %>%
  sample_n_keys(size = 10) %>%
  ggplot(aes(x = xp,
             y = ln_wages,
             group = id)) + 
  geom_line()

Filtering observations

You can combine sample_n_keys() with add_n_obs() and filter() to only show keys with many observations:

library(dplyr)
wages %>%
  add_n_obs() %>%
  filter(n_obs > 5) %>%
  sample_n_keys(size = 10) %>%
  ggplot(aes(x = xp,
             y = ln_wages,
             group = id)) + 
  geom_line()

(Note: sample_frac_keys(), which samples a fraction of available keys.)

Now, how do you break these into many plots?

Clever facets: `facet_strata`

brolgar provides some clever facets to help make it easier to explore your data. facet_strata() splits the data into 12 groups by default:

set.seed(2019-07-23-1936)
library(ggplot2)
ggplot(wages,
       aes(x = xp,
           y = ln_wages,
           group = id)) +
  geom_line() +
  facet_strata()

But you could ask it to split the data into a more groups

set.seed(2019-07-25-1450)
library(ggplot2)
ggplot(wages,
       aes(x = xp,
           y = ln_wages,
           group = id)) +
  geom_line() +
  facet_strata(n_strata = 20)

Getting Started