---
title: "Pure panels: the longitudinal weight and gross flows"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Pure panels: the longitudinal weight and gross flows}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(weightflow)
set.seed(20260910)
```

A **pure panel** follows one sample over several waves without rotating it. Nobody
enters after the first wave; people only leave. That makes it the right instrument
for questions a series of cross-sections cannot answer -- who moved between states,
how long spells last, whether the same households are poor two years running -- and
it makes its weight a different object from the cross-sectional one, with a
different target population, a different denominator and a different failure mode.

This vignette builds a longitudinal weight over four waves and uses it for gross
flows. The rotating case, where the sample is partly renewed each period and the
quantity of interest is usually the net change, is
`vignette("rotating-panels")`.

## The longitudinal file is an intersection

```{r merge}
waves <- split(panel_puro, panel_puro$wave)
names(waves) <- paste0("T", 1:4)

wide <- panel_merge(waves, by = c("household_id", "person_no"), require = "all")
c(T1_sample = nrow(waves$T1), linked_all_four = nrow(wide))
```

`require = "all"` keeps the units observed in **every** wave; `require = "any"`
would keep the union. Each repeated variable is suffixed by wave, and a
`.wf_in_<wave>` indicator records presence. Adding a wave can only remove units,
never add them, which is the defining property of the design and worth checking as
you go:

```{r shrink}
vapply(2:4, function(k)
  nrow(panel_merge(waves[1:k], by = c("household_id", "person_no"), require = "all")),
  integer(1))
```

## Three ways to be missing, and only one of them is nonresponse

The field disposition in these data distinguishes three things that a single
"missing" flag would collapse:

```{r disposition}
table(wave_4 = wide$disposition_T4)
```

* **`OS` -- out of scope.** The unit left the target population: it died, emigrated,
  the dwelling was demolished. It is not a nonrespondent; it is no longer a member
  of the universe. Its weight must be **removed**, not redistributed, or the
  estimated population grows every wave.
* **`UNK` -- unknown eligibility.** Nobody knows whether the unit is still in scope.
  Treating it as eligible inflates the population; treating it as out of scope
  deflates it. The standard answer is to split it in the observed proportion of
  known-eligible units, which is what `step_unknown_eligibility()` does.
* **`NR` -- nonresponse.** The unit is still in the universe and did not answer. Its
  weight **is** redistributed, among the units that remain.

The order of the steps follows the same logic: leave the universe first, resolve
what is unknown second, and only then adjust for the nonresponse of those who are
certainly in.

```{r cascade}
resp <- vapply(1:4, function(k) wide[[paste0("disposition_T", k)]] == "R", logical(nrow(wide)))
wide$responded_always <- rowSums(resp) == 4

lw <- weighting_spec(wide, base_weights = pw_T1) |>
  step_drop_ineligible(disposition_T4 == "OS", reason = "left the target population") |>
  step_unknown_eligibility(disposition_T4 == "UNK", by = "region_T1") |>
  step_attrition(respondent = responded_always, method = "propensity",
                 formula = ~ age_T1 + sex_T1 + region_T1) |>
  prep()

lw
```

Two things to read there. The **sum of the weights** barely moves at the attrition
step (293,772 to 293,769, one part in a hundred thousand) while the active sample
falls from 1,980 to 1,363: that is what an adjustment by the inverse of an
estimated response propensity is supposed to do, and `weighting_alerts()` says so
out loud if the total moves by more than 5%. The **effective sample size** is the
price: 1,820 to 1,252.

Note also how the respondent condition is defined. "Responded in **all** waves" is
the definition of the longitudinal respondent set: someone who answered three of
four waves contributes no complete trajectory and is a nonrespondent here, however
useful they are for the cross-section.

### Attrition compounds

```{r attrition}
data.frame(through = paste0("T", 1:4),
           complete = vapply(1:4, function(k) sum(rowSums(resp[, 1:k, drop = FALSE]) == k), 0L))
```

Each wave keeps between 84% and 89% of the previous one, so by the fourth only two
of every three initial respondents are left. This is the structural reason a
longitudinal weight carries more adjustment than a cross-sectional one, and why
the propensity model deserves more care: by the fourth wave it is doing the work
of a much larger correction.

## Calibrate to the **first** wave

A panel adds no elements over time, so it is representative only of the population
of the period in which it was selected. ECLAC's household-survey manual (ch. XVI,
sec. B.2.c) is explicit that the auxiliary totals must therefore represent the
population of the **first** period of interest.

```{r calibrate}
t1 <- waves$T1
sex_tab <- data.frame(sex_T1   = names(tapply(t1$pw, t1$sex, sum)),
                      Freq      = as.numeric(tapply(t1$pw, t1$sex, sum)))
reg_tab <- data.frame(region_T1 = names(tapply(t1$pw, t1$region, sum)),
                      Freq      = as.numeric(tapply(t1$pw, t1$region, sum)))

lw <- weighting_spec(wide, base_weights = pw_T1) |>
  step_drop_ineligible(disposition_T4 == "OS", reason = "left the target population") |>
  step_unknown_eligibility(disposition_T4 == "UNK", by = "region_T1") |>
  step_attrition(respondent = responded_always, method = "propensity",
                 formula = ~ age_T1 + sex_T1 + region_T1) |>
  step_calibrate(method = "raking", totals = list(sex_tab, reg_tab), count = "Freq") |>
  prep()

c(calibrated = sum(lw$final_weight), T1_population = sum(t1$pw))
```

(In production those totals are demographic projections; here the design-weighted
totals of the full first wave stand in for them.)

This is the one convention in the whole cascade that **nothing can check for you**.
A vector of control totals carries no label saying which period it belongs to, so
passing the fourth wave's projections converges just as happily, closes just as
cleanly, and produces weights that represent a population the sample never had a
chance to cover. The recipe is auditable; the provenance of the totals is yours.

## Gross flows

A net change is a difference of aggregates: it cannot tell an immobile population
from one where equal numbers enter and leave employment. The longitudinal weight
can.

```{r flows}
STATES <- c("emp", "unemp", "inact")
transition_matrix(lw, from = "lf_status_T1", to = "lf_status_T4",
                  states = STATES, format = "row")
```

`format = "row"` gives the conditional distribution of the destination given the
origin -- each row sums to one. `"joint"` gives the joint distribution and
`"counts"` the weighted counts. For standard errors, resample the whole recipe and
read the transitions off the replicates:

```{r boot}
b <- bootstrap_weights(lw, replicates = 100, strata = "stratum_T1", psu = "psu_T1",
                       seed = 1, progress = FALSE)
boot_transition(b, "lf_status_T1", "lf_status_T4", states = STATES, format = "row")
```

Every replicate re-runs the eligibility drop, the unknown-eligibility split, the
propensity model and the calibration, so the standard errors include the cost of
having estimated the adjustments -- not just the sampling of the units.

`boot_flows()` reports the same information as population totals, adds the **net**
flow `i -> j` minus `j -> i`, and gives the margins: how many started in each
state, how many ended in each, how many stayed and how many moved.

```{r grossflows}
boot_flows(b, "lf_status_T1", "lf_status_T4", states = STATES)
```

A caveat about the illustration, not the method: the shipped panel datasets are
synthetic, and in all of them **inactivity is an absorbing state** -- nobody enters
or leaves it. So the matrices above show movement only between employment and
unemployment, and the `inact` row and column are degenerate. On real microdata the
transitions in and out of inactivity are usually the interesting ones.

## Cross-sectional estimates from a longitudinal file are reference only

They will not reproduce the published cross-sectional figures, and they should not.
The target population of a four-wave panel combination is the set of units that
were in the universe at T1 **and stayed** through T4; the target population of the
T4 cross-section is everyone in the universe at T4, including those who entered
after T1. Two different populations, two different weights, both correct for their
own question. Use the longitudinal weight for flows, transitions and durations, and
the cross-sectional weight for levels.

`step_cross_sectional()` marks a recipe as the second kind, which is what stops the
panel-specific guards from firing on a recipe that is not trying to be
longitudinal.

## Where to look next

`?panel_merge` for the linkage, `?step_attrition` for the adjustment families,
`?step_unknown_eligibility` and `?step_drop_ineligible` for the two things that are
not nonresponse, `?transition_matrix` and `?boot_flows` for the flows, and
`?step_cross_sectional` for the ECLAC conventions in full. The rotating case, net
change and the chained production workflow are in
`vignette("rotating-panels")`.