Visualizing Normed Score Precision with normref

Klazien de Vries, Casper J. Albers, and Marieke E. Timmerman

2026-11-20

Introduction

The normref package provides tools for continuous norming of psychological test scores using GAMLSS models. In continuous norming, norms are estimated as smooth functions of a continuous norm predictor, typically age. This contrasts to classical norming, where norms are estimated separately within discrete age groups. Applying these norms results in standardized scores such as percentiles, \(z\)-scores, T-scores, or IQ scores, allowing an individual’s performance to be interpreted relative to the reference population of the same age.

Because the norms are estimated from a finite normative sample, every resulting normed score has sampling uncertainty. This vignette describes bootstrap methods for studying that uncertainty at the subtest and composite levels with the normref package. The vignette demonstrates how to:

The examples use subtest data from the Intelligence and Development Scales–2 (IDS-2), included in normref. Three cases are considered:

  1. Case 1: Precision of Subtest Norms
  2. Case 2: Precision of Composite Norms
  3. Case 3: Sample Size Planning for Subtests

Bootstrap computations are time-consuming, so the results are precomputed and stored in the package.

Bootstrap implementation

normref implements bootstrap procedures at two levels:

Both functions return two complementary types of results:

For subtests, normtable-level and sample-level results often have similar interpretations. For composites, they do not. Composite sample-level results propagate uncertainty from the subtest norms into each person’s composite score. Composite normtable-level results condition on a fixed composite score; therefore, they do not represent the full uncertainty of an observed person’s composite score. In this application, the composite $normtable result is best treated as a sanity check on the stability of the mapping of the model to the composite, not as the primary estimate of person-level diagnostic uncertainty. The composite $sample result is the more relevant quantity for the interpretation of the precision of an individual’s score.

Nonparametric and parametric bootstrap

For subtests, boot_subtest() supports two bootstrap types:

Use the nonparametric bootstrap for post-hoc precision analysis when norming data are available. Use the parametric bootstrap primarily for simulation-based planning. Cases 1 and 2 use the nonparametric bootstrap. Case 3 uses the parametric bootstrap.


Case 1: Precision of subtest norms

Model fitting

The following example uses the Naming Categories subtest (y7) from the IDS-2 data (ids_data). The response is bounded between 0 and 34. The nonparametric bootstrap depends on the collected normative sample and the selected model. Therefore, we first perform model selection on the collected sample. For a more detailed explanation of model selection with normref, see the normref package introduction vignette.

We begin by inspecting the relationship between age and raw test scores.

# Load data
data(ids_data)

# Visualize scores
plot(ids_data$age, ids_data$y7,
     xlab = "Age",
     ylab = "Raw test score",
     pch = 15,
     cex = 0.5,
     col = gray(0.7))

The raw scores show a clear age-related trend, with both the location and spread of the scores changing across the age range. This motivates the use of flexible continuous norming models with GAMLSS.

Model fitting

Candidate distributional families can include the discrete betabinomial (BB) and continuous distributions, such as a truncated normal or truncated Box–Cox Cole Green (BCCG) distribution. The betabinomial distribution is directly suitable for discrete (bounded count) responses, but the NO and BCCG assume a continuous response. For relatively wide test score ranges, continuous distributions can provide a good approximation to the discrete test scores. To account for the test score bounds when using continuous distributions, we truncate those distributions at the observed minimum and maximum.

# BB
mydata.BB <- shape_data(
  ids_data,
  age_name = "age",
  score_name = "y7",
  family = "BB"
)

mod.BB <- fb_select(
  data = mydata.BB,
  age_name = "age",
  score_name = "shaped_score",
  family = "BB",
  selcrit = "BIC"
)
#> FB-select iteration 0: Polynomial degrees = 2,1 : BIC = 9138.345
#> FB-select iteration 1: Polynomial degrees = 2,2 : BIC = 9099.919

# NOtr
gamlss.tr::gen.trun(par = c(0,34), family = "NO", name = 'tr34', type = "both")
#> A truncated family of distributions from NO has been generated 
#>  and saved under the names:  
#>  dNOtr34 pNOtr34 qNOtr34 rNOtr34 NOtr34 
#> The type of truncation is both 
#>  and the truncation parameter is 0 34

mydata.NOtr34 <- shape_data(
  ids_data,
  age_name = "age",
  score_name = "y7",
  family = "NOtr34"
)

mod.NOtr34 <- fb_select(
  data = mydata.NOtr34,
  age_name = "age",
  score_name = "shaped_score",
  family = "NOtr34",
  selcrit = "BIC"
)
#> FB-select iteration 0: Polynomial degrees = 2,1 : BIC = 9091.904

# BCCGtr
gamlss.tr::gen.trun(par = c(34), family = "BCCG", name = 'tr34', type = "right")
#> A truncated family of distributions from BCCG has been generated 
#>  and saved under the names:  
#>  dBCCGtr34 pBCCGtr34 qBCCGtr34 rBCCGtr34 BCCGtr34 
#> The type of truncation is right 
#>  and the truncation parameter is 34

mydata.BCCGtr34 <- shape_data(
  ids_data,
  age_name = "age",
  score_name = "y7",
  family = "BCCGtr34"
)

mod.BCCGtr34 <- fb_select(
  data = mydata.BCCGtr34,
  age_name = "age",
  score_name = "shaped_score",
  family = "BCCGtr34",
  selcrit = "BIC"
)
#> FB-select iteration 0: Polynomial degrees = 2,1,0 : BIC = 9071.163
#> FB-select iteration 1: Polynomial degrees = 3,1,0 : BIC = 9069.03

# Compare model fit in terms of BIC
BIC(mod.BB, mod.NOtr34, mod.BCCGtr34)%>%arrange(BIC)
#>              df      BIC
#> mod.BCCGtr34  7 9069.030
#> mod.NOtr34    5 9091.904
#> mod.BB        6 9099.919

In this example, we select the truncated BCCG model because it has the lowest BIC. The BIC is the recommended model selection criterion for continuous norming with GAMLSS in (Voncken, Albers, and Timmerman 2019). A low BIC indicates that a model has a good balance between model fit and complexity. However, it does not, by itself, establish that a model is adequate. Worm plots, centile curves, and residual diagnostics should also be inspected. In this example, these diagnostics (not) indicate that the truncated BCCG model has better model fit than the the NO of BB distributions. This is not surprising, because the truncated BCCG, unlike the NO and BB distributions, additionally allows test score skewness to vary with age.

After selecting the model, we construct a normtable on an age–score grid using normtable_create().

normtable1 <- normtable_create(
  mod.BCCGtr34,
  mydata.BCCGtr34,
  normtype = "IQ",
  min_age = 5,
  max_age = 20,
  age_name = "age",
  score_name = "shaped_score"
)

normmat <- normtable1$norm_matrix

normmat %>%
  filter(age == 10)
#>   age  1  2  3        4        5        6        7        8        9      10
#> 1  10 55 55 55 57.06819 59.71555 62.29403 64.88999 67.55676 70.32859 73.2276
#>         11       12       13       14      15       16       17       18
#> 1 76.26766 79.45698 82.79983 86.29791 89.9512 93.75862 97.71845 101.8286
#>        19       20       21      22       23       24       25       26  27  28
#> 1 106.087 110.4911 115.0388 119.728 124.5563 129.5218 134.6225 139.8566 145 145
#>    29  30  31  32  33  34
#> 1 145 145 145 145 145 145

At age 10, a raw score of 21 corresponds to an IQ score of approximately 115. That value is estimated from a finite normative sample and therefore has sampling variability. We will use boot_subtest() to provide estimates of the precision around the estimated normed scores in the normtable.

Running boot_subtest()

The function boot_subtest() estimates the precision of normed scores, given a certain selected GAMLSS distribution. In each bootstrap replicate, boot_subtest() refits the model on a bootstrap sample and reconstructs the norm table. The variation across replicates is used to estimate the standard error of the normed score estimates.

The function boot_subtest() follows the same workflow and grid specifications as normtable_create(): normtype specifies the reporting scale ("Z", "T", or "IQ"). The age and score ranges and their grid spacing can be controlled with min_age, max_age, min_score, max_score, step_size_age, and step_size_score. Scores beyond a selected z threshold can be excluded with trim.

An important option is reselect: with reselect = FALSE (the default), the polynomial structure selected in the original model is held fixed, while model parameters and smoothing parameters are re-estimated in each replicate. With reselect = TRUE, polynomial degree selection is repeated in each replicate, thereby incorporating model-selection variability. Additional fb_select() options can be supplied through fb_control. Because repeated fitting can be computationally intensive, parallel = TRUE can distribute replicates over CPU cores.

Below, we run B = 1000 bootstrap replicates without and B = 2000 with model reselection.

# Bootstrap without model reselection, IQ-scale
boot.BCCGtr34 <- boot_subtest(
  model = mod.BCCGtr34,
  age_name = "age",
  normtype = "IQ",
  B = 1000,
  min_age = 5,
  max_age = 20,
  parallel = TRUE,
  seed = 1
)


# Bootstrap with model reselection, Z-scale
boot.BCCGtr34.reselect <- boot_subtest(
  model = mod.BCCGtr34,
  age_name = "age",
  normtype = "Z",
  B = 2000,
  min_age = 5,
  max_age = 20,
  parallel = TRUE,
  reselect = TRUE,
  fb_control = list(method = "RS(30)"), # limit the maximum amount of iterations per fit to speed up calculations
  seed = 2
)

The result contains bootstrap standard errors and percentile confidence intervals in $normtable and $sample. For instance, inspecting $normtable around age 10 shows the original point estimate alongside the bootstrap standard error and bias-corrected percentile confidence intervals. When the corresponding normed score has been trimmed, these values are set to NA.

# Table-level grid results
boot.BCCGtr34$normtable %>%
  filter(age == 10) %>%
  filter(`Subtest Score` %in% c(1,20,21,22,34))
#>   age Subtest Score original boot_mean boot_median   boot_se boot_ci_lower
#> 1  10             1  55.0000        NA          NA        NA            NA
#> 2  10            20 110.4911  110.5003    110.5106 0.6282775      109.2820
#> 3  10            21 115.0388  115.0606    115.0628 0.7032947      113.6734
#> 4  10            22 119.7280  119.7646    119.7599 0.8011500      118.1955
#> 5  10            34 145.0000        NA          NA        NA            NA
#>   boot_ci_upper boot_n
#> 1            NA     NA
#> 2      111.7144   1000
#> 3      116.3869   1000
#> 4      121.3818   1000
#> 5            NA     NA

# Sample-level individual results
head(boot.BCCGtr34$sample)
#>   id       age Subtest Score  original boot_mean boot_median   boot_se
#> 1  1 10.971461            17  92.13566  92.11728    92.11260 0.4909682
#> 2  2 11.674196            20  99.35255  99.34844    99.34471 0.5238299
#> 3  3 11.444216            22 108.56602 108.58034   108.58120 0.5849699
#> 4  4 10.049315            14  86.05549  86.02867    86.03428 0.5256726
#> 5  5 14.182067            20  87.97204  87.96295    87.94188 0.6267867
#> 6  6  6.554415            11  94.57839  94.57396    94.59500 0.6801125
#>   boot_ci_lower boot_ci_upper boot_n
#> 1      91.13268      93.12665   1000
#> 2      98.37400     100.36799   1000
#> 3     107.42885     109.75569   1000
#> 4      84.91009      87.04666   1000
#> 5      86.79067      89.29659   1000
#> 6      93.22407      95.90411   1000

Inspecting Monte Carlo error

A bootstrap standard error is itself estimated from the finite number of bootstrap replicates. Therefore, sufficiently many bootstrap replicates are needed to produce precise bootstrap standard errors. For bootstrap estimates of the standard error, Efron and Tibshirani (1993) indicate \(B=200\) replicates are typically enough. When using the bootstrap to construct confidence intervals however ($boot_ci_lower and $boot_ci_upper), many more replicates are needed (e.g., at least \(B=1,000\)). Furthermore, tail regions and analyses with reselect = TRUE may require more replicates. Therefore, we generally recommend setting \(B\) as high as computationally feasible (e.g., \(B \geq 1,000\) for standard errors, \(B \geq 10,000\) for confidence intervals, see Hesterberg (2015)).

Additionally, in $mcse_normtable and $mcse_sample, we can inspect the Monte Carlo standard errors (MCSE) of the bootstrap standard errors in both the normtables and the sample. The MCSEs ($boot_mcse_se) give an estimate of the precision of the bootstrap estimates themselves, and their use is recommended in Morris, White, and Crowther (2019). Additionally, Efron and Tibshirani (1993) recommend investigating the Coefficient of Variation (CV) ($boot_cv_se), which divides the MCSE by the value of the estimate itself. The CV of the bootstrap standard error is dimensionless, which is typically more informative, but it can fail where the bootstrap standard error is close to zero. A \(CV \le 5\%\) corresponds roughly to \(B \ge 200\) and can be suitable for preliminary exploration.

# Inspect relative MCSE of the bootstrap SE
boxplot(boot.BCCGtr34$mcse_normtable$boot_cv_se,
     main = "Distribution of Relative MCSE",
     ylab = "Relative MCSE of Bootstrap SE",
     col = "gray85")


# Inspect largest relative mcse
head(boot.BCCGtr34$mcse_normtable%>%arrange(desc(boot_cv_se)))
#>        age Subtest Score original  boot_se boot_mcse_se boot_cv_se
#> 1 14.69697             7 55.03709 1.437804   0.03779242 0.02628482
#> 2 14.54545             7 55.30689 1.415692   0.03718982 0.02626972
#> 3 13.48485             6 55.22510 1.332115   0.03497355 0.02625414
#> 4 14.39394             7 55.57925 1.394274   0.03659542 0.02624694
#> 5 14.24242             7 55.85420 1.373544   0.03600928 0.02621634
#> 6 14.09091             7 56.13178 1.353495   0.03543154 0.02617781

Model Selection History

When reselect = TRUE, we can inspect the frequency of selected polynomial structures, both in a table $model_freq_table and for each individual replicate ($model_selection_history).

boot.BCCGtr34.reselect$model_freq_table
#> 
#> 3-1-0 2-1-0 4-1-0 5-1-0 2-3-0 2-4-0 1-0-1 4-0-0 2-0-0 2-5-0 4-0-2 2-1-1 3-3-0 
#>   926   617   290    59    39    39     4     4     3     3     3     2     2 
#> 5-0-0 1-0-0 2-0-1 2-4-1 3-0-0 3-0-2 3-1-2 4-1-2 
#>     2     1     1     1     1     1     1     1

This provides insight into the stability of the selected polynomial structure across bootstrap replicates.

Precision maps

Once the bootstrap procedure has produced estimates of uncertainty across the age–score grid, the results can be visually summarized in a precision heatmap with boot_heatmap().

The heatmap displays a grid where each block corresponds to a specific age–score combination. The blocks are shaded according to the bootstrap standard error of the standardized score. Yellower/lighter colors indicate smaller standard errors (higher precision) and purpler/darker colors indicate larger standard errors. Centile curves and observed data can be overlaid using centiles and add_points = TRUE.

boot_heatmap(
  boot.BCCGtr34,
  type = "se",
  add_points = TRUE,
  centiles = c(1, 5, 25, 50, 75, 95, 99),
  centile_label_nudge = 2.0
)

The precision map provides an overview of the norm stability across the age–score space. Scores outside the trimming threshold are NA and appear blank, giving the heatmap a jagged boundary. Standard errors are commonly smallest near the centre of the distribution and larger in the tails. They may also increase near the age boundaries, because the polynomial and spline curves lack neighboring data to stabilize estimates there.

Traditional discrete-norming benchmark

Traditional norming guidelines (such as the COTAN guidelines; Egberink et al. (2026)) often describe minimum sample sizes per discrete age group, such as \(n = 200\), \(n = 300\), or \(n = 400\). These sample sizes result in a benchmark standard error of the mean test score, under the assumption of normally distributed test scores. Because precision should be evaluated not only at the level of the mean test score, but across the entire normed score region of interest, we extend these benchmarks from the mean to the entire normed score region. This will allow us to compare the sampling errors from the bootstrap against the sampling error benchmarks from traditional norming.

Traditional benchmark across the normed score region.

When estimating a standard score (\(z_0\)) within a discrete group of size \(n\) under the assumption of normality, uncertainty arises from estimating both the group mean (\(\hat{\mu}\)) and the group standard deviation (\(\hat{\sigma}\)). As derived by Cole (2021), the standard error of an estimated \(z\)-score \(\hat{z}_0\) under discrete norming is:

\[\begin{equation} SE_{\text{benchmark}}(\hat{z}_0 ; n) = \sqrt{\frac{1 + \frac{z_0^2}{2}}{n}} \end{equation}\]

The benchmark is smallest at the mean and increases toward the tails because uncertainty in the estimated standard deviation contributes more strongly for large \(|z_0|\). The benchmark is naturally expressed on the z scale, but it can be transformed to any normed score that is a linear transformation of the \(z\)-score. For a standardized score \(S = a + bz\), the benchmark standard error is \(SE_{\mathrm{benchmark},S}(S;n) = b\,SE_{\mathrm{benchmark},z}(z;n)\).

Visualizing the comparison with boot_heatmap

By comparing the empirical bootstrap standard error \(\widehat{SE}_{\text{boot}}(\text{age}, z)\) with the benchmark standard error \(SE_{\text{benchmark}}(z; n)\), we can evaluate across the entire continuous age–score space whether the continuous norming model estimates meet the precision benchmarks imposed on traditional norming.

Two metrics are available:

  • Difference: \(\Delta SE(age, z) = \widehat{SE}_{\mathrm{boot}}(age, z) - SE_{\mathrm{benchmark}}(z; n)\)
  • Relative Ratio: \(R(age, z) = \frac{\widehat{SE}_{\mathrm{boot}}(age, z)}{SE_{\mathrm{benchmark}}(z; n)}\)

A negative difference or a ratio below 1 indicates smaller estimated uncertainty than the benchmark.

# Difference map for fixed-parameter model
boot_heatmap(
  boot.BCCGtr34,
  type = "difference",
  benchmark_n = 300,
  add_points = TRUE,
  add_contour = TRUE,
  centiles = c(1, 5, 25, 50, 75, 95, 99),
  centile_label_nudge = 2.0
)
#> Contour threshold: difference < 0 (n = 300)
#> Percentage of age-score range below contour: 93.59% (2160 / 2308 cells)


# Relative ratio map
boot_heatmap(
  boot.BCCGtr34,
  type = "ratio",
  benchmark_n = 300,
  add_points = TRUE,
  add_contour = TRUE,
  centiles = c(1, 5, 25, 50, 75, 95, 99),
  centile_label_nudge = 2.0
)
#> Contour threshold: ratio < 1 (n = 300)
#> Percentage of age-score range below contour: 93.59% (2160 / 2308 cells)


# Reselected model specification across bootstrap samples
boot_heatmap(
  boot.BCCGtr34.reselect,
  type = "difference",
  benchmark_n = 300,
  add_points = TRUE,
  add_contour = TRUE,
  centiles = c(1, 5, 25, 50, 75, 95, 99),
  centile_label_nudge = 2.0
)
#> Contour threshold: difference < 0 (n = 300)
#> Percentage of age-score range below contour: 63.47% (1465 / 2308 cells)

In these heatmaps, green regions (\(\text{Difference} < 0\) or \(\text{Ratio} < 1{,}0\)) indicate regions where the continuous norming model meets the precision benchmarks of traditional norming with a norm group of size \(n = 300\). This typically happens near the centre of the distribution.

The blueish regions indicate regions where continuous norming estimates are less precise than the traditional norming benchmark. This typically occurs near the extreme ages or at the outer centiles where data density is lowest. The dashed contour line indicates the exact boundary where the continuous norming is as precise as the benchmark. Comparing boot.BCCGtr34 against boot.BCCGtr34.reselect illustrates how accounting for model selection variance alters this boundary.

Note on Interpreting the Benchmark Comparison

The comparison between the bootstrap standard error SE and the reference SE should not be viewed as a direct comparison between continuous and traditional norming. Instead, it serves as a practical benchmark to evaluate whether the precision of the continuous norming model satisfies established quality criteria (i.e., guidelines based on norm group sizes for traditional norming).

This comparison cannot be taken as a direct comparison between both methods for several reasons:

  • It does not account for bias: This comparison focuses solely on variance (SE) and ignores estimation bias. Traditional discrete norming often introduces systematic bias (such as end-of-cohort bias) by collapsing continuous age into discrete intervals. Continuous norming avoids this bias by modeling age smoothly. While continuous models may exhibit higher variance due to model flexibility, their overall mean squared error (\(\text{MSE} = \text{Bias}^2 + \text{Variance}\)) is frequently lower.
  • The standard errors capture different sources of variability: The traditional reference SE is a theoretical formula derived under the assumption of normally distributed test scores within each norm group. In contrast, the empirical bootstrap SE reflects real-world complexities such as skewness, kurtosis, and floor and ceiling effects. Furthermore, when reselect = TRUE, the bootstrap also captures the uncertainty inherent in the data-driven model selection process. Because the bootstrap SE incorporates more sources of variation than the theoretical reference SE, the two metrics are not strictly comparable.

Precision at selected percentiles

The function boot_centiles() provides a focused view of how precision evolves across age along specific centile curves (e.g., the 5th percentile, median, or 95th percentile).

boot_centiles(
  boot.BCCGtr34,
  centiles = c(5,50,95),
  benchmark_n = c(200, 300, 400)
)


boot_centiles(
  boot.BCCGtr34.reselect,
  centiles = c(5,50,95),
  benchmark_n = c(200, 300, 400)
)

At the lower age boundary (near age 5), standard errors increase steeply, especially for the 95th percentile.

Case 2: Precision of Composite Norms

Composite scores combine standardized subtest scores. Their precision depends on the uncertainty of each subtest norm, the cross-subtest dependence structure, and the composite norming model.

The function boot_composite() jointly resamples participants, refits the subtest models, recomputes subtest normed scores, refits the composite model, and summarizes uncertainty. The $sample output propagates the complete chain of uncertainty for each observed individual. The $normtable output conditions on a fixed composite score grid. Therefore, it is best used as a diagnostic or residual-stability check.

Fitting the Subtest and Composite Models

We illustrate composite norming using two IDS-2 subtests: Naming Categories (y7) and Naming Antonyms (y14). First, we fit the subtest and composite models. In the construction of the normtables, we set trim = Inf so that the subtest normed scores are not rounded before the composite model is fit. Rounding may result in deviations from normality in the composite score distribution, potentially resulting in a poorer model fit of the composite score model.

mydata1 <- shape_data(ids_data, age_name = "age", score_name = "y7",  family = "BCCGtr34")
mod1 <- fb_select(mydata1, age_name = "age", score_name = "shaped_score", family = "BCCGtr34")
#> FB-select iteration 0: Polynomial degrees = 2,1,0 : BIC = 9071.163
#> FB-select iteration 1: Polynomial degrees = 3,1,0 : BIC = 9069.03
mydata2 <- shape_data(ids_data, age_name = "age", score_name = "y14", family = "BCCGtr34")
mod2 <- fb_select(mydata2, age_name = "age", score_name = "shaped_score", family = "BCCGtr34")
#> FB-select iteration 0: Polynomial degrees = 2,1,0 : BIC = 8399.433

normtable1 <- normtable_create(mod1, mydata1, age_name = "age", score_name = "shaped_score", trim = Inf)
normtable2 <- normtable_create(mod2, mydata2, age_name = "age", score_name = "shaped_score", trim = Inf)
comp_data <- composite_shape(list(y7 = normtable1, y14 = normtable2))
comp_model <- fb_select(comp_data, age_name = "age", score_name = "z_sum", family = "NO")
#> FB-select iteration 0: Polynomial degrees = 2,1 : BIC = 6647.217
#> FB-select iteration 1: Polynomial degrees = 2,0 : BIC = 6640.389
#> FB-select iteration 2: Polynomial degrees = 1,0 : BIC = 6633.873
#> FB-select iteration 3: Polynomial degrees = 0,0 : BIC = 6626.51

Running boot_composite()

We pass the fitted subtest models and composite model to boot_composite(). Options include specifying custom subtest or composite score labels with subtest_score_names and composite_score_name, setting custom weighting via weights (defaulting to equal weights), selecting the output metric (normtype = "Z", "T", "IQ", or "SS"), and selectively enabling model reselection for subtests or the composite model with reselect.

boot_comp <- boot_composite(
  subtest_models = list(y7 = mod1, y14 = mod2),
  composite_model = comp_model,
  age_name = "age",
  composite_score_name = "z_sum",
  normtype = "Z",
  step_size_score = 0.1,
  B = 1000,
  parallel = TRUE,
  seed = 3
)

Sample-level results

The sample-level result is the primary result for evaluating the stability of observed composite scores. It contains one row per observed individual and propagates uncertainty through the subtest norms, the observed subtest scores, and the composite model.

head(boot_comp$sample)
#>   id       age y7_testscore y14_testscore       z_sum    original  boot_mean
#> 1  1 10.971461           17            17 -0.80164937 -0.45012945 -0.4506782
#> 2  2 11.674196           20            19  0.06315429  0.03770244  0.0380113
#> 3  3 11.444216           22            23  2.19983630  1.24299524  1.2460976
#> 4  4 10.049315           14            14 -1.79249457 -1.00906081 -1.0104793
#> 5  5 14.182067           20            19 -1.46350027 -0.82347659 -0.8253445
#> 6  6  6.554415           11            11 -0.76693216 -0.43054563 -0.4305513
#>   boot_median    boot_se boot_ci_lower boot_ci_upper boot_n
#> 1 -0.45133369 0.03336660   -0.51487133    -0.3829690   1000
#> 2  0.03823916 0.03397369   -0.02810428     0.1030786   1000
#> 3  1.24570633 0.04483677    1.16215515     1.3330474   1000
#> 4 -1.01194151 0.03764512   -1.08243408    -0.9332077   1000
#> 5 -0.82469941 0.03814654   -0.90041357    -0.7526586   1000
#> 6 -0.43101823 0.04553871   -0.51970344    -0.3411091   1000

The function boot_sample() visualizes these observed sample-level results across age and facets them into composite-score bins (z_bins). The main option is type = c("ratio", "difference", "se"). For type = "se", it plots the raw bootstrap SE per individual. For type = "se", reference lines are omitted because individual normed scores vary within each bin. Types type = "ratio" and type = "difference" plot the ratio and difference, respectively, between the sample-level bootstrap SEs divided by their corresponding traditional-norming reference SE with a set benchmark_n. A LOESS smoothing curve can be added with smooth = TRUE; note that its approximation can be off, because tail bins may contain few observations.

boot_sample(
  boot_comp,
  type = "ratio",
  benchmark_n = 400,
  z_bins = seq(-3, 3, by = 1)
)


boot_sample(
  boot_comp,
  type = "se",
  benchmark_n = 400,
  z_bins = seq(-3, 3, by = 1),
  smooth = TRUE
)

The plot typically shows the same broad pattern as the subtest precision results: composite scores are generally more stable around the middle of the age range, while standard errors are larger at the youngest and oldest ages. Boundary inflation occurs because the fitted age trends have fewer neighbouring observations at the edges of the norming range. Standard errors can also be larger in the score tails, where the normative sample is less dense.

Limitations of the sample-level analysis

The sample-level result is the most relevant result for the observed individuals, but it does not paint a complete picture of composite-score uncertainty. A given composite score can be produced by many different combinations of subtest scores, while the observed sample contains only the combinations that happened to occur. This limitation is especially important in the tails, where there may be only a few observations. Therefore, boot_sample() is a useful empirical proxy for deciding whether observed composite scores are stable enough for use, but it does not guarantee stability for every possible subtest-score profile in the population.

Normtable-level results

The composite $normtable output evaluates precision across a fixed evaluation grid of age and composite sum score combinations. This primarily serves as a secondary check to assess whether composite normtable results are stable across bootstrap refits. They should not be used to compute confidence intervals around individuals’ composite scores, because they do not take subtest score uncertainty into account.

When comparing composite norms against traditional norming benchmarks with boot_heatmap of boot_centiles, the reference SE of composite scores is calculated under the assumption of multivariate-normally distributed subtest scores with age-independent correlations. Because the composite score is constructed by summing standardized subtest scores whose residuals are already centered around zero with approximately unit variance, the resulting theoretical benchmark standard error is substantially smaller than for single subtests. In fact, at the 50th percentile (e.g., \(z = 0\)), the theoretical benchmark standard error approaches zero because the mean of standardized normal sum scores is theoretically fixed at zero across all ages.

head(boot_comp$normtable)
#>        age   z_sum original boot_mean boot_median boot_se boot_ci_lower
#> 1 4.449543 -6.2679       -3        NA          NA      NA            NA
#> 2 4.798260 -6.2679       -3        NA          NA      NA            NA
#> 3 5.146977 -6.2679       -3        NA          NA      NA            NA
#> 4 5.495694 -6.2679       -3        NA          NA      NA            NA
#> 5 5.844411 -6.2679       -3        NA          NA      NA            NA
#> 6 6.193128 -6.2679       -3        NA          NA      NA            NA
#>   boot_ci_upper boot_n
#> 1            NA     NA
#> 2            NA     NA
#> 3            NA     NA
#> 4            NA     NA
#> 5            NA     NA
#> 6            NA     NA

boot_heatmap(boot_comp, type = "se", benchmark_n = 400, add_points = TRUE)

boot_heatmap(boot_comp, type = "difference", benchmark_n = 400, add_points = TRUE)
#> Contour threshold: difference < 0 (n = 400)
#> Percentage of age-score range below contour: 95.28% (5151 / 5406 cells)

boot_centiles(boot_comp, centiles = c(5, 50, 95))

Notes on the interpretation of composite normtable-level results.

Normtable-level bootstrap standard errors for composites are typically very small across the entire grid because they evaluate stability conditional on a fixed sum score. If the subtest models have successfully removed all age effects and the inter-subtest correlations are stationary over age, the composite score model should be flat across age.

Any systematic age trend or instability across bootstrap refits (particularly when reselect = TRUE) signals unmodelled age-dependency. Checking the composite model’s selection history ($model_selection_history or $model_freq_table) reveals whether this age-dependency enters through the location (\(\mu\)) or the scale (\(\sigma\)) parameter:

  • Age-dependency in the mean (\(\mu\)): Indicates redundant age-dependency that was not properly captured by one or more subtest models, leading to biased composite norms across age groups. Re-evaluate the subtest models.
  • Age-dependency in the scale/variance (\(\sigma\)): Can reflects genuine developmental shifts in inter-subtest covariance (such as cognitive differentiation or dedifferentiation across age spans). In this case, the multivariate-normal reference SE are no longer valid, because they do not account for additional uncertainty from age-dependent covariance estimation.

Case 3: Sample Size Planning for Subtests

While the nonparametric bootstrap resamples the observed empirical data, the parametric bootstrap simulates new raw scores directly from a (presupposed or fitted) GAMLSS model. If a plausible model is available (e.g., from a previous version of the test), this mechanism allows a priori sample size planning and study design. Specifically, test developers can simulate candidate sample sizes (\(N\)) and sampling distributions across age (sampling frames) to evaluate the precision of the resulting normtables before data collection begins.

Designing and Simulating Alternative Sampling Frames

Using the Naming Categories subtest as an example, we evaluate three candidate recruitment strategies using boot_subtest() with boot_type = "parametric":

  1. Small Uniform Allocation (\(N = 500\)): Equal recruitment across ages 5 to 20.
  2. Large Uniform Allocation (\(N = 1000\)): Doubled sample size under uniform sampling across ages 5 to 20.
  3. Boundary-Enriched Allocation (\(N = 1000\)): 80% uniform recruitment across ages 5 to 20, 10% oversampling at ages 5–8, and 10% oversampling at ages 17–20.

As demonstrated in the nonparametric bootstrap, continuous norming models naturally have more variance near the youngest and oldest ages, because the polynomial and/or spline curves lack neighboring observations at the boundaries. By intentionally oversampling the age boundaries (e.g. allocating 10% extra to both boundaries), test developers can flatten these boundary standard error spikes without increasing the total sample size \(N\) (Borghi et al. 2006). Alternatively, if the test remains valid outside of the target age range, sampling a small number of observations beyond the target age range can also stabilize the curves (Timmerman, Voncken, and Albers 2021).

# Define candidate age sampling frames
frame_uniform <- function(n) {
  runif(n, min = 5, max = 20)
}

frame_boundary_enriched <- function(n) {
  n_mid <- round(n * 0.80)
  n_low <- round(n * 0.10)
  n_high <- n - n_mid - n_low
  c(
    runif(n_mid, min = 5, max = 20),
    runif(n_low, min = 5, max = 8),
    runif(n_high, min = 17, max = 20)
  )
}
# Simulate Plan 1: Uniform N = 500
plan_unif_500 <- boot_subtest(
  model = mod.BCCGtr34,
  age_name = "age",
  score_name = "y7",
  boot_type = "parametric",
  n_sim = 500,
  age_sampling_frame = frame_uniform,
  normtype = "Z",
  min_age = 5,
  max_age = 20,
  B = 1000,
  parallel = TRUE,
  seed = 123
)

# Simulate Plan 2: Uniform N = 1000
plan_unif_1000 <- boot_subtest(
  model = mod.BCCGtr34,
  age_name = "age",
  score_name = "y7",
  boot_type = "parametric",
  n_sim = 1000,
  age_sampling_frame = frame_uniform,
  normtype = "Z",
  min_age = 5,
  max_age = 20,
  B = 1000,
  parallel = TRUE,
  seed = 123
)

# Simulate Plan 3: Boundary-Enriched N = 1000
plan_boundary_1000 <- boot_subtest(
  model = mod.BCCGtr34,
  age_name = "age",
  score_name = "y7",
  boot_type = "parametric",
  n_sim = 1000,
  age_sampling_frame = frame_boundary_enriched,
  normtype = "Z",
  min_age = 5,
  max_age = 20,
  B = 1000,
  parallel = TRUE,
  seed = 123
)

Comparing Simulated Precision Across Study Designs

To evaluate how each design controls estimation uncertainty, we plot the expected standard error curves across age for several important percentiles:

# Uniform N = 500: Notice the SE spikes at ages 5 and 20
boot_centiles(
  plan_unif_500,
  centiles = c(5, 50, 95)
) + ggplot2::labs(title = "Uniform, N=500")


# Uniform N = 1000: Lower standard errors across the entire continuum
boot_centiles(
  plan_unif_1000,
  centiles = c(5,50, 95)
) + ggplot2::labs(title = "Uniform, N=1000")


# Uniform N = 1000: Boundary spikes are flattened with the same total N
boot_centiles(
  plan_boundary_1000,
  centiles = c(5,50, 95)
) + ggplot2::labs(title = "Boundary-enriched, N=1,000")

Comparing these simulated curves against accuracy benchmarks from traditional norming helps researchers and test constructors empirically justify sample size requirements and optimize age recruitment allocations prior to data collection.

The resulting plots indicate that a sample size of \(N = 500\) under uniform allocation is insufficient, as standard error curves across most age ranges exceed the benchmark for traditional norming with \(n = 200\). Doubling the sample size to \(N = 1{,}000\) yields acceptable precision across the entire range. Furthermore, comparing the two \(N = 1{,}000\) designs shows that the boundary-enriched strategy successfully reduces standard errors at both age extremes compared to a strictly uniform design.

Funding

Development of normref was supported by Dutch Research Council NWO, grant 406.27.GO.022.

References

Borghi, E., M. de Onis, C. Garza, J. Van den Broeck, E. A. Frongillo, L. Grummer-Strawn, S. Van Buuren, et al. 2006. “Construction of the World Health Organization Child Growth Standards: Selection of Methods for Attained Growth Curves.” Statistics in Medicine 25 (2): 247–65. https://doi.org/https://doi.org/10.1002/sim.2227.
Cole, T. J. 2021. “Sample Size and Sample Composition for Constructing Growth Reference Centiles.” Statistical Methods in Medical Research 30 (2): 488–507. https://doi.org/10.1177/0962280220958438.
Efron, B., and R. J. Tibshirani. 1993. An Introduction to the Bootstrap. 1st ed. Chapman; Hall/CRC.
Egberink, I., W. De Leng, A. Evers, B. Hemker, and W. Lucassen. 2026. COTAN Beoordelingssysteem Voor de Kwaliteit van Tests. Geheel Herziene Versie 2026. Nederlands Instituut van Psychologen.
Hesterberg, T. C. 2015. “What Teachers Should Know about the Bootstrap: Resampling in the Undergraduate Statistics Curriculum.” The American Statistician 69 (4): 371–86.
Morris, T. P., I. R. White, and M. J. Crowther. 2019. “Using Simulation Studies to Evaluate Statistical Methods.” Statistics in Medicine 38 (11): 2074–2102. https://doi.org/10.1002/sim.8086.
Timmerman, M. E., L. Voncken, and C. J. Albers. 2021. “A Tutorial on Regression-Based Norming of Psychological Tests with GAMLSS.” Psychological Methods 26 (3): 357. https://doi.org/10.1037/met0000348.
Voncken, L., C. J. Albers, and M. E. Timmerman. 2019. “Model Selection in Continuous Test Norming with GAMLSS.” Assessment 26 (7): 1329–46. https://doi.org/10.1177/1073191117715113.