Pass double matrices

R stores a matrix of whole numbers as integers. The backend binds double storage, so an integer matrix, a data frame or a logical matrix is converted first, and the conversion is a copy by R’s own rules. The conversion happens visibly in R before the call, once, and a double matrix passes through untouched. To keep a large design copy-free, create it as double.

Xi <- matrix(sample(0:2, 60 * 20, replace = TRUE), 60, 20)
storage.mode(Xi)
#> [1] "integer"
storage.mode(Xi) <- "double"     # one explicit conversion, then zero-copy from here on
storage.mode(Xi)
#> [1] "double"

Selectors: in place at construction, back at the end

A selector centers and scales X in place when it is constructed and writes the original values back when select() returns. In between, X holds the working design. restore() writes back early for a selector that will not run, and a selector that is garbage collected without running writes back in its finalizer.

set.seed(1)
n <- 80
p <- 40
X <- matrix(rnorm(n * p), n, p)
y <- as.numeric(X[, 1] * 2 + rnorm(n))
X_before <- X + 0
sel <- TRexSelector$new(X, y, tFDR = 0.1, seed = 2, verbose = FALSE, control = trex_control(K = 10))
round(sqrt(colSums(X^2))[1:3], 6)     # the working design: unit columns
#> [1] 1 1 1
sel$select()
max(abs(X - X_before))                 # the original values are back
#> [1] 4.440892e-16

T-solvers: writers and readers

A T-solver with normalize = TRUE or intercept = TRUE is a writer. It transforms X and D in place, keeps the statistics, and holds a claim on both matrices until restore() or garbage collection writes them back. A second writer on a claimed matrix is refused, because a second transformation would move the data by rounding.

D <- matrix(rnorm(n * p), n, p)
w1 <- TLARS_Solver$new(X, D, y)
tryCatch(TLASSO_Solver$new(X, D, y), error = function(e) cat("error:", conditionMessage(e), "\n"))
#> error: TLASSO: the input matrix X is already held by another live T-solver or T-Rex selector that preprocessed it in place. A second in-place preprocessing is not idempotent (it moves the data by rounding) and would corrupt both. Pre-scale once in front and construct the solvers with normalize = false and intercept = false (readers never claim a buffer), or restore() the previous solver, or let it go out of scope first.
w1$restore()

A solver with normalize = FALSE and intercept = FALSE is a reader. It runs on the data as it finds them and never writes, so any number of readers can share one design. A pipeline scales the design once with a scaler and constructs readers.

sx <- MLLpNormScaler$new(norm_type = 2, center = TRUE)
invisible(sx$fit_transform_inplace(X))
sd <- MLLpNormScaler$new(norm_type = 2, center = TRUE)
invisible(sd$fit_transform_inplace(D))
y_c <- y - mean(y)
opts <- tsolver_options(normalize = FALSE, intercept = FALSE, precentered = TRUE)
r1 <- TLARS_Solver$new(X, D, y_c, options = opts)
r2 <- TLASSO_Solver$new(X, D, y_c, options = opts)
r1$execute_step(T_stop = 2)
r2$execute_step(T_stop = 2)
invisible(sx$inverse_transform_inplace(X))
invisible(sd$inverse_transform_inplace(D))
max(abs(X - X_before))
#> [1] 8.881784e-16

y is never written. The solver centers an internal copy of the response.

Loading a saved solver

load() restores a solver into one constructed on the same data in the same state. Load into a reader on pre-scaled data, or into a writer constructed with the same options on the same raw data, which then keeps the duty to write back.

Mapped files are never read into memory

A memory-mapped matrix is bound through the same interface as an R matrix. The backend touches the pages it needs and the file keeps the data, so the working set of a run is the backend’s own buffers, never a copy of the design. The R-level accessors ([ and as.matrix()) do copy the requested block into R memory, by design.

The low-memory engines

The ridge and elastic-net classes have a low_memory engine that avoids every allocation of the size of X.

Class	`low_memory = FALSE`	`low_memory = TRUE`
`MLRidgeCV`	each fold gathers and standardizes its training rows and takes a thin SVD, about three to four copies of `X` in flight	the dual (kernel) form, \(n \times n\) kernels accumulated from column blocks read in place, memory \(O(n^2)\)
`MLElasticNet`	coordinate descent on a centered and standardized working copy, one \(n \times p\) matrix	coordinate descent on `X` itself, no allocation of the size of `X`
`MLElasticNetCV`	one working copy per fit	every fold on `X` itself with a training mask

The two engines compute the same estimator on the same grid and agree to rounding. The selectors that tune \(\lambda_2\) by cross-validation (GVS, SPCA, Tikhonov) use these engines, and the elastic-net tuner of the sparse PCA selector keeps its peak memory under 0.3 times the size of X.

cv_a <- MLRidgeCV$new()
cv_a$fit(X, y, n_folds = 5, n_lambda = 30)
cv_b <- MLRidgeCV$new(low_memory = TRUE)
cv_b$fit(X, y, n_folds = 5, n_lambda = 30)
all.equal(cv_a$get_cv_errors(), cv_b$get_cv_errors())
#> [1] TRUE

Measuring the footprint

get_memory_bytes() reports what a T-solver holds on the heap. X and D are borrowed and not counted.

r1$get_memory_bytes()
#> [1] 14140

Working without data copies

The rule