--- title: "Token efficiency" description: "Budgets, caches and descriptions that keep context small." output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Token efficiency} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- gptr logo Keep data and computation in R, and send models enough context to choose useful actions. For inline work, an object around 4 GB can remain in the same R session across turns. The model receives size-limited descriptions, previews and tool output that may include data values, while its R tool computes on the object. No repeated export and reload cycle is required; R copies, computation time and request latency still depend on the workflow. Workers receive serialized context rather than sharing the parent workspace. Four practices help: - Refer to live objects with a compact description rather than printing large tables into a prompt. The model can compute on those objects with the `r` tool. - Compose several deterministic operations in one R call and return a bounded result. - Reuse a session's stable prompt and growing conversation when the provider supports prompt caching. Changing models, tools, compaction or rewind can change that prefix. - Inspect usage and set budgets before starting a costly workflow. The executed examples use fake providers. Their request counts and estimates demonstrate the interfaces; they are not measurements of a real model's speed, quality or bill. ## Inspecting the prompt ``` r library(gptr) view = gptr_prompt(preset = "minimal") attr(view, "tool_names") #> [1] "read" "r" "edit" "write" view$sections #> name tier tokens #> 1 preamble T0 42 #> 2 tools T0 86 #> 3 rules T0 348 #> 4 modes T0 84 #> 5 context T0 152 view$total_tokens #> [1] 1920 ``` `gptr_prompt()` inspects the system sections and tool definitions with estimated token counts; printing it shows their text. It does not query a provider. The `standard` preset describes most capabilities as functions in `peter$`, with the core model-visible `r`, `read`, `edit` and `write` tools and `ask` when applicable. `view$total_tokens` already counts the tool array and a preview of the first message; your prompt and the objects you attach add to it. ## Object descriptions and result size ``` r writeLines(gptr_describe(iris)) #> 150 x 5, 6.8 KB #> $ Sepal.Length min 4.3, median 5.8, max 7.9; no NA #> $ Sepal.Width min 2, median 3, max 4.4; no NA #> $ Petal.Length min 1, median 4.35, max 6.9; no NA #> $ Petal.Width min 0.1, median 1.3, max 2.5; no NA #> $ Species 3 levels, top: setosa (50), versicolor (50), virginica (50); no NA writeLines(gptr_describe(iris, budget = 60)) #> 150 x 5, 6.8 KB #> Sepal.Length:dbl Sepal.Width:dbl Petal.Length:dbl Petal.Width:dbl Species:fct ``` `gptr_describe()` summarizes class, shape and values within a budget (150 tokens by default). Packages can register methods for their own classes. Generative-model context normally uses these descriptions and names; **System 1 questions send their input states**, so their egress and size rules differ. See [Typed decisions](system-one.html). ``` r work = new.env() session = peter("Compare mpg and weight by cylinder count.", mtcars, model = fake, envir = work, mode = auto) work$slopes #> 4 6 8 #> -5.647025 -2.780106 -2.192438 ``` The calculation runs in R once the model requests it. Return a small table, statistic or plot rather than a full intermediate object. Long `r` output keeps a bounded head and tail; the full spilled text can be read through `peter$out(id)` when an output ID was supplied. ## Reported usage and estimated components ``` r gptr_usage(session, by = "model") #> group requests input output cache_read cache_write cost #> 1 fake/fake-1 2 3460 48 0 0 0 #> #> # 1 row #> 2 requests, 3.5k tokens in, $0.0000 ledger = gptr_usage(session, detail = TRUE) aggregate(tokens ~ component, data = ledger, FUN = sum) #> component tokens #> 1 attached 132 #> 2 environment 156 #> 3 images 0 #> 4 other 344 #> 5 project 0 #> 6 t0 2850 #> 7 t1 566 #> 8 tool_results 53 #> 9 tools 2068 #> 10 transcript 70 #> 11 workspace 58 ``` The usage summary contains provider-reported input, output, cache reads/writes and cost. `by = "session"`, `"agent"`, `"model"` or `"route"` groups it; child requests are included once. A typed System 1 decision made by an agent's R code is charged to that agent's session, its budget and its ancestors once. `gptr_usage()` without a session also includes the process-level System 1 log of decisions that no session owns. The detailed ledger estimates components such as tools, workspace, attached objects, transcript and tool results. Its sum can differ from reported usage because tokenizers, provider accounting, images and fake-provider behavior differ. Unknown reported usage or cost stays `NA`, not zero. A local model's known zero metered charge does not imply zero hardware cost or latency. ## Budgets and partial results ``` r budget_error = tryCatch( peter("Summarize mpg.", model = small, budget = list(tokens = 500)), gptr_error_budget = function(e) e ) class(budget_error)[1] #> [1] "gptr_error_budget_tokens" budget_error$session$status #> [1] "budget" ``` The estimated input for the first request already exceeds 500 tokens, so this call stops before that request. A top-level run's defaults are 2 million tokens and 5 US dollars; `budget = list(tokens = ..., cost = ..., turns = ...)` supplies limits for that call. Change settings deliberately when different defaults are needed: ``` r gptr_config(budget = list(tokens = 20000, cost = 1, turns = 10), .scope = "session") session = peter("Review this analysis.", budget = list(tokens = 12000, turns = 6)) ``` Checks happen before requests using estimated next-input tokens and usage already recorded. The next output or final provider charge can exceed a remaining allowance; unknown usage cannot establish a hard monetary ceiling. A budget is a stopping policy, not a billing reservation. At the console the UI can ask to extend it; without a person present the call signals a budget condition carrying the partial session as `e$session`. Inspect that session's status, transcript, workspace and usage before deciding whether to continue. Teams and fan-outs share the root call's budget; launching more children does not give each one a fresh root allowance. With `gptr_parallel()`, member `budget =`, `tools =` and per-run `.opts` are not applied. Set the intended limits through the governing settings or enclosing run, cap concurrency and inspect the returned children's statuses. See [Teams and background work](teams-and-background.html). ## Compaction At the chat prompt, `/compact` replaces older model context with a summary. Add a focus when particular details must survive, for example `/compact retain the model comparison and next steps`. Automatic compaction follows the configured context and cold-cache thresholds; see [Configuration](configuration.html). Compaction can make a model request and add usage. It keeps the R objects already in your session, while the model no longer sees every older turn verbatim. Preserve important decisions in project files and named R objects. A changed request prefix can also affect provider cache reuse. ## Three different caches | Mechanism | What is reused | Inspect | |---|---|---| | Provider prompt cache | A repeated request prefix | `gptr_usage(session)` | | System 1 answer cache | Matching input, question and model | Metadata and `gptr_cache()` | | Replay / System 2 cache | Recorded code and saved answer | `gptr_blocks(path)`, `gptr_cache()` | Each request to the same model repeats the previous request and adds to its end: the session freezes its prompt and tool list, and messages are only appended. The provider can then read the repeated prefix from its cache. Compaction, image elision, rewind and changed resources can start a new prefix. gptr compares each request with the previous one to the same model; a difference it did not plan is recorded in the session as a cache break, and `options(gptr.check_prefix = "warn")` also warns. ![Prompt caching, one request per row. Each request repeats the one before and appends the model's reply and the tool result; after compaction a summary replaces the older messages, so only the prompt and tools are repeated.](figures/prompt-prefix.svg){alt="Diagram: requests that repeat a cached prompt prefix"} Providers decide whether a cache hit occurs, subject to their support, minimum size, expiry and pricing; gptr does not guarantee a hit after the first request. Document replay makes no model request for a usable recorded call but still runs its R code. Clearing an answer cache changes what can be reused. ## Explicit ways to reduce context These live-call examples assume a configured model. For unattended use of a remote provider, save its egress acknowledgment first, as in [Language models](language-models.html). `context = "none"` still requires that consent because project instructions and environment details remain; offline providers and local loopback routes are exempt (Ollama only while local-only inference is on). ``` r peter("Summarize the model fit.", fit, .opts = list(context = "names")) peter("Find the mean of x. Return one number.", .opts = list(context = "none")) peter("Compute the mean.", x, .opts = list(preset = "minimal")) ``` `context = "names"` lists workspace names and uses headers for explicitly attached objects. `"none"` omits both built-in workspace and attached-object descriptions; the R bindings still exist and can be used by tool code, as `x` is in the second example. Prompt text, tool results, images and other context blocks can still reach the model. System 1 input states use a separate encoding route. [Execution model](execution-model.html) explains these boundaries. `preset = "minimal"` reduces the tool/prompt surface; it does not authorize actions or remove required safety checks. `{identifier}` in a literal prompt can insert a short atomic value when that is sufficient. For a long workflow, continue one session, keep intermediate data in R and request concise reports between stages. Use a new session or an explicit fork when a separate analysis is intended.