---
title: "Search plans and quota-aware retrieval"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Search plans and quota-aware retrieval}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = FALSE, comment = "")
# Console colour carries no meaning on a rendered page. pkgdown turns it on for
# its own build, and the escape sequences then reach the reader as literal text,
# so colour is switched off here for a plain vignette render and a site build
# alike. The fixed width keeps tibbles inside the documentation column.
options(cli.num_colors = 1, cli.hyperlink = FALSE, crayon.enabled = FALSE,
        width = 80)
# Print data frames and tibbles as formatted tables.
local({
  kp <- function(x, ...) {
    if (any(vapply(x, is.list, logical(1)))) return(knitr::normal_print(x))
    knitr::knit_print(knitr::kable(x))
  }
  for (cls in c("data.frame", "tbl_df", "tbl")) {
    registerS3method("knit_print", cls, kp, envir = asNamespace("knitr"))
  }
})
```

```{r setup}
library(scopusflow)
```

The Elsevier Scopus Search API is generous but bounded. A weekly quota limits how
many requests you may make, a short-term rate limit caps how fast you may make
them, and under the ordinary offset paging no single query will return more
than its first 5000 records. This article shows how scopusflow works within
those bounds so that a large retrieval
is reproducible, efficient and resumable. The steps that contact the API need a
key and are not run here. Everything else runs offline.

## A query, built safely

Most queries combine a few terms under a field tag. `scopus_query()` assembles
them without the bracket and tag mistakes that creep in when fragments are pasted
together by hand.

```{r}
q <- scopus_query("language learning", "effect size", .field = "TITLE-ABS-KEY")
q
```

The recognised field tags, and what each one searches, are listed by
`scopus_field_tags()`.

```{r}
scopus_field_tags()
```

## Describing the search as a plan

A plan records exactly what will be fetched, so it can be saved, reviewed and
re-run. Partitioning by year is the recommended way to stay under the
5000-record ceiling, since each year becomes its own cell.

```{r}
plan <- scopus_plan(q, years = 2010:2020, partition = "year")
plan
```

Each cell carries the query, the year, the view and the page size. The page size
deserves a moment's attention, because it is where quota is won or lost.

## Why page size is a quota decision

Scopus charges quota by the request, whatever a request brings back. A page may
hold up to 200 records under the `STANDARD` view, or 25 under `COMPLETE`, so
retrieving a thousand records in pages of 200 costs five requests where pages of
25 would cost forty. For that reason `page_size` defaults to the largest the view allows,
which is the same efficiency `rscopus` relies on, and is in no sense an evasion
of the quota. Every request is counted, and the 5000-record ceiling still holds.

```{r}
scopus_plan(q, view = "STANDARD")$page_size[1]
scopus_plan(q, view = "COMPLETE")$page_size[1]
```

## Sizing before spending

Counting is cheap and does not download records, so it is worth doing first. The
count comes back with the parsed quota attached, which lets a workflow decide
whether it has the allowance to proceed.

```{r eval = FALSE}
n <- scopus_count(q, years = 2010:2020)
n
attr(n, "quota")
```

That allowance is parsed from the response headers by `scopus_quota()`. To show
its shape without a network call, apply it to a constructed response.

```{r}
resp <- httr2::response(
  status_code = 200,
  headers = list(
    `X-RateLimit-Limit`     = "20000",
    `X-RateLimit-Remaining` = "19987",
    `X-RateLimit-Reset`     = "1700000000"
  )
)
scopus_quota(resp)
```

## Fetching, with caching and resume

`scopus_fetch_plan()` runs each cell in turn. Given a cache directory it writes
each cell to disk as it completes, so a run interrupted halfway, or stopped by
the quota, resumes from where it left off and never pays twice for the same
cell.

```{r eval = FALSE}
records <- scopus_fetch_plan(
  plan,
  cache_dir = scopus_cache_dir(),
  resume = TRUE
)
records
```

A cache directory serves one plan. Cells are checkpointed by their position in
the plan and their year, so a second plan pointed at the same directory could
otherwise be paired with the first plan's checkpoints. Before loading one,
`scopus_fetch_plan()` checks that the query, year, view, page size and record
cap it was written under all match the cell being run, and refetches with a
warning when they do not. A checkpoint holding more records than the current
`max_results` asks for is served trimmed to that cap, with the fuller set left
on disk for a later, wider request. A checkpoint that stopped at a smaller cap
than is being asked for now is a mismatch, refetched with a warning. One that
ran to the end of its result set is not, however few records that turned out to
be, because there is nothing further to fetch. The clean arrangement is still a
separate directory per plan.

The cache lives under `scopus_cache_dir()`. To force a fresh retrieval, empty it
with `scopus_cache_clear()`. Both are shown but not run, so the article does not
touch a real cache.

```{r eval = FALSE}
scopus_cache_dir()    # where completed cells are written
scopus_cache_clear()  # remove them, so the next run re-fetches from scratch
```

The result is a `scopus_records` tibble, the same shape returned by
`scopus_fetch()` for a single query. Without a key, the bundled
`example_records` stands in for it: 138 real journal articles in that same
schema, shipped because Scopus records may not be redistributed.

```{r}
head(example_records)
```

## Watching progress

Per-cell progress is silent by default and switched on with `verbose = TRUE`,
worth doing for a harvest spanning many years.

```{r eval = FALSE}
records <- scopus_fetch_plan(
  plan,
  cache_dir = scopus_cache_dir(),
  verbose = TRUE
)
```

A line is reported as each cell is fetched or loaded from cache.

## Combining separate retrievals

Results gathered in separate runs combine safely with `scopus_combine()`, which
renumbers the records and can drop duplicates by Scopus identifier or DOI. This
is preferable to `rbind()`, which would leave duplicate entry numbers. Here a
baseline retrieval that stopped at 2023 is merged with a later one covering the
whole period.

```{r}
baseline <- example_records[example_records$year <= 2023, ]
combined <- scopus_combine(baseline, example_records, dedupe = TRUE)
nrow(combined)
```

The 138 distinct articles come back as 149 rows, which is worth understanding
before reaching for a workaround. De-duplication needs something to match on.
These records carry no Scopus identifier, never having come from Scopus, so it
falls back to the DOI, and the eleven that arrived without one cannot be matched
to their own copies. A live Scopus harvest carries an identifier on every
record, so the same call would return 138.

## Writing the search up

A harvest is rarely the end of the work. A systematic review has to report the
search itself, in enough detail that a reader can repeat it, and the reporting
standard for that is PRISMA-S (Rethlefsen et al., 2021). `scopus_search_report()`
assembles the record from what the plan and the harvest already carry, so the
methods section is written from the objects and never from memory.

A plan on its own can be reported before it is run, which is useful when a
protocol has to be registered in advance. The plan below describes the search
that produced the bundled corpus, so that the record and the records match.

```{r}
graphene <- scopus_plan("graphene supercapacitor", years = 2015:2024,
                        field = "TITLE-ABS-KEY", partition = "year")
report <- scopus_search_report(graphene)
report
```

Notice how much of it says "unrecorded". Nothing has been retrieved yet, so
there is nothing to state, and the report says so in words, since a blank there
would be read as a zero. That is the governing rule throughout: the record
states only what the objects hold. It never substitutes the current time
for a retrieval that did not record one, never gives a completeness figure for a
harvest whose reported total is unknown, and never counts duplicates unless a
merge recorded removing them.

After a harvest the picture fills in. `scopus_fetch_plan()` attaches the plan,
the retrieval time, the version, the paging mode and the per-cell accounting, so
the report has everything it needs and you never set any of it yourself. The
bundled corpus stands in for a harvest here, since Scopus records may not be
redistributed, so those attributes are written out below to show what each one
contributes.

```{r}
records <- example_records
attr(records, "plan") <- graphene
attr(records, "retrieved_at") <- as.POSIXct("2026-07-22 09:15:00", tz = "UTC")
attr(records, "scopusflow_version") <- "0.3.0"
attr(records, "paging") <- "offset"
per_year <- as.integer(table(example_records$year))
attr(records, "cell_totals") <- tibble::tibble(
  cell = 1:10, date = as.character(2015:2024),
  n_records = per_year, reported_total = as.numeric(per_year)
)

report <- scopus_search_report(records)
report
```

The completeness lines are worth a moment. Each cell is shown against the number
of records the API reported for it, so a cell that came back short stays visible
where a total would have hidden it, and the overall figure is given only because
every cell reported one. Drop any of those attributes and the corresponding line says
so instead.

The methods paragraph is the same record as prose, ready to paste into a
manuscript and edit.

```{r}
cat(format(report, style = "paragraph"))
```

Supplying a `file` writes the whole record as Markdown, including a runnable
snippet that rebuilds the plan, which makes a natural supplementary file.

```{r eval = FALSE}
scopus_search_report(records, file = "search-record.md")
```

Five of the sixteen PRISMA-S items are answered here from the objects, and a
sixth, de-duplication, would be too had these records been merged with
`scopus_combine()`. The rest, among them peer review of the strategy, grey
literature and any other database searched, are listed as yours to supply,
because the package has no way to know them.

## When the ceiling bites

Under offset paging, a query matching more than 5000 records cannot be retrieved
in full from a single call. `scopus_fetch()` returns the first 5000 and warns.
One remedy is to split the search by year, or by any other facet, so that each
cell stays under the ceiling, and `scopus_count()` tells you in advance whether
a split is needed. The other is `scopus_fetch(cursor = TRUE)`, which follows the
API's cursor in place of an offset and retrieves the whole set in one call, with
the records arriving in deep-paging order. The *Analysing a literature* article
weighs the two. A plan gives cached, resumable cells, the cursor a complete set
in a single pass.

## Handling interruptions

Network and API problems are raised as typed conditions, all inheriting from
`scopus_error`, so a long retrieval can catch them and carry on.

```{r eval = FALSE}
result <- tryCatch(
  scopus_fetch_plan(plan, cache_dir = scopus_cache_dir()),
  scopus_error_rate_limit = function(e) {
    message("Rate limited; the cached cells are safe. Try again later.")
    NULL
  }
)
```

Because each completed cell is already cached, resuming after such a pause costs
nothing for the work already done.