Cache and downloads

datasusr can cache DATASUS downloads in a local directory so that repeated calls do not hit the DATASUS FTP again. This is especially useful when developing analysis pipelines interactively.

How caching works

When you call datasus_download() with use_cache = TRUE (the default), files are stored in a structured subdirectory tree under the cache folder. On subsequent calls for the same files, the cached versions are reused without any network access.

library(datasusr)

downloads <- datasus_fetch(
  source    = "SIHSUS",
  file_type = c("RD", "SP"),
  year      = 2024,
  month     = 1,
  uf        = c("PE", "PB")
)

Cache layout

Each file is stored as

<cache_dir>/<source>/<file_type>/<period>/<file_name>

for example SIM/DO/current/DOPE2022.dbc and SIM/DO/prelim/DOPE2022.dbc. <period> is the FTP tree the file came from (current, historical or prelim), so a preliminary file never shadows the final file of the same name. The same layout is used under dest_dir when you pass one.

Downloads are first written to a temporary <file_name>.part file and only renamed to the final name once the transfer succeeds, so an interrupted or failed download never ends up in the cache. datasus_download() reports the outcome of each file in the success and error columns, and datasus_fetch() skips failed files with a warning.

With use_cache = FALSE and no dest_dir, files go to a fresh temporary subdirectory of tempdir() instead of the cache.

Configuring the cache directory

By default, downloads are placed in a session-scoped subdirectory of tempdir() (which R cleans up automatically when the session ends), so the package never writes outside the user-controlled tempdir unless you opt in.

The cache location is resolved in the following order:

  1. The cache_dir function argument
  2. The DATASUSR_CACHE_DIR environment variable
  3. The datasusr.cache_dir R option
  4. The session default (file.path(tempdir(), "datasusr-cache"))

To enable a persistent cache that survives across sessions, point one of the above to a directory of your choice — for example tools::R_user_dir("datasusr", "cache") — and the cache becomes truly persistent.

To set it globally, add a line to your .Renviron:

DATASUSR_CACHE_DIR=/path/to/my/cache

Or in R:

options(datasusr.cache_dir = "/path/to/my/cache")

Inspecting the cache

# Quick summary
datasus_cache_info(verbose = TRUE)

# Detailed listing of all cached files
datasus_cache_list()

Forcing a re-download

Pass refresh = TRUE to datasus_download() (or datasus_fetch()) to re-download files even when they exist in the cache:

datasus_download(files, refresh = TRUE)

datasus_fetch("SIM", "DO", year = 2022, uf = "PE", refresh = TRUE)

Timeouts

The DATASUS FTP can be slow, so downloads have no hard limit on the total transfer time by default. Instead, a connection must be established within 60 seconds and a transfer is aborted when it stays below 1 byte/s for 120 seconds. Pass a finite timeout (in seconds) to datasus_download() or datasus_fetch() to also impose a hard cap per file.

Pruning and clearing the cache

Over time the cache can grow large. Two functions help manage its size:

# Remove files older than 90 days
datasus_cache_prune(older_than_days = 90)

# Keep the total cache under 5 GB
datasus_cache_prune(max_size_bytes = 5 * 1024^3)

# Remove everything
datasus_cache_clear()

When pruning by size, the least-recently-accessed files are removed first.