Download ICES DATRAS survey data

Download ICES DATRAS survey data for one or more surveys and years, and save each survey-year combination as a zipped exchange file in a survey-specific subdirectory.

Usage

download_datras(
  path = NULL,
  surveys = NULL,
  years = NULL,
  overwrite = FALSE,
  download_hl = TRUE,
  download_ca = TRUE,
  method = c("api", "webservice", "php"),
  use_php = FALSE,
  years_per_request = 10,
  include_flagged = FALSE,
  return_data = TRUE,
  strict = TRUE,
  verbose = TRUE,
  timeout = 10,
  ...
)

Arguments

path: Character string giving the directory where downloaded files should be stored. Survey-specific subdirectories are created within this directory. If NULL, the current working directory is used.
surveys: A character vector of DATRAS survey names to download, for example "NS-IBTS" or "BITS". If NULL, all available surveys are used.
years: An integer vector of years to download. If NULL, all available years for each selected survey are used.
overwrite: Logical. If FALSE (default), survey-year files that already exist in path are skipped. Set to TRUE to re-download and overwrite existing files.
download_hl: Logical. If TRUE (default), length-frequency data are also downloaded where available. This option is not used when method = "php".
download_ca: Logical. If TRUE (default), age-length keys and age data are also downloaded where available. This option is not used when method = "php".
method: Character string naming the download route: "api" (default) for the ICES DATRAS Download API, "webservice" for DATRAS::getDatrasExchange(), or "php" for the legacy DATRAS::downloadExchange(). See Details.
use_php: Logical. Kept for backward compatibility: use_php = TRUE is the same as method = "php". Default is FALSE.
years_per_request: Integer. With method = "api", the maximum number of years fetched in one request per record type. Larger values mean fewer requests but more memory for large surveys. Default is 10.
include_flagged: Logical. If FALSE (default), known test surveys are skipped when the survey list is taken from the server. Set to TRUE to download them anyway.
return_data: Logical. If TRUE (default), the function returns the downloaded data by running read_datras() on the specified path.
strict: Logical. Passed to read_datras() when return_data = TRUE, and ignored otherwise. Controls how records in CA without a haul identifier are matched back to a haul: if TRUE (default), a record matching several candidate hauls is left as NA; if FALSE, it is assigned one of them at random. It does not affect the files written to disk, which always hold the exchange data as delivered by ICES.
verbose: Logical. If TRUE (default), progress messages are printed.
timeout: Numeric. Maximum number of seconds to wait for the ICES DATRAS server to respond when retrieving the list of available surveys or years. If the server does not respond within this time (e.g. because a firewall blocks the connection), the function falls back to locally cached information. Default is 10 seconds.
...: Additional arguments for the function read_datras(), used when return_data = TRUE. The most useful ones control how much is held in memory: prune = TRUE drops non-essential columns, drop_hl = TRUE and drop_ca = TRUE omit the length-frequency and biological tables from the returned object, and ncores sets the number of workers used to read the files. None of them affect what is written to disk. See read_datras() for the full list.

Value

Invisibly returns NULL.

Details

If surveys is NULL, all available surveys returned by icesDatras::getSurveyList() are used. If years is NULL, all available years for each selected survey are downloaded.

By default (method = "api"), data are downloaded from the ICES DATRAS Download API, the service behind icesDatras::getDatrasUnaggregated(), and written to disk with write_datras(). The previous route, DATRAS::getDatrasExchange() on the DATRAS web service, is available as method = "webservice", and the legacy PHP-based route from DATRAS::downloadExchange() as method = "php".

Files are saved as zipped exchange files named "<survey>_<year>.zip" inside survey-specific subfolders, whichever method is used.

method = "api" is considerably faster than method = "webservice". It fetches several years and all quarters in one request per record type (HH, HL, CA) and receives a zipped CSV file, where the web service needs one XML request per record type, year and quarter plus several availability checks. The API delivers the new ICES field names; they are mapped back to the exchange names so that the files are the same in layout whichever route wrote them. Values are written as delivered by ICES, read as text so that codes such as ICES rectangle "37E9" are kept exactly. Missing values (-9) are written as empty fields, as with method = "webservice", so that both routes give the same haul identifiers.

The following surveys are treated as test surveys and are skipped unless include_flagged = TRUE: "Test-DATRAS" and "NS-IBTS_UNIFtest". The filter applies only when surveys is NULL, i.e. when the survey list is taken from the DATRAS server; a survey named explicitly is always downloaded.

Each downloaded survey-year is recorded in an archive manifest, DATRAS_manifest.csv, written in the root of path. The manifest holds the extraction date, the checksum of each exchange file, the ICES calculation date of the records it contains, and the package versions used, so that a snapshot can later be verified with verify_extraction() and so that data read from the archive can report where they came from. Existing manifest entries for the same survey, year and quarter are replaced.

Downloading a large part of the database and returning it in one call can exhaust memory, because every survey-year is read back into a single object. Either set return_data = FALSE and read the archive in pieces afterwards, or pass the memory-reducing arguments of read_datras() through ..., for example prune = TRUE and drop_ca = TRUE.

Survey-year-quarters that have hauls but no length data (HL) at all are listed in a warning. Their hauls would otherwise be read as empty hauls with zero catch; DATRAS holds a few such cases, for example test entries (see https://github.com/ices-tools-prod/icesDatras/issues/59). The files are still written as delivered; remove the cases from an analysis with clean_datras(drop_without_hl = TRUE). The check needs download_hl = TRUE and is not made with method = "php".

No manifest entries are written when method = "php", because DATRAS::downloadExchange() writes the files through an external script and reports nothing about what it retrieved. Run write_manifest() on the directory afterwards to describe such an archive.

Examples

if (FALSE) { # \dontrun{
## Download all available years for one survey into the current directory
dat <- download_datras(surveys = "NS-IBTS")

## Download selected years for multiple surveys
dat <- download_datras(
  surveys = c("NS-IBTS", "BITS"),
  years = 2010:2012,
  path = "data/datras"
)

## Re-download existing files
dat <- download_datras(
  surveys = "NS-IBTS",
  years = 2020,
  overwrite = TRUE
)
} # }

Usage

Arguments

Value

Details

See also

Examples