Read ICES DATRAS survey data from zipped exchange files

Read one or more ICES DATRAS exchange files, or all zipped exchange files in one or more directories, into a single datras_raw / DATRASraw object.

Usage

read_datras(
  path,
  surveys = NULL,
  years = NULL,
  recursive = TRUE,
  min_file_size = 0,
  prune = FALSE,
  drop_hl = FALSE,
  drop_ca = FALSE,
  strict = TRUE,
  verbose = TRUE,
  ncores = 1
)

Arguments

path: A character vector of file or directory paths. Each element can point either to an individual DATRAS .zip exchange file or to a directory containing such files.
surveys: Optional character vector of survey acronyms to read (e.g. c("NS-IBTS", "BITS")). When supplied and path contains directories, only zip files whose immediate parent folder name exactly matches one of the specified strings are read. This avoids false matches between similarly named folders (e.g. "NS-IBTS" will not match "NS-IBTS_old"). Matching is case-sensitive.
years: Optional integer vector of years to read. When supplied and path contains directories, only zip files matching those years are read.
recursive: logical. Should the listing recurse into directories? (Default: TRUE).
min_file_size: Minimum file size in bytes. Files of this size or smaller are skipped and reported. Defaults to 0, which skips only empty files; raise it to exclude small archives without trying to read them.
prune: Logical. If TRUE, only core columns are retained using prune_datras() before combining files. This can substantially reduce memory use when reading many files.
drop_hl: Logical. If TRUE, the HL (length-frequency) table is set to NULL after reading each file. Use this when only haul metadata is needed, as HL is often the largest table. Can be combined with prune.
drop_ca: Logical. If TRUE, the CA (biological sampling) table is set to NULL after reading each file. Can be combined with prune and drop_hl.
strict: Logical. Controls how records in CA without a haul identifier are matched back to a haul by DATRAS::readICES(). If FALSE, a record that matches several candidate hauls is assigned one of them at random. If TRUE (default), such ambiguous records are left as NA and are dropped by any subsequent subsetting. Use strict = TRUE when individual biological records must not be attributed to an arbitrary haul.
verbose: Logical. If TRUE (default), progress messages are printed.
ncores: Integer. Number of parallel workers to use when reading zip files. Defaults to 1 (sequential). Values greater than 1 use parallel::mclapply() and are only effective on non-Windows systems.

Value

A combined DATRAS survey object with classes datras_raw and DATRASraw, carrying an extraction record retrievable with extraction().

Details

The function can read:

one or more individual .zip files,
one or more directories containing .zip files,
optionally only files matching selected years.

Empty zip files are skipped. A file that cannot be read is skipped with a message giving the reason, and the remaining files are still read.

Reading a large number of DATRAS files into R can require substantial memory, especially when combining multiple surveys or many years. The following options can substantially reduce peak memory use:

drop_hl = TRUE drops the length-frequency table (HL) immediately after each file is read. HL is typically the largest table and can be omitted when only haul-level metadata is needed.
drop_ca = TRUE drops the biological sampling table (CA) in the same way.
prune = TRUE trims all three tables to a compact set of core columns. Can be combined with drop_hl / drop_ca.

When loading a very large database (many surveys, many years) in a single call still exceeds available memory even after using the options above, consider loading in parts and combining with c():

x1 <- read_datras("~/data/DATRAS", surveys = c("NS-IBTS", "BITS"),
                  drop_ca = TRUE)
x2 <- read_datras("~/data/DATRAS", surveys = c("EVHOE", "IBTS-MED"),
                  drop_ca = TRUE)
x_all <- c(x1, x2)
rm(x1, x2)

If you need a different set of retained columns than provided by prune_datras(), you may wish to apply your own pruning function after reading or adapt the pruning code.

Records in the CA table frequently lack the station and haul numbers that make up haul.id, and are matched back to a haul by survey, year, quarter, country, ship, and statistical rectangle. When that match is not unique, the strict argument decides what happens: strict = FALSE picks one of the candidate hauls at random, whereas the default strict = TRUE leaves the record unmatched. The number of records affected can be checked afterwards with sum(is.na(x[["CA"]]$haul.id)). The matching is not stored in the exchange files, which hold the data as delivered by ICES, so it is redone every time an archive is read. download_datras() therefore takes the same strict argument and passes it on when it returns the downloaded data.

Empty fields in the exchange files are returned as NA, like the -9 code for missing values, rather than as an empty string or an empty factor level. haul.id is left as built by DATRAS, so a haul without a station number keeps an identifier such as "BTS:2022:1:GB:74E9:BT4P::5".

Examples

if (FALSE) { # \dontrun{
## Read all zip files from a survey folder
x <- read_datras("data/NS-IBTS")

## Read selected years from a folder
x <- read_datras("data/NS-IBTS", years = 2018:2020)

## Read selected surveys from a folder containing the whole database
x <- read_datras("data/DATRAS", surveys = c("NS-IBTS", "BITS"))

## Combine survey and year filtering
x <- read_datras("data/DATRAS", surveys = "NS-IBTS", years = 2018:2020)

## Read multiple zip files directly
files <- c("data/NS-IBTS/NS-IBTS_2020.zip",
           "data/NS-IBTS/NS-IBTS_2021.zip")
x <- read_datras(path = files)

## Read and prune to reduce memory use
x <- read_datras("data/NS-IBTS", prune = TRUE)

## Load only haul metadata (HH) -- drop HL and CA to minimise memory use
x <- read_datras("data/DATRAS", drop_hl = TRUE, drop_ca = TRUE)

## Prune columns and also drop the CA table
x <- read_datras("data/NS-IBTS", prune = TRUE, drop_ca = TRUE)

## Attribute ambiguous CA records to an arbitrary haul
x <- read_datras("data/NS-IBTS", strict = FALSE)
} # }