Find files on the local filesystem#

Find files on the local filesystem.

Example configuration to find CMIP6 data on a personal computer:

Listing 10 Contents of data-local.yml#
projects:
  CMIP6:
    data:
      local:
        type: "esmvalcore.local.LocalDataSource"
        rootpath: ~/climate_data
        dirname_template: "{project}/{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}"
        filename_template: "{short_name}_{mip}_{dataset}_{exp}_{ensemble}_{grid}*.nc"

The module will find files matching the glob.glob() pattern formed by rootpath/dirname_template/filename_template, where the facets defined inside the curly braces of the templates are replaced by their values from the Dataset or the recipe plus any facet-value pairs that can be automatically added using augment_facets(). Note that the name of the data source, local-data in the example above, must be unique within each project but can otherwise be chosen freely.

To start using this module on a personal computer, copy the example configuration file into your configuration directory by running the command:

esmvaltool config copy data-local.yml

and tailor it for your own system if needed.

Example configuration files for popular HPC systems and some supported climate models are also available. View the list of available files by running the command:

esmvaltool config list

Further information is available in Data sources.

Classes:

DataSource(*args, **kwargs)

Data source for finding files on a local filesystem.

LocalDataSource(name, project, priority, ...)

Data source for finding files on a local filesystem.

LocalFile(*args, **kwargs)

File on the local filesystem.

Data:

GRIB_FORMATS

GRIB file extensions.

Functions:

find_files(*[, debug])

Find files on the local filesystem.
class esmvalcore.local.DataSource(*args, **kwargs)[source]#

Bases: LocalDataSource

Data source for finding files on a local filesystem.

Deprecated since version 2.14.0: This class is deprecated and will be removed in version 2.16.0. Please use esmvalcore.local.LocalDataSource instead.

Methods:

find_files(**facets)

Find files.

get_glob_patterns(**facets)

Compose the globs that will be used to look for files.

path2facets(path, add_timerange)

Extract facets from path.

Attributes:

regex_pattern

Get regex pattern that can be used to extract facets from paths.
find_files(**facets) → list[LocalFile][source]#

Find files.

Return type:

list[LocalFile]

get_glob_patterns(**facets) → list[Path][source]#

Compose the globs that will be used to look for files.

Return type:

list[Path]

path2facets(path: Path, add_timerange: bool) → dict[str, str][source]#

Extract facets from path.

Parameters:
Return type:

dict[str, str]

property regex_pattern: str#

Get regex pattern that can be used to extract facets from paths.

esmvalcore.local.GRIB_FORMATS = ('.grib2', '.grib', '.grb2', '.grb', '.gb2', '.gb')#

GRIB file extensions.

class esmvalcore.local.LocalDataSource(name: str, project: str, priority: int, rootpath: ~pathlib.Path, dirname_template: str, filename_template: str, ignore_warnings: list[dict[str, ~typing.Any]] | None = <factory>)[source]#

Bases: DataSource

Data source for finding files on a local filesystem.

Attributes:

debug_info

A string containing debug information when no data is found.

dirname_template

The template for the directory names.

filename_template

The template for the file names.

ignore_warnings

Warnings to ignore when loading the data.

name

A name identifying the data source.

priority

The priority of the data source.

project

The project that the data source provides data for.

rootpath

The path where the directories are located.

Methods:

find_data(**facets)

Find data locally.
Parameters:
debug_info: str = ''#

A string containing debug information when no data is found.

dirname_template: str#

The template for the directory names.

filename_template: str#

The template for the file names.

find_data(**facets) → list[LocalFile][source]#

Find data locally.

Parameters:

**facets – Find data matching these facets.

Returns:

A list of files.

Return type:

list[LocalFile]

ignore_warnings: list[dict[str, Any]] | None#

Warnings to ignore when loading the data.

The list should contain dict`s with keyword arguments that will be passed to the :func:`warnings.filterwarnings function when calling LocalFile.to_iris().

name: str#

A name identifying the data source.

priority: int#

The priority of the data source. Lower values have priority.

project: str#

The project that the data source provides data for.

rootpath: Path#

The path where the directories are located.

class esmvalcore.local.LocalFile(*args, **kwargs)[source]#

Bases: PosixPath, DataElement

File on the local filesystem.

Attributes:

attributes

Attributes are key-value pairs describing the data.

facets

Facets are key-value pairs that were used to find this data.

ignore_warnings

Warnings to ignore when loading the data.

Methods:

prepare()

Prepare the data for access.

to_iris()

Load the data as Iris cubes.
property attributes: dict[str, Any]#

Attributes are key-value pairs describing the data.

property facets: Facets#

Facets are key-value pairs that were used to find this data.

property ignore_warnings: list[dict[str, Any]] | None#

Warnings to ignore when loading the data.

The list should contain dict`s with keyword arguments that will be passed to the :func:`warnings.filterwarnings function when calling the to_iris method.

prepare() → None[source]#

Prepare the data for access.

Return type:

None

to_iris() → CubeList[source]#

Load the data as Iris cubes.

Returns:

The loaded data.

Return type:

iris.cube.CubeList

esmvalcore.local.find_files(*, debug: bool = False, **facets: FacetValue) → list[LocalFile] | tuple[list[LocalFile], list[Path]][source]#

Find files on the local filesystem.

Deprecated since version 2.14.0: This function is deprecated and will be removed in version 2.16.0. Please use esmvalcore.local.LocalDataSource.find_data() instead.

The directories that are searched for files are defined in esmvalcore.config.CFG under the 'rootpath' key using the directory structure defined under the 'drs' key. If esmvalcore.config.CFG['rootpath'] contains a key that matches the value of the project facet, those paths will be used. If there is no project specific key, the directories in esmvalcore.config.CFG['rootpath']['default'] will be searched.

See Input data for extensive instructions on configuring ESMValCore so it can find files locally.

Parameters:

  • debug (bool) – When debug is set to True, the function will return a tuple with the first element containing the files that were found and the second element containing the glob.glob() patterns that were used to search for files.

  • **facets (FacetValue) – Facets used to search for files. An '*' can be used to match any value. By default, only the latest version of a file will be returned. To select all versions use version='*'. It is also possible to specify multiple values for a facet, e.g. exp=['historical', 'ssp585'] will match any file that belongs to either the historical or ssp585 experiment. The timerange facet can be specified in ISO 8601 format.

Return type:

list[LocalFile] | tuple[list[LocalFile], list[Path]]

Note

A value of timerange='*' is supported, but combining a '*' with a time or period as supported in the recipe is currently not supported and will return all found files.

Examples

Search for files containing surface air temperature from any CMIP6 model for the historical experiment:

>>> esmvalcore.local.find_files(
...     project='CMIP6',
...     activity='CMIP',
...     mip='Amon',
...     short_name='tas',
...     exp='historical',
...     dataset='*',
...     ensemble='*',
...     grid='*',
...     institute='*',
... )
[LocalFile('/home/bandela/climate_data/CMIP6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/tas/gn/v20181214/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc')]
Returns:

The files that were found.

Return type:

list[LocalFile]

Parameters:

  • debug (bool)

  • facets (FacetValue)