c3s_sm package

Submodules

c3s_sm.cli module

c3s_sm.const module

c3s_sm.const.check_api_read() → bool

c3s_sm.download module

Module to download c3s soil moisture data from the CDS

c3s_sm.download.download_and_extract(target_path, startdate=datetime.datetime(1978, 1, 1, 0, 0), enddate=datetime.datetime(2024, 10, 21, 15, 53, 52, 692099), product='combined', freq='daily', version='v202212', keep_original=False, dry_run=False)

Downloads the data from the CDS servers and moves them to the target path. This is done in 30 day increments between start and end date. The files are then extracted into yearly folders under the target_path.

Parameters:

• target_path (str) – Path where the files are stored to

• startdate (datetime, optional (default: datetime(1978,1,1))) – first day to download data for (if available)

• enddate (datetime, optional (default: datetime.now())) – last day to download data for (if available)

• product (str, optional (default: 'combined')) – Product (combined, active, passive) to download

• freq (str, optional (default: 'daily')) – ‘daily’, ‘dekadal’ or ‘monthly’ averaged data to download.

• version (str, optional (default: 'v202212')) – Dataset version to download.

• keep_original (bool, optional (default: False)) – Keep the original downloaded data in zip format together with the unzipped files.

• dry_run (bool, optional (default: False)) – Does not download anything, returns query, success is False

Returns:

queries – List[dict]: All submitted queries

Return type:

list

c3s_sm.download.download_c3ssm(c, sensor, years, months, days, version, target_dir, temp_filename, freq='daily', keep_original=False, max_retries=5, dry_run=False)

Download c3s sm data for single levels of a defined time span Parameters. We will always try to download the CDR and ICDR!

Parameters:

• c (cdsapi.Client) – Client to pass the request to

• sensor (str) – active, passive or combined. The sensor product to download

• years (list) – Years for which data is downloaded ,e.g. [2017, 2018]

• months (list) – Months for which data is downloaded, e.g. [4, 8, 12]

• days (list) – Days for which data is downloaded (range(31)=All days) e.g. [10, 20, 31]

• version (str) – Version string of data to download, e.g. ‘v202212’

• target_dir (str) – Directory where the data is downloaded into

• temp_filename (str) – filename of the zip archive that will be downloaded

• freq (str, optional (default: daily)) – daily, dekadal or monthly. Which of the three aggregated products to download.

• keep_original (bool, optional (default: False)) – Whether the original file retrieved from CDS should be kept. If False, then only the extracted images are kept.

• max_retries (int, optional (default: 5)) – When a download failes, try again up to max_retries times

• dry_run (bool, optional (default: False)) – Does not download anything, returns query, success is False

Returns:

• success (dict[str, bool]) – Indicates whether the download was successful, False for dry_run=True

• queries (dict[str, dict]) – icdr and cdr query that were submitted

c3s_sm.download.first_missing_date(last_date: str, freq: str = 'daily') → datetime

For a product, based on the last available date, find the next expected one.

c3s_sm.download.logger(fname, level=10, verbose=False)

c3s_sm.interface module

Readers for the C3S soil moisture products daily, dekadal (10-daily) and monthly images as well as for timeseries generated using this module

class c3s_sm.interface.C3SImg(filename, parameters=None, mode='r', subgrid=<smecv_grid.grid.SMECV_Grid_v052 object>, flatten=False, fillval={'sm': nan, 'sm_uncertainty': nan, 't0': nan})

Bases: ImageBase

Class to read a single C3S image (for one time stamp)

close(*args, **kwargs)

Close file.

flush(*args, **kwargs)

Flush data.

read(timestamp=None)

Read a single C3S image, if it exists, otherwise fill an empty image.

Parameters:

timestamp (datetime, optional (default: None)) – Time stamp of the image, if this is passed, it is compared to the time stamp from the loaded file and must match

write(*args, **kwargs)

Write data to an image file.

Parameters:

image (object) – pygeobase.object_base.Image object

class c3s_sm.interface.C3STs(ts_path, grid_path=None, remove_nans=False, drop_tz=True, **kwargs)

Bases: GriddedNcOrthoMultiTs

Module for reading C3S time series in netcdf format.

iter_ts(**kwargs)

read_cell(cell, var='sm') → DataFrame

Read all time series for a single variable in the selected cell.

Parameters:

• cell (int) – Cell number as in the c3s grid

• var (str, optional (default: 'sm')) – Name of the variable to read.

write_ts(*args, **kwargs)

class c3s_sm.interface.C3S_Nc_Img_Stack(data_path, parameters='sm', subgrid=<smecv_grid.grid.SMECV_Grid_v052 object>, flatten=False, solve_ambiguity='sort_last', fntempl='C3S-SOILMOISTURE-L3S-SSM{unit}-{product}-{freq}-{datetime}-{record}-{version}.{subversion}.nc', subpath_templ=('%Y', ), fillval=None)

Bases: MultiTemporalImageBase

Class for reading multiple images and iterate over them.

read(timestamp, **kwargs)

Return an image for a specific timestamp.

Parameters:

timestamp (datetime.datetime) – Time stamp.

Returns:

image – pygeobase.object_base.Image object

Return type:

object

tstamps_for_daterange(start_date, end_date)

Return dates in the passed period, with respect to the temp resolution of the images in the path.

Parameters:

• start_date (datetime) – start of date range

• end_date (datetime) – end of date range

Returns:

timestamps – list of datetime objects of each available image between start_date and end_date

Return type:

Iterator

c3s_sm.metadata module

class c3s_sm.metadata.C3S_SM_TS_Attrs(sensor_type, version)

Bases: object

Default, common metadata for daily and monthly, dekadal products

atts_sensor_type(sensor_type='active')

dn_flag()

flag()

freqbandID_flag()

mode_flag()

sensor_flag()

class c3s_sm.metadata.C3S_daily_tsatt_nc(cdr_type: str, sensor_type: str, version: str, cls)

Bases: C3S_SM_TS_Attrs

class c3s_sm.metadata.C3S_dekmon_tsatt_nc(freq: str, cdr_type: str, sensor_type: str, version: str, cls)

Bases: object

Attributes for c3s dekadal and monthly for active, passive and combined tcdr and icdr timeseries files.

c3s_sm.misc module

c3s_sm.misc.collect_ts_cov(data_path: str, n_proc=1, progressbar=False)

Open all time series files in a directory (slow) and detect the temporal coverage.

Parameters:

• data_path (str) – Path where the cell files are stored.

• progressbar (bool, optional (default: False)) – Show progress bar when looping through files.

Returns:

periods – Periods coverged by the time series files. {(start, end): [cell, cell, …], …}

Return type:

dict

c3s_sm.misc.delete_empty_directories(path: str)

Delete empty dirs in path

c3s_sm.misc.get_first_image_date(path: str, fntempl: str = 'C3S-SOILMOISTURE-L3S-SSM{unit}-{product}-{freq}-{datetime}-{record}-{version}.{subversion}.nc') → str

Parse files in the given directory (or any subdir) using the passed filename template. props will contain all fields specified in the template. the datetime field is required and used to determine the first image date.

Parameters:

• path (str) – Path to the directory containing the image files

• fntempl (str, optional) – The filename template used to parse image file names. Must contain a field datetime that indicates the location of the image time stamp in the filename.

Returns:

date – Parse date from the first found image file that matches fntempl.

Return type:

str

c3s_sm.misc.get_last_image_date(path: str, fntempl: str) → str

Parse files in the given directory (or any subdir) using the passed filename template. props will contain all fields specified in the template. the datetime field is required and used to determine the last image date.

Parameters:

• path (str) – Path to the directory containing the image files

• fntempl (str (optional)) – The filename template used to parse image file names. Must contain a field datetime that indicates the location of the image time stamp in the filename.

Returns:

date – Parse date from the last found image file that matches fntempl.

Return type:

str

c3s_sm.misc.img_infer_file_props(path: str, fntempl: str = 'C3S-SOILMOISTURE-L3S-SSM{unit}-{product}-{freq}-{datetime}-{record}-{version}.{subversion}.nc', start_from='last') → dict

Parse file names to retrieve properties from c3s_sm.const.fntempl(). Does not open any files.

c3s_sm.misc.read_summary_yml(path: str) → dict

Read image summary and return fields as dict.

c3s_sm.misc.update_image_summary_file(data_path: str, out_file=None, fntempl: str = 'C3S-SOILMOISTURE-L3S-SSM{unit}-{product}-{freq}-{datetime}-{record}-{version}.{subversion}.nc')

Summarize image metadata as yml file

Parameters:

• data_path (str) – Root path to the image archive

• out_file (str, optional (default: None)) – Path to summary file. File will be created/updated. If not specified, then data_path is used. If a file already exists, it will be overwritten.

• fntempl (str, optional) – The filename template used to parse image file names. Must contain a field datetime that indicates the location of the image time stamp in the filename.

c3s_sm.misc.update_ts_summary_file(data_path, props=None, collect_cov=False, **kwargs)

Create a summary yml file that contains the most relevant information for a dataset. This information is then also used to update/extent the record.

Parameters:

• data_path (str) – Path where the time series files are stored and where the file overview.yml will be stored.

• props (dict, optional) – Additional information to write down

• collect_cov (bool, optional (False)) – Loop through all available cell files and detect their tempora coverage. Will add one or multiple periodX fields to the summary file, which indicate the temporal coverage of each cell.

• kwargs – Additional keyword arguments are passed to collect_ts_cov

c3s_sm.reshuffle module

Module for a command line interface to convert the C3S data into a time series format using the repurpose package

c3s_sm.reshuffle.extend_ts(img_path, ts_path, fntempl='C3S-SOILMOISTURE-L3S-SSM{unit}-{product}-{freq}-{datetime}-{record}-{version}.{subversion}.nc', startdate=None, freq=None, n_proc=1)

Append any new data from the image path to the time series data. This function is only applied to time series file that were created using the img2ts function. This will use the start date from each cell, and process only parameters that are already present in the time series files.

Parameters:

• img_path (str) – Path where the image files are stored

• ts_path (str) – Path where the time series files to append to are stored

• fntempl (str, optional (default: see c3s_sm.const.fntempl)) – Template that image files follow, must contain a section {datetime} where the date is parsed from.

• startdate (str, optional (default: None)) – Date of the first image to append. If None, then we use the next available image to the last date in the time series.

• freq (str, optional (default: None)) – DAILY, DEKADAL or MONTHLY. If None is passed, freq must be inferable from metadata.

• n_proc (int, optional (default: 1)) – Number of parallel processes to read and write data.

c3s_sm.reshuffle.img2ts(img_path, ts_path, startdate, enddate, parameters=None, land_points=True, bbox=None, cells=None, ignore_meta=False, fntempl='C3S-SOILMOISTURE-L3S-SSM{unit}-{product}-{freq}-{datetime}-{record}-{version}.{subversion}.nc', overwrite=False, imgbuffer=250, n_proc=1)

Reshuffle method applied to C3S data.

Parameters:

• img_path (str) – input path where c3s images were downloaded.

• ts_path (str, optional (default: None)) – Output path.

• startdate (datetime or str) – Start date. If None is passed, then we will try to detect the start of the available image files in img_path

• enddate (datetime or str) – End date. If None is passed, then we will try to detect the date of the last available image file

• parameters (list, optional (default: None)) – parameters to read and convert to time series. If None is passed, then we use all available parameters from the input files.

• land_points (bool, optional (default: True)) – Use the land grid to calculate time series on. Leads to faster processing and smaller files.

• bbox (tuple, optional (default: None)) – Min lon, min lat, max lon, max lat BBox to read data for. Data outside the bbox is ignored in the conversion step. This option is incompatible with the cells keyword.

• cells (tuple, optional (default: None)) – To limit the processing to certain cells. This option is incompatible with the bbox keyword.

• ignore_meta (bool, optional (default: False)) – Ignore metadata and reshuffle only the values. Can be used e.g. if a version is not yet supported.

• fntempl (str, optional (default: see c3s_sm.const.fntempl)) – Template that image files follow, must contain a section {datetime} where the date is parsed from.

• overwrite (bool, optional (default: False)) – If this option is activated, then any existing files on the output directory will be deleted before the conversion takes place. Otherwise the program will try to append new data to existing files. To extend existing time series, it is recommended to use the extend_ts function instead.

• imgbuffer (int, optional (default: 250)) – How many images to read at once before writing time series.

• n_proc (int, optional (default: 1)) – Number of parallel processes to read and write data.

c3s_sm.reshuffle.parse_filename(data_dir, fntempl='C3S-SOILMOISTURE-L3S-SSM{unit}-{product}-{freq}-{datetime}-{record}-{version}.{subversion}.nc')

Take the first file in the passed directory and use its file name to retrieve the product type, version number and variables in the file.

Parameters:

• inroot (str) – Input root directory

• fntempl (str, optional (default: c3s_sm.const.fntempl)) – Filename template

Returns:

• file_args (dict) – Parsed arguments from file name

• file_vars (list) – Names of parameters in the first detected file

c3s_sm.reshuffle.reshuffle(*args, **kwargs)

Module contents