The `dataregistry` package

Reference documentation for the core objects within the dataregistry package. Demonstrations of their usage can be found in the tutorials section.

The DataRegistry class

The DataRegistry class is the primary front end to the dataregistry package. This should be the only object users have to import to their code.

It connects the user to the database, and serves as a wrapper to both the Registrar and Query classes.

class dataregistry.DataRegistry(owner=None, owner_type=None, config_file=None, root_dir=None, logging_level=20, site=None, namespace=None, schema=None, entry_mode='working', query_mode='working')

Primary data registry wrapper class.

Each DataRegistry instance has as members an instance of the Registrar class, to register/modify/delete datasets, and of the Query class, to query existing datasets.

Access to the database is handled automatically using:

the users config file (if None defaults are used)
the passed schema (if None the default schema is used)

The root_dir is the location the data is copied to. This can be manually passed, or alternately a predefined site can be chosen. If nether are chosen, the NERSC site will be selected as the default.

Parameters:

owner (str) – To set the default owner for all registered datasets in this instance.
owner_type (str) – To set the default owner_type for all registered datasets in this instance, one of “production”, “project”, “group”, “user”. Defaults to “user”.
config_file (str) – Path to config file, if None, use default NERSC config.
root_dir (str) – Root directory for datasets, if None, default is assumed.
logging_level (int, optional) – Level for the logger output (default is logging.INFO)
site (str) – Can be used instead of root_dir. Some predefined “sites” are built in, such as “nersc”, which will set the root_dir to the data registry’s default data location at NERSC.
namespace (str, optional) – Namespace to connect to. If None, the default namespace (“lsst_desc”) will be used
schema (str, optional) – Schema to connect to, to connect directly to a chosen schema, bypassing the namespace.
entry_mode (str, optional) – Which schema (“working” or “production”) within the namespace to use when writing/modifying/deleting entries.
query_mode (str, optional) – Which schema(s) (“working” or “production”) to probe when querying. By default query_mode=”working”, however this can be set to either “production” to search only that schema or “both” to search both.

Registrar.get_owner_types()

Returns a list of allowed owner_types that can be registered within the data registry.

Returns:: - – Set of owner_types
Return type:: set

Query.find_datasets(property_names=None, filters=[], return_format='property_dict', strip_table_names=False, schema_mode=None)

Get specified properties for datasets satisfying all filters. Both schemas (i.e., the working and production schema) are searched, with the results combined.

If property_names is None, return all properties from the dataset table (only). Otherwise, return the property_names columns for each discovered dataset (which can be from multiple tables via a join).

Filters should be a list of dataregistry Filter objects, which are logic constraints on column values.

These choices get translated into an SQL query.

Parameters:

property_names (list, optional) – List of database columns to return (SELECT clause)
filters (list, optional) – List of filters (WHERE clauses) to apply
return_format (str, optional) – The format the query result is returned in. Options are “DataFrame”, or “proprety_dict”. Note this is not case sensitive.
strip_table_names (bool, optional) – True to remove the table name in the results columns This only works if a single table is needed for the query
schema_mode (optional) – May be “production”, “working” or None. Defaults to None, in which case query mode established at connection time is used. Ignored unless query mode was “both”

Returns:

result – Requested property values

Return type:

dict, or DataFrame (depending on return_format)

add_keywords_to_dataset(dataset_id, keyword)

Add keywords to a dataset entry.

Parameters:

dataset_id (int) – Dataset id to add keyword to
keyword (list[str]) – Keywords to add to dataset

create_keywords(keywords, user_type='user', system=False, commit=True): Keyword.create_keywords for complete description of arguments

delete_dataset(name, version_string, owner, owner_type, confirm=False): Convenience function which just calls DataRegistry.registrar.dataset.delete. See DatasetTable.delete for complete argument and return description.

fetch(dataset_id, schema_type='working', destination_path=None, destination_endpoint='NERSC DTN', no_cfs_copy=False)

Fetch a registered dataset. This is just a wrapper which calls Registrar.fetch, supply the Query object as an argument.

Behavior depends on arguments and whether dataset is available in cfs or only from archive, but archiving is not yet implemented, so the only possibility is:

If destination_path is not None, copy from cfg to user-specified path, at user-specified globus endpoint.

Parameters:

dataset_id (int) – id of dataset to be retrieved
schema_type (string) – one of “working” (the default) or “production”
destination_path (string) – where to put the dataset. If None, defaults to absolute path in cfs assigned to this dataset
destination_endoint (string) – globus endpoint to which dataa will be written. Defaults to “NERSC DTN”
no_cfs_copy (boolean) – If True and dataset was absent from cfs, write directly to to the destination requested; do not also restore to cfs.

Return type:

Absolute cfs path of dataset when it was registered

find_datasets(**kwargs): Convenience function which just calls the find_datasets function of the Query object. See full documentation there.

get_all_columns(table='dataset', include_table=True, include_schema=False): See Query.get_all_columns for complete description

get_all_tables(): See Query.get_all_tables for complete description)

get_dataset_absolute_path(dataset_id, schema=None, silent=True): See Query.get_dataset_absolute_path for complete description.

get_keyword_list(query_mode=None): See Query.get_keyword_list for complete description

get_modifiable_columns(table='dataset'): See full documentation under DatasetTable.get_modifiable_columns()

modify_dataset(dataset_id, update_dict): Convenience function which just calls DataRegistry.registrar.dataset.modify. See DatasetTable.modify for complete argument and return description.

modify_execution(execution_id, update_dict): Convenience function which just calls DataRegistry.registrar.execution.modify. See ExecutionTable.modify for complete argument and return description.

register_dataset(name, version, **kw): Convenience function which just calls DataRegistry.registrar.dataset.register. See DatasetTable.register for complete argument and return description.

register_execution(name, **kwargs): See full documentation under ExecutionTable.register

remove_keywords_froom_dataset(dataset_id, keyword)

Remove keywords from a dataset entry.

Parameters:

dataset_id (int) – Dataset id to remove keyword from
keyword (list[str]) – Keywords to remove from dataset

replace_dataset(name, version, **kw): Convenience function which just calls DataRegistry.registrar.dataset.replace. See DatasetTable.replace for complete argument and return description.

simple_query(return_format='list_of_dicts', columns=None, **conditions)

Run a query on the registry with a simple syntax. For example, you can do:

# everything belonging to a specific owner results = registry.simple_query(owner=”jbogart”)

# a specific dataset results = registry.simple_query(dataset_id=30)

# you can combine search terms, which are ANDed together: results = registry.simple_query(owner=”jbogart”, version_major=2)

Current supported search terms are:: access_api access_api_configuration archive_date archive_path contact_email creation_date creator_uid data_org dataset_id delete_date delete_uid description execution_id is_overwritable location_type move_date name nfiles owner owner_type register_date register_root_dir relative_path replace_id replace_iteration status total_disk_space url version_major version_minor version_patch version_string path

Parameters:

return_format (str) – The format to return the results in. Options are “list_of_dicts”, “dataframe”, “dict_of_lists”. The default is “list_of_dicts”. “dict_of_lists” matches the format return by find_datasets.
columns (list of str, optional) – If not None, only return these columns in the results. The column names should be from the list of search terms above, without the “dataset.”
**conditions (dict) – The query parameters. These should be of the form field=value, where field is one of the search terms listed above.

Returns:

A list of datasets matching the query, where each dataset is represented as a dict of its metadata fields. The keys of the dicts will be the same as the search terms listed above. Note that internally the keys are prefixed with “dataset.”, but this function will remove that prefix for convenience.

Return type:

list of dict

DatasetTable.register(name, version, creation_date=None, description=None, execution_id=None, access_api=None, access_api_configuration=None, is_overwritable=False, old_location=None, owner=None, owner_type=None, execution_name=None, execution_description=None, execution_start=None, execution_site=None, execution_configuration=None, input_datasets=[], input_production_datasets=[], max_config_length=None, keywords=[], location_type='dataregistry', url=None, contact_email=None, test_production=False, relative_path=None, kwargs_dict=None)

Create a new dataset entry in the DESC data registry.

Any args marked with ‘**’ share their name with the associated column in the registry schema. Descriptions of what these columns are can be found in schema.yaml or the documentation.

First, the dataset entry is created in the database. If success, the data is then copied (if old_location was provided). Only if both steps are successful will there be “valid” status entry in the registry.

Parameters:

name** (str)
version** (str)
creation_date** (datetime, optional)
description** (str, optional)
execution_id** (int, optional)
access_api** (str, optional)
is_overwritable** (bool, optional)
old_location (str, optional) –
Absolute location of dataset to copy into the data registry.

If None, dataset should already be at correct relative_path within the data registry (for datasets of location_type “dataregistry”).
owner** (str, optional)
owner_type** (str, optional)
execution_name** (str, optional)
execution_description** (str, optional)
execution_start** (datetime, optional)
execution_site** (str, optional)
execution_configuration** (str, optional)
input_datasets (list, optional) – List of dataset ids that were the input to this execution
input_production_datasets (list, optional) – List of production dataset ids that were inputs to this execution
max_config_length (int, optional) – Maxiumum number of lines to read from a configuration file
keywords (list[str], optional) – List of keywords to tag dataset with. Each keyword must be registered already in the keywords table.
location_type** (str, optional) – If location_type=”external”, either url or contact_email must be supplied
url** (str, optional) – For location_type=”external” only
contact_email** (str, optional)
test_production (boolean, default False. Set to True for testing) – code for production owner_type
relative_path** (str, optional. Always None for datasets with) – location_type “external” or “meta_only”
kwargs_dict (dict) – Stores all the keyword arguments passed to this function (and defaults). Automatically generated by the decorator, do not pass manually.

Returns:

prim_key (int) – The dataset ID of the new row relating to this entry (else None)
execution_id (int) – The execution ID associated with the dataset

DatasetTable.replace(name, version, creation_date=None, description=None, execution_id=None, access_api=None, access_api_configuration=None, is_overwritable=False, old_location=None, owner=None, owner_type=None, execution_name=None, execution_description=None, execution_start=None, execution_site=None, execution_configuration=None, input_datasets=[], input_production_datasets=[], max_config_length=None, keywords=[], location_type='dataregistry', url=None, contact_email=None, test_production=False, kwargs_dict=None)

Replace a dataset in the registry.

This is so a user can keep the same name/version/ower/owner_type combination as a previous dataset. Note the original dataset must have is_overwritable=True to allow the replace to work.

The process is as follows:

The original dataset is deleted, and the entry in the database tagged accordingly
A new entry is made with the same name/version combination as before, and the data goes into the same relative_path as before. All other properties are what the user specifies in the replace function
The old dataset gets pointed to the new dataset saying it is the most up to date iteration

Returns:

prim_key (int) – The dataset ID of the new row relating to this entry (else None)
execution_id (int) – The execution ID associated with the dataset

DatasetTable.modify(entry_id, modify_fields)

Modify an entry in the DESC data registry. Only certain columns are allowed to be modified after registration, this is defined in the schema yaml file.

Parameters:

entry_id (int) – The dataset/execution/etc ID we wish to delete from the database
modify_fields (dict) – Dict where key is the column to modify (must be allowed to modify) and value is the desired new value for the entry

DatasetTable.delete(name, version_string, owner, owner_type, confirm=False)

Delete an dataset entry from the DESC data registry.

This will also remove the raw data from the root dir, but the dataset entry remains in the registry (now with an updated status field).

Parameters:

name/version_string/owner/owner_type (str) – Identifiers for dataset we want to delete from the registry
confirm (bool) – Will ask for a confirmation

DatasetTable.add_keywords(dataset_id, keyword)

Add keywords tags to a dataset entry.

Parameters:

dataset_id (int) – Dataset id to add keyword to
keyword (list[str]) – Keywords to add to dataset

DatasetTable.get_modifiable_columns()

Return a list of all columns in this table that are “modifiable”.

As defined in the schema yaml file.

Returns:: mod_list
Return type:: list[str]

ExecutionTable.register(name, description=None, execution_start=None, site=None, configuration=None, input_datasets=[], input_production_datasets=[], max_config_length=None)

Create a new execution entry in the DESC data registry.

Any args marked with ‘**’ share their name with the associated column in the registry schema. Descriptions of what these columns are can be found in schema.yaml or the documentation.

Parameters:

name** (str)
description** (str, optional)
execution_start** (datetime, optional)
site** (str, optional)
configuration** (str which is path to coonfiguration file for) – the code executed, optional
input_datasets** (list of ids (int) of registered datasets, optional)
input_production_datasets** (list of ids (int) of registered) – production datasets, optional
max_config_length (int, optional) – Maxiumum number of lines to read from a configuration file

Returns:

my_id – The execution ID of the new row relating to this entry

Return type:

int

DatasetAliasTable.register(aliasname, dataset_id, ref_alias_id=None, access_api=None, access_api_configuration=None, supersede=False)

Create a new dataset_alias entry in the DESC data registry. It may refer to a dataset (default) or another alias

Any args marked with ‘**’ share their name with the associated column in the registry schema. Descriptions of what these columns are can be found in schema.yaml or the documentation.

Parameters:

aliasname (str alias name)
dataset_id** (int not None if alias refers to dataset)
ref_alias_id** (int not None if alias refers to) – another alias
access_api** (str api, if any, which can read the) – dataset
access_api_configuration** (str extra information for access_api)
supersede (bool if True, create a new entry with) – this alias name even if old ones exist

Returns:

prim_key – The dataset_alias ID of the new row relating to this entry

Return type:

int

KeywordTable.create_keywords(keywords: list[str], owner_type: Literal['user', 'group', 'project', 'production'] = 'user', system: bool = False, commit: bool = True) → None

Add multiple keywords to the registry.

Parameters:

keywords (list[str]) – The keywords to add.
owner_type (string.) – A known owner type. Not currently used for anything
system (bool.) – system keywords can only be created by a privileged user. They are intended to be global
commit (bool) – if True (default) immediately commit change to db

KeywordTable.disable_keyword(keyword: str)

Disable a keyword from the registry.

keyword must exist and must be owned by the current user.

Parameters:: keyword (str) – The keyword to disable.

KeywordTable.enable_keyword(keyword: str) → None

Enable a keyword in the registry.

Keyword must exist and must be owned by the current user.

Parameters:: keyword (str) – The keyword to enable.

KeywordTable.get_keywords_from_dataset(dataset_id: int) → list[str]

Get the list of keywords associated with a dataset.

Parameters:: dataset_id (int)
Returns:: keywords
Return type:: list[str]

KeywordTable.add_keywords_to_dataset(dataset_id: int, keywords: list[str]) → None

Add/append keywords to an already existing dataset.

First check the keywords are valid, then append. If the dataset already has one or more of the passed keywords attributed to it, the keyword(s) will not be duplicated.

Parameters:

dataset_id (int)
keywords (list[str])

KeywordTable.remove_keywords_from_dataset(dataset_id: int, keywords: list[str]) → None

Remove keywords from a dataset.

Parameters:

dataset_id (int)
keywords (list[str])

The dataregistry package

The DataRegistry class

The `dataregistry` package