data_types
- class txpipe.data_types.base.DataFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
object
A class representing a DataFile to be made by pipeline stages and passed on to subsequent ones.
DataFile itself should not be instantiated - instead subclasses should be defined for different file types.
These subclasses are used in the definition of pipeline stages to indicate what kind of file is expected. The “suffix” attribute, which must be defined on subclasses, indicates the file suffix.
The open method, which can optionally be overridden, is used by the machinery of the PipelineStage class to open an input our output named by a tag.
- add_provenance(key, value)[source]
Concrete subclasses (for which it is possible) should override this method to save the a new string key/value pair to file
- static generate_provenance(extra_provenance=None)[source]
Generate provenance information - a dictionary of useful information about the origina
- classmethod open(path, mode)[source]
Open a data file. The base implementation of this function just opens and returns a standard python file object.
Subclasses can override to either open files using different openers (like fitsio.FITS), or, for more specific data types, return an instance of the class itself to use as an intermediary for the file.
- read_provenance()[source]
Concrete subclasses for which it is possible should override this method and read the provenance information from the file.
Other classes will return this dictionary of UNKNOWNs
- suffix = None
- supports_parallel_write = False
- class txpipe.data_types.base.Directory(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.DataFile
- read_provenance()[source]
Concrete subclasses for which it is possible should override this method and read the provenance information from the file.
Other classes will return this dictionary of UNKNOWNs
- suffix = ''
- class txpipe.data_types.base.FileCollection(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.Directory
Represents a grouped bundle of files, for cases where you don’t know the exact list in advance.
- path_for_file(filename)[source]
Get the path for a file inside the collection. Does not check if the file exists or anything like that.
- suffix = ''
- class txpipe.data_types.base.FitsFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.DataFile
A data file in the FITS format. Using these files requires the fitsio package.
- missing_columns(columns, hdu=1)[source]
Check that all supplied columns exist and are in the chosen HDU
- read_provenance()[source]
Concrete subclasses for which it is possible should override this method and read the provenance information from the file.
Other classes will return this dictionary of UNKNOWNs
- required_columns = []
- suffix = 'fits'
- class txpipe.data_types.base.HDFFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.DataFile
- read_provenance()[source]
Concrete subclasses for which it is possible should override this method and read the provenance information from the file.
Other classes will return this dictionary of UNKNOWNs
- required_datasets = []
- suffix = 'hdf5'
- supports_parallel_write = True
A data file in the HDF5 format. Using these files requires the h5py package, which in turn requires an HDF5 library installation.
- class txpipe.data_types.base.PNGFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.DataFile
- read_provenance()[source]
Concrete subclasses for which it is possible should override this method and read the provenance information from the file.
Other classes will return this dictionary of UNKNOWNs
- suffix = 'png'
- class txpipe.data_types.base.PickleFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.DataFile
- read_provenance()[source]
Concrete subclasses for which it is possible should override this method and read the provenance information from the file.
Other classes will return this dictionary of UNKNOWNs
- suffix = 'pkl'
- class txpipe.data_types.base.TextFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.DataFile
A data file in plain text format.
- suffix = 'txt'
- class txpipe.data_types.base.YamlFile(path, mode, extra_provenance=None, validate=True, load_mode='full')[source]
Bases:
txpipe.data_types.base.DataFile
A data file in yaml format. The top-level object in TXPipe YAML files should always be a dictionary.
- read_provenance()[source]
Concrete subclasses for which it is possible should override this method and read the provenance information from the file.
Other classes will return this dictionary of UNKNOWNs
- suffix = 'yml'
This file contains TXPipe-specific file types, subclassing the more generic types in base.py
- class txpipe.data_types.types.CSVFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.DataFile
- suffix = 'csv'
- class txpipe.data_types.types.ClusteringNoiseMaps(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
- class txpipe.data_types.types.FiducialCosmology(path, mode, extra_provenance=None, validate=True, load_mode='full')[source]
- class txpipe.data_types.types.LensingNoiseMaps(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.types.MapsFile
- required_datasets = []
- class txpipe.data_types.types.MapsFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.HDFFile
- required_datasets = []
- write_map(map_name, pixel, value, metadata)[source]
Save an output map to an HDF5 subgroup.
The pixel numbering and the metadata are also saved.
- Parameters
group (H5Group) – The h5py Group object in which to store maps
name (str) – The name of this map, used as the name of a subgroup in the group where the data is stored.
pixel (array) – Array of indices of observed pixels
value (array) – Array of values of observed pixels
metadata (mapping) – Dict or other mapping of metadata to store along with the map
- class txpipe.data_types.types.NOfZFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.HDFFile
- required_datasets = []
- class txpipe.data_types.types.PhotozPDFFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.HDFFile
- required_datasets = []
- class txpipe.data_types.types.RandomsCatalog(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.HDFFile
- required_datasets = ['randoms/ra', 'randoms/dec']
- class txpipe.data_types.types.SACCFile(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.DataFile
- read_provenance()[source]
Concrete subclasses for which it is possible should override this method and read the provenance information from the file.
Other classes will return this dictionary of UNKNOWNs
- suffix = 'sacc'
- class txpipe.data_types.types.ShearCatalog(*args, **kwargs)[source]
Bases:
txpipe.data_types.base.HDFFile
A generic shear catalog
- property catalog_type
- class txpipe.data_types.types.TomographyCatalog(path, mode, extra_provenance=None, validate=True, **kwargs)[source]
Bases:
txpipe.data_types.base.HDFFile
- required_datasets = []