Utilities

class txpipe.utils.calibration_tools.HSCCalculator(selector)

This class builds up the total response calibration factors for HSC-convention shear-calibration from each chunk of data it is given. At the end an MPI communicator can be supplied to collect together the results from the different processes.

add_data(data, *args, **kwargs)

Select objects from a new chunk of data and tally their responses

Parameters

• data (dict) – Dictionary of data columns to select on and add

• *args – Positional arguments to be passed to the selection function

• **kwargs – Keyword arguments to be passed to the selection function

collect(comm=None, allgather=False)

Finalize and sum up all the response values, returning calibration quantities.

Parameters

comm (MPI Communicator) – If supplied, all processors response values will be combined together. All processes will return the same final value

Returns

• R (float) – R calibration factor

• K (float) – K = (1+m) calibration

• N (int) – Total object count

class txpipe.utils.calibration_tools.LensfitCalculator(selector, input_m_is_weighted=False)

This class builds up the total calibration factors for lensfit-convention shears from each chunk of data it is given. Note here we derive the c-terms from the data (in constrast to averaging values derived from simulations and stored in the catalog.) At the end an MPI communicator can be supplied to collect together the results from the different processes.

add_data(data, *args, **kwargs)

Select objects from a new chunk of data and tally their responses

Parameters

• data (dict) – Dictionary of data columns to select on and add

• *args – Positional arguments to be passed to the selection function

• **kwargs – Keyword arguments to be passed to the selection function

collect(comm=None, allgather=False)

Finalize and sum up all the response values, returning calibration quantities.

Parameters

comm (MPI Communicator) – If supplied, all processors response values will be combined together. All processes will return the same final value

Returns

• K (float) – K = (1+m) calibration

• C (float array) – c1, c2 additive biases (weighted average of g1 and g2)

class txpipe.utils.calibration_tools.MetacalCalculator(selector, delta_gamma)

This class builds up the total response and selection calibration factors for Metacalibration from each chunk of data it is given. At the end an MPI communicator can be supplied to collect together the results from the different processes.

To do this we need the function used to select the data, and the instance this function to each of the metacalibrated variants automatically by wrapping the data object passed in to it and modifying the names of columns that are looked up.

add_data(data, *args, **kwargs)

Select objects from a new chunk of data and tally their responses

Parameters

• data (dict) – Dictionary of data columns to select on and add

• *args – Positional arguments to be passed to the selection function

• **kwargs – Keyword arguments to be passed to the selection function

collect(comm=None, allgather=False)

Finalize and sum up all the response values, returning separate R (estimator response) and S (selection bias) 2x2 matrices :param comm: If supplied, all processors response values will be combined together.

All processes will return the same final value

Returns

• R (2x2 array) – Estimator response matrix

• S (2x2 array) – Selection bias matrix

txpipe.utils.calibration_tools.read_shear_catalog_type(stage)

Determine the type of shear catalog a stage is using as input. Returns a string, e.g. metacal, lensfit. Also sets shear_catalog_type in the stage’s configuration so that it is available later and is saved in output.

txpipe.utils.fitting.fit_straight_line(x, y, x_err=None, y_err=None, m0=1.0, c0=0.0, nan_error=False, skip_nan=False)

Use scipy to fit a straight line, with errors or covariances allowed on both x and y.

Parameters

• x (array) – x-coordinate

• y (array) – y-coordinate

• m0 (float) – optional, default=1, guess at gradient

• c0 (float) – optional, default=1, guess at intercept

• x_err (array/float) – optional, default=None, errors either 1D std. dev., 2D covariance, or scalar constant, or None for unspecified

• y_err (array/float) – optional, default=None, errors either 1D std. dev., 2D covariance, or scalar constant, or None for unspecified

Returns

• m (float) – gradient

• c (float) – intercept

txpipe.utils.hdf_tools.create_dataset_early_allocated(group, name, size, dtype)

Create an HdF5 dataset, allocating the full space for it at the start of the process. This can make it faster to write data incrementally from multiple processes. The dataset is also not pre-filled, saving more time.

Parameters

• group (h5py.Group) – the parent for the dataset

• name (str) – name for the new dataset

• size (int) – The size of the new data set (which must be 1D)

• dtype (str) – Data type, One of f4, f8, i4, i8

txpipe.utils.hdf_tools.repack(filename)

In-place HDF5 repack operation on file.

txpipe.utils.healpix.dilated_healpix_map(m)

Dilate a healpix map - every pixel with a neighbour that is UNSEEN gets set to unseen as well

Parameters

m (array) – Healpix float map

Returns

m2 – Matching-sized map with edge pixels UNSEEN

Return type

array

txpipe.utils.misc.hex_escape(s, replace_newlines=False)

Replace non-printable characters in a string with hex-code equivalents so they can be printed or saved to a FITS file header. Newline characters will be replaced if replace_newlines is set to true

Based on: https://stackoverflow.com/a/13935582

Parameters

• s (str) – The initial string

• replace_newlines (bool) – Whether to include newline characters in the replacement

Returns

Same string with unprintable chars replaced with hex codes, e.g. the bell character becomes “”

Return type

str

txpipe.utils.misc.rename_iterated(it, renames)

Rename the items in dictionaries yielded by an interator.

In several places we load data from catalogs chunk by chunk, yielding a dictionary of data each time. This renames columns in each chunk.

Parameters

• it (iterator) – Must yield triplets of (any, any, data)

• renames (dict) – dictionary of old names mapped to new names

txpipe.utils.misc.unique_list(seq)

Find the unique elements in a list or other sequence while maintaining the order. (i.e., remove any duplicated elements but otherwise leave it the same)

Method from: https://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-whilst-preserving-order

Parameters

seq (list or sequence) – Any input object that can be iterated

Returns

L – a new list of the unique objects

Return type

list

txpipe.utils.mpi_utils.mpi_reduce_large(data, comm, max_chunk_count=1073741824, root=0, op=None, debug=False)

Use MPI reduce in-place on an array, even a very large one.

MPI Reduce is a reduction operation that will, (e.g.) sum arrays from different processors on a single process.

It fails whenever the size of the array is greater than 2**31, due to an overflow. This version detects that case and divides the array up into chunks, running reduction on each one separately

This specific call does in-place reduction, so that the root process overwrites its own array with the result. This minimizes memory usage.

The default is to do a sum of all the arrays, and to collect at process zero, but those can be overridden.

Parameters

• data (array) – can be any shape but must be contiguous

• comm (MPI communictator) –

• max_chunk_count (int) – Optional, default=2**30. Max number of items to allow to be sent at once

• root (int) – Optional, default=0. Rank of process to receive final result

• op (MPI operation) – Optional, default=None. MPI operation, e.g. MPI.PROD, MPI.MAX, etc. Default is to SUM

• debug (bool) – Optional, default=False. Whether to print out information from each rank

class txpipe.utils.nmt_utils.MyNmtBin(*args: Any, **kwargs: Any)

class txpipe.utils.nmt_utils.MyNmtBinFlat(*args: Any, **kwargs: Any)

txpipe.utils.provenance.find_module_versions()

Generate a dictionary of versions of all imported modules by looking for __version__ or version attributes on them.

txpipe.utils.provenance.get_caller_directory(grandparent=False)

Find the directory where the code calling this module lives, or where the code calling that code lives if grandparent=True.

txpipe.utils.provenance.git_current_revision()

Run git diff in the caller’s directory, and return stdout+stderr

txpipe.utils.provenance.git_diff()

Run git diff in the caller’s directory, and return stdout+stderr

txpipe.utils.theory.theory_3x2pt(cosmology_file, tracers, nbin_source, nbin_lens, fourier=True)

Compute the 3x2pt theory, for example for a fiducial cosmology

Parameters

• cosmology_file (str) – name of YAML file

• tracers (dict{str:obj}) – dict of objects (e.g. sacc tracers) containing z, nz. keys are source_0, source_1, …, lens_0, lens_1, …

• nbin_source (int) – number of source bins

• nbin_lens (int) – number of lens bins

Returns

theory_cl – theory c_ell for each pair (i,j,k) where k is one of SHEAR_SHEAR = 0, SHEAR_POS = 1, POS_POS = 2

Return type

dict{str:array}