Creating New Catalogs

Catalogs - both the binary data files and yaml config - for some object types can be created using the script create_main.py and create_flux.py. Creation of the main catalog must precede creation of the flux catalog for the same sky region. For other object types the data are created by other means, but it is still necessary to make a suitable config fragment so that the skyCatalogs API can access the data.

The script create_main.py and its options

create_main.py has a plethora of options, but few are of interest for most invocations and even fewer are required. The options determine

• object type for which files will be created (required)

• region of the sky for which files will be created.

• where the output goes

• inputs (usually defaulted, depending on object type)

• numerous options conditioning exactly how the output is to be created, most of which are usually defaulted

One particularly handy option, --options-file, allows you to specify everything else as key-value pairs in a yaml file.

Options table

Here is the complete list of options as they appear in an options file. On the command line, prepend -- and change all underscores to hyphens.

Name	Datatype	Default	Description
object_type	string		Required. One of {star, sso, cosmodc2_galaxy, diffsky_galaxy}
catalog_dir	string	“.”	Location of catalog relative to skycatalog_root (see below)
catalog_name	string	“skyCatalog”	Name of top-level yaml config file
config_path	string	None	where to write config. If None, same folder as data
dc2	boolean	False	Use dc2 conventions
galaxy_magnitude_cut	float	29.0	Discard galaxies above cut. Ignored for non-galaxy object types
nside	int	32	nside for healpixels
stride	int	1_000_000	Max objects output per row group
truth	string	None	Default depends on object_type
knots_magnitude_cut	float	27.0	Omit knots component from galaxies with i-mag above cut
log_level	string	“INFO”	Log level
no_knots	boolean	False	Omit knot component
options_file	string	None	Path to file where other options are set
pixels	int list	[9556]	healpix pixels for which catalog will be created
skip_done	boolean	False	do not overwrite existing files
skycatalog_root	string	None	Abs. path. See catalog_dir
sso_sed	string	None	Path to file to SED to be used for all SSOs. Defaults to solar_sed_thin.txt, included in repo.
star_input_fmt	string	“sqlite”	Format of star truth

The script create_flux.py and its options

create_flux.py has most of the same options as create_main.py plus a couple new ones. Only object_type is required but several others are specified for most invocations.

Options table

Here is the complete list of options as they appear in an options file. On the command line, prepend -- and change all underscores to hyphens.

Name	Datatype	Default	Description
object_type	string		Required. One of {star, sso, cosmodc2_galaxy, diffsky_galaxy}
catalog_dir	string	“.”	Location of catalog relative to skycatalog_root (see below)
catalog_name	string	“skyCatalog”	Name of top-level yaml config file
config_path	string	None	where to write config. If None, same folder as data
flux_parallel	int	16	# processes to run in parallel when computing fluxes
iclude_roman_flux	boolean	False	If True calculate & store Roman as well as Rubin fluxes.
log_level	string	“INFO”	Log level
options_file	string	None	Path to file where other options are set
pixels	int list	[9556]	healpix pixels for which catalog will be created
skip_done	boolean	False	do not overwrite existing files
skycatalog_root	string	None	Abs. path. See catalog_dir
sso_sed	string	None	Path to file to SED to be used for all SSOs. Defaults to solar_sed_thin.txt, included in repo.

Example options files

Create cosmodc2-style galaxies main file. This file was one I wrote primarily for testing the creation code. In order to speed things up, I set galaxy_magnitude_cut down to 20.0. The output file is about 1000 times smaller than with the default cut.

object_type:          cosmodc2_galaxy
catalog_dir:          just_testing
pixels:               [9683]
galaxy_magnitude_cut: 20.0    # Default is 29.0
log_level:            DEBUG   # Default is INFO

Suppose the file is called main_galaxy.yaml. It can be invoked as follows:

python cosmodc2_galaxy --options-file main_galaxy.yaml

Note

Since object_type is required it must be specified both on the command line and in the options file.

Create star flux file for a couple healpixels. The default value of flux_parallel is rather conservative for Perlmutter, so use something higher to make the process go faster.

catalog_dir:          just_testing
pixels:               [9683, 9684]
flux_parallel:        24      # Default is 16

Note

The star main files for both healpixels must already exist in the output directory since they are input to the flux generation.