DC2 Tutorials

This directory contains tutorial and demonstration notebooks convering how to access and use the DC2 datasets. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.

Notes for Tutorial Users

(Notes for tutorial contributors are at the bottom of the page.)

• If you are unsure on exactly how to start to use these tutorials, you may want to look at this step-by-step guide (requires Confluence login).

• If you want to obtain an overview of the different types of DC2 data products and their access methods, take a look at DC2 Data Product Overview (requires Confluence login).

• Many of the tutorials below use the Generic Catalog Reader (GCR) to access the catalogs. To learn more about GCR, visit LSSTDESC/gcr-catalogs and see the information therein.

• You are very encouraged to use these tutorials as a starting point for your own projects! And you can contribute your analysis notebooks following the instruction here. It is expected that some of your analysis notebooks would be based on these tutorials and hence have duplicated code snippets. You don’t need to worry about that for now; we can always identify commonly used pieces and refactor them as standalone tools at a later time.

Notebook	Short description	Links	Owner
Object catalog GCR Tutorial Part I: GCR access	Use the GCR for simple access to the object catalogs	ipynb, rendered	Francois Lanusse, Javier Sanchez
Object catalog GCR Tutorial Part II: Lensing Cuts	Use the GCR to access the object catalog and build a lensing sample similar to the HSC Y1 shape catalog	ipynb, rendered	Francois Lanusse, Javier Sanchez
Object catalog GCR Tutorial Part III: Guided Challenges	Use the GCR to access the object catalog and solve some typical data analysis problems	ipynb, rendered	Francois Lanusse, Javier Sanchez
Object catalog GCR Tutorial Part IV: Photo-z information	Use the GCR to access the Photo-z information that are provided as an “add-on” to the object catalog	ipynb, rendered	Yao-Yuan Mao, Sam Schmidt
Object catalog with Spark	Introduction of using Spark to access the object catalogs	ipynb, rendered	Julien Peloton
Object catalog with pandas, color-color stellar locus	Directly access the Run 1.1p object catalogs using pandas and explore the stellar locus	ipynb, rendered	Michael Wood-Vasey
DM DeepCoadd Catalog with Butler Access: lensing cuts	Use the Butler to build a lensing sample similar to the HSC Y1 shape catalog	ipynb, rendered	Jim Chiang
DM Postage Stamps with Butler	Make some small cutout images and visualize them	ipynb, rendered	Michael Wood-Vasey
Plotting skymap in DM catalogs using Butler	Use the data butler to obtain information on the skyMap used in the coadd analyses performed by the DRP pipeline.	ipynb, rendered	Jim Chiang
Matching catalogs using the LSST Stack matching code	Spatial matching of objects using the DM Stack	ipynb, rendered	Jim Chiang
Matching catalogs using FoF algorithm	Using the Friends-of-Friends algorithm to match the extragalactic, truth, and object catalogs	ipynb, rendered	Yao-Yuan Mao
Truth Catalog with GCR	Example of accessing DC2 truth catalog with GCR	ipynb, rendered	Scott Daniel
Truth Catalog with GCR: variables and transients	Example of accessing variables and transient objects in the truth catalog with GCR	ipynb, rendered	Yao-Yuan Mao, Scott Daniel
Extragalactic catalog with GCR: redshift distributions	Extract, plot and explore the differential number counts of galaxies	ipynb, rendered	Eve Kovacs
Extragalactic catalog with GCR: Halo Occupation Distribution	Compute and plot the HOD for the extragalactic catalog	ipynb, rendered	Yao-Yuan Mao
Extragalactic catalog with GCR: mass relations	Compute and plot the relations between halo mass and other quantities in the extragalactic catalog	ipynb, rendered	Yao-Yuan Mao
Extragalactic catalog with GCR: cluster colors	Access the extragalactic catalog with the GCR, and explore colors in galaxy clusters	ipynb, rendered	Dan Korytov
Extragalactic catalog with GCR: cluster members	Extract cluster member galaxies from the extragalactic catalog and plot them on the sky	ipynb, rendered	Dan Korytov
DC2 Run 2.1i DR1 Object Catalog	Preliminary Validation of DR1 Object Catalog	ipynb, rendered	Michael Wood-Vasey
Difference Image Analysis - Stamps and Lightcurves	Exploring a test Run 1.2p DIA run on (tract, patch) = (4849, ‘6,6’)	ipynb, rendered	Michael Wood-Vasey
Difference Image Analysis - Supernova Example	Comparing supernova lightcurves to variable+transient truth catalog for test Run 1.2p DIA run on (tract, patch) = (4849, ‘6,6’)	ipynb, rendered	Michael Wood-Vasey
Using PostgreSQL Introduction - Object table	Use object table as first example of how to access PostgreSQL database. Also includes mini SQL primer	ipynb, rendered	Joanne Bogart
PostgreSQL - Object table part 2	More advanced queries of object table	ipynb, rendered	Joanne Bogart
PostgreSQL - Forced source	Use PostgreSQL fourcedsource view to access forced source, plot light curves	ipynb, rendered	Joanne Bogart
PostgreSQL - Truth data	Use PostgreSQL to access star truth summary and variability tables	ipynb, rendered	Joanne Bogart

---

Notes for Tutorial Contributors

Both tutorial and demo notebooks are hugely useful resources - pull requests are most welcome! A detailed instruction can be found here.

• Before you commit a notebook, please make sure that a) it runs to completion and b) the outputs are cleared (to avoid both repo bloat and conflicts on every run due to semantically equivalent but bitwise-distinct output blobs).

• Please update the index table above, carefully copying and adapting the URLs, and making sure that all the lines in the table are correctly aligned (or else the table will not display). Pro-tip: use the “preview changes” tab in the online GitHub editor to check that the table is correct before committing. Adding your notebook to the table will trigger the automatic testing of it once your PR is merged (see the “Semi-continuous Integration” section below).

• The “owner” of a notebook (that’s you, as its contributor!) is responsible for accepting proposed modifications to it (by collaboration), and making sure that it does not go stale (by fixing issues posted about it).

• Every tutorial notebook needs an owner/last verified header, a statement of its goals (learning objectives) in the first markdown cell, and enough explanatory markdown (with links to docs, papers etc) to make the notebook make sense.

• Before August 2018, these tutorials were developed in the DC2-production repo.You can follow this link to see issues and PRs that were related to these tutorials before they being moved here.

Semi-continuous Integration

All the notebooks listed in the table above (and on the master branch) can be run on Cori using the beavis-ci script, which then pushes them to an orphan “rendered” branch so that the outputs can be viewed. Our ideal is that this will be run automatically daily, but that is not currently active.

If the link to a rendered notebook yields a 404 error, please check the corresponding log file (by clicking on the “build:failing” badge) and issue the notebook’s owner. If it looks like something has gone wrong overall (like, none of the notebook builds are passing, or the logs indicate some problem with the run environment, issue @drphilmarshall.