Getting Started

Basic Notation

Throughout this document, [night] will be used to designate the standard designation of night following the format yymmdd[n/s]. Sidebars are used to indicate shortcuts I have made, often in the form of aliases in my /home/hurt/.cshrc file; feel free to examine and use any useful code from it. My csh scripts all are kept in /home/hurt/bin.

Scripts that invoke editors assume that in your .cshrc file there is a line of the form:

Making a New QAreview Document

The QA review template document is found in:

Finding the Data

The location (on barney) can be found by reading the bottom of Ron's .report e-mail or using the command:

Starting Up

After rlogining into the appropriate computer, cd to the appropriate night. One useful shortcut is to go to the datequal directory and use the command:

qastart

This will summarize calmon overrides, dark & flat status, and open up all the commonly-viewed plot files.

Web Pages

2MASS Data Processing Status and History

This is the master document linking to all of the processed nights.

The index page for a processed night can be found at:

• [night].lgo Text file with the night's log file
• q[night].html 24-hour Monitor file
• night.html Nightly summary page

These may all be found in Davy's public 2MASS directory:

Usually the web pages are updated once as a standard step in processing. The web pages are stored on the machines:

Check the Report

Before preparing the recommendation it is good to check out the processing to see if there are any obvious problems that might be fixed by rerunning the processing with minor tweaks.

The first hint that problems exist are in the processing report file. This will indicate whether any scans have failed processing and why.

Check Diagnostic Plots

On the q[night]_night.html page one should check the cal-instrument mag vs. time plot. Here validly-calibrated survey blocks will be obvious between color-coded calibration blocks. Ideally survey blocks should extend through the entire evening, but this is sometimes not the case. Some common problems (and solutions) include:

Not Enough Calibrators

On a short night or when observations have been interrupted and resumed during the evening, it is possible to drop below the required # of calibrators (6 for old strategy, 5 for new). One may work around this by overriding the default by setting the calmon.nl parameter minnfinblk[1] to a smaller number.

Poor Calibrator Photometry

If the dispersion of the calibration stars exceeds the primary threshold value for one or more scans one may choose to set more lenient thresholds if the rest of the calibrators look good. To do this set maxdspnscn[1] to a larger value and remember to downgrade the quality of the scans in the photometry section of the QA review.

Too Much Time Between Calibrators

Sometimes extra time elapses between cal blocks due to camera timeouts or other operator decisions. If the data look photometric (if the dispersion in the calibrators is low, if there appear to be no clouds in the backgrounds and jump counters, if the overlap scatter looks good) then rerunning with a larger maxsep[1] may be warranted.
 

Processing Notes

CALMON Namelist

The easiest way to find differences between the standard parameter namelist and the one used in the run is to diff the file used with the standard template. From the root night level directory type (for northern and southern hemispheres, respectively):

This information comes directly from Ron's processing report file.

Scans Failing ACT Reconstruction

Failing scans are automatically listed here. If there are failed scans one should open the summary file [night].QApm and record the # of Tycho stars in each scan (nTycho column) and the percent of the scan spanned by the Tycho stars (PctSpn column). If the % spanned value is larger than ~40% then it is worth checking further why the ACT reconstruction failed.

The ACT status also can be found on the Astrometric Summary web page; look for red flags in the first column.

Check of Darks & Flats

General success of flats are indicated in the web Night page, summarized at the top. Lore has it that this info is sometimes in error and should be checked against the file darks/fort.44 which summarizes dark and flat (called responsivity) processing. Look for whether darks and flats were taken in the morning or evening, and if any passed and were used.

In questionable circumstances, it is also good to check the actual dark images, comparing them with previous and subsequent nights for consistency. The archive of the dark history from files run through 24 hr QA can be found at:

Observer Report

This should be digested from the observer log found in:

These should be noted whenever seen during the QA evaluation. Note the scan number and coadd tile of the artifact (i.e. meteor streak).

In the case of bright stars (often identified by looking at background statistics) one should display the affected tile and make a coordinate overlay. A simple script to do this is:

One may also wish to note other things of interest in the scans (e.g. galaxies, nebulae), that will not in fact require clean-up, in a separate section here. A summary of all catalog galaxies that should lie within a given scan (and an indication of whether or not they were processed by galworks) can be produced from the night root directory by using:

Photometricity

Calibration Strategy

Specify "old" (sets of 2 calibrators separated by 2 hour interval) or "new" (sets of 1 calibrator separated by 1 hour interval).

CALMON Status

This is summarized on the night page. One should look both at whether the summary indicates minimum photometricity requirements have been met as well as the links to figures.

The cal minus insrument mag vs. time gives a quick indicator of which cal sets failed photometric specs (#s on the plot are airmass, calibrator source #, and sigma). Red symbols indicate failure for that calibrator scan. The sigma vs. time shows how the night progresses. Often if only one cal spills over the sigma limit in one or two bands then it makes sense to request a calmon/quality rerun for the night after resetting the namelist default maxdspnscn (max dispersion in calibration scan) to larger than 0.04.

*** Need to insert instructions on how to identify failure error codes and what they mean.

Photometric quality factor is affected by small number of cal scans in block, poor photometry for field stars (***where is this info?***), and poor standard star photometry.

Here one should explain any overrides to the calmon namelist and justify why the quality was or was not degraded by this.

Repeatability in Overlaps

A useful check of photometric stability is to see how close in magnitude stars appear in overlapping scan regions. These plots (which must be copied by hand into Davy's QA web page area later) may be found in:

Photometric quality factor is affected by repeatability in overlaps. One looks at the peak-to-peak scatter in overlap repeat differences when evaluating the quality factor on Davy's scoring rules. Quality degradations due to scatter in the overlaps should apply to the entire block of survey scans. Summaries of the overlaps values found in these plots can be found on the phtometricity QA web page; this is useful for entering exact ranges when downgrading scans.

Scan Summary

The survey science scan ranges should be entered here (read out of any of the night tables) noting which quality factors apply to which ranges of scans as determined by the previous checks. Survey blocks which are not bracketted by calibrators are automatically given a quality factor of 0.0 as they are by definition non-photometric.

Sensitivity

Sensitivity Plot

The sensitivity plot is automatically generated by the pipeline processing.

If for some reason the plot needs to be generated by hand use the following procedure:

The older way of doing this task (with colors not corresponding directly to the quality grades) was to invoke the script:

The survey scans are ranked individually by sensitivity according to Davy's scoring rules and the fiducial rules on the plot. The horizontal rules represent quality factor thresholds of, from bottom to top, 100%, 75%, 50%, 25%, and 0%. Each survey block should be broken into sub-blocks of corresponding quality grades.

Backgrounds/OH Airglow/Jump Counters

Background Statistics

This is also a useful time to examine how the backgrounds have changed over the evening to help substantiate the comments of the observer, particularly in quantifying the effect of clouds. One hint of encroaching clouds is an increase in all backgrounds, while airglow variations show up primarily at H. To look at the background levels for a night cd to the datequal directory and run:

A summary of known indicators of clouds and airglow may be found at:

The jump counter results should be summarized in this table. Every entry should be examined and explained as completely as possible (typically stars, clouds, and airglow are the culprits, but sometimes one may find airplanes or ladybugs!).

Historically there had been a problem with RA slippage in the north, which could also trigger the jump counters. A useful follow-up diagnostic is to look at the pointing offsets using the following commands in the datequal/ directory:

Untracked Seeing

This information is found in the scan-by-scan page in the Background/seeing/focus summary in the seeing tracker score column. List any survey scans that fail (are flagged red) in any band.

2^nd Image Moment Ratios

This may be found in the av. of 2^nd moment ratio column of the same table.

Quality Recommendation

Sci Scans

The overall ratings for sci scans are computed by multiplying the photometricity and sensitivity quality factors by the default score (usually 10) and rounding to the nearest integer. Be sure to note maximum allowed scores from untracked seeing and 2^nd image moment ratios.

These numbers are automatically entered for sensitivity; any photometric quality factors need to be updated by hand.

Note that if any scans in a science block are affected by clouds then the entire block is set to quality 0.

Cal Scans

Cal scan ratings are judged to be either 0 or 10 depending on whether they are photometric. Note that any cal scan that is bounded by non-photometric survey blocks (this may be due to clouds) should be set to quality 0.
 

Send Report

The QA recommendation form should be e-mailed to the 2MASS team for review and approval. The standard subject line should be:

Once the recommendation has been approved by UMASS then the QA transaction files should be updated so the night can be removed from disk. The local database transaction file may be found in:

The template telescope transaction file may be found in

A useful shortcut for updating the transaction files is:

Useful Utilities & Procedures

COADDS

To view an entire scan's set of images, cd to the directory sNNN/image (where NNN represents the scan number) and run the script:

This will also mark the extended source candidates with blue circles (to remove them enter "er bo" for erase blue overlays). This leaves you in skyview so you can use its functionality to further examine the images.

Point Source Overlays

It is often handy to look at the point sources from the pipeline overlaid on a scan image. This may be done using the following script while within a single scan's image directory:

To look at the background levels for a night cd to the datequal directory and run:

Examine Coadds & Coadd Backgrounds

To examine a series of coadds and allow subtraction of GALWORKS background models and examiation of backgrounds (useful for finding J banding artifacts) you can use Tom Jarrett's script:

/home/jarrett/proj/2mass/dev/src/galworks/3chan/cflash_single.csh

Calling it without parameters gives the syntax. Its useful features include subtracting background models and plotting median right/left coadd background and fourier transforms thereof.

Note that one must have the line "setenv PGPLOT_DIR /usr/local/pgplot" in ones .cshrc for the plotting to function.

Examine Seeing History of a Scan

To see the seeing as a function of position within a scan (useful for tracking down problems induced by brief periods of bad seeing) use Tom Jarrett's script:

Reprocessing a night

Often a night with some processing failures can be salvaged by rerunning the pipeline software with slightly different parameters. The most common parameters to reset are:

• minnfinblk1 [6] Minimum # of calibrators to make a block
• minnfdsets [2/1] Min # of calibrators in a block (doesn't work?)
• maxsep1 [2.2] Max separation in hrs between calibrators
• maxdspfd1 [0.06] Max dispersion of ???
• minncs1 [4] Min # of calibrators in a set
• maxdspscn1 [0.04] Max dispersion in a cal scan

Checking Archival Processing