2MASS "Hot-Pixel" Analysis

1. A list of scans taken in regions of low source density is produced for hot pixel analysis.
2. The "freak" program is executed on the frame extraction files for each scan to produce 256^2 images containing the source frame extraction (FREX) count (number of frames where a source was identified) for sources brighter than 14th mag in each pixel.
3. The FREX images for each scan are thresholded at FREX>=4.
4. The thresholded FREX images for each scan are coadded (summed=stacked) to learn which pixels are the "hottest" (having the most frame extractions overall), and how many pixels on the array are effected.
5. Hot pixel lists are extracted to reduce data volume and simplify visualization.
6. Plots of the hot pixel lists and scan counts are generated.
7. Histograms, cumulative distribution functions (CDFs), and histogram/CDF comparison statistics are generated.
8. Key plots and statistics are presented in an HTML report generated by the software. See for example 970418n and 970423n.

A more detailed outline of the steps in this processing can be found here.

3. Review of Stages in the Analysis

2) The disturbing finding of mostly intermittent events in seemingly random pixels indicated the need for further work. Therefore, the initial analysis (1) was expanded (Jun 1997) to include detailed statistics and distribution plots for 22 science scans from 970418n. The primary result was 3 hot pixels persistent in all 22 science scans (AND mask), and 89 unique hot pixels in at least 1 scan (OR mask). The bottom line was given in an analysis summary for 970418n.

3) The analysis first done for 970418n (2) was repeated and reported (June 1997) for 970423n and 970521n. The bottom line was given in an analysis summary for 970423n and an analysis summary for 970521n.

4) A comparison and over all summary for 970418n, 970423n, and 970521n was made (June 1997).

5) The findings reported through stage (4) above indicated that an improved strategy is to perform hot pixel characterization using calibration scans instead of science scans. This brings two primary advantages: a) calibration scans are processed first every night to evaluate whether science scans should be processed (e.g., photomety checks); b) the calibration scans are only 1 degree long compared to 6 degrees long for the science scans, so the processing runs faster and takes up much much less disk space.

First, a comparison was made between hot pixels for 68 970424n calibration scans and 28 970424n science scans, and also for 35 970622n calibration scans and 28 970622n science scans. The results are given in a summary for calibration versus science scan hot pixels .

In also became necessary to produce a turn-key driver script where only the night has to be specified, rather than a labor intensive process of running a few scripts interactively. To make this possible, first the process of identiying low-density scans had to be automated. This was done by studying the number of sources in the *.bfpts files for 970424n (formerly 970423n) and 970622n, and cross-checking against the number of source extractions in the results of running the 'freak' program. A program was written to generate a list of all available scans for a night, separate science or calibration scans, and threshold on an input lower limit which defines a "low-density" scan. The process uses logic based on the following observations:

• cal scans (1 deg) have (#sources <2000) in their *.bfpts files
• sci scans (6 deg) have (#sources >5000) in their *.bfpts files
• low-density cal scans have (#sources <~ 1400) in their *.bfpts files
• low-density sci scans have (5000<~ #sources <~10000) in their *.bfpts files

4. Recipe for Hot Pixel Characterization

1. Run do_hot_pixels, the main driver script, for the night in question.

Usage:   do_hot_pixels rodan_dir night_hemisphere scan_type
Example: do_hot_pixels /o1/TEST 970418n cal > & do_hot_pixels.970418n.cal.log

To keep track of what has been done, add the command-line used to the top of file do_hot_pixels.driver.csh and run this shell script with the new run(s) uncommented. Example:

         do_hot_pixels /o1/TEST 971205n cal >& do_hot_pixels.971205n.cal.log
        #do_hot_pixels /o2/TEST 971105n cal >& do_hot_pixels.971105n.cal.log
        #do_hot_pixels /o2/TEST 970717n cal >& do_hot_pixels.970717n.cal.log

Notes:

• Timing: It takes 10-20 minutes to run a single night's cal scans.
• Disk space: 1.0 - 3.0 MB; typically 1.5 MB per night for cal scans. This is after deletion of the intermediate files (which is done by the scripts automatically.)

Here is a summary of what do_hot_pixels does:

• Makes list of bfpts files on rodan.
• Edits each line to look like: rsh rodan "wc -l /s*/src/*.bfpts"
• Collates line (source) counts for *.bfpts files (running wc many times).
• Finds low-density cal scans using counts in *.bfpts files.
• Copies frex files and runs 'freak' program to make FITS images (running do_frex).
• Builds an IRAF scipt to threshold all frex images above 4 source extractions and builds an image stack (running threshold_frex_images).
• Runs the IRAS script.
• Generates plots and statistics in an HTML report for the night (running img2lst_plot_stats).
• Removes frex and FITS files.

2. Add the new night(s) processed with do_hot_pixels to the lists of all nightly mask files generated to date for each band (J, H & K). This is done by editting the following files and adding new line(s) for the new night(s) at the top of each file:

         mask_tables.k
         mask_tables.h
         mask_tables.j

For example, each line in the file 'mask_tables.k' looks like this:

         970424n.cal/k.fex.stack.hot.tbl

Be sure to check that all 3 files have the same number of lines via:

       wc -l mask_tables.*

3. Run do_merge_masks. This csh script runs merge_masks on each of the mask lists as follows:

#!/bin/csh -v

rm merge_masks.j.log
rm merge_masks.h.log
rm merge_masks.k.log
merge_masks mask_tables.j J > & merge_masks.j.log
merge_masks mask_tables.h H > & merge_masks.h.log
merge_masks mask_tables.k K > & merge_masks.k.log

ls -l merge_masks.*.log
ls -l Cum*.html
echo '\n Please view Cum*.html using Netscape to see if merge_masks succeeded.'

Currently, it takes only 7-20 seconds for merge_masks to run on each band. The primary output of merge_masks includes 3 HTML reports with plots and links to further details for each night:

• Cum_K_mask.html
• Cum_H_mask.html
• Cum_J_mask.html

NOTE: If these HTML files, when viewed with Netscape, are empty missing plots, or have broken links, there was a problem with the run.

4. Tar newly created files to a file, ftp to spider:/data/Http/Docs/staff/mazz/2mass/hot-pixels, unpack it, and remove the tar file from spider. Note that depending on the number of cal scans, the final report for one night's hot pixel processing takes 1-3 MB, but typically only 1.5 MB.

The following tar command can be used (or a csh script):

     setenv NIGHT = "971105n"
     tar -cvfh /tmp/$NIGHT.spider.tar $NIGHT* Cum* Nights* do* img* *.html mask* merge*

This stage of the automation was performed 6-8, 18-22, 25-26, & 28 August 1997. Debugging and ducumentation so others can run this "pipeline" was produced on 4-5 Sept. 1997, with minor revisions on 3 April 1998.