IRAC Artifact Mitigation Info
Introduction
This page contains the current (02 September 05) version of the post-bcd IRAC
artifact correction codes. These codes are written in csh scripts (for data
handling and module control) and IDL (image processing). To run the
correction scripts, you must have IDL version 5.6 or higher (version 5.4 will
not work due to a problem with the strsplit function) in your path and your
IDL_PATH must include a directory containing the IDL Astronomy User's Library,
available from http://idlastro.gsfc.nasa.gov/.
If you download this code, please send an email to Sean Carey
(carey@ipac.caltech.edu), if you
would like to be notified of code updates and patches. Your feedback on the
code is also appreciated. Please indicate the version of IDL you are running
and AOR request keys of the data if you encounter problems.
What the code corrects
For a complete description of artifacts in IRAC data please consult the
IRAC Data Handbook. The code does not work on subarray data. In
general, subarray data is free of the artifacts this code will correct.
The artifact mitigation code attempts to correct:
- Muxbleed: Residual signal in multiplexers seen in 3.6 and 4.5 micron
data
- Column pulldown/pullup: A bias shift in a column containing a bright
source. Effects are seen in data from all arrays
- Electronic banding: A bias shift in a row containing a bright source.
This effect is seen in the 5.8 and 8.0 micron arrays only.
- First frame effect: The overall bias of each array has a temporal
dependence (the bias depends on the readout history of the array). The bias
variation between overlapping images is corrected in a least-squares fashion.
The SSC provides another method of correcting for bias variations as part of
the post-BCD software package (MOPEX task overlap.pl). In many cases the
overlap.pl script will produce comparable results to the overlap correction
provided by this software.
What the code does not correct
The code does not currently correct:
- Muxstriping: A bias variation (3.6 and 4.5 micron arrays only) seen after
observing very bright sources which manifests itself as a per readout bias
shift which affects only a fraction of the array.
- Optical banding: Some of the light from point sources observed at 5.8 and
8.0 microns is scattered into the columns and rows near the peak of the source.
The optical component of banding is only partially corrected by this code
as the code assumes that a source does not create artifacts in rows and
columns other than the one containing the triggering pixel.
- Stray light: Stray light is masked at a later stage in the processing
using the mask_straylight procedure written by Mark Lacy. The straylight
masked is bundled with the artifact correction routines, but is not used in the
make_corrected_bcds.tcsh script.
- Ghost images: Ghost images are not mitigated by this code. The stray
light masking code will also mask filter ghosts in all 4 arrays. The optical
ghosts are currently included in the PSFs provided by the SSC.
How to get the code
The scripts and procedures are located in a single directory, irac_preprocess.
You will also need some cal files, namely the pmasks and rough versions of the
PRFs. The PRFs are only used to determine if a source is more pointlike than
radhit like. The pmasks have been modified slightly from the current versions
on the SSC webpage. This version includes the time period that the mask
applies to as keywords in the pmask header. I also have a directory of
sample cdfs that I use for different data sets. The cdf directory is not
necessary for running the artifact correction software. Tarballs of all three
directories are here:
- irac_preprocess tarball
: Artifact correction code
- cal tarball : Directory of
calibration files, please consult the SSC website for current PRFs
- cdf tarball : Directory of
namelists for Mopex, these namelists give some examples for processing
IRAC data.
How to run the code
First, download your data to some disk. The scripts assume that the filenames
are in the archive format (e.g. SPITZER_I1_6050048_0000_0000_1_bcd.fits).
For the code to run, the beginning of the filenames must be in capital
letters, check to see that your version of unzip does not translate uppercase
filenames to lowercase filenames.
Copy the irac_preprocess, cal and cdf tarballs to the directory where the
data resides (./r6050048, for example). Untar the 3 tarballs in your working
directory (example is working from a UNIX machine, sscsci100.
sscsci100> gunzip ./irac_preprocess_02sep05.tar.gz
sscsci100> gunzip ./cal_23aug05.tar.gz
sscsci100> gunzip ./cdf_23aug05.tar.gz
sscsci100> tar xvf irac_preprocess_02sep05
sscsci100> tar xvf cal_23aug05
sscsci100> tar xvf cdf_23aug05
Now from the directory with the data,
copy the main driver script to the data directory.
sscsci100> cp ./irac_preprocess/make_corrected_bcds.tcsh .
make_corrected_bcds.tcsh controls the
execution of all the correction modules applied. By default, the script
cleans up a previous run, copies the BCDs into the current
working directory, creates lists of files to process and then operates on the
files performing various corrections. Some sets of corrections create new
instances of the files so that the entire script does not need to be rerun.
You should edit make_corrected_bcds.tcsh to run the modules you want or modify
the path to the correction routines if you have an alternate path. The
default processing of the script is for non-HDR data. If you have HDR data,
then you should uncomment the HDR processing steps. If your data contains
repeats, then other processing steps will need to be enabled. The
./irac_preprocess/get_originals.csh may also
need to be edited if the data are not downloaded using Leopard.
To execute the code:
sscsci100> ./make_corrected_bcds.tcsh
the status of the correction will be printed to the screen and also to a log
file make_corrected_bcds_#.log where # is a unique process ID number.
By design, there is a copy of the code in a subdirectory of the working
directory where your data resides. Keeping local instances of the code for
each of your data processings will help keep track of what has been done to
each data set. If you prefer keeping the code in a single location, you will
have to modify the paths specified in the .idl files as they assume a relative
path of ./irac_preprocess to all of the procedures and functions.
Results
The correction code will create several types of new files.
- sfx_SPITZER_I[1-4]...._bcd.fits : BCD corrected for saturations
(correction is only for HDR data at present)
- sfx_SPITZER_I[1-4]...._bunc.fits : saturation-corrected uncertainty
images
- est_sfx_SPITZER_I[1-4]...._bcd.fits : BCD with estimate of "true"
sky in artifact affected pixels. This is an intermediate product which should
not be used for photometry.
- cor_sfx_SPITZER_I[1-4]...._bcd.fits : artifact corrected BCD
- cor_sfx_SPITZER_I[1-4]...._bunc.fits : updated uncertainty image,
contains estimate of uncertainty in correction
- SPITZER_I[1-4]...._bimsk.fits : updated mask file in
imask format, also contains the contents of the pmask.
- over_cor_sfx_SPITZER_I[1-4]...._bcd.fits : overlap and artifact
corrected BCD. These files along with the SPITZER_I[1-4]_...._bimsk.fits and
cor_sfx_SPITZER_I[1-4]...._bunc.fits files are the final products of the code.
You should inspect the cor_sfx_SPITZER[1-4]*_bcd.fits images to ensure
that the artifact corrector performed as expected. The artifact corrector
will fail if there are too many pixels flagged to perform proper sky
estimation. It will also fail to recognize when the artifacts are saturated
and cancelled out in the pedestal reads.
To facilitate browsing of the results, the irac_preprocess/extras directory
contains a script to create byte images with a side-by-side comparison of the
cor_sfx_SPITZER_I[1-4]_...._bcd.fits and sfx_SPITZER_I[1-4]_...._bcd.fits. To
make the byte images:
sscsci100> idl ./irac_preprocess/extras/make_quick_compare.idl >& make_quick_compare.log &
The byte images will take a while to make. The byte images will have
filenames like cor_sfx_SPITZER_I[1-4]_...._bcd.byte. Once the code has
completed, you can use the idl procedure ./irac_preprocess/quick_cor_browse to
examine the byte images. From your working directory:
sscsci100> idl
IDL> .run ./irac_preprocess/extras/quick_cor_browse
IDL> spawn, 'ls cor_sfx_SPITZER_I1_*_bcd.byte', files1
IDL> quick_cor_browse, files1, 'keep_corrected_ch1.log'
where files1 is a list of all the channel 1 side-by-side byte images and
'keep_corrected_ch1.log' is a log file that will contain your choice of
corrected or not for each BCD in the files1 list. The procedure will bring up
a plot window with the corrected BCD on the left and the uncorrected on the
right. Smaller versions of each image are displayed using a logarithmic
stretch at the bottom of the plot window. To write the filename of the
corrected BCD to the log file, left click. For the uncorrected BCD,
right click. The procedure will automatically advance to the next byte image
in the list. quick_cor_browse displays the images much faster than using an
image browser such as ds9, atv or skyview.
How to know if the code works
If the code has run properly, you will have a set of cor_sfx, est_sfx and
sfx_ images corresponding to your input BCDs in the directory that you execute
make_corrected_bcds.tcsh in. For non-HDR observations, the
first frame will not be processed for frame times greater than 2 seconds and
the second frame will not be processed for frame times greater than 30 seconds.
The initial frames in non-HDR mode for frame times greater than 2 seconds are
taken in HDR mode and have a different frame time than the rest of the data.
If you are missing any of the output files, then one or more of the processing
steps has failed. Inspect the *.log files for IDL error messages. You will
see a message "% Program caused arithmetic error: Floating illegal operand" in
most of the log files. This message is due to the handling of NaNs in the
images and can be ignored.
Algorithm Overview
A general description of the correction algorithm is listed below. For more
detailed information on a module, click on the link (if it exists).
The correction scheme consists of:
- Adding estimate of sky subtracted by skydark back to data,
- Creation of new mask files which have placeholders for the various types
of IRAC artifacts,
- Identifying saturated pixels and estimating their true surface
brightness,
- Flagging artifact-affected pixels,
- Estimating the sky in artifact-affected pixels,
- Determining a model for the muxbleed, optional if you already have a model
that will work,
- Correcting for artifacts by fitting (in a least-squares sense) models of
each artifact to the difference of the pixel and true sky estimate.
Corrections are performed separately for
- Column pulldown/pullup: Currently a separate DC offset is applied to
the affected column above and below the triggering source
- Electronic banding: A separate DC offset if applied to the affected
row left and right of the triggering source.
- Muxbleed: Either a lookup table or an exponential fit is applied
simultaneously to overlapping instances of muxbleed
Things to do / planned code improvements
- Estimate the surface brightness of pixels that are super-saturated in
non-HDR data.
- Upper limit threshold for muxbleed in channel 1 -- actually for all
artifacts in all channels. At a certain surface brightness (much greater than
saturation), the electronic artifacts disappear from the data.
- Lower threshold for banding in channels 3+4. The threshold for banding
may be too conservative
- Set thresholds in an external table. Then observers can tune parameters
more specifically to their own data sets.
- Better estimation of sky when there are a lot of masked pixels -- may
have to try bootstrap correction list at a time. The sky estimation produces
poor results when a lot of pixels in a single region are masked.
- Need to correct the brightest cases of muxbleed. From the data that I have
processed, it appears that the muxbleed has a different behavior as a function
of triggering surface brightness.
- Develop library of muxmodels
- Need bandwidth correction for super-saturated pixels. The electronic
bandwidth correction is not correctly applied for triggering pixels that
saturate in the pedestal read.
Sample results
Good results
Here are some examples of the artifact corrector working well.
Under construction -- Examples will go here.
Bad results
Here are some examples of the artifact corrector performing poorly.
Under construction -- Examples will go here.
More detailed module descriptions
IMASK creation
Under construction
Sky dark sky addition
Under construction
HDR bias matching
Under construction
Saturated source estimation
Currently works for HDR mode only. Section under construction
Artifact flagging
Section under construction
Sky estimation
Section under construction
Artifact correction
Section under construction
Overlap correction
overlap.pl in the MOPEX software provides equivalent results in most cases.
Section under construction
Version notes
30 Aug 2005 version -- first release
01 Sep 2005 version -- fixed bug in muxbleed flagging in pixel 65535
02 Sep 2005 version -- edited comments in make_corrected_bcds.tcsh
13 Oct 2005 version -- removed repeated call to make_artifact_masks.idl
Last update 13 October 2005 by Sean Carey