Reduction Process
The "home base" of the reduction process is in pawnee:/data. Each night's directory can be found here, in the appropriate YEAR_OTZFQ directory. There are eight steps to the reduction process. The first consists of preliminary set-up tasks. The other seven are often called OTZDFQI: Overscan, Trim, Zero, Dark, Flats, Fringing, and Illumination.
Preliminary Tasks
-->COPY THE UNPROCESSED FILES FOR EACH NIGHT TO A SPECIAL DIRECTORY IN /DATA/RAW BEFORE YOU DO ANYTHING. This is very important.
--->When we take data at WIRO, the images are saved as .fit files. We first need to change them to .fits files and change their datatype to make them usable. However, while the files will now technically be .fits files, they need to remain named *.fit, as this is what our iraf recognizes.
1. Make two lists of files: "all.list" will be a list of file names with the .fit extension. "all2.list" will be an identical list with the .fits extension.
2. Change file types (enter in iraf): cl> rfits @all.list 0 @all2.list datatype=u
3. Delete the .fit files: cl> imdel @all.list
4. Rename the .fits files as .fit files (this doesn't change the files' intrinsic type): cl> imrename @all2.list @all.list
--->Now, we need to add headers to our images, using a fortran code called mkheader. This file can be found in the /data directory.
1. Copy master mkheader executable file from /data to working directory.
2. Run a.out by typing "./mkheader". The program will prompt you for the night's log file name, the output ascii file (enter header.bin), and observers' names.
3. Pipe header.bin into iraf: cl> cl < header.bin
4. Check status of images (enter in iraf): cl> ccdlist @all.list. If you've done everything correctly, next to each filename it should say [ushort] (NOT [short]...if it only says [short], something is wrong), and the type of file ("object" for object frames, "zero" for biases, "dark" for darks, and "flat" for flats).
-->Remove bad images from the processing directory: images with low flux counts, high seeing, etc.
Summary of commands: (cl> refers to IRAF terminal)
>ls *.fit > all.list
>ls *.fit > all2.list
cl> rfits @all.list 0 @all2.list datatype=u
cl> del @all.list
cl> imrename @all2.list @all.list
>cp /data/mkheader .
>./mkheader
'monthdaydayyear'.log
header.bin
observer names
cl> cl < header.bin
cl> ccdlist @all.list (check [ushort])
Overscan, Trim, Zero, Dark, and Flat (OTZDF)
1. Combine bias images using zerocombine. Notable parameters:
input = b*.fit
output = Zero
combine = average (avg has slightly lower noise than median)
reject = minmax
ccdtype = zero
process, delete, clobber = no
2. Determine the overscan and trim sections using implot on any image (look at the help page for implot for appropriate keystrokes). When you look at the plot, you'll notice the left and right sides of the images have "dead" sections, also known as overscan. Zoom into the largest of the two (should be the right side) and note the dimensions of the overscan (should always be the same for any night for WIRO images, I use [2100:2200,1:2048]). Then, determine the usable part of the image (i.e., minus both overscans)--this is called the trim. Again, it should always be the same. I use [55:2100,1:2048]. the 1:2048 comes from the fact that the ccd chip is 2048 pixels tall; the only part that is determined from implot is the dead zones on the left and right hand side of the image (overscan).
3. Process flats, and objects (each separately) using ccdproc. !NO LONGER USING DARKS! Notable parameters:
images = t*.fit or a*.fit
ccdtype = flat or object, respectively as above
fixpix = no
oversca, trim, zerocor = yes
darkcor, flatcor, illumco, etc. = no
biassec = [x1:x2,y1:y2] (see #2)
trimsec = [x1:x2,y1:y2] (see #2)
zero = Zero.fits
Sort and make lists of images:
>ls a*.fit > object9155.list
>ls t*.fit > flat9155.list
4. !NO LONGER USING DARKS! Combine darks using darkcombine. MAKE SURE TO COMBINE SEPARATE EXPOSURE TIMES SEPARATELY. Notable parameters:
input = list of equal-exptime darks
output = Dark30 or Dark300, for example
combine = median
reject = minmax
ccdtype = dark
process, delete, clobber = no
5. Combine flats using flatcombine. MAKE SURE TO COMBINE SEPARATE FILTERS SEPARATELY. Notable parameters:
input = list of same-filter flats
output = Flat8615 or Flat8685, for example
combine = median
reject = avsigclip
ccdtype = flat
subsets = yes
process, delete, clobber = no
scale = mode
6. !NO LONGER USING DARKS! Correct objects for darks using ccdproc. MAKE SURE TO CORRECT SEPARATE EXPTIMES SEPARATELY (i.e. use Dark30 for standards and Dark300 for objects). Notable parameters:
images = list of equal-exptime images
ccdtype = object
darkcor = yes; everything else in that section = no
dark = correct exptime dark (Dark30.fits, for example)
7. Correct objects for flats using ccdproc. MAKE SURE TO CORRECT SEPARATE FILTERS SEPARATELY. Notable parameters:
images = list of same-filter images
ccdtype = object
flatcor = yes; everything else in that section = no
flat = correct filter flat (Flat8615.fits, for example)
Fringing and Illumination (QI)
***MAKE PREFRINGING DIRECTORY***
Remember this section only applies if your images have fringing or needs illumination correction. Also, the fringe pattern will change if you have switched from filter to filter multiple times throughout the night. The filter can be placed in a slightly different positions, so if you come
back to a filter later in the night it may not be in the exact same postion as it was previously. Therefore, it is best to lump
all observations of a particular filter into one portion of the night without switching the filters.
List of filters that need illumination and fringe correction:
7598:
7661:
8132:
8199: fringe and Illumination correction
8615: fringe and Illumination correction
8685: fringe and Illumination correction
9155: fringe and Illumination correction
9233: fringe and Illumination correction
U: Niether
B: Niether
V: Niether
R: Illumination correction
I: fringe and Illumination correction
Summary of commands: (cl> refers to iraf terminal)
>cp object300.list object_good.list
cl> imstat @object_good.list
cl> disp "highest count".fit
cl> imstat @object_good.list low=0 up="saturation count" > count
>sort -n count +2 > object_sort.list
>n object_sort.list (erase all but image name and sorted counts)
>cp object_sort.list imarith
cl> cl < imarith
cl> epar imcom
Input: @fringe.list
output=FringeFILTER_1
combine=average
lthresh=0
hthresh=50-100 above sat of highest count fringe image
nlow=0
nhigh=2
nkeep=1
cl> disp FringeFILTER_1.fits
*****cl> epar ccdp (set all to no)*****
cl> mkfringecor FringeFILTER_1 out=FringeFILTER_1_cor
cl> mkskycor FringeFILTER_1 output=illumcorFILTER_1
cl> imhist FringeFILTER_1_cor z1=-# z2=# (close to 10^3)
cl> epar imcom:
Input: FringeFILTER_1_cor
output=FringeFILTER_1_cor2
combine=average
lthresh=z1
hthresh=z2
nlow=0
nhigh=0
nkeep=1
cl> epar ccdp
Input: @group.list
output= Leave blank
ccdtype =object
fringec=yes ;illuco=yes
all others in that section = no
fringe= FringeFILTER_1_cor2 ;illum=illumcorFILTER_1
cl> prevu @group.list
>cp object_good.list rename.list
>n add a001"_monthdaydayyear".fit to rename.list
cl> imcopy @object_good.list @rename.list
1. Make a "good" list without the calibration images. Imstat the "good" list, display image with highest counts, redo imstat with lower=0 and upper=just below "saturation limit" in image with highest counts. example: highest count = 345.8; upper=345. Sort this list by counts.
---> There are two different ways to proceed from here and it depends on whether the night you are reducing has 2 or less filters or 3 or more filters. When reducing a night with 2 or less filters that have fringing it becomes very easy and can usually get rid of the diffraction effect with only 2 sets of fringes; the other becomes more difficult and tedius. Keep this in mind when observering.
2 OR LESS FILTERS
2.For each filter take 3 random count images, all three of which have to be different fields; also try NOT to use fields with large saturated stars:any stars larger than 1/8th the size of the entire image. These stars have residual effects on the final corrected images. Put these three in one list (fringelist), and the rest of the group in another list (grouplist).
3.Use the iraf command 'imarith' to normalize each images' counts and bring them up to the counts of the highest count image. This alows all of the images to be corrected virtually at once. Make a text file called "imarith" by copying "object_sort.list" to "imarith". The syntax is :
imarith a001.fit * ("highest count"/"a001 count") a001_temp.fit
cl> cl< imarith
4. Use imstat to find the "saturation limit" of the highest-count image in fringelist. Epar into imcombine.
Input: @FRINGELIST
output=FringeFILTER_1
combine=average
reject=minmax
lthresh=0
hthresh=50-125 above "saturation limit" of highest-count image in fringe list, (for low counts, around 200-400, add 50 or 75 or so; for higher counts add 100 or more)
nlow=0
nhigh=2
nkeep=1
Run imcombine.
5. Make sure you go into ccdproc and set all corrections to no.(second section) The next step somehow uses ccdproc and may accidently correct for fringing, dark, flat, etc. This has a noticeable effect on your images. Take a look at fringeFILTER_1. If it looks ok (i.e., with no noticable very bright or very dark spots, and no noticable traces of stars), do this: "mkfringecor FringeFILTER_1 output=FringeFILTER_1_cor." NOTE: If you get an error, it's probably because a parameter in ccdproc is set to "yes." Epar into ccdproc and make sure every parameter in the second section is set to "no." Mkfringecor centers the counts of your fringe image around 0, so the bright fringes are positive and dark fringes are negative (or vice versa).
6. Now the corrected fringe image must be trimmed so only the fringe pattern remains; no remnants of stars or anything else. To do this, use imhist and set z1 and z2 so each cuts the curve off slightly lower than 10^3 counts. If done correctly, the absolute values of z1 and z2 should be very close, i.e. within 2 or 3. Remember these values.
7. To trim our corrected fringe, we can be sneaky and use imcombine on only one image, namely the corrected fringe. So, epar into imcombine.
Input: FringeFILTER_1_cor
output=FringeFILTER_1_cor2
combine=average
lthresh=z1
hthresh=z2
nlow=0
nhigh=0
nkeep=1
Run imcombine.
8. Take a look at the new corrected image. If it looks ok, by the same standards as above (it should, if the above image was ok), you can now correct the group list for fringing. Epar into ccdproc.
Input=@GROUPLIST
Output: leave output blank (i.e., this will overwrite the images in grouplist)
ccdtype=object
fringec=yes
all others in that section=no
fringe=FringeFILTER_1_cor2
Run ccdproc
9. Finally, illumination correction. We'll use mkskycor to create an illumination image, i.e. an image that contains only the underlying illumination differences in all the images. Do this: "mkskycor FringeFILTER_1 output=illumcorFILTER_1" Now epar into ccdproc. Everything remains the same except: illumco=yes and fringec=no, and illum=illumcorFILTER_1 Run ccdproc.
-->There are certain filters that only require illumination correction and can be very difficult to correct for. The broadband filter (R) is almost always difficult to correct for illumination. If no combination of images are properly correcting for illumination you can try to change the "min box size" in the parameter file. I have now permanently changed my paremeters from 5 to 0.25 pixels in both x and y minimum box size at the edges.
10. Use prevu to peruse grouplist. Repeat above steps on any images in which fringing still exists, and on other groups. You'll have to play around with which images, when combined, will result in successful fringing correction on which images.
11. Undo your imarith on the newly corrected images.
12. Make copies of the processed images with a different name to be moved to the photometry directory. The nomenclature will be similar to the names of the log files: a001_jul228292008.fit. Now move all of the images to the corresponding field directory inside the photometry directory.
3 OR MORE FILTERS
2. Divide this sorted list by counts, keeping roughly similar counts together. Keep in mind that now the fringing pattern usually changes over the course of the night so you may need group images according to time of night for each filter as well.
3. follow the steps above.