Super Grand Canonical Flats

Final Production Grand Canonical Flats

Pipeline 3 processing will employ flats compiled from long-period runs of nightly flats to improve the statistics over the pipeline 2 method of a simple average over a floating 5 night window. A side benefit will be a simplified processing step. For more discussion on the basis of this proposal, please refer to the earlier discussion on the topic.

To minimize effects of biases introduced by anomalous nights included in the canonicals we have generated the super grand canonical flats from medians of the nightly canonicals. We have started with the original hardware break periods used for flats/darks, looking for which required further subdivision and which could be combined (if they lacked any true distinction in structure).

The conclusion was that only several flats hardware periods were required. Each band of the north warrants 6 periods (though they are not exactly the same for any two bands) and all south bands fall into 4 periods (the same for all bands).

The final production canonical flats summaries may be found on the following pages. Each page contains images of the median flats, the deviations about the median, and the difference between averages and medians for the flats. Note that differences between the two methods are trivial to levels well below 1% and are mostly driven by a scattering of morning flats with slightly different illumination patterns (mostly at J) weighting the averages.

Analysis for Grand Canonical Break Periods

Comparisons between flats from the original hardware break periods follow. Some periods have been further subdivided because of their long duration and slow drift of the flats. These medians have been compared with direct averages, and they only differ at most by less than half of a percent. More advanced trimmed averages would seem to be overkill. The largest deviations (strongest in J band) are related to the following issues:

Images of the super grand canonical flats and deviations about the mean may be found on the following pages:

Examination of the souther residual images above supports Mike's suggestion that the long term low-level variation of the flats may be related to seasonal shifts in illumination, not detector drift. Examining the third column of difference images (super-grand canonicals minus the super-duper-grand canonicals), there are similar illumination patterns seen in periods 06a/07 and 06b/08 which cover similar date ranges in different years (see table below). Therefore adopting a more uniform canonical flat averaged over a longer period may give more consistent cross-scan biases.

Following this suggestion, it looks like the southern data may be separated into 4 flats periods: 01-02, 03, 04-05, 06-08, with breaks falling at real differences in electronics and window cleanliness (the differences between 01-02 and 03 are not obvious at K band, but it would be less confusing to use the same breaks for all bands).

Canonical flats periods are summarized below:

Overall, the original "hardware break" periods have been used for the super grand canonical flats. In cases where the deviations from the median appeared much higher than other periods, they were further subdivided by looking at the individual nightly differences by eye and noting when flat structure seems to have changed.

In several northern nights, sharp edges are seen in the deviation maps in the upper right corners. This appears to be caused by individual nights in which there was some edge vignetting (perhaps caused by a slighly closed shutter?). It may be appropriate to mask out extra pixels on these nights if the problem potentially affects the data aquisition on these nights as well.

Breaks for Final Grand Canonicals

Examining the flats summaries above, paying special attention to the third column of images (the difference between each canonical period and a single super-duper grand canonical) we can identify the minimum number of necessary hardware breaks mandated by actual changes in electronics/chips/etc.

The situation appears to be relatively straightforward for the south, which breaks into 4 obvious periods at all bands (though there are a few specks at J that appear to change in period 02b, which might suggest one additional subdivision here).

The north, however, is a lot more complex, with different bands showing breaks at different times. The outstanding question is whether to adopt different flat periods for each band or to synchronize the dates in all bands, resulting in a total of 8 flats periods for everything (indicated by A1-8 in the table below). Alternately some of the breaks might be consolidated if the differences are deemed to be sufficiently small to not warrant separate periods.


01  970000 970802  J1  H1  K1  A1
                   --          --
02a 970903 970922  J2          A2
                   --  --  --  --
02b 970927 971026  J3      K2  A3
                   --      --  --
02c 971027 971224          

03  971225 980411      H2  K3
04  980412 980604  J4

05  980605 980918          
                       --  --  --
06  980919 981015
                       H3  K4  A5
07  981016 981020          
                   --  --  --  --
08  981021 981117          

09a 981118 990129
                   J5  H4  K5  A6
09b 990130 990427

10  990428 990720          
                   --  --  --  --
11a 990721 991030          

11b 991031 000126
                       H5      A7
11c 000127 000321  J6      K6

11d 000323 000901          
                       --      --
12  000902 050000      H6      A8


01  970000 980502

02a 980503 981030  J1  H1  K1

02b 981031 990117
                   --  --  --
03  990118 990226  J2  H2  K2
                   --  --  --
04  990227 990507
                   J3  H3  K3
05  990508 990624
                   --  --  --
06a 990625 991010

06b 991010 000218

06c 000219 000417  J4  H4  K4

07  000418 000908

08  000909 050000

Discussion of J-band speckles may be found elsewhere.
Column averages of grand canonical flats are have been computed for the comparison flats.

R. Hurt  Last updated: 2/2/01