Pipeline 3 processing will employ flats compiled from
long-period runs of nightly flats to improve the statistics over the
pipeline 2 method of a simple average over a floating 5 night window. A
side benefit will be a simplified processing step. For more discussion
on the basis of this proposal, please refer to the
earlier discussion on the topic.
To minimize effects of biases introduced by anomalous nights included in the canonicals we have generated the super grand canonical flats from medians of the nightly canonicals. We have started with the original hardware break periods used for flats/darks, looking for which required further subdivision and which could be combined (if they lacked any true distinction in structure).
The conclusion was that only several flats hardware periods were required. Each band of the north warrants 6 periods (though they are not exactly the same for any two bands) and all south bands fall into 4 periods (the same for all bands).
The final production canonical flats summaries may be found on the following pages. Each page contains images of the median flats, the deviations about the median, and the difference between averages and medians for the flats. Note that differences between the two methods are trivial to levels well below 1% and are mostly driven by a scattering of morning flats with slightly different illumination patterns (mostly at J) weighting the averages.
Comparisons between flats from the original hardware break periods follow. Some periods have been further subdivided because of their long duration and slow drift of the flats. These medians have been compared with direct averages, and they only differ at most by less than half of a percent. More advanced trimmed averages would seem to be overkill. The largest deviations (strongest in J band) are related to the following issues:
Examination of the souther residual images above supports Mike's suggestion that the long term low-level variation of the flats may be related to seasonal shifts in illumination, not detector drift. Examining the third column of difference images (super-grand canonicals minus the super-duper-grand canonicals), there are similar illumination patterns seen in periods 06a/07 and 06b/08 which cover similar date ranges in different years (see table below). Therefore adopting a more uniform canonical flat averaged over a longer period may give more consistent cross-scan biases.
Following this suggestion, it looks like the southern data may be separated into 4 flats periods: 01-02, 03, 04-05, 06-08, with breaks falling at real differences in electronics and window cleanliness (the differences between 01-02 and 03 are not obvious at K band, but it would be less confusing to use the same breaks for all bands).
Canonical flats periods are summarized below:
01 970000 970802 02a 970903 970926* 02b 970927 971026 02c 971027 971224 03 971225 980411 04 980412 980604 05 980605 980918 06 980919 981015 07 981016 981020 08 981021 981117 09a 981118 990129 09b 990130 990427 10 990428 990720 11a 990721 991030 11b 991031 000126 11c 000127 000321** 11d 000323 000901 12 000902 050000 * flats had a gap through 970923-970926, but hardware work on the telecope conclude on 970926 so this break period was extended through this date. ** followed by nights w/o flats; will need to decide which canonical these nights go with. Original periods subdivided above: 02 970803 971224 09 981118 990427 11 990721 000901
01 970000 980502 02a 980503 981030 02b 981031 990117 03 990118 990226 04 990227 990507 05 990508 990624 06a 990625 991010 06b 991010 000218 06c 000219 000417 07 000418 000908 08 000909 050000 Original periods subdivided above: 02 980503 990117 06 990625 000417
Overall, the original "hardware break" periods have been used for the super grand canonical flats. In cases where the deviations from the median appeared much higher than other periods, they were further subdivided by looking at the individual nightly differences by eye and noting when flat structure seems to have changed.
In several northern nights, sharp edges are seen in the deviation maps in the upper right corners. This appears to be caused by individual nights in which there was some edge vignetting (perhaps caused by a slighly closed shutter?). It may be appropriate to mask out extra pixels on these nights if the problem potentially affects the data aquisition on these nights as well.
Examining the flats summaries above, paying special attention to the third column of images (the difference between each canonical period and a single super-duper grand canonical) we can identify the minimum number of necessary hardware breaks mandated by actual changes in electronics/chips/etc.
The situation appears to be relatively straightforward for the south, which breaks into 4 obvious periods at all bands (though there are a few specks at J that appear to change in period 02b, which might suggest one additional subdivision here).
The north, however, is a lot more complex, with different bands showing breaks at different times. The outstanding question is whether to adopt different flat periods for each band or to synchronize the dates in all bands, resulting in a total of 8 flats periods for everything (indicated by A1-8 in the table below). Alternately some of the breaks might be consolidated if the differences are deemed to be sufficiently small to not warrant separate periods.
North
-----
01 970000 970802 J1 H1 K1 A1
-- --
02a 970903 970922 J2 A2
-- -- -- --
02b 970927 971026 J3 K2 A3
-- -- --
02c 971027 971224
03 971225 980411 H2 K3
A4
04 980412 980604 J4
05 980605 980918
-- -- --
06 980919 981015
H3 K4 A5
07 981016 981020
-- -- -- --
08 981021 981117
09a 981118 990129
J5 H4 K5 A6
09b 990130 990427
10 990428 990720
-- -- -- --
11a 990721 991030
11b 991031 000126
H5 A7
11c 000127 000321 J6 K6
11d 000323 000901
-- --
12 000902 050000 H6 A8
South
-----
01 970000 980502
02a 980503 981030 J1 H1 K1
02b 981031 990117
-- -- --
03 990118 990226 J2 H2 K2
-- -- --
04 990227 990507
J3 H3 K3
05 990508 990624
-- -- --
06a 990625 991010
06b 991010 000218
06c 000219 000417 J4 H4 K4
07 000418 000908
08 000909 050000