======================================================================= MIPS Downlink Processing Meeting Minutes #1 (Wed. Sep. 3, 2003): Attendees: M. Pesenson, V. Mannings, D. Frayer, D. Padgett, S. Wachter, M. Moshir, D. Makovoz, D. Henderson, M. Aronsson, S. Hemple, T. Hesselroth, F. Masci Action Items and Concerns ------------------------- 1). People requested that the meeting time be changed due to conflict with the weekly IPAC science seminars. Stay tuned for new time. 2). F. Masci/D. Padgett reported a 43-47 arcsec discrepancy between predicted and reconstructed pointing in the only photometry-mode AOR of campaign A. Furthermore, a few cases showed discrepancies of ~7 deg. D. Padgett and F. Masci will look into this. M. Moshir suggested looking directly at the boresight pointing history data for anomalies. 3). F. Masci reported that later versions of pointing history files for the same time-range contained (more) anomalous data contrary to expectations. He will look into this and submit an ISA if needed. 4). Post-BCD processing of MIPS photometry AOR (6413824) encountered a failure in the image interpolation step. D. Makovoz said this was due to bad pointing in BCD headers as alluded to above. He will re-test this on the dev. side and put in the necessary ARs for S8.2 if needed. 5). D. Henderson reported that only the pipe0+pointing can be run on the ops system right now due to re-design of modules. MIPS-Ge is currently undergoing a complete overhaul of the pipeline picking/ensemble creation infrastructure. Furthermore, he said the dark pipeline will be out of action until S8.2 (to be deployed on ops ~Sep. 20). 6). D. Frayer reported that we have lost a 4x8 pixel block in the 70um array due to a malfunction in read-out electronics. 7). D. Padgett reported that MIPS campain-D1 (scheduled ~Sep. 22) will be used to test the on-board SUR-mode slope fitting algorithm by comparisons with interleved RAW-mode DCEs. She said that a flaw was found in the SUR-mode simulator to be used for this analysis. F. Masci will look into fixing the module. 8). D. Padgett also brought to our attention that the ~45" predict - actual pointing discrepancy could be due to an earlier update to the SODB fov/frame-table used by uplink. She will look into this. 9). F. Masci complained about the arduous task of finding the correct pointing history file used in pointing reconstruction for a given DCE. He will put in an AR to ensure the correct path/filename is included in the processing log. 10). F. Masci also reported a logic flaw in the pointing thread when writing scan-mirror position keywords to FITS product headers. Only the bcd header was found to be updated and not the accompanying raw-tranheaded file. He has put in a fix to be deployed for S8.2 (Sep. 20). ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #2 (Tue. Sep. 9, 2003): Attendees: M. Pesenson, V. Mannings, D. Frayer, D. Padgett, S. Wachter, M. Moshir, D. Makovoz, M. Aronsson, S. Hemple, T. Hesselroth, F. Masci, J. Rho, D. Shupe Updates on Actions from last Meeting ------------------------------------ 1. Large (>7 deg) discrepancies between predicted and reconstructed pointing seem to have dissapeared with later pointing history file versions. 2. The persistent 43-47" predict - recon'd pointing discrepancy seen in a MIPS/campaign-A photometry AOR was not seen in IRAC data (reported by M. Moshir). This rules out this discrepancy being due to different ephemerides for which the predicts were made and what the S/C actually saw. D. Padgett will run T. Goldina's ephemeris updater to expolore this further. 3. F. Masci updated the SUR-mode simulator module used to derive slope images from RAW-ramp data. The activity of comparing RAW-ramp data with slope (SUR-mode) data will occur in campaign-D1. F. Masci verified that later versions of pointing history files are indeed "better" in terms of constancy/expected-range of values. 4. Some people reported the difficulty of associating the correct pointing history file with a given DCE. F. Masci reported that this can be done by finding the start time of a DCE (SCLK_OBS keyword value) and searching for this in the file: "/sos/logs/S8.0.2/ptnserver.9984.log". Note that both the release (S*) and id (9984) will change with time. F. Masci also reported that there will be a fix for including this filename in the output processing logs. Campaign-B Updates ------------------ 5. D. Padgett/D. Frayer reported that the two IERs of campaign-B were successfully downlinked and processed through the pipe-zero and pointing threads with a quick turn-around. F. Masci reported that no pointing what so ever could be associated with the dark-IER (request 6604544) since first, predicts are not available for IERs (per design) and actuals (from pointing reconstruction) cannot be written to headers since the scan-mirror range for darks is not covered in the pixel-scale/angle specification file for pointing transfer. 6. M. Moshir reported that Uplink could be including predicted pointing in the database for IERs in week 9 of IOC. 7. D. Padgett reported that we have at least two new bad pixels in the mips24 array. She also showd some plots of image data run through the science pipeline. Since these were "saturated dark" images, they showed a sharp boundary where pixel replacement of saturated pixels with difference values was performed. F. Masci will explore whether the results are as expected. 8. The issue of what to do with non-primary (flanking field) observations with MIPS was brought up. i.e. when mips-70 is the primary field, mips-24 is looking at another piece of sky which could also be scientifically useful. Currently, all data is being processed and archived as normal. It wasn't clear however whether the mirror-emulator will correctly associate the correct pointing to non-primary DCEs for them to be immediately useful. 9. D. Frayer reported that the 4x8 pixel block in the 70um array previously thought to have been lost has come back alive. 10. D. Padgett ran into a problem running the offline pipeline under Linux with Redhat 9.0 installed. T. Hesselroth said he will look into it if he can, otherwise we will have a limitation of using only Redhat 8.0 or earlier. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #3 (Tue. Sep. 16, 2003): Attendees: M. Pesenson, D. Frayer, D. Padgett, S. Wachter, M. Moshir, D. Makovoz, M. Aronsson, F. Masci 1. D. Padgett reported that analysis of 24um campaign-B data by the IT revealed a decrease in responsivity of ~1% (over the array on average) over a time span of ~3 hrs from the beginning of an anneal or instrument turn-on. She also showed a plot where the median slope dropped by a few % in the first 30 minutes from an anneal and stabilized there-after. She suggested that science observations should be commenced after such a period where stability is achieved. 2. Campaign-C data (acquired 9/15 at ~18:00hrs PDT) revealed that we are not cooling fast enough as expected and this is consistent with HK telemetry. Campaign-C consisted of darks and scattered background measurements most of which are still saturated although saturation is not as severe as it was in campaign-B (acquired 9/8 at ~22:00hrs PDT). Ramp-data images show that first-to-second sample differences are starting to drop below the nominal saturation threshold. The 4-second exposures indicate that ~30% of the 24um array is unsaturated. 3. D. Frayer reported that there is at least one pixel in the 70um array with higher than normal responsivity which hits saturation sooner than all the others. He also reported that a 5-pixel block in the 160um array shows a lower than normal readout than other surrounding readout blocks. Overall, the characterization from campaigns B & C is the same as expected from the ATLO tests. 4. F. Masci showed some mips24 pipeline results from campaign-C processing. The main concern is that the main bcd image shows a discontinuity where saturated pixels were replaced with corresponding values from the difference plane. He speculated that either: the linearization is not being computed correctly in the difference plane, that the (quadratic) non-linearity model is not appropriate for the data at hand, or that the non-linearity calibration derived from ground tests is not applicable in cases of extreme saturation. D. Padgett was going to explore whether the IT pipeline showed similar behavior and whether the models need to be revised. F. Masci will also embark on doing some sanity checks. 5. F. Masci also brought up the need for a simple quick-look analysis tool to display a variety of histogram cuts and diagnostic plots for a FITS image. He will look into whether such a tool can be acquired from the IRAC team or whether we should write our own specific to the MIPS data format. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #4 (Tue. Sep. 30, 2003): Attendees: M. Pesenson, D. Frayer, D. Padgett, M. Moshir, M. Aronsson, F. Masci, S Hemple, T. Hesselroth, J. Rho, D. Henderson, D. Hines, G. Rieke, J. Muzerolle 1. Campaigns D1, D2 and E were successfully executed. D1 contained mips24 first list, while campaigns D2 and E primarily consisted of mips24 flatfield/dark calibrations and focus confirmation activities. S8.2 processing on ops revealed a bug in the mips24 science pipelines which had to do with a delivery error. This problem has been fixed and a large portion of AORs will be re-processed using the S8.3 s/w release on Oct. 1. 2. The secondary mirror was hovering at 29K on 27 Sept. and the cooling rate is as expected. D. Frayer reported that MIPSGe was behaving nominally but that they were still quite warm to carry out new characterisations. 3. D. Pagdett showed some mips24 first light images which appeared to show some distinct holes or dark regions of order 5-10 pixels across. She (and others) believed this was due to dust particles on the pick-off mirror. It was briefly discussed how these should be handled in processing. D. Padgett suggested these should be flat-fielded out, however, G. Rieke claimed that this may affect the background estimation and that they should be masked out at the end of processing instead. M. Moshir also mentioned that these "dust-spots" may leave dust streaks in scan-mode and to properly calibrated them out, we will need to aquire calibration data at all observable mirror positions and scan-rates. 4. F. Masci reported three anomalies in the pointing reconstruction: i. There was a persistent ~20 arcsec discrepancy between predicted and actual (reconstructed) pointing following frame table update version _6a_ for campaings D1 through E. D. Padgett reported that FTU version _07f_ may rectify the problem (we hope!). ii. Since pointing reconstruction depends on the position of the scan mirror, the mirror angle-range within which the distortion has been calibrated was found not to be big enough. D. Frayer suggested that the observable range be made larger in the distortion/ pixel scale file by replicating all parameters corresponding to the extremeties of the mirror positions. iii. The most serious one that the IT discovered was that first, pixel 1,1 does not correspond to the first read-out pixel (actually currently it's pixel 128,1) and second, the WCS keywords lead to a declination which decreases as one goes in a direction of increasing Y in BCDs. F. Masci was going to put in a temporary fix to the first problem by incorporating a flipping module in all pipelines. This will be tested by the ISTs and then MIPL will be notified on how the pixel ordering and FITS packaging should be done. The second problem deals with configuration settings in D/L software and is a simple one. 5. F. Masci and J. Rho showed some flats made with the IT (DAT) and SSC pipeline for mips24 and both were remarkably similar. Debbie also reported that the DAT and SSC science pipelines are slowly converging to the same answers. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #5 (Tue. Oct. 7, 2003): Attendees: D. Frayer, D. Padgett, M. Moshir, F. Masci, S Hemple, T. Hesselroth, D. Henderson, S. Wachter, D. Makovoz 1. Campaign F (executed on 10/6/03) revealed some interesting results. The 70um array showed some bizarre pattern noise correlated with readout channel. It's not sure whether some of this could be due to transients or non-optimal bias settings. The IT is looking into whether this could be fixed via bias settings before Campaign-G (MIPS-Ge first light - this Thursday). A 4 x 8 pixel block in the 70um array also appears to be in a dormant state. 2. Auomated (SSC) processing of campaign-F data revealed two types of failures: first, database deadlocks were occuring sparodically causing pipeline-SODB interactions to time-out and eventually faili in making products; second, an anomaly was seen in MIPS-Ge processing in that the "pfl" exposuretype (renamed from pht by the cal-AOR plscriptid converter) was not recogized by MIPS-Ge specific s/w. All campaign-F data will be re-processed using the new S8.4 deplyment once the SODB deadlock problem is resolved. 3. D. Padgett reported that the on-board SUR-mode slope estimator was giving slightly wrong results as to what was expected using the downlink SUR-mode simulator s/w. The IT is going to re-install (if they havn't already) a patched software version on board. 4. There still appears to be a ~20 arcsec discrepancy between requested and actual (reconstructed) pointing. D. Padgett reported this could be due to the fact that the scan-mirror angle as computed by downlink's mirror-emulator s/w is not giving the same value as uplink's. The discrepancy could be due to the definition of the zero-point from which angles are computed. Furthermore, it's not clear whether we are using the correct sky-to-mirror axis gain factor in pointing reconstruction. She will look into these anomalies and issue a CR for S8.5. 5. D. Pagdett also mentioned that the IT did some analysis of flatfield data taken at different mirror positions and indeed there is a dependence. In particular, the presence of dust particles on the pick-off mirror creates dark blotches on the array whose position vary with scan-mirror angle. One way to disguise these is to flat-field them out. F. Masci will look into making sure the downlink infrastructure is in place to handle the application of flats as a function of mirror angle and scan rate. Such functionality will be available in the S8.9 patched delivery. 6. F. Masci brought up the existence of downlink software which can report estimates of the Zody, Cosmic Infrared and ISA plate background and write them to the FITS headers of BCDs. The IST reported that this will be an extremely useful thing to have. This feature will be available in the S10.0 delivery. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #6 (Tue. Oct. 14, 2003): Attendees: D. Frayer, D. Padgett, M. Moshir, F. Masci, S. Hemple, D. Henderson, S. Wachter, R. Laher, D. Shupe, M. Aronsson, J. Rho, M. Pesenson, F. Masci 1. Campaign-H processing revealed three major anomalies: first, there were missing telemetry (science) packets which manifested themselves as missing keywords in DCE headers causing pipelines to abort downstream. Overall, 4 out of 9000 DCEs from campaign-H had corrupted headers. Second, MIPS-Ge post-BCD pipelines were failing due to problems in the interpolation step. An AR was submitted to debug this. Third, some post-BCD pipelines were aborting for RAW-readoutmode IERs. This was due to missing calibration entries for Point Response Function (PRF) files in the fallback calibration files. 2. MIPS-Ge is still showing ugly pattern noise over the array (before it was half the array, now it's most of the array) which could be correlated both temporally and spatially. Both D. Henderson and R. Laher were asked to see if any special filtering could be done in the Fourier domain to characterize the noise behavior/pattern and hence ameliorate the problem. 3. D. Padgett reported that requested pointings generated by Uplink are of order ~20" discrepant from 2MASS reference stars. On the other hand, actual pointings from pointing reconstruction are of order ~4" from known reference stars. D. Padgett also reported that the requested pointing discrepancies were worse for observations near the ecliptic pole where abberation effects are most severe. She thinks there was a bug in the S8.0 Uplink release to do with the aberration correction when updates were simultaneously made to the scan-mirror emulator. She claims that the correct version will not be deployed until after IOC/SV. 4. D. Shupe was given an action to generate new Point Response Functions (PRFs) with the appropriate sampling for all MIPS channels. 5. R. Laher requested that prior to the next re-processing run of campaign-H on ops, that all the pmid*.timestats logs be re-initialized to obtain a clean set of timing benchmarks. 6. Mosaics made from photometry mode AORs with the SSC post-BCD pipeline are showing slightly elongated sources in the scan-direction. It is suspected that this is due to improper pointing reconstruction in the input BCDs. F. Masci will look into this and see which parameters if any could be responsible. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #7 (Tue. Oct. 21, 2003): Attendees: D. Frayer, D. Padgett, M. Moshir, D. Henderson, S. Wachter, D. Shupe, J. Fowler, J. Rho, M. Pesenson, F. Masci, T. Hesselroth, D. Makovoz 1. Campaign-I processing revealed problems when attempting to detect radhits from long MIPS-Ge ramps due to memory limitations. There will be a new module in S9.0 which will partition the data into shorter ramps to avoid the problem. MIPS-70 is still showing pattern noise in its "good" half. Problems to do with scale/distortion parameters being incorrectly assigned to SED-mode BCDs were traced to incorrect calibration files. These were recently updated and should give expected results after data has been re-processed. 2. Frame Table Update version 10_8b will be received soon (possibly Wednesday) which will be used to re-synch the ops SODB:fov table with missing SED aperture names and also include new apertures for MIPS-Ge scan mode. 3. D. Shupe took an action to ensure existing PSFs made from Tiny Tim are close enough to those derived from flight data and notify ISTs if any changes need to be made. 4. D. Frayer was going to ensure the values in mirrorparameters.tbl for pointing transfer were going to be updated using the latest FPS/IPF results. J. Morrison/J. Keene are the points of contact. 5. D. Padgett is going to look into whether the existing zero points which define the "optical-zero" position of the scan-mirror in downlink cdfs are correct and notify D. Frayer if changes are necessary. 6. To enable efficient ensemble generation for making mirror-dependent calibration flats as well as picking the correct flat to apply according to the mirror position of a specific DCE, a new parameter needs to be added to the SODB:dces table. D. Pagdett has taken an action to discuss the matter with Uplink to see if they can populate this new field. What is needed is the effective mirror position (in DAC) for each DCE ("csm_pred") which is computable from existing parameters in the exposures table and the specific DCENUM. 7. D. Frayer took an action (from D. Makovoz) to ensure the BCD-masks in MIPS-Ge post-BCD processing have the correct bits set to indicate whether a BCD corresponds to a stimflash DCE and that these are excluded in processing. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #8 (Tue. Oct. 28, 2003): Attendees: D. Frayer, D. Padgett, M. Moshir, S. Wachter, J. Fowler, J. Rho, M. Pesenson, F. Masci, S. Hemple, D. Makovoz 1. D. Frayer reported that reception of an AOR consisting of ~5000 DCEs (~7 GBytes) from MIPL took a very long time (from campaign K). The data trickled in at a rate ~12 DCEs/min for an equal admixture of 24um, 70um and 160um DCEs. Furthermore, based on the time it took to process this AOR (~50 min) he deduced that the ops system is processing at a rate ~ 4 times the acquisition rate. This was not explicitly mentioned at the meeting (only ball-park time estimates were given) but was later verified privately. 2. F. Masci reported two types of failures for MIPS-Ge post-BCD processing, first, the mosaicker thread is timing out (intentionally) after 1hr (usually for 70um) and second, outlier rejection is failing when it was specifically told not to perform this operation from namelist settings. D. Makovoz advised D. Frayer that the first problem can be fixed by setting the pixel scale in the mopex namelist for 70um to the smallest possible value (~4.99 arcsec/pixel - i.e. as for fine-scale) until a fix is put in for S9.0 to make the pixel-scale picking more robust. The second problem was decided to be resolved by actually turning on outlier-rejection and seeing where we stand. This will be tested on the DEV side before making the necessary changes on ops if needed. F. Masci will provide D. Makovoz with the specific test cases. 3. D. Padgett reported that some 160um mosaics made on the ops system (from "slow" scan rates) showed gaps between successive BCDs. This was not seen in the IT's enhancer software. Bad pointing reconstruction was ruled out since the IT's enhancer used the same set of BCDs w/ same pointing. The relevant request is IER-7674624 and will be ingested on the DEV side for D. Makovoz to explore. Furthermore, a request was made to see what would happen if all IERs are pushed through the post-BCD processing on the ops system. This will be tested on the DEV side. 4. The timeline for handling scan-mirror dependent calibrations for mips24 was hashed out. Downlink will go ahead and make the necessary upgrades for S9.0 to handle calibration-picking at the "fallback" level. However, since uplink won't be populating two new mirror-specific fields in the flight SODB until the December time frame, M. Moshir suggested that Ron Beck could manually populate these fields for selected requests using the engineering database. F. Masci will talk to Ron Beck to see if this is feasable. FLS processing will be affected if no action is taken to apply scan-mirror dependent calibrations to science data. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #9 (Tue. Nov. 4, 2003): Attendees: D. Frayer, D. Padgett, M. Moshir, S. Wachter, J. Fowler, M. Pesenson, F. Masci, S. Hemple, D. Makovoz, D. Shupe, T. Hesselroth, D. Henderson 1. Campaign-O processing revealed some memory issues on the ops drone machines. The problem was believed to be related to processing of image cubes containing long (30 sec) ramps, equivalent to 60 sample planes for mips24, 240 samples for MIPS-Ge. The problem can be localized primarily to mips-Ge processing where some modules are particularly memory savvy. D. Henderson is going to put in a fix for S9.0 to split the ramps into 80-sample sub-ramps so they can be processed through the system. In the meantime, D. Frayer suggested that the long-ramp data can be run through pipe0 until such a fix is deployed. 2. There was also a database lock-up problem in which M. Moshir reported (actually reported by T. Handley at the ops status) that the number of "threads" or simultaneous processes/transactions in the flight SODB was exceeding the maximum (allowed) number of 50. F. Masci was going to look into the maximum number of threads that can be initiated by downlink processing at any given time and see if processing was the root cause. 3. D. Padgett mentioned that there was a gigantic solar flare towards the end of campaign-O which temporarily fried all MIPS arrays. 4. The problem where gaps were being seen in 160um mosaics was resolved by D. Makovoz. It turned out that outlier rejection was doing it's job as intended but with incorrect namelist parameters set for this step. D. Frayer is going to deliver new post-bcd namelists to ops before processing of campaign-Q (tomorrow). 5. At the request of ISTs, F. Masci is going to put in an AI to have the ensemble creation rules for mips-24 updated. There will be two major updates. First, the ensemble creation of SUR-mode flats and darks will be extended to include 3 sec exposures, and second, all calibration products should be made when mips24 was indeed the primary FOV. Prior to this, flats were being made even when the scan-mirror was in the dark position and this will pose problems when automated cal-picking is turned on in science pipelines in the long term. 6. F. Masci implemented a new design for handling pointing reconstruction for MIPS. He presented a design which allows one to specify one of two methods to compute the scan-mirror position. The first is based on a model which assumes a quadratic dependence of mirror-encoder (DAC) position on it's projected sky position "The Intrument Team model" (or IT model) and second, a model which uses a constant value for this conversion for the observation mode of interest: "The Focal Plane Survey model" (or FPS model). F. Masci is going to run some offline tests to see which of the models for each mode is most robust and correct. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #10 (Tue. Nov. 11, 2003): Attendees: D. Frayer, D. Padgett, S. Wachter, J. Fowler, M. Pesenson, F. Masci, S. Hemple, D. Makovoz, D. Shupe, D. Henderson, J. Rho 1. Processing of campaign-Q revealed that requests containing of order 1000 DCEs were failing in MIPS-Ge ensemble processing on the ops system. Furthermore, the mosaicker with this many inputs was timing-out (for all MIPS channels) due to the processing time exceeding an intentional time-out limit of 1 hour. F. Masci will look into increasing the time-out limit to 2 hours. D. Henderson is going to look into why the MIPS-Ge piplines failed in the BCD processing. 2. There was a request (from D. Padgett) to mask out, or avoid inclusion of MIPS-70um pixels from the noisy half of the array in mosaic generation. This also applies to the dead readouts at 160um. D. Padgett is going to talk to the Instrument Team to see what the best solution is. i.e. should such pixels be entirely excluded from mosaicking, or, should they simply be flagged in accompanying masks to say they are noisy and still include them when making mosaics? 3. D. Makovoz has updated some outlier-detection modules to make the radhit detection/rejection more robust for mips24. This is planned for S9.0. Currently, the mosaics are riddled with radhits, although at a much lower level than MIPS-Ge. 4. D. Padgett has reported that due to an on-board software bug, the slope estimate for the first DCE of every mips-24 exposure is incorrect and that these DCEs should be excluded when mosaicking. This will eventually be fixed on-board and in the meantime, F. Masci is going to put in a request to exclude all "first"-DCEs when creating ensembles for post-BCD processing. 5. F. Masci brought up the issue of smearing (in-scan elongations) seen in sources in "fast" scan-map mosaics. The smearing is not as noticeable in the slow and medium scans. Smearing at the DCE level appears to be minimal if at all present. This leaves us with something going on during pointing reconstruction. F. Masci performed relative pointing refinement on the BCDs prior to mosaicking and the smearing dissapeared (as expected). However, the smearing is of order 12-15 arcsec and applying "artificial" pointing refinement is not a robust solution without first removing what we know is a systematic effect going on upstream in pointing reconstruction. F. Masci is going to explore this further. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #11 (Tue. Nov. 18, 2003): Attendees: D. Frayer, D. Padgett, S. Wachter, J. Fowler, M. Pesenson, F. Masci, S. Hemple, D. Shupe, D. Henderson, M. Moshir 1. Processing updates: the 160um pipelines showed some failures in campaigns V and X1 to do with flatfield generation. D. Henderson attributed this to the presence of NaNs in the input data causing the median combine step to fail. A fix has been put in for S9.0. There were also sporadic MIPS-Ge failures when attempting to process "non-primary" 70um or 160um IERs. The non-prime observations neglect the fact that stims need to be taken for the stim-interpolation step in pipelines. Such modes are not taken very often and a fix was decided to be unecessary. 2. 24um also had some failures when processing RAW-mode photometry mode IERs of unexpected exposure time. This mode was scheduled with an exposure time not recogized by D/L software. The failure occurs when matching the correct RAW-mode calbration dark for the exposure-time of interest. This mode will primarily be used for "Total Power Mode" observations and the details on the types of exposures possible still has to be hashed out by the IST. 3. There was a discussion on tuning the "radhit" module for MIPS-Ge. Currently, there appear to be many false radhit detections believed to be due the absence of any electronic non-linearity correction upstream. D. Frayer will look into tuning the module in the presence of all the necessary corrections upstream. 4. There was a discussion on increasing the "PIPETIMEOUT" environment variable to 3hrs on the ops system to accomodate intensive MIPS-Ge ensemble processing jobs. This also has implications for the MIPS-24um post-BCD processing. There were requests in campaign R which consisted of slow scan maps of order 3000 DCEs and post-BCD pipelines seemed to exceed the existing "pipetimeout" limit of 2 hours. With proper tuning of the post-BCD pipelines, it may be possible to accomodate such processing within the proposed 3 hour time limit. Offline testing will go on to find an optimal setting, even if this means introducing pipeline-dependent timeouts on the ops system. 5. F. Masci found the cause for the source smearing in scan mosaics (which is worst for "fast" scans"). It turns out the time at which the scan-mirror starts moving is not synchronized with the "universal" SCLK time counter. The BPHF is tied to SCLK while the scan mirror is doing its own thing in a different timing system. F. Masci always assumed that the mirror started moving at the "SCLK_OBS" of a DCE. It turns out the SCLK-to-mirror time offset between consequetive DCEs in a scan is not constant but depends on the DCENUM along the scan. Computing these offsets along a single scan leg and synchronizing these to the boresight-history, the smears dissapeared. D. Frayer was given an action to revise the MIPS-SCLK formulas and how these relate to the scan-mirrow motion. A patch is planned for S9.0 when a robust solution is found. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #12 (Tue. Nov. 25, 2003): Attendees: D. Frayer, D. Padgett, S. Wachter, J. Fowler, M. Pesenson, F. Masci, S. Hemple, D. Shupe, D. Henderson, M. Moshir, D. Makovoz, S. Stolovy, T. Hesselroth 1. D. Frayer had reported failures in the 160um BCD pipeline and D. Henderson believed this was due to bad input data to the qatool module. D. Henderson is going to attempt to reproduce this on the dev system and work with R. Laher to resolve the issue. A problem to do with MIPS-24 post-BCD pipelines timing out after a pre-specified limit of 2 hours was alleviated when D. Makovoz suggested changes to some namelist parameters in the "source-estimation" step. He will forward the new namelist settings to D. Frayer for deployment on ops. 2. D. Padgett had asked the IT how to handle MIPS-70 BCDs when making mosaics - i.e. whether the good side or both (good and bad) should be used. She said that the default should always be to use the good side. This will be done by reading a bit flag from the input bmasks (bcd-masks). The user/IST will have the option of including both sides in mosaicking by setting a namelist parameter. D. Frayer is going to ensure that only the good side (side-A) is currently used for mosaic generation on the ops system. 3. M. Moshir brought to our attention some problems with mosaics from the ops system: first, there appears to be some cross-talk between stimflash DCEs at 70um and DCEs at 24um taken at the same time. People were asked to keep a look-out for the effect; second, not many sources are noticeable in the 70um mosaics as one would expect from the individual BCDs. This will require further examination and some of this could be removed when S9.0 goes online with an upgraded pointing reconstruction scheme to remove image smears/mis-registrations in scan maps. 4. The issue of how to generate Fiducial Image Frames (FIFs) for use by the mosaicker using a more robust/efficient method was brought up. Currently, FIFs are made using predicted pointing from the database. However, it turns out that the predicts are not always reliable due to incorrect computation of position angles for scan maps and other reasons to do with scan-mirror alignment parameters in the slew/emulator model. In the end, this causes mosaics to be "cut off" unexpectedly. A suggestion was made to use the actual (reconstructed) pointing instead and default to predicts if these are not available. Everyone had agreed that the predicts are likely to get better and the problem can be circumvented by making the "edge padding" in the FIF bigger - as is currently being done. 5. F. Masci had done some analysis on the reconstructed mips24 pointing by comparison with 2MASS references. He found that the median 2MASS-to-24um extracted-source separation was about 3 arcsec radial and distributed within 0.5 to 4.5 arcsec. This is a preliminary result as no work on eliminating random 2MASS-to-24um matches was done. This was done using the "pointingrefine" module where a plot of the 2MASS-to-24um source separation before and after refinement can be found in /ssc/pipe/docs/working_notes/mips24/mips24.ps ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #13 (Tue. Dec. 2, 2003): Attendees: D. Frayer, D. Padgett, S. Wachter, J. Fowler, M. Pesenson, F. Masci, S. Hemple, D. Shupe, D. Henderson, M. Moshir, D. Makovoz, S. Stolovy, T. Hesselroth, J. Rho 1. Processing updates: D. Frayer reported that the data transfer rate of MIPS campaign-W from MIPL was extremely slow. Also, there were sporadic glitches in the pointing assigned to BCDs using the S8.9 (online) software where the WCS was way off. This happens in about one out of every few hundred cases and M. Moshir attributes this to a semaphore locking problem in the alogrithm used by the pointing server. S9.0 s/w contains a fix to this. There are still problems in processing large (few thousand DCE) 160um requests. D. Henderson claimed this cannot be reproduced on the dev system and will ask I and T to process the same test case. He is confident that S9.0 does not have this problem. 2. F. Masci resolved the "inter scan-leg" smearing problem in mosaics whereby forward and reverse scan legs were being mis-registered with respect to each other. It was found that this was arising from an out of synch "mirror-SCLK" time and boresight SCLK. There is an offset of ~0.93 seconds between these times which when accounted for, properly registers adjacent scan legs and minimizes 24um-to-2MASS point source discrepancies in the in-scan direction (or makes them consistent with cross scan separations). This change can be made via a namelist parameter and no s/w changes are required. It is not understood how this time offset relates to known mirror/instrumental timing parameters. D. Frayer and D. Padgett will talk to an instrument team member about this. 3. Following the recently revised schedule for S9.0, S9.5 (new) and S10.0 deployments to operations, a list of items was put forward to IST members and prioritized. A list of some high priority items and their desired deliveries is as follows: - mipsGe high pass/noise filtering (S9.0 patch if possible) - mipsGe mirror-dependent illumination corrections (S9.5) - mips24 mirror-dependent flat-fielding (S9.0 as planned) - new mips24 rowfluxcorr module (S9.5) - new pointing infrastructure (S9.0 as planned) - mips24 saturation dependent readout/droop correlation removal (S10.0) 4. In the iterim, downlink will be happy to process FLS/SWIRE validations offline on the dev system using the latest (S9.0) software. M Moshir has put in a request for an additional five drones and more disk space on the dev system, bringing the total number of drones to 12. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #14 (Tue. Dec. 9, 2003): Attendees: S. Wachter, J. Fowler, M. Pesenson, F. Masci, D. Shupe, D. Henderson, M. Moshir, D. Makovoz, S. Stolovy, T. Hesselroth, M. Aronsson 1. Re-processing of MIPS galactic/X-galactic FLS data using S9.0 was briefly discussed. F. Masci and D. Henderson agreed to process at most 2 AOR requests per day (at night actually to avoid clashes). Most of time is spent prepping the DEV database and ingesting the data. The first galactic FLS request was processed this morning and there is a huge difference with the S8.9 on-line version. F. Masci also requested help from the ISTs by running the offline/portable version of the mips24 pipeline. 2. A new tool called "fitsplot" was disccussed (written by M. Aronsson). This tool is used to plot various QA diagnostics for a single input FITS image. Histogram plots and column/row average cuts are made. Suggestions were also given on extending its versatility. 3. D. Henderson mentioned a number of updates to the MIPS-Ge pipelines for the S9.5 delivery: the use of mirror/time-dependent calibration files and the ability to reject DCEs in downstream processing due to bad stim-latent corrections. He will delay this development until the FLS re-processing is complete. 4. The presence of systematic "high flux" pixel patterns in mips24 mosaics was discussed. D. Makovoz found that this arises from uncertainty values being exactly zero in the input BCDs which affect the interpolation step in post-BCD processing. F. Masci discovered that these zero uncertainty pixels arise in BCD processing when creating the flux-calibrated image via the dntoflux module. The dntoflux module calls the uprop library to compute uncertainties. It turns out that the zero uncertainty pixels occur when the input uncertainty image pixels are NaNs. F. Masci has put in a CR to get this fixed as a S9.0 patch. 5. S. Stolovy discussed the need for additional identifiers/keywords in the mosaic headers to aid the user. She will come up with a list of desired keywords and forward it to D. Makovoz. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #15 (Tue. Dec. 30, 2003): Attendees: S. Stolovy, D. Frayer, D. Padgett, F. Masci, D. Henderson 1. Processing using the newly installed S9.0 D/L system appears to be going smooth. Some glitches in final-product generation in campaign MIPS002600 (in the final formatting of FITS headers) were reported by R. Beck. F. Masci will investigate the problem. 2. MIPS s/w development for S9.5 was discussed. For MIPS-Ge there are requests for fixing the stim-latent interpolation (the spline interpolation has a bug), optimizing the fusion of multi-plane data to create non-negative slope images and most important of all, inclusion of median/high-pass filtering to improve image quality at 70um. For MIPS-24, the plan is to include final product generation in the creation of calibration products on-line and also an algorithm to remove a persistent "row-flux" dependent gradient. 3. Creation of scan-mirror position/rate dependent flats to mitigate spots in images from dust on the pick-off mirror is currently on hold. Large amounts of data are required to make the flats and D. Padgett is in contact with the IT on how to automate the process using their scripts. Other dependencies of the shape/position of spots were noticed by the IT, such as the time from when the instrument is first turned on. This is currently under investigation. Also, the issue with AIRE incorrectly populating mirror-position fields in the SODB:Dces table for non-prime observations needs to be sorted out before mirror-dependent flat picking can be fully automated. 4. A request was made to re-format mosaic product FITS headers using the FPG so that more observation/scheduling-specific information can be retained. F. Masci discussed the matter with D. Makovoz and put in a CR for S10.0 (online ~end of Feb.), however, the ISTs thought this was too late. S. Stolovy will take an action to discuss the urgency with D. Makovoz. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #16 (Tue. Jan. 6, 2004): Attendees: S. Stolovy, D. Frayer, D. Padgett, F. Masci, D. Henderson, D. Makovoz, M. Moshir, J. Fowler, M. Pesenson, D. Shupe, T. Hesselroth, S. Hemple 1. S9.0 online processing is progressing fine except for some minor glitches. D. Frayer reported that one mips24 DCE (out of ~1000) failed with exit status code 202. F. Masci tracked this down to missing telemetry packets in MIPL headers causing pipelines which depend on these to fail. There were also some failures in point source extraction for mips-Ge due to configuration file mixups. D. Frayer will look into this. 2. F. Masci reported that some mip24 mosaics from the S9.0 online system are showing systematic blotchy patterns. This may be due to bad flatfielding. F. Masci will investigate this further. 3. F. Masci also reported that point source positional uncertainties (derived from PSF profile fits) appear to be under-estimated by about an order of magnitude based on reduced chi-square values from both the single frame and coadd-based source extractors. This could be due to under-estimated pixel flux uncertainties from bcd processing. F. Masci will probe the problem further. 4. The issue was brought up on how to optimize point source extaction for mipsGe. Due to incomplete knowledge of uncertainties in bcds coupled with the loss of information due to median/high-pass filtering, it was decided to simplify the process and perform source extraction directly off the mosaic instead of simultaneously on all input bcds based on mosaic detections (currently the default). D. Frayer will put in a CR to get something in for S10.0. 5. D. Frayer was asked to revise ensemble creation rules for mipsGe to ensure post-BCD products were not being made for non-primary/flanking field observations where the array is not looking at the sky. 6. The ISTs requested a simple offline script to sort DCEs/BCDs according to scan-mirror position/rate and make corresponding file lists. This will speed up creation of mirror-dependent flats offline. F. Masci offered to put something together. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #17 (Tue. Jan. 13, 2004): Attendees: S. Stolovy, F. Masci, D. Henderson, D. Makovoz, M. Moshir, J. Fowler, M. Pesenson, D. Shupe, S. Hemple, J. Rho, S. Wachter 1. D. Henderson reported that he has implemented median/high-pass filtering for 70 micron BCDs. This is the primary BCD that will be used for mosaicking since it brings out more small-scale structure and sources. Also, there is an ancillary BCD produced which is not filtered and written to the archive. The new s/w will be in effect for S9.5. 2. The accuracy of using scan-mirror position/rate dependent flats to reduce mips24 science BCDs was discussed. S. Stolovy presented some analysis and showed that indeed, mirror-dependent flat-fielding is definitely the way to go to mitigate dark blotches on the array from grit on the pick-off mirror. J. Rho took an action to convert a set medium-scan flats made by D. Padgett to the format required by SSC pipelines. 3. F. Masci presented some results on analysis of uncertainties from mips24 pipelines. He looked at chi-square distributions in profile fits from the SSC point source extractor and showed that BCD-pixel uncertainties appear to be over-estimated by about 14%. He traced the dominant source of uncertainty as coming from an incorrect computation of the initial noise (Poisson+readnoise components) in DCE processing in that it does not account for "non-photon" electronic readout artifacts such as "droop". This makes negligible difference overall although a fix will be put in for robust uncertainty estimation for S10. 4. J. Rho reported an occurance of imperfect distortion corrections by s/w which uses SSC distortion coefficients from FITS headers. D. Shupe mentioned that the current distortion coefficients give pixel position accuracies of ~0.06 and 0.2 of a pixel at the center and edges of a mips24 frame respectively. This correction accuracy may not be enough to avoid non-linear frame registration effects during mosaicking. He took an action to ask J. Morrison if more distortion characterizations have been done. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #18 (Tue. Jan. 27, 2004): Attendees: S. Stolovy, F. Masci, M. Moshir, J. Fowler, M. Pesenson, D. Shupe, S. Hemple, J. Rho, H. McCallon, V. Mannings, D. Padgett, T. Hesselroth 1. S9.1 processing on ops appeared to have some glitches to with the database which affected processing of a large chunk of MIPS002700 campaign data last weekend. The glitches may have to do with an intensive cave run last Friday and failure to "update statistics" thereafter (R. Beck private communication). Things are now back on track with the record length 13-day MIPS campaign. 2. Actions were given to F. Masci to notify D. Makovoz of two issues. First, D. Padgett reported that signal-to-noise ratio thresholds for source extraction do not appear to trace the locally varying background and noise. i.e. a nominal input SNR appears to operate on a scale larger than that in which the noise is seen to vary wildy. It is advised that D. Padgett speaks to D. Makovoz about this issue due to possible mis-interpretation by the writer of these minutes. Second, S. Stolovy suggested that we should have a flag in the offline version of mopex which ignores BCDs with DCENUM = 0 for mips24 since these are affected by incorrect pre-processing on-board in the derivation SUR-mode slopes. 3. Still on the subject of "DCENUM=0" DCEs, S. Stolovy requested that separate darks be made specifically for these DCEs and used on corresponding science DCEs to alleviate "jail-baring" effects. She said that the data is available and that an IST member will take an action to make them. For an entirely different issue, perhaps to do with the on-board slope-fitting algorithm, it appears that the first 2 or 3 DCEs of each exposure look ratty with strong jail-baring. There was also a discussion on whether uplink should modify exposure times in AOTs to accomodate the loss of data due anomalous first DCEs. No consensus was reached, however, downlink is prepared to make the DCEs usable if an algorithm can be provided. 4. The issue of ratty sporadic BCDs appearing in mosaics with strong jail-baring in the presence of bright/saturating sources was brought up by ISTs. This is believed to be due to a read-droop cross-talk effect and S. Stolovy reported that an algorithm will be provided once the effect is properly validated. V. Mannings reported that the effect is rare and hardly noticeable to declare AORs as failures. 5. F. Masci reported the existence of a systematic ~1.3 arcsec offset in mosaics relative to the 2MASS sky. This was discovered in some SWIRE and XLFS data and is probably present in a lot of other data as well. He will work on finding the root cause of this. There are many places that could bring about such an effect. People suggested bad frame-table values, an unknown star-tracker to boresight offset or badly calibrated scan-mirror zero points and axis orientations. 6. The need to properly calibrate mips24 distortion coefficients was discussed. H. McCallon and F. Masci will attempt to do this from all the science data available from the last three campaigns with comparisons to 2MASS data. Proper distortion calibration is necessary to perform robust source matching, registration and pointing refinement, which up until now, have been limited by a "hard" systematic limit of 0.5 arcsec at worst. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #19 (Tue. Feb. 10, 2004): Attendees: S. Stolovy, F. Masci, M. Moshir, J. Fowler, S. Hemple, D. Padgett, T. Hesselroth, R. Laher, D. Henderson, D. Frayer 1. To support re-processing of campaign MIPS002700 Ge data, D. Frayer reported that he will prepare two types of namelists. One which creates "median-filtered" primary products (with a factor of ~2 higher SNR) and another which doesn't. Observers are supposed to get back to him by this Wednesday night to request their preference. The S9.5 pipeline versions will unconditionally generate/archive both products. 2. The issue of mipsGe processing not selecting correct calibration products according to "time since last anneal" was discussed. First, there appeared to be a problem on the DEV system in that bogus anneal times were returned. This was traced by R. Laher as due to a campaign-ID referencing problem since database campaign-IDs are internal counters and expected to differ between databases. Second, it was noticed that the time since last anneal was computed using predicts (from the requestbodystart time as populated by cave) and SCLK_OBS (actual times reported in FITS headers). M. Moshir mentioned that the predicts could be wrong by large amounts relative to when the observation was executed. In fact, D. Frayer reported that the requirement for commensurability between predicts and actual times should be <~10 minutes. R. Laher suggested that the stored function which returns the time since last anneal can be modified to compute it in a self-consistent/relative manner, i.e. use predicted times for both the anneal IER and the DCE observation time of interest. Also, this change is contingent on uplink not re-caving the database for selected requests in existing/scheduled campaigns since it will mess up the relative timing between AORs and dependent IERs needed for this purpose. R. Laher will put in a CR to get the change in for S10.0. 3. F. Masci mentioned the presence of stray-light patterns in flatfield and BCD products and said that depending on their relative strengths, they can manifest themselves in the final BCDs (like seen in IRAC). The effect however is negligible and the IST said we shouldn't worry about it now. 4. F. Masci described a new tool to make mips24 scan-mirror position/rate dependent flatfields. This essentially does everything the IST will need to do manually offline and provides flexibility in making these on the fly for offline self-calibration purposes if needed. 5. M. Moshir mentioned the possibility of introducing errors/biases in flatfield products when made from BCDs with different background levels (e.g. varying zody in long scan maps). He suggested that each BCD/ input should have its illumination normalized out so as to make it statistically indistinguishable from the rest in the ensemble before averaging. F. Masci will explore whether such effects are important and if so, put in a CR for S11.0. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #20 (Tue. Feb. 24, 2004): Attendees: D. Frayer, D. Padgett, S. Stolovy, S. Wachter, M. Moshir, M. Pesenson, S. Stolovy, F. Masci, D. Henderson, H. McCallon 1. S9.5 processing updates: reprocessing of MIPS0002700 and processing of MIPS0002800 has revealed some failures for 24um BCD science pipelines. This is due to missing telemetry groups in DCE FITS headers causing necessary keywords derived therefrom to be absent. M. Moshir reported that MIPL/FOS is supposed to send new versions with complete telemetry packets. The fact that pipeline re-processing revealed the same anomalies indicates that either no new FOS versions were sent to SSC, or, the new versions did not contain the previously missing ancillary data. F. Masci will attempt to trace the history of DCEs with missing telemetry groups within the current campaign and notify Ingolf Heinrichsen if the problem persists. 2. The ISTs raised a concern with the existing help documentation provided along with archived SSC products to the user. The problem for MIPS is that there are both "primary" and "non-primary" (ancillary flanking-field) observations. A photometry AOR will contain an equal admixture of BCDs for all bands. Users are finding it confusing in making associations between what they scheduled as "primary" and what products they receive from the SSC. It was suggested that the README files contain a more thorough description on what "is prime" and what's not. Also, it was requested that such information should be indicated in the product filenames themselves. The information is in the headers, but users don't know unless they're explicitly told. 3. S. Stolovy mentioned the need to have scan-mirror dependent flats for MIPS-24 "non-prime" ancillary data since corresponding DCEs are still being processed, mosaics made and disctributed to users. F. Masci voluntered to do a database search for non-prime DCEs for use in making "non-prime" mips24 flats. 4. The IST mentioned that we will need to add a new entry to the plscripts table to support a new mipsGe dark-calibration exposuretype. S. Wachter voluntered to submit a CR for S11.0 to get this into a future database release. 5. The current processing throughput on ops was briefly discussed. M. Moshir and IST members reported that existing MIPS downlinks consist of significantly more DCEs than what was previously assumed by the Tiger-Team to meet the 5xAcquisition_rate goal. It is not known exactly what the current throughput is and some suspected that there may be a (temporary) lull due to currently large (SWIRE) AORs. F. Masci also reported that there has been significant additions to the pipelines since the Tiger-Team benchmark analyses. 6. A plan to rejuvinate and tune a latent-detection/correction tool for offline use on mipsGe data was discussed. M. Pesenson will look into testing the tool on real flight data. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #21 (Tue. Mar. 2, 2004): Attendees: D. Frayer, D. Padgett, S. Stolovy, S. Wachter, M. Moshir, M. Pesenson, F. Masci, D. Henderson, H. McCallon, J. Fowler, S. Hemple, D. Makovoz 1. Ops processing update: There are sporadic problems in the creation of 160um mosaics. D. Makovoz has looked into it and saw that all BCD uncertainty images had NaNs. This is privy to problems when attempting to create mosaics and coadds using input pixel uncertainties as statistical weights. An example from MIPS002800 is AOR 4884992. D. Henderson was given an action to look into this. 2. D. Frayer reported that the MIPS-Ge stim-interpolation algorithm to fix/alleviate transients seems to be doing more harm than good when the illumination is high. He suggested that simple consequetive "4-frame" averaging may do a better job. He said this option is yet to be tested and will explore it further with help from M. Pesenson. 3. H. McCallon presented some results on an attempt to characterize/calibrate mips24 distortion using available 2MASS information. His work shows the presence of two distinct 2MASS populations correlated with mips24 detections: one at K < 11 mag. and another at K > 15 mag. His analysis of MIPS002500, MIPS002600 and MIPS002700 data also confirms previous findings of a systematic offset in mips24 pointing by upto 1.3 arcsec in the "in-scan" direction. This bias is currently being explored by F. Masci and he will have an answer once all the degeneracies in the parameter phase space are distentangled. 4. F. Masci reported that he has created "non-primary" field mips24 calibration flats for use on science data taken in the same mode. DCEs corresponding to "non-primary" ancillary data are still being processed, mosaics generated and distributed to users. Such products are currently of very low quality due to the unavailable calibration data when mips24 is not the intended target field. 5. The presence of elusive gradients in mips24 products was discussed. There is a persisent gradient in the (+Y) or in-scan direction in BCD products which shows up noticeably in small field (photometry-mode) coadds. This has been tracked down to residual gradients in the mirror- dependent calibration flats which leave their mark in the final BCD products. The IST is currently looking at ways to come up with a more representative set of flats for the data at hand. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #22 (Tue. Mar. 9, 2004): Attendees: D. Frayer, D. Padgett, S. Stolovy, S. Wachter, M. Moshir, M. Pesenson, F. Masci, D. Henderson, H. McCallon, J. Fowler, S. Hemple, D. Makovoz, D. Shupe 1. Processing/development updates: D. Henderson tracked down the problem of persistent NaNs in 160um BCD uncertainty images which was causing mosaic generation to fail. This was due to an uplink scheduling problem in that "stim-specific" exposures were given different exposure-types and were not being picked up correctly in processing. Alberto has agreed to schedule consistent exposure types. A potential new module to perform non-linearity flux calibration on mipsGe BCDs was discussed and D. Frayer requested that we should aim for S11 or as soon as requirements are available. 2. M. Pesenson presented some analysis on mips-Ge latents/response from stim exposures. There were large variations in the latent strengths which defied characterization using any sort of model. He will re-analyse the data as it may be contaminated from effects of high-pass filtering. 3. F. Masci presented his analysis on mips24 BCD uncertainties by examining chi-square distributions from profile-fits output by the source extractor (APEX). He had to modify the BCD error-estimation module to account for non-photon counting events such as "droop". After prior removal of droop to compute Poisson noise correctly, downstream uncertainties gave reduced chi-square distributions very close to expected mean values of one. 4. H. McCallon presented some new results on his analysis of mips24 distortion. He found that prior distortion esimates derived by the IT and validated against IPF results were way off from what he derived. This needs to be further examined as there is likely to be a flaw in the assumptions or analysis. F. Masci suspected that the source extraction information given to Howard may be flawed due to use of CD matrices containing hidden linear distortion terms. J. Fowler requested that F. Masci send around plots of undistorted source extractions as a first step in the investigation. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #23 (Tue. Mar. 23, 2004): Attendees: D. Frayer, D. Padgett, S. Wachter, M. Moshir, M. Pesenson, F. Masci, D. Henderson, H. McCallon, J. Fowler, S. Hemple, D. Makovoz 1. Campaign MIPS002900 updates: There were sporadic failures in the ceation of 160um mosaics. D. Makovoz traced some of the failures as due to smaller than expected FIF (Fiducial Image Frame) sizes when made from predicted pointings. It turns out that the predicted positional angles were wrong by about 90 degrees (relative to actuals) causing BCDs to be mapped/interpolated to non-existent mosaic regions. FIFs are made using predicts in S9.5 software. In S10.0, they will be made using actuals and these problems are expected to dissapear. Some of the 160um mosaic failures are also suspected to be due to saturated BCDs whose pixels are effectively all NaNs as set by the upstream slope fitting module. D. Henderson will look into whether the BCDs were indeed saturated. Furthermore, the mosaicker will choke if given uncertainty images with NaNs. To avoid such occurances, D. Frayer offered to turn off the use of uncertainty images in mosaic generation for 160 and use an equal-weighting option for the input pixels instead. 2. D. Frayer also reported that we want to ensure that non high-pass filtered MIPS-Ge BCD products are labelled as the "primary" BCDs in pipeline processing. The filtered product will become an acillary product. He also reported that currently, the filenames actually indicate the reverse and that he will put in a CR to have this fixed for S11.0. 3. F. Masci reported that he has made a new batch of mirorr-dependent flats for mips24 non-primary observations (effectively photometry mode) from MIPS002800 campaign data. He will ensure these get propagated on ops in the next call for calibration updates. 4. H. McCallon presented his new findings in mips24 distortion analysis. He found that after correcting for the infamous pointing bias of ~1.2 arcsec in BCDs prior to matching with astrometric references, the distortion magnitudes are closer to the code-V (lab-test) model. His distortion estimates still appear to under-estimate the a-priori code-V results by about a fifth of a pixel at two corners. People suggested that he re-fit the results using linear terms included and see what this gives us. If this doesn't improve things, then allowance must also be made for the skew term hidden in the CD-matrix. It was also suggested that he show the magnitude of the dispersion in binned data used as a function of array position. 5. M. Pesenson presented some of his recent analysis on Ge stim latents and their correction in BCDs. He found that stim latent corrections make little difference if performed on median filtered or non-filtered products. However for robustness, M. Moshir suggested that it's best if stim-latent corrections were made before the filtering step to mitigate against latent levels and structure therein being propagated downstream through filtering. It was suggested that he look at an AOR of sufficiently low background to explore the details. D. Frayer also mentioned that the new on-board bias levels for the 70um arrays are likely to alleviate strong stim latents following anneals. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #24 (Tue. Mar. 30, 2004): Attendees: D. Frayer, S. Wachter, M. Moshir, M. Pesenson, F. Masci, D. Henderson, D. Makovoz, T. Hesselroth 1. M. Pesenson presented some preliminary analysis on MIPS-70 stim latents. For the data set used, he found that the recent change in on board bias-level helped little to alleviate latents. However, long term transient behavior is less apparent over the entire array. It was suggested that he repeat the analysis with data of moderately low background (e.g. SWIRE) that has _not_ been median (high-pass) filtered. Given enough data with the new bias setting, there is now a good chance for stim latents to be characterized and corrected to the desired level. D. Makovoz also suggested for robustness to perform the model fitting not just between stims, but also between a stim and the next point source (or outlier) in a ramp. 2. D. Henderson brought up some issues on naming (and renaming) conventions for mips-Ge calibration products in configuration files. The issue was whether the renaming of a calibration product filename (and associated ancillary files) in configuration files will affect simultaneous PAO processing and re-processing of a past campaign in future. M. Moshir suggested that he go through his code and check for any hard-coded filenames before allowing the option of changing cal. filenames in configuration files. If names are likely to change in config. files, then measures should be taken to also update caltrans fallback tables in the database for self-consistency. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #25 (Tue. Apr. 13, 2004): Attendees: S. Wachter, M. Moshir, M. Pesenson, F. Masci, D. Henderson, J. Rho, H. McCallon, J. Fowler, D. Shupe 1. S9.5/S10 I&T updates: everything appears to be progressing along fine. S10 deployment is scheduled for this Thursday. Products from S10 I&T were briefly scanned for expected upgrades and everything looks good. The decision of changing BUNIT for MIPS to MJy/sr to a later TBD S10 patch was brought up. ISTs were made aware of the various procedures for cal/cdf changes (documented in AIs/CRs) that need to take place. 2. H. McCallon reported on his continued analysis of mips24 distortions. His main conclusions are that correlations of 24um extractions with 2MASS reveal that the distortion derived therefrom is consistent with the code-V model characterization to within +/- 0.3 of a pixel for mirror positions +/- 0.3 degree of optical zero. F. Masci will provide Howard with more data so to validate the distortion at larger mirror angles against code-V. 3. F. Masci briefly reported on his creation of new mips24 mirror-dependent flats made from coadds of several thousand BCDs. In addition to the two infamous dark spots (from dust on pick-off mirror) at top right, there are many other dark blotches (at the 0.5-1.5% level) whose position correlates with mirror angle. S. Stolovy reported that correction for the two large dark spots using flats amounts to a photometric (pixel scale) correction of ~20%. J. Fowler also mentioned that it may be worth looking at the uncertainty images themselves to see how uncertainties in the dark-spot regions compares with that in neighboring regions. F. Masci will take an action to look at this. 4. M. Pesenson presented his recent results on analysis of mipsGe stim-latents using data with initial (un-optimized) on-board bias settings. He found that coaddition of DCEs between stims revealed four separate vertical strips (each consisting of four pixel-wide columns) in the coadd. D. Frayer then successfully applied column filtering to remove associated latent signatures from 70um mosaics. The following conclusions were also drawn (written by M. Pesenson): i. The second and fourth strips ('sensitive' strips) have significant latents (40% drop in the amplitude), while the first and the third one ('robust' strips) practically don't exhibit latents. The difference between the robust and sensitive strips diminishes from frame to frame (the further from the stim DCE the less is the difference). So, it's not that the strips are fundamentally different, but rather they respond to stims differently (an illumination pattern?). ii. There is no visual read-out dependence. Mehrdad had suggested to characterize this quantitatively. Absence in variation of the medians of the readouts confirmed the visual picture. iii.The latent's time constant of more illuminated pixels (belonging to the sensitive stripes) is greater then that of pixels exposed to less flux. It agrees with the fact that the time constant is inversely proportional to the background. iv. However, even a brightest pixel from the robust strips still shows a negligible latent comparing to a much darker pixel from the sensitive strips. 5. D. Shupe expressed a concern about MIPS photometric calibrations and questioned the validity of using 10,000K blackbody for K-giant calibration stars. He will talk to J. Rho regarding the details who had left the meeting prior to when this was brought up. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #26 (Tue. May. 4, 2004): Attendees: D. Frayer, D. Padgett, M. Moshir, M. Pesenson, F. Masci, D. Henderson, J. Rho, H. McCallon, J. Fowler, D. Makovoz 1. Creation of PRFs (Point Response Functions) from calibration IERs was discussed. D. Makovoz presented a PRF created from ~700 BCDs provided by J. Rho. The coadd showed noticeable jailbars at the 0.01% level. This was traced to the lack of cross-scan dithering in observations from which the PRF was made. D. Padgett similarly showed a a PSF (not PRF) made from a dithering strategy designed by Karl Stapelfeldt which sampled down to ~1/8 of a pixel. It looked very impressive and showed no jail-baring. D. Padgett will forward F. Masci the relevant IER IDs that were used so that we can make some good PRFs for use in SSC post-BCD pipelines. 2. D. Frayer briefly summarized his work on validating uncertainty propagation in MIPS-Ge pipelines. He's quite happy with the algebraic flow and is looking at ways to optimize the signal to noise when combining BCDs to make mosaics. Some downlink members suggested we hold a splinter meeting to discuss some avenues of approach. 3. Contributed by M. Pesenson: M. Pesenson analyzed stim latents in AOR 8808512_2 (originally suggested by Mehrdad). DCE's between stims (each 25 DCE's) were averaged (four files, 400 DCE's each) on a pixel by pixel basis and then used to find parameters of the latents by fitting. This averaged correction didn't lead to a satisfactory removal of latents. However, this approach is hoped to be more effective for data taken with new bias and this will be analyzed next. In the meantime, a low pass filter has been developed to be applied in the vicinity of stims. The window size W should be determined based on a particular observation (depending on how many point sources is expected) with the rule of thumb W=2d+1, where d is the number of DCE's which will be smoothed without affecting photometric accuracy. It's found that median filtering is equivalent to solving a nonlinear partial differential equation. This connection should be very useful for segmentation of two-dimensional images. 4. F. Masci briefly touched on a method for creating scan-mirror dependent flats and their application. Since the Instrument Team showed quite convincingly that the position of dark spots (due to absorbing dust specs on the pick off mirror) are constant for a single mirror position within a single campaign (and usually differs in the next campaign), it was suggested that prior to each campaign re-processing run, a set of "campaign/mirror-dependent" flats be created offline and then used as fallbacks. This assumes there is sufficient time between initial processing and reprocessing to create the flats. For this matter, all that is required from the initial PAO processing are "tranheaded" FITS files (pointing information isn't even necessary). This will continue for the next N campaigns (where N is TBD) until we are confident we have built a large enough library of flats to provide good sampling of spot positions. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #27 (Tue. May. 11, 2004): Attendees: D. Frayer, S. Wachter, M. Moshir, T. Hesselroth, M. Pesenson, F. Masci, D. Henderson, J. Fowler, D. Makovoz This meeting was motivated by D. Frayer's finding that aperture fluxes derived from SSC-generated MIPS-Ge mosaics are systematically underestimated by ~10% if inverse variance weighting is used in the mosaicker with variances computed from pixel uncertainties in BCD pipelines. Flux estimates from mosaics made with no weighting applied to input pixels were generally as expected within calibration errors and repeatability requirements. At this time, it is not known whether existing assumptions going into the initial noise computation in BCD pipelines is correct and whether uncertainties are really dominated by transient and systematic effects over the period in which data is acquired. D. Frayer reported that the instrument team suggested a ballpark systematic error of ~4% at the BCD level over the time span of an AOR. In lieu of these unknowns, the following action items and tests are suggested: 1. D. Frayer: to look into the scatter in raw counts from repeated measurements of calibration stars (spanning several AORs) at 70um. With good statistics, this will allow one to ascertain whether the dispersion is dominated by systematic effects, are as expected from (uncorrelated) Poisson statistics in the Gaussian limit or both. It is suggested that different calibration stars (with preferably different brightnesses) be used. 2. D. Makovoz: to add an option to mopex to perform equal weighting of input coadded pixels for mosaic generation _and_ generate an uncertainty mosaic estimated from the sample standard deviation divided by sqrt(N); 3. D. Henderson: to update uncertainty estimation for median/high pass filtered BCDs. Currently, the uncertainties are assumed to be same as in the unfiltered case. He will compute a sample variance about the median from pixels within the window filter and combine this with the input pixel uncertainty to estimate the effective uncertainty. In the end, the reported uncertainty for filtered products will be larger. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #28 (Tue. May. 18, 2004): Attendees: D. Padgett, S. Stolovy, D. Shupe, M. Moshir, M. Pesenson, F. Masci, D. Henderson, D. Makovoz, H. McCallon 1. F. Masci received from D. Padgett a list of IERs for derivation of PSFs and PRFs. The IERs were specifically designed with an optimal dither strategy for measuring the PSF. F. Masci will process the data and forward products to D. Makovoz for creation of PRFs required by SSC post-BCD software. 2. F. Masci reported the current problems in attempting to perform pointing refinement for mips24 at the BCD level. Point source match statistics are too low at the required signal to noise ratio to give reasonable extraction centroids. In the end, there is little or no refinement. The existing pipeline will be replaced with a more robust version for S11. This will attempt to refine individual scan-leg mosaics (or exposure-num based mosaics) where the signal-to-noise ratio and hence potential point source match statistics (both relative and with 2MASS) are higher. The refined pointing information will then be written back to BCD headers. For the time being, F. Masci will submit an AI for instructions on how to _temporarily_ terminate existing threads on the ops system. 3. D. Padgett asked what typical mips24-to-2MASS point source match numbers are expected for the different (or most widely used) exposure times and galactic latitudes. F. Masci was happy to look into this in the long term. 4. The issue of how to handle Final Product Generation (FPG) of mosaic products (headers) in a flexible (namelist-driven) manner was discussed. S. Stolovy asked whether we want to keep the existing scheme and have current mosaicking software read-in a configuration file, or, update the existing FPG module. M. Moshir had suggested to look into the capabilities of the "calkeywords" module in passing through/computing averaged keywords as specified in a namelist and then using the existing fpg to re-format product headers. F. Masci will look into whether such a scheme is feasable for S11. 5. ISTs requested that the APERNAME keyword be removed from scan-mode mosaics since in this mode, the keyword value in all channels refers to the channel FOV name which is leading the scan. Observers not familiar with the design could be confused when reading the keyword. It was decided to continue writing the keyword to BCD headers, but use the FPG with appropriate logic downstream to remove it from mosaic headers once a design is in place for S11 (see 4 above). 6. MIPS-Ge filtering/noise mitigation analysis; contributed by M. Pesenson: Hi/low pass linear filtering (in time) was done in the log intensity domain. It revealed that low pass filtering is more effective (in the sense that it produces a smoother result) in the log domain. Filtering in the log domain when applied to a two-dimensional image, increases local contrast. Based on filtering in the log domain, an adaptive modification of the local contrast and local luminance module is being developed as a tool for enhancement of an image with varying amounts of luminance (extended structure). ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #29 (Tue. May. 25, 2004): Attendees: D. Padgett, D. Shupe, D, Frayer, M. Moshir, M. Pesenson, F. Masci, D. Henderson, D. Makovoz, H. McCallon 1. Status on mips24 scan-mirror dependent flats for campaign reprocessing was discussed. MIPS003100 (MC7) and MIPS002000 are ready albeit with some more sanity checking by the IST. M. Moshir suggested if there's a more robust way of generating the flats. For instance, generating a set of superflats for each mode with the spots and crud removed, and then artifically adding the crud signatures (with appropriate weighting) to the individual flats. F. Masci will think about how feasable this is. He is looking for a faster/robust method to keep up with campaign reprocessing. 2. MIPS-Ge Filtering and long-term transients (contributed by M. Pesenson): Low pass filtering produces an output which consists of variations due to variable background and long term transient. When the time constant of the transient is much larger than characteristic time of background change it should be possible to separate the variation of the background from the slow transient. Indeed, subtracting the difference between two consecutive filtering from the original signal, showed (on the current plots) mitigation of the long transient. 3. H. McCallon presented a summary of the new boresight pointing history refinement plan. The idea is to use the results of pointing refinement from bands with good 2MASS/match statistics, (in this case MIPS-24 or IRAC bands 1 and 2) and then using the corrections therefrom to refine the initial pointing history file. The refined pointing history file will then be used in campaign/PAO _reprocessing_ so that all other bands (e.g. MIPS-2,3 and IRAC-3,4 which could not be refined during initial processing) will be refined. H. McCallon and M. Moshir outlined the various SIS's that need to be written before any coding takes place. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #30 (Tue. June 01, 2004): Attendees: D, Frayer, S. Wachter, S. Stolovy, M. Moshir, M. Pesenson, F. Masci, D. Henderson, D. Makovoz, H. McCallon, J. Rho, T. Hesselroth, J. Fowler, R. Laher 1. There has been recent talk about minimizing the number of products copied to the sandbox and keeping only those destined for the archive. MIPS ISTs and CogEs have agreed that this is not possible since all products (mostly intermediate) are needed for propagation between the different chains of pipeline threads. Furthermore, besides from some QA files, a majority of products initially copied to the sandbox for mips24 are also ending up in the archive. D. Frayer took an action to see if there are any unnecessary sandboxed products for MIPS-Ge. 2. The issue of using the calkeywords module together with the existing fpg to propagate BCD keywords (and reformat) mosaic headers was discussed. The calkeywords module needs to be updated to pass user specified keyword value strings from the first listed FITS image to the output image. F. Masci will put in a CR to get this updated. 3. F. Masci presented some anomalies from mips24 uncertainty images where entire rows were being filled by NaNs by the rowdroop module. This happened to rows which initially contained a NaN uncertainty pixel due to the saturated core of a bright source. The first problem has been fixed (by Iffat Khan) by skipping over NaN'd uncertainty pixels in a particular row when computing the uncertainty in rowdroop correction and the second will be fixed by F. Masci by ensuring that the correct bit mask is set for saturated cases and that this is set in the fatal bit mask template for mopex. 4. J. Rho had reported that mopex was underestimating the flux for bright sources in mips24 mosaics. D. Makovoz performed some analysis and found that in order to have flux conservation (from BCDs to mosaics), the mosaics should be made with no inverse variance weighting where the variances here are BCD pixel uncertainties. It is not clear whether the BCD uncertainties are as expected from (uncorrelated) Poisson statistics in the Gaussian limit, or whether there is some temporal or spatial systematic effect amongst the coadded pixels. F.Masci and J. Fowler found it somewhat nauseating that the decision to abandon use of BCD uncertainties as statistical weights was based on some very circumstantial and inconclusive evidence. F. Masci will look into the assumption of Poisson statistics by looking at repeated observations of sources in raw images. For the time being, he will update the mopex namelist to avoid using sigma weighting. 5. Super-boresight pointing history (refinement) update: (the purpose for this was summarized in the last meeting minutes). H. McCallon presented a different strategy where in addition to refining the boresight pointing, that the boresight-to-FOV Euler angle offsets be simultaneously refined by using data from all bands (e.g. IRAC1-4). This idea was shot down by first noting that 2MASS/match statistics are not robust in bands-3 and 4 and we can't rely on all BCDs being refined from the pointing refinement module, and second, J. Fowler mentioned that it would be redundant to additionally refine the Euler offsets since the refinement can all be absorbed in boresight pointing corrections alone. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #31 (Tue. June 08, 2004): Attendees: S. Wachter, S. Stolovy, D. Padgett, M. Pesenson, J. Fowler, F. Masci, D. Henderson, D. Makovoz, H. McCallon, J. Rho 1. There has been a recent burst of failures in online PAO processing of campaign MIPS003200 (MC-7) in mips24 flatfield calibration pipelines. The problem has to do with ensembles consisting of >~ 2000 DCEs and exceeding a hard-coded limit of 1000 in the software. Products from _online_ flatfield calibration pipelines are _not_ currently used in science processing. Calibration flats are made and refined offline according to scan-mirror position. The IST has asked that this be fixed in the long term (S11). Incidently, F. Masci has already put in a CR to update the "flatfield" module. 2. S. Stolovy reported that she has made superflats to support MIPS003100 (MC-6) reprocessing to accompany the mirror-dependent set made by F. Masci. She also wrote an IDL script to filter out radhits from the set of fast scan flats due to poor sampling at the different mirror positions. MIPS002100 (EROs) followed by MIPS002600 (MC-2) are next in our reprocessing queue. 3. The issue of using BCD uncertainties as weights during mosaicking was briefly discussed. Following the controversy from last week's meeting regarding use of weighted averaging in a badly calibrated mips24 dataset, J. Fowler performed some thought experiments and wrote up his findings in a document. The document is attached below. It can also be found in /ssc/pipe/docs/working_notes/mips24/ called PoissonProperties.PDF. I encourage uninterested parties to ignore the attachment, however, for those deeply involved in data processing, it elucidates several important key points not obvious to some. 4. Contributed by M. Pesenson: The problem of brighter sources having larger uncertainties has been approached as signal-dependent noise and a module for removing such noise is being developed. It is based on the following algorithm. Due to the nonlinear coupling of signal and noise, linear filters are not useful for removing signal-dependent noise. Even nonlinear median filters do not remove such noise. Therefore decoupling of the noise from signal is highly desirable. Strictly speaking, such exact decoupling possible only for a multiplicative noise. However, it can be achieved approximately by passing the observed image through a certain nonlinearity. Having done this, the noise is removed by applying an appropriate filter, depending on the probability function of the noise. After this, the inverse nonlinear function restores the signal. 5. H. McCallon presented another approach to refining the FOV offset angles along with the boresight. This approach addresses the concerns raised at last week's meeting where a plan to simultaneously solve FOV offset refinements along with the boresight refinements was presented. The new plan solves for FOV offset refinements after the boresight refinements have been determined. It uses the difference between the combined boresight position and what that refined position would be using only the channel of interest and it's apriori FOV offsets as input for a least-squares fit. Although a little less optimal than a simultaneous solution, it appears more practical. It's advantages include: 1) The time range of the FOV offset fits can be extended indefinitely to increase the input point count, assuming the offsets are truly static. Also sub-sums can be output periodically to check for variability. 2) In addition to picking up any FOV offset biases, it would characterize the uncertainties (needed, but not presently well known) on these parameters. It was generally agreed that this approach has promise and should be pursued at least to the point of characterizing FOV offset biases and uncertainties for a set of test data. F. Masci said he could provide a test set with very high density such that even IRAC channels 3 and 4 could be easily refined. In other comments regarding testing D. Henderson suggested also using simulation data. H. McCallon agreed and stated he planned to use Monte Carlo to test the uncertainty modeling. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #32 (Tue. June 15, 2004): Attendees: S. Wachter, D. Frayer, D. Padgett, M. Pesenson, F. Masci, D. Henderson, H. McCallon, M. Moshir 1. Updates: F. Masci has made mirror-dependent flats for campaign MIPS002600 (MC-2) and said we should be ready to proceed with reprocessing once Susan gets back and performs a quality check. Flats for other campaigns in the queue will be made once Susan gets back. F. Masci will show her how to make them starting from online MIPL data. 2. F. Masci presented an explanation for some recent failures regarding Ge pointing transfer in reprocessing of campaign MIPS002100 (mostly EROs). It turns out that the affected dces were commanded for TPM, while pointing transfer was treating them as SED mode. The s/w uses CSM_PIDX=41 (exactly) to trigger special SED mode handling. This case had TPM with CSM_PIDX=41. D. Frayer said this is a minor detail pertaining only to IERs taken during IOC/ERO scheduling. Since dces were comanded as TPM, they have different aperture IDs from SED. Thus, the software had trouble writing/reading specific RA_SLT, DEC_SLT keywords downstream. D. Frayer voluntered to investigate this further. 3. D. Henderson brought up the status and whereabouts of a "real" electronic non-linearity calibration file to use in Ge processing. D. Frayer said that the IT (K. Gordon) is still characterizing the data and that the IST might also be involved in the analysis. D. Frayer indicated that this is not a high priority since it's buried in other more important sources of systematics. 4. M. Pesenson presented some recent analysis regarding ways to mitigate high frequency systematic (noise) contributions to backbround estimation in Ge data. People were concerned that the method might corrupt the "true" noise and signal components in BCDs and that he may be better off using the available pixel redundancy. Work on this continues. 5. H. McCallon briefly summarized his approach to refining the boresight pointing (and FOV angles) using a "sequential algorithm" (see Minutes #31). F. Masci suggested that we may first want to verify the accuracy of FOV angles and refine them offline (if needed) using high density data (e.g. GLIMPSE for IRAC). This will minimize potential systematic biases in pointing and leave us with the problem of refining the boresight pointing alone. ----------------------------------------------------------------------- MIPS Downlink Processing Meeting Minutes #33 (Tue. Sep. 7, 2004): Attendees: D. Padgett, M. Moshir, S. Stolovy,H. McCallon, M. Pesenson, J. Fowler, F. Masci, D. Henderson, T. Hesselroth 1. S. Stolovy said whe will prepare a Final Product Generator (FPG) template cdf to use on mips24 mosaics. The template will be generic enough so that it can be used for mipsGe and IRAC. 2. D. Henderson is currently grappling with a scheme to ensure that a mipsGe thread which generates flatfield calibration products can also be used to process science data by keying off the respective exposure-type. 3. F. Masci talked about methods on how to ameliorate the existence of bit-mask settings in the mips24 science pipeline which say things like: "flatfield applied using _questionable_ value". The word _questionable_ has raised a barrage of questions from the public through user support. This comes from the module which applies the flat to science data. It checks the calibration "c-mask" to see if while the flat was made, one or more outliers were detected in the pixel stack. If so, the "questionable" bit-mask is set in the b-mask which accompanies the BCD. F. Masci suggested that CRs should be submitted to make the setting of c-mask bits by the flatfield module namelist configurable, together with some threshold for the fraction of radhits/outliers above which the c-mask bit should be set. Since mips24 flats are made offline, J. Fowler proposed the following quick fix. He has volunteered to write a simple module which sets the desired (user specifiable) bits in an input c-mask according to whether the input SNR of pixels in the flat falls below some user specifed threshold. The bits should be reset to zero if the condition fails. 4. M. Pesenson presented some of his recent work on nonlinear and adaptive filtering. The aim is to improve the overall SNR in 70um mosaics. The main concern is that real "faint" point sources were being lost in the filtering process. The following additional analyses were suggested: - Histogram the pixel data before and after filtering to see the distribution shape. - Start with a "raw" noisy mosaic with original background, add artifical point sources and then try to extract these with the source extractor, both before and after filtering. i.e., perform a completeness and reliability analysis. - Just simply perform source extractions on filtered and unfiltered mosaics and compare source statistics at low SNR levels. - Perform filtering on BCDs first, then mosaic them. Will there be a difference? 5. F. Masci distributed a plan for the new "superboresight" design which will be used to refine the pointings of IRAC and MIPS during campaign reprocessing runs. Due to time constraints, the plan was not discussed. =======================================================================