SCANSCI High-Level Documentation

Version 2.6.2
27 October 2000 

Purpose

SCANSCI is just one small part of the 2MASS pipeline process. It is intended to gather information compiled by other subsystems, perform quality analysis on the nightly data acquired, and then print an HTML report summarizing the scan-by-scan QA performed.

High-Level Organization

The purpose outlined above suggests an overall design split into three clear, distinct parts: Input, Q/A, Report. Accordingly, the program is organized into these three areas, and (with few exceptions) no overlap occurs between these functional zones.

Although SCANSCI is written entirely in C, it is designed in an object-oriented manner (to the extent that the problem domain and language choice allow). For the most part, the code is not organized by function, but rather by data types (grouped with the functions which operate on them). This organization breaks down somewhat in the input and report routines (I/O is always messy!) because the input routines are strongly influenced by the files to be read, while the report routines are strongly influenced by the HTML format and other concerns which may cross data boundaries (for example, links to external plots).

The error-handling strategy we employ is this: If a nonfatal error occurs (such as one of the input files in an unexpected format), then we simply note the error and continue operation, printing the error message in the final report file. If a fatal error occurs (out-of-memory, or unable to open the final report file) then we print an error message to stderr and abort. (Currently, however, all error messages are just printed to stdout.)

File-Level Organization

SCANSCI consists of fourteen header files (with the suffix .h) and sixteen source files (with the suffix .c), plus this HTML documentation and a Makefile. A project file for use with the Integrated Development Environment (IDE) Metrowerks Codewarrior is also available upon request. Header files are used primarily for comments (explaining the purposes of individual functions and data structures) and function and data declarations. Source files are used primarily for implementing these declarations, but occasionally also include their own declarations (for information private to the implementation).

In particular, the main structure is the scanlist, which consists of individual substructures (scan), each of which is in turn made up of substructures (tracking, focus, focus_quality, etc.) Each of these substructures has its own header and source files, containing typically three functions: One for initializing a new structure, one for performing Q/A on it, and one for printing the structure to stdout (essentially a debugging feature). Functions for reading these structures from file or writing them out to a final report tend to have overlap with one another, and consequently are placed elsewhere.

The header and source files are listed below, along with an overview of their contents:
 
 
 
 
 
 

lines of code  filename  purpose and contents 
102 
errors.h
 global error-handling facilities 

contains two custom types (my_error and error_report) for error-reporting, plus function declarations 
 
 
 
 

 
142 
errors.c
 global error-handling facilities 

contains implementations of the functions declared in errors.h, along with custom functions for handling certain fatal errors. 
 
 
 
 

 
76 
debug.h
 global debugging facilities 

includes support for Design-By-Contract (preconditions, etc.). All debugging tools are routed through the global error-handling facilities. Debugging checks can be turned off by defining the global variable NDEBUG (either in the source, or through a compiler directive like gcc's -D flag). 
 
 
 
 

 
77 
globals.h
 constants and macros used throughout the program 
15 
globals.c
 necessary only to give a definition for the Q/A names (green, yellow, etc.). (Could be done in any source file.) 
148 
scan.h
 definition of the overarching structures holding the individual scan data (and Q/A results) and the entire scan list. This file is needed by almost every part of scansci, but is not so primitive as to be a part of globals.h. 

It also includes declarations of the three primary functions (read_scanlist, qualify_scanlist, and summary_report) as well as functions for operating on individual scans or the scanlist as a whole. 
 
 
 
 

 
102 
scan.c
 Implementations of the functions operating on individual scans or the scanlist as a whole. 
68 
tracking.h
 declaration of tracking data and Q/A results, plus functions for working with them 
86 
tracking.c
 implementation of the functions declared above 
74 
focus.h
 declaration of background/seeing/focus data and Q/A results, plus functions for working with them 
128 
focus.c
 implementation of the functions declared above 
84
astrometric.h
 declaration of astrometric data and Q/A results, plus functions for working with them 
161 
astrometric.c
 implementation of the functions declared above 
50 
photometric.h
 declaration of photometric data and Q/A results, plus functions for working with them 
65 
photometric.c
 implementation of the functions declared above 
86 
galaxy.h
 declaration of galaxy data and Q/A results, plus functions for working with them 
86 
galaxy.c
 implementation of the functions declared above 
51 
quad_jump.h
 declaration of quadrant jump/planet hazard data and Q/A results, plus functions for working with them 
71 
quad_jump.c
 implementation of the functions declared above 
48 
file_util.h
 functions for making working with files a little more sane 
56 
file_util.c
 implementation of the functions declared above 
86 
input.h
 declaration of functions used to read in all the data, along with the relevant directory names 
1191 
input.c
 implementations of all of these functions and their helpers. Big and messy, but file I/O in C always is. 
67 
qualify.h
 Convenient macros used by all the QA routines. 
107 
qualify.c
 Implementation of the main QA routine (which calls all the others) and grading routine (which grades each scan). These functions were declared in scan.h 
43 
report.c
 Implementation of the main reporting routine (declared in scan.h) 
43 
debug_report.c
 Primarily useful for debugging purposes. Uses the print functions defined in each substructure's .c file to print an overall summary to standard out. 
139 
html.h
 Declarations of useful macros when working with HTML, along with constants defining the URLs used in the resulting pages (especially links to images or plots) and function declarations for the HTML generators. 
721 
html.c
 Implementation of the functions declared above. Big and messy, like input.c 
57 
main.c
 Contains the main() routine which extracts whatever command-line parameters are passed to SCANSCI and then drives the entire process. 
4230 total lines of code 

Error Handling

To the extent possible, I trap all errors encountered and store them in a global list (error_list, defined in the files errors.h and errors.c). Fatal errors (such as running out of memory, or encountering a programming error, or being run with the wrong number of command-line parameters) will cause program execution to stop immediately and a message to be printed to stderr (or possibly stdout). Otherwise (if, for example, an input file could not be opened, or a file was in an unexpected format) then execution continues as best it can.

Once the program is ready to generate the HTML report, a summary of the errors encountered is listed at the top of the report in an "Error Summary" table. You can suppress the generation of this table by recompiling scansci with the OMIT_ERRORS flag defined (either in html.h, or as a -D parameter to the gcc compiler).

Anticipated Changes

A roadmap broadly covering changes that may occur and their effects on the source code may help those who are left with the job of maintaining this code. Following is a list of anticipated changes (in both the requirements and implementation) and the parts of the code which must change to meet them:

Input files change

In this case, the files input.h and input.c will be the only ones affected. Filenames are mostly hard-coded in input.c, but are only used in one place each (namely, the reading functions read_*). Directory names are defined as constants in input.h.
  You will need to create a new reading function (read_whatever) and make sure it is called from the function read_scanlist at the top of input.c. Your function should be declared in input.h, along with the other reading functions. Most likely, you can emulate one of these existing functions (choosing either a function such as read_psf_see, which reads one file containing one entry for every scan, or a function such as read_bmg which reads one file for each scan). These functions in turn call helper functions (_read_whatever) which do the actual work. So you would probably also emulate one of these helper functions (e.g., _read_psf_see or _read_bmg). The main thing you would need to change would be the format string describing the data being read, and of course which data fields in the scan structure (or its substructures) are affected.
  As above, you will need to modify the format string used in the reading helper function. Any good C reference will explain the use of the library function fscanf in detail. Additional data being read may also necessitate changes in the data structures themselves, their Q/A routines, their initialization routines (e.g., in tracking.h and tracking.c) and possibly also the report format (as in html.h, html.c).

Q/A process changes

If, for example, the data structure affected is focus_data, then you would edit the files focus.h and focus.c to introduce these changes. The macros defined in qualify.h may be of help (see focus.c for examples of use).
  Depending on the change, this might be as simple as just changing a value, or using a different macro (e.g., CHECK_UPPER instead of CHECK_LOWER). Again, see the individual substructure files (such as focus.c) for details.

Report generation changes

Changes in the HTML report format are made in the files html.h and html.c. Depending on the kind of change to be made, this could be as simple as editing one of the URLs defined in html.h, or as complicated as changing every implemented routine in html.c (if, for example, every table is to be laid out vertically instead of horizontally).

If you want to change whether the text is printed in color or the table entry backgrounds are (or neither), change the definitions of USE_TEXT_COLOR or USE_ENTRY_COLOR in html.h. The actual color codes used can also be found there.

The output routines in html.c are complex and somewhat intertwined; I recommend you examine them thoroughly before embarking on significant changes to them.

Additional Notes

SCANSCI was developed entirely on a Power Macintosh G3 (266 MHz) minitower, using the Metrowerks CodeWarrior development environment, in about 54 hours (including time spent in meetings, designing, and writing documentation, testing, and debugging). It compiles and executes to completion under both the MacOS and Solaris.

Thanks to Davy Kirkpatrick and Linda Fullmer for their extensive help and direction during the development phase, and to Roc Cutri for the introduction to/overview of 2MASS. Thanks also to Dave Van Buren for loaning me to 2MASS for a week, and providing me with such a great work environment.

As of August 1998 maintenance of scansci has passed on to Robert Hurt who sadly is developing entirely under unix and not his beloved Mac.

Version History

2.6.2- 27 October 2000 2.6.1- 12 October 2000 2.6- 8 August 2000
  • Added Bla0 override in .qagrade file to allow flagging of H electronics glitches. This represents an incompatibility with previous versions of this file.
  • The contents of Gene's *.badfsig files are appended to the end of the review templates (used to help identify possible electronics glitches).
    • 2.5.1- 10 May 2000 2.5- 10 May 2000 2.4.3- 17 April 2000 2.4.2- 4 April 2000 2.4.1- 22 March 2000 2.3- 4 February 2000 2.2- 3 January 2000 (R. Hurt). 2.1 (PARC)- 4 November, 1999 (R. Hurt). 2.0- 4 November, 1999 (R. Hurt).


    1.9- 17 September, 1999 (R. Hurt).


    1.8- 31 August, 1999 (R. Hurt).


    1.7- 17 March, 1999 (R. Hurt).


    1.6 - 10 February, 1999 (R. Hurt).


    1.5 - 14 January, 1999 (R. Hurt).


    1.4- 30 September, 1998 (R. Hurt); numerous minor improvements.
     

    1.3 - 4 September, 1998 (R. Hurt).


    1.2 - 2 September, 1998 (R. Hurt).


    1.1 - 17 March, 1998.


    1.0 - 6 March, 1998.