2MASS Tape Operations
2MASS Tape Operations
Roc Cutri - IPAC
Revised 14 August 1996
I. INTRODUCTION
In an earlier memo projecting the growth rate of the 2MASS raw and processed
data volume, an estimate was made for the total number of tapes required
during the survey. Based on nominal photometric time percentages of 30% in
the north and 50% in the south, a simple baseline tape operations plan requires
a total of ~4700 DLT 4000 Type IV tapes for the survey. At the current unit
cost of DLT Tape IV of ~$90-$100, the total tape budget would approach a half
million dollars. The hardware budget for 2MASS data processing will not
support this amount.
In this memo, I propose a tape operations plan that significantly reduces
the survey tape costs, while maintaining minimum standards for raw data
integrity and archiving. Additional costs associated with this plan do
include increased coding time and some hardware to be installed at the
observatories and IPAC.
II. EXISTING TAPE PLAN
The baseline operations plan calls for data to be taken on all possible
nights. A conservative estimate is that the number of observable nights
will be approximately twice the number of photometric nights. Two copies of
each night's data are written, one per DLT 4000 Tape IV (20GB native
format), and one is shipped to UMASS and one to IPAC. At IPAC, nightly
quality assessment would take place during processing, and the final decision
to accept or reject a night is based on this assessment. Over the life
of the survey, this implies that data tapes would be written and processed
for approximately 1875 telescope nights (2 x 46 months x 0.3 phot. =
840 nights north, and 2 x 34 months x 0.5 phot. = 1035 nights south). Thus,
nearly 3700 DLT tapes will be required to store just the raw data archive.
The baseline plan calls for one night's processed data to be written to one
archive DLT tape. Thus, if there are ~940 photometric telescope nights during
the survey, the total processed data tape archive will require the same number
of tapes.
III. PROPOSED TAPE PLAN
1. Observatory Tape Writes
Under my new observatory tape operations plan, data would still be
acquired on every possible night. However, a minimal version of the 2MAPPS
pipeline would be installed on mountain computers, and the calibration scans
from each night would be processed on-site and a basic assessment of the
photometric quality of the night made immediately following a nights
operations. Only raw data for nights that exhibit at least 4 consecutive
hours (TBD) of photometric conditions would be written to DLT
tape. Two uncompressed copies of the raw data tapes will be written;
one copy will be shipped to IPAC and one will remain on the mountain.
This will reduce the estimated number of nights for raw data tape writes
by approximately a factor of two, to ~940. It also reduces the number of
potentially bad nights that will have to be processed thereby improving the
capability of the pipeline processing to keep up with the survey pace.
2. Raw Data Archiving at IPAC
Within 2 weeks of receiving a raw data tape from the observatories, the
tape will be read and verified for content and integrity at IPAC, and the
data written to disk. Once on disk, the data will be "pre-conditioned" using
the algorithm devised by Gene Kopan and Nian-Ming Chiu (see Appendix I).
The pre-conditioning consists of simply evaluating the differences between
successive frames in each scan. The differencing reduces the entropy of the
data, and the resulting frames will realize much larger lossless compression
ratios using simple LZ algorithms, such as those employed by unix compress,
gnu gzip, or most tape hardware compression chips. The first raw frame and
following 259 differenced frames and all ancillary text files from each scan
will be compressed, and written to tape using standard tar formats.
As described in Appendix I, this method generally achieves lossless compression
ratios of 2:1. Therefore, on average two nights raw data can be archived onto
each DLT 4000 Tape IV. Efficient tape "packing" will further reduce the number
of raw data archive tapes. The general rules for "packing" the raw data
archive tapes are:
1) Nights will be written sequentially in date order on each tape.
2) Nights will not be split between two tapes.
3) Northern and Southern raw data will not be mixed on archive tapes.
One copy of each raw data archive tape will be sent to UMASS and one will
be stored on-site at IPAC.
Once an observatory tape has been read and verified, and the two archive copies
have been written and verified, the original observatory tape will be returned
to the mountain. Upon receipt of the returned original tape at the
observatory, both that tape, and the second stored copy will be freed for
recycling for raw data.
If we conservatively estimate that the turn-around from data acquisition
to return of tapes to the observatory will be ~4 weeks, then there must
be a rotating stock of approximately 110 tapes per observatory. Over the life
of the survey, each of these tapes will be cycled 6-8 times.
3. Processed Data Archiving
By far the largest component of the processed data archive will be the
Atlas Images. Over the life of the survey, the image archive will grow
to ~11.5 TB in size, while the source database will be at most ~0.2TB.
Therefore, efficient packing of the image data will result in the greatest
savings in media costs.
One night's processed data occupies only ~70% of the volume of the
raw data, or an average of 12.7GB. With only a very samll amount of
compression at least two nights processed data can be fit on each archive
DLT tape. The coadd 1995 ProtoCam coadd images currently on-line on
lugosi and karloff are actually compressed versions of the original images
that have been pre-conditioned by binning using the FITS bzero, bscale
capability, and compressed using standard unix "compress." This is not
strictly a lossless procedure, but the loss can be limited to a very small,
low range of bits through judicious selection of bscale. The coadd ProtoCam
images on disk now have realized a factor of ~4 compression.
I assume that a factor of 2 compression can be achieved for the survey
Atlas Images after processing, either through the bscale or some other
pre-conditioning scheme. If that is the case, then on average three nights
processed data can be written to one DLT 4000 tape using the same
packing rules as for the raw data archive.
IV. PROJECTED TAPE COSTS
a. Tape Cost - The current unit cost for DLT 4000 Tape IV has been negotiated
at $87. Tape vendors have estimated the price to drop by 10-15% over the
next year, according to reports from their suppliers. Longer term price
profiles are not available. It is recommended that tapes be purchased in
minimum volume to take full advantage to any future cost reductions unless
a substantial savings can be obtained through large volume purchase.
b. Tape Numbers -
i. Rotating Telescope Stock: 110 tapes per obs. 220 tapes
ii. Raw Data Archive : 0.5 DLT/night x 937.5 = 469
iii. Processed Data Archive: 0.33 DLT/night x 937.5 = 309
iv. Miscellaneous: 100
TOTAL 1098 tapes
c. Cost -
1098 tapes @ $87/tape (current cost) $95.5k
@ $78/tape (1997 cost) $86.0k
@ $67/tape (avg. for 10% decline
per year) $73.9k
V. IMPACT
To realize the tape cost savings outline above, there will be compensatory
costs in hardware and software. Before any decision is may to adopt
all or part of this proposal, we must decide if the potential savings
exceed these new costs.
- The proposed plan requires the installation of a workstation running
a truncated version of the pipeline at each observatory. The cost of the
workstations and installation and maintenance of the pipelines must be
determined.
- A premium will be placed on tape I/O at IPAC. In order to avoid critical
impact on the processing pipeline, the tape I/O should probably be handled
by separate computers (one for the north, one for the south) that are
cross-mounted to the production cluster disks. The role of these computers
would be to read raw mountain DLT tapes, pre-condition and compress the data,
and write the two raw data archive tapes for each input tape. These Tape
Control computers would also be responsible for reading the raw data archive
tapes when processing is to be carried out, and if reprocessing is necessary.
Archiving the processed data to tape may also be handled by these machines.
Depending on architecture and system cost, these machines may provide some
backup on processing power. Alternatively, one of these machines, if of
sufficient capacity, could act as the fileserver for the 2MASS sub-net.
- A minimum of 2 DLT drives per production machine will be necessary to handle
the increased I/O function. A strong case may be made for 3 drives.
- The TAPELOAD subsystem would have to undergo significant revision to handle
input from the raw data archive tapes.
- A new DLTLOAD process would have to be written. This subsystem would have
the responsibility of doing nothing by reading, writing and verifying tapes,
and providing some quality measures for feedback to the observatory.
APPENDIX I. - Gene Kopan and Nian-Ming Chiu's Memo on 2MASS Raw Data
Compression
26 June 1995
Preliminary results are in on compression, thanks to Nian-Ming
and Dave Gregorich.
2mass raw data does not compress significantly, however, with
simple data pre-conditioning, compression ratios of 2-2.5 are easily
achieved with the standard unix compress program or the gnu gzip program.
These programs use variants of Lempel-Ziv coding which has the dual
advantages of speed and single pass operation. In addition, the code
is available for the gnu gzip program and it has been ported to many
architectures including Intel/DOS/Linux.
Nian-Ming Chiu has implemented a pre-conditioning/ compression
engine under Solaris which has demonstrated the feasibility. He has
written a program which pre-conditions the data, taking advantage of
the frame-pair periodicity to reduce the entropy prior to compression.
The method is to simply replace the raw data with the frame pair
differences for all frame pairs after the first. This operation is
quite fast, and allows typical compression ratios of 2-2.5.
This technique could be used to compress the data on-site and
reduce the time taken to write the data to tape, and the number of
tapes. If the compression can be done while the data is written to
tape, the total time might be reduced. The time to read the data at
IPAC will be reduced by a factor of 2-2.5, and the raw data could be
left in the compressed form until used, reducing io and saving disk
space. The final decision may depend on the practicality at the
observatory. The major gains would be reducing the tape inventory and
io time at IPAC. With a 90Mhz Pentium, it will be tricky saving time
at the observatory.
The following table summarizes compression results using data pre-
conditioning on a 50Mhz sparc (karloff) and a 90Mhz pentium. The
pentium numbers were provided by Dave Gregorich.
machine compression file size % cpu time throughput time
-----------------------------------------------------------------------
none 68157440 100
sparc(50Mhz) compress 57335171 84 3:22 / 1:48
sparc(50Mhz) tzip -g 30961812 45 3:36 / 1:31
sparc(50Mhz) tzip -u 27473874 40 156 / 72 2:39 / 1:43
sparc(50Mhz) tzip -1u 29215162 43 3:02 / 1:55
sparc(50Mhz) tzip -1g 31715902 46 3:49 / 1:29
pentium(90) compress* 27473874 40 156 / 63 4:01 / 2:34
pentium(90PCI) compress* 27473874 40 155 / 62 3:16 / 1:32
time required to write 68MB > 8mm tape ~ 2:20
Notes: - tzip is Nian's combined pre-conditioning and compression
with several options (see attachment).
- the numbers given are for compression / decompression
- the pentium tests were preformed on pre-conditioned data
and under Linux.
- the pentium(90PCI) run used an IDE drive attached to the
PCI bus.
- the data pre-conditioning takes just a few seconds of cpu
This process would be more attractive with a 133Mhz or dual processor
Pentium at the site.
gene
I attach Nian's info below:
Gene,
Here is what I have so far.
2mass scan data compression program.
compression:
tzip [-[1|2] -[u|g]] input_file output_file
uncompression:
tunzip [-[1|2] -[u|g]] input_file output_file
Examples:
tzip 2mass_data 2mass_data.Z
tunzip 2mass_data.Z 2mass_data
Options compress uncompress
program program
--------------------------------------------
1 sub1 add1
2 sub add
u compress uncompress
g gzip -1 gunzip
sub (sub1) is a simple filter to preprocess a data file before compression.
It can increase compression for data whose points tend to be close to the last
point. The output is the difference of successive bytes of the input.
The add filter is used to undo what sub does.
sub method:
C1 = B1 - B0
C2 = B2 - B1
Cn = Bn - Bn-1
sub1 method:
C1 = B1
C2 = B2 - B1
C3 = B3 - B1
Cn = Bn - B1
Examples:
sub < 2mass.data | gzip -1 > 2mass.data.gz
sub1 < 2mass.data | gzip -1 > 2mass.data1.gz
add (add1) is a simple filter to postprocess a data file after decompression.
It can increase compression for data whose points tend to be close to the last
point. The output is the sum of successive bytes of the input.
add method:
B1 = C1 + C0
B2 = C2 + C1
Bn = Cn + Cn-1
add1 method:
B1 = C1
B2 = C2 + C1
B3 = C3 + C1
Bn = Cn + C1
To expand, use the reverse operation, as in:
Examples:
gunzip < 2mass.data.gz | add > 2mass.data
gunzip < 2mass.data1.gz | add1 > 2mass.data
Test tzip
Machine: karloff
uncompress data: /k9/gene/data/sa57ah3
size: 68157440
%: 100.0
compress time: N/A
uncompress time: N/A
command: compress /k9/gene/data/sa57ah3
compress data: /k9/gene/data/sa57ah3.Z
size: 57335171
%: 84.1
compress time: 3:22
uncompress time: 1:48
command: tzip -g
compress data: /scr/data/sa57ah3.gz
size: 30961812
%: 45.4
compress time: 3:36
uncompress time: 1:31
command: tzip -u
compress data: /scr/data/sa57ah3.z
size: 27473874
%: 40.3
compress time: 2:39
uncompress time: 1:43
command: tzip -1u
compress data: /scr/data/sa57ah3.1z
size: 29215162
%: 42.8
compress time: 3:02
uncompress time: 1:55
command: tzip -1g
compress data: /scr/data/sa57ah3.1g
size: 31715902
%: 46.5
compress time: 3:49
uncompress time: 1:29
--
Nian-Ming Chiu Phone: 595
E-Mail: nmc
office: 149