Go to the previous, next section.
The native format of UniPOPS for storing and retrieving data is called SDD. The file consists of three parts: bootstrap, index, and scans, and is a direct-access file. The file has the following layout,
SDD File Format _________________________________________________________ | BOOTSTRAP |-------------------------------------------------------- | INDEX | | | | |-------------------------------------------------------- | SCANS | | | | | |_________________________________________________________
The binary representation used in an SDD file is that of the IEEE standard. The checkfile.exe utility will list the contents of the bootstrap and index section of the file while makeindex.exe will reconstruct the index and some of the items in the bootstrap part of the file by looking at the scan section of the file. The makefile.exe utility will create an empty SDD file with a specified size.
As of the fall of 1993, a new bootstrap and index format are in use. All of the primary UniPOPS programs are able to read both the old and the new forms transparently to the user. The new format primarily involves switching from 16-bit integers to 32-bit integers. The makeindex.exe utility can be used to generate a new bootstrap and index for an old-format SDD file. The makeoldindex.exe utility can be used to generate an old-format bootstrap and index.
The bootstrap occupies one record. The structure and contents of the bootstrap record is:
SDD Bootstrap Structure
-------------------------------------------------------------
Word Size Description
-------------------------------------------------------------
Integer*4: Number of records in index including the bootstrap
record.
Integer*4: Number of data records which follow index records.
Integer*4: Bytes per record.
Integer*4: Number of bytes per index entry.
Integer*4: Number of index entries already used.
Integer*4: Counter which indicates whether the file has
been updated.
Integer*4 Type of SDD file (0 if data, 1 if individual records)
Integer*4 SDD version number (this is version 1)
Integer*4: Zeros to pad out the record to its full size.
-------------------------------------------------------------
Old SDD Bootstrap Structure
-------------------------------------------------------------
Word Size Description
-------------------------------------------------------------
Integer*2: Number of records in index including the bootstrap
record.
Integer*2: Number of data records which follow index records.
Integer*2: Bytes per record.
Integer*2: Number of bytes per index entry.
Integer*2: Number of index entries already used.
Integer*2: Counter which indicates whether the file has
been updated.
Integer*4: Zeros to pad out the record to its full size.
-------------------------------------------------------------
The actual number of bytes in each record of the file can be different for each file. Typically, it is 512 bytes per record and all records in the file have the same size.
Each scan in the scan section of the file is summarized by an entry in the index part of the file. The bootstrap gives the number of bytes occupied by each index entry and has a minimum value of 64 bytes. The number of bytes in each record of the file must be evenly divisible by the number of bytes in each index entry.
As scans are stored in the file, the number of index items already used will give the number of the highest index entry that describes a scan. It need not be the entry describing the last scan written to the file and not all index entries lower than that one need be filled (i.e., index entries need not be filled sequentially). This number is zero for an empty SDD file. Also, as scans are stored in the file, the number of records in the data section of the file is stored in the bootstrap as well. This entry in the bootstrap is zero for an empty SDD file.
If there are N-1 index records (so that there are N records in the bootstrap plus index), and if each index entry takes up M bytes, and if the file has L byte records, the number of scans which can be stored in the file will be: (N-1)*L / M. Typically, N is 129 or 513, so the number of scans which can be stored in an SDD format is 1024 or 4096. The number of records in the index thus dictates the maximum number of scans which the index can describe and the limit on how many scans can be stored in a file. When an SDD file is created (with the makefile.exe utility), the user can specify the maximum number of scans which the file will contain. The expandsdd utility can be used to expand the index size of an existing SDD file. The mergesdd utility can be used to merge two SDD files into one SDD file.
The new bootstrap can be differentiated from the old format by assuming the new format and looking for a 1 at the eighth 32-bit integer. If that value is 0, then it must be the old 16-bit integer format. If that value is neither 1 or 0 then there must be a problem with the file. Individual records data (indicated by a 1 in the seventh 32-bit integer location) is currently only available for Green Bank data.
One index entry describes one scan in the scan section of the file. Each index entry occupies the number of bytes given in the bootstrap. The number of index entries depends upon the number of records in the index, the number of bytes per record, and the number of bytes per index entry. In most cases, there will be something like 1024 or 4096 index entries.
Each index entry must contain at least 64 bytes of the following information:
SDD Index Entry Structure
-------------------------------------------------------------
Word Size Description
-------------------------------------------------------------
Integer*4: Starting record number for the scan about to be
described in the index.
Integer*4: Last record number for that scan.
Real*4: Horizontal Coordinate in degrees.
Real*4: Vertical Coordinate in degrees.
Character*16: Source name.
Real*4: Scan number.
Real*4: For spectral-line scans, the frequency resolution
in MHz.
For continuum scans, the slew rate of the
telescope in degrees/sec.
Real*8: For spectral-line scans, the rest frequency of
the observation in MHz.
For continuum scans, the integration time per
point.
Real*4: LST in hours.
Real*4: UT date in YYYY.MMDD format.
Integer*2: Observing mode (Coded).
Integer*2: (Record_number*64 + Phase_number)
Integer*2: Position Code.
Integer*2: Zeros for padding out index item to the number
of bytes in each index entry.
-------------------------------------------------------------
Old SDD Index Entry Structure
-------------------------------------------------------------
Word Size Description
-------------------------------------------------------------
Integer*2: Starting record number for the scan about to be
described in the index.
Integer*2: Last record number for that scan.
Integer*2: Magic number indicating format of the data.
Integer*2: Position Code.
Real*4: Horizontal Coordinate in degrees.
Real*4: Vertical Coordinate in degrees.
Character*16: Source name.
Real*4: Scan number.
Real*4: For spectral-line scans, the frequency resolution
in MHz.
For continuum scans, the slew rate of the
telescope in degrees/sec.
Real*8: For spectral-line scans, the rest frequency of
the observation in MHz.
For continuum scans, the integration time per
point.
Real*4: LST in hours.
Real*4: UT date in YYYY.MMDD format.
Integer*2: Observing mode (Coded).
Integer*2: (Record_number*64 + Phase_number)
Integer*2: Zeros for padding out index item to the number
of bytes in each index entry.
-------------------------------------------------------------
Notes for the current SDD format and the old SDD format:
Continuum = type_code + 256 Spectral-Line = type_code + 512
The possible type_codes, and what they represent (as of the data this document was prepared), are:
SDD Observational Type Code
-------------------------------------------------------------
Type_Code Short Form Description
-------------------------------------------------------------
0 No mode present
1 PS Position Switched
2 APS Absolute Position Switched
3 FS Frequency Switched
4 BSP Beam_Switch + Position_Switch
5 TPON Total Power On
6 TPOF Total Power Off
7 ATP Absolute Total Power
8 PSM Position Switched Map
9 APM Absolute Position Switched Map
10 FSM Frequency Switched Map
11 TPMO Total Power Map On
12 TPMF Total Power Map Off
13 DRF Drift Map
14 PCAL Position Calibrate
15 BCAL Beam Calibrate
16 BLNK Blanking
17 SEQ Sequence
18 FIVE Five Point
19 MAP Continuum Map
20 FOC Focalize
21 NSFC North-South Focalize
22 TTIP Total Power Tip
23 STIP Switched Power Tip
24 DON Continuum On
25 CAL Calibration
26 FSPS Freq Switch + Position Switch
27 BSPS Beam Switch + Position Switch
28 ZERO Zero Check
29 TLPW Total Power
30 FQSW Frequency Switched
31 NOCL No Calibration
32 PLCL Pulse Cal
33 ONOF Continuum On-Off Scan
34 BMSW Nutation
35 PSSW Position Switched, Tucson, old
36 DRFT Continuum Drift Scans, Tucson
37 OTF On-the-fly, Tucson
38 SON See Tucson for an explanation
39 SOF See Tucson for an explanation
40 QK5 See Tucson for an explanation
41 QK5A See Tucson for an explanation
42 PSS1 PS flip or PS-1 mode, Tucson
43 VLBI VLBI, Tucson
44 PZC See Tucson for an explanation
45 CPZM See Tucson for an explanation
46 PSPZ PS - polarization, Tucson
47 CPZ1 See Tucson for an explanation
48 CPZ2 See Tucson for an explanation
-------------------------------------------------------------
Note: The short form of the modes is that which makes up part of the OBSMODE header word stored within the header of the SDD scan.
The scans section of the file usually has zero length when it is created. In principle, the scans section of the file can be of various formats (as described by the index entry of each scan). Usually, however, the format of the data will be SDD version 1. This section is the same for both the old and new SDD bootstrap and index formats.
Each scan in the data section can occupy more than one record. If the scan doesn't fill a record, the record is padded with zeros.
Records may be empty within the data section. Empty records are created when UniPOPS tries to overwrite a scan in the file with one that takes up less space. Also, if the scan to be written occupies more records than the scan to be overwritten, the new scan is usually tacked onto the end of the file and the old scan is overwritten with zeros. Thus, SDD files can have holes in them.
If the stored scan is in SDD (version 1) format, the scan has the following graphical format,
SDD Scan Format _________________________________________________________ | PREAMBLE |-------------------------------------------------------- | HEADER-Class 1 | -Class 2 | -Class 3 | ......... | ......... | -Class 15 |-------------------------------------------------------- | DATA | | | | | |_________________________________________________________
The preamble describes the locations in the header where the various classes begin and end, as well as the number of classes present. Its structure is,
SDD Preamble Structure ------------------------------------------------------------- Word Size Description ------------------------------------------------------------- Integer*2 Number of classes Integer*2 Starting Real*8 word for classes (15) -------------------------------------------------------------
Up to 15 classes of header words can exist in the header. A minimum of 13 are needed. Each class describes a particular aspect of the scan (pointing, mapping, etc). Each class must contain a minimum amount of information. This minimum amount must be in a certain format and must follow prescribed conventions. Extra information may exist in each class after the minimum information is provided. Each entry in each class occupies a multiple of 8 bytes. The memo, dated October 21, 1986 and entitled "Single Dish FITS tape", by Betty Stobie and Lorrie Morgan contains a lengthier, though sometimes out-of-date discussion of the uses of the various classes.
The data section contains the data values. The format, and information on what the data structure is like, is stored in the header. UniPOPS dictates an upper limit of 10240 data points but the SDD file format has no real limits.
Class 1 : Basic Information
---------------------------------------------------------------------
Description | Keyword | Size | Units
---------------------------------------------------------------------
Length of Header | HEADLEN | R*8 | Bytes
Length of Data | DATALEN | R*8 | Bytes
Scan Number | SCAN | R*8 |
Observer Initials | OBSID | C*8 |
Observer Name | OBSERVER | C*16 |
Telescope Descriptor | TELESCOP | C*8 |
Project Identification | PROJID | C*8 |
Source Name | OBJECT | C*16 |
Type of Data and Observing Mode | OBSMODE | C*8 |
Frontend Descriptor | FRONTEND | C*8 |
Backend Descriptor | BACKEND | C*8 |
Data Precision of Spectrum | PRECIS | C*8 |
Save number from which scan was retrieved| SAVENUM | R*8 |
Number of records for indiv. records scan| NORECORD | R*8 |
Or the tot. # of OTF spectra in this row
The record number for this indiv. record | RECORDID | R*8 |
Or the # of this OTF spectra
Class 2 : Pointing Parameters
---------------------------------------------------------------------
Description | Keyword | Size | Units
---------------------------------------------------------------------
Total Az/RA Pointing Correction | XPOINT | R*8 | arcsec
Total El/Dec Pointing Correction | YPOINT | R*8 | arcsec
User Az/RA Pointing Correction | UXPNT | R*8 | arcsec
User El/Dec Pointing Correction | UYPNT | R*8 | arcsec
Pointing Constants(1) | PTCON | R*8 | arcmin
Pointing Constants(2) | PTCON+1 | R*8 | arcmin
Pointing Constants(3) | PTCON+2 | R*8 | arcmin
Pointing Constants(4) | PTCON+3 | R*8 | arcmin
Receiver Box or Secondary Orientation | ORIENT | R*8 | Degrees
Radial Focus | FOCUSR | R*8 | mm
North-South Focus | FOCUSV | R*8 | mm
East-West Focus | FOCUSH | R*8 | mm
Pointing Model | PT_MODEL | C*8 |
Class 3 : Observing Parameters
---------------------------------------------------------------------
Description | Keyword | Size | Units
---------------------------------------------------------------------
Universal Time Date | UTDATE | R*8 | YYYY.MMDD
Universal Time | UT | R*8 | Hours
LST | LST | R*8 | Hours
Number of Receiver Channels | NORCHAN | R*8 |
Number of Switching Variables | NOSWVAR | R*8 |
Number of Phases per Cycle | NOPHASE | R*8 |
Length of Cycle | CYCLLEN | R*8 | Seconds
Length of Sample | SAMPRAT | R*8 | Seconds
Class 11 type | CL11TYPE | C*8 |
The phase number of this indiv. scan | PHASEID | R*8 |
Class 4 : Positions
---------------------------------------------------------------------
Description | Keyword | Size | Units
---------------------------------------------------------------------
Epoch | EPOCH | R*8 | Years
Commanded Source X | XSOURCE | R*8 | Degrees
Commanded Source Y | YSOURCE | R*8 | Degrees
Commanded Reference X | XREF | R*8 | Degrees
Commanded Reference Y | YREF | R*8 | Degrees
Commanded Epoch Right Ascension | EPOCRA | R*8 | Degrees
Commanded Epoch Declination | EPOCDEC | R*8 | Degrees
Commanded Galactic Longitude | GALLONG | R*8 | Degrees
Commanded Galactic Latitude | GALLAT | R*8 | Degrees
Commanded Azimuth | AZ | R*8 | Degrees
Commanded Elevation | EL | R*8 | Degrees
Indicated X Position | INDX | R*8 | Degrees
Indicated Y Position | INDY | R*8 | Degrees
Descriptive Origin(1) | DESORG | R*8 | Degrees
Descriptive Origin(2) | DESORG+1 | R*8 | Degrees
Descriptive Origin(3) | DESORG+2 | R*8 | Degrees
Coordinate System Code | COORDCD | C*8 |
Class 5 : Environment
---------------------------------------------------------------------
Description | Keyword | Size | Units
---------------------------------------------------------------------
Ambient Temperature | TAMB | R*8 | C
Ambient Pressure | PRESSURE | R*8 | cm-Hg
Ambient relative humidity | HUMIDITY | R*8 | %
Index of Refraction | REFRAC | R*8 |
Dew Point | DEWPT | R*8 | C
Mm of Water | MMH2O | R*8 | mm
Class 6 : Map Parameters
---------------------------------------------------------------------
Description | Keyword | Size | Units
---------------------------------------------------------------------
Map Scanning Angle | SCANANG | R*8 | Degrees
X Position at Map Reference Position Zero| XZERO | R*8 | Degrees
Y Position at Map Reference Position Zero| YZERO | R*8 | Degrees
Delta X or X Rate | DELTAXR | R*8 | (arcsec or
Delta Y or Y Rate | DELTAYR | R*8 | arcsec/sec)
Number of Grid Points | NOPTS | R*8 |
Number of X Grid Points | NOXPTS | R*8 |
Number of Y Grid Points | NOYPTS | R*8 |
Starting X Grid Cell Number | XCELL0 | R*8 |
Starting Y Grid Cell Number | YCELL0 | R*8 |
XY Reference Frame Code | FRAME | C*8 |
Class 7 : Data Parameters
---------------------------------------------------------------------
Description | Keyword | Size | Units
---------------------------------------------------------------------
Beam Halfwidth at Half Maximum | BFWHM | R*8 | arcsec
Off Scan Number | OFFSCAN | R*8 |
Bad Channel Value | BADCHV | R*8 | K
Velocity Correction | RVSYS | R*8 | km/sec
Velocity with respect to Reference | VELOCITY | R*8 | km/sec
Velocity Definition and Reference | VELDEF | C*8 |
Type of Calibration | TYPECAL | C*8 |
Class 8 : Engineering Parameters
---------------------------------------------------------------------
Description | Keyword | Size | Units
---------------------------------------------------------------------
Antenna Aperture Efficiency | APPEFF | R*8 |
Antenna Beam Efficiency | BEAMEFF | R*8 |
Antenna Gain | ANTGAIN | R*8 |
Rear Spill & Scat Efficiency | ETAL | R*8 |
Forward Spill & Scat Efficiency | ETAFSS | R*8 |
Class 9 : Telescope Dependent Parameters- Green Bank
------------------------------------------------------------------------
Description | Keyword | Size | Units
------------------------------------------------------------------------
L1 | L1 | R*8 | MHz
L1F1 | L1F1 | R*8 | MHz
L1F2 | L1F2 | R*8 | MHz
L2 | L2 | R*8 | MHz
L2F1 | L2F1 | R*8 | MHz
L2F2 | L2F2 | R*8 | MHz
LA | LA | R*8 | MHz
LB | LB | R*8 | MHz
LC | LC | R*8 | MHz
LD | LD | R*8 | MHz
Level Correction | LEVCORR | R*8 | Volts
Pointing Fudge(1) | PTFUDGE | R*8 | arcmin
Pointing Fudge(2) | PTFUDGE+1| R*8 | arcmin
RHO | RHO | R*8 | Degree
THETA | THETA | R*8 | Degree
Center Frequency Formula | CFFORM | C*24 |
Class 9 : Telescope Dependent Parameters- Tucson 12 M
------------------------------------------------------------------------
Description | Keyword | Size | Units
------------------------------------------------------------------------
Synthesizer Frequency | SYNFREQ | R*8 | MHz
LO Factor | LOFACT | R*8 |
Harmonic | HARMONIC | R*8 |
LOIF | LOIF | R*8 | MHz
First IF | FIRSTIF | R*8 | MHz
Reference Azimuth Offset | RAZOFF | R*8 | arcsec
Reference Elevation Offset | RELOFF | R*8 | arcsec
Beam Throw | BMTHROW | R*8 | arcsec
Beam Orientation | BMORENT | R*8 | Degree
Baseline Offset | BASEOFF | R*8 | K
Observing Tolerance | OBSTOL | R*8 | arcsec
Sideband | SIDEBAND | R*8 |
Wavelength | WL | R*8 | mm
GAIN Scan Number | GAINS | R*8 |
+ Beam (1) | PBEAM | R*8 | arcsec
+ Beam (2) | PBEAM+1 | R*8 | arcsec
- Beam (1) | MBEAM | R*8 | arcsec
- Beam (2) | MBEAM+1 | R*8 | arcsec
RA/Dec Offsets (1) | SROFF | R*8 | arcsec
RA/Dec Offsets (2) | SROFF+1 | R*8 | arcsec
RA/Dec Offsets (3) | SROFF+2 | R*8 | arcsec
RA/Dec Offsets (4) | SROFF+3 | R*8 | arcsec
Frequency switching signal offset | FOFFSIG | R*8 | MHz
Frequency switching reference offset 1 | FOFFREF1 | R*8 | MHz
Frequency switching reference offset 2 | FOFFREF2 | R*8 | MHz
Class 10 : Open Parameters (Data Reduction)
----------------------------------------------------------------------
Description | Keyword | Size | Units
----------------------------------------------------------------------
Open Parameter Values(1) | OPENPAR | C*8
Open Parameter Values(2) | OPENPAR+1| C*8
Open Parameter Values(3) | OPENPAR+2| C*8
Open Parameter Values(4) | OPENPAR+3| C*8
Open Parameter Values(5) | OPENPAR+4| C*8
Open Parameter Values(6) | OPENPAR+5| C*8
Open Parameter Values(7) | OPENPAR+6| C*8
Open Parameter Values(8) | OPENPAR+7| C*8
Open Parameter Values(9) | OPENPAR+8| C*8
Open Parameter Values(10) | OPENPAR+9| C*8
Class 11: Phase Block -- CL11TYPE = "ORIG " or unset -- Original
----------------------------------------------------------------------
Description | Keyword | Size | Units
----------------------------------------------------------------------
Variable Value | VARVAL | R*8
Variable Descriptor | VARDES | C*8
Phase Table | PHASTB | C*8
Class 11: Phase Block -- CL11TYPE = "PROTO12M" -- 12-m Prototype
---------------------------------------------------------------------
Description | Keyword | Size | Units
---------------------------------------------------------------------
Number of Fast Switching Variables | NOSWVARF | R*8 |
Number of Slow Cycles per Scan | NUMCYC | R*8 |
Number of Fast Cycles per Scan | NUMCYCF | R*8 |
Number of Fast Phases per Cycle | NOPHASEF | R*8 |
Length of Fast Cycle | CYCLENF | R*8 | Seconds
Length of Fast Phase Sample | SAMPTIMF | R*8 | Seconds
Variable Value nn | VARVALnn | R*8 |
Variable Descriptor nn | VARDESnn | C*8 |
Phase Table nn | PHASTBnn | C*32 |
(last 3 repeat (NOSWVAR+NOSWVARF) times)
(nn is currently limited to 01 through 10)
Class 12 : Descriptor Block for Each Receiver Channel
------------------------------------------------------------------
Description | Keyword | Size | Units
------------------------------------------------------------------
Observed Frequency | OBSFREQ | R*8 | MHz
Rest Frequency | RESTFREQ | R*8 | MHz
Frequency Resolution or Scale Fcator | FREQRES | R*8 | MHz or
| | | K/Count
Bandwidth | BW | R*8 | MHz
Receiver Temperature | TRX | R*8 | K
Calibration Temperature | TCAL | R*8 | K
Source System Temperature | STSYS | R*8 | K
Reference System Temperature | RTSYS | R*8 | K
Source Temperature | TSOURCE | R*8 | K
RMS of Mean | TRMS | R*8 | K
Reference Point Number | REFPT | R*8 |
X Value at the Reference Point | X0 | R*8 | km/sec or
| | | degrees
Delta X | DELTAX | R*8 | km/sec
| | | degrees
Total Integration Time | INTTIME | R*8 | seconds
Number of Integrations | NOINT | R*8 |
Starting Point Number | SPN | R*8 |
H2O Opacity | TAUH2O | R*8 |
H2O Temperature | TH2O | R*8 | K
O2 Opacity | TAUO2 | R*8 |
O2 Temperature | TO2 | R*8 | K
Polarization | POLARIZ | C*8 |
Effective Integration Time | EFFINT | R*8 |
Receiver Information | RX_INFO | C*16 |
Class 13: Reduction Parameters
------------------------------------------------------------------
Description | Keyword | Size | Units
------------------------------------------------------------------
Number of scans stacked | NOSTAC | R*8 |
First Scan in stack | FSCAN | R*8 |
Last Scan in stack | LSCAN | R*8 |
Line Amplitude | LAMP | R*8 | K
Line Width | LWID | R*8 | km/sec
Integrated Line Intensity | ILI | R*8 | K km/sec
RMS Noise | RMS | R*8 | K
Go to the previous, next section.
Webmaster: Ronald J. Maddalena