Go to the previous, next section.
The native format of UniPOPS for storing and retrieving data is called SDD. The file consists of three parts: bootstrap, index, and scans, and is a direct-access file. The file has the following layout,
SDD File Format _________________________________________________________ | BOOTSTRAP |-------------------------------------------------------- | INDEX | | | | |-------------------------------------------------------- | SCANS | | | | | |_________________________________________________________
The binary representation used in an SDD file is that of the IEEE standard. The checkfile.exe utility will list the contents of the bootstrap and index section of the file while makeindex.exe will reconstruct the index and some of the items in the bootstrap part of the file by looking at the scan section of the file. The makefile.exe utility will create an empty SDD file with a specified size.
As of the fall of 1993, a new bootstrap and index format are in use. All of the primary UniPOPS programs are able to read both the old and the new forms transparently to the user. The new format primarily involves switching from 16-bit integers to 32-bit integers. The makeindex.exe utility can be used to generate a new bootstrap and index for an old-format SDD file. The makeoldindex.exe utility can be used to generate an old-format bootstrap and index.
The bootstrap occupies one record. The structure and contents of the bootstrap record is:
SDD Bootstrap Structure ------------------------------------------------------------- Word Size Description ------------------------------------------------------------- Integer*4: Number of records in index including the bootstrap record. Integer*4: Number of data records which follow index records. Integer*4: Bytes per record. Integer*4: Number of bytes per index entry. Integer*4: Number of index entries already used. Integer*4: Counter which indicates whether the file has been updated. Integer*4 Type of SDD file (0 if data, 1 if individual records) Integer*4 SDD version number (this is version 1) Integer*4: Zeros to pad out the record to its full size. -------------------------------------------------------------
Old SDD Bootstrap Structure ------------------------------------------------------------- Word Size Description ------------------------------------------------------------- Integer*2: Number of records in index including the bootstrap record. Integer*2: Number of data records which follow index records. Integer*2: Bytes per record. Integer*2: Number of bytes per index entry. Integer*2: Number of index entries already used. Integer*2: Counter which indicates whether the file has been updated. Integer*4: Zeros to pad out the record to its full size. -------------------------------------------------------------
The actual number of bytes in each record of the file can be different for each file. Typically, it is 512 bytes per record and all records in the file have the same size.
Each scan in the scan section of the file is summarized by an entry in the index part of the file. The bootstrap gives the number of bytes occupied by each index entry and has a minimum value of 64 bytes. The number of bytes in each record of the file must be evenly divisible by the number of bytes in each index entry.
As scans are stored in the file, the number of index items already used will give the number of the highest index entry that describes a scan. It need not be the entry describing the last scan written to the file and not all index entries lower than that one need be filled (i.e., index entries need not be filled sequentially). This number is zero for an empty SDD file. Also, as scans are stored in the file, the number of records in the data section of the file is stored in the bootstrap as well. This entry in the bootstrap is zero for an empty SDD file.
If there are N-1 index records (so that there are N records in the bootstrap plus index), and if each index entry takes up M bytes, and if the file has L byte records, the number of scans which can be stored in the file will be: (N-1)*L / M. Typically, N is 129 or 513, so the number of scans which can be stored in an SDD format is 1024 or 4096. The number of records in the index thus dictates the maximum number of scans which the index can describe and the limit on how many scans can be stored in a file. When an SDD file is created (with the makefile.exe utility), the user can specify the maximum number of scans which the file will contain. The expandsdd utility can be used to expand the index size of an existing SDD file. The mergesdd utility can be used to merge two SDD files into one SDD file.
The new bootstrap can be differentiated from the old format by assuming the new format and looking for a 1 at the eighth 32-bit integer. If that value is 0, then it must be the old 16-bit integer format. If that value is neither 1 or 0 then there must be a problem with the file. Individual records data (indicated by a 1 in the seventh 32-bit integer location) is currently only available for Green Bank data.
One index entry describes one scan in the scan section of the file. Each index entry occupies the number of bytes given in the bootstrap. The number of index entries depends upon the number of records in the index, the number of bytes per record, and the number of bytes per index entry. In most cases, there will be something like 1024 or 4096 index entries.
Each index entry must contain at least 64 bytes of the following information:
SDD Index Entry Structure ------------------------------------------------------------- Word Size Description ------------------------------------------------------------- Integer*4: Starting record number for the scan about to be described in the index. Integer*4: Last record number for that scan. Real*4: Horizontal Coordinate in degrees. Real*4: Vertical Coordinate in degrees. Character*16: Source name. Real*4: Scan number. Real*4: For spectral-line scans, the frequency resolution in MHz. For continuum scans, the slew rate of the telescope in degrees/sec. Real*8: For spectral-line scans, the rest frequency of the observation in MHz. For continuum scans, the integration time per point. Real*4: LST in hours. Real*4: UT date in YYYY.MMDD format. Integer*2: Observing mode (Coded). Integer*2: (Record_number*64 + Phase_number) Integer*2: Position Code. Integer*2: Zeros for padding out index item to the number of bytes in each index entry. -------------------------------------------------------------
Old SDD Index Entry Structure ------------------------------------------------------------- Word Size Description ------------------------------------------------------------- Integer*2: Starting record number for the scan about to be described in the index. Integer*2: Last record number for that scan. Integer*2: Magic number indicating format of the data. Integer*2: Position Code. Real*4: Horizontal Coordinate in degrees. Real*4: Vertical Coordinate in degrees. Character*16: Source name. Real*4: Scan number. Real*4: For spectral-line scans, the frequency resolution in MHz. For continuum scans, the slew rate of the telescope in degrees/sec. Real*8: For spectral-line scans, the rest frequency of the observation in MHz. For continuum scans, the integration time per point. Real*4: LST in hours. Real*4: UT date in YYYY.MMDD format. Integer*2: Observing mode (Coded). Integer*2: (Record_number*64 + Phase_number) Integer*2: Zeros for padding out index item to the number of bytes in each index entry. -------------------------------------------------------------
Notes for the current SDD format and the old SDD format:
Continuum = type_code + 256 Spectral-Line = type_code + 512
The possible type_codes, and what they represent (as of the data this document was prepared), are:
SDD Observational Type Code ------------------------------------------------------------- Type_Code Short Form Description ------------------------------------------------------------- 0 No mode present 1 PS Position Switched 2 APS Absolute Position Switched 3 FS Frequency Switched 4 BSP Beam_Switch + Position_Switch 5 TPON Total Power On 6 TPOF Total Power Off 7 ATP Absolute Total Power 8 PSM Position Switched Map 9 APM Absolute Position Switched Map 10 FSM Frequency Switched Map 11 TPMO Total Power Map On 12 TPMF Total Power Map Off 13 DRF Drift Map 14 PCAL Position Calibrate 15 BCAL Beam Calibrate 16 BLNK Blanking 17 SEQ Sequence 18 FIVE Five Point 19 MAP Continuum Map 20 FOC Focalize 21 NSFC North-South Focalize 22 TTIP Total Power Tip 23 STIP Switched Power Tip 24 DON Continuum On 25 CAL Calibration 26 FSPS Freq Switch + Position Switch 27 BSPS Beam Switch + Position Switch 28 ZERO Zero Check 29 TLPW Total Power 30 FQSW Frequency Switched 31 NOCL No Calibration 32 PLCL Pulse Cal 33 ONOF Continuum On-Off Scan 34 BMSW Nutation 35 PSSW Position Switched, Tucson, old 36 DRFT Continuum Drift Scans, Tucson 37 OTF On-the-fly, Tucson 38 SON See Tucson for an explanation 39 SOF See Tucson for an explanation 40 QK5 See Tucson for an explanation 41 QK5A See Tucson for an explanation 42 PSS1 PS flip or PS-1 mode, Tucson 43 VLBI VLBI, Tucson 44 PZC See Tucson for an explanation 45 CPZM See Tucson for an explanation 46 PSPZ PS - polarization, Tucson 47 CPZ1 See Tucson for an explanation 48 CPZ2 See Tucson for an explanation -------------------------------------------------------------
Note: The short form of the modes is that which makes up part of the OBSMODE header word stored within the header of the SDD scan.
The scans section of the file usually has zero length when it is created. In principle, the scans section of the file can be of various formats (as described by the index entry of each scan). Usually, however, the format of the data will be SDD version 1. This section is the same for both the old and new SDD bootstrap and index formats.
Each scan in the data section can occupy more than one record. If the scan doesn't fill a record, the record is padded with zeros.
Records may be empty within the data section. Empty records are created when UniPOPS tries to overwrite a scan in the file with one that takes up less space. Also, if the scan to be written occupies more records than the scan to be overwritten, the new scan is usually tacked onto the end of the file and the old scan is overwritten with zeros. Thus, SDD files can have holes in them.
If the stored scan is in SDD (version 1) format, the scan has the following graphical format,
SDD Scan Format _________________________________________________________ | PREAMBLE |-------------------------------------------------------- | HEADER-Class 1 | -Class 2 | -Class 3 | ......... | ......... | -Class 15 |-------------------------------------------------------- | DATA | | | | | |_________________________________________________________
The preamble describes the locations in the header where the various classes begin and end, as well as the number of classes present. Its structure is,
SDD Preamble Structure ------------------------------------------------------------- Word Size Description ------------------------------------------------------------- Integer*2 Number of classes Integer*2 Starting Real*8 word for classes (15) -------------------------------------------------------------
Up to 15 classes of header words can exist in the header. A minimum of 13 are needed. Each class describes a particular aspect of the scan (pointing, mapping, etc). Each class must contain a minimum amount of information. This minimum amount must be in a certain format and must follow prescribed conventions. Extra information may exist in each class after the minimum information is provided. Each entry in each class occupies a multiple of 8 bytes. The memo, dated October 21, 1986 and entitled "Single Dish FITS tape", by Betty Stobie and Lorrie Morgan contains a lengthier, though sometimes out-of-date discussion of the uses of the various classes.
The data section contains the data values. The format, and information on what the data structure is like, is stored in the header. UniPOPS dictates an upper limit of 10240 data points but the SDD file format has no real limits.
Class 1 : Basic Information --------------------------------------------------------------------- Description | Keyword | Size | Units --------------------------------------------------------------------- Length of Header | HEADLEN | R*8 | Bytes Length of Data | DATALEN | R*8 | Bytes Scan Number | SCAN | R*8 | Observer Initials | OBSID | C*8 | Observer Name | OBSERVER | C*16 | Telescope Descriptor | TELESCOP | C*8 | Project Identification | PROJID | C*8 | Source Name | OBJECT | C*16 | Type of Data and Observing Mode | OBSMODE | C*8 | Frontend Descriptor | FRONTEND | C*8 | Backend Descriptor | BACKEND | C*8 | Data Precision of Spectrum | PRECIS | C*8 | Save number from which scan was retrieved| SAVENUM | R*8 | Number of records for indiv. records scan| NORECORD | R*8 | Or the tot. # of OTF spectra in this row The record number for this indiv. record | RECORDID | R*8 | Or the # of this OTF spectra Class 2 : Pointing Parameters --------------------------------------------------------------------- Description | Keyword | Size | Units --------------------------------------------------------------------- Total Az/RA Pointing Correction | XPOINT | R*8 | arcsec Total El/Dec Pointing Correction | YPOINT | R*8 | arcsec User Az/RA Pointing Correction | UXPNT | R*8 | arcsec User El/Dec Pointing Correction | UYPNT | R*8 | arcsec Pointing Constants(1) | PTCON | R*8 | arcmin Pointing Constants(2) | PTCON+1 | R*8 | arcmin Pointing Constants(3) | PTCON+2 | R*8 | arcmin Pointing Constants(4) | PTCON+3 | R*8 | arcmin Receiver Box or Secondary Orientation | ORIENT | R*8 | Degrees Radial Focus | FOCUSR | R*8 | mm North-South Focus | FOCUSV | R*8 | mm East-West Focus | FOCUSH | R*8 | mm Pointing Model | PT_MODEL | C*8 | Class 3 : Observing Parameters --------------------------------------------------------------------- Description | Keyword | Size | Units --------------------------------------------------------------------- Universal Time Date | UTDATE | R*8 | YYYY.MMDD Universal Time | UT | R*8 | Hours LST | LST | R*8 | Hours Number of Receiver Channels | NORCHAN | R*8 | Number of Switching Variables | NOSWVAR | R*8 | Number of Phases per Cycle | NOPHASE | R*8 | Length of Cycle | CYCLLEN | R*8 | Seconds Length of Sample | SAMPRAT | R*8 | Seconds Class 11 type | CL11TYPE | C*8 | The phase number of this indiv. scan | PHASEID | R*8 | Class 4 : Positions --------------------------------------------------------------------- Description | Keyword | Size | Units --------------------------------------------------------------------- Epoch | EPOCH | R*8 | Years Commanded Source X | XSOURCE | R*8 | Degrees Commanded Source Y | YSOURCE | R*8 | Degrees Commanded Reference X | XREF | R*8 | Degrees Commanded Reference Y | YREF | R*8 | Degrees Commanded Epoch Right Ascension | EPOCRA | R*8 | Degrees Commanded Epoch Declination | EPOCDEC | R*8 | Degrees Commanded Galactic Longitude | GALLONG | R*8 | Degrees Commanded Galactic Latitude | GALLAT | R*8 | Degrees Commanded Azimuth | AZ | R*8 | Degrees Commanded Elevation | EL | R*8 | Degrees Indicated X Position | INDX | R*8 | Degrees Indicated Y Position | INDY | R*8 | Degrees Descriptive Origin(1) | DESORG | R*8 | Degrees Descriptive Origin(2) | DESORG+1 | R*8 | Degrees Descriptive Origin(3) | DESORG+2 | R*8 | Degrees Coordinate System Code | COORDCD | C*8 | Class 5 : Environment --------------------------------------------------------------------- Description | Keyword | Size | Units --------------------------------------------------------------------- Ambient Temperature | TAMB | R*8 | C Ambient Pressure | PRESSURE | R*8 | cm-Hg Ambient relative humidity | HUMIDITY | R*8 | % Index of Refraction | REFRAC | R*8 | Dew Point | DEWPT | R*8 | C Mm of Water | MMH2O | R*8 | mm Class 6 : Map Parameters --------------------------------------------------------------------- Description | Keyword | Size | Units --------------------------------------------------------------------- Map Scanning Angle | SCANANG | R*8 | Degrees X Position at Map Reference Position Zero| XZERO | R*8 | Degrees Y Position at Map Reference Position Zero| YZERO | R*8 | Degrees Delta X or X Rate | DELTAXR | R*8 | (arcsec or Delta Y or Y Rate | DELTAYR | R*8 | arcsec/sec) Number of Grid Points | NOPTS | R*8 | Number of X Grid Points | NOXPTS | R*8 | Number of Y Grid Points | NOYPTS | R*8 | Starting X Grid Cell Number | XCELL0 | R*8 | Starting Y Grid Cell Number | YCELL0 | R*8 | XY Reference Frame Code | FRAME | C*8 | Class 7 : Data Parameters --------------------------------------------------------------------- Description | Keyword | Size | Units --------------------------------------------------------------------- Beam Halfwidth at Half Maximum | BFWHM | R*8 | arcsec Off Scan Number | OFFSCAN | R*8 | Bad Channel Value | BADCHV | R*8 | K Velocity Correction | RVSYS | R*8 | km/sec Velocity with respect to Reference | VELOCITY | R*8 | km/sec Velocity Definition and Reference | VELDEF | C*8 | Type of Calibration | TYPECAL | C*8 | Class 8 : Engineering Parameters --------------------------------------------------------------------- Description | Keyword | Size | Units --------------------------------------------------------------------- Antenna Aperture Efficiency | APPEFF | R*8 | Antenna Beam Efficiency | BEAMEFF | R*8 | Antenna Gain | ANTGAIN | R*8 | Rear Spill & Scat Efficiency | ETAL | R*8 | Forward Spill & Scat Efficiency | ETAFSS | R*8 | Class 9 : Telescope Dependent Parameters- Green Bank ------------------------------------------------------------------------ Description | Keyword | Size | Units ------------------------------------------------------------------------ L1 | L1 | R*8 | MHz L1F1 | L1F1 | R*8 | MHz L1F2 | L1F2 | R*8 | MHz L2 | L2 | R*8 | MHz L2F1 | L2F1 | R*8 | MHz L2F2 | L2F2 | R*8 | MHz LA | LA | R*8 | MHz LB | LB | R*8 | MHz LC | LC | R*8 | MHz LD | LD | R*8 | MHz Level Correction | LEVCORR | R*8 | Volts Pointing Fudge(1) | PTFUDGE | R*8 | arcmin Pointing Fudge(2) | PTFUDGE+1| R*8 | arcmin RHO | RHO | R*8 | Degree THETA | THETA | R*8 | Degree Center Frequency Formula | CFFORM | C*24 | Class 9 : Telescope Dependent Parameters- Tucson 12 M ------------------------------------------------------------------------ Description | Keyword | Size | Units ------------------------------------------------------------------------ Synthesizer Frequency | SYNFREQ | R*8 | MHz LO Factor | LOFACT | R*8 | Harmonic | HARMONIC | R*8 | LOIF | LOIF | R*8 | MHz First IF | FIRSTIF | R*8 | MHz Reference Azimuth Offset | RAZOFF | R*8 | arcsec Reference Elevation Offset | RELOFF | R*8 | arcsec Beam Throw | BMTHROW | R*8 | arcsec Beam Orientation | BMORENT | R*8 | Degree Baseline Offset | BASEOFF | R*8 | K Observing Tolerance | OBSTOL | R*8 | arcsec Sideband | SIDEBAND | R*8 | Wavelength | WL | R*8 | mm GAIN Scan Number | GAINS | R*8 | + Beam (1) | PBEAM | R*8 | arcsec + Beam (2) | PBEAM+1 | R*8 | arcsec - Beam (1) | MBEAM | R*8 | arcsec - Beam (2) | MBEAM+1 | R*8 | arcsec RA/Dec Offsets (1) | SROFF | R*8 | arcsec RA/Dec Offsets (2) | SROFF+1 | R*8 | arcsec RA/Dec Offsets (3) | SROFF+2 | R*8 | arcsec RA/Dec Offsets (4) | SROFF+3 | R*8 | arcsec Frequency switching signal offset | FOFFSIG | R*8 | MHz Frequency switching reference offset 1 | FOFFREF1 | R*8 | MHz Frequency switching reference offset 2 | FOFFREF2 | R*8 | MHz Class 10 : Open Parameters (Data Reduction) ---------------------------------------------------------------------- Description | Keyword | Size | Units ---------------------------------------------------------------------- Open Parameter Values(1) | OPENPAR | C*8 Open Parameter Values(2) | OPENPAR+1| C*8 Open Parameter Values(3) | OPENPAR+2| C*8 Open Parameter Values(4) | OPENPAR+3| C*8 Open Parameter Values(5) | OPENPAR+4| C*8 Open Parameter Values(6) | OPENPAR+5| C*8 Open Parameter Values(7) | OPENPAR+6| C*8 Open Parameter Values(8) | OPENPAR+7| C*8 Open Parameter Values(9) | OPENPAR+8| C*8 Open Parameter Values(10) | OPENPAR+9| C*8 Class 11: Phase Block -- CL11TYPE = "ORIG " or unset -- Original ---------------------------------------------------------------------- Description | Keyword | Size | Units ---------------------------------------------------------------------- Variable Value | VARVAL | R*8 Variable Descriptor | VARDES | C*8 Phase Table | PHASTB | C*8 Class 11: Phase Block -- CL11TYPE = "PROTO12M" -- 12-m Prototype --------------------------------------------------------------------- Description | Keyword | Size | Units --------------------------------------------------------------------- Number of Fast Switching Variables | NOSWVARF | R*8 | Number of Slow Cycles per Scan | NUMCYC | R*8 | Number of Fast Cycles per Scan | NUMCYCF | R*8 | Number of Fast Phases per Cycle | NOPHASEF | R*8 | Length of Fast Cycle | CYCLENF | R*8 | Seconds Length of Fast Phase Sample | SAMPTIMF | R*8 | Seconds Variable Value nn | VARVALnn | R*8 | Variable Descriptor nn | VARDESnn | C*8 | Phase Table nn | PHASTBnn | C*32 | (last 3 repeat (NOSWVAR+NOSWVARF) times) (nn is currently limited to 01 through 10) Class 12 : Descriptor Block for Each Receiver Channel ------------------------------------------------------------------ Description | Keyword | Size | Units ------------------------------------------------------------------ Observed Frequency | OBSFREQ | R*8 | MHz Rest Frequency | RESTFREQ | R*8 | MHz Frequency Resolution or Scale Fcator | FREQRES | R*8 | MHz or | | | K/Count Bandwidth | BW | R*8 | MHz Receiver Temperature | TRX | R*8 | K Calibration Temperature | TCAL | R*8 | K Source System Temperature | STSYS | R*8 | K Reference System Temperature | RTSYS | R*8 | K Source Temperature | TSOURCE | R*8 | K RMS of Mean | TRMS | R*8 | K Reference Point Number | REFPT | R*8 | X Value at the Reference Point | X0 | R*8 | km/sec or | | | degrees Delta X | DELTAX | R*8 | km/sec | | | degrees Total Integration Time | INTTIME | R*8 | seconds Number of Integrations | NOINT | R*8 | Starting Point Number | SPN | R*8 | H2O Opacity | TAUH2O | R*8 | H2O Temperature | TH2O | R*8 | K O2 Opacity | TAUO2 | R*8 | O2 Temperature | TO2 | R*8 | K Polarization | POLARIZ | C*8 | Effective Integration Time | EFFINT | R*8 | Receiver Information | RX_INFO | C*16 | Class 13: Reduction Parameters ------------------------------------------------------------------ Description | Keyword | Size | Units ------------------------------------------------------------------ Number of scans stacked | NOSTAC | R*8 | First Scan in stack | FSCAN | R*8 | Last Scan in stack | LSCAN | R*8 | Line Amplitude | LAMP | R*8 | K Line Width | LWID | R*8 | km/sec Integrated Line Intensity | ILI | R*8 | K km/sec RMS Noise | RMS | R*8 | K
Go to the previous, next section.
Webmaster: Ronald J. Maddalena