Go to the previous, next section.

Data Access and Manipulation

This chapter tells you about UniPOPS data. It describes the data files available to the UniPOPS user, and their access and manipulation. You will be told how to find which files and scans are currently available to you and how to change or delete the files, or create new files. You will find out how to change the project code for which you can currently access data. The chapter will also describe the operations of reading data from a disk file to a UniPOPS working array, moving and copying it between working arrays, and writing it out to an appropriate disk file. Access to the header parameters of a scan, and its individual data values, are described in Sections 9.3 - 9.5. This Chapter will concentrate on simple one-dimensional spectra and scans. The user who is interested in two- or three-dimensional data (Matrices and Arrays) should consult Chapter 16.

Data Files, their Access and Manipulation

Presently, the UniPOPS program has simultaneous access to six data files on disk. These files all hold data scans (often referred to in this COOKBOOK as "spectra") in identical formats.

If you are running gbline or gbcondar, the program will attempt to attach the Green-Bank "on-line data" file, while tucline and tuccondar will attempt to attach the Tucson "on-line" files. If you are running cvline or cvcondar, no attempt will be made to attach any "on-line" file.

The "off-line", "save" and "keep" files are interchangeable between themselves, if the user so desires. Thus, yesterday's "save" file can become today's "off-line data" file, and so on. Their maximum file size is site dependent, but it is usually 1024 scans, although bigger files can be easily accommodated (see Appendix D-9).

If any of these six files are absent, or undefined by user choice, the only effect on the program will be an inability to use the verbs that access that particular file.

The files are,

"on-line data"

The "on-line data" file available to any user with either direct, or network, access to the telescope computers at NRAO Green Bank or Tucson. This file contains the observational data taken most recently with the telescope. Though the actual name, size, and location of the file is site dependent, this should be transparent to the user. This file is READ ONLY. At Green Bank, when the user enters their project code at login, access within the "on-line" file is restricted to just the scans taken for that project. However, the project-of-interest can be changed at will via the verb CHNGPRJ (see Section 5.4.1).

Scans in this file (and the "off-line data" file) are accessed by the scan number, plus the receiver (or feed) number using the scheme,

		scan no. + (rx no. / 100.)

If the scan number alone is used, the first receiver (or feed) is assumed.

"off-line data"

The "off-line data" file contains scans which the user has previously stored in it using the KEEP or SAVE facilities (see Sections 5.9 and 5.10), or has imported from tape. This file can have a user-selected name, (the UniPOPS defaults are LDATA and CDATA, for LINE and CONDAR respectively), and can be changed at any time using the verb CHNGFILE (see Section 5.3). This file is also READ ONLY and scan access is by the same convention as for the "on-line" file (see below for how ambiguities between these two files are resolved during access.)

"save" file

The "save" file is a file where the user can READ and WRITE (or OVERWRITE) scans. It can have any user-selected name, (UniPOPS default LSAVE, or CSAVE), and the "save" file to be accessed can again be changed by the user via CHNGFILE. Reading and writing to this file is via the pair of verbs, SAVE and RECALL (see Section 5.9). The "save" file can be used to store scans temporarily for later recall and further processing.

"keep" file

The "keep" file is a file where the user can READ and WRITE (or OVERWRITE) scans. It can have any user-selected name, (UniPOPS default LKEEP, or CKEEP), and the current "keep" file can again be changed by the user via CHNGFILE. Spectra are written to and read from this file using the verbs KEEP and KGET, respectively (see Section 5.10). Since two scans with the same scan number cannot be stored in the "keep" file, the file is most often used for storing scans that have been fully reduced.

"gain" file

The off-line "gains" file, which is relevant only to Tucson data calibration. This file is READ ONLY, and can have any user-selected name (UniPOPS default LGAINS). The currently-accessed file can be changed via CHNGFILE.

"records" file

The off-line "records" file, which is relevant only to Green Bank data. This file contains individual records data (which can be created using the cvt.tele-recs utility). This file is READ ONLY, and can have any user-selected name (UniPOPS default RDATA). The currently-accessed file can be changed via CHNGFILE. The file must be in the individual records flavor of the UniPOPS data format.

Listing the Current Data Files

When you start up UniPOPS, you will be shown a list of the currently-attached external files (see Section 2.2). The first six in this list are the data files, just described. This list of current files can always be reviewed using the pseudo verb FILES, which must always be the only command on the line, i.e.,

	>FILES

Changing the Current Data Files

To change any of the above files, (except the "on-line data" file), or the "printout", or "command-logging" files (see Sections 12.3 and 16.3), the pseudo verb CHNGFILE should be used. This must be the only command on the line. The syntax for CHNGFILE is,

	>CHNGFILE   action-type   file-interface-id   [filename]

NOTE : The filename attribute is not required if the "action-type" is SUBTRACT.

The action-types available are the following (these can be literal strings or string adverbs, they should NOT be enclosed in quotes):

	CREATE   - create the disk file "filename" and attach it to the
		   specified file-interface, replacing the existing
		   input.

	SUBTRACT - remove the file currently attached to the specified
		   file-interface, leaving the interface unattached.

	CHANGE   - replace the current file attached to the specified
		   file-interface, with the existing disk file
		   "filename".

The action-type need only consist of enough characters to distinguish between the three possible types. For example, the character S as the second argument can only imply the SUBTRACT action type while CR and CH are needed to distinguish the other two action types.

The file-interface-ids specify how you want to access the data file. You specify the file-interface-ids by one of two methods. You either give an integer number from the following table that represents the desired access method. Or, you give the name of a pointer adverb that already has the correct value. The pointer adverbs are often easier to remember than the actual value:

	ID   Pointer Adverb	Access method
	--------------------------------------
	 1 = DSCANS		Off-Line data file
	 2 = KSCANS		Keep data file
	 3 = SSCANS		Save data file
	 4 = GSCANS		Gains data file (Tucson Only)
	 5 = RSCANS		Individual Records data file (Green Bank Only)
	11 = PRINTFILE		Printout file
	12 = LOGFILE		Command-Logging file

Suppose it is desired to,

	a) attach the existing file LSAVE.A02 as the "off-line" file,
	b) detach the command-logging file completely,
	c) create a file LKEEP.B02, and attach it as keep file.

Then type,

	>CHNGFILE CHANGE DSCANS LSAVE.A02
	>CHNGFILE SUB PRINTFILE
	>CHNGFILE CR KSCANS LKEEP.B02

(NOTE : If you just type,

	>CHNGFILE

then you will be prompted by the program for the respective attributes.)

Changing Project Code or Spectrometer Type

Changing Project Code : Green Bank Only

At Green Bank, if you have data in the "on-line" file under a number of different project codes, you may wish to change project codes from time to time without exiting from UniPOPS. The pseudo verb CHNGPRJ allows you to do this. CHNGPRJ takes a single attribute, namely the new project code. It must be the only instruction on the command line. CHNGPRJ is a "site-dependent feature", so if it does not work when used as described here, consult the local UniPOPS guru.

Suppose you are reducing data from project code A260, but wish at a certain point to begin looking at data you took under project code A270. Then type,

	>CHNGPRJ  A270

From this point, data access will be permitted to the data of project A270, but the scans of A260 will no longer be available.

A special project code of NULL indicates that the user does not want to access on-line data. The FILES verb will indicate your current project code.

Changing from Hybrid-Correlator to Filter-Bank Data : Tucson Only

At Tucson, there are always 4 files that may receive raw data from the telescope: sdd, gsdd, sdd_hc, and gsdd_hc (the actual names have your initials and the file version number appended to the above names). The first two contain either filter bank or continuum data and the last two contain hybrid spectrometer data. A typical 12-meter observing run may use several files (although with the new SDD data format, files can hold more data so that the need for several files is not as pressing as it once was). In addition, several UniPOPS verbs need to make assumptions about subscan number when none is supplied (e.g. GET, GGET, SELECT, SUMMARY, and TELLS) and many of the standard 12-meter procedures need to make similar assumptions (F, S, C1, C2, etc). The current convention for 12-meter subscan numbers (the fractional part of the scan number) is that subscans numbers for the filter bank data files are in the range .01 to .04 while the hybrid spectrometer data files are in the range .11 to .18. The default subscan number is either .01 or .11. Four verbs are useful for specifying or displaying the version number and default subscan number: CHNGONLINE, CHNGVER, FILES and OLFILES.

To find out which raw data files are currently opened, use OLFILES.

The project code displayed by the FILES verb indicates your initials and the type of file that is the current default for SELECT, SUMMARY, TELLS, etc. (FB indicates filter bank data, HC indicates hybrid spectrometer data).

To switch from accessing filter-bank data to hybrid-correlator data, use the verb CHNGONLINE. Type,

	> CHNGONLINE(HCTYPE, HC_VER)

HC_VER is a pointer adverb that is always set to the hybrid spectrometer file version number currently opened.

To switch back to filter-bank data, type,

	> CHNGONLINE(FBTYPE, FB_VER)

FB_VER is a pointer adverb that is always set to the filter bank file version number currently opened.

In CONDAR, the available file types are CONTYPE and CON_VER.

To switch to a different version you can use CHNGONLINE if you know what the available version are or CHNGVER if you need a list of available version numbers.

To switch to version 3 of the filter-bank data file, type,

	> CHNGONLINE(FBTYPE, 3)

To change the version of the hybrid spectrometer file currently opened when you don't know what versions are available type,

	> CHNGVER HCTYPE

You will be presented with a list of version numbers to choose from.

The version numbers of the gsdd and gsdd_hc files are forced to be the same as the corresponding sdd and sdd_hc files.

There are two standard spectral line procedures that help when switching between filter bank and hybrid spectrometer data: FBDATA and HCDATA. These also set the PROMPT adverb to indicate what the last action was. For example, to switch to indicating that the filter bank data is the default data, type:

	> FBDATA

(the prompt should switch to be "LineF>"). To switch to indicating that the hybrid spectrometer data is the default data, type:

	> HCDATA

(the prompt should switch to be "LineH>").

There is no way to change the initials that the program uses in constructing the names of the on-line data files. If you need to change initials for any reason, you must exit the program and log out and log back in to the computer and restart the program.

What Data is Available on Disk ?

Two verbs, TELL and SUMMARY, exist in UniPOPS to give the user a synopsis of the scans currently available to them in either the "on-line" or "off-line data" files, the "save" file, or the "keep" file. TELL is the more succinct, although SUMMARY provides more information.

TELL

TELL takes a single attribute, which identifies the data to be listed. These attributes are,

	TELL DISK    - prints the total number of scans in the "on-line
		       data" file, the "off-line data" file, the "save"
		       and "keep" files, and (when available) the
		       "gains" file.

	TELL DSCANS  - lists by number the scans presently available to
		       the user in the "on-line data" file, and the
		       "off-line data" file.

	TELL KSCANS  - lists by number the scans in the "keep" file.

	TELL SSCANS  - lists by number the scans in the "save" file.

	TELL RSCANS  - lists by number the scans in the "individual records"
		       file (on and off-line Green Bank data only).

	TELL ONDATA  - lists by number the scans in the "on-line data"
		       file.

	TELL OFFDATA - lists by number the scans in the "off-line data"
		       file.

	TELL GSCANS  - lists by number (Tucson only) the scans in the
	 	       "gains" file.

Suppose that a synopsis is required giving the total number of scans in each data file, and then a list of scan numbers for the "on-line" and "off-line data" files and the "keep" and "save" files, type,

	>TELL DISK
	>TELL DSCANS
	>TELL KSCANS
	>TELL SSCANS

SUMMARY

SUMMARY prints a line of information for each receiver of each scan in a data file. This comprises, for spectral-line data: the scan no., rx no., source name, observing coordinates, frequency resolution, and rest frequency. For continuum: the scan no., rx no., source name, observing coordinates, rate of telescope motion, and integration time. The rx no., or receiver number, is simply the subscan number (i.e. the fractional part of the scan number).

SUMMARY takes a single attribute, the options being DSCANS, ONDATA, OFFDATA, KSCANS, SSCANS, GSCANS and RSCANS, defined as for TELL above. SUMMARY uses the adverbs BSCAN and ESCAN to determine the lowest and highest scan numbers to be listed. The defaults for BSCAN and ESCAN are both zero, implying the lowest and highest scan numbers in the file, respectively.

Suppose that a summary of the scans in the "on- and off-line data" files is desired. Then type,

	>SUMMARY DSCANS

NOTE : Both TELL SSCANS and SUMMARY SSCANS will "map" the position of a scan to show its "save bin" position (see Section 5.9).

The UniPOPS Data Arrays

In using UniPOPS, observational data and their associated header parameters, (called scans or spectra in this Cookbook), are read from one of the attached disk files. These are then processed with the facilities of UniPOPS, and can be output to another (or the same in the case of the "save", "keep" or "off-line data" file) attached disk file. While the scan is within UniPOPS, ten data arrays, named Array (0) to Array (9), are available to the user for holding scans. Scans can be read into any of the ten arrays, and moved, or copied, between these arrays as desired.

The UniPOPS commands, or verbs, that perform operations on the scans often assume that the required data will be available in specific arrays, and will also write the processed data into specific arrays. For example, to use the verb TEMP (see Section 9.7) on Tucson data, an "on-scan" is expected in Array (0), an "off-scan" in Array (1), and a "gains" array in Array (2). The result of TEMP is placed in Array (0). For a complete list as to which verbs use what arrays, see Appendix E, which a "novice" UniPOPS user is advised to keep handy at all times. Often, you will move a partially processed spectrum into a particular array for temporary storage, and return it to a "working" array later for further processing. (Note that the "save" file could also be used for this, but would be slower.)

Scans can be read from the "on-line data" file, the "off-line data" file, the "save" and "keep" file, (in Tucson) the "gains" file, and (for Green Bank data) the "individual records" file. Scans can be written to the "save" and "keep" files. In the remainder of this chapter, we will describe the operations of reading data from a disk file to a UniPOPS array, moving and copying it between arrays, and writing it out to a disk file.

Reading Data from On-line and Off-line Data Files

The basic set of verbs used for reading data from the "on-line" and "off-line data" files into Array (n) are GET0 - GET9. Each of these verbs takes a single attribute whose syntax is,

	GETn  [-]scan-number[.(rx-number/100)]

where square brackets [] denote optional extensions. The option of giving a negative value to the attribute is used to define precedence between the "on-line" and "off-line data" files, as follows,

	Negative attribute = the "off-line" file is searched before the
			     "on-line" file when looking for the
			     demanded scan.

	Positive attribute = the "on-line" file is searched before the
			     "off-line" file.

If the fractional part of the attribute is omitted, then the data for the first receiver is assumed to be required (for Tucson on-line data, see Section 5.4.2 for further information).

Suppose that you wanted to read data for,

	a) the first rx for scan 700 held in the "off-line" file into
	   Array (0),
	b) the third rx for scan 800 held in the "on-line" file into
	   Array (4),
	c) the eleventh rx for scan 900 held in the "off-line" file
	   into Array (9).

Then type,

	>GET0  -700		or  	>GET0  -700.01
	>GET4   800.03
	>GET9  -900.11

A number of pseudonyms exist for GET0 and GET1. GET0 can be replaced by GET or ON, and GET1 by OFF, without changing the operations. The commands ON and OFF exist because the verb TEMP (see Section 9.7) requires a total-power ON scan to be present in Array (0), and an OFF scan in Array (1). GET, which is simpler to type than GET0 when reading data into Array (0), is the most-used read operation.

It you wanted to read the first rx of scan 700 from the "on-line data" file into Array (0) and scan 701 into Array (1), the following command lines would all be equivalent,

	>GET0 700;   GET1 701
	>GET0 700;   OFF  701
	>GET  700;   GET1 701
	>GET  700;   OFF  701
	>ON   700;   GET1 701
	>ON   700;   OFF  701

A Tucson-specific verb GGET copies a "gains" scan from the "on-line" or "off-line data" files (using the rule for file selection given above for GETn) and places it in Array(2). This is required for Tucson data by the verb TEMP (see Section 9.7). Suppose you wished to bring the gains data from disk for the second receiver of scan 550, and display it on the screen. Then type,

	>GGET 550; COPY(2,0) PAGE SHOW

A Green-Bank-specific verb GETIR gets an individual record from either the on-line individual record data or from the off-line individual record file (which can be constructed using the cvt.tele-recs utility from a telescope archive tape). GETIR places an individual record into Array (0). This verb should be used if you need to edit individual records of a scan. It is not intended for use by new users. Consult the documentation on GETIR for details. You should also consult the Green Bank staff for help as well as for information on useful procedures to help in your editing.

A Tucson-specific verb GETOTF retrieves data from the on-line file from a row of an on-the-fly spectral-line map. On-the-fly data is currently processed primarily outside of UniPOPS. The flow of data from an OTF scan to a map is currently being developed. You should get the most up-to-date information from the 12-meter staff. The GETOTF verb places one spectra from the row (or any of the 5 ancillary information for that row) into Array (0). Consult the documentation on GETOTF for details.

Reading the Latest Scan, or Partial Scan

At the Green Bank or Tucson telescopes, it is possible to read the most recently-completed scan, or even the present partially-observed scan (available in Green Bank only), into Array (0) via the verb CGET. CGET takes a single attribute equal to the rx number of the required spectrum.

Suppose that it is wished to look at the present scan in progress, or that just completed, for the first and fifth rx, then type,

	>CGET 1; PAGE SHOW
	>CGET 5; PAGE SHOW

A similar Tucson-specific verb CGGET is available for retrieving the most-recently completed gains scan from the "on-line data" file into Array (2). Its single attribute again equals the required rx number. Thus, to look at the most-recently completed gains scan for the second receiver, type,

	>CGGET 2; COPY(2,0) PAGE SHOW

The Save File

As detailed above, the "save" file is a READ/WRITE file, and hence offers a convenient way to store spectra, which may be recalled at a later time. The "save" file can be considered as consisting of many numbered "slots", usually called "save bins", into each of which a scan can be stored, and from which it can be recalled later. The verb SAVE will store the scan in Array (0) into the "save bin" numbered the same as the contents of the adverb NSAVE. The verb RECALL will retrieve the scan stored in the "save bin" pointed at by NSAVE and place it in the Array (0).

A further adverb, SPROTECT, will prevent the overwriting of an already full "save bin" if set equal to TRUE (its initial value). If you wish to overwrite the scan stored in "save bin" number 65, first type,

	>SPROTECT = FALSE
	>NSAVE = 65 ; SAVE

Suppose you wish to save a partially reduced scan held in Array (0) and an incomplete accumulation of scans held in Array (2), and also guard against overwriting already-used "save bins", then type,

	>SPROTECT = TRUE
	>NSAVE = 65 ; SAVE
	>NSAVE = 66 ; COPY(2,0) SAVE

Later, when you wish to restore the "status quo", type,

	>NSAVE = 66 ; RECALL COPY(0,2)
	>NSAVE = 65 ; RECALL

The verb CHECK will print the scan number of the spectrum held in the "save bin" pointed at by NSAVE, or report it empty if no spectrum is present. For example, to find the contents of "save bin" number 66, or whether it is empty, type,

	>NSAVE = 66 ; CHECK

The Keep File

The "keep" file is a READ/WRITE file. It is a convenient location to store both fully-processed spectra, and original data that you wish to export to your home institute. Further, a "keep" file can be reattached as the "off-line" or "save" data file. The verb KEEP will write the scan in Array (0) into the "keep" file.

If the adverb KPROTECT is set to TRUE, then a spectrum cannot be written to the "keep" file if one with the same scan number is already present there. If you wish to overwrite a given scan in the "keep" file with one of the same number, then type,

	>KPROTECT = FALSE
	>KEEP

To recall a scan from the "keep" file you should use KGET which accepts one argument, the scan number that you want to retrieve. The convention for specifying a scan number is the same as described above for the on-line and off-line data files. For example, to retrieve scan 1427.03 from the "keep" file, type,

	> KGET 1427.03

Shifting Spectra between UniPOPS Arrays

There is frequent need to shift data from one UniPOPS array to another. The verbs COPY(m,n) and MOVE (m,n) are provided to achieve this. Both take two arguments, where Array (m) and Array (n) are the array from which the data comes and the destination array, respectively. COPY leaves the same spectrum in both arrays on completion, while MOVE clears the array from which the spectrum is taken.

Suppose you wish to shift a scan in Array (0) to Array (5), read scan 700 into Array (0) and display it, and then read the original scan back into Array (0), leaving Array (5) cleared. Then type,

	>COPY(0, 5)
	>GET 700; PAGE SHOW
	>MOVE(5, 0)

Clearing a UniPOPS Array

If you want to create your own data distribution in a UniPOPS array, you may prefer to start with a cleared array (as left behind by MOVE). The verb REMOVE(n) will completely clear Array (n).

For example, if you wish to plot the system temperatures of scans 700 to 720, they you could type,

	>REMOVE(3)
	>H3(NOINT) = 21
	>SCALAR P, Q
	>FOR P = 1 TO 21; Q = 699 + P; GET Q; D3(P) = H0(STSYS); END
	>COPY(3,0)
	>PAGE SHOW

Note that, MOVE(m,n) is equivalent to,

	>COPY(m,n)
	>REMOVE(m)

Getting Data from Tape into UniPOPS and Vice Versa

You will often wish to read data from a magnetic tape and put it into a UniPOPS data file for further processing, or write data from a UniPOPS data file on to tape for export or storage purposes. This can be done to and from a number of formats, such as IEEE-Keep, PCPops, SDD, FITS, etc. These operations are described in detail in Appendix D of this Cookbook, to which we direct the user. Reading and writing from tape is dealt with in Section D-11 for FITS-tapes. For other formats, you should read Sections D-2 for writing from disk to magnetic tape, and D-3 for reading from tape to disk.

Go to the previous, next section.



Webmaster: Ronald J. Maddalena
Send questions or comments to


Cookbook table of contents               NRAO information