./toolbox/io
index_file__define.pro

Last modification date:

Wed Sep 28 13:27:14 2016

Base Class used for profiling contents of sdfits files inside an ASCII file. File has two main sections: the header section contains generic info, while the rows section is a mapping of 'data containers' (spectra or continua) to their location in sdfits files. See UML for all IO Classes, or INDEX UML for just index classes. This class is virtual: acutal index files will be handled by children of this class for spectral line or continuum index files. This class is responsible for: basic searching, file creation, modifiction gateways, basic rows translations, and index file verifications.

Fields Summary
FILE_NAME string	file name of ASCII index file.
FILE_LOADED longword integer	flag that determines that this object is up to date with the index file
FILE_PATH string	full path to the index file
HEADER object reference	object that manages the header section
ROWS object reference	object that manages the rows section
ROWS_CLASS string	string specifiying the class used for the rows object
ROW_LINES pointer	pointer to array of structures, each representing row in index file (local copy of row object's rows).
VERSION string	version number passed in on class construction
SPROTECT longword integer	.
INFO structure [1]	.
DEBUG longword integer	flag that determines verbosity

Routine Summary  66 routines

procedure index_file__define
Base Class used for profiling contents of sdfits files inside an ASCII file.

function INDEX_FILE::init([file_name=string], [version=string])
Class Constructor - flags and constants set here.

procedure INDEX_FILE::cleanup	private
Class Destructor

procedure INDEX_FILE::set_file_name, file_name
Sets file name of index file

procedure INDEX_FILE::reset
Sets up object to read/create new index file

procedure INDEX_FILE::set_file_path, file_path
Sets path to index file

function INDEX_FILE::get_file_name()
Retrieves file name of index file

function INDEX_FILE::is_file_loaded()
Finds if object contains index file's contents in memory

procedure INDEX_FILE::set_version, version
Sets version number of I/O modules

function INDEX_FILE::get_version()
Retrieves the version number of the I/O module using this object

procedure INDEX_FILE::list, _EXTRA=_EXTRA
Prints the rows section of the index file for those rows specified.

procedure INDEX_FILE::read_file[, ver_status], [file_name=string], [max_nrows=integer]
Loads contents of index file into memory.

procedure INDEX_FILE::list_header
Prints contents of index header section

function INDEX_FILE::search_for_row_info(indicies, _EXTRA=_EXTRA)
Given the search parameters used by search_index, returns the structures representing those rows in the index file

function INDEX_FILE::search_row_location(_EXTRA=_EXTRA)
Given the search parameters used by search_index, returns the structures representing the locations of each row in the index file in their sdfits files

function INDEX_FILE::find_values(search_arr, values[, status])	private
Multiplexes searches according to the array type and the query type.

function INDEX_FILE::find_string_matches(array, values)	private
Queries the input array with an array of canidates, and allows for wildcards ('*') at the begining and end of canidate strings.

function INDEX_FILE::find_float_ranges(array, values)	private
Queries the input array with an array of canidates, where canidates can be a string representing a single float, or a range of the form: '0.

function INDEX_FILE::find_exact_matches(search_arr, values, total_count)	private
A wrapper around a 'where' so that the locations of several values can be found in an array

function INDEX_FILE::find_int_ranges(array, values)	private
Queries the input array with an array of canidates, where canidates must be a string representing a range of integers of the form: '0:10' or ':0', or '0:' Warning: double quotes ("") don't work with ranges - use single quotes ('')

function INDEX_FILE::refine_search(arr, values, arr_index, name, count)

procedure INDEX_FILE::find_values_plus_and, array, values, and_result, name	private
Used for making a search in an array and ANDing the results with previous searches

procedure INDEX_FILE::new_file, observer, backend, tcal_table, file_path, [file_name=string]
Creates a new index file, initializing the header section

function INDEX_FILE::new_header_string(observer, backend, tcal_table, file_path)	private
Creates a string array of key-value pairs from the input info; used to create the header section

function INDEX_FILE::get_column_values(column, [/unique], [subset=array], _EXTRA=_EXTRA)
Retrieves values found for a column in the index file (all the files for exampler).

procedure INDEX_FILE::write_index_file, header_strings, [rows_info=array]
Takes a given header section and list of rows, and writes them to a file

procedure INDEX_FILE::update_index_file, rows_info
Appends new rows to the end of the rows section of index file

procedure INDEX_FILE::overwrite_row, index_num, new_row
Overwrites a row in the index file with a different row structure

function INDEX_FILE::get_procedure_from_obsmode(obsmode)	private
Extracts procedure name from sdfits OBSMODE column

function INDEX_FILE::translate_polarization(polarization)	private
Translates sdfits value of polarizaiton to a char representation

function INDEX_FILE::get_col_variability(rows, tag_name, names, default_value)	private
Given rows from an sdfits file, how much does the value of a given column vary? Used when translating contents of an sdfits file into contents of the index file.

function INDEX_FILE::get_polarization_numbers(scan_rows, names)	private
Algorithm for determining polarization numbers (0-based), given the sdfits rows of a scan.

function INDEX_FILE::get_row_value(row, tag_name, virtuals, names, default_value)	private
Method for attempting to extract a value from an sdfits row.

function INDEX_FILE::get_full_file_name(file_name)	private
If this object is using the file path, returns full path name of index file

function INDEX_FILE::check_index_with_reality([expanded], [/verbose])
Checks the contents of the index file with the actual charcteristics of the files it lists, looking for inconsistencies

function INDEX_FILE::check_io_version_number()
Compares io version number found in index file with number coded in object

function INDEX_FILE::file_exists([file_name=string])	private
Checks if file exists

function INDEX_FILE::search_range([start][, finish], search_result)	private
If the start and finish parameters are used, it pares down the search indicies to just cover the range requested.

function INDEX_FILE::get_uniques(arr)	private
Sorts and uniques an array

procedure INDEX_FILE::set_debug_on
Makes object verbose

procedure INDEX_FILE::set_debug_off
Makes object quiet

function INDEX_FILE::check_file_properties(file_name[, expanded], [/verbose])	private
Checks basic file properties to see if they agree with what the index file has listed.

function INDEX_FILE::get_nsave_index(nsave)
Retrieves the unique index number for a given unique nsave number.

procedure INDEX_FILE::set_nsave, index_num, nsave
Sets the nsave number for a given index number in the index file

procedure INDEX_FILE::set_sprotect_on
Sets index file so nsave numbers cannot be overwritten

procedure INDEX_FILE::set_sprotect_off
Sets index file so nsave numbers can be overwritten

function INDEX_FILE::get_sprotect()
Retrieves the state of nsave protection

function INDEX_FILE::are_index_file_indicies_unique()
Diagnostic function to determine if index file indicies are unique (as they should be)

procedure INDEX_FILE::set_more_format_on
Sets the object to print rows using the interactive 'more' format

procedure INDEX_FILE::set_more_format_off
Sets the object NOT to print rows using the interactive 'more' format

procedure INDEX_FILE::list_available_columns
Prints the available columns from the rows section for list; these are also the valid search keywords

procedure INDEX_FILE::set_user_columns, columns
Sets what columns should be used for user listing

procedure INDEX_FILE::list_user_columns
Prints the columns currently selected for the user specified listing

function INDEX_FILE::get_available_columns()
Returns the available columns from the rows section for list; these are also the valid search keywords

function INDEX_FILE::get_param_types()

procedure INDEX_FILE::read_new_rows[, num_new_lines], [max_nrows=long]
For reading into memory the new rows that may have been written to the index file ( by hand, or by an online process ).

function INDEX_FILE::get_info()
Retrieve the structure that has info on the index file - updated only on reading file, or read_new_rows

procedure INDEX_FILE::set_info, info
Sets the structure that has info on the index file

function INDEX_FILE::validate_search_keywords(keyword_struct)
Takes in an _EXTRA structure, and validates tags according to the columns listed in the format.

function INDEX_FILE::get_num_index_rows()
Retrieves the number of rows currently in index file

function INDEX_FILE::validate_column_names(columns)
Checks to make sure that all the column names submitted are valid.

function INDEX_FILE::get_num_chans_from_dims(dimensions)
Determines number of channels from the dimensions keyword value (TDIM#) This is the first number in the string: '(1,1,1,1)'

function INDEX_FILE::get_row_structs()
Returns the array of structures representing the lines in the rows sections

function INDEX_FILE::update_file_with_row_structs(row_structs)
Instead of parsing the rows from an sdfits table, and using that info to update the index file, this takes in rows from another index object, changes the index numbers for those rows, and appends them to this objects index file

function INDEX_FILE::get_base_index_for_file(filename)

function INDEX_FILE::file_match(files)
Find the longest string with a trailing wildcard that matches all of the filenames in the input string array.

Routine Details

index_file__define

procedure index_file__define

Base Class used for profiling contents of sdfits files inside an ASCII file. File has two main sections: the header section contains generic info, while the rows section is a mapping of 'data containers' (spectra or continua) to their location in sdfits files. See UML for all IO Classes, or INDEX UML for just index classes. This class is virtual: acutal index files will be handled by children of this class for spectral line or continuum index files. This class is responsible for: basic searching, file creation, modifiction gateways, basic rows translations, and index file verifications.

INDEX_FILE::init

function INDEX_FILE::init([file_name=string], [version=string])

Class Constructor - flags and constants set here.

Keywords
file_name in, optional string	file name of index file
version in, optional string	version number

INDEX_FILE::cleanup

private

procedure INDEX_FILE::cleanup

Class Destructor

INDEX_FILE::set_file_name

procedure INDEX_FILE::set_file_name, file_name

Sets file name of index file

Parameters
file_name in string	name of index file

INDEX_FILE::reset

procedure INDEX_FILE::reset

Sets up object to read/create new index file

INDEX_FILE::set_file_path

procedure INDEX_FILE::set_file_path, file_path

Sets path to index file

Parameters
file_path in string	path to index file

INDEX_FILE::get_file_name

function INDEX_FILE::get_file_name()

Retrieves file name of index file

Returns

file name of index file

INDEX_FILE::is_file_loaded

function INDEX_FILE::is_file_loaded()

Finds if object contains index file's contents in memory

Returns

0,1

INDEX_FILE::set_version

procedure INDEX_FILE::set_version, version

Sets version number of I/O modules

Parameters
version in string	version number

INDEX_FILE::get_version

function INDEX_FILE::get_version()

Retrieves the version number of the I/O module using this object

Returns

version number (string)

INDEX_FILE::list

procedure INDEX_FILE::list, _EXTRA=_EXTRA

Prints the rows section of the index file for those rows specified. For keywords, see: ROWS_INDEX_SECTION::list

Keywords
_EXTRA	.

INDEX_FILE::read_file

procedure INDEX_FILE::read_file[, ver_status], [file_name=string], [max_nrows=integer]

Loads contents of index file into memory. Sets file loaded flag to True.

Parameters
ver_status out, optional variable	whether the index has the correct version number

Keywords
file_name in, optional string	file name of index file
max_nrows in, optional integer	Maximum number of index rows to read - to be used for the online case only.

INDEX_FILE::list_header

procedure INDEX_FILE::list_header

Prints contents of index header section

INDEX_FILE::search_for_row_info

function INDEX_FILE::search_for_row_info(indicies, _EXTRA=_EXTRA)

Given the search parameters used by search_index, returns the structures representing those rows in the index file

Returns

array of structures representing the found rows of the index file

Uses

INDEX_FILE::search_index

INDEX_FILE::search_row_location

function INDEX_FILE::search_row_location(_EXTRA=_EXTRA)

Given the search parameters used by search_index, returns the structures representing the locations of each row in the index file in their sdfits files

Returns

array of structures representing the locations of each found row

Uses

INDEX_FILE::search_index

INDEX_FILE::find_values

private

function INDEX_FILE::find_values(search_arr, values[, status])

Multiplexes searches according to the array type and the query type.

Returns

array of indicies where the values can be found in search array

Examples

 >arr = [1,3,5,7]
 >values = [3,7]
 >print, index->find_values(arr,values)
 >1,3
 
 >arr = ['dog','mouse','cat','moose']
 >values = ['dog','mo*']
 >print, index->find_values(arr,values)
 >0,1,3
 

Uses

INDEX_FILE::find_int_ranges INDEX_FILE::find_exact_matches INDEX_FILE::find_float_ranges INDEX_FILE::find_string_matches

INDEX_FILE::find_string_matches

private

function INDEX_FILE::find_string_matches(array, values)

Queries the input array with an array of canidates, and allows for wildcards ('*') at the begining and end of canidate strings.

Returns

array of indicies where the values can be found in search array

Examples

 >arr = ['dog','mouse','cat','moose']
 >values = ['dog','mo*']
 >print, index->find_string_matches(arr,values)
 >0,1,3
 

INDEX_FILE::find_float_ranges

private

function INDEX_FILE::find_float_ranges(array, values)

Queries the input array with an array of canidates, where canidates can be a string representing a single float, or a range of the form: '0.0:1.0' or ':0.0', or '0.0:' Warning: double quotes ("") don't work with ranges - use single quotes ('')

Returns

array of indicies where the values can be found in search array

Examples

 >arr = [0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0]
 >values = ':1.0,2.5:4.0'
 >print, index->find_string_matches(arr,values)
 >0,1,6,7
 

INDEX_FILE::find_exact_matches

private

function INDEX_FILE::find_exact_matches(search_arr, values, total_count)

A wrapper around a 'where' so that the locations of several values can be found in an array

Returns

array of indicies where the values can be found in search array

INDEX_FILE::find_int_ranges

private

function INDEX_FILE::find_int_ranges(array, values)

Queries the input array with an array of canidates, where canidates must be a string representing a range of integers of the form: '0:10' or ':0', or '0:' Warning: double quotes ("") don't work with ranges - use single quotes ('')

Returns

array of indicies where the values can be found in search array

Examples

 >arr = [0,5,10,15,20,25,30,35,40]
 >values = ':10,25:40'
 >print, index->find_string_matches(arr,values)
 >0,1,6,7
 

INDEX_FILE::refine_search

function INDEX_FILE::refine_search(arr, values, arr_index, name, count)

Parameters
arr	.
values	.
arr_index	.
name	.
count	.

INDEX_FILE::find_values_plus_and

private

procedure INDEX_FILE::find_values_plus_and, array, values, and_result, name

Used for making a search in an array and ANDing the results with previous searches

Examples

 >a = [1,2,3,4]
 >b = ['dog','cat','mouse','moose']
 >;init the search results
 >and_result = [0,1,2,3]
 >io->find_values_plus_and,a,'1:',and_result
 >print,and_result
 >[1,2,3]
 >io->find_values_plus_and,b,'dog,mo*',and_result
 >print,and_result
 >[2,3]
 

Uses

INDEX_FILE::find_values

INDEX_FILE::new_file

procedure INDEX_FILE::new_file, observer, backend, tcal_table, file_path, [file_name=string]

Creates a new index file, initializing the header section

Uses

INDEX_FILE::set_file_name INDEX_FILE::new_header_string INDEX_FILE::write_index_file

INDEX_FILE::new_header_string

private

function INDEX_FILE::new_header_string(observer, backend, tcal_table, file_path)

Creates a string array of key-value pairs from the input info; used to create the header section

Returns

string array of key-value pairs from input params.

INDEX_FILE::get_column_values

function INDEX_FILE::get_column_values(column, [/unique], [subset=array], _EXTRA=_EXTRA)

Retrieves values found for a column in the index file (all the files for exampler).

Returns

the values found for a column in the index file

Uses

INDEX_FILE::get_row_info_strct

INDEX_FILE::write_index_file

procedure INDEX_FILE::write_index_file, header_strings, [rows_info=array]

Takes a given header section and list of rows, and writes them to a file

Parameters
header_strings in array	string array of key-value pairs for header section

Keywords
rows_info in, optional array	array of structures representing rows for index file

INDEX_FILE::update_index_file

procedure INDEX_FILE::update_index_file, rows_info

Appends new rows to the end of the rows section of index file

Parameters
rows_info in array	array of structures representing rows to be appended

INDEX_FILE::overwrite_row

procedure INDEX_FILE::overwrite_row, index_num, new_row

Overwrites a row in the index file with a different row structure

Parameters
index_num in, required long	index number of row which is to be overwritten
new_row in, required struct	new row to write in index file at index_num

INDEX_FILE::get_procedure_from_obsmode

private

function INDEX_FILE::get_procedure_from_obsmode(obsmode)

Extracts procedure name from sdfits OBSMODE column

Returns

procedure name

INDEX_FILE::translate_polarization

private

function INDEX_FILE::translate_polarization(polarization)

Translates sdfits value of polarizaiton to a char representation

Returns

polarization in char representation

INDEX_FILE::get_col_variability

private

function INDEX_FILE::get_col_variability(rows, tag_name, names, default_value)

Given rows from an sdfits file, how much does the value of a given column vary? Used when translating contents of an sdfits file into contents of the index file.

Returns

the variablity of this column in the fits file

INDEX_FILE::get_polarization_numbers

private

function INDEX_FILE::get_polarization_numbers(scan_rows, names)

Algorithm for determining polarization numbers (0-based), given the sdfits rows of a scan.

Returns

an array for polarization numbers of each row in index file for this scan (ex: [0,0,1,1,0,0,1,1,...])

Uses

INDEX_FILE::get_col_variability

INDEX_FILE::get_row_value

private

function INDEX_FILE::get_row_value(row, tag_name, virtuals, names, default_value)

Method for attempting to extract a value from an sdfits row. If the row contains the tag name requested, that value is passed back. If that tag name actually specifies a keyword in the extension-header, and NOT a column, then that value is returned. Finally, if the tag name mathes one of the expected column names that were not found in this extension, the default value is returned.

Returns

either the value of row.tag_name, virtauls.tag_name, or default_value

INDEX_FILE::get_full_file_name

private

function INDEX_FILE::get_full_file_name(file_name)

If this object is using the file path, returns full path name of index file

Parameters
file_name	.

INDEX_FILE::check_index_with_reality

function INDEX_FILE::check_index_with_reality([expanded], [/verbose])

Checks the contents of the index file with the actual charcteristics of the files it lists, looking for inconsistencies

Returns

0,1

Uses

INDEX_FILE::get_full_file_name INDEX_FILE::check_io_version_number INDEX_FILE::is_file_loaded INDEX_FILE::file_exists INDEX_FILE::check_file_properties

INDEX_FILE::check_io_version_number

function INDEX_FILE::check_io_version_number()

Compares io version number found in index file with number coded in object

Returns

0,1

Uses

INDEX_FILE::get_header_value

INDEX_FILE::file_exists

private

function INDEX_FILE::file_exists([file_name=string])

Checks if file exists

Returns

0,1

INDEX_FILE::search_range

private

function INDEX_FILE::search_range([start][, finish], search_result)

If the start and finish parameters are used, it pares down the search indicies to just cover the range requested.

Returns

long array which is the result of the range on the search

INDEX_FILE::get_uniques

private

function INDEX_FILE::get_uniques(arr)

Sorts and uniques an array

Returns

uniqe values of array

INDEX_FILE::set_debug_on

procedure INDEX_FILE::set_debug_on

Makes object verbose

INDEX_FILE::set_debug_off

procedure INDEX_FILE::set_debug_off

Makes object quiet

INDEX_FILE::check_file_properties

private

function INDEX_FILE::check_file_properties(file_name[, expanded], [/verbose])

Checks basic file properties to see if they agree with what the index file has listed.

Returns

0,1

INDEX_FILE::get_nsave_index

function INDEX_FILE::get_nsave_index(nsave)

Retrieves the unique index number for a given unique nsave number.

Returns

the unique index number for the row where this nsave number is found

INDEX_FILE::set_nsave

procedure INDEX_FILE::set_nsave, index_num, nsave

Sets the nsave number for a given index number in the index file

Parameters
index_num in, required long	the index number for which we are setting the nsave number
nsave in, required long	the nsave number to be written to the index file

INDEX_FILE::set_sprotect_on

procedure INDEX_FILE::set_sprotect_on

Sets index file so nsave numbers cannot be overwritten

INDEX_FILE::set_sprotect_off

procedure INDEX_FILE::set_sprotect_off

Sets index file so nsave numbers can be overwritten

INDEX_FILE::get_sprotect

function INDEX_FILE::get_sprotect()

Retrieves the state of nsave protection

Returns

0 - nsave numbers cannot be overwritten; 1 - they cannot be overwritten

INDEX_FILE::are_index_file_indicies_unique

function INDEX_FILE::are_index_file_indicies_unique()

Diagnostic function to determine if index file indicies are unique (as they should be)

Returns

0 - bad, 1 - good

INDEX_FILE::set_more_format_on

procedure INDEX_FILE::set_more_format_on

Sets the object to print rows using the interactive 'more' format

INDEX_FILE::set_more_format_off

procedure INDEX_FILE::set_more_format_off

Sets the object NOT to print rows using the interactive 'more' format

INDEX_FILE::list_available_columns

procedure INDEX_FILE::list_available_columns

Prints the available columns from the rows section for list; these are also the valid search keywords

INDEX_FILE::set_user_columns

procedure INDEX_FILE::set_user_columns, columns

Sets what columns should be used for user listing

Parameters
columns in, required string array	array of columns to print on list command

INDEX_FILE::list_user_columns

procedure INDEX_FILE::list_user_columns

Prints the columns currently selected for the user specified listing

INDEX_FILE::get_available_columns

function INDEX_FILE::get_available_columns()

Returns the available columns from the rows section for list; these are also the valid search keywords

INDEX_FILE::get_param_types

function INDEX_FILE::get_param_types()

INDEX_FILE::read_new_rows

procedure INDEX_FILE::read_new_rows[, num_new_lines], [max_nrows=long]

For reading into memory the new rows that may have been written to the index file ( by hand, or by an online process ). First checks file size to see if there might be any new lines, then jumps to previous last line, and reads new rows.

Parameters
num_new_lines out, optional variable	number of new rows found in index file

Keywords
max_nrows in, optional long	maximum number of total rows to read (new rows + old rows <= max_nrows)

INDEX_FILE::get_info

function INDEX_FILE::get_info()

Retrieve the structure that has info on the index file - updated only on reading file, or read_new_rows

Returns

a {file_info} structure

INDEX_FILE::set_info

procedure INDEX_FILE::set_info, info

Sets the structure that has info on the index file

Parameters
info in, required file_inf	structure

INDEX_FILE::validate_search_keywords

function INDEX_FILE::validate_search_keywords(keyword_struct)

Takes in an _EXTRA structure, and validates tags according to the columns listed in the format.

Returns

0 - invalid, 1 - valid

INDEX_FILE::get_num_index_rows

function INDEX_FILE::get_num_index_rows()

Retrieves the number of rows currently in index file

Returns

number of rows in index file

INDEX_FILE::validate_column_names

function INDEX_FILE::validate_column_names(columns)

Checks to make sure that all the column names submitted are valid. Unlike search keywords, must be an exact match, though not case sensitive

Returns

0 - not valid, 1 - valid

INDEX_FILE::get_num_chans_from_dims

function INDEX_FILE::get_num_chans_from_dims(dimensions)

Determines number of channels from the dimensions keyword value (TDIM#) This is the first number in the string: '(1,1,1,1)'

Returns

an integer, the first number in the string '(1,1,1,1)'

INDEX_FILE::get_row_structs

function INDEX_FILE::get_row_structs()

Returns the array of structures representing the lines in the rows sections

Returns

array of structures

INDEX_FILE::update_file_with_row_structs

function INDEX_FILE::update_file_with_row_structs(row_structs)

Instead of parsing the rows from an sdfits table, and using that info to update the index file, this takes in rows from another index object, changes the index numbers for those rows, and appends them to this objects index file

Returns

0 - failure, 1 - success

INDEX_FILE::get_base_index_for_file

function INDEX_FILE::get_base_index_for_file(filename)

Parameters
filename	.

INDEX_FILE::file_match

function INDEX_FILE::file_match(files)

Find the longest string with a trailing wildcard that matches all of the filenames in the input string array. Returns "*" if there are no characters in common. This assumes that these are fits files and hence all should end in ".fits". Used in get_scan_info in the derived classes for the case where there are multiple FITS files in use.

Returns

string guaranteed to match all elements of files