Checklist for VLBI observing on the GBT

(not so) Very Brief Project Startup Checklist

Some (but not all) VLBI experiments will be scheduled dynamically, which means that they will be scheduled for one of two possible dates, a "primary" date, and an "alternate" date. Thus the experiment will appear twice on the resource calendar. This information will also appear on the GBT VLBI Project Summaries web page, which will show both the prime and backup dates.

The PI needs to make the dynamic decision by 11 AM for a project that takes place between the following 5 PM and a day later. This is just the same as the decision process for all dynamically scheduled GBT observing.

The operator should check Carl's daily e-mail which tells the results of the daily decisions. If the decision is for the primary day, then the procedures for running the VLBI experiment are the same as before.

But if the decision is for the backup date, then the operator needs to check some things:

To re-generate the OBSERV.TX file for the alternate schedule, log into "gbvlbi" and type : "setalt [projcode]"
for example, to set the alternate schedule for project bb240b, type:

Then one should check the OBSERV.TX file to make sure the "alt" project has been set and that the start and stop times are reasonable.

Note that setalt should be run before the primary schedule would have run; otherwise the Mark5 will start recording on the primary schedule.

If you want to set OBSERV.TX back to the primary schedules, just type:

Note also that when you have changed OBSERV.TX by running setalt (or by editing it directly) you have to re-boot the station computer.

Astrid Files The files for the alternate schedule should have been loaded into the Astrid database. The scheduling blocks will have "alt" in their names. The operator should run these "alt" blocks instead of the normal ones whenever the backup schedule has been chosen.

More changes

You will notice there is a new "alt/" directory under ~gbvlbi/obs/ -- this is where the alternate versions of the schedules will be put. Normally the retrieval of project schedules into these directories will be taken care of by the "VLBI friend" (Frank or Jules) well before the projects are supposed to be run. The procedures "fetchvobs" and "vlbprep" have been modified to download both the primary and alternate schedules from Socorro and put them in the ~gbvlbi/obs/ or ~gbvlbi/obs/alt/ directories. Both primary and alternate scheduling blocks will be imported into the Astrid database, and again this normally will have been done by the "VLBI friend".

to top

Check Disk or Tape Supply

• Try to get disks delivered at least a few days before the experiment so there is time to let the temperature stabilize.
• Communicate with the VLBA tape librarian using the e-mail address 'vlbatape@aoc.nrao.edu'
• When disks or tapes are received, and when disks or tapes are shipped out to Socorro, use the TRACK system to record the receiving and shipping of tapes.
• To get into the TRACK system, use the web browser, go to
  the TRACK web page: http://magnolia.nrao.edu/track/
  Login: gbvlba
  Enter data as directed by the web page.

to top

Download Schedules

  There are 3 methods of downloading files:
	a) new automatic method using program "fetchvobs"
	b) download through the aspen web page.
	c) download EVN schedule from "vlbeer"
	d) download from Goddard.
  
  Note that you should check for file updates a few hours before the 
  experiment just in case the schedules have been changed.

  First, log into any unix machine as "gbvlbi".  Contact the GBT operator
  for the password.
  $ cd obs


  -----------------------------
  A. Automatic file downloading
  -----------------------------
   Run the new python script "fetchvobs" which downloads schedules from Aspen.
   "fetchvobs" will fetch all files from the present month; or you may
   give it the month and year as "mmmYY" (e.g., "jan06").

   - log into a Linux machine as "gbvlbi"

   $ cd ~gbvlbi/obs/incoming
	[we use the 'incoming' directory to receive the files]

     [clear out the incoming directory:]
   $ rm *.*

     [Fetch the files: 
	
   $ fetchvobs

     or : fetchvobs mmmYY

     [wait a few minutes for the file download to finish]

     The incoming directory will contain all the *.gb files for the
	projects that use the GBT in the selected month.

	Copy the '[project]*.*' files up one level to  ~gbvlbi/obs/

     Now you are ready to set up the schedules.

  Print out the schedule files:
  $ a2ps -f6.5 [project]*.gb  

  -------------------------------------
  B. Downloading through the web page --
  -------------------------------------
  Use the link for the 
  VLBA-Aspen web site 

   Click on the desired month/year
	for example "nov02" will get November of 2002.
   Click on the project name.
   
   The files of interest are the ones with a '.gb' extension.
   Click on the   '---crd.gb' file, choose 'save as' in the file menu.
   Click on the   '---sch.gb' file, choose 'save as' in the file menu.

  $ a2ps -f6.5 [project]*.gb     {print out schedule files}

  
  -----------------------------------------------
  C. Download EVN schedule from "vlbeer"
  -----------------------------------------------
    The standard place for EVN schedules is "vlbeer"
    Sometimes if a project is using the EVN and the GBT is the only NRAO
    telescope, the files will not have been put on aspen.
    Of course, the user should really put the files on aspen.

    But just in case they don't, use the  link for the "vlbeer" web site. 


  ------------------------------------------------------
  D. Downloading schedules from NASA (geodetic projects):
  ------------------------------------------------------
        ftp cddisa.gsfc.nasa.gov   {user=vlbigeo }
        cd mmmyy
        get [project].skd

Set up Schedules

  After you have downloaded the schedule files:

  Look in the .key file to find which correlator (Socorro or JIVE)
   will be used, the PI name and email address.

  Look through the schedule files; the [project]sch.gb file is easier
  to read than the [project]crd.gb file.

  Take note of: 
	a) start and stop times and dates
	b) how many tapes are required
	c) when are tape changes needed?
	d) which receiver is required

  Check the start and stop times versus the GBT schedule: tell Carl about
  any discrepancies.  

  Edit the project summary web page to summarize the basic information.

Run the handy python script "vlbprep" :

• Remove earlier *crd.gb, *cr.gb, and *.turtle files from the ~gbvlbi/obs directory.
• Copy the "crd" files for all the upcoming experiments into the ~gbvlbi/obs/ directory.
• type: vlbprep

• the OBSERV.TX file
• a series of Astrid scheduling blocks for each project.

The following steps in the box are what has been replaced by running "vlbprep". Ignore the stuff in this box!

- Edit the "OBSERV.TX" file: - Log in to any linux machine as "gbvlbi" cd obs vi OBSERV.TX {or use your favorite editor} {duplicate the last line of OBSERV.TX, then modify the line to contain the name of the *crd.gb file, change the start time to now, and the stop time to a few minutes after the end of the schedule.} ** Note that the *crd.gb file names must contain no more ** than 8 characters before the dot! ** Shorten the file name if necessary! - Make the Observing Blocks for astrid. Run the program "vlbastrid" and give it the name of the [project]crd.gb file. vlbastrid [project]crd.gb (this will produce 3 or more files whose name starts with the project code and ends with "obsnn.turtle" in which the "nn" is a sequence number.)

  - print a summary of the astrid files:
	   a2ps [project}__sum 

  - Print the schedule files:

	a2ps -f6.5 [project]crd.gb
	a2ps -f6.5 [project]sch.gb   

to top

Notes on Frequencies

  Have a look in the  "....crd.gb" file at the "extlo" and "extlosideband"
   parameters.  These will each be set to a pair of parameters, for example:

	extlo = (1,  5.6000000000)
	extlosideband = (1,L)
	extlo = (3,  5.6000000000)
	extlosideband = (3,L)

   - the first of the pair is a number 1,2,3, or 4, which corresponds
	to IF input ports A, B, C, or D of the DAR.  

   - the "extlosideband" 2nd parameters should be Lower (L) for 
	receivers of 10 GHz and less (up thru X-band),
     and should be Upper (U) for higher frequency receivers (KU band and above).

   - For Upper sideband receivers, the extlo value will be the desired
	sky frequency (Fsky) minus the BBC frequency, for all backend
	channels.

   - For Lower sideband receivers, the extlo value will be the desired
	sky frequency (Fsky) plus the BBC frequency, for all backend
	channels.
   
   Note that in the "...sch.gb" file, near the top, will be the desired
	sky frequency (Fsky) for each channel.  
	These are labelled: "Next scan frequencies"

	Below these Fsky values are the BBC frequencies for each channel.
	These should of course agree with the BBC values in the "...crd.gb"
	file.

    GBT LO settings: 

	The nominal center of the IF3 band for VLBI observing is 750 MHz.
	Assuming the midpoint of the BBC frequencies is 750, then
	we set the GBT rest frequency (Frest) as follows:
	   Frest = extlo + 750MHz for upper-sideband receivers,
	 and
	   Frest = extlo - 750MHz for lower-sideband receivers,

	We set the LO2 frequencies in the converter rack to
	   either  12750 MHz for receivers that user IF1=3000,
	   or      15750 MHz for receivers that user IF1=6000,
	   or      10830 MHz for receivers that user IF1=1080.

	This will make Frest convert to 750 MHz going into the VLBI
	backend, and the BBC settings in the "...crd.gb" file will
	tune in the correct sky frequencies.

        In most cases, the midpoint of the users set of BBC frequencies
	will be something other than 750, hence one would alter the above
	accordingly.

     The program "vlbastrid" will determine Frest and lo3freq from the
     ...crd.gb file to put Frest at the midpoint of the desired set of
     frequencies and to make the effective LO equal to "extlo".
  

Do System Prechecks

  These checks should be done a day or two before the experiment so 
  any problems that show up can be fixed.
  Some of these checks can also be done an hour or two before the 
  experiment, if there is time, just to make sure everything is ok.

  First check the the station clock,
  and set up the screens  as described in the following 
	sections.
  - After the schedule file is downloaded, and the OBSERV.TX file
	has been edited, reboot the VLBA station computer to start
	the observing system.

  If you have control of the GBT receiver and LO/IF system,
	run Astrid and submit the first non-peak scheduling block for the 
	experiment.

	- check the desired receiver channels connect to the 
	   correct VLBA DAR channels.
	- adjust the attenuators in the converter rack to put the
	   power levels in the VLBA IF distributors in the range
	   about 10000 to 20000.
	- Turn on the test tone in LO1 and use a spectrum analyzer
	   to check that the tone is at the right frequency in the
	   BBCs.

	The following can be done with the VLBA back end, even if you
	  do not have control of the GBT front end and LO/IF system:

	- Check that the BBCs are set to the correct frequency and
	   that their LOs are all locked.
	- Check the LO signals with a spectrum analyzer.
	- Check the "formatter errors" and "system check" displays.
	

• Make sure all the hardware is powered on:

   - Check that the DAR, Recorders, station computer (VLBAGB), Mark5,
     and "spock" are powered on.
   - The computer "spock" runs the status checker (vgstatus), so make sure
     it is up and running.
   - If power has been off, refer to  power off/on instructions. 
   - Everything should be powered on at least an hour before an experiment starts.
   - If no experiment has been run for a few days, it is probably a good idea
  	to re-boot the station computer (VLBAGB).
  

• Files should be set up: "crd" file and Astrid files should be present in ~gbvlbi/obs/ directory, and OBSERV.TX file should have been edited to include the current experiment "crd" file. (Refer to "Set Up Schedules", above).
• Check that status file is updating: bring up the web page GBTVLBAStatus and check that the UTC is within the last 10 minutes (if not, type "stopvgstat" and wait a few minutes), and also that the "Clock" value is less than a few thousand (if not, the station timer on the VLBA_DAR rack may need to be reset).

B. GBT things:

• Correct receiver is rotated in place, and selected in the ScanCoordinator.
• If C-band or higher frequency, check that dynamic corrections are running.and that the active surface is running in FEM mode.
• If S-band or lower frequency, active surface should be in "zero points" mode and dynamic corrections should be off.
• Check that the "gbtstatus" display is updating and showing the correct antenna coordinates, source, and time.

Check clocks; set station clock.

• First, check that 'Site Time' is synchronized with GPS.
  • Compare the GPS display with the IRIG display. They should be on the same second. (also check that the year, month, and day is correct!)

• Next, check that the VLBI formatter is synchonized with SiteTime.
  • Check "clock" on the web page: GBTVLBAStatus
  • The value should be less than +/- a few thousand.
  • If not, push the reset button on the VLBA timing module.

• Set the VLBA station clock.
  • (logged into any unix machine) Type 'rs' to bring up the RSCREENS.
  • Bring up the 'OBS-->CLOCK' screen.
  • Check the year, day number, hour, minute, and second:
    they should agree exactly with the GPS display.
  • To double check, or if there is no GPS display:
    Call up the USNO master clock ( 202-762-1401 )
  • Use the CLOCK screen to adjust the station clock, if necessary.
    Make sure the station clock agrees with USNO or GPS.

Start Screens and check "gvstat"

• Log in to a convenient Linux machine ('naiad' for example) as 'gbvlbi'
• Check that the "gvstat" demon is running.
  • type : gvstat
  • (gvstat shows the current source in the schedule;
    check that the UTC time is within the last few minutes).
    (If not, type : "stopvgstat" and check again in 3 minutes)

• Type 'rs' to start RSCREENS.
  • OBS-->CLOCK to check that the clock is on the right second.
  • FORMAT-->FORMATERR to check the formatter has no errors.
    (if there are formatter errors, see "clearing formatter errors")
    
  • after initial checks, go to OBS-->DAS
    keep the OBS-->DAS screen up during the run to monitor the experiment.
  • Also bring up OBS-->FLAG screen to monitor the on-source status.
  • Check that the DAS screen shows the correct project and source:
    if not, do OBS-->REOBS.
    if OBS-->REOBS does not fix it, reboot the station computer.

• Start the CLEO Status, IFRack, and ConverterRack screens.

Recording on Mark 5 disks:

Fill Out the Disk Labels

• Disk serial number
• station code: GBT_VLBA
• UT Date and time when the Disk is mounted.
• UT Date and time when the Disk is dismounted.
• names of the projects that are on the disk.
• If the disk is not going to Socorro, make a note of it.

to top

   Make sure that 'gbvlbi' is in the Gateway.

   Log in to one of the Linux machines ('naiad' for example) as 'gbvlbi'
	-- type: cd ~gbvlbi/obs/
	-- type: astrid
	-- select "work online with control of the telescope"

	-- in the astrid "Edit" tab, select the project code.
           If the correct project code is not in the list, then select "AVLB".

	-- Import the scheduling blocks and save them to the database, if
		this had not already been done.

   Ask the operator to set the scan number (usually to 1).

   Make sure the antenna drives are activated and it is ready to run scans.

	-- select the "run" tab in astrid.
	-- set the project code if necessary.

	-- in the "Session" box, type the session code,
           or if you had used project code AVLB then type in the 
           VLBA project code as the session:
	    This is the first 5 or 6 characters of the turtle file name. 

	-- For the "Observer", select the project PI if his name is there,
	   otherwise use your own name.

   - Submit the first observing block,  whose name ends in _obs00_pk.turtle
   - This will do a peak and optionally a focus at a position near the 
	first source in the observing list.
	(Focus only for observing at frequencies >= 8 GHz).
   - Check that the peak and focus look good.
   - Check that the antenna is on the correct Azimuth wrap !

   - Run the first non-peak observing block, 
	whose name probably ends in _obs01.turtle
   - This will configure the system for VLBI and will move to the first
      source to be observed.

   Make sure the antenna starts on the correct wrap!! 
   --To do this, consult the [projname]sch.gb file which lists
     az and el coordinates.

Set IF Levels

• In the IF Rack CLEO screen, balance the IFs.

• In RSCREENS, bring up the OBS-->DAS screen.
• Use the Converter Rack CLEO screen to adjust the attenuators:
  put the VLBA IF power levels in the range 10,000-20,000.
  

   Note: Usually Rack A, converter modules 1 and 5, are used for 600MHz, 
	800 MHz, L-, C-, Ku-, K-, and Q-band receivers.
	Rack B, converter modules 9 and 13, are used for S-band and X-band receivers.

     ***This is how the VLBA IF channels connect to the GBT converter rack:
	-----------------------------------------------------------------
	IF 1:1 is VLBA_DAR:A  (RCP, from Cnv.rack A, CM5)
	IF 1:2 is VLBA_DAR:B  (RCP, from Cnv.rack B, CM13)
	IF 2:1 is VLBA_DAR:C  (LCP, from Cnv.rack A, CM1)
	IF 2:2 is VLBA_DAR:D  (LCP, from Cnv.rack B, CM9)
	-----------------------------------------------------------------

   For example, if using S-band or X-band receivers, adjust the attenuator
   of converter module 9 to bring the IFPower 2:2 to near 16000,
   and CM13 to bring IFPower 1:2 to near 16000.

   For the other receivers, CM1 adjusts IF 2:1 and CM5 for 1:1.

  - In SCREENS, run OBS-->CHECKALL  : there should be very few messages.
 

Check IF paths and frequencies

• Use the IF Manager either by itself or in the CLEO STATUS screen to check that frequencies and bandwidths are correct.
• Consult the file "[project]_setup.py" which is one of the files that the program "vlbastrid" produced. In that file will be a set of configuration commands labeled "Vsetup1".
• The "restfreq" value should be the sky frequency in the IF Manager; the "lo3freq" should be the "IF" in the IF Manager. The bandwidth in the IF Manager should be greater than or equal to the bandwidth in Vsetup1.
• The polarization for VLBI is almost always circular, and the sense (RCP or LCP) should be as in the above table.

• For hardcore users you may also :
• Inject a test tone at the sky frequency for each BBC; use a spectrum analyzer to confirm it appears at the right place in the BBC spectrum.
• Check the phase cal level (if requested).

Check the Logging system

  Log in to arcturus as 'gbvlbi' :
	$ ssh arcturus -l gbvlbi
	$ cd 
	$ cd mdata/MDA
	$ ls -lt | head

	Check that there is a '.mda' file named with today's date
	that has been updated within the last few minutes.

  Now check that this file is also being written to the jansky
	machine in Socorro:

	$ ssh jansky -l gbvlba
	$ lsgb

	(this will list the most recent files in the log area on jansky.
	the most recent file should have been updated in the last few
	minutes -- note time is Socorro is 2 hours less than Green Bank)

  Viewing the VLBA log file 
      Logged into "gbvlbi" on arcturus or vortex, 
	(Note "seelog" runs only on solaris machines)

	cd /users/gbvlbi/mdata/MDA
	ls -lt | head                 {get the name of the current log file}
	seelog MMMdd.mda | less   {lists the log in ascii}

	seelog MMMdd.mda | grep GPS  {check GPS data is being logged}
	seelog MMMdd.mda | grep WEA  {check weather data is being logged}

	-- if phase cals are being used:
	seelog MMMdd.mda | grep PCALX | less   {levels should be 2-5 %}

	-- Check Tsys
	  Look up the Tcals for the frequency being used.
	
	  seelog MMMdd.mda -cal tcal tcal | grep POWER | less

to top

Check Progress of Experiment.

  - in RSCREENS, use OBS-->DAS
  -            then: OBS-->FLAG

  - Use the printed *sch.gb  and  *crd.obs  schedule to follow the observations.

  - Check that the OBS-->FLAG screen is showing the telescope on source
    ("DATA OK") when it is on source and 
    also that it shows "Telescope position error too large" when off source
    or slewing between sources.

  - Every few minutes, check the IF power levels in the OBS-->DAS screen,
    and re-balance the IFRack if they go below 10000 or above 20000.

  About every half hour:
   - check that the tape is spinning when it is supposed to be,
	and that the footage is about right.
   - do CHECKALL (in the 'rs' screen)
   - do FLAG (in the 'rs' screen)
   - is the GBT on the right source?

Log weather conditions and other information.

  - Log unusual weather conditions that might affect the data, such as snow,
  heavy rain, and high winds.  The temperature, pressure, wind speed, and dewpoint 
  are being automatically logged, so no need to mention them.

  - Log any interruption in the observing schedule.

  - Log anything that compromises the data, such as a receiver warming up.

  - Include these messages in an e-mail that you send after the
	experiment is over.

to top

Dealing with Problems.

   A. Powering down 
	If the VLBA equipment needs to be powered down, there
	is no particular sequence that is required.
	DAR: turn off all the switches on the power supplies, which
		are near the top of the rack.

        -- Mark 5: refer to  Mark 5 Operations Guide 
	for powering the Mark 5 down and up.

   B. Powering up  
	Turn on the VLBA DAR rack by turning on all the DAR power supplies.
	Turn on the recorders by pushing the green button just to the
	lower right of the door.
	Turn on the power switch for the station computer
	  ("VLBAGB" in the rack to the left of the recorder)
	Turn on the power for the Linux machine "spock", which is
	in the rack to the right of the recorder.

   C. Logging in spock 
	Log in to spock using the login name "gbvlbi"

   D. Open a console window for the station computer 
	Open an xterm by pressing the right mouse button.

	In an xterm, type: 'minicom vlba'

	-- this opens the console for the VLBA station computer
	
	Hit the CR, and you should see  the  'GB->' prompt.

   E. Re-booting the station computer (vlbagb) 
	If power has been off, wait until everything is back on,
	especially machines 'vortex' and 'arcturus',
	then reboot vlbagb.

	If you see a 'GB->' prompt in the console window, type:  reboot 

	If you see '[VxWorks Boot]:'  ,  type:  @  

   F. Initialize the hardware 
	After vlbagb is up and running, bring up another xterm on spock,

	ssh arcturus

	and type 'rs' to bring up the screens package.

	To set the MCB interfaces, do  MCB-->SETALL

	For each of the two recorders, do  PARM-->RECPARM-->SEND

	Then re-sync the formatter 1pps to the site time 1pps by 
	  pushing the reset button on the Station Timer module.
	  (Lower left corner of the front of the DAR rack).
	  Hold the button down for 1 second.

   G. Clearing Formatter Errors 

	! NOTE: the following formatter errors are ok; !
	!    no action is needed if you see :          !
        ! 	Quality Analysis Module Error          !
	!	  Channel A Clock Error                !
	!	  Channel B Clock Error                !

	 1. in RSCREENS, FORMAT-->FMTERR screen, push the RESETMCB button.
        
	 if errors do not clear, 
	 2. in FORMAT-->FORMAT screen, reset SRATE, wait for formatter
		to finish configuring, then repeat step 1.

	 if errors still do not clear, CALL FOR HELP !

	 The following might help if you are desparate:
	 3. reboot the formatter controller single-board computer.
	    After that you have to do:
		MCB-->SETALL
		OBS-->REOBS
	    and repeat steps 2 and 1.

   Other weird stuff (part1) 
	You should mount the tapes after you reboot the station computer.
	If the tapes have already been mounted and there is some need
	to reboot, the station computer will pick up the tape VSN number
	from drive 1 during the reboot, because it has a bar code reader.
	It will not pick up the VSN from drive 2.  So you should 
	re-run the "Mount" procedure on the tapes after re-booting.

	Alternately, you can give the station computer the VSN numbers
	without having to re-run the mount command by typing at the
	vlbagb console:  tapetypeset(tapenum, VSN)
	for example, to set the VSN for drive number two: 
		tapetypeset( 1, "EVNT0262") 

   Other weird stuff (part2) 

	Sometimes the Mark5 disk packs have both a VSN and 4-digit CRCC code
	on them (you need both for entering in the TRACK system).
	But sometimes there is no CRCC code label on the disk pack.
	You can find a CRCC code for any VSN number by typing the "crcc"
	command to the vxWorks prompt on the station computer.
	For example, type:   crcc("EVNT0262")
	the reply includes several numbers, one of them is a hex number;
	that is the crcc code.

   Other weird stuff (part3) 
	When not using the Mark5 recorder, do the following,
	(logged into arcturus as gbvlbi: )
	
	rcmd disc stmode gb

	(this sets the Mark5 to pass 32 tracks directly through
	to tape drive 1.)

Dismount and Send Tapes to Socorro.

  - The schedule will usually dismount the tape after it is done.
    If not, then in RSCREENS, use TAPE-->MOTION screen to UNLOAD.

  - Make sure you put a red dot on the label and that the
	label information is filled out.
  - The station name is GBT_VLBA.
  - Make sure that the tape serial number is written on the outside
    of the case.  Put a note on the case that it should be sent to 
    Socorro, and put it in the mail room.  
  - Tell Sherry to give you the airbill numbers.
    After FedEx picks it up, get the airbill numbers and use the 
    TRACK system  to record the shipment.

  - (remember the TRACK login is 'gbvlba' with pw= track)

Send email to Socorro.

  Write an e-mail message describing any problems that happened during
  the run, missed scans if any, and the tape numbers.
  - address it to the PI and :  vlbaops, analysts, gbtops, gbtobssum.

Shut Down.

   - To shut down the SCREENS package, type "CNTRL-C A"
   - Shut down all processes and log out.

to top