GBT Operations and Commissioning Report 14 January 2005 AGENDA 1. Az Track, Antennae Report -- Bob 2. Observing News -- Toney 3. Scheduling Status -- Carl 4. Microwave Electronics Status -- Roger 5. Digital Electronics Status -- Rich 6. Software Status -- Nicole 7. Project Planning Status -- John 8. Any other Business REPORTS 1. Az Track, Antennae Report We sent a sample of 4340 to the lab late last week to determine physical and mechanical properites. We have ordered 8 additional wear plates. 2 are still in manufacture, and we have 1 on hand. We continue with track replacement planning, obtaining cost estimates for the various items suggested by the panel, and making comparisons. Art continues with analyses recommended by them as well. -- RAA 2. Observing News During the week of Dec 16 (7pm) through Dec 23 (7am) we supported 9 projects in 21 observing sessions. 142 hours were devoted to observing, 7 hours to PTCS, 2 hours to tests and 17 hours to maintenance. The average setup time was 47 minutes. Lost time amounted to 20% of the observing and test time. The causes of lost time were: 11.42 hrs : Temp. Below 1 F 5.69 hrs : Network slowness 3.62 hrs : Observer error (not zeroing rates) 3.23 hrs : Wind 2.70 hrs : Wheel Bearing warmup (after Temp. Below 1 F) 0.75 hrs : long setups, inexperienced observer 0.50 hrs : Active surface manager 0.17 hrs : vortex 0.12 hrs : Grail 0.10 hrs : ACS hung in stopping 0.05 hrs : Az brakes, circuit breaker 0.05 hrs : SP hung in stopping 0.04 hrs : ACS lag jumps 0.03 hrs : config_tool Highlights from this weeks observing include a very successful Mercury radar run. A very large amount of lost time was recorded this week due to the cold temperatures and the network problems. It was found that after a prolonged shutdown due to low temperatures that the antenna has to be exercised for several hours to warm up the oil in the Azimuth wheel bearings before normal operation of the telescope can commence. The network problems continued to make it very difficult to use standard utilities like Config_tool, Iards, GFM. Remote observers had a very difficult time with this problem. Some improvement occurred towards the end of the week. ------------------------------------------------------------------------------ During the week of Dec 23 (7am) through Dec 30 (7am) we supported 9 projects in 11 observing sessions. 109 hours were devoted to observing, 5 hours to observing tests, 15 hours to software tests, 12 hours to maintenance, and 36 hours to the Christmas shutdown. The average setup time was 35 minutes. Lost time amounted to 1.8% of the observing and test time. The causes of lost time were: 0.87 hrs : Wind 0.74 hrs : VLBA Tape Drive 2 stopped recording 0.20 hrs : Antenna stop trajectory failure 0.07 hrs : Az brakes, circuit breaker 0.05 hrs : DCR samplers not updating This week saw a light load due to the Christmas shutdown. Many VLBI observations were done. The amount of lost time was back to normal. The network continued to be noticeably slow, but not slow enough to cause any lost observing time as had happened in the previous two weeks. ------------------------------------------------------------------------------ During the week of Dec 30 (7am) through Jan 6 (7am) we supported 14 projects in 21 observing sessions. 139.5 hours were devoted to observing, 6 hours to observing tests, 15 hours to software tests, and 14.5 hours to maintenance. The average setup time was 43 minutes. Lost time amounted to 2.7% of the observing and test time. The causes of lost time were: 1.36 hrs : Az brakes, circuit breaker 0.85 hrs : galaxy survey had no available sources 0.65 hrs : long setup, setting up every pulsar backend 0.41 hrs : BCPM manager dead, slowed config_tool and GO 0.33 hrs : long setups, inexperienced observer 0.33 hrs : victor hungup 0.25 hrs : lost due to not following spigot scripts exactly 0.21 hrs : virgo rebooted itself, antenna stopped 0.17 hrs : spigot tic error 0.13 hrs : long setup, sources not up for spigot calibration 0.06 hrs : Spectral Processor hung in stopping 0.02 hrs : while spigot in use, 2nd person tried to copy data, spigot tic err Highlights included two sessions of target of opportunity VLBI/spigot observations for a Gamma-ray repeater near the Galactic center. A bad network switch was discovered to be the cause of the network slowness. This has problem has now been rectified and it is business as normal. ------------------------------------------------------------------------------ During the week of Jan 6 (7am) through Jan 13 (7am) we supported 17 projects in 26 observing sessions. All 168 hours were devoted to observing or observing tests. The average setup time was 42 minutes. Lost time amounted to 6% of the observing and test time. The causes of lost time were: 4.75 hrs : could not setup due to OR2->CM2 bad cable (1/4th data lost) 2.81 hrs : wind 0.50 hrs : unhooked cables from ODM6 to Converter Rack (undocumented) 0.40 hrs : Antenna moving on its own 0.29 hrs : fire needed rebooting 0.23 hrs : Antenna failed to produce stop trajectory (2 times) 0.22 hrs : GO/IARDS connection failed or glishd failed 0.19 hrs : grail failed during spigot setup 0.19 hrs : spectrometer did not configure properly 0.07 hrs : spectral processor hung in running 0.05 hrs : LO1 hung in stopping 0.03 hrs : bad lags in spectrometer This week saw a larger than normal number of observations. Toney managed to survive 84 hours of continuous support and does not recommend anyone else ever having to do this again. We had a network outage between Green Bank and Charlottesville but managed to avoid lost time by having the remote observer log in through the support person's Socorro account. We also had a spigot disk space crisis when the weather conspired to stack many spigot observations together along with several users who had not been moving their data off the spigot raids after their observations. Moving data off the spigot has become enough of an issue that we should start discussions on how to manage spigot disk space more effectively. -- TM 3. Scheduling Status RCB Note: I have added a section below for email addresses for PI on projects scheduled the week after next. Last 4 weeks ============ Observations for proposals GBT04A-030, GBT04C-040, BP116 , GBT03C-031, GBT04B-011, BG155 , GBT02B-020, GBT03A-011, GBT04C-022, BJ054 , GBT04C-023, GBT04C-056, GBT05A-016, GBT02C-033, GBT05A-005, GBT04C-013, GBT04B-014, GBT04A-005, GBT04B-026, GBT04C-058, GBT02A-069, GBT04C-036, BK121 , GBT04C-059, GBT04A-029, GBT04B-028, GBT04B-029, BB191 , BW076 , GBT04C-039, GBT04C-029, BH128 , BU027 , GBT04C-008 Completed proposals GBT04A-030, GBT04C-040, BP116, GBT03C-031, BG155, GBT03A-011, GBT04C-022, BJ054, GBT02C-033, GBT05A-005, GBT04C-013, GBT04A-005, GBT04C-058, BK121, GBT04C-059, GBT04A-029, BB191, BW076, GBT04C-039, GBT04C-029, BH128 Next Week ========= Observations scheduled for ** GBT05A-034 [ B] John Hibbard Ultracompact Blue Dwarfs: Galaxy Formation in the Local Universe? GBT04C-031 [ P ] Paul Kondratko Monitoring of Five NGC4258-like Water Megamasers Discovered with the GBT and the DSN GG057 [F ] Leonid Gurvits VLBI and Doppler tracking of the Huygens Titan Probe GBT04C-054 [ PB] Christopher Carilli A search for molecular absorption in the lensing galaxy toward the red gravitational lens J0134-0931 GBT04C-056 [ P ] Paul Demorest Precision Timing of Binary and Millisecond Pulsars GBT04C-036 [F B] Rajagopalan Ramachandran Searching for young pulsars in the Cygnus Super Bubble region GBT02A-069 [F ] Richard Fisher Galaxy Survey of HI emission GBT04C-015 [F ] William Cotton Continuum imaging of the very extended FRI radio sources 3C31, NGC315, NGC6251 GBT04B-029 [ B] Ingrid Stairs Timing New Binary and Millisecond Pulsars from the Parkes Multibeam Survey GBT04C-008 [ B] Yurii Pidopryhora Mapping the Galactic Halo HI: Evidence of Outflow from the Galactic Plane? GBT02C-017 [ P ] Alberto Bolatto Probing the Physical State of High Redshift Galaxies (CO) GBT02A-008 [ B] Morton Roberts A Study of the Hydrogen Reservoir Surrounding Galaxies ** Scheduled as F-ixed, P-rimary, B-ackup Contact Information for week after next ======================================= Proposal PI Bands Email ---------- ------------------ ----- --------- GBT04C-041 James Braatz K jbraatz@nrao.edu GBT04C-054 Christopher Carill UK ccarilli@nrao.edu GBT05A-015 Paul Kondratko K pkondrat@cfa.harvard.edu GBT02C-054 James Braatz K jbraatz@nrao.edu GBT04C-056 Paul Demorest L8 demorest@astron.berkeley.edu GBT04B-014 Paul Kondratko KU pkondrat@cfa.harvard.edu GBT04C-039 Geoffrey Bower X gbower@astro.berkeley.edu GBT04C-008 Yurii Pidopryhora L ypidopry@nrao.edu GBT04C-019 Ji-hyun Kang L kjh@astro.snu.ac.kr GBT02C-017 Alberto Bolatto U bolatto@astro.berkeley.edu Scheduled hours [backup] ======================== Category/Month-> January February March April Astronomy ~ 470 [203] 445 [250] 415 [198] 432 [108] Maintenance ~ 31 [068] 0 [135] 13 [143] 30 [111] Test & Comm ~ 222 [026] 196 [084] 262 [063] 205 [024] Un-assigned ~ 23 31 55 53 Proposal Checkouts ================== Awaiting checkout - 0 Active checkout - 15 On Hold - 8 Schedule - 62 Completed - 274 Current backlog [hours prior to 04C*] = 762 Total time to discharge [hours] = 2971 * Includes projects that are on hold for trimester 04C -- RCB 4. Microwave Electronics Status Operational issues: A weak 5MHz rail was seen in the S-band receiver during Huygen's checkout. Traced to the Phase Cal Generator (even though it was "OFF", and output port terminated in the TT Router). Disabled temporarily by removing its 500MHz reference. Ka Receiver - Design work continues on a new LO Multiplier/Distributor module. The split-block waveguide splitter prototype (version 2), was completed and tested. Performance is satisfactory, although Galen may attempt to improve it further. Shop fab of the prototype microstrip-waveguide adapter has been completed, and assembly in the lab is beginning. Assembly of 14 of the 32 planned detector video amplifier boards has been completed (includes enough for the 3mm receiver). 3mm Receiver - The initial M&C specification (sans calibration optics M&C) is completed. Mechanical drawings for the cardcage were completed and the shop is scheduled to start fab next week. Prep work for assembly of the matched quartz windows nears completion, and test assemblies will be started soon. A progress/status meeting was held 12/20/04. Other - An automated mixer test program is being developed in Labview. A preliminary version is in the lab being evaluated in testing of Q-band mixers. R. Norrod 1/13/2005 5. Digital Electronics Status Spectrometer Report 16 December 2004 - 13 January 2005 *****Operations, Problem summary***** No significant problems to report in the last 4 weeks! *****Development***** *LTA replacement: No report. Assume work continued on LTA firmware. *Cross-Correlation Test Fixture: The second filter module has been tested. The enclosures for the cross-correlation modules have been hung up in the shop since mid-Dec. They are not expected until early February. One of the boards is completely built. We will wait to build the second until the first is tested. Drafting work on the chassis is complete. This will now also have to wait for the shop. The front panel interface board is complete. *****Plans***** Continue LTA replacement and Cross-Correlation Text Fixture development. -- Rich 6. Software Status Single Dish Development IPT #110 - Friday, January 14, 2005 Today ends week 2 in C1, the first development cycle of 2005. The Plan of Record is available from the wiki at Software.PlanOfRecordC12005. Automated nightly builds and stress testing are now in place for IDL. New data columns for the index file as well as navigation methods for the index file have been created. The project documentation has been drafted to a state that is consistent with the rest of the project, and is located on the wiki at Data.IDLMain - all IDL documentation can be accessed through this main location. The root cause of the Pyro protocol error and the MMConverter problems has been identified. They are both due to a bug in the new Redhat Posix threading library. Setting an environmental variable disables the NPTL, so the workaround is straight-forward. Both problems will be fixed in the release after the completion of the VLBA run on Friday morning. PTCS work the week included created an agile service wherein M&C sampler values are available as ASCII encoded data streams from a port; see Software.ModificationRequest2C804 for more information. The new Daisy petal scan type in Turtle is completed and will be tested by Richard Prestage sometime next week. Work also continued on the porting of the remainder of Managers currently running on gbtaio1 (VxWorks) to Linux. In support of the porting activity, work was also done to gather data on equipment requirements for new Linux machines. Operational support over the past week included: * Network: Joe Brandt spent about a day assisting Chris and Wolfgang in evaluating the impact of a faulty gigabit card in the omni-NL switch on the network. -- AS 7. Project Planning Status January 10, 2005 Planning Meeting Minutes Rm 137 10:45 A.M. [0] Observer comments & operator log items resulting in lost time: Observer's comments received from: None. Operator's log comments Significant failures the past week stemmed from: Computing: Victor reboot Chris reports that this was an automounter problem that was likely triggered by rebooting of the network switches earlier. He will reboot victor from now on when a network switch is rebooted during maintenance, and this will likely prevent a recurrence. Software: Spectral Processor, iards/go link, reboot of fire, antenna manager gateway problem (did the operator really mean "MonCntrl@wind" (sic)? ), antenna trajectory error, spigot setup problem The "fire" reboot was caused by a filesystem problem. The antenna manager gateway problem is being looked at, and the other errors are understood, and are nothing new. Electronics: Active surface IIOP 3 warning, spigot setup, Spectrometer power levels No discussion of these. They are all known issues. Telescope/Servo: Winds Other: Spigot disk space got into a crunch this weekend. We need to revamp how we manage the spigot disk space. Other Problems from Observing Reports: None [1] Resource calendar schedule conflicts and discussions Frank needs the time he has signed up for in order to test out the Huygens hardware and software. Chris would like to exchange a bad board in the network switch. It will be done at 8:00 on the first maintenance day. [2] Observing Schedule discussions Some backup time is available Wednesday and Friday [3] GBT development planning CCC work over the past few weeks, and future plans See: http://wiki.gb.nrao.edu/bin/view/Projects/ChangeControlCommittee for details. John presented the salient parts of the minutes of the past 2 meetings. attendees were invited to visit the wiki for additional details. [4] AOB OD6 cables -- It was discovered this weekend that 2 of the output cables to Optical Receiver #6 had been disconnected and terminated. Nobody could be found who knew when this had been done. A log sheet and change procedure will be developed to help keep track of the changes to the cabling. -- JMF 8. Any other Business