GBT Operations and Commissioning Meeting 08 October 2004 AGENDA 1. Az track, antenna Report -- Phil for Bob 2. Observing News -- Ron 3. Scheduling Status -- Carl 4. Frontends, IFs, and Spectral Baselines -- Roger 5. Spectrometer Status -- Rich 6. Software Status -- Amy, Nicole 7. Project Planning Status -- John 8. Any other Business REPORTS 1. Az track, antenna Report Bob, Dennis, and Art traveled to Brownsville, Texas this week to inspect a new radar antenna array that has wheel loading only a factor of two or so below that of the GBT. The azimuth track is a two-layer track similar to the GBT's, except that the wear plate is held down by lateral and circumferential restraints rather than bolts into the base plate. Bob will give a report on the visit as well as other track and antenna news next week. Phil 2. Observing News This week we supported seven different observing projects in nine observing sessions (a light week), Ka-band tests, K-band calibration, and a few hours of software tests. About ten hours were devoted to tests and 110 to observing. Average setup time was 45 minutes. The major loss of observing time this week came from a problem with the recently-released Spectrometer code (3 hrs) and user inexperience with the spigot (2.5 hrs). The remaining lost time was 1 hr from other Spectrometer problems and 1 hr from miscellaneous software and hardware problems (GFM, Prospero, IF Power levels, LO1/aio2, GO, ...). In total, about 7% of the time scheduled for observing and tests was lost. In other news, Mike Stennes, Roger, Chelen, and I are working on the K-band calibration problem that Jim reported a few weeks ago. We think we have good hot-cold tests which show the left polarizations of the K-band receiver has changed by about 50% from when they were last measured. Jim's observations suggest the diode values are off by this amount. We also have astronomical data that, when combined with the hot-cold data, will provide Tcal and Tsys values at 1 MHz resolution across the lower and upper band of the receiver. Next week, our workload will be high due to a significant number of projects and observing sessions. Brian and Frank will be working on Ka-band commissioning. -- RJM 3. Scheduling Status Last Week ========= Observations for proposals GBT02B-020, GBT04A-045, GBT04C-036, GBT04B-026, GBT02B-004, GBT04A-029, GBT01A-004, GBT04C-018 Completed proposals GBT02B-004, GBT01A-004 Next Week ========= Observations scheduled for GBT04C-009 PI Cintia Quireza Using HI absorption to resolve kinematic distance ambiguity of Galactic HII reigons GBT04B-040 PI Jeron Stil Re-Observations of Galactic Plane Clouds GBT02B-020 PI Dominic Benford Search for Low Excitation Molecular Gas in High Redshift Quasars (CO) GBT02C-043 [P] PI Douglas Finkbeiner Microwave Emission from Spinning Dust GBT04C-056 [B] PI Paul Demorest Precision Timing of Binary and Millisecond Pulsars GBT04C-013 [P] PI Bryan Jacoby Precision Pulsar Timing GBT04C-027 [B] PI Esteban Araya A GBT Search for H2CO 6cm Emission from Massive Protostars GBT04B-029 PI Ingrid Stairs Timing New Binary and Millisecond Pulsars from the Parkes Multibeam Survey GBT04C-018 PI Alberto Bolatto A Search for Cosmological HI Absorption Systems Toward Radio Selected Flat-Spectrum Sources GBT04C-008 PI Yurii Pidopryhora Mapping the Galactic Halo HI: Evidence of Outflow from the Galactic Plane? Scheduled hours [backup] ======================== Category/Month-> October November December January Astronomy ~ 373 [251] 425 [294] 425 [194] 425 [312] Maintenance ~ 91 [051] 30 [060] 28 [077] 0 [136] Test & Comm ~ 265 [053] 216 [063] 187 [044] 249 [062] Shutdown ~ 0 36 36 0 Un-assigned ~ 16 14 69 70 Proposal Checkouts ================== Awaiting checkout - 0 Active checkout - 11 On Hold - 5 Schedule - 63 Completed - 231 Current backlog [hours prior to 04C*] = 1168 Total time to discharge [hours] = 2565 * Includes projects that are on hold for trimester 04C -- RCB 4. Frontends, IFs, and Spectral Baselines Operational issues: Hot/Cold load tests and astronomical observations on Wednesday were used to get new K-band cal data. Data reduction is underway, but the weather was excellent and the tests appeared to go well. Baselines: Four of the converter modules in Rack B were removed for mods and bench testing. Several instabilities were located and corrected. This work will continue next week. Q-band: Fabrication of waveguide runs is underway. Ka Receiver: During the first run last Saturday, there was confusion about beam identification and the Sig/Ref phases. Galen has worked on the Sig/Ref control logic, and we believe the situation is straightened out. Two commissioning runs are scheduled in the next week. R. Norrod 10/7/2004 5. Spectrometer Status Spectrometer Report 24 September - 30 September 2004 *****Operations***** There were two significant spectrometer problems this week, both related to the installation of M&C revision 4.6. Early Saturday morning, persistent Data Interrupt Failures as well as self-test failures were traced to an old version of the "intAdapter" program running in the VME computer in the spectrometer. The job of this computer and software is to forward interrupts from the LTAs to Earth. It is not clear why the system worked for over 30 hours before crashing so spectacularly. When checking future software revisions, it would be worthwhile re-booting this single-board. A second problem had to do with missing programs for the spigot. Somehow something required for spigot observing always sneaks under the radar during a software upgrade, and we say we'll get it right next time. This week Nicole, with a little help from Rich, has written up a procedure to run a test spigot observation. Performing this test during integration testing should be adequate to insure that the spigot will indeed perform correctly following the next upgrade. Ramesh also had spigot problems. We found that these were most likely procedural ones: he was not aware of all the instructions that had to be executed for spigot calibration. In the process of investigating this problem, we found that there were several things that could be done to make the process less error-prone. They are on the list of things to do... Ramesh also promised to make some recommendations to Karen about the spigot instructions. New firmware in the Correlator Control Cards was required for the 200 MHz spigot modes. To make sure that this firmware did not adversely affect spectroscopy operation, test observations were carried out using a few randomly selected modes. One of the modes 50 Mhz, 3-level, 2 inputs with 2 phases, did not work correctly: one of the phases of one of the IFs always came out with all zeroes in the autocorrelation function for one phase of one IF. Fortunately this mode is rarely, if ever used. It is most unlikely that the problem is in the new firmware. Additional tests will have to be carried out to find and fix the cause of this most unusual failure. *****Development***** LTA replacement: LTA PCB design is finished. Final check of the design is about 50% done. I would like to have the one who has experience at PCB design to check the design as well, especially, check the size of the costumer designed package of the components before send to make the board. Cross-Correlation Test Fixture: The filter board construction is complete except for installing it into its box. The drawing for the filter box case is in the shop. The schematic for the combiner/output module complete; printed circuit design is continuing. Chassis design has begun. Spigot Mode Tests: (Report from Ray) Rich Lacasse and I tested spigot card modes last week and here is a summary of our efforts. 1) We have now tested all modes on the mode chart (except for mode 87 which we somehow missed). All modes have passed our engineering tests with the following exceptions: a. Modes 37 and 3B did not pass and we did not have time to find out why. b. All 16-bit and 8-bit modes with 512-lags or 256-lags failed. A spigot card FPGA change will be required to fix this problem 2) Since Scott had problems with mode 6, we tested it very thoroughly and it seems to work OK. 3) The new 2048-lag, 8-bit modes all tested OK and should be ready to use after Rich installs new correlator control card software. 4) All new 200 MHz modes tested OK (except A1-A4 and B1-B4 as above) and should be ready to use (again, after the correlator control card software update). *****Plans***** Continue LTA replacement and Cross-Correlation Text Fixture development. Test various upgrades to the spigot: 8- and 16-bit 512- and 256-lag modes testing Add FIFO to spigot interface Astronomical test of upgraded Xilinx personalities Test modes 37, 3B and 87. -- Rich, Ray and Holly 6. Software Status Single Dish Development IPT #101 - Friday, October 8, 2004 We are finishing week one of the seventh development cycle (C7) in 2004, which will run until mid-November. The Plan of Record is available from the wiki at Software.PlanOfRecordC72004. Work on the Observing API and the Turtle GUIs is being wrapped up, in preparation for a training session and release. This should happen within the next week or two. IDL work continues this week. Jim Braatz and Bob Garwood attended an IDL training this week in Vienna, which should greatly assist their efforts. Just a reminder to anyone interested in the IDL Data Container which is under development - it is documented at Data.IDLDataContainer and Data.IDLDataContainerTranslation. Anyone interested in data model development should follow the progress of these pages, and feel free to add comments. A strategy for unit testing has also been identified and is being put to use. The config_tool is being enhanced to handle "expert" keywords. It is expected that this new capability will be available next Tuesday, October 12. Please contact Frank Ghigo or Melinda Mello if you have any questions. Work continued this week on resolving issues with the Samplers/Transporter under Linux. This work includes some maintenance on the current code, but mostly focuses on redesign of the Sampler/Transporter. There is now a working version of the new Sampler and Transporter currently being tested, which uses the kernel module to connect the two together. So far, these tests have been encouraging as we can independently restart either the manager, or transporter and the whole system reconnects properly. However during testing, issues with the Accessor have arisen, which require further investigation. With regard to PTCS activities, Richard Prestage has tested the 'Model 4' residual correction engine in the antenna and now is seeing the behavior he expects. This week's operational support included: 1. Getting the receivers to reliably clear all messages when set to the "off" state. Getting the receivers to use the configuration files to determine their default state (i.e. on/off/standby). Fixes for these two issues are ready for testing and installation next Tuesday, October 12. 2. Ka Band commissioning support. 3. config_tool issues identified by Toney Minter. A fix is in place. 4. Installation of the new intAdapter code for the Spectrometer. 7. Project Planning Status October 4, 2004 Planning Meeting Minutes Rm 137 10:45 A.M. [0] Observer comments & operator log items resulting in lost time: Observer's comments received from: Tom Troland: Summary: Overall success of observations: Very successful Observations will result in a journal paper: Yes Description of success and problems: The runs were a success, only minor problems with winds. Description of RFI environment No problem The summary was followed by suggestions for areas needing improvement in the observing system and our infrastructure. Much discussion on this topic, with the basic conclusion that these concerns are being addressed with the ongoing api and data reduction projects. Converting to using the config tool would be helpful, and once the new observing api is released (in a year or so) the remaining concerns having to do with doing pointing and focusing will be addressed. Operator's log comments Significant failures the past week stemmed from: Computing: Prospero locked up Software: Spectrometer Manager failed, Scan Coordinator failed, GFM not updating, Spectrometer interrupt control failure, software release failure, BCPM Manager, GO lite, Spigot losing setup(?) Regression tests for the spigot software are being added. This should prevent a recurrence of the spigot software release failure. The problems with the spigot calibration were due to observer error. There were some steps slkipped in the calibration process. Electronics: Spigot losing setup(?) See above Telescope/Servo: HVAC failure Other Problems from Observing Reports: F. Ghigo: PF1 focus scan testing bugs. This was traced, we think, to a chenge in units wrought last January, but the PF system was using the old units, and obviously never was used. T. Minter: 1) gfm didn't reduce pointing data correctly 2) no way to tell GO to skip waiting for gfm 3) config_tool did not route signal properly 4) DCR not setup properly when using DCR_AFR in the config_tool 5) Mysterious scan start 6) GO confused over start/stop state 7) aips++ stable version changed during observing 8) receiver hung in running SDD is looking into some of these reports. Of particular concern are #7 and #8. As the aips++ system is essential to observing, it should only be changed during a maintenance day. The receiver hung in running problem is a new bug introduced with the latest M&C version, and needs to be squashed. Yurii Pidopryhora: There is a bug in the GBT software which results in a loss of scan. It happens when either you or the system aborts the scan during either "stopping" or "activating" phase (I'm not sure which). Today it happened after the next scan failed to activate because of an antenna error. Mark Clark has attempted to reproduce the bug unsuccessfully. It will be kept on the radar screen. [1] Resource calendar schedule conflicts and discussions Antenna Manager testing Richard will use time this week to check out the new pointing model problems. [2] Observing Schedule discussions Network outage tonight? Scheduled for 5 to 11 P.M. EDT. [3] GBT development planning Discussion on decisions from last week Deferred to next week due to a lack of time. [4] AOB None 8. Any other Business