GBT Operations & Commissioning Meeting 12 March 2004 AGENDA 1. Az Track and Telescope Status -- Bob 2. Observing News -- Ron 3. Scheduling Status -- Carl 4. Frontends, IFs, and Spectral Baselines -- Roger 5. Spectrometer Status -- Rich 6. Software Status -- Nicole 7. Project Planning Status -- John 8. Any other Business PRELIMINARY REPORTS 1. Az Track and Telescope Status Az Track * Long discussions with SGH on the results of Model 4 last week. Model 4 is supposed to model the modification. While it shows the fretting wear, and gives a peek at what is happening below the weld, the results presented did not show what was causing the defects in the wear plate. SGH is reperforming some runs, and we may get new results today or next week. * We also reviewed and discussed results from Model 6. Model 6 is a larger scale two dimensional model of the area around the hole through the root of the weld. Stresses are slightly below, or slightly above the AISC Code allowable stresses for fatigue, depending on the expected number of cycles. * We also reviewed and discussed results from Model 3. Model 3 is a solid track section tied to grout. The forces on the grout are at 50% of allowable, which is good. We asked for more resolution of the stresses in the one piece section. This will give us an idea of what would be necessary if we were to change to a piece of this design. * A crack was found in Plate 42 yesterday. It will need to be replaced next week. Structural Inspection * We were able to get a Phase 2 authorization letter from Procurement to Modjeski and Masters this week. We can expect a plan of attack from them for next summer's work in the next couple of weeks. * We are contacting suppliers to mobilize for repair of the elevation shaft, tentatively in May. Telescope Operation * We developed a fault in the large emergency generator yesterday during testing. A generator service rep is here this morning to guide our troubleshooting efforts. Bob Anderson NOTES: VLBI observers will be asked to expect the slower Elevation rates when they schedule their upcoming run. This is meant only as a fall back if the elevation drive replacement isn't done by the time of the run. 2. Observing News It's been a very busy week for support staff. We serviced 10 proposals, including a VLBI run, checked out two proposals, had a calibration run, and system checkouts after yesterday's power outage. In all, there were sixteen observing sessions. (The only things that were not up and running after the outage was the BCPM and about 90 actuators.) The number of failures this week were huge. We saw probably about five dozen instances of problems which I can organize into the following categories: o Winds: 8.75 hrs o Telescope Servo: 7.5 hrs o VLBI: 2.5 hrs due to schedule/tape-handling problems o Grail, GFM, Glishd, GO: 2.75 hrs o Spectrometer's typical problems: 2.25 hrs o Spectral Processor's typical problems: 1 hr o DCR software: 45 min o Network: 40 min o LO1 software: 30 min o Electrical power dips over the weekend: 30 min. o Active Surface power supply: 20 min About 6% of the time allocated to astronomy or tests was lost due to winds and 12% to problems that were hardware or software related. -- RJM NOTES: John suggested we need to improve our ability to diagnose servo problems 3. Scheduling Status Last Week ========= Observations for proposals GBT03C-009, BS131 , GBT03B-013, GBT04A-026, GBT02A-069, GBT02A-025, GBT01A-058, GBT02A-028, GBT02C-038 Completed proposals GBT01A-058, GBT02C-038, GBT02B-016 Next Week ========= Observations scheduled for GBT04A-030 [B] PI Ingrid Stairs High-Precison Timing of Binary Pulsars at the GBT GBT04A-045 [B] PI Mallory Roberts Timing of Three Binary Pulsars Discovered in a Survey of Mid-Latitude EGRET Error Boxes GBT02C-030 [P] PI Fred Lo GBT Search for Very Luminous H2O Megamasers GBT03C-031 [B] PI Bryan Jacoby Timing the pulsars in M62, NGC 6544, and NGC 6624 and Search for Ultra-fast pulsars GBT03B-011 [P] PI Susanna Widicus A search for sugars in hot cores GBT04A-026 [B] PI Yurii Pidopryhora Mapping the Galactic Halo HI: Correlated halo clouds and an HI plume GBT02C-012 [P] PI Mike Hollis A Search for the First Nucleic Acid Base Biomarker: Interstellar Pyrimidine GBT02C-002 [B] PI Christopher Carilli Redshifted HI 21cm Absorption towards Red Gravitational Lenses (J0134-0931, J1004+1229) GBT03B-019 [B] PI Di Li The GBT HI Narrow Self Absorption Survey of Star Forming Regions March ===== Scheduled hours [backup] Astronomy ~ 472 [237] Maintenance ~ 37 [85] Tests & Comm ~ 227 [43] Shutdown ~ 4 Un-assigned ~ 5 April ===== Scheduled hours [backup] Astronomy ~ 443 [168] Maintenance ~ [157] Tests & Comm ~ 220 [34] Un-assigned ~ 58 May === Scheduled hours [backup] Astronomy ~ 166 Maintenance ~ 88 [61] Tests & Comm ~ 64 Un-assigned ~ 427 Proposal Checkouts ================== Awaiting checkout - 0 Active checkout - 29 On Hold - 7 Schedule - 41 Completed - 152 Current backlog [hours] = 2520 -- RCB NOTES: Some discussion on how we can get more proposals available for Carl to schedule. There's a huge, 500-hr K-band proposal that might soon be moved to the "Schedule" list. Carl might have to use X-band proposals as backup. Scheduling more general proposal checkouts would help too. 4. Frontends, IFs, and Spectral Baselines Baselines: Outfitting of a dewar for baseline testing of Q-band components continues; fabrication of waveguide vacuum windows, SS waveguide, and dewar wiring all underway. I have requested some cryo HFET amplifiers from CDL for coaxial testing also. Some progress has been made on improvement of the Equipment Room temperature stability, by tweaking the PID controller parameters. Ka-band and mmConverter: RF testing of the breadboarded mmConverter and fabrication of filters and amplifiers for this system continues. Work orders for most of the mmConverter packaging have been submitted to the shop. Testing of prototype detector video amplifier is underway, with only minor design tweaking required to this point. Other: Frank questioned the S-band cal values, and last week we did a hot/cold measurement on the GBT that confirmed the values we have in the Tcal database are not correct. The calibration is not straight-forward because of the SDARS signals in the passband and other factors. We are still debating how best to address the issue. RDN 3/12/2004 NOTES: Installation of Ka-band is still scheduled for April 15. We still don't understand why Roger last week measured two different Trcvr for the same S-band channels when using different I.F. Rack detectors. The suspect optical drivers looked good when they were bench tested. 5. Spectrometer Status *****Status***** The spectrometer worked fairly reliably this week. Data interrupt as well as other failures seemed to occur less frequently. This may be due to recent code enhancements by the SDD. One exception was (again) Dana's test observations on Wednesday night. He was plagued with several Data Interrupt Failures. Since this was test time, it would not have hurt to have called in the troops. Spikes at the lower edge of the band in 50 MHz modes were traced to the Analog Filter Rack. These finding will be discussed with Norrod and Watts soon. We have made no progress on determing the difference between the 12/1/99 and 12/30/99 firmware in the Sample Distributor boards. We have two copies of each in the system. A similar problem cropped up with the spigot xilinx personalities. There are two and both seem to work. Ray will try to verify which is the latest and greatest. Some effort was made to understand the cause of intermittent serial errors. It was found that these come from the sample distributor boards, while reading duty cycles. *****Development***** This week, start LTA schematic design. currently, is building components, such as Xilinx, etc that are not in the library provided along with board design tool. The design of the noise source for the Cross-Correlation Test Fixture continued. The schematic is complete. Most parts are ordered. Board layout and enclosure remain to be designed for this module. A workable test procedure for the spigot cards in the system was defined and tested. The test uses both pseudo-random data to test for frame-to-frame bit-for-bit data integrity, and the Artificial Pulsar to look at something very similar to real-world signals. The Artificial Pulsar signals are analyzed with a variety of programs provided by Kaplan. The test procedure still needs to be cleaned up a bit. *****Plans for next week***** Continue design of improved LTA and Cross-Correlation Test Fixture. Clean up test procedure for spigot cards. Test correlator cards that were repaired last year. Update all spigot cards to the same revision of xilinx personality. Try to resolve the issue of the different versions of firmware in the Sample Distributor Cards. -- Rich and Holly NOTES: Ray soon will have some 200 MHz Spigot modes that can be scheduled for testing. 6. Software Status Single Dish Development IPT #73 - Friday, March 12, 2004 Today ends week 4 of 6 in development cycle C2 in 2004. The Plan of Record is available from the wiki at Software.PlanOfRecordC22004. Release documentation from previous cycles can be found at Software.SoftwareReportCentral on the wiki. Incremental work was performed this week on most of this cycle's commitments. Extensive testing on the ported Converter Rack and GPIB server has continued on live hardware this week, with excellent results. One of the tests was to ensure a clean recovery when rebooting after pulling the plug - the results were very good. Design work for the Caltech Backend receiver manager has resulted in class diagrams, sequence diagrams, a design for threading and callbacks, and a completed DDL. Michael will use these as guidelines to begin coding the CCB Manager next week. The code review report for Grail, the generic interface to M&C, was written and posted to the wiki by Ray. This will provide guidance for reliability improvements. Additional work was completed on the CLASS beta and its documentation, and Paul and Jim are now producing some spectrometer output. One interesting outcome is that CLASS can only digest a subset of GBT's spectrometer data. Antenna position calculations were changed in the SDFITS generator to be more efficient, and timing tests will be run over the weekend converting GBT projects into SDFITS. Work has commenced on completing the header information so that the files will be useful.   A meeting was held on Monday to discuss some high-impact fixes to Dish that will be done during the remainder of this cycle and the next. This should help fix some of the larger pain points associated with that package while reengineering is underway. Details can be found on the wiki at http://wiki.gb.nrao.edu/bin/view/Data/DishRequests SDD operational support this week included: + Patches added to M&Cv4.1 Weather2, updating the wind calculation to take into account new output voltage range. + Support for bringing software systems back up after Thursday's power outage + Virgo (machine that runs Antenna manager) replaced by another machine after it failed to come up after the power outage       -- NMR NOTES: Ron read the report since most of the programmers couldn't attend the meetings. There remains some questions as to what was meant in the report about the limitations of using CLASS on GBT data. An observer this weekend will try using CLASS but we're not making any promises as to what will or will not work. 7. Project Planning Status March 8, 2004 Planning Meeting Minutes Rm 137 10:45 A.M. [0] Observer comments & operator log items to discuss Operator's log comments Significant failures this week stemmed from: Spectrometer, Spigot, Spectral Processor, Wind, Power surges, software (GO, Grail), Servo, Active Surface, VLBI schedule fouled up. Note that the majority of Spectrometer and Spigot failures occurred early in the week before significant fixes were installed. There are still problems with the serial command stream. More work will be done on this system this week. We are making good progress on both Spectrometer and Spigot card. Spectral Processor failures are the known nagging problems. The wind was very strong over the weekend. Nothing we could do here, I think. The most time lost was due to strong gusts of wind, and not the "mean wind speed" algorithm. A couple of brief power dips caused some equipment to go insane, necessitating reconfiguration. Software faults were minimal. The servo system failures were aggravated by the software not giving clear indications of the fault. Seems that the Q band VLBI run was a fiasco from the start: weather, schedule problems, Converter rack manager. Observing Summaries *** Action Item: John to talk to Chris about what the problem with *** this machine may be. (actually it was the *font* server that *** crashed) Chris and Wolfgang need to be called in to witness problems and trouble shoot it. Phil informed us that the problem of slowdowns/lockups also happens on titania. *** Action Item: John to ask Chris to keep machine names the same *** when upgrading machines in the control/equipment rooms. Done. Chris is agreeable to this. Q band checkout, R. Maddalena Although today wasn't the greatest day for Q-band, I was able to check out three proposals, all of which look like they can be scheduled on the GBT. All of the data are in TQBAND_03MAR04. The support person for each proposal should take it from here (e.g., reconfirm my optimistic conclusions, reduce the data, send it off to the PI's, conferring with Carl about scheduling, etc.) 3C09, J. Darling, 450 MHz, spectrometer support by T. Minter, Mar 5 16:30 - Mar 6 5:00 Problem summary: ================ 1) 450 MHz receiver/Optical Drivers have lower than expected power 2) Power surge - spectrometer and LO1 went down 3) Spectrometer data interrupt failure Observing summary: ================== Jeremy will have to comment on how the observing went. He was again observing remotely. The new instructions for keeping the spectrometer alive after "stops" and "aborts" were "tested" and they all worked! 2C38, S. Gibson, L-band, 450 MHz, spectral processor Support: T. Minter, Mar 7 4:30-15:00 and 16:30-21:30 Problem summary: ================ 1) Spectral Processor: Accumulator output FIFO overflow errors 2) GO: RaLongMap and DecLatMap restart not correct when offsets used 3) Spectral processor did not get stop event from GO 4) Real time display flagging data when no flagging is desired 5) IF Rack not balancing 6) IF Rack starting late 7) config_tool changing converter rack attenuations? 8) Spectral Processor didn't write correct FITS file Observing summary: ================== The observations were to make maps around Cas A in the 408 MHz continuum, the HI 21cm line and 1420 MHz continuum polarization. Because of the many interuptions due to wind, snow, the spectral processor, etc. plus the map restart problems with GO, the mapping required an enourmous amount of human intervention. *** Action Item: Ron to organize an effort to find and fix *** mapping problems. GBT Observing Summary: G. Langston Chris Carilli's High Redshift H1 Abs. Data seem reasonably good, but target source is very weak. Details ======= Jeff Acre and I examined the RFI environment and found that the 80 MHz wide band pass filter worked well for his observation at 389 MHz. We found a very bright RFI line at 432.05, that is due to a Ham band transmitter on a mountain in VA. Jeff called Gary Anderson, who called the transmitter owner. They turned if off and removed that RFI source for the duration of the observation. Thanks to Jeff and Gary. GBT Observing Summary: G. Langston Shen's Q band observations of Sgr A Snow and high winds made Q band pointing very difficult. Pointing data seemed to be consistent with small pointing offsets. [1] Resource calendar schedule conflicts and discussions No conflicts for either day yet. Tuesday will be a maintenance day. Thursday will have a power outage from 8:00 to 11:00. [2] Observing Schedule discussions Carl has developed a WIKI page for showing the status, planned and current, of the telescope equipment. It is hoped that this will eventually grow to encompass all the receviers, IF, LO, Backends, etc. Carl mentioned the need for NRAO to notify parties to the coordination agreements that we have with potential interferers, including a new one for Boeing's Connexion service (A Ku band satellite based broadband data service offered from airliners). There was a short discussion followed by general agreement that this is a problem to be solved between Carl and the RFI group. [3] GBT development planning The March planning meeting will assist Phil in preparing the visiting committee report. The planning meeting will be held early next week to allow iteration of the report before the deadline. [4] AOB 1) Ron reported that the S band calibration test pointed up some possible problems with the IF rack and/or detectors in the IF rack. 2) Red Hat 9 is going out of support soon, and the computing division is deciding what to replace it with. In any case, we will need to carefully plan the migration to the new OS. As part of this, we need to modify/upgrade the LEDA library that the IF manager is based on. This code is obsolete and unsupported, so another solution is desired. 3) There is a Project Office review scheduled for next week. We will go through the project office bug reports and assign priority to them. People should go through the project office and see what is still there that may already fixed, or irrelevant. 4) John got assigned to coordinate the power outage recovery for Thursday. -- JMF NOTES: John reminded us of the "Project Office" meeting on Wednsday of next week. The causes of the continuum power dips from L.O. Doppler tracking was discussed. Either turning off Doppler tracking or turning on L.O. Blanking should work. Our experiences suggest that turning on L.O. Blanking isn't working, at least for the DCR, and we may need to set aside some effort to verify this. (Turning off Doppler tracking does work.) 8. Any other Business