GBT Operations & Commissioning Meeting 05 March 2004 AGENDA 1. Az Track and Telescope Status -- Bob 2. Observing News -- Ron 3. Scheduling Status -- Carl 4. Frontends, IFs, and Spectral Baselines -- Roger 5. Spectrometer Status -- Rich 6. Software Status -- Nicole 7. Project Planning Status -- John 8. Any other Business PRELIMINARY REPORTS 1. Az Track and Telescope Status Azimuth Track * No change in the crack in plate 35. * We received over 200 pages of results from SG&H this week from models 3 and 4. Team has been busy digesting them. Conference call with SG&H on Friday (tentatively). -- RAA 2. Observing News Over the last week we had nine observing projects, another night of holography, and a couple of sessions of proposal checkouts, including a few Q-band proposals. Q-band baselines look like they will be reasonable for various projects that are looking for narrow spectral lines. In my opinion, about 50 hrs of Q-band observing can be scheduled. We have two Q-band test periods next week which we will devote to testing how well the system will work for wide-width lines. We had no weather-related lost time this week. The lost time, which amounted to 6% of the 150 hrs of observing and test time, came in longer chunks than has been typical. That is, the number of incidents were fewer but lasted longer. The significant sources of lost time were: BCPM/Spigot (2.5 hrs wiping out project 4A45), GFM (1 hr), config_tool (0.75 hr), GO (0.75 hr), DCR (0.75 hr), Spectrometer reboots (a typical dozen incidents of ~10 min each), and Spectrometer (two longer incidents of about 1 hr each). In some cases, the lost time could have been shortened using 'standard' fixes but was extended to allow staff the opportunity to debug a problem. We don't have enough experience to say how much the recent Spectrometer software changes have improved the reliability of that device. We're crossing our fingers! -- RJM 3. Scheduling Status Last Week ========= Observations for proposals GBT02A-063, GBT04A-045, GBT03C-031, GBT03A-014, GBT02A-046, GBT03B-013, GBT02A-025, GBT01A-058, GBT02C-002 Next Week ========= Observations scheduled for GBT03C-009 PI Jeremy Darling A Direct Measurement of Fine Structure "Constant" Evolution from OH and HI Absorption Lines BS131 PI Zhi-quiang Shen VLBA Observations of Sgr A ? at 3 and 7 mm GBT03B-013 [B] PI Min Yun An Unbiased HI Survey of the Coma Cluster and Beyond GBT03A-014 [B] PI Jay Lockman Halo HI Clouds: Distribution and Properties GBT04A-026 [B] PI Yurii Pidopryhora Mapping the Galactic Halo HI: Correlated halo clouds and an HI plume GBT02A-025 [P] PI Mike Hollis A Search for the Next Interstellar Aldehyde Sugar: Glyceraldehyde GBT02A-069 [B] PI Richard Fisher Galaxy Survey of HI emission GBT01A-058 PI Frank Briggs High redshift Damped Lyman Alpha Systems against background Radio Galaxies GBT02A-028 PI James Braatz Exploring the Radio Spectrum of Orion A and W51 GBT02C-038 PI Steven Gibson Interstellar Matter in the Vicinity of Cassiopeia A (HI) March ===== Scheduled hours [backup] Astronomy ~ 464 [290] Maintenance ~ 20 [120] Tests & Comm ~ 257 [50] Un-assigned ~ 11 April ===== Scheduled hours [backup] Astronomy ~ 377 [8] Maintenance ~ [149] Tests & Comm ~ 123 [6] Un-assigned ~ 228 May === Scheduled hours [backup] Astronomy ~ 92 Maintenance ~ 88 [71] Tests & Comm ~ 58 Un-assigned ~ 506 Proposal Checkouts ================== Awaiting checkout - 0 Active checkout - 31 On Hold - 7 Schedule - 42 Completed - 149 Current backlog [hours] = 2619 -- RCB 4. Frontends, IFs, and Spectral Baselines Baselines: Gary and Jonah are outfitting a test dewar to use for in-depth tests of the front-end, to be used to attempt to identify the receiver noise temperature structure which we believe is major contributor to baseline structure on continuum sources. We anticipate this will be used initially at C and Q-bands. Ka-band and mmConverter: RF testing of the breadboarded mmConverter and fabrication of filters and amplifiers for this system continues. Initial results look good, although Mike and Tracy are investigating noise spikes that seem to be RFI related specific to the lab test setup. The prototype LO amplifier is now being used in the breadboard - bodies for additional units are in the shop. Work orders for some of the mmConverter packaging have been submitted to the shop; the baseplate design is in progress. All 7-14GHz , three of the 13-20GHz, and one of the 19-26GHz preselection filters are completed. The prototype detector video amplifier nears completion. Zotefoam vacuum windows were evaluated (vacuum properties), and a preliminary design arrived at. -- RDN 3/4/2004 5. Spectrometer Status *****Status***** There were a few bad experiences with Data Interrupt Failures during the past 10 days or so. Many intermittent serial errors were also reported. The recent spigot observations have been unsuccessful. Other than that operation has been routine. Spurred on by the above-mentioned Data Interrupt Failures, both the software and hardware groups put in quite a bit of effort to eliminate these errors. Brandt and Ford reviewed the spectrometer code and found some coding errors which may have contributed to this problem. Fixes for the errors were patched in. Similarly, some coding errors were found in the serial communication routine and these were patched. Holly is working on a minor upgrade to the LTAs to improve the reliability of their serial communication link. The Data Interrupt Failure message merely indicates that the control computer expected data and none showed up. Many types of errors could cause this error. Clark, with input from quite a few folks, has greatly improved the "Spectrometer Recovery Procedure". The document now provides common nomenclature for various procedures. If the operators follow the procedures outlined in this document and document their actions using the terms defined in the procedure, it will assist in diagnosing the cause of failures. The problem with the pulsar spigot was traced to old firmware which managed to find its way into a quadrant containing a spigot card. A thorough review of the firmware revisions on all spectrometer boards was initiated. Several cards were found to be out-of-date. All Correlator Control Cards have been upgraded. There is an outstanding issue with firmware in the Sample Distributor boards which should be resolved on Friday. The system was tested Thursday with a lot of help from Kaplan and Ghigo by looking at a real pulsar. The pulsar came booming in! *****Development***** The timing simulations for the upgraded LTA board are nearly complete. Board design should begin next week. Design of the Cross-Correlation Test Fixture continued. In particular the printed-circuit layout of the noise source is in progress. *****Plans for next week***** - Continue design of LTA upgrade and Cross-Correlation Test Fixture. - Finish (hopefully) the spigot card test procedure. - Get the Sample Distributor cards to the same, correct, firmware revision. - Test a few correlator cards that have had their heat sinks repaired. -- Rich & Holly 6. Software Status Single Dish Development IPT #72 - Friday, March 5, 2004 Today ends week 3 of 6 in development cycle C2 in 2004. The Plan of Record will be available shortly from the wiki at Software.PlanOfRecordC22004. C1 culminated with the release of M&Cv4.1 on Tuesday afternoon; release documentation for M&Cv4.1 is at Software.SoftwareReportCentral on the wiki. New Sparrow functionality is also under development as part of the Observing API. This week, Mark and Amy are adding the production-quality interface to the control system, to support tests on the live antenna (December tests were done with a prototype interface). These procedures will be handed over to Frank next week for those live tests, as Amy and Mark complete coding on the FocusSubReflector procedure. Extensive testing on the ported Converter Rack and GPIB server has commenced on live hardware this week, with excellent results. A block of testing is also underway today. Work on the Converter Rack & Analog Filter Rack port has helped to uncover issues in the Spectrometer code as well, which is helping operational support efforts this week. In other news: Design work for the Caltech Backend receiver manager has commenced. Some small tasks for GFM were completed and pushed into the test version of the program this week. To help in planning out Dish Reengineering, Bob completed a description of functionality and how it fits together in the current product at Data.DishDesign. Jim also started updating the regression tests to include all the new and prototype Python products, including GFM, and recording this on the wiki at Software.PerformRegressionTests. SDD operational support this week included: + Operate.SpectrometerRecoveryProcedures was reviewed and updated + Mark worked extensively with electronics on spectrometer troubleshooting related to data interrupt failures + Patches added to M&Cv4.1 to aid in resolution of recent spectrometer issues. First, Joe revised timeout calculation in ConditionVar.cc and RealInterrupt.cc - problem was that the values were overflowing, and the equation was just plain wrong. Also, in Quadrant.cc calls to condition_signal were outside of the 'mutexed' code, which is incorrect. + Melinda found and fixed some errors in the Active Surface unit tests -- NMR 7. Project Planning Status March 1, 2004 Planning Meeting Minutes Rm 137 10:00 A.M. [0] Observer comments & operator log items to discuss Action item review: partial bad device flagging Nicole reports that this will take 1/2 person for a development cycle. Operator's log comments The following are what I gleaned from the logs. I think the significant ones are the spectrometer failures, grail lockup,antenna manager fits file failure, GFM failure, and the servo faults. (the GFM and antenna manager failures were one and the same) Spectrometer: Data interrupt failure increasing in frequency. DCR: One failure, reboot fixed it. Spectral Processor: one instance of racks not getting cleared Miscellaneous unconnected software failures: Cleo permissions problems Manager lockups Naiad lockups Antenna manager FITS file failures Config tool lockup (Grail?) Servo faults Observer's comments: GBT Observing Summary: G. Langston Min Yun's HI observation of the Coma Cluster Observations went well. GBT Observing summary: T. Minter GBT03B-015, S. Ransom 1) Naiad file server crashed 2) Naiad was slow due to cleo/segeste The observing went smoothly. *** Action Item: John to talk to Chris about what the problem with *** this machine may be. (actually it was the *font* server that *** crashed) *** Action Item: John to ask Chris to keep machine names the same *** when upgrading machines in the control/equipment rooms. GBT2A63, A. Wootten No problems. GBT Test Summary: R. Prestage 1. Fix to GO_LITE to allow exiting from a repeat loop has apparently broken the ability to reliably use GO tables. We request this feature be removed until it has been more reliably tested. 2. GO_LITE does not appear to be able to control the Q-bamd beamswitching transfer switch. 3. Spectral Processor accumulator overflows (known problem). GBT Observing summary: 02A_025, P. Jewell Our observations on Sunday for program GBT02A_025 went pretty well on the whole. Data quality is good and my colleague, Mike Hollis, and I are both impressed with the sensititivies we are reaching. We did lose 2.5-3 hours out of 14 yesterday for various problems, although fortunately, they did not occur during our most critical source time. The problems were: 1) DCR problem 2) Spectrometer power levels This is the known power slope problem 3) Config tool This could be related to the Naiad problems 4) GFM [1] Resource calendar schedule conflicts and discussions Holly needs 4 hours for spectrometer testing We decided to redirect the spectrometer time on Tuesday for investigation of the "Data Interrupt Failure" problems. [2] Observing Schedule discussions Action item review: Getting more backup scheduled We discussed Carl's mail to the group, and decided that we should try and get a few projects with many hours out of the checkout queue. There are several LF projects in there that may provide short-term relief from the problem. Action item review: Frank Ghigo's plans for calibration Frank sent out a plan. Ron needs to make sure that enough time is allocated for the work to get done. [3] GBT development planning No activity [4] AOB Action item review: Spectrometer reanimation instruction update The operations group has the latest on their page. It's right, I think. http://wiki.gb.nrao.edu/bin/view/Operate /SpectrometerRecoveryProcedures This page has been extensively reworked, and I think that everyone involved is comfortable with it. Redirecting the Monday meeting We are going to revise the format of this meeting to cut down the length. We will not try and solve problems, rather we will assign responsibility and have the responsible parties get back to us with some ideas. The meeting time will be 10:45 to 11:45, and the agenda will be revised. Support Scientist Scheduling Support Scientist schedules will be tied to the 6 week cycles. Summer receiver maintenance schedule: Roger will provide a rough schedule of when we wish to remove receivers over the summer. Q and Ka band will be removed when the weather goes away, X band when the MM converter modules are completed, and the others can follow. Meeting minutes These minutes will appear on a WIKI page as soon as I get my ducks in a row. The page will include the action items, and action item owners will be expected to fill in the details of their action items. -- JMF 8. Any other Business