GBT Operations & Commissioning Meeting 19 December 2003 AGENDA 1. Az Track and Telescope Status -- Bob 2. Observing News -- Karen 3. Frontends, IFs, and Spectral Baselines -- Roger 4. Spectrometer Status -- Rich 5. Software Status -- Nicole 6. Scheduling Status -- Carl 7. Project Planning Status -- John 8. Any other Business PRELIMINARY REPORTS 1. Az Track and Telescope Status Az Track: * The modified joint was re-inspected by ultrasonic methods on Tuesday. No indications were found in the weld. We also asked for an examination of the wear plate. Suprisingly, 2 small, long defects were found in the wear plate, running parallel to the line of the weld seam beneath, 5 and 7 inches from it, and about 1/4-inch deep from the top surface. A mechanism for this has not been identified yet. It is very unlikely that it was a manufacturing defect; it is running 45 degrees to the original hot rolling direction, and even if present, would have been found during UT at the plant. No indication of fretting was found on the new plate, although corrosion type pitting was found on the noses of the chevrons. * A torque check is being made on the bolts this week. * We can expect results from FEA model 2 any day. This is the midspan model of a wear plate, baseplate, and a load patch. * The 2 recently ordered wear plates have a target delivery of 1/31/04. We still have 3 on hand. Elevation Shaft * Still awaiting the report from Peter Keating. Wind Limits * Recommendations will be presented at the meeting. -- RAA 2. Observing News Projects: 8am Friday Dec 12 - 10am Friday, Dec 19: Q-band checkout, Dec 12 - O'Neil 8.5 hours (Qband, ACS/DCR) GBT02C-048, Dec 12/13 - Kondrakto 20.5 hours (Kband, ACS) GBT02A-063, Dec 13,14 - Claussen 6.5 hours (Kband, ACS) GBT03C-009, Dec 13-19 - Darling 36.5 hours (L/800, ACS) GBT03B-026, Dec 14 - Roberts 2 hours (l/800, BCPM) GBT03A-015, Dec 15 - Lane 8.5 hours (800/ACS) GBT03C-031, Dec 15 - Jacoby 8 hours (L/800 BCPM) GBT01A-061, Dec 15/16 - Lane 15 hours (800/ACS) GBT03B-016, Dec 16-18 - Walsh 15.5 (Kband, ACS) GBT02A-046, Dec 18 - Braatz 5.5 hours (Kband, ACS) GBT03C-041, Dec 18/19 - Ransom 6.5 hours (L/800, spigot) Total astronomy: 116.5 + 8.5 commissioning Total lost time: 15 (12%) + 7 (6%) for weather Problems: Front end receivers - Qband cal diode failed, Q-band call controller card has been replaced. Computer software - GO: - hung when talking to antenna due to "primary segments' error. Fixed when antenna was moved by operator (Dec 13 - Hung during observing, had to be restarted - total of ~1 hour lost - GO.fits files were not written for all of one night of J. Darling's observations. This was not caught since iards also wasn't working. Bob Garwood re-created the needed files and the data was saved. - GO/GO_LITE does not update the Delta_frequency, only the center frequency, when updating the velocity. This resulted in about 3 hours of lost time. - IF Rack power levels went to zero, and fixed by turning off and then on the IF rack manager - approx 10 minutes lost - Could not run the /home/gbt/gbt.sh script over the weekend - got "no such file" error for /home/gbt/bin/ygor.sh (fixed monday morning) - IARDS: - For 8 IFs, 1s dumps, aips++ takes 4-10 minutes to reduce 5 minutes of data, resulting in iards being unusable - Iards hung a number of times - DCR hung in aborting - problem handed to Mark, Tim, and Joe - LO1Router activate failed Computer Hardware: - /home/gbtdata disk filled up twice (two different nights & projects \) and observing had to halt until space was cleared. This lost a total of ~2 hours of observing. Note, too, that the cleo messages screen and GO did not report any problems in spite of being unable to write data. - gbtaio1 had to be rebooted regularly Backends - Spectrometer: - Spectrometer has reverted to having one of the chips 'dropping out' - CLEO spectrometer window froze and had to manually killed. This appears to be caused by leftover cleo windows from old observing sessions. When those are killed the spectrometer manager is fine. this occurred at least twice over the week, resulting in ~ 5-10 minutes of lost time. - Twice the spectrometer was in a confused state on start-up & had to be rebooted (~10-15 minutes lost) - DCR had 'missed tick' problem Other: - Cal values in the PF800 receiver file appear to be reversed - Notch filter in place without the obs, support (me) realizing it, which caused ~1 hour lost time. The new configuration tool Active Surface/Metrology: - Active Surface/Metrology failure on software (1 hour lost) - Reports of antenna time jumps of 100 - 400 ms all night on the 17/18 - Total lost time (from ops log): Weather 6.41 Front end receivers 2.95 Computer Software 3.36 Computer Hardware 1.13 (really should be 1.8) Other 0.69 Backends 4.00 Active Surface/Metrology 1.56 Not included in logs : spectrometer 'drop-outs' < 1 hour -- KO'N 3. Frontends, IFs, and Spectral Baselines 12/19/2003 Baseline, Receivers, IF Weekly Report Things were quiet with the GBT receivers this week. The only major effort was a rebuild of the Q-band receiver cal card in an attempt to solve the pesky cal problem in that receiver. This was completed and the receiver is back operational. There now are eight cold receivers, plus the holography front-ends, on the GBT. Rick's analysis on last week's baseline tests, done to evaluate the ACS sampler revisions, failed to show any significant differences between units with and without the revisions. The 26-40GHz receiver has now been successfully cooled, and testing is beginning. Initial integration tests with the M&C manager may begin next week. -- RDN 12/18/2003 4. Spectrometer Status *****Status***** A few problems to report this week 1 - There were occasional 1024-lag-long pulses in the correlation function. For one experiment a negative pulse was followed 2 integrations later by a positive pulse. These usually have been caused by the LTAs and are one of the reasons we want to redesign the LTAs. I've got some ideas about what could be going on in the LTA card to cause this, but that is beyond the scope of this report. The present configuration of LTAs has worked pretty well and I'm hesitant to shuffle them around in hopes of fixing this problem. Possibly, power supplies can be tweeked to improve the performance. It seems like it would be easy to identify and edit out the offending scans. Some plots will be brought in to illustrate this. 2 - One of the two spigot boards failed. It could not get a good xilinx load. The spare card worked! *****Development***** The FPGA design for the new LTA is complete and awaiting simulation tools to test it. Karen performed a day's worth of dynamic range testing with modified software courtesy of David Kaplan. The bottom line is that we can now accommodate "normal" system temperature variations and not saturate the system. It is not clear how steep a sensitivity price this exacts, however. Some sky testing will need to be performed to assess this. Karen may wish to add a word or two. *****Plans for next week***** Merry Christmas! -- Rich & Holly 5. Software Status Single Dish Development IPT #63 - Friday, December 19, 2003 Today ends week 5 of 6 in the current development cycle, which is the 9th and last in 2003. The Plan of Record for the current development cycle is available from the wiki at Software.PlanOfRecordC92003 along with the most up-to-date status information. An interim milestone was met last night in the control room! As part of the Ease of Use project, Mark and Amy were able to successfully slew to a location on the sky, and then track, using the new prototype Python modules that will become the new Observing API and GO functionality. There was a slight problem initially setting the coordinate mode, but all in all the tests went very smoothly. This work is truly a team effort, as most of the members of the software division have made valuable contributions to the activity, particularly over the past week. In particular, Eric provided valuable leadership on the design, Ray supported use of the standard M&C interface for this new purpose, Joe was consulted regarding proper antenna usage, and Melinda also provided support for some of the programming. It's highly promising that the design has been used to produce code so quickly and easily. The first round of integration tests was conducted on the 17th, and results were promising. The spectrometer had never before been tested on RH9, but performed well. The new version of M&C (v3.19) was cleared for regression tests to be completed by Monday the 22nd. Although the testing is being conducted now, the new software developed during this cycle will be released on 1/12/04, after astronomers return from their January conferences. Major enhancements pertain to the antenna, which will write new (but backwards compatible) FITS files, and the addition of dynamic focus corrections using temperature values. The program which will become the replacement for IARDS is now ready for preview; peak and focus are currently supported but not spectral line data analysis. There is a rudimentary module to analyze spectrometer data in place, though, so if you would like to participate in the review to help us determine what additions are necessary you are welcome to review this part as well. The name of the application is rtfm and the beta documentation is at http://wiki.gb.nrao.edu/bin/view/Data/RtfmBeta The port of gbtaio1 to Linux continues. This week, a test connection was successfully made to a GPIB test device, and the MR was reviewed for the remainder of the work. An alpha MCB server is also up and running, and some additional enhancements are scheduled to be completed today to light up an LED which will indicate that things are working as expected. Work was done to offload gregorian receivers onto gbtaio3, but as of Thursday the effort had not been completed. Joe also installed a program to monitor the number of connections between gbtaio1 and vortex, which is being examined often as work is being done in this area. Dish documentation is in the process of being overhauled, and is nearly complete except for the images that accompany the text. This activity will be tied up by the end of the first week of January, and can be viewed at http://wiki.gb.nrao.edu/bin/view/Data/DishUsersGuideForGBT on the wiki. Operational support occupied much of the SDD's work over the past week. Joe changed out the earth (spectrometer) disk, to add a RH7/RH9 capability, so the spectrometer will work on RH9. A similar effort was carried out for other new machines. When the PF1 receiver was reinstalled, Melinda restarted the samplers and verified cryo monitoring. Some additional support was also provided to get the first spiral and daisy scans running and tested. -- NMR 6. Scheduling Status Last Week ========= Observations for proposals GBT03C-009, GBT01A-061, GBT02A-063, GBT03C-041, GBT03C-031, GBT02A-046, GBT03A-015, GBT02A-069, GBT03B-026, GBT03B-016, GBT02C-048 Next Week ========= Observations scheduled for GBT03C-009 [B] PI Jeremy Darling A Direct Measurement of Fine Structure "Constant" Evolution from OH and HI Absorption Lines GBT03C-041 [B] PI Scott Ransom RRS Observations of the Double Binary Pulsar J0737-3039 GBT02A-065 [P] PI Lincoln Greenhill Detection of AGN in Apparently "Normal" Galaxies GBT02A-069 [B] PI Richard Fisher Galaxy Survey of HI emission GBT03B-015 PI Scott Ransom Timing the Pulsars in the Globular Cluster M30 GBT03A-016 PI Ingrid Stairs The Physics of a Massive Pulsar System GBT02C-058 [B] PI Ingrid Stairs Characterizing the Polarization Changes in an Intermediate-Mass Pulsar GBT03B-016 [P] PI Andrew Walsh A Complete Picture of Cluster Formation in NGC 1333 December ======== Scheduled hours [backup] Astronomy ~ 345 [75] Maintenance ~ 121 Tests & Comm ~ 279 [11] January ======= Scheduled hours [backup] Astronomy ~ 404 [295] Maintenance ~ 24 [106] Tests & Comm ~ 234 [19] Un-assigned ~ 83 February ======== Scheduled hours [backup] Astronomy ~ 264 [141] Maintenance ~ [141] Tests & Comm ~ 244 [18] Un-assigned ~ 188 Proposal Checkouts ================== Awaiting checkout - 0 Active checkout - 25 On Hold - 5 Schedule - 49 Completed - 124 -- RCB 7. Project Planning Status December 15th Planning Meeting Minutes ***** Rm 241 10:00 A.M. ***** [0] Observer comments & operator log items Lots of actrivity here. We should plan to spend a good deal of time discussing this. I will bring excerpts of observing reports to the meeting for discussion. We discussed 6 pages of email comments from staff observers. These were mostly from tests, but some from observing. The following actions came from it: 1) Ron's problem with GO hanging will be looked at by SDD 2) Rich Lacasse will look at the bad spectrometer scans (This is the known LTA bug that gives steps in ACF) 3) The active surface actuators will be repaired (Done!) 4) Config tool bug with alternate ODM's (Fixed!) 5) GBTAIO3 to be installed. (this was not completed) 6) Project ID's changed to "test" spontaneously. (We think this is a scan coordinator core dump and restart) 7) GO_LITE does not update the delta_frequency, only center frequency. This is to be investigated and proposed for new development. 8) Look at methods for determining when disk space is low. 9) gbt.sh file was lost from the computers. Look at changing permissions. [1] Resource calendar schedule conflicts and discussions Wednesday/Thursday conflict with moving telescope on as measurements and temperature sensor repairs. It was explained that only one of the 2 days are needed for moving measurements. [2] Observing Schedule discussions Dynamically scheduled maintenance days Bob Anderson will draft a policy for review by the planning committee. (done!) [3] GBT development planning Impact of software operation support on development plans Richard and John stated that the operational support must be used to fix the above problems, and to fix reliability issues. Even if delivered projects must suffer. [4] AOB Richard brought up the topic of the wind shutdowns. We discussed it a bit without any progress. -- JF 8. Any other Business