GBT Operations & Commissioning Meeting 9 May 2003 AGENDA 1. Az Track Status -- Bob A. 2. Spectrometer status -- Rich 3. Commissioning and Observing Checkout News -- Ron 4. Spectral Baseline, Front-end, and IF work -- Roger 5. Software status -- Nicole 6. Data reduction report -- Bob G., Steve 7. Project scheduling -- John 8. Observing schedule -- Carl 9. Any other business REPORTS & DISCUSSION 1. Az Track Status * Lee King continues with the flex plate analysis. * Lab results were received for the gray powdered material that migrated up through some of the splices.  It was 90% zinc metal. This indicates that the material failed due to fretting rather than electrolsys or chemical oxidation. * Preparations continue to make the shop welding demonstration the week of May 19, and the field trial the week of June 2. -- RA Discussion. Bob added that he and Dennis are planning to travel to Savannah for the shop weld to begin on the 19th. The teflon shim material appears to be winning out as the material of choice. 2. Spectrometer status Microprocessor development tools for the spectrometer microprocessors were researched. A development system from Keil Software was purchased. This will allow for the testing of code in a simulation environment instead of requiring the use of the spectrometer hardware. A lot of time was spent trying to find the cause of an intermittent failure in LTA S/N 2. No luck - it fixed itself again. However, it was gratifying to find good timing margin in the areas that were tested. Design of the hardware reset continues. Most of the effort has concentrated on understanding the system monitor firmware, since it will need to be upgraded as part of this project. During this process a potential bug in the firmware which might cause system shut down was uncovered. (In some cases the A/D which checks power supplies and rack temperatures may be read before it is ready.) This will be double-checked with Escoffier before a modification is made. A new hypothesis for the cause of pulsar spigot data phase jumps was formulated. Attempts to test it were unfruitful due to changes in Kaplan's software. With additional instructions from Kaplan, the hypothesis will be tested on Friday. The LTA schematics were updated to reflect some recent modifications related to synchronizing the pulsar spigot. Work continued on the LTA firmware upgrade which is intended to increase it's reliability. System logs have been studied to try to glean some clues about recent failures. Plans for next week (comments appreciated): - continued work on firmware upgrades - continued work on hardware system reset - fix problems as required -- Rich and Holly Discussion. John noted that most of the crashes he has looked into appear to be the "Quadrant 0 got no data" problem. Rich felt that his work to research and implement a hardware reset should be completed in the next few days. 3. Commissioning and Observing Checkout News This week's report: Commissioning and Observing Checkout News The commissioners have been involved in a number of tasks this week: o Proposal checkouts -- Karen, Glen, Frank and I are collecting information from the authors of the latest round of proposals. Some time has already been spent with the telescope on checking out older proposals. o Observer support -- Noticeably less than previous weeks due to the smaller number of proposals on the schedule. o Ease-of-Use -- Frank, Karen, and Toney will complete by the end of next week their specification document for the interface for configuring the GBT hardware. Some of this has already been prototyped in Frank's GLISH configuration scripts. o Spectrometer checkouts -- Karen is on the Thursday-night schedule for mode checkouts and we'll know how well things went by the time of the meeting. o "Traditional" holography -- Steve and I spent three nights resurrecting the holography system. We had a problem with the L.O. going out of lock when we pointed at strong satellites, something Steve was able to fix. We still have saturation problems in the block converter. Possible remedies are: placing absorbers in the waveguide, changing to the low-gain block converter that we used on the 140-ft, or finding a weak satellite. When we point away from the satellite all is OK and I have used phase measurements to predict the expected accuracy of the maps we can make. Under benign weather conditions, my tests indicate we can achieve an rms noise of 70 microns with a spatial resolution of 0.5m in about four hours. I won't know about systemic errors until after we have a few maps. My attempts at testing the system on methanol masers indicates we will need to convert the correlator into its wideband mode for measuring the surface at elevations away from the rigging angle. I've taken a few 64x64 and 100x100 maps to help reblaze the data path and to test various thoughts I have of overcoming the saturation by some analysis legerdemain. -- RJM 4. Spectral Baseline, Front-end, and IF work The S-band receiver was installed on the GBT on May 7, and engineering checkout time has been reserved for next Monday during maintenance. There are now eight cold receivers on the GBT. Good progress was made this week in searching for the cause of 60MHz baseline ripples within the IF system. We can now eliminate these (apparently) by terminating three of four Optical Receiver output ports. This indicates some interaction between Converter Modules through the four-way splitter within the Optical Receivers. The task now moves toward finding an acceptable solution. Baseline tests were run this week with the C-band receiver. We found that water droplets on the feedhorn radome (left from a day's showers at access) had a significant affect on the receiver noise. Tsys went from about 40K to 18K when the radome was wiped off. The water also introduced significant ripple in the Tcal/Tsys spectra. Spectral baselines were also measured with various sized metal objects lying in the center of the radome. A 2.5 inch diameter disc (having area about 1% of the feed aperture area) induced baseline ripples up to 7% p-p. A dime (having area about 0.1% of the feed aperture) induced baseline ripples about 0.5% p-p. These tests were undertaken to better understand how reflections within the feedhorn - LNA region affects baseline stability. A full report of these tests is in preparation. -- RDN 5/8/03 Discussion. Roger showed spectra with the optical receiver ports connected and terminated. The ripple is quite strong when connected normally and is flattened when 3 of the 4 ports are terminated. Roger also described in a bit more detail the problem of water droplets on the feed radomes and some possibilities for reducing the problem. 5. Software status Software Development Division #35 - Friday, May 9, 2003 We are ending week 2 of a 5-week development cycle, which will culminate with the release of M&C v.3.14 on 6/4/03. The plan of record which shows goals for M&C v3.14 is available at http://tryllium.gb.nrao.edu/images/POR_May03.pdf. Key tasks this month include finishing up software work on the spectrometer in response to the recent platform change, a batch of active surface items in support of PTCS, continued EMS development and analysis tasks including review of GBT configuration algorithms from a control systems perspective. 12 out of 15 MRs have been written and are approved. Of the 12 approved MRs, 6 are under development and 6 are ready for internal testing (this includes all EMS-related tasks). The first trilateration experiment using EMS is on schedule to be supported by 5/16/03. As a result of applying a kernel fix to the new spectrometer platform this week, there are direct benefits for observers as well as engineers. If the spectrometer crashes and Taskmaster restarts it, the user no longer has to reboot the spectrometer. If the spectrometer hangs, the user should be able to normally stop and start the device with Taskmaster. Interleaving the data from different quadrants is now complete and tested, but this will not be made available on the spectrometer to allow slightly higher data rates until further electronics work is done to stabilize the device. Applying this improvement to the system may take place with the regular release. The Linux Migration Project Charter has been completed and is awaiting sponsor review, before it will be used to guide migration activities for the remainder of the year. Knowledge transfer for structural vibration monitoring, which was conducted last week, will be followed up by a summary report to close the task next week. Also, the SDD will be holding a tutorial on FQL (FITS Query Language) during the week of May 19-23. Please send Nicole email if you wish to attend. -- NMR Discussion. Nicole added that for work on the Spectrometer, about the only development work left to do on the software side was to improve the communication between the hardware and software. The trajectory preprocessor work is scheduled for completion by the end of June. Rich asked if the self-test code for the Spectrometer was planned to be ported to the Linux computer. Nicole replied that that was scheduled for completion the week after next. 6. Data reduction report Bob G. reported that a new release was scheduled for the end of the month. He also asked that when glishd must be restarted, that as much information as possible be reported. A email with more information on this was sent following the meeting. Steve noted that the data reduction group is continuing to work on existing targets and defect fixes in Dish. Further development work is on hold pending the outcome of discussions on data analysis reorganization. 7. Project scheduling May 5th 28th Planning Meeting Minutes 0) Observer comments AGBT02A_063 A. Wootten Al's major problems were with wierdness in GO, and woes in data reduction. Quitting and restarting programs was the ticket out of the morass. 1) This week's schedule Approved all scheduled activities 2) Next week's schedule All but possibly Thursday's spectrometer tests were approved. We may substitute spigot card testing for that day. 3) June Observing Schedule discussions Proposed shutdown a) ALMA tests b) power outage We decided to firmly schedule the ALMA test time and the power outage. The track experiment is still scheduled to happen these weeks. The power outage will be scheduled with the power company by Butch Wirt for the Saturday the 7th. ALMA LO testing will be done during the week in the daytime. c) During the June shutdown, we will attempt to do much of the cable and mount installation of the temperature sensor system. The boxes themselves will not be ready to install at that time. Maintenance time will be scheduled for 5 days, 12 hours per day. 4) July Observing Schedule discussions The 8th through the 21st are being held open for the track experiment, should the contractor fail to be able to get things ready by the first of June. 5) PTCS Temperature Sensor Installation Overview The temperature sensor installation was reduced in scope, and the installation techniques re-engineered to yield an affordable plan. Details upon request. 6) 6 month plan review Scheduled for Thursday morning, 10:00 8) AOB A) Receiver project updates: W band -- shelved until next year Ka band -- progress OK. CDL support forthcoming, Galen finishing up baseline stuff Penn Array -- Software condition red, packaging problems (weight, bulk) We will be having a meeting to assess our position on all these things. K band -- We will do the minimal amplifier upgrade. B) Observing problems. 1) FITS files were written into the wrong directory. The cause of this is known, and the solution is being thought about. 2) The trajectory preprocessor problem was mentioned. Richard replied that the problem is well known, and is on the schedule to be fixed, and that the workaround should be more highly publicized. 3) The C band receiver is installed. Questions were raised over the amount of testing needed befor observers come to use it. C) John is still working on the spectrometer improvement project organization. Al Wootten's comments: Description of successes and problems The system operates much more smoothly than in December when I was here last. Despite the rain and clouds on the second night we were able to see lines over the 5 Jy in strength necessary for successful VLBA imaging, and hence to cull currently inactive sources from the candidate list. Software On the second night, GO did not start successfully; Clark and friends manually created a GO directory. iards would not work automatically and was abandoned. Despite thunderstorms and wet conditions, we managed to survey most of the target sources. Dish continued to baffle. Example of frustrations: Reduced a spectrum and showed in on screen with LSRK Radio velocity. Clicked on the Gfit button and got my window. Unaccountably, the horizontal axis reverted to frequency. I need VLSR but could not figure out how to get that. Conclusion: using Gfit is vexing and the result is unuseable. Tried adding new scans to the ms: - d.import(\'/home/gbtdata/AGBT02A_063_02\',\'yso\',,105,164) LocalExec::SetStatus: abnormal child termination for /home/aips++/stable1/linux/bin/sditerator SEVERE: Could not open /users/awootten/OH/yso_SPECTROMETER_A as a working set T - d.lscans() SEVERE: Bad/Unselected sditerator: Use filein and try again F Since it said it could not open the ms I tried to open it using - d.open(\'yso_SPECTROMETER_A\') which completely hung dish. so I did control C and - d.files() Current filein is : not set Current fileout is : averageSpectra Hmmm. I tried - d.open(\'yso_SPECTROMETER_B\') which also completely hung dish. Conclusion: This is also vexing. I guess I have to create another new ms? So...rm -r on the three ms in my directory...after which d.import(\'/home/gbtdata/AGBT02A_063_02\',\'yso\',,37,184) still, dish is hung. Control C gets me a prompt from which I attempt to exit. That worked. Upon rebeginning dish, I opened the spectrometer dataset mentioned above successfully. Baffling behavior. Services Those with overnight observing should be bedded in the rooms away from the construction. The black blinds are sorely missed! -- JF Discussion. John added that discussions Friday morning with the Penn Array team had been productive and that a number of issues had been resolved. 8. Observing schedule Last week ======= Observing for: GBT02C-005, GBT02A-031, BR085, GBT03A-001, GBT02C-019, GBT01A-014 Completed: GBT02C-005, GBT03A-001 May === Astronomy ~ 347 hours Maintenance ~ 134 hours Tests & commissioning ~ 264 hours June === Shutdown ~ 260 hours Astronomy ~ 162 hours Maintenance ~ 157 hours Tests & commissioning ~ 72 hours Un assigned ~ 70 hours July === Astronomy ~ 188 hours Maintenance ~ 218 hours Tests & commisioning ~ 31 hours In reserve ~ 228 hours -- RCB Discussion. Carl noted that the 12 hour maintenance days for painting and structural inspections are scheduled to begin on June 30. Bob A. noted the painting will begin sooner than that, but will be confined initially to stairwells and walkways. Once the structural inspection team arrives, the painters will follow the areas they have inspected. 9. Any other business Roger noted that the 2.4 MHz ripple from the fiber optical modulators is buried below the 60 MHz ripple, so if the latter is fixed, the former will probably be visible. In response to a question, Bob A. described the structural inspections that will be done this summer (box structure, horizontal feedarm, and vertical feedarm). The alidade and half the backup structure will be done during summer 2004. PRJ 12 May 2003