GBT Commissioning Meeting 25 April 2003 AGENDA 1. Az Track Status -- Bob A. 2. Spectrometer status -- Rich, John 3. Commissioning news -- Ron 4. Holography Results -- Claire 5. Spectral Baseline, Front-end, and IF work -- Roger 6. AIPS++ report -- Joe M. 7. Software status -- Amy 8. Project scheduling -- John 9. Observing schedule -- Carl 10. Any other business REPORTS & DISCUSSION 1. Az Track Status The results of the telescope weighing have been compiled, and the total weight is 16,727,000 pounds. The total error is +/- 30,000. The highest load of any of the 8 wheel sets is 2,199,348 pounds. This the load that will have to be considered in the analysis and performance of track components, wheel assemblies, flex plates, etc. A full seminar is scheduled right after this meeting to present the methodology and results. Continental Field Systems has had a delay in getting the tooling to cut the weld grooves. The shop welding demonstration has been delayed, possibly as late as the end of May. The field weld would slip accordingly. We and the contractor are looking for ways to make this time up. A preliminary work scope has been sent to them to begin estimating. Data taken from the bolt strain gauge test has been reduced, and will be presented at the commissioning meeting. -- RA Discussion. Bob added that the samples from the cracked plate are at the metallurgy lab and we may get some results from their tests by the end of next week. The bolt strain gauge results show that one side of the bolts are being unloaded by as much as 5000 lbs. Since June of 2002, there have been 21,000 wheel passes over a given bolt. This suggests that the repeated loading and unloading provides a fatigue mechanism that could account for the breakage. The team is looking at two approaches to mitigate this fatigue. One is to provide a better method of torquing the bolts to uniform values. The other is a possible design modification that could be implemented as part of the track project. Bob noted that one problem is bowing of the wear plates across the short dimension -- the bolts cannot pull this down. Ron asked if the increased astronomy usage of the GBT would add to the wheel passes. Bob noted that this might be possible, although we have put a lot of wheel passes on as part of maintenance and commissioning. Bob is striving to reduce the wear during maintenance periods. Dave asked if an acceptable strategy for the bolts is simply to plan to replace a certain number of them each year. Bob replied that this might indeed prove the best strategy, although the bolts are $5 each and there are 1728 bolts around the track. 2. Spectrometer status The M&C group provided new messages for failed Xilinx loads. These messages identify the board that caused the problem. These messages were tested by slight board modifications which forced bad loads on certain boards. The messages for the LTA's worked the first time. Testing of the Memory Boards showed that the test for a bad load was faulty. This was repaired and re-tested. The second iteration of the above test worked so well that it identified a real card problem. The card was tested on the bench and worked fine. It was then replaced in the system and worked well there also. Another intermittent problem... The multi-integration problem (spikes in acf after the second integration in a scan) in LTA S/N 8 was corrected by adjusting its clock threshold. The board has been monitored in the system for almost a week with no recurrence of this problem! Upgraded System Monitor firmware mod was tested in system. The upgrade prevents the microprocessor firmware from jumping to incorrect locations in response to incorrect characters received over the VME serial port. There has been only one instance since the upgrade where the system monitor shut the system down. Maybe it's helping?!? The shut-down occurred on 4/24 at about 0700, with the System Monitor complaining about bad system supply voltages. A serious problem occurred over Easter weekend. Ringing was seen in the power spectrum, and spikes were seen in the acf. The self-test showed that one card in each quadrant had problems. A day's worth of trouble-shooting by Ford yielded no improvement. Follow up tests on Monday evening (after Lacasse returned) found a different problem: only one card in the system had a problem. This problem was traced to a bad pin on a socket on an LTA. The socket was replaced and the self-test passed. Subsequent scans with test noise signals and real telescope signals looked good as well. Two important delay numbers were measured and passed on to M&C for possible code upgrades. The delay from an LTA generated interrupt to LTA data being available to the VME computer measured 47 +/- 3 usec. This is very close to the VME bus time-out and should be compensated for somewhere in the system (firmware or software) ASAP. The delay from a Memory Card "Load Xilinx" command to the completion of the command measured 6.7 seconds. The code should be checked to assure that at least this much time is allowed before checking the Xilinx Load completion status. Follow-up testing of the LTA board which started the whole computer replacement episode have been performed. The board passed the bench test flawlessly. The board has also since performed flawlessly in the system for a few days now. This failure could be due to an intermittent contact or to the marginal situation with the VME bus time-out. The spectrometer recovery procedure was updated and passed on to Pete Chestnut. Priorities for next week (comments appreciated!): - astronomer support as required - upgrading all cards to the same firmware revision - spigot card troubleshooting - implementation of a hardware/firmware reset on the system monitor - continue design of the new test signal source -- RL Discussion. Rich added that the Spectrometer appears to be in good shape at the moment, except for a problem with the pulsar spigot card. The engineering group is considering adding a hardware reset button to try to avoid power cycling as a means of resetting. A test signal for cross-correlation mode testing is also under development. 3. Commissioning news o All have spent a substantial amount of time on observer support. o Frank, Karen, Toney, and I have started the next stage in the specification of the user interface for configuring the GBT hardware. Even after only a few days of work, the group has the broad picture pretty well defined and is already working on the definitions of the command names, their arguments and their actions, which the user will want in the interface. o We found a defective transfer switch in the 18-22 GHz section of the K-band receiver that probably was a significant cause of the calibration problems observers have been seeing over the past month. (We also have suggested changes to the Aips++ calibration routines that might also have contributed.) The failure was very low isolation between the two channels -- a source would seem to be in both beam simultaneously, each channel would 'see' the other channel's noise diode, a source would seem to be weaker than expected, and the signal to noise would be lower than expected. We don't yet know how many observers have been affected by the bad switch but Jim tells us he does not see the problem in his March data. All affected observers should not trust the intensities in their left polarizations. Luckily, most affected observations seem to be detection experiments which merely lost a bit in sensitivity. The switch has been removed and won't be replaced until the receiver is in the shop for upgrades this summer. The consequences of the removal for observers over the next few weeks are: . Only L1 and L2 (18-22 GHz) receivers are affected. The 22-26 GHz section seems OK as does the Right polarizations at 18-22 GHz. . Beam-switched continuum observations should only be done with the R1/R2 channels. Observers should deselect L1/L2 in the DCR for pointing and focusing. . Nodding the telescope for line observations still works but one should not pick "Beam-switching" as the switching mode. Instead, one should pick "Total Power". Nodding without "Beam-switching" still will removes LNA and I.F. bandpasses, fast changes in the atmosphere, etc. The only thing that "beam-switching" adds is the ability to take out fast drifts in I.F. gain and bandpass shapes due to such things as cable changes in the equipment room due to temperature changes. This should only affect those doing long integrations with wide bandwidth observations and can be mitigated by shortening the Nodding cycle time to a couple of minutes. There's only a couple of projects that might be affected by the removal of the switch. -- RJM Discussion. Ron noted that the staff astronomers are surviving the increased load of visiting observer support so far. Dave asked if observers will be notified about the calibration problem in the K-band system. Ron replied that we are in the process of narrowing down when the problem first occurred, and whose data are affected. We will notify observers when we have this information. 4. Holography report Claire gave a report on the OOF holography results. A number of holography maps have been obtained, and the technique looks quite promising. Tests have been done in turning the FEM corrections on and off, and this is clearly evident in the maps. With the FEM look-up table off, there is a significant horseshoe pattern of errors in the dish. With the FEM table on, these errors are greatly reduced, although the horseshoe pattern is still faintly apparent. This suggests that improvements in the FEM would improve the surface accuracy. A test was made in which a group of panels in one quadrant was moved. This was easily apparent in the maps, and has helped to resolve the orientation of the maps. The K-band images show that the center of the amplitude pattern is considerably offset from the center of the dish. This is not the case in the Ku-band maps. The cause of this is under investigation. Claire noted that a few problems in the data-taking techniques were apparent, including the effects of feedarm vibration. A further set of maps at Q-band are planned for Monday night, weather permitting. 5. Spectral Baseline, Front-end, and IF work Lab calibration of the S-band receiver continues - installation on the GBT has been tenatively scheduled for April 29. A brief observing period was available and used to check the C-band receiver baselines, after the recent receiver work. Initial results indicate that the performance on strong continuum sources is similar to the previous performance. It appears the surface wave suppression material that was tried on the thermal transition had little if any effect. Phase stable IF cables were recently installed between the Optical Receiver and the Converter Modules in a couple of channels. This cable length corresponded to ripple frequency near 60MHz that has been particularly bothersome. Testing showed however that these channels continued to show baseline ripple near 60MHz, so it appears that another factor also introduces this ripple frequency. Several hours of time using the spectrometer and a noise source input shows that from channel-to-channel the dominant baseline ripple frequency varies between 58 and 69 MHz, and the ripple amplitude after 10-15 minutes by more than a factor of ten. The source of the ripple was isolated to something within the 1-8 GHz Converter Modules. Work to identify the exact cause and a means to mitigate it continues. -- RDN 4/24/03 Discussion. Roger noted that the 60 MHz ripple appears be be associated with a variable step attenuator in the converter modules. The phase-stable coax does not appear to help this problem, but Roger still feels that we should be using these cables to eliminate the potential for further problems in this area. Roger stated that the leakage of the 5 MHz comb from the LO reference source had been suppressed by about 30 dB and asked that observers see if the comb is still present in their data. John Hibbard sent a follow-up email after the meeting stating that the RFI appears to be absent in his data of last weekend. 6. AIPS++ report o Stable Snapshot. For details see: New stable available. 18.625. http://aips2.nrao.edu/docs/reference/updates.html o Project Office The address is: http://projectoffice.aips2.nrao.edu o Key Targets: garwood, bob complete multi-bank MS->single MS 60 inc 5/23/2003 garwood, bob flat table->ms converter design 60 inc 5/23/2003 braatz, jim handle new m&c events for iards 60 inc 5/23/2003 mcmullin, joe implement revised multi-feed cal 40 inc 5/23/2003 o Key Support K-band calibration problem Velocity widget Ylva, John o Other Consortium disbands. Note is on home page: aips2.nrao.edu Project as a named entity ends. Another note pending. Calendar: June 13-14: User's Committee August 10-17: Single Dish Summer School -- JMcM Discussion. Joe added that his name was changing to Sally next week. 7. Software status Software Development Division #33 - Friday, April 25, 2003 We are completing week 6 of a 6-week development cycle. We released M&C v3.13 on 4/23/03. Next week begins a new development cycle, which will last for 5 weeks and culminate with the release of M&C v.3.14 on 6/4/03. Release notes for M&C v3.13 are available at http://tryllium.gb.nrao.edu/release/MC_v3.13 The plan of record for M&C v3.14 is available at http://tryllium.gb.nrao.edu/images/POR_May03.pdf The SDD has continued to focus on providing operational support for the Spectrometer. In the next few days, we expect to test a fix to the kernel bug which requires us to reboot the Linux host on which the Spectrometer Manager runs every time the Manager must restart. Software support continues for the OOF beam map activities. EMS activities will continue as part of the new development cycle in the form of 5 new MRs. Specifically, we will be adding data translation modules for Wit2Matlab and Matlab2Wit, enhancing the Matlab interface WiT operator, enhancing the database insertion WiT operator, enhancing the Database Query WiT operator, and adding the streaming of PSM data into EMS. -- AS 8. Project scheduling April 14th Planning Meeting Minutes (Actually held in 2 sessions, Apr 14 and 16) 0) Observer comments -- John Hibbard, April 9th. Discussed John'ss comments, and found that nothing in there was an unknown problem. John lends his weight to the people who want a more friendly telescope. 1) This week's schedule Approved activities for maintenance periods. 2) Next week's schedule No activities for maintenance periods were yet scheduled. 3) May Observing Schedule discussions Still one weekend to be filled. Trouble getting observers to come that weekend. 4) June Observing Schedule discussions The track welding experiment is scheduled for the 2 weeks starting June 2nd. In case of slippage, it will slip until the first weeks of July. Still looking for observers to fill in the schedule. 5) K band calibrations and upgrades A transfer switch was found to be bad, causing leakage between the beams. The switch will be removed and cables will replace it. It is unknown what effect this problem has on data already taken. Ron will attempt to determine how long the switch has been defective. The minimal K band upgrade will be done this summer due to budget constraints. The full 4 beam upgrade is deferred until at least 2004. 6) Spectrometer discussion for near and far term hardware and software improvements. The hardware and software improvements cited in Ford's memo of March 28th will be turned into a formal project proposal. We must begin getting better reliability out of the spectrometer. This will likely require money and staff effort that are not currently budgeted. For now, the software improvements already on the books will continue as planned. 7) Phase 2 of observing improvements Part 1: The algorithm development phase of the project, with the notable exception of the balancing problem, was completed. The balancing problem was handed off to the baselines group. Part 2: Frank, Karen, and Toney will begin to define the look and feel of the system. It will be initially limited to a command-line interface. GUIs will follow where appropriate. Ron will supervise this effort. The effort will be "done" May 9th. "Done" is defined as having a substantial part of the interface specified. Part 3: The third part of this task is to cost out the system specified in Part 2, and then the team will iterate on the design and cost until it is a satisfactory compromise. We did not set a deadline for this iteration to complete. 8) Software task prioritizing and planning. The M&C software plan for the next release was approved. It may be found in the usual place on the project office Web site. The securityu problem was brought up and discussed. Currently, the security only works if all software is written to use the cleo gateway file. Currently GO and the Configuration Tool do not use it. Ron offered to share the recipe with anyone who asks. In the longer term, the security needs to migrate down into the system to avoid all user interfaces, scripts, etc. from being able to accidentally or maliciously interfere with the telescope's operation. Other software tasks: We must migrate miscellaneous cruft and observing scripts out of the gbtops account into a common user area. We must also fix IARDS so that it works from all accounts. These tasks must be done by May 6th. After that time, nobody will be permitted to run as gbtops except for the operators. 9 AOB Spectrometer use by Thacker and Payne Approval was made to offer time during the track welding shutdown to Thacker and Payne for ALMA tests. They will need 4 long days of time on the spectrometer to make their measurements. They will come to GB beforehand to do some feasibility tests and determine the proper setup for the experiments. ==================================================================== April 21st Planning Meeting Minutes 0) Observer comments -- Barry Turner Barry had a successful run. His main complaints were, again, the difficulty of setup and the unfriendliness of the system. He experienced first-hand the security problems mentioned in last week's report. 1) This week's schedule Elevator is out of service at least until Wednesday sometime. Other work appproved as is. M&C and feed installation will need to coordinate on Wednesday morning. 2) Next week's schedule Nothing scheduled yet 3) May Observing Schedule discussions None 4) June Observing Schedule discussions None 4A) July observing discussions 5) K band calibrations and upgrades Beam switch will be replaced at earliest opportunity. 2 are on order, one to replace defective switch, and one for a spare. 6) Spectrometer discussion for near and far term hardware and software improvements. We agreed that we'd like to see the improvements in place for the fall high-frequency season. John will develop the plan for these improvements. 7) Ku Band receiver. Ku Band will be rotated to its proper position for cross-elevation beam switching. 8) Software task prioritizing and planning. This devolved into a general discussion of where we are headed in our efforts, along with some thinking about our overall Green Bank Telescope priorities. In order to get the telescope working well at W band, we have much to accomplish in the next 2 years. We will be meeting to map out all our projects to determine if there is a shortfall in any area. 9) AOB None -- JF Discussion. John noted that the project planning group has been making a point of going through the observer comments in detail each week to make sure we are aware of all the issues being raised and that they are being dealt with on a priority basis. 9. Observing schedule Last week ======= Observing for: GBT03A-020, GBT02C-061, GBT02B-010, BC132, GBT03A-016, GBT02C-034, GBT02A-062, GBT02B-021 Proposals completed this week: GBT03A-020, GBT02C-061, GBT02B-010, GBT02B-021, BC132 April ==== Astronomy remaining ~ 80 hours Maintenance remaining ~ 26 hours Tests & commissioning remaining ~ 22 hours May === Astronomy ~ 320 hours Maintenance ~ 134 hours Tests & commissioning ~ 290 hours June === Astronomy ~ 122 hours Shutdown (track work) ~ 256 hours Maintenance ~ 155 hours Unscheduled ~ 187 hours -- RCB Discussion. Carl stated that about 120 proposals have been accepted so far for observations on the GBT and about 50 of these have been completed. This tally does not include VLBA proposals or ad hoc requests. 10. Any other business None.