GBT Commissioning & Operations Meeting 6 June 2003 AGENDA 1. Az Track Status -- Bob A. 2. Spectrometer status -- Rich 3. Commissioning and Observing Checkout News -- Ron 4. Spectral Baseline, Front-end, and IF work -- Roger 5. Software status -- Nicole 6. Data reduction report -- Joe McM (or proxy) 7. Project scheduling -- John 8. Observing schedule -- Carl 9. Any other business PRELIMINARY REPORTS 1. Az Track Status * The track modification outage started Monday.  The contractor's activities have been going well; the initial vertical welds have been made, the weld groove has been cut, and horizontal joint welding has started.  We are pretty close to schedule, with some activity durations shorter, and some longer.  The contractor's work should finish up Saturday, and the attachment blocks will be installed by our people.  A cooldown period is needed Sunday, and then surveying and layout of the wear plate bolt holes can begin.  We should finish up on schedule or nearly so late next week. * We received the results of the metallurgical tests of wear plate 44 - the one with the big crack in it.  The ductility of the plate at low temperatures was very, very low.  It is highly unlikely that it is an original condition of the plate, and far more likely that it got this way through fatigue from the high stresses.  The sample came from an area about 3 feet from the end, hence, this is indicative of the plate as a whole.  The metallographic images of the grain structure were very good; there was no indication of an original material defect that could have led to the crack.  This means that we will have to either find a new material with much better fatigue resistance and ductility at low temperatures, or change the geometry of the track further to reduce fatigue, or find a way to reduce the fatiguing forces.  This should not be a surprise - we knew we have to do something.  The surprise is how low the numbers actually were. -- RA 2. Spectrometer status Spectrometer Status 5/23/03 to 6/5/03 The spectrometer hardware has been fairly reliable lately; it has not failed self-test in quite a while. Call-ins have been few and far between. The problem with spigot card synchronization has been solved. The System Monitor has been successfully upgraded to 32 MHz. A work around for the "CCC Xilinx load from ROM" problem has been found. The "DMA error" problem persists. Details: The problem with the spigot board was fairly obvious when one finally looked in the right place. To start the scan from a VME command, the new firmware waits on the 1PPS and then counts 1.3 msec interrupts for one second. The problem was that the 1.3 msec interrupts were not designed to start until the scan started, a Catch-22! All three spigot cards were modified to always get 1.3 msec interrupts. Also, all of them have had their software upgraded and have been tested in the system. One useful and unexpected result of the spigot card debugging was that we noticed that the duration of the site 1 PPS (1 usec) is about equal to the length of the spectrometer firware instruction that waits on the 1 PPS. This could cause the system to miss a pulse and start late on occasion, or worse, some parts of the spectrometer could start one second ahead of the others. To remedy this situation, the length of this pulse could be stretched, either site-wide or in the spectrometer. It might be useful to discus whether a site-wide upgrade would be useful, and what a good pulse width would be. Alternatively, the 32 MHz microprocessor upgrade would take care of this. The spectrometer firmware waits for the falling edge of the pulse The System Monitor board was successfully upgraded to a 32 MHz operating frequency and has been running in the system for a week. This upgrade should make it less likely to miss commands from the control computer. The upgrade of the LTA microprocessor is still facing some challenges. The error message "Failed to initialize CCC xilinx personality from ROM." still appears fairly often. A work-around has been found for this message: it should be possible to ignore it and start observing. In a future software release the message will be cleared when a xilinx personality is successfully loaded from file. The root cause of the problem really is a failed load. It is fairly likely that the problem is associated with hardware in the memory boards, and/or with firmware on the Correlator Control Card (CCC). Some investigation will be required to sort this out. The spectrometer still fails occasionally with a "DMA error". We were fortunate to observe one of these failures a few weeks ago and gleaned a few clues. It appears that the interrupt that generates the failure is not spurious and that about 25% of the data is transferred before the error is generated. It is not clear who drops the ball in the transfer. When these errors occur, a phone call would be appreciated if time allows. It would probably still be useful to have time tags added to entries in the log file. Work on the hardware reset continued at a low level. Errors were found in the system monitor firmware that was upgraded last week. These have been corrected and are awaiting test. The testing of the microprocessor development software of Keil Software continues. Keil has provided an add-on diskette which has allowed us to run simulation software, but there are still some problems in running it. More work needs to be done. Plans for next week. - spigot tests - continued work on the microprocessor speed upgrade - continued work on hardware reset testing - continued work on documentation updates - continued work on Keil Software package. -- Rich and Holly 3. Commissioning and Observing Checkout News 4. Spectral Baseline, Front-end, and IF work There are two significant baseline effects not yet completely understood, which are the main focus of the current efforts.  One is a nearly sinusoidal ripple with period of 55-70MHz which occurs in the common IF system.  It has been isolated to the Converter Racks, and is well correlated to the air stream temperature variations in those racks.  At first thought to be associated with cable length temperature sensitivity, the major effect now seems not to be directly related to that.  Crosstalk between the 1-8GHz converter modules seems to play a role in the generation of this ripple, but the mechanism is not well understood.  A couple of test periods last week were spent investigating this ripple, but testing this week has been spent on the second effect.   The second not well understood effect is that which leads to sharp baseline variations with frequency in observations of sources with significant continuum power.  During the track shutdown, we have removed the X-band receiver and are testing it in the Equipment Room as access to the spectrometer allows.  A test which I believed would show the frequency structure seen on the GBT failed to do so when the feedhorn was replaced with a shorting plate on the waveguide vacuum window.  We have now begun to investigate possible problems with the feedhorn construction which might generate the observed effects.   Work has started on the amplifier upgrade to the K-band receiver.   R. Norrod 6/5/2003   5. Software status Software Development Division #38 - Friday, June 6, 2003 We are ending week 5 of a 5-week development cycle, and now have the release of M&C v.3.14 queued up to become available on 6/12/03. The plan of record which shows goals for M&C v3.14 is available at http://tryllium.gb.nrao.edu/images/POR_May03.pdf. Key accomplishments this month will be described on the release notes, which will be made available on or about June 10. The change in release date, which is being done to accommodate the shutdown period, does not affect the start of the new development cycle next week. A new Plan of Record has been identified for the month of June 2003 and is available at http://tryllium.gb.nrao.edu/docs/POR/POR_June03.pdf. M&Cv3.15 will be released on or about July 2, 2003, along with the Metrology system and a fully tested new component for EMS. Phase 1 of the Antenna Test Range software, built on the foundation that was coded last summer during Phase 0 of the project, is scheduled for completion in the next (August) release. Several bug fixes extracted from the Project Office were also completed: Request 733, adding a campler for LO lock; Request 827, S-band polarization switch removed; Request 815, mistake in holography FITS file fixed; and Request 836, elimination of segmentation fault from TaskMaster. -- NMR 6. Data reduction report 7. Project scheduling June 2nd Planning Meeting 10:00 A.M. 0) Observer comments GBT02C_063 -- Gary Fuller My summary: Spectrometer problems, aips++ problems, and an anomalous subreflector position. Spectrometer problems are an ongoing thing, and we are working to come up with a plan to address these. We will not have enough money this fiscal year to address the LTA and sample distributor problems. We will attempt to correct any firmware problems, and other problems that we can solve with minimal cash. The aips++ problems are known. The subreflector problem bears a bit of investigation. The 2" offset is suspiciously close to what would be observed if a focus scan was incorrectly completed. 1) This week's schedule Track work This has begun as scheduled. The new wear plates are in hand. Alma tests Will begin in earnest tomorrow morning. Slight snafu on LN2. Spectrometer Spigot card tests Will piggyback on pulsar drift scan time this week. PTCS tests will pre-emp some of the drift scan time. This is TBD in discussions with Ron Maddalena. Power outage over the weekend The power outage may only be for one day. We will meet at 1:15 and discuss plans for restarting the site. Several checkpoints will be established where we could call in people to restart everything. 2) Next week's schedule More PTCS tests will be scheduled. See the calendar for final results of these discussions. 3) June Observing Schedule discussions Some time may come available if the track welding experiment goes off exactly as planned. The schedule is very tight, however, and nobody should count on it getting done precisely on time. 4) July Observing Schedule discussions July observing has been filled, except for some time the first weekend. 5) 6 month plan review Please get any comments back to me. I'll be finishing this the week of June 2nd. 6) AOB We discussed the Software Group's priorities for the next cycle. A report on this will be made at the commissioning meeting by Nicole. We discussed the problem that happened last week regarding a tour of the telescope that was scheduled was scheduled on top of setup time for an observing run. Due to many errors, this was not caught by anyone. We will be more vigilant about this in the future. Fuller's comments: Overall success of observations: Very successful Observations will result in a journal paper: Yes Description of success and problems Success: We completed our program of identifying and confirming OH masers toward high mass protostellar objects. Problems: On some nights the spectrometer was unstable and had to be rebooted (sometimes several times). On one night we lost nearly 3hours because of this. Success; On some nights the spectrometer didn\'t give any problems. Problem: Running dish under the same username as observing and running iards would hang up iards. Problem: On the night of 22May the azimuth pointing was different by 3arcmin to the previous nights. Also the telescope was about 20% less efficient(the calibrator lines were 20% weaker). After the observing finished the daytime operator suggested it was because the subreflector was in the wrong position by 2inches. No one seems to know why it may have moved. Description of RFI environment No problems. Hardware Please make the spectrometer more stable. It appears that some parameters don\'t always set when first a configuration. This was seen as a shift in the velocity of a calibration linre by 10km/s. Running our setup twice more back-to-back and taking a new spectrum fixed the problem. Software aips++ can do some nice stuff IF you can get someone who really knows to tell you have to do it. On the other hand trying to figure out how aips++ works from the documentation is impossible. The documents are incomplete, inaccurate and written in a language which is not easily comprehensible to astronomers. It is ridiculous that aips++ can\'t write out fits files to be imported into other packages. ASCII file aren\'t a substitute for proper files with full headers. Other comments Despite the problems, it was a very good run and visit. Thanks very much ! -- JF 8. Observing schedule Last Two Weeks ============ Observations for: GBT03B-039, GBT03b-042, GBT02A-014, GBT02C-007, GBT02C-023, GBT03B-037, GB036, GBT03B-023, GB046, GBT02C-065, GBT03B-007, GBT03B-009 Completed: GBT01A-014, GBT02A-014, GBT03B-037, GBT03B-039, GBT03B-041, GBT03B-042, BT069 June === Shutdown ~ 226 hours Astronomy ~ 393 hours * Maintenance ~ 190 hours Test & comm ~ 126 hours * 215 hours are scheduled during the track work shutdown period as background projects when the GBT cannot be moved July === Astronomy ~ 324 hours Maintenance ~ 218 hours Tests & Commissioning ~ 192 hours August ===== Astronomy ~ 323 hours Maintenance ~ 192 hours Tests, Commissioning & Calibration ~ 209 hours -- RCB 9. Any other business