GBT Commissioning Meeting 28 February 2003 AGENDA 1. Az Track Status -- Bob A. 2. Spectral Baseline work -- Rick 3. K-band and other commissioning -- Ron 4. Project scheduling -- Phil (for John) 5. Observing schedule -- Carl 6. Spectrometer status -- Rich 7. Software status -- Nicole 8. AIPS++/Dish status -- Joe M. 9. Any other business REPORTS & DISCUSSION 1. Az Track Status We have received two quotations for doing the welding and machining demonstration. We may receive another one Friday, and another company has expressed interest and has been forwarded information. An independent method for providing a load versus output voltage has been executed for the bolt strain gauge data. Dennis Egan has returned from Effelsberg. He gained some information that may be beneficial in a new track design, and our experiences were probably helpful to the Germans as well. -- RA Discussion. Bob gave an update on the order of wear plates from Gadsden Tool. The plates arrived from the heat treatment firm and have been sent to Birmingham for finish grinding. They are expected back in Gadsden early next week for cutting, and the first batch should be shipped out to us by the end of next week. This is about a week behind the schedule we were expecting, and indicates that the plates will be replaced by about 14 March. Harry has worked out a method to plot a hole template from his survey data and has done a trial run of this. The plate replacement work will be done as a maintenance day activity and will not require an overnight shutdown. Tim Weadon and co-workers are making track deflection measurements today at Splice 45, where the trial retrofit will be done. This will provide the "before" measurements. 2. Spectral Baseline work Reduction of data from two long (5 and 6 hours) IF noise spectrum stability tests is complete. The most frequently observed and generally strongest ripples have periods close to 63 MHz, which is the period associated with reflections in the IF cables connecting the optical receivers to the converter modules. These cables carry the highest IF frequencies (3 or 6 GHz) in the system, which may be why they are the dominant cause of IF baseline instability. Roger and Galen will concentrate on improving this IF link over the next couple of weeks after which we will repeat our stability tests. The other ripple periods of 23 and 21 MHz seen in the cable reflection test did not show up in the 5-hour stability tests. Hence, we probably won't spend time on the cables between the converter modules and the analog filter rack and between the analog filter rack and the samplers until the OR to CM cables are under control. Absorber was placed around the Ku-band receiver's waveguide thermal gap within the dewar in a trial to suppress apparent weak cavity resonances observed in the system temperature spectrum. No change was measured so the search continues. The point was raised that the TM01 circular waveguide cutoff frequency is within the passband of many of our receivers. An inspection of the Tsys spectra of several receivers showed no obvious difference above and below this cutoff, except possibly at X-band. In addition to preparing a summary talk and report following the designated 3 months of investigation, we are drawing up a recommended plan for how to proceed with baseline improvements on the basis of what we've found. -- JRF Discussion. The baseline group will be giving a presentation on their results in CV on 11 March, and will also give a presentation in GB about the same time. Roger stated that a set of evaluation cables have been received to try between the optical receivers and converter modules. Ron asked if the baselines would improve if we took shorter observing samples, before temperature variations could change the cable properties. Roger replied that the Equipment Room temperature varies on a fairly fast time scale, and Rick added that the cables tended to take a jump in characteristics (perhaps owing to thermal properties of the dielectric) rather than varying smoothly with temperature. 3. K-band and other commissioning Commissioning this week concentrated on proposal checkouts by Frank and myself and Spectrometer checkouts by Karen. Frank has concluded that his list of proposals (2C61 and 2C62) have been sufficiently checked out. However, the observers for 2C62 would like additional tests before deciding whether the project is technically feasible. I've finished checking out 2B02, which also looks like it will be technically possible. Bad weather and hardware or software problems have prevented me from finishing the checkouts for 2C12. The data I've taken for 2C12 isn't as promising as I would have liked and I wouldn't be surprised if we had to postpone this project until baselines are improved. Phil is on the 2C12 project and is examining the data. This week, Karen started the checkout of six different Spectrometer modes. There's a report on the results on one of the modes already on the web. This weekend, we have another two runs for Karen working on Spectrometer checkouts. Frank will attempt to complete the requested 2C62 experiments. And, Glen has three proposals to checkout, all using narrow bandwidths and very similar to experiments that have been successful in the past. After this weekend, there are no commissioning activities scheduled for the telescope until March 13. Instead of collecting data, we'll be in our offices reducing data, writing reports, or working on certain projects centered around making the GBT easier to use. -- RJM Discussion. Ron added that the Spectrometer tests that Karen is doing should be finished by the end of April. Karen noted that some of the old test results are now invalid because of the recent cabling changes in the Spectrometer; some of the modes don't exist any more. Jim asked whether the mode he needs for his project could be checked out again. Karen will do this during her test session on Saturday. Rich stated that it would be very helpful to have a faster way to do engineering checkouts and confirmations. Engineering faults have been developing in modes that have been already checked out. Phil noted that we should try to do checkouts during maintenance days to the extent possible and practical. Roger noted that the Q-band Receiver was going on the telescope on next Tuesday. 4. Project scheduling Priority Projects: Az Track New plates are to be installed ASA they arrive and are drilled. Probably late next week/early the week after. Last 2 weeks of April set up for track welding. No azimuth motion those weeks. Baselines Electronics is working on various baseline related prblems in the Ku band receiver, the IF system, and other parts of the system. PTCS Conceptual design review is being scheduled for April. PTCS will share some of the assigned March Q band time with commissioning. Vibration testing is being scheduled. 6 month planning results Although not yet set in stone, the following are some of the discussions and plans that are in motion. Note that PTCS, Baseline improvements, and AZ track still hold the priority positions in the queue. PTCS progress is being impacted by AZ track on the metrology front, and may be impacted by the Ease Of Use project on the software front. Ease Of Use project: We will divide this into 3 phases. Ron and Frank will write out a scope of work for the project before starting. Deadlines were agreed to as follows for the three phases: 1) Algorithm development -- March 28th -- Ron 2) Software requirements spec. -- April 30th -- Ron 3) Software development plan -- May 30th -- Nicole Documentation project definition -- Phil will take a cut at defining what to do. Ron and Phil will flesh this out by March 15th. Bug list consolidation and prioritization -- Nicole will have level of effort to complete bugs on the list in 4 weeks. Maybe a little sooner. Software Planning -- Antenna Test Range software development was discussed. Options range from getting Michael Lacasse in to Electronics taking over the project. It's not clear that the BFA will attain critical mass this year. Electronics Planning -- Electronics is fully occupied with PTCS, Baselines, K band receiver upgrades, and maintenance throughout the next few months. Shop Planning -- Dennis did not have anything sorted out yet for us to discuss. He will report at the next meeting. Commissioning for the next 6 months -- Spectrometer mode checkout, Q band, K band, PF-2, Observer support, SW testing and specification generation, documentation, pointing runs. -- JF 5. Observing schedule Last week ======= Observations of GBT02A-049, GBT02A-031, GBT02C-023, GBT02C-034, GBT02A-062, GBT03B-027 (TOO), GBT02B-019 February ====== Astronomy ~ 206 hours Maintenance ~ 171 hours Tests & commissioning ~ 231 hours Shutdown ~ 64 hours March ===== Astronomy ~ 345 ( + 241 hours backup) Maintenance ~ 146 hours * Tests & commissioning ~ 257 * If all goes well with track repair April ** === Maintenance ~ 153 hours Tests & commissioning ~ 150 hours Astronomy ~ 140 hours ** Numbers very incomplete because preliminary scheduling has just started -- RCB Discussion. Phil noted that the percentage of time scheduled for astronomy programs was about 31% in February and is 46% in March, which is moving in the right direction. April should also have a majority of time for astronomy when scheduling is completed. Carl noted that the actual time for observing in February was much lower, owing mostly to the bad weather (about 150 hours were lost to weather, and ~40 hours to system problems). Some of the K-band time that was weathered out in February will be rescheduled in March. 6. Spectrometer status The main thrust this week has been to attempt to fix nagging problems. The first of these is the small, short steps in the auto-correlation function (acf). These were found to originate in the LTA cards. The worst of these was removed from the system to be examined on the bench. Testing on the bench has been impeded by a problem that has develped in the test fixture. Most of the effort this week has concentrated on getting the LTA test fixture to work. A long e-mail and some test data were forwarded to Escoffier summarizing problems we have had lately mostly with the LTAs. One especially vexing problem is that LTA boards tend to behave differently in different locations in the spectrometer. (Braatz's problem Sunday was fixed by simply swapping the LTAs in quad 0 and 1!). The sample distributor board that failed last week was examined on the bench this week. A small sliver of copper was found near the chip which we had concluded was failing in the system. Moreover, several burned out (charred) capacitors were found on the board, and a fair amount of green and white residue which may be copper oxide. The capacitors were replaced; the residue is probably best left alone. The board was tested in the system and the original problem is still there. It needs additional troubleshooting. If this board continues to give problems, we may have to seriously consider building a new one, using printed circuit technology rather than wire-wrap. With regards to pulsar spigot testing, there were a few developments. First a meeting was held. It resulted in a commitment by David Kaplan of CIT to write a test plan for the upcoming few months. Second, several short scripts were written and tested. These enabled Kaplan to do most of his testing without assistance. The only script that does not work is one that involves down-loading a Xilinx personality to the memory boards. This would take some effort by the computing division to solve. Third, upgraded firmware was installed and tested; it fixed the "loss of synchronization" problem reported last week. The actual testing by Kaplan this week found a problem with the data frames received by the spigot computer: extra, erroneous data seems to get spliced in to some frames after 50 to 500 frames. Tests to isolate the source of this problem were unsuccessful. Some test time was also spent following up on problems reported by Maddalena and Braatz. No new hardware problems were added to the list. -- RL 7. Software status Software Development Division #26 - Friday, February 28, 2003 Regression testing scheduled for Friday, February 21st was conducted as planned to re-release M&C v3.11, whose release was aborted on Thursday, February 13th. Prior to the 13th, scheduled regression tests were abandoned due to inclement weather, and the SDD attempted to release the software without regression tests. Fatal build problems delayed the release, and once in production, the IF Manager failed during commissioning. M&C v3.10 was restored to allow continued operation. The IF Manager software was essentially reengineered for v3.11, and several errors were uncovered only when the software was put in production on the telescope. Additional investigation showed that these errors could have been revealed through unit testing, but the SDD was unaware that certain combinations would be used along the IF path. Our tests were implemented, the software was adjusted for the new set of cases, and these were independently verified through regression tests plus retrying the observations Ron had originally uncovered the problems through. SDD has determined that to mitigate problems like this in the future, three things must be done: a release cannot take place without complete regression tests, there should be two blocks of regression testing separated by a few days to allow some time for adjustments, and receivers should be cross tested with all backends prior to release. The first two items will be implemented immediately, and the third will be implemented over the next two to three cycles. One possibility for future work is to decouple GO releases from M&C releases to bring capabilities to the commissioners regardless of M&C challenges. M&C v3.11 was re-released on Monday, February 24th. SDD believes it has captured all of the scenarios that might cause the IF Manager to crash, but please call Mark Clark if any critical issues arise. The key updates for the March release are adjustments in the RMS calculations for the active surface, the completion of fixes to ensure spectrometer software reliability, and the astronomical verification and subsequent release of the resolution for the active surface 100ms problem. Work is still on schedule for the completion of GO improvements that were initiated last month. Work continued on the EMS this week, culminating in the completion of 6 MRs which are now awaiting review and signoff. To meet the goal of having a "show and tell" piece available for the PTCS Design Review in April, the team is responsible for compiling a work plan that will show incremental progress through the month of March towards this goal. -- NR Discussion. Ron suggested that decoupling section of the M&C system would allow independent releases, and Nicole agreed that this would be a very good thing. Rich stated that a preview of the new active surface software showed it to be much better than the old version. 8. AIPS++/Dish status o Installations Reminder that http://www.gb.nrao.edu/~aips2mgr/ has information on AIPS++ installation status, AIPS++ usage hints, sketch of the Glish/IARDS interaction, and the protocol for the M&C/AIPS++ installation coordination. We are searching for an appropriate location within the GB web pages to place a link. The M&C/AIPS++ installation coordination has just been beefed up due to some confusion in the agreements which though agreed upon, were not well documented. This should remedy the situation. This includes the details of the Solaris support (used exclusively in GB for the VLBA backend communications and 85-3 pulsar monitoring). The release version has been updated in support of Solaris needs. An additional document on the IARDS operation and recovery will be available this weekend (at this location). o Observing There were some problems encountered in the operation of IARDS. The problems weren't fully reproducible; the reboot of naiad seems to have cleared up the difficulties. Reconstructing the state at the time of the problem indicates several possible concerns. First, the use of glishd should occur solely through the use of the sudo protocol; if this is actually created directly by someone with root privilege, problems can not be cleared - this has happened in the past but it is not clear if this is an issue on the night(s) of the problem. Darrell examined the run-glishd logs and detected potential confusion in the use of the script, particularly the flags. Many starts occurred over short periods of time. To address this, we have updated the documentation on recovery and Darrell has updated the usage of run-glishd to be more facile. Finally, Wolfgang indicated that several vestigial AIPS++ processes were stalled on the system which could have interacted with the event passing required. Jim has written concise documentation on the use of IARDS and a recovery procedure in the event of difficulties. Hopefully these steps will prevent this from recurring. o Stable Snapshot. For details see: http://aips2.nrao.edu/daily/docs/reference/updates.html o Targets joe migrate glish access utilties 40 28/2/03 bob gbtmsfiller: multibank data to single MS 80 28/2/03 joe gbtms average/decimate routines 40 28/2/03 bob gbtmsfiller: support weight spectrum 40 28/2/03 jim iards: display 1 spectrum/spw 40 28/2/03 jim iards: append to one ms 40 28/2/03 postponed bob gbtmsfiller: document 40 30/3/03 added jim Enable AzEl pointing 40 28/2/03 Done jim Focus reduction 40 30/3/03 Nearly complete. Waiting on focus model (Ron), M&C function call (Toney). Calendar: 3/5-3/7 AIPS++ External Technical Review o User Support updated in final minutes o Defects resolved: Use http://aips2.nrao.edu/ddts/ddts_main to examine updated in final minutes -- JMcM Discussion. Toney noted that the operators don't always seem to get the latest documentation on start-up and problem resolution procedures. Jim has just written some updates. Bob asked that when such updates are available, that they be sent first to Pete Chestnut (mchestnu@gb.nrao.edu) for initial review and comments, and then Pete will forward them to the 'gbtops' email list which will get to all the operators. 9. Any other business Richard has produced some template web pages for the PTCS project, which can be found at http://www.gb.nrao.edu/ptcs or by following the links GB -> GBT -> GBT Projects -> PTCS Richard ask that those working on Q-Band receiver tests stay for a discussion after the meeting. Ron noted that he is going to suggest some changes in the way commissioners file their nightly reports so that priority items are emphasized more (he is working with Bob Garwood on this). Jim noted that he has a source selection tool that is complementary to Ron's tool for use while the azimuth restrictions are on. See Jim for further info. PRJ 28 Feb 2003