GBT Commissioning Meeting 11 April 2003 AGENDA 1. Az Track Status -- Bob A. 2. Spectrometer status -- Rich, John 3. PTCS CoDR summary -- Richard 4. Spectral Baseline, Front-end, and IF work -- Roger 5. Commissioning news -- Ron 6. AIPS++ report -- Joe M. 7. Software status -- Nicole 8. Observing schedule -- Carl 9. Project scheduling -- John 10. Any other business REPORTS & DISCUSSION 1. Az Track Status * Wear plates 27 and 31 have been replaced this week, and the tilts across them look good.  The azimuth restrictions are lifted, except for not operating below 0 degrees F. * The bottom side of plate 44 was sandblasted and checked for additional cracks near the edges of the wheel path using magnetic particle testing.  None were found, which is very good news.  When the plate was lifted, 2 small, previously unseen, cracks were found near the big crack.  They were visible only on the end of the plate and underneath.  This plate was cut up and samples have been sent to the met lab for analysis.  A section of the uncracked end was cut off for the new visitor's center. * The field work for the weighing was completed Wednesday.  We are planning to have results next week. * I drafted a work scope for the field modifcation coming in June and circulated it for comments.  Those have been incorporated and sent to Continental to start estimating. A final number will not be received until after the shop demonstration. * Tim, Dennis, and I have been analyzing the bolt strain gauge data. * An official request for quotation has been sent to Simpson, Gumpertz, and Heger for the finite element analysis. * Dennis has started work on design of a new track. -- RA Discussion. Carl, Phil and others expressed their appreciation for the hard work and technical accomplishments of the track crew in getting the wear plates replaced. Dave asked about the status of tilts. Bob replied that the worst tilt is 0.25 inches, which is not too bad. We continue to monitor these. Ron noted that the wind restrictions because of the tilts are causing lost observing time. Phil replied that we should get further information on this from the planned flex plate analysis. 2. Spectrometer status Most of the effort during the past week has concentrated on getting the new computer installed, debugged and tested. Additional effort went into upgrading firmware to make it less likely to crash, and into trying to get to the root of the data transfer problem pointed to by the new software. We finished up the brain transplant. The Spectrometer now sports a 2.4 GHz Pentium 4 CPU with 512 MB of Ram, gobs of disk, etc. The computer is connected to the Spectrometer hardware via a Bit-3 bus adapter and fiber optics. The software, which previously had been run in simulation mode on Linux, was adapted to run with the new hardware. This involved understanding and installing the DMA and interrupt drivers for the Bit-3 cards, modifying the DMA and Interrupt classes in the Spectrometer software, and fixing some latent bugs in the serial interface library that were exposed by the different byte order of the Intel processor. We also installed a serial card that supported the odd baud rate used by the spectrometer. At the same time, new debugging routines were added to the code to allow us to debug the DMA problems even in the face of complete process lockups. Better error handling was also incorporated. The good news is that the LTA failures that were causing the lockups can now be identified and corrected, because the new hardware does not lock up when an error happens, and instead it spits out an informative message. The bad news is that the LTA failure that was at the root of this particular fire drill is quadrant dependent. An entire day was spent trying to isolate the failure mechanism without success. Work will continue at a fairly intense pace since we are down to zero LTA spares again (if you count a spare as a card that works in any LTA slot in the machine). The new computer system has been a big help in trouble-shooting. It specifically lists the anomalous condition and then does not crash or hang! A few warts remain on the system. For instance, the system becomes unstable after an abort sometimes. A couple of informational error messages don't propagate correctly. The pesky LTA problem that caused this effort in the first place is still there, although it is hidden and manageable. We did receive one of our Themis Sparc CPU cards back from the vendor, repaired. We will test it at our earliest opportunity, and put it in the cabinet for backup. We never intend to use it, though. The firmware for the System Monitor card was modified. We hope to see an improvement in robustness as a result. System testing is still required. Plans for the coming week, from the hardware perspective, include the attempted repair of the two quadrant dependent LTA cards, testing of upgraded firmware and component specification for the cross-correlation test fixture. John, Holly, Rich Discussion. We are all appreciative of the very hard work the team put in last weekend to get the port to Linux working. The results were impressive. John noted that there were still lots of details to clean up. Rich and John stated that a new, intermittant problem was found by Frank in which the 3rd and 4th integrations occasionally have steps. We will try to catch this problem again. 3. PTCS CoDR summary The PTCS Conceptual Design Review was held on 8/9th April. At least from my perspective, it seemed very successful. Without wishing to prejudge the formal report, the review committee had a number of specific technical concerns, but seemed to feel that the basic approach proposed was acceptable. They do appear concerned about the ambitious schedule, especially given the available resources. We will need to set "inchstones" as well as milestones, and work hard to achieve these. The formal committee report should be available in approximately two weeks. Apart from the review itself, considerable technical progress has been made over the last few weeks. Ray and Paul have completed the first implementation of the EMS (which they demo'ed at the review with archive data). We plan to start real experiments with this within the next few weeks. Jason, J.D. and Randy are making good progress on the temperature sensor system, which we hope will have initial operating capability by the beginning of June. Claire Chandler has arrived from Socorro, and has already installed the OOF data reduction software. Initial observing tests will start this weekend. -- RMP 4. Spectral Baseline, Front-end, and IF work The C-band receiver was installed on Wednesday, and is now available for use. Lab calibration of the S-band receiver continues but has been slowed by the need to respond to other issues. A RFQ is being prepared for Andrews heliax to replace coaxial jumpers in the paths between the Optical Receivers and the Spectrometer high-speed samplers. The DCR was used to do routine stability checks on the IF system, and tests using the Spectrometer are planned for Thursday evening. A written report on the former is available at: http://www.gb.nrao.edu/gbt/baseline/subpages/reports/TBASErdn030407/reportOn IF.pdf --RDN Discussion. Roger added that the coax cables being replaced included 16 cables from the converter modules and another 8 between the filters and sampler inputs. 5. Commissioning news The commissioners have been involved in a number of activities this week: o Most of us were involved in the PTCS review, o Frank and I finished off the "Configuration Algorithm" memo, o Karen, Frank, Glen, and I spent a fair fraction of all our time on observer support, o Glen tried checking out a proposal but was cut short because of winds, o Claire Chandler, Dana, and I have planned out this weekend's holography experiment o Karen successfully checked out the 'astronomical' side of the Spectrometer after it had its recent brain transplant. It has been a very full week of diverse activities. Next week we will concentrate on observer support and holography and probably will start working on the specifications for the user interface for the next generation of hardware configuration program. -- RJM Discussion. In addition to the above, Karen distributed an email this morning identifying several new Spectrometer modes that have passed checkout. 6. AIPS++ report o Stable Snapshot. For details see: End of cycle - see the following url for resolved targets, resolved defects, enhancements (Monday) http://aips2.nrao.edu/docs/reference/updates.html o Project Office The address is: http://projectoffice.aips2.nrao.edu o Key Targets: Bob: gbtmsfiller: fill multi-bank data to single MS Jim: o Key Support o removal of VIDEO_POINT helped o IARDS problems - (incomplete scans, missing IF info) o K-band calibration Comments on velocity widget? Comments on benchmarks? Key Defect work: AOCso04440 no error recovery from bad spectrometer scan AOCso04349 the zero line is ephemeral AOCso04345 import crashed when adding to an existing MS AOCso03941 previous problems leave import non-functional AOCso04422 Failing to fit pointing data AOCso04432 old DCR data seems to fail AOCso04400 d.aver function freezes and fails o Other Calendar: NRAO VC: 4/14-4/15 (4/8- 4/16: Joe, Steve, Tim: CV, GB) -- JMcM Discussion. Ron noted that it was better to use "Stop" than "Abort" if you need to stop a scan (avoids data corruption). However, Karen said that if you use Stop, the Spectrometer will carry on with the next observing phase. Ron said this must be a bug in GO. Bob G. said that they were working to make some of the AIPS++ error messages give similar information to those of the comparable ones in M&C, so that the information was consistent. Jim stated that the calibration error in which the data appear to have an extra, artificial scale factor was a serious problem for several observers. This matter will be taken up in the program planning meeting on Monday. 7. Software status Software Development Division #32 - Friday, April 11, 2003 We are completing Week 4 of a 6-week development cycle, to culminate in the release of M&C v3.13 on 4/23/03 +/- 1 day. The SDD has continued to focus on providing operational support for the spectrometer in the form of porting the software to a new platform that makes it easier for Electronics to detect and solve problems. As a result of the effort, astronomers will find that in the event of a spectrometer crash, it will take up to 5 minutes for the system to be rebooted, in contrast to the 15-20 minutes it took before the port. Efforts over the past week have been focused on getting the spectrometer online on the new Linux platform. However, the spectrometer software will not be running from the same source code location as the rest of the M&C system until the release date of April 23. During the remainder of our development cycle we will continue performing software tests to ensure that the spectrometer on the new platform is tested and reliable. Due to continued spectrometer work, there is one MR that may not be successfully delivered this month, regarding adding feedback parameters from the spectrometer. There are no changes from last week regarding the status of other MRs to be released around the April date. A python interface to M&C was created that performs essentially the same function as ygor_g (the glish interface) and segeste (the tcl interface). This will be used by the SDD for rapid prototyping of development concepts. Utility software for the OOF beam maps activity has been installed in GB for Claire Chandler's use. A change to GO was made that allows the choice of the offset coordinate mode on the GO main GUI and to have all the functions use this value. This is needed for the OOF holography measurements that will be done this weekend. Project Office Request #781, which was a resubmittal of Bugzilla Bug #349, was also completed and released at the same time. This changed the labeling of the GO repeats to iterations and added an iteration counter below the scan counters. The first phase of EMS work culminated successfully in a demonstration of the system's data processing capabilities on laser rangefinder values from last August, which was presented on Tuesday afternoon at the PTCS review. During the remainder of this development cycle, EMS work will focus on refining the vision for the completed system, refactoring current code, and preparing some limited documentation. -- NMR Discussion. None. 8. Observing schedule Last week ======= Observed: GBT02B-002, BP103, GBT02C-061, GBT02A-012 GBT02C-002 was completed. Three projects were cancelled and one project required at least twice allotted time (using spectral processor) because of the spectrometer failure. Spectrometer back into operation Tue April 8. Track restrictions lifted Wed April 9. April ==== Remaining in April => Astronomy ~ 300 hours Maintneance ~ 101 hours Tests & Commissioning ~ 116 hours May === Astronomy ~ 296 hours Maintenance ~ 134 hours Tests & Commissioning ~ 310 hours June === Astronomy ~ 26 hours Maintenance ~ 199 hours Shutdown ~ 244 hours -- RCB Discussion. None. 9. Project scheduling minutes April 7th Planning Meeting 10:00 A.M. 1) This week's schedule Weighing impacts on maintenance and access Weighing may commence after lunch Monday. 2 weighing sessions are needed to fully complete the plan. Money is running out, so we may abort the final data collections and go with what we already have. Wear plate replacement Plates arrived Sunday afternoon. Work started this morning to install them. Due to the spectrometer problems and the weather, no observing this evening until the VLBI run, so the mechanics will work late on the plates. Should be completed by Wednesday afternoon. 2) Next week's schedule No discussion 3) May Observing Schedule Not much backup time is scheduled. Need more astronomy backup projects. We can probably use some of the time for tests. 4) June Observing Schedule The shop weld test is scheduled for April 29th. The field weld test is scheduled for June 2nd. We will block out 2 weeks for that, and schedule astronomy for the last 2 weeks of June. In case the weld schedule slips, we will hold the astronomy schedule, and use the first 2 weeks of July for the weld. We will make the decision on June 2nd after the shop weld is performed. 5) AOB Spectrometer Discussion The new Linux based spectrometer code is mostly working. There are still lots of problems to solve, and almost the entire software division is working on it. The basic flaw that caused the system lockups seems to be present in the spectrometer hardware, but the new computer hardware/software platform seems to gracefully recover from the fault. Spectrometer tests are scheduled for Tuesday evening, provided these bugs are worked out. We are also having the old CPU cards repaired as a backup to this new effort. A Spectrometer Upgrade project will be organized as soon as the current crisis has been solved. This will address other reliability problems and data quality problems. PTCS Temperature sensor installation over the summer is going to require much manpower. A meeting will be held April 14th to figure out what to do with this. Holography is slated for May. John will check with Roger and Steve White to ensure the holography system is ready. Useability Project Ron and Frank have finished a document outlining the configuration algrothms to be implemented. This document will be released April 7th. Following on to this will be discussions on GUI vs command line, staffing required for the project, etc. -- JF Discussion. Phil added that those who attend the Monday planning meeting are John, Richard, Carl, Bob A., Nicole, Ron, Dennis, and he. If you have problems that need addressing urgently, please make sure that one of these people are aware of it. 10. Any other business Joe McMullin stated that he would be handling Jim Braatz's observer support duties while Jim is away next week. Brian raised the issue of which receivers should be taken down seasonally to allow some of the high frequency receivers to be mounted in the winter. He described the issues and proposed some options in an email that followed the meeting. Comments are coming in. Phil noted that we had a little confusion a few nights ago when an observer showed up somewhat unexpectedly late at night to begin observations. A few people got called in late at night to support the observers and solve some problems. To try to avoid this in the future, several things should happen. Carl is presently working on a web-accessible reservation system that will allow observers to make lodging reservations and will indicate what program they are associated with. This should be ready for testing in 2-3 weeks. The support astronomers should also make positive contact with the PI at least a week prior to the run to find out who is coming and when, what their needs are, etc. PRJ 13 April 2003