GBT Commissioning Meeting 21 March 2003 AGENDA 1. Az Track Status -- Bob A. 2. Spectral Baseline, Front-end, and IF work -- Roger 3. K-band, Q-band, and other commissioning -- Ron 4. Spectrometer status -- Rich 5. Project scheduling -- John 6. Observing schedule -- Carl 7. AIPS++ report -- Joe M. 8. Software status -- Amy 9. Any other business REPORTS & DISCUSSION 1. Az Track Status Azimuth track: We received the first replacement plate Tuesday night, and as of this writing, expect it to be in place Thursday evening.  Azimuth restrictions have been reduced accordingly. And now, in other news: * The "weighing" of the telescope is in progress.  We want to recollect data taken yesterday;  winds exceeded our accuracy limit.  We also lost a day Thursday due to rain.  This means we will need maintenance time through next Friday as scheduled. * We measured the hardness of the top surface of wear plate 44 Monday. The values were consistent across the wheel path, contrary to expectations, and near the nominal value specified for the plate material.  We also measured it at one of the deformed places at the edge of a plate, and did get a value about 5 Rockwell C points higher, as expected. * We had a metallurgy consultant (Dr. Walter Sperko) do a site visit Thursday, and gained insight into the consistent hardness readings, among other things.  The hardness would only increase if we have 10% elongation of the top surface;  we are significantly less than that. He also confirmed a number of assessments made by Dennis and myself; among them: o The bolt fracture surfaces indicate unidirectional fatigue. o The wear plate cracks are due to fatigue caused by high rolling stress.  This is indicated by site of initiation and the presence of plate bending.  The cracks are most likely to occur at the ends, but given enough passes, could also occur midplate (emphasis on could). o There is no feasible field repair of a wear plate. o Low temperatures could facilitate propagation of the wear plate cracks, but are not the root cause. * Dr. Sperko also gave guidance on tests to perform on the cracked plate, recommendations on the proposed welding procedure for the base plate, will help us determine the need for a post weld heat treatment, and suggested that we obtain ultrasonic inspection equipment to use to monitor existing cracks and periodically check for new ones. * We hope to award a contract for the demonstration weld today, pending resolution of a couple of items. -- RA Discussion. Bob gave an update on Plate 44: The replacement was successfully installed on Thursday, and the tilt measurements at the splice looked good. The azimuth restriction is now reduced over that splice, which will give about 60 degrees more sky coverage. Ron will change the CLEO tool that displays this. Bob reported in a follow-up email that that the replacements for Plates 27 and 31 will probably not ship until ~1 April. 2. Spectral Baseline, Front-end, and IF work The Ku-band receiver was installed on Tuesday. TP checks with the DCR showed no stability problems, but low level in one channel was traced to a broken IF cable in the receiver room. That will be replaced Thursday or Friday, access allowed. Checks with the spectrometer are planned for next Tuesday. Cold testing of the revamped C-band receiver got underway this week. Gain and stability tests have gone well. Gain and phase balancing for circular polarization, and thermal cals remain to be done. Installation by April 1 still seems feasible. Phase stability vs. temperature testing of several cable types has been done, and we have found that the manufacturer's stated specs tend to be "optimistic". The outcome of these investigations will guide decisions for replacement of IF cables in the Equipment Room. Baseline test observations with the Q-band receiver were scrubbed this week due to weather. -- RN Discussion. Roger noted the Ku-band Rx is scheduled for use at the end of next week, and asked if it could be checked out early in the week. Phil asked about the phase stability of the cables Roger is testing. He replied that the manufacturer is quoting stabilities of ~10 parts per million, but he is seeing more like 25 ppm. The cables we presently have show stabilities of ~70-100 ppm. Other types of cables are on order. 3. K-band, Q-band, and other commissioning Because of the wet weather, commissioning activities this week have not made much use of the telescope. Toney and Mike Stennes spent a few hours investigating receiver instabilities at X-band; Frank and I worked on determining settling times for VLBI-type phase reference observations. Chelen Johnson is in town during her spring break and has volunteered to help with some calibration data I took a few months ago. Frank is making good progress on the "algorithms" for configuring the GBT hardware. Over the weekend we hope the weather will be good enough for continuing Q-band commissioning. Next week Karen returns from Arecibo and is planning on more Spectrometer mode checkouts. -- RJM Discussion. Jim noted an anomaly in his recent K-band observations. For observations in 4-bank, 200 MHz mode, the Tsys in the two polarizations of Bank B agree well, but the Tsys in the 2 polz of Bank A differ by ~50%. Ron and Jim planned to discuss this off-line. Glen noted that the 300 MHz observations he has been doing for the Nord program checkout have gone pretty well. He believes that further work should now shift to the PI. Phil asked Ron and Frank if they could give a "percent complete" on their algorithm development project for system configuration -- they replied 30-50%. 4. Spectrometer status There were numerous call-outs regarding the spectrometer over the weekend. These may be related to M&C 3.12. We also had numerous problems with the software hanging during our test time. It's not clear what the source of these problems is. M&C made a minor change today which may help. We worked for a day to verify that slot dependency and multi-integration problems still existed in the system. We characterized these as best we could in preparation for Ray's visit. Then we worked with Ray for 2 days trying to find the cause of multi-integration problems and slot dependency problems of one particular LTA. We had no luck. We plan to continue to compare signals on a working versus non-working card to try to find the needle in the haystack. We may also swap all the chips between two boards to see if it is a chip or board problem. We're down to zero spares on the LTAs again. We also made no progress in a half hour of searching for pulsar spigot problems; in this case we have a couple leads to follow. -- RL Discussion. The problems Jim had last weekend were discussed at some length. The software hangups tended to go away after switching from 2.5 to 5 minute observations. John had some ideas on this, and further discussion was deferred until after the meeting. Rich stated that he had rewritten the "Spectrometer Recovery Procedure," which now has 5 steps of increasing magnitude. He would like to know how far down the list you have to go to clear problems. Some of the commands that must be typed are rather long. Rich noted that Eric had made some aliases for some of these, and perhaps some more were needed. Ron noted that the operators may need to work more from central, authoritative documentation rather than their own notes to make sure the best and most current procedures are being followed. 5. Project scheduling March 17th Planning meeting minutes This week's schedule a) Weighing schedule and impact on maintenance tasks The weighing will take place as the weather allows, all week. Setup will happen in the morning as early as possible. While setup is taking place, personnel will be allowed aloft on the antenna. Once jacking commences, no access will be allowed. b) Impact of track restrictions on observing Q band commissioning is basically impossible with the new azimuth restrictions. The time could be profitably used for baseline tests with the Q band receiver, so in the event that we get any Q band weather, that's what it will be used for. There was pessimism about the weather this week. Warm and rainy is the forecast. There is a backup program available M-W, with PTCS and possible engineering and software tests later in the week. Pulsar drift scans are also a possibility. Next week's schedule Weighing will continue. The worst of the cracked plates should be replaced, and a reassessment of the azimuth restrictions should allow more observing. We will revisit this Monday morning. April Observing Schedule The end of April track welding program was discussed. Much wringing of hands about the probability of meeting that date, so that observers can be notified, and scheduled. The bottom line is that we are somewhat at the mercy of the successful bidder on the shop weld test, and we will not know of the schedule until the shop weld is complete and the process proven. We decided to continue on with the current schedule until better information becomes available. -- JF 6. Observing schedule Observing summary for tomorrow: Last week ======= Observations for: GBT02A-065, GBT02A-031, GBT03B-009 Azimuth restrictions had significant impact on program completion. Remaining in March ============== Astronomy ~ 70 hours Maintenance ~ 70 hours Tests & Commissioning ~124 hours April === Astronomy ~ 314 hours Maintenance ~ 152 hours Tests & commissioning ~ 196 hours Shutdown for track work ~ 39 hours Astronomy backup time and makeup for time lost due to azimuth restrictions will be scheduled shortly. May === Astronomy ~ 16 hours Maintenance ~ 197 hours Shutdown for track work ~ 25 hours -- RCB 7. AIPS++ report o Stable Snapshot. For details see: In third week of five week cycle, ending April 4th (extended by one week). Current Stable Snapshot available: http://aips2.nrao.edu/docs/reference/updates.html o Project Office In collaboration with Nicole and Carl, we've developed a project office to provide transparent access to planning and accounting. The address is: http://projectoffice.aips2.nrao.edu It contains information on: o Project Book o Key decision logs o Request/Defect submission/requests o Project Calendar o Stable Development Cycles (Plans and results) o Overall Release Plan o Technical Review material Material presented to panel Presentations to panel Panel draft report Project Response (soon) Development - no release; continue with stable snapshots through until at least end of May; at that time have requirements and use cases for planning. It's critical that we move ahead with GB on the SSR document to facilitate this. Key Targets: Bob: gbtmsfiller: fill multi-bank data to single MS Joe: average and decimate design/implementation Jim: Observing support Backlog: Documentation improvements (imaging, etc), Defects Calendar: ALMA PDR: 3/18-3/21 NRAO VC: 4/14-4/15 (4/8- 4/16: Joe, Steve, Tim: CV, GB) Bob in GB: 3/26 -- JMcM Discussion. Glen noted that the flagging tool in AIPS++ was very useful for the program checkout observations he had been doing. He had some trouble finding the right image convolution tools to use for his Cas A map -- Joe and he will work offline. Fred S. also noted that he had some ideas on convolution algorithms and would work on Glen's data if he would send it to him. Joe asked Fred to communicate these to him, as well. 8. Software status Software Development Division #29 - Friday, March 21, 2003 The SDD requests that you submit all new software requests through the software Project Office at http://projectoffice.gb.nrao.edu and provide us with comments to help us make our system work better for you. You can also view and search the 137 aforementioned items from this location. We are completing Week 2 of a 6-week development cycle, to culminate in the release of M&C v3.13 on 4/23/03 +/- 1 day. M&C development work continues on schedule. 7 out of 12 MRs are approved; 2 are being written. 1 MR is completed and ready for sponsor testing; 2 MRs are completed and ready for integration testing within the SDD. EMS work continues on schedule; a limited working example is now available which makes the group confident that the completed demo will be useful at the PTCS review in April. Out of the 8 MRs that comprise this project, 4 are complete and the remainder are in various stages of development, representing an expected level of progress. The SDD is also moving forward reviewing Frank's list of outstanding bugs from Bugzilla, moving towards its goal of completing all reviews and providing responses and/or high level level-of-effort (LOE) estimates on all 137 requests by the end of March. This is in support of the ease of use initiative. Once LOEs are in place, the SDD will work with the program planning group to divide and conquer the remaining bugs that have not been solved in previous cycles. Toney has begun work on implementing fixes to the GO bugs identified by Frank. We spent quite a bit of time looking into the Spectrometer failures reported over the weekend. It was determined that the most likely cause of the problems was that they software was receiving interrupts from the hardware when there was no integration data present. If the software tries to read data from the hardware when there are no data, the sparc-board hangs and must be rebooted. We are working with Electronics to track down the cause(s) of the problem. We did try switching to v3.11 on Thursday afternoon to see if the problem went away, but they only got worse. In support of the weighing of the GBT, we fixed a bug in the weather taking functionality of the metrology system that was induced when the samplers for the weather stations were changed a few months ago. We has halted work on upgrading to VxWorks tornado2. This upgrade cannot be completed because the VxWorks libraries were compiled with a compiler that did not support the functionality that we hoped to gain with the upgrade (C++ standard library, exceptions); we do not have the source and so we cannot recompile the libraries. -- AS Discussion. Phil asked if there was a work-around for the VxWorks problem described. Amy said not so far, and this may be a further impetus to move toward RT-Linux. Ron suggested that the Project Office "Request" feature have a way to auto-email the bug to the relevant person. There was some discussion about pros and cons of this; one idea is to make an optional email field so that if you know the person who should be notified, you could fill that field out. 9. Any other business Dana stated that there was some confusion over the callout lists the operators should be using for various problems. There was some discussion on this and some follow-up emails clarifying the procedure after the meeting. PRJ 21 March 2003