GBT Commissioning and Operations Meeting 20 August 2003 2pm EDT AGENDA 1. Az Track Status -- Bob A. 2. PTCS news -- Richard 3. Observing news -- Karen (for Ron) 4. Spectrometer status -- Rich 5. Spectral Baseline, Front-end, and IF work -- Roger 6. Software status -- Nicole 7. Schedule -- Carl 8. Project scheduling -- John 9. Any other business Reports & Discussion 1. Az Track Status We have received a revised work plan from SG&H for phase 2 of the track Finite Element Analysis, and are working to get it through procurement. -- RA D. Hogg asked about the status of the Phase I finite element contract. This work has met all requirements of the statement of work and has been closed out. The Phase II contract is in the procurement phase. 2. PTCS news All eighteen antenna temperature sensors are now installed and working reliably. This project has been a great success, and the data are being routinely logged to /home/gbtlogs for anyone who is interested. We have started the process of correlating astronomical peak/focus measurements with structural temperatures; unfortunately our commissioning time to date has occurred under conditions of thick cloud cover and very stable temperatures! We are continuing to have difficulties with the new python pointing/focus processing scripts. Hopefully, these will be resolved in the very near future; this is now definitely on our critical path. A second feed-arm tip trilateration experiment (slewing from 95deg to 5deg in 5 deg interval, down and back up) was performed on August 16th, with great success. 11 of the 12 ground rangefinders were operational, and the data were processed in near real time. The results were extremely encouraging, although full digestion of them will take some time. A simultanous conventional (range, angle, angle) survey of one target was performed to give an independent (albeit much lower precision) estimate of the absolute result. These data are still being processed. A considerable amount of PTCS commissioning time is scheduled for late August/September. This is flexibly scheduled, so that we will not continue to acquire data under mediocre conditions (we have plenty of that already). The goals for this time are to refine the elevation dependent terms of the pointing and focus tracking models; to acquire data to begin to understand the thermal dependencies, and to continue to characterize the performance of the rangefinders (in part by simultanous rangefinder/astronomical measurements). -- RMP Discussion. D. Hogg asked when the next out-of-focus holography run was scheduled. We are expecting to have another round of OOF holography this fall, probably in November. 3. Observing news Ten different projects were run over the last 1 1/2 weeks. These included the Troland, Heiles, & Robishaw runs using the spectral processor, four different pulsar runs, two spectrometer L-band runs, an L-band spectral processor run which just took place this morning, and a slew of different projects which fall under the SDS (Single Dish School) heading. Ignoring the morning of Friday, August 8, the Heiles, et.al runs went fine bar the usual problems with the Spectral Processor. No improvement has been seen with the Spectral Processor crashes, which still occurred 1-2 times in a 12 hour shift. While it typically takes only 3-5 minutes to recover after a crash, considerably more time is lost due to 'useless data' (Tom Troland's term). That is, the group is typically observing using a pattern which is hard-wired into the GO tables/GO procedure code. When any part of that pattern is interrupted, they are forced to dump all the data from that pattern and re-start. This means that if the spectral processor crashes in the last scan of a pattern, all the data from that pattern is tossed out. On 08 August, Heiles, et.al lost a few hours of observing due to equipment (the even-numbered optical receivers) being removed from the system during their observing run. Although in theory this equipment was not being used by the observers, its removal caused a major software crash, and also led to serious difficulties in routing the receiver signal down a usable path. This problem has been discussed extensively elsewhere, and so I will not dwell on it further except to state that an improved telescope and equipment access policy has been sent out to prevent problems along these lines from happening in the future. The lost time was recovered later in the afternoon thanks to the PTCS group being unable to observe due to weather, and giving their observing time to Heiles, et.al. The four pulsar runs which took place over the last ten days were 3B15 (Ransom), 2A62 (Camilo), 2C260 (Nice), and 2A12 (Minter). The 3B15 and (I believe) 2A12 observing runs went fine. All of 2A62's observing time was lost due to LO1 not having, or accepting, valid commanded frequencies. No reason for this problem was found, but approximately 90 minutes after the problem began it fixed itself. The 2C260 run was a replacement for David Nice's earlier observing which had been ruined due to RFI from the power lines. That RFI was not observed at all during last week's run, and many thanks are due to the IPG group. Bar a couple of minor problems with GO, the 2C260 was successful and David was able to see the scintillation effects he was looking for. Two spectrometer L-band projects were run last week - 2B6 (Yusef-Zadeh) and 3B13 (Yun). The 2B6 project suffered from a number of problems, namely: the allocated telescope time began 3 hours before the source rose; their observer's directory from their June run had been deleted, resulting in loss of their observing files (Chris Clark was called in to restore the files); the spectrometer had intermittent DMA problems; and the TaskMaster had NFS locking problems. In all, approximately 25% of 2B6's observing time was lost. The 3B13 project observed for only two of its allotted days. During that time it suffered from two spectrometer failures. The project was canceled, though, due to the source of interest (the Coma Cluster) being too near the sun during the observations. This resulted in large, unstable baseline wiggles. A full report in these wiggles can be seen online at http://wwwlocal.gb.nrao.edu/~glangsto/gbt/coma. A total of seven different observing projects took place under the SDS (Single Dish School) heading last week. Each project was scheduled for approximately 3 hours (1 hours for set-up and two for observing). Combined, the projects used the DCR, spectral processor, and spectrometer, and used the L-Band and C-band receivers. The first project was canceled due to problems with the power supply in the sub-reflector. This was fixed by 10pm on 11 August, in time for all subsequent SDS observing projects. Fortunately, the group was able to make up some of its lost time the next day, and all GBT SDS groups were able to take data. The only other problem encountered was that the X-band receiver warmed-up during the week and could not be used as planned. The groups planning to use the x-band were able to switch to C-band, and no observations were lost. No other serious problems were encountered during these observations, and in all the projects went well. The 3B30 project (Benjamin) took place this morning and I do not know how the observations went. -- KO'N 4. Spectrometer status The "DMA error" continues to rear its ugly head periodically. It is hoped that the LTA upgrade will have some beneficial effect here. LTA modifications are nearly complete and are being tested in the system. Work on pulsar requirements continues. For pulsar timing modes, only two bandwidths, 800 MHz and 50 MHz are presently designed. How serious a limitation is this? Also, Escoffier will be quite busy until the end of the year and will not be available to offer significant support in the pulsar effort. He will have time to help next year, however. With regards to the spectral processor, new data cables have been specified and ordered. These should improve the data integrity and hopefully reduce the frequency of spectral processor "hangs". Plans for the coming week: The pulsar spigot will be used for some test observations. We will support this as required. Work will continue on the LTA clock upgrade and pulsar programming requirements. -- Rich Discussion. It was agreed that the 200 MHz bandwidth mode was important for the Spectrometer pulsar timing capability, and Karen will add that to the list for organizing work on the Spectrometer. 5. Spectral Baseline, Front-end, and IF work Six ripple-compensated fiber modulators have been received back from the fiber splice contractor. Lab testing begins this week. For installation, at least a full maintenance day at access will need to be scheduled after successful lab testing and alignment. One session of Converter Rack baseline testing with the new heliax cabling was completed last week. The results still show some of the "60MHz" ripple, although at the level of about 3X the p-p noise in a 5 minute integration. We have seem much higher levels in the past but it does come-and-go. More experience will show if there truly has been an improvement. We still plan to incorporate a LO2 buffer amp. The shop is working on a prototype housing, so we're still at least a couple of weeks away from results to report on that front. The K-band upgrade proceeds. The rebuilt receiver is now cold, and RF noise and general testing is underway. Work on the 1cm Receiver proceeds. The square-law detectors mentioned in the 8/8 report were received this week. Design work on several fronts proceeds. The C-band receiver comes off for a few days on August 20, for reflection tests on the feedhorn, and for cryo maintenance. Emergency cryo repair of the X-band receiver was accomplished last week, without loss of any scheduled observations. RDN 8/19/03 6. Software status Single Dish Development IPT #47 - Wednesday, August 20, 2003 The development cycle ended last week, so M&Cv3.16 will be released at the end of the maintenance day today. Regression tests delayed from Sunday, August 17 were instead conducted last night, August 19. These showed several small issues with the observing interface that were fixed this morning. Release notes for v3.16 are available for review at http://tryllium.gb.nrao.edu/release/MC_v3.16 on the web. This week begins Week 1 of a new 6 week development cycle, which is the 7th cycle in 2003. The Plan of Record for the current development cycle is available from the Project Office web site at http://tryllium.gb.nrao.edu/docs/POR/POR_Sept03.pdf. SDD activities that were not focused on tweaking and sponsor testing this past week instead turned to the new cycle's commitments. Activities for next cycle will include continued work on Data Export and Accessibility, the Ease of Use Initiative, PTCS activities, completion of the current phase for the Antenna Test Range software, and other continuing maintenance and enhancement activities including the addition of a new quicktest for the spectrometer. A key activity this week was major troubleshooting on the python script for az/el pointing in support of PTCS experiments. These are still not fully resolved, but we are pursuing this activity as our highest priority until data handling for peak and focussubreflector procedures are of sufficient quality for PTCS experiments to proceed regularly, in part because this commitment was not delivered on time by SDD. Operational support this week has included minor fixes for the ODBC interface to EMS, participation in PTCS experiments on Saturday, resolving Archivist/sampler issues, and responding to issues that arose during integration testing of newly released software all through the week. Many of the group members participated in the summer school sessions last week; Jim Braatz spent time moderating a session and leading a hands-on project as well. -NMR 7. Schedule Last Week (and a half) ====================== Observations for: GBT02C-009, GBT02C-046, GBT02A-062, GBT02B-006, GBT02A-012, GBT03B-013, GBT02C-060, GBT03B-030 Completed: GBT02C-013, GBT02C-046, GBT02B-006? Cancelled (technical reasons): GBT03B-013 August ====== Remaining time in hours [backup time]: Astronomy: 60 [35] Maintenance: 74 Tests & comm: 146 [6] September ========= Scheduled in hours [backup time]: Astronomy: 225 [92] Maintenance: 147 [48] Tests & comm: 245 Not scheduled: 103 Ocotober ======== Scheduled in hours [backup time]: Astronomy: 46 Maintenance: 124 Tests & comm: 141 Not scheduled: 434 -- RCB Discussion. Carl also reported that at the Scheduling Committee Meeting earlier in the week, 35 proposals requesting 1200 hours of time were considered. 19 proposals amounting to 500 hours were selected. The committee went over the 400 hour target to ensure that there were enough low frequency proposals to serve as backup programs during the high frequency season this fall and winter. 8. Project scheduling August 11th Planning Meeting Rm 241 1:30 P.M. <0> Observer comments None available online. We discussed some problems that occurred over the weekend. One problem was the LO1 system going insane, and for no apparent reason fixing itself. Ron reported that this could have been a prblem that had been seen before that was easily solved by starting a scan, which would bring the LO1 system back to reality. After further investigation, it turns out that it was not this problem, rather some other unknown problem. Another problem was that the message window did not show the LO1 messages generated by the error. More problems occurred with the Spectral Processor. It seems to be a continuation of a nagging problem that has been there fro a long time. We have requested a quote for better data cables to transfer the data to the computer. This may cure the problems. Another problem that happened Friday was due to electronics maintenance disrupting the observations. It is unacceptable for this to happen, and we will draft a memo to outline the strict conditions under which people can work on the telescope systems during observations. A comment from Carl Bignell was that the observers are mixing data from several projects in one project directory, making it difficault to sort out how much time has been observed on each project. A discussion folloewd and we decided that this is probably something that needs to be changed. It was felt that it would be better not to segregate the data by proposal number, rather each scan's data should be tagged with the proposal ID that it belongs to. Carl Bignell, as project scientist for the archive project, was charged with further defining the problems and possible solutions. <1> This week's schedule Tim Weadon needs time to test the azimuth servo system due to a failure of an azimuth drive. <2> Next week's schedule C band will be removed on the 20th for tests and cryogenics work. <3> August Observing Schedule discussions None. <4> September Observing Schedule discussions September schedule is filling up. We discussed the status of the Spigot card, and if the card would be ready for expert mode observations in September. That question will be answered next week after David Kaplan's tests are completed. Rich Lacasse will be asked to decide, in conjunction with tests David is doing, if the system is suitable for use. There was agreement that once it is available for expert mode, that David Kaplan will need to be a collaborator on the proposals until experience is gained on the system. Once the system is declared operational, the proposals will be scheduled in normal fashion. <5> GBT development planning Review of items from project planning meeting (1) Comments from anyone in the meeting From an Electronics person: "I think the concept of including a lot of people in these meetings so that people will have a good understanding of management's priorities and their roles in them is a good one. I did receive some enlightenment in this regard while viewing Phil's proposed RE budget for next year. However, I received very little enlightenment with regards to what everybody was going to be working on over the next 6 weeks. Instead, it seemed like the project managers went around the table saying they had no problems with the existing slate of things to do. In a few cases some potential problems with the slate of things to do were discussed. I think the meeting would be a lot more useful to engineers who are not project managers if they came away with a good understanding of what was supposed to happen over the next 6 weeks. A couple of suggestions to improve this: 1 - along with the meeting agenda, provide a link to the relevant wiki pages that detail what people will be doing. 2 - during the discussion by each of the project managers, provide a view graph that lists the people in their projects and their assigned tasks." We talked about this comment, and decided that if we publish the web page addresses, that people could go through there and get the information that would be provided by (2). <2> Comments on draft minutes (at the end of this marked Appendix A) No comments. <3> Ka vs Q band priority We decided that Q band is our highest priority, and that getting Ka band done should be given some thought. Ford and Norrod to explore other avenues for ramping up effort on Ka band project. <4> Baseline scientific staffing Ford proposed that a 2 day per week for 6 months sort of effort is what is needed to bring reasonable closure to the current investigations. It seems out of the question that anyone here at GB would be assigned to the project. Richard argues strongly that Dana stay on PTCS, and his arguments are compelling. Ford will consult with Norrod and Fisher on using some help from other sites. <5> Penn Array and CCB staffing for 2004 The efforts in the preliminary version of the program plan are too light. Ford will get with the program managers for this program and provide better estimates. Discussion of any Latest Ideas None submitted since last week <6> AOB Power outage scheduling We discussed timing of the power outage needed to finish repairs of the power line feeding the site. A 4 hour outage is needed. We would like to do it on a weekend or in the evening after working hours to minimize disruption of the workday for employees. Ford will check with Mike Holstine on the cost to us to have the work done outside normal business hours. Cryo test for Penn Array W/O September 22nd? From Roger Norrod: "From the posted September GBT schedule, the week of September 22 might work well for the Penn cryo testing. We would want access on Monday and Wednesday with backup Thursday (at least the mornings). So far there are no low frequency observations scheduled, and maybe Carl could avoid PF and L-band on Monday through Wednesday nights." C band receiver removal August 20-26th We would like to remove this receiver for baseline tests and cryogenic work. It is on the resource calendar schedule. Update on ease of use and data processing projects Phil requested that Richard and Nicole update him on our progress in these areas. He will arrange to meet with Nicole. Appendix A Minutes of 4th August 2003 Project Planning Meeting 1:30 - 3:30pm New Conference Room R. Prestage 8/4/03 Present: Jeff Acree, Bob Anderson, Carla Beaudet, Jody Bolyard, Holly Chen, Pete Chestnut, Chris Clark, Kim Constantikes, Dennis Egan, Rick Fisher, John Ford (Chair), Frank Ghigo, Rich Lacasse, Ron Maddalena, Brian Mason, Randy McCullough, Roger Norrod, Karen O'Neil, Richard Prestage, Nicole Radziwill, Jason Ray, Bob Simon, Mike Stennes, Galen Watts, Steve White. [1.] Review of planning process and discussion of active projects John Ford gave a brief summary of the essential points of the project planning process, particularly the roles of the Project Managers, and the role of this meeting in the process (the intent is to resolve resource conflicts as far as possible, and flag any remaining for further action). John reviewed the 2004 section of the 2004 program plan, specifically the list of proposed projects. It was noted that this assume that the budget request is fully allocated. John then briefly reviewed all of the currently active and proposed projects (new ones are discussed below). Actions/Discussion: - wiki server should be moved to a faster computer at some convenient time (Chris Clark) - wiki should be made publically available (Chris Clark) - a link needs to be made to the wiki from some appropriate static GB/GBT pages (Carl Bignell?) - it was noted that Commissioning no longer exists as a separate activity (ongoing commissioning work is tied to a specific project). [2.] Review and Discussion of Functional Division Requests: Electronics: - There is a conflict for Galen's time between Ka-band construction versus Q-band refurbishment. Roger will assume Q-band is highest priority until he is explicitly told otherwise. This conflict needs to be discussed at the next weekly planning meeting (John Ford). Other electronics division requests are acceptable given adjustments made with PMs prior to the meeting. Mechanical: The Az-track request for 25% of Jeff Cromer's time may be a problem, but Dennis anticipates this is probably a over-estimate in any case. - The Ka-band dewar may conflict with the Penn Array dewar. This could cause a one-to-two week slip to the Penn Array dewar. Brian Mason to contact the Penn Array Group and inform them of this. Scientific Services: - The allocation of Dana Balser of only 0.5 days per week to the baselines project will mean that he will not be able to do any significant work, merely keep appraised of developments. Richard Prestage to ensure Phil Jewell is aware of this. There were no other conflicts for scientific services effort for the cycle. Ron Maddalena proposed a list of staff effort allocations; the wiki pages will be updated appropriately. Software Development: The PTCS Quadrant Detector M&C Upgrades will be descoped to analysis work only this cycle (2 people for 1/2 week). Implementation will be postponed to the following cycle. With the completion of the Spectrometer "T3" self-test, very little additional SDD effort will be required for this activity. The requested SDD effort is achievable assuming a 20% overhead for Operational support. Operations: The submitted requests for antenna access time are acceptable assuming the structural inspections are completed on schedule. There was a lengthy discussion concerning the process by which antenna maintenance/ access time requirements are agreed and prioritized. "Antenna Access" is one of the categories for which we have asked Project Managers to specifically anticipate requirements in their six-week requests, although this has not received as much attention in the planning process as staff resource requirements. Given the liberal access during the summer, and the almost total lack of access during structural inspections, this issue has not been a major problem to date. It will become more important as we move into planned three-day maintenance weeks. There is intense pressure to maximize the amount of time the antenna is scheduled for observing, so we will need to become more efficient at scheduling and executing all other activities (maintenance work; project work; commissioning). These issues will be addressed further in future cycles. Within a cycle, resolution of access conflicts is via requests made to the resource calendar, and the Monday 1pm antenna scheduling meeting, with a two-week forward look and one-week review by the Project Planning Committee. [3.] Coffee Break [4.] Project Team Additional Discussion and Comments No additional discussion. [5.] Review of potential future projects John Ford briefly reviewed the status of proposed potential new projects (over and above the existing "future projects" list. These include: - extending the phase cal to K and Q band receivers. This is believed to be electronics division effort only, and will be added to the Electronics CM & E project. - GBT Fire Detection System. Will be considered further by project planning committee. - GBT Fire Suppression System. Will be considered after above. - Additional Ladders and associated fall protection on GBT. Will be addressed as part of Operations CM & E. - Video survellience for GBT. Will be considered further by project planning committee. - SDD support of VLBI software. This was brought up at the meeting by Frank Ghigo, and will be considered further by Nicole. [6.] Any Other Business John noted that a preliminary estimate of Cycle 8 requirements would be extremely helpful; he will contact individual PMs directly. -- JF 9. Any other business