GBT Commissioning and Operations Meeting 5 September 2003 3 PM EDT AGENDA 1. Az Track Status -- Bob A. 2. Observing news -- Ron 3. Spectrometer status -- Rich 4. Spectral Baseline, Front-end, and IF work -- Roger 5. Software status -- Nicole / Amy 6. Schedule -- Carl 7. Project scheduling -- John 8. Any other business PRELIMINARY REPORTS 1. Az Track Status There was no work on the azimuth track this week. The purchase order for the Phase II finite element analysis (3D, quasi-dynamic) is about to be issued. Follow-up work on structural inspections continue. -- PRJ/RA 2. Observing news Since Friday, the GBT hosted a hefty nine different projects, most of which used the Spectral Processor, a couple used the BCPM, and one was a VLB experiment. Sometimes four different projects were run in a 24-hour period. Karen's reports from the last few nights also illustrate how it has been an interesting week as she lists a good number of problems or issues that need to be addressed. The most significant reported problems were: o The operator's logs list about seven 'crashes' of the Spectral Processor, most of which cost the observers about 15 minutes per incident. One failure was compounded by human error in resetting the hardware and Tim Robishaw and Co. lost over an hour. The new 'configurator' and/or the refactoring of the Spectral Processor manager might reduce the possibility for this kind of human error. The new Spectral Processor cables, we hope, will reduce the backend's failure rate. o Tim R. lost his setup time when the Switching Signal Selector was unresponsive for 40 min until the manager was restarted. o This was followed by an hour of what Joe labeled a 'security' problems when it looked like someone was trying to reset the system while the observer was also trying to set up the system. To me, the symptoms are similar to what we've seen when the Scan Coordinator "reboots" itself and falls back into its default configuration. Although this latter possibility might be what was happening, the security hole that can be exploited by users of Glish still needs to be plugged. o The loss of Sadira on Tuesday night stopped the observing for two hours. Even though Sadira is not on the GBT's network, the GBT software apparently still has undesired dependencies that stretch into the general site network. o Thursday night's observers lost almost all of their setup time when the antenna was inadvertently left in its "simulator' configuration. Other items for possible discussion are: o Karen reports FITS files were again being written into the wrong directories. Carl previously reported that this happens fairly frequently -- I forget the exact figures he gave but they were impressively high. This compounds the problem in archiving data and frustrated Robishaw's ability to process his data. I understand a patch will be placed into the system that will not allow observations to begin until the system is set up so that files will go into a single directory. o GO apparently has difficulty aborting from loops within observing tables. The operators sometimes spend five to ten minutes repeatedly pressing the "Stop" or "Abort" buttons in order to terminate a sequence of observations. I believe the SDD is assessing the cost of fixing this problem. -- RJM 3. Spectrometer status There were no problems reported on the GBT Spectrometer this week, mainly because it was used very little. Hardware status is unchanged from last week. This week work has concentrated on learning the Xilinx Development System (Holly) and writing pulsar software requirements (Rich). The Xilinx Development System will be used in the LTA re-design. Pulsar software requirements are about 80% complete. The Spectral Processor was used a fair amount during the past week, and occasionally experienced the usual problem with data read out. We are still awaiting delivery of the new data cables. Plans for next week: - Continued work on Xilinx Development System - Additional testing of LTAs - Meet with SDD to discuss draft of pulsar software requirements - Spigot tests - High Frequency workshop -- Rich & Holly 4. Spectral Baseline, Front-end, and IF work Work on the ripple-compensated fiber modulators continues. Four of the new units are now installed. While we are not quite ready to release them for general use (one unit shows some instability not yet understood), quick initial tests showed reduced 2.4MHz ripple as expected. The K-band upgrade proceeds. Initial testing found quite good noise performance in most cases. However, one beam showed a fairly large noise peak in the middle of the band that has been isolated to the vacuum window. Initial rebuild of the unit was not very successful, still showing high loss. The problem has been isolated to the final assembly step, when the window foam and membrane are epoxied in. A revised assembly procedure has been worked out and will be completed this weekend. The shop did a quick response to some modifications that were required to replace the resonating hermetic coaxial feedthrus, so once the window problems are fixed, we'll be ready to reassemble and complete receiver testing. Work on the new Ka-band 1cm receiver, prep of the Q-band receiver for this winter's installation, and reconstruction of the feed defrost system continue. --RDN 9/5/03 5. Software status Single Dish Development IPT #49 - Friday, September 5, 2003 This week ends Week 3 of the 6 week development cycle which is the 7th cycle in 2003. The Plan of Record for the current development cycle is available from the Project Office web site at http://tryllium.gb.nrao.edu/docs/POR/POR_Sept03.pdf. As of today we have completed 39% of our progress measures in 48% of the cycle, meaning we are slightly behind schedule. This will only become a concern if we continue to be behind schedule next week. Activities for the current cycle include PTCS work on Python data displays, enhancements to the laser rangefinders, expansion of EMS to include the ability to include data from other parts of the telescope control system, several updates to support PTCS antenna characterization, the development of a configuration alpha from prototype developments over the past year, a new gbtlogview, validation of data preprocessing components, and continued support for electronics to resolve spectrometer issues. Configuration work continues to proceed on schedule, and several issues were uncovered this week to make the process more efficient programmatically. Melinda and Frank conducted initial tests earlier this week. Two software items in support of troubleshooting for the spectrometer DMA problems were patched into M&Cv3.16 this week; another is expected next week. PO#904 has been completed to support this weekend's PTCS experimentation (http://tryllium.gb.nrao.edu/cgi-bin/pro-case.pl?ID=904&request=view). Also, the problem with some FITS files going into the wrong directory when switching between project IDs (which was raised at last week's meeting, details at http://tryllium.gb.nrao.edu/cgi-bin/pro-case.pl?ID=880&request=view) has also been addressed and sponsor tested. One adjustment related to the DCR was identified in the testing, and will be addressed as time permits next week. Operational support this week has included participation in a brainstorming meeting for turning up the 140 ft telescope, and fixing a bug that prevented samplers in Refrigerator Power Supply Managers on the Motor Rack from updating. -NMR 6. Schedule Last week ======= Observing for: GBT02A-031, GBT02C-007, GBT02A-062, BM193, GBT03A-014, GBT01A-007, GBT02C-010, GBT02A-031, GBT02C-056 Completed: GBT02A-062, GBT02C-007 September ======= Scheduled hours [backup] Astronomy ~ 269 [113] Maintenance ~ 159 [36] Test & Comm ~ 293 [17] October ======= Scheduled hours [backup] Astronomy ~ 139 [53] Maintenance ~ 124 Test & Comm ~ 230 [18] Un assigned ~ 231 -- RCB 7. Project scheduling September 1st (held September 2!) Planning Meeting Minutes Present: J. Ford, P. Jewell, R. Prestage, R. Maddalena, A. Shelton, C. Bignell 0) Observer comments Observing report: 3am Sept 1 - 7am Sept 2 Projects: 1A7, 2C10 (Robishaw) Observing Support: Karen O'Neil Problems Summary (Described below): (1) Fits files being written to the wrong directory (2) Spectral Processor failures (3) Cannot exit a GO table (4) Spectral Processor "prepare" does a balance even with balance button turned off (5) Go tables error messaging woefully incomplete (6) Lack of adequate warning message when IF rack balance completely fails Observing Summary: Overall things went well. With the help of Paul Marganian the go procedures in MC3.16 were checked the night before observing to insure all the changes needed by the observers had propagated into the new version (they had). Consequently there were no problems associated with the specific observing method of this group. Instead all the problems encountered seemed to be of a more random nature. The project did start on time (approx. 4am), and a total of probably 2 hours were lost over the 27 hour observing period due to problems. (Details at the end of these minutes) Actions: Item (1): A patch to prevent this is in hand, and will be released this week. (Shelton) Item (2): New cables for the data transfer are on order, and will be installed as soon as they arrive. This may help the failures. (Lacasse) Item (3): More detail will be collected from Karen about this, and a judgement made as to the cost/benefit of fixing this. (Maddalena/O'Neil) Item (4): We will attempt to reproduce this and fix it if it is reproducible. (M&C/O'Neil) Item (5): Darrell will be asked to look at this and come up with an estimate to improve the error messages. Even just a line number with the offending non-parsed text would be great. (Shelton) Item (6): This also needs to be reporduced. the messages need to stay up as long as the power levels are out of range. (M&C/O'Neil) 1) This week's schedule Install and test new modulators A Target of Opportunity observation is being inserted into the schedule this week. Wednesday and Thursday evenings. 2) Next week's schedule Moving telescope checkout Monday Sept. 8th Need to possibly rearrange friend of the observer duties a bit. Not seen to be a problem. 3) September Observing Schedule discussions Ku band will be removed for baseline work Sept. 8th K band checkouts not scheduled until later in the month. K band should be reinstalled by September 15th. 4) October Observing Schedule discussions Q band checkouts planned for this month. Receiver will be installed about the first of the month. 4B) November Observing Schedule discussions Pulsar Spigot observations will be scheduled. A simultaneous Arecibo/GBT observation is planned and scheduled. Observers understand that this is still experimental. Spectrometer group will come up with written policy and instructions for experimental use of the Spigot. 5) GBT development planning 140 and 20 meter Discussions are underway to develop a cost/benefit analysis for us to take on the job of readying the 140 to track the SOHO satellite for several weeks this winter and spring. This has a short fuse and the cost/benefit analysis must be completed in a week. Spectrometer Spigot card tests to date have been favorable. Karen is polling the potential users to determine which modes are needed for coming observations. Debugging the nagging DMA problems will require closer cooperation between Lacasse/Ford/Chen and the software people. We will endeavor to work interactively instead of in batch mode. Alma test correlator We will attempt to estimate the amount of time needed to get the Alma test correlator running in order to use it to test our front end and IF components in the lab. Ford and Norrod will handle this. 6) AOB Power outage scheduled for September 10th at 4:00 Karen's report of 9/2/03 Observing report: 3am Sept 1 - 7am Sept 2 Projects: 1A7, 2C10 (Robishaw) Observing Support: Karen O'Neil Problems Summary (Described below): (1) Fits files being written to the wrong directory (2) Spectral Processor failures (3) Cannot exit a GO table (4) Spectral Processor "prepare" does a balance even with balance button turned off (5) Go tables error messaging woefully incomplete (6) Lack of adequate warning message when IF rack balance completely fails Observing Summary: Overall things went well. With the help of Paul Marganian the go procedures in MC3.16 were checked the night before observing to insure all the changes needed by the observers had propagated into the new version (they had). Consequently there were no problems associated with the specific observing method of this group. Instead all the problems encountered seemed to be of a more random nature. The project did start on time (approx. 4am), and a total of probably 2 hours were lost over the 27 hour observing period due to problems. Notes: (1) The converter rack could use a balance button which allows the desired power levels to be entered. (Time is often wasted during set-up while observers tweak the attenuation settings here.) Problem Descriptions: (1) Fits files being written to the wrong directory: Robishaw loaded a cleo configuration file from last January. After making a few changes to the file, observations were begun. However, roughly half the device fits files were written to the project id from his January run, while the other half were written to the correct directory. When we used the CLEO device explorer to look at the project id associated with, e.g. the spectral processor (one of the device being written to the wrong directory), it gave the right project id. Following the advice given to the operator by Mark Clark, we then went to the CLEO scan coordinator, unselected all the devices, re-selected all the devices, and then hit the "conform parameters" button. This sequence (or maybe just the conform params part of the sequence) did the trick, and the next observation had all the files in the correct location. (2) Spectral Processor failures; The spectral processor died 3 times over the 28 hours of observing. The first and third failure lost approximately 15 minutes. The second failure lost over 1 hour. The lengthy amount of time lost for the 2nd failure was due to mis-configuring the spectral processor when re-setting up all the various parameters ("square" mode was chosen instead of "square-cross"). Note that a significant amount of the 15 minutes of time usually lost is due to the inability to properly abort a GO table (described below). (3) Cannot exit a GO table: This error (like most of these) has been reported before. The gist is that when a GO table is being run there is no means to abort the observing sequence. Instead, the abort button must be hit a number of times equal to the number of commands being run in the GO table. If there is a "repeat" command in the table (which is usual), you have to continue hitting abort for each command in each repeat, This can take 5-10 minutes, and is a large waste of time. (4) Spectral Processor "prepare" does a balance even with balance button turned off: After carefully changing the attenuators to give the right power levels with the spectral processor, and insuring the "balance" button was turned off, Robishaw hit the "prepare" button on the spectral processor to clear the error status. The prepare button then re-balanced the spectral processor (but not to the desired values). This should not have happened. (5) Go tables error messaging woefully incomplete There was an error in the observer's GO table - the "ls" command was accidentally typed on one line out of the ~200 lines in the table. Go, of course, could not run that command but instead of giving a sensible error along the lines of "cannot parse command 'ls'" or something, it just crashes with the error "cannot read header". The observer is then required to waste time going through each line of code trying to figure out what the error is. Additionally, you must remember to close the GO table window and re-open it after an error has been found. If this is not done, the GO table will continue to repeat the same error regardless of whether or not the current GO table is o.k. Lots of time is easily lost in this process. (Note this file was not run in the offline version of GO simply because the table had been working two weeks ago, and sometime during the intervening two weeks the mistake had been unknowingly made.) (6) Lack of adequate warning message when IF rack balance completely fails: The first source observed by Robishaw was Cass A, an extremely bright object. The observers had desired to observe the object with the 320MHz filter in the IF rack. Unfortunately, Cass A is simply too strong to allow this to happen, and the IF rack lacked enough attenuation to bring its power levels down to a reasonable value. However, when the ifbalance command failed (and sent the attenuators to 31 and set the power levels to what I assume is the default value of 10), the message sent to the messages screen cleared itself immediately, as if the problem had been solved. Yet the data at that point was completely useless. Fortunately, we were concerned this would happen and were carefully monitoring the power levels. Has we not been monitoring the power levels (and often observers do not pay enough attention to them), we could easily have lots a considerable amount of time while observing while the IF rack power levels were off the scale. (The IF rack filter was changed to 20MHz after this was seen and the power was then low enough to allow the observations to proceed. No science was lost because of the change in filter bandwidth.) -- JF 8. Any other business