GBT Operations & Commissioning Meeting 16 April 2004 AGENDA 1. Az Track and Telescope Status -- Bob 2. Observing News -- Ron 3. Scheduling Status -- Carl 4. Frontends, IFs, and Spectral Baselines -- Roger 5. Spectrometer Status -- Rich 6. Software Status -- Nicole 7. Project Planning Status -- John 8. Any other Business PRELIMINARY REPORTS 1. Az Track and Telescope Status Az Track The delivery of the replacement plate for plate 45 will not be ready until 4/26. Work continues both here and at SGH on computer analyses. SGH Team members will be here Monday for an inspection and meeting. Telescope Operations The failure of Elevation Motor 3 may be significantly linked to water ingress. A large amount of water was observed to come off of the counterweight during the heavy rainstorm Wednesday. A trial modification is being implemented and evaluated. A thorough PM of each elevation and azimuth motor is about 1/3 complete. -- RAA 2. Observing News Over the last two weeks, things were very busy. We serviced 14 different proposals in 21 observing sessions, 2 receiver checkouts, 4 program checkouts, 3 PTCS tests, and spigot and non-linearity tests. There were about 205 hours scheduled for observing, 90 hours for the various tests, and the remaining hours to maintenance. Failures were very numerous over the last two weeks with about 20 interruptions due to the weather and 80 interruptions due to software failures. That's about one failure every three to four hours. Although most incidents lasted five to ten minutes, their numbers just added up. The major sources of lost time were: o Snow/Wind: 13.5 hours o Spectrometer: 4.0 hours o Wind/Converter Rack/Analog Filter Rack: 3.5 hours o Grail/GFM: 2.0 hours o Network: 2.0 hours o Others: 2.0 hours plus a large number of problems with accessors, dynamic corrections, etc. The rash of problems required a larger that typical number of callouts. In all, ~28 hours were lost, about 9.5% of the observing and test time. Although the average setup time was 45 minutes (which is also rather high), most problems occurred during observing and not the setup time. Carl Bignell and the support scientist met on Thursday to assign support duties for May and to discuss program checkouts. About 25 new proposals have been added this week after the TAC met on Tuesday. The various PTCS tests look like that have proved fruitful. I believe our non-linearity tests will help with our problems with baseline shapes and with what sometimes appears as discrepant calibrations or Tsys. I'm organizing an informal meeting for early next week to discuss our linearity findings. -- RJM 3. Scheduling Status Last 2 Weeks ============= Observations for proposals GBT03C-009, BW070 , GBT04A-041, GBT01A-061, GBT02A-063, GBT02A-031, GBT04A-003, GBT04A-025, GBT03B-013, GBT03A-015, GBT02A-069, GBT02C-012, GBT03B-015, GBT03A-016, GBT03C-026, GBT02A-028 Completed proposals GBT01A-061, GBT02A-031, GBT04A-025, GBT03A-015, GBT03C-026 Next Week ========= Observations scheduled for GBT03C-030 PI Jayaram Chengalur Constraining the variation of fundamental constants using 18cm OH lines GBT04A-045 [B] PI Mallory Roberts Timing of Three Binary Pulsars Discovered in a Survey of Mid-Latitude EGRET Error Boxes GBT03C-031 [B] PI Bryan Jacoby Timing the pulsars in M62, NGC 6544, and NGC 6624 and Search for Ultra-fast pulsars GBT04A-025 [B] PI Harvey Liszt Physical Properties of Halo HI Clouds GBT02C-043 [P] PI Douglas Finkbeiner Microwave Emission from Spinning Dust GBT02C-054 [P] PI James Braatz Measuring Nuclear Disks in NGC 1386 and IC 2560 (H2O) GBT02C-012 [P] PI Mike Hollis A Search for the First Nucleic Acid Base Biomarker: Interstellar Pyrimidine April ===== Scheduled hours [backup] Astronomy ~ 430 [45] Maintenance ~ 52 [53] Tests & Comm ~ 223 [49] Un-assigned ~ 15 May === Scheduled hours [backup] Astronomy ~ 389 [56] Maintenance ~ 88 [61] Tests & Comm ~ 185 [12] Shutdown ~ 26 Un-assigned ~ 56 June ==== Scheduled hours [backup] Astronomy ~ 221 Maintenance ~ 214 Tests & Comm ~ 129 [24] Un-assigned ~ 7 Proposal Checkouts ================== Awaiting checkout - 25 Active checkout - 15 On Hold - 7 Schedule - 45 Completed - 164 Current backlog [hours] = 3136 -- RCB 4. Frontends, IFs, and Spectral Baselines Ka-band and mmConverter: The new receiver was not installed on the GBT this week as planned. There was a problem with a cooldown after the feedhorns and vacuum window were installed, and not all of the assembly work for components outside the dewar was completed. On the positive side, four detector video amps were completed and integrated into the receiver, and MMIC 26-40GHz postamps and the MMIC LO multipliers were completed and function. Two production mmConverter modules were completed and tested. Mounting brackets have been fabricated and trial fit in the receiver room. Other: A solution for the 300kHz spurs generated in the Converter/Filter modules was developed, and all sixteen modules modified. Spectrometer scans after the mods appear to be clear of the spurs. Some measurements on the Q-band components in the test dewar were made using the Spectrometer. Analysis of the data is on-going. A draft memo reviewing the status of the 3mm Module 1 receiver development status was produced, outlining the work required to complete the project. RDN 4/15/2004 5. Spectrometer Status *****Operations***** Wendy Lane experienced quite a few Data Interrupt failures. Using clues found in the logged information, Ford and Clark modified the spectrometer software. Most of the modifications had to do with duty cycle monitoring. Since the modification, the spectrometer has been used 8 hours and has experienced no failures! Ford can provide more information at the meeting if desired. Other than the Data Interrupt failures mentioned above, operation has been quite smooth. *****Development***** Work has continued on the 200 MHz spigot modes. In particular efforts are continuing to produce new Xilinx personalities for the Memory Cards which store and re-order samples for the correlator cards. Thusfar, bench tests have not been completely successful. Ray is expected to come up next week to pursue this further. Progress has been made on the Cross-Correlation Test Fixture. The printed circuit boards for the noise sources have been received. A case for the boards has been designed and submitted to the shop. The design of a voltage regulator for the noise sources is complete. The printed circuit design has been submitted to the vendor; all other components are NRAO stock. Bandpass-defining filters have been designed and simulated as lumped components. Some additional work is required to figure out exactly how we will implement these. On the LTA upgrade, schematic design is in progress. We had a number of different spigot tests over the last 2 weeks. The first, using the antenna, was to test the spigot IF path set-up, center frequency, band flips, etc. at all receivers from prime focus through X-band. Things were mostly successful, although we ran into a few problems: - the 200 MHz modes are aliased, at least at high frequencies. - its is _Extremely_ difficult to get the 4-bit summed pol. modes to have a decent calibration - this needs to be worked on During this time we also tested the AF/CF balance program, which worked successfully, and the 0-lag (power level) strip chart, which also proved extremely useful. The problem with calibrating the summed polarization, lower-bit modes has been looked into a bit by David Kaplan, and more work is scheduled for next week. Finally, we also have successfully tested the new if & spectrometer balance commands (run from the configuration api). In other news - the new spigot computer should be arriving any moment, and we have test time set aside next week to begin testing it out. Also, we should be testing remote observing with the spigot during the week of May 3. *****Plans***** Continue the above-mentioned development efforts -- Rich, Karen, Holly 6. Software Status Single Dish Development IPT #76 - Friday, April 16, 2004 Today ends week 3 of 6 in development cycle C3 in 2004. The Plan of Record is available from the wiki at Software.PlanOfRecordC32004. Release documentation for our latest release and other previous cycles can be found at Software.SoftwareReportCentral on the wiki. PTCS represents the bulk of our activity this development cycle. Over the past two weeks, several items have been accomplished. Amy has assumed the role of Product Manager for GFM/DEAP and has resolved 7 out of 33 total items required by PTCS in this area. These are all going into test April 15. Eric integrated an array handling library that provided up to 100x performance improvements in plotting within GFM/DEAP and spent much time this week educating Amy on GFM internals. Richard was performing some of his own benchmarks on April 15 to quantify what additional performance requirements are needed; these can be viewed at Data.GfmBenchmarks. Many other areas of work are active, and several working group sessions have been conducted. An MR plus development has been completed to provide a glish client to compare refraction algorithms. Another MR is in process for changes to the antenna refraction model. MRs are under development as well for improvements to the Antenna Characterization manager, and new managers, including one for Inclinometers and another for Accelerometers. An analog to Grail for engineering use is also being scoped out in an MR. Requested updates to configuration have been made, and were released on Wednesday, April 14. Configuration now computes delta frequencies in the user's selected velocity frame and adds the delta frequencies to these computed values. Configuration for the new Ka receiver is supported, and details on usage will be sent to scientists once this cycle's commissioning work is complete. Additional work was also done to make it easier to configure and diagnose problems with the spectrometer. If a spectrometer error is detected, the configuration tool will send an error message to the user with directions on how to use the config tool to fix the problem. Also, commands to execute simple balancing were also released with these configuration updates. This was done to support commitments to Karen on behalf of pulsar spigot remote users this cycle, and the commands simply replicate what can now be done only from within GO or by using CLEO. Complete needs for balancing backends and the IF system will be completed as part of the IF Balancing project, which has been designated a potential future project but will not be launched until later this year at the earliest. Reliability of Grail, the standard interface to the M&C system, has been questioned over the past several weeks. Ray has collected data on the number of times Grail has crashed, the number of times it has been killed and restarted, and the traffic using the service since early February. We found that two things significantly improved the perceived performance of grail: a) a patch to the IF Manager on March 22, which prevented clients of the IF manager from crashing unexpectedly, and b) the Grail improvements that were released as part of M&Cv4.2 on March 31. Even though traffic on grail has increased significantly since the end of March, during the first two weeks of April the utility only failed once, and was restarted manually only 6 times, as compared to the last two weeks of March where Grail crashed 5 times and was restarted manually 20 times. Stats will be kept throughout C3, but the early returns are very promising. Operational support over the past two weeks has included: + Joe and Mark worked with John Ford in response to the surge of data interrupts that have been seen lately with the spectrometer. Fixes were patched in to repair a duplicate start, to prevent signal level monitoring to extend into a scan, and within the control libraries. + Support for the Ka-band receiver has been ongoing, and will continue when the receiver goes on the telescope. M&Cv4.2 was patched with some required changes on 4.9.04. + The Converter Rack and Analog Filter Rack were also patched to fix potential problems which could be impacting Antenna Characterization or dcmon performance. --NMR 7. Project Planning Status April 5, 2004 & 12 Combined Planning Meeting Minutes [0] Observer comments & operator log items to discuss Operator's log comments Significant failures these weeks stemmed from: Computing: Network switch, prospero Chris reported that the network switch had been up for more than a year, and was successfully restarted after it crashed. This network switch failure caused prospero's NFS to go nuts, resulting in the need to reboot it. Chris is working on a system to prevent prospero from taking down the GBT when it goes down. Running out of Memory on titania and Naiad. (Doubled memory to 2 GB!) Software: GFM, Converter rack, Analog Filter Rack, Dynamic Corrections, weather 3, Antenna Characterization, Archivist SDD is looking into why the AFR and CR are giving trouble. It was stated that the DCMON program has problems, and cleo should be used instead until it is fixed. Weather 3 needs looked at to figure out why the data is not correct sometimes. GFM, Dynamic corrections and Archivist problems are all known, and are being addressed this cycle as part of PTCS activities (see SDD Plan of Record). Converter rack and AFR a combination of various problems: Linux Transporter problems on wind, a convert rack "reset" auto-parameter which appears to get involved when wind loses connection and reconnects to the network, and potentially accessor problems on vortex. The first two have fixes which will be patched in Tuesday 13th, the third is under investigation by Mark Clark. Spectrometer configuration issues Nicole needs more concrete information on the failures so that they can be tracked down. John will send out email requesting info, and will go through the logs to track down possible instances of the failures. Active surface work will take the hit to provide personnel to work this problem. Spectrometer issues have been traced to problems getting the Spectrometer configured from a "Standby" state, and the fact that paramaters need to be set and activitated in groups in the correct order. These problems have become more prevelant recently due to increased use of the config tool, and a tendency to put the Spectrometer into Standby when not in use to avoid IF Manager errors. Melinda has a work-around in the config tool, which should circumvent most of these problems. Ron made an eloquent plea, agreed by all, that we should devote some effort to improving the underlying system, rather than creating workarounds at the user interface level. This applies to both the spectrometer, and the IF Manager. This issue to be addressed at the next cycle planning meeting. Electronics: Spectrometer -- Data Interrupt Failure (3), setup failures Spectral Processor -- Hung in stopping Telescope/Servo: Az blower motors, weather, CCU IRIG card failure Observing Summaries G. Langston: GBT Q band Commissioning and program checkout Summary: ======== GBT Tests of Q band system found: 1) Confirmed that the spectral baselines have the same shape with 200 MHz an 800 MHz bandwidth spectrometer modes. 2) Observed Orion A and found reach 0.015 K RMS noise level for four 800 MHz bands centered on 44.5 +/- 1600 MHz in a 2 hour On-Off series of Scans. R. Maddalena: PCO 3C33 Summary Early Friday morning, Doug Roberts, F. Yusef-Zadeh, and I tried out the Q-band proposal 3C33. The prospects for the success of this experiment, and others like it, aren't great. The most significant issues are baselines and poor antenna efficiency at low elevations. If it weren't for the baseline problems, our tests would have been corrupted by the Dynamic Corrections problem Al Wootten experienced the next day. There were no other significant hardware or software issues we saw. T. Minter: PCO GBT01A_004 Comment: In checking out baselines for the GBT01A_004 experiment I believe I have isolated the continuum baseline ripples to arise between LO1 and LO2. There are further tests that should be done that can help determine where the ripples arise and if anything can be done in the near future to suppress them. F. Ghigo: Radar Project BB178 OK F. Ghigo: Checkout of 600 MHz OK. G. Langston: GBT Pulsar Observing Summary, AGBT03A_016_14 Summary: Ingrid Stairs carried out the observations remotely. Kevin G. and Ingrid did the complete setup of the observations, with minimal assistance from the "friend". Observations are on-going but appear to be successful. K. O'Neil: Project: Spigot tests: Preliminary indications are that 4 bit mode tests were successful. T. Minter: Checkout of 1A4, X-band, spectrometer Problem summary: ================ 1) prospero in a funny state 2) Az Truck 3 blower 3) config_tool does not setup DCR correctly when dev_health.conf file used 4) titania does not always show gfm (and other) displays 5) gbtsnowtool caused extra Antenna scan 6) spectrometer still does not stop at end of next integration 7) spectrometer start 1 second late and then immediate data interrupt failures 8) Active Surface got wrong elevation at beginning of scan T. Minter: 4A3, M. Murphy, 600 MHz, spectrometer Problem summary: ================ 1) Observers account recreated with different UID - breaks GO-IARDS comm. 2) config_tool did not setup spectrometer correctly 3) vortex and wind accessor errors 4) spectrometer data interupt failures 5) Converter Rack and Analog Filter Rack power monitors all read -100 6) config_tool did not setup receiver properly 7) Moon! T. Minter: Checkout of 2A44, Hardware: Ku-band, spectrometer Problem summary: ================ 1) Converter Rack transporter 2) Accessor errors for vortex connection to transporter on xena 3) gfm didn't reduce 1st scan of pointing data 4) gfm doesn't always update values - it is too slow 5) CLEO Status didn't have correct monitors for IF, Conv., A. F. Racks 6) spectrometer: bad lags in one scan Observing summary: ================== I was checking for Ku-band RFI from satellite TV and if the noise at Ku-band integrates down with time. RFI: present at all frequencies below 12.7 GHz when pointing between an Azimuth of 90 to 270 degrees (South) and basically at all elevations. T. Minter: 2A69, 3C26, L-band, spectral processor, K-band, spectrometer Problem summary: ================ 1) config_tool / spectrometer setup problem 2) titania Xwindows weird behavior 3) dynamic corrections stooped updating Observing summary: ================== The observations went smoothly. F. Ghigo BW070 VLBI at Q band Summary: Weather did not cooperate, CCU IRIG failure [1] Resource calendar schedule conflicts and discussions [2] Observing Schedule discussions 300 KHz spurs smacked down. Steve White has discovered the cause of the spurs and is going to fix them as maintenance day time permits. Spectrometer 6 minute scan hoo-haa fixed. We believe that the problems manifesting themselves as failures in 6 minute scans are band-aided up. A real fix will be released at the end of this development cycle. (there is a bug in the control library.) Ka band receiver installation was scrapped for this week. Carl needs test time requirements through end of Feburary by Friday 23rd of April. [3] GBT development planning [4] AOB Richard will hold a meeting this week to discuss long-term (~1 yr) plans for commissioning and test time requirements. This will be broken into 2 parts, the June to September trimester, for which very concrete requirements are needed now, and the October to April period, for which we need to develop the requirements as soon as possible. Carl mentioned that backup time for the end of April is still available. Carl mentioned that the Program Checkout web page has been expanded. All checkouters are encouraged to contribute to the page. Antenna software modifications -- Motor fault mods. Now or later? Bob described the problem in more detail. Overtemperature faults in the motors need to be software-controllable, so that if necessary motors can be used to drive the antenna to stow dispite an overtemp fault. We agreed that Tim Weadon would further characterize the problem and summarize the required solution before Joe would get involved. Also, Melinda rather than Joe might be able to work on this (Joe's planned PTCS activities have a specific deadline to meet a commissioning time slot; Melinda's do not). Spectrometer configuration problems Melinda's config tool hacks will be used to fix most of the woes, spectrometer manager fixes should be done eventually. We agreed this should be considered during the planning meeting for next cycle. Encoder R&R during welding of axle T Weadon and J. Shelton will work on this. There will need to be some quick zero-point checks after each encoder removal; these will be scheduled in to the end of maintenance day. The "general test" time after the second removal will be scheduled for a full pointing run. IF system non-linearities. Ron described the results of tests performed by he and Toney. The IF system appears to have a non-linearity which may explain system temperature, receiver temperature, line intensity and baseline variations! We will ponder on the implications for a while and then formulate a plan. -- JMF 8. Any other Business