GBT Commissioning Meeting 28 March 2003 AGENDA 1. Az Track Status -- Bob A. 2. Spectral Baseline, Front-end, and IF work -- Roger 3. Q-band and other commissioning news -- Ron 4. Spectrometer status -- Rich 5. Software status -- Amy 6. AIPS++ report -- Joe M. 7. Project scheduling -- John 8. Observing schedule -- Carl 9. Any other business REPORTS & DISCUSSION 1. Az Track Status * A contract was awarded last Friday for the shop welding demonstration.  We are obtaining official quotes on the steel for the mockup and will award that contract as soon as possible.  We are also waiting on a starting date. * We have developed the work scoping document for the analysis of the existing track and forwarded it to purchasing for requests for quotation. * The weighing effort continues, and the team has refined their method and obtained results with better repeatability.  They have requested the opportunity to remeasure some of the first weighings, which will extend into next week. -- RA Discussion. Bob talked with Gadsden Tool this morning about the status of the other wear plates. One has left the straighteners and is at the grinders, and the other will be at the grinders by noon today. They will get to Gadsden's shop by Monday or Tuesday next week for finish grinding and cutting. They should ship to us by the end of next week, and be installed on the GBT the week of April 7 (replacing plates 27 and 31). Seven additional plates will come later, including the 3 needed for the bridging wearplate retrofit test. Dennis has drawn up some possible geometries for this retrofit. Bob said that the weighing is going well and that the technique is being refined. Jacking the trucks up and down a few times seems to get the stiction out of the system, and gives a more repeatable weight. The _preliminary_ estimates are pointing to a total weight of between 17 and 17.5 million pounds. John F. noted that what really matters is the peak weight of the heaviest corner, as this determines the maximum stress on the track. Dave Hogg asked if some of the earlier measurements would be repeated now that the technique has been refined, and Bob replied that they would be. As of today, the team has measured each corner once, and will do some repeat measurements next week. Fred S. asked how the new wear plate is holding up. Bob replied that it looked very good with no wear. At the splice, it is a little low, apparently because of the crown in Plates 43 and 45. The contract for the demonstration shop weld for the retrofit has been awarded. 2. Spectral Baseline, Front-end, and IF work Testing of the revamped C-band receiver continues to go well, although interviews for the technician opening have taken away significant time this week. Still looks feasible to install it by the middle of next week. Work on the S-band receiver has been paused while the C-band testing is completed. Commissioning of the Q-band receiver was supported. There have been mysterious changes in apparent Tsys and efficiency that still are not understood. Some bugs in related software, and a couple of hardware bugs were exterminated. Research into cable phase temperature coefficients continued. We should be ready next week to make a couple of changes for evaluation in the system. Dana looked into some of the raw data from 02A-065, which gave hints on where to search in the system for causes of inconsistant baselines in the K-band #1 and #2 beams. Indications are that the instabilities arise before the beam transfer switches, and may involve the switches themselves. But, these conclusions were based on examining only two scans. The receiver is scheduled for observations most every day through early May, so it is not clear whether any hardware work can proceed. No progress on the thermal transistion issue, except Sri gathered information in preparation for commencement of EM modeling, and we ordered ferrite absorber materials which are advertised to be useful for surface current control. -- RN Discussion. Roger noted that there was not any time available on the schedule in April to check out the K-band baseline problems. We will take a look at this at the Monday planning meeting to see if any time can be arranged. 3. Q-band and other commissioning news Changes to the telescope schedule allowed for more Q-band commissioning than was originally planned. Two of the four scheduled commissioning slots were cancelled because the weather was too bad. On one night we encountered problems with unexplained high system temperatures, too low atmospheric opacities, and the receiver's noise diode turning itself off. On the last night, the system temperatures and opacities were good but the same problems with the noise diode cut our plans short. We still have only a rough estimate of telescope efficiency but all indications are that the telescope's performance (pointing, sidelobes, focus, ...) match those we expected from our lower frequency measurements. I'm currently trying to process some thermal/maser SiO maps of Orion. At first glance, the baselines seem to have the same qualities as with the K-band system. The SiO observations were real easy since the lines are whoppers and the I.F./backends provide plenty of versatility. Karen had one terrible and one very productive night checking out five Spectrometer modes. She says that she has about 14 more to check out which we are hoping to provide time for in April. Rich and company will probably want to discuss the Spectrometer failures we have been experiencing that have made difficult both observing and commissioning. Frank had a few hours to check out some details concerning today's VLBI experiment. He spent a fair fraction of his time putting together the memo describing the algorithm for configuring the GBT from soup to nuts. Unfortunately, the extra Q-band commissioning this week will delay the memo by about a week. Next week we have one more opportunity for Q-band tests before I think we need to call it quits for this season. April is very full of observing with mostly short-term projects. A consequence will be that most of us will be working about half of the time on observing support and, in comparison with previous months, very little on commissioning activities. In April, Claire Chandler and Bojan Nicolic will be visiting and helping with some commissioning and PTCS projects. -- RNM Discussion. Ron showed some Q-band beam cuts and both thermal and maser SiO spectra in Orion. There is no evidence of coma lobe to <-20 dB, although there is some evidence of a little beam broadening at the base (astigmatism?). Dave asked if there was any evidence of an error pattern. Ron said no, but that he would like to explore this more with cuts onto the limb of the Moon. Phil noted that the 35% efficiency at Q-band is consistent with a surface accuracy of 450 microns. 4. Spectrometer status As nearly everyone knows, for the past few weeks, the spectrometer has seemed to hang every couple of hours. Most of the effort this week went into an intensive investigation of the robustness of the spectrometer. Hardware and software test points were added and monitored to try to determine the cause(s) of the crashes. We replaced the "VME Interface" board on Tuesday afternoon, and the spectrometer has not hung since. On Friday, the original Interace Board will be re-installed to see if the problems re-occurs. Of course, if it does, the "working" board will be re-installed before the weekend. A few aspects of the spectrometer design were studied in detail, looking flaws that would result in the system hanging. Hardware areas of concentration included the interface between the LTAs and VME chassis and the interface between the VME interface board and VME computer. In addition, the firmware was studied. The goal is to improve it to minimize the chance of communications failures between the main contol computer and the microprocessors. The spectrometer recovery procedure was reviewed with two operators. Improvements were sugggested and incorporated into the procedure. A Spectrometer trouble shooting menu was started. Draft procedures for 'Spectrometer shut down', and 'Spectrometer hangs' were written. Our goal is to write down as complete as possible trouble shoot procedures on regular spectrometer failures. So, operators and technicians can solve these problems according the menu with a little help. At mean time, we will work on the projects of improving reliability and stability of the spectrometer. LTA S/N 9 was repaired. Plans for next week: If necessary, continue robustness testing and repair/enhance as required . If not, spend some time on some or all of the following: o LTA S/N 8 trouble-shooting (multi-integration and acf step problems) o Test-signal source design. o Pulsar spigot card trouble-shooting o Hardware reset design. -- RL Discussion. John Ford reported on behalf of Rich. He noted that the system hung again with the new VME interface board installed, so alas, that was not the problem. We will keep working on the Spectrometer problem as a priority. John is writing a memo on Spectrometer reliability, which was issued shortly after the meeting. 5. Software status Software Development Division #30 - Friday, March 27, 2003 The SDD requests that you submit all new software requests through the software Project Office at http://projectoffice.gb.nrao.edu and provide us with comments to help us make our system work better for you. We are completing Week 3 of a 6-week development cycle, to culminate in the release of M&C v3.13 on 4/23/03 +/- 1 day. M&C development work continues on schedule. 10 out of 11 MRs are approved; one of these MRs has been deferred to the June Release. 1 MR is completed and sponsor tested; 4 MRs are completed and ready for integration testing within the SDD. EMS work continues on schedule; a limited working example is now available which makes the group confident that the completed demo will be useful at the PTCS review in April. Paul and Ray plan on meeting with Kim today to begin the process of finalizing the demo. Out of the 8 MRs that comprise this project, 4 are complete and the remainder are in various stages of development, representing an expected level of progress. The SDD is also moving forward reviewing Frank's list of outstanding bugs from Bugzilla, moving towards its goal of completing all reviews and providing responses and/or high level level-of-effort (LOE) estimates on all 137 requests by the end of March. This is in support of the ease of use initiative. Once LOEs are in place, the SDD will work with the program planning group to divide and conquer the remaining bugs that have not been solved in previous cycles. Some of Frank's bugs have already been addressed and fixed as patches to M&C v3.12 or for the v3.13 release. We spent quite a bit of time looking into the recent Spectrometer failures with the Electronics Division this week. Progress has been slow. Mark Clark and Joe Brandt dedicated a portion of the week to the support of PTCS review preparations with Richard Prestage. Some Q Band support was provided. -- AS Discussion. Amy noted that some of the MRs in the next release may have to be deferred to a future release because of the time being spent on the Spectrometer problem. Glen mentioned the issue of getting an email acknowledgement when a bug report or request is submitted to Project Office. Amy replied that an email generation option is being considered, but that for now, you should send an email explicitly if urgent action is needed. 6. AIPS++ report o Stable Snapshot. For details see: In fourth week of five week cycle, ending April 4th (extended by one week). Current Stable Snapshot available: http://aips2.nrao.edu/docs/reference/updates.html o Project Office The address is: http://projectoffice.aips2.nrao.edu o Key Targets: Bob: gbtmsfiller: fill multi-bank data to single MS Joe: average and decimate design/implementation Jim: Observing support o Key Support Support for some Q band activity - there have been a number of problems encountered in this area, however, to avoid distracting scheduled work, Phil has made it clear that we maintain focus on existing targets. Calibration problem in K-band - Jim has discovered a problem in multi-bank K band observing. The problem has been isolated to either the filler on to the raw data. This should be understood soon. Data check out. Glen has been running through tests for a project in April. He encountered a problem with the SRcal procedure. This was fixed. The resultant Cas A image is available at: http://www.aoc.nrao.edu/~jmcmulli/casa.ps The script to reduce this data is: d.calib(28) #do basic calibration d.SRcal([28:44],[28,44]); #use ends of map as reference positions #average them and subtract d.makeim(28,500,600,step=100,imname='casa_im3',nx=40,ny=36,cellx='7arcmin', celly='7arcmin',gridfn='PB'); There were problems with this data however; the switching state and signal were incorrect and the Trx and Tcal were listed as 1.0 - it should be understood if this was just an observing problem or something more serious. IARDS issues - There were several problems encountered by Ron et al. this week. Jim has mostly resolved these. He has found a work-around to the problem of running IARDS from some accounts - a full solution is pending. GBT-SSR issues will be followed up by Steve on his visit. The initial use cases are important to obtain within the project on a short time scale for the development planning changes indicated by the external review and for the overall support of the GBT. Benchmarking results: Run on atlas: 2392 MHz, 512 KB cache, 1 GB RAM 42256 - Spectral Processor data - [17,18] - OffOn PSWITCH 121k - Spectral Processor data - [36,37] - Track FSWITCH 114k - Spectrometer data - [20,21] - OffOn PSWITCH 224k - Spectrometer data - [43,44] - OffOn PSWITCH 128k - Spectrometer data - [48,48] - Track FSWITCH 1216k - Spectrometer data - [22,23] - OffOn PSWITCH 12125k- Spectrometer data - [51,52] - OffOn PSWITCH The code delineates: first number: number of IFs (1, 2 or 4) second number: number of polarizations (1 or 2) third number: number of channels (256, 1k, 4k, 8k, 16k, 125k) gbt42256: import : 1.73397601 Filling gbt42256: getr : 0.242471933 Access gbt42256: plotr : 0.656038046 Plotting gbt42256: calib : 0.157655001 Calibration gbt42256: average : 0.185328007 Averaging gbt42256: base : 0.180756927 Baselining gbt121k: import : 1.0938009 gbt121k: getr : 0.0801990032 gbt121k: plotr : 0.231571913 gbt121k: calib : 0.299522996 gbt121k: average : 0.269976974 gbt121k: base : 0.22657907 gbt114k: import : 1.26607692 gbt114k: getr : 0.0942389965 gbt114k: plotr : 0.325878978 gbt114k: calib : 0.270769954 gbt114k: average : 0.254790902 gbt114k: base : 0.211865902 gbt224k: import : 5.38499594 gbt224k: getr : 0.0977519751 gbt224k: plotr : 0.327937007 gbt224k: calib : 1.44732797 gbt224k: average : 0.724479079 gbt224k: base : 0.284914017 gbt128k: import : 10.71129 gbt128k: getr : 0.0980050564 gbt128k: plotr : 0.402778983 gbt128k: calib : 1.42480397 gbt128k: average : 0.768871069 gbt128k: base : 0.337412 gbt1216k: import : 13.654997 gbt1216k: getr : 0.125509024 gbt1216k: plotr : 0.531210065 gbt1216k: calib : 1.51756597 gbt1216k: average : 0.703106046 gbt1216k: base : 0.514420986 gbt12125k: import : 94.200578 gbt12125k: getr : 0.523063898 gbt12125k: plotr : 5.46040606 gbt12125k: calib : 12.8003939 gbt12125k: average : 4.89967299 gbt12125k: base : 4.83111095 Defect work: AOCso04422 Failing to fit pointing data o Other GBT/AIPS++ coordination meetings - we would like to have at least monthly coordination/commissioning meetings to help with communications between the two groups. We propose a meeting next Tuesday at 3pm EST. Please let me know if this works. Calendar: NRAO VC: 4/14-4/15 (4/8- 4/16: Joe, Steve, Tim: CV, GB) Bob in GB: 4/16? --JMcM Discussion. Joe noted that with the present system, it was unclear whether the observing programs on the schedule had been checked out from the AIPS++ data reduction standpoint. Ron replied that he thinks that they are in good shape. There have been follow-up discussions on this since the meeting, and some steps to improve this are under consideration. 7. Project scheduling March 24th Planning Meeting Minutes 0) Lincoln Greenhill's comments The group read through the list of Lincoln Greenhill's observing comments. (See http://wwwlocal.gb.nrao.edu/internal/obscom/history/summary.html) We also included email comments from Jim Braatz that were mailed out Monday morning before the meeting. Our goal was to sort out the list into things we already knew about and things that were new, and to make sure that all the items were on somebody's list somewhere. We discussed the highest priority items, and decided what to do with them. (High and Urgent in Lincoln's terms) Following is an edited list of the comments and our understanding of the situation involved in each: Spectrometer faults are common. Symptoms include banks individually, hanging of the whole backend in a stopping, committed, or other states. Barry notes that logging into gbtspec-ws sometimes requires a wait of a few minutes or 20 minutes after the reset switch is depressed on the VME. Overall, the spectrometer is the single largest cause of time lost. Evidence is ambiguous as to whether spectrometer is unstable because nod cycle time is short or for other reasons. Jim feels problems do not occur when cycle times are 5m and do occur when cycle times are 2-2.5m. Unfortunately, IF gain instability requires short times at least some of the time to achieve good baselines. I have noted long periods in which short cycle times have not caused problems. I believe any cycle time will be affected by instru- ment errors if enough cycles are executed. Priority: Urgent We have been working to increase spectrometer reliability for months now. Much progress has been made, and most of the faults that were happening before are fixed. This problem seems to be something new. We are working on reproducing the problem with oscilloscopes and logic analysers attached to the hardware, and instrumented code. An effort is underway by John Ford to come up with concrete proposals for spectrometer reliability enhancements. GO GUI: swiping is not permitted in fields and can destroy the user interface In one instanc, after freezing source name field by accidently swiping, it was nec. to exit GO. Getting obs system working again required > 2h. This is even more maddening than the spectrometer. The basic windowing machinery from MIT is not good but it is not so bad as to require this. Priority: Urgent An (unsuccessful) effortwas made on this problem last week. A more vigorous effort will be made this week. [ N.B. This was successful!] RCP channel gain stability in RF or IF stages is << for LCP channel. 2.5m nodding time works well enough most of the time. However, shorter intervals would be desirable to obtain optimal baselines. It is not clear the spectro- meter can manage these shorter nodding times (e.g., see 14 Mar log). Ripple appears to by 6 peaks across 200 MHz -> ~5 m path. Priority: High The Microwave group is investigating this. This is a new bug. Calibration of Tcals applied to spectrometer data are not mutually consistent LCP vs RCP. Differences can be as large as a factor of two. A complete catalog of cal temps is needed ASAP. For broadband data, proper application of frequency variable cal values is needed. As it stands, I do not believe it is possible to calibrate GBT Tsys and data without separate observations of recomb lines, a painful state of affairs. Priority: Urgent This is a known problem and is on the aips++ to-do list. No solution date is known. SUGGESTIONS ============================================================================= Need real-time display of Tsys (LCP & RCP) at all times, not just during pointing and gbtspec plotting - esp. since the latter gives Tsys values that are suspect. Priority: Very High This item is known, and the capability sort-of exists. It will be made straightforward as a part of the "useability enhancements". Need access to catalogs from inside GO window and the ability to designate a source on deck. Priority: High Also a part of the useability enhancements 02A65 lost hours because no one was available for call out. An improved commo system is needed - something that would work on a nice spring saturday PM. Priority: Urgent Management is in discussions of what can be done here. We have callout lists with backups for all systems. We do not require 24/7 support from our scientists and engineers. In GO, when J2000/B1950 coordinate modes are chosen with, e.g., procedure NOD, rates should be set to zero as a first offering to users. On beam=1 as well. Priority: High Another useability enhancement. Expansion of iards line tool is needed ASAP. A complete quick-look pseudo real-time display is needed for all bands in all line modes. That only one 200 MHz band of the two observed by us is readily available is a serious loss of information. All data should be accessible. The notion of what is most efficient for users depends on the project. I strongly suggest one have the option of displaying all banks at once. The display should also be able to show an accumulation of scans that can either be selected (i.e., a gui selector in which one can set a range of scans, or simply all the scans since the last retune and/or change of source. AIPS++ processing of data is too time consuming to be done at observe time for some programs. The iard quick look is therefore very important if one is to efficiently plan observations (e.g., did I detect anything, and do I need to stay on source?) Priority: Urgent Useability again For the iards display discussed above, many header values are displayed in pairs when observing with both CP channels. There is no indication which number, is LCP and which is RCP. Priority: High Useability again Additional Comments from Jim Braatz: Spectrometer: Regular crashes happened on ~2 hour intervals. Active Surface: It is not clear to me or the operator when the active surface is behaving normally. Can we replace the existing GUI with a simple on/off switch? We were seeing blips in the commanded RMS where it would change from its usual ~35um to ~150um for a cycle, then go back to ~35um. This will be investigated this week. Ron has seen this behaviour before. GO: The "swipe bug" is still there, contrary to reports. We were running with KDE and using the default window manager. This will be attacked with more vigor this week. [ N.B. This was successful!] Antenna: There is apparently feed-arm vibration or some other instability at the start of a first focus scan. It seems to damp out after 5-10 secs and does not appear again if the focus scan is repeated. Perhaps we need to build in a delay so that the focus scans do not need to be repeated as often. This is a known problem with the antenna trajectory preprocessor system. It's on the list of things to fix, but is below our radar. 1) This week's schedule Weighing impacts on maintenance and access Access will be permitted when not actively jacking the telescope. Azimuth Track restrictions Plate 44 was replaced last week, and that opened up some azimuth range. No new plates are expected until April 4th at the earliest. Spectrometer diagnostics are at the top of this priority list for this week. We are cancelling other spectrometer work to try and find the cause of the failures. 2) Next week's schedule Due to weather, weighing will spill over into next week. We expect 2 new plates to be available late next week or the week after. 3) April Observing Schedule Track Weld schedule and effects We were unable to award the contract last week, and so the end of april is probably not going to happen. More information will be gathered and a scheduling decision will be made ASAP. 4) May Observing Schedule Institution of long maintenance days We will be starting 4 12 hour maintenance days in May. Personnel schedule changes (9/40?, 8/10?) Discussions on this topic were deferred due to a lack of time. 5) AOB None. -- JF Discussion. Carl asked that the support astronomers remind observers to use the comment submission web page, as this then records their comments into the history database (this is better in this regard than an ad hoc email). Phil noted that we will try to discuss all comments received at the Monday planning meeting, and that we will try to give each observer a formal response to their comments. 8. Observing schedule The observing summary: Last week ======= Observations for: GBT02A-065, GBT01A-057, BG134, GBT03A-016 Azimuth restrictions relaxed on March 21 after wear plate replaced. March (remaining time) ================ Astronomy ~ 44 hours Maintenance ~ 14 hours Tests & Commissioning ~ 20 hours April ==== Astronomy ~387 hours ( ~ 223 hours backup) Maintenance ~ 155 hours Tests & commissioning ~ 178 hours May === Astronomy ~ 82 hours Shutdown ~ 48 hours Maintenance ~ 197 hours -- RCB Discussion. Jim asked if Spectrometer checkouts could be slotted into backup time on the observing schedule, if the time has not been filled otherwise. Karen replied that she did this last week. 9. Any other business Glen asked if all the RF shielding retrofits going on upstairs would be finished in time for a low frequency observing run in mid-April. John replied that it probably would not. He said there were some difficult issues with humidity condensation on the insides of the windows that were being dealt with. Jim noted that his recent water maser observations were highly successful. 7 detections were made, 5 of which were in systems that were previously unknown water maser sources. This is the most productive detection session ever. Phil noted that in April, the telescope schedule will be dominated by observing programs for the first time. This is likely to flush out a lot of bugs, and will probably result in a lot of support work for the engineering, M&C, science, and AIPS++ staff. Nevertheless, this is what we have been working toward, and is a very good sign of progress. We look forward to the scientific results. PRJ 28 March 2003