GBT Commissioning Meeting 2 May 2003 AGENDA 1. Az Track Status -- Bob A. 2. Spectrometer status -- Rich, Holly 3. Commissioning and Observing Checkout News -- Ron, Frank 4. Spectral Baseline, Front-end, and IF work -- Roger 5. Software status -- Amy 6. Data reduction report -- Joe M. 7. Project scheduling -- John 8. Observing schedule -- Carl 9. Any other business REPORTS & DISCUSSION 1. Az Track Status * We inspected Splice 12 this week.  The zinc shim material had been completely eaten through near the middle of plate 13, and it was this product that had been migrating up through the joint.  The teflon under plate 12 was still intact.  We reshimmed this joint using teflon.  A sample of the material was sent to a lab for chemical analysis. * Continental reports that we are still on track for the weld demo the week of May 19. * We awarded a change in contract to Gadsden Tool to machine the trial overlapping wear plates this week. * Tim Weadon did a simple correlation analysis between wheel passes and cracked wear plates, and individual truck passes and cracked wear plates.  No significant correlation was found.  This would indicate that some other factor either causes or mitigates the cracking problem. -- RA Discussion. Bob displayed the drawings for the wear plates to be used in the trial retrofit. The joints are chevron-shaped, with one end having a 90 degree angle and the other a 120 degree angle. Part of the test is to see which of these works the best. The points are rounded and beveled to try to eliminate sharp edges that might gouge the wheel or bind on the adjoining plate. Lee gave a brief discussion of his models of the flex plates. In short, it looks like there is considerable margin available before any buckling would occur, with the tilts we have now. Dave asked if this implied that we could relax the wind limits. We will try to evaluate this and reach a conclusion as soon as possible. Dave Parker completed the analysis of the rolling weight of the GBT. The final number is 16.727 +/- 0.048 million pounds. 2. Spectrometer status Progress was made on several fronts this week. These include hardware reset, spigot debugging and firmware upgrades. The function of the hardware reset is to reset all microprocessors in the spectrometer. To date resetting the microprocessors has been accomplished by cycling the rack power; this is far from optimal for long term reliability. As with most things, resetting the microprocessors is not as simple as it might seem. When the system monitor's microprocessor is reset, one of the first things it does is turn off the rack power. It's duties also include powering down the rack in the event of an out of spec temperature or voltage. Clearly firmware modifications will be required. Therefore, the work so far has concentrated on understanding the existing firmware. The pulsar spigot transmits frames of data to the spigot computer. During the last testing session, the frames seemed to "jump phase" every once in a while. It appeared as if extra data were inserted between the frames. A day was spent this week looking at various signals that might be marginal. Nothing remarkable was found. Also a hunch that certain important (at least in this context) modifications were not performed on all LTAs was investigated. This was also fruitless. The rack self-test revealed a faulty LTA card. It was replaced on Thursday. The card is currently on the bench undergoing testing. Plans for next week - continued firmware development - continued work on hardware reset - continued pulsar spigot debugging - continued design work on the improved noise test fixture -- Rich Modified the firmware of LTA and have been trying to test it on the systems. need more system time to test it. Prepare for changing microprocessor clock from 16 MHz to 32 MHz : ordered 32 MHz oscillator, and modified LTA firmware code for using 32 MHZ clock. -- Holly Discussion. John mentioned that it would be helpful if the scientists and operators would log as much information as possible about the circumstances of spectrometer crashes to aid in debugging. 3. Commissioning & Observing Checkout News I'll be observing with Harvey tonight and probably will not make the 10 AM meeting. I've asked Frank to represent the commissioning team. Commissioning news ------------------ Except for a few hours here and there, the last week has been devoid of any commissioning time that required the telescope. Instead, since last Friday we've had a very busy time supporting about ten different projects, many with multiple sessions. I think this number of projects in so short a time is unprecedented in the history of Green Bank. And, even more importantly, a good number of these projects have been finished. Karen, Toney, and Frank are making progress in the user-interface specifications for configuring the GBT. Frank may want to discuss the group's progress. Most of my time has been in scheduling the commissioning activities in May, a much more difficult problem than normal because of the large number of constraints we have in May and because of the new set of 3B proposals that had to be quickly assessed. A by product of this has been a re-evaluation of all the old proposals in the queue, some of which have been moved into the "Scheduled" or "Active Checkout" lists. The commissioners now have updated lists of proposals they are supporting. There are over 90 proposals in the queue. Over 50 can be scheduled and at some level require our support. Ten projects cannot be scheduled or checked out because of missing hardware most often this being the spigot card, BCPM2; only a few require PF2 and Q-band. At most five may need RFI work, many of these will probably be moved to the 'Can be scheduled' list soon. About 15 - 20 proposals cannot be scheduled or would be fruitless to checkout because of issues with wide bandwidths and baseline shapes. As of Tuesday, and after the reassessment, rough numbers were: Can be scheduled: 54 proposals Active Checkout: 12 (4-9 baseline problems; 2 RFI, 1 Hardware) On Hold: 11 (3 RFI problems; 1 Baseline; 1 pol; 6 pulsar hardware) Waiting checkout: 16 (3 Q-band required; 2 RFI problems; 1 Pol; 10 baseline) In May we will again have heavy user support; we will continue work on the user interface; check out at least eight 3B and older proposals, start work on traditional holography, and occasionally checkout a wide-bandwidth spectrometer mode. -- RJM Discussion. Frank gave the report for Ron, who was up late observing last night. Phil asked how much time the supporting scientists found that they needed to spend with the observers. Frank replied that after a few hours, they could usually leave the observers on their own. Glen stated that it typically takes about 2 hours to give the "introductory lecture." Glen also stated that the use of individual login accounts by the observers as opposed to gbtobs common account was causing some slow-downs and confusion. We will look into this. 4. Spectral Baseline, Front-end, and IF work Lab calibration of the S-band receiver continues. Some minor problems were found that called for more testing. Installation on the GBT has been tenatively scheduled for May 6. Work continues to identify and correct the source of ~60MHz baseline ripples within the Converter Modules. Nothing new to report this week. EM modeling of a new thermal transition using "photonic crystal joint" technology has begun. -- RDN 5/1/03 Discussion. Roger added that a loaner transfer switch had been obtained from Socorro and had been re-installed into the K-band receiver. The S-band polarization transfer switch has also now failed and needs replacing at some time. Roger showed some drawings of the thermal transition that Sri and he are modeling. After a design is settled on, this will be built and installed in the Ku-band receiver and tested, probably this summer some time. There will also be further experiments with the ferrite absorber, this time on the X-band feed. 5. Software status Software Development Division #34 - Friday, May 2, 2003 We are starting week 1 of a 5-week development cycle, which will culminate with the release of M&C v.3.14 on 6/4/03. The plan of record for M&C v3.14 is available at: http://tryllium.gb.nrao.edu/images/POR_May03.pdf 11 out of 15 MRs have been written and are approved. Of the 11 approved MRs, 6 are under development and 3 are ready for internal testing. We completed the "Structural Vibration Monitoring Knowledge Transfer" commitment this week by meeting with Kim Constantikes and Satish Ravichandran. The Linux Migration Project Charter is approximately 75% complete. This week, the SDD has continued to focus on providing operational support for the Spectrometer. Work has included testing a new algorithm that will interleave data reads from the quadrants, which will increase the data throughput of the Spectrometer. Software support continued for the OOF beam map activities and for various other observing runs. -- AS Discussion. Phil asked if the new algorithm for interleaving data reads was to address a particular bug. Amy and John replied that it is to allow faster data dumps. John said that the fastest data dumps were now about 0.5 seconds, although by reducing the number of phases recorded and playing some other tricks this could possibly be made faster. 6. Data reduction report o SSG = Scientific Software Group Conceptually a significant change, as the AIPS++ project as an entity to itself is gone and replaced with project-based requirements and solutions. In practice, this isn't observatory wide yet, but is composed of the former-AIPS++ crew. Details of this are still being worked out. In the mean time, commitments are being carried through. In particular, the GBT targets listed below will be fulfilled and continued support for existing modes will be maintained. New development will follow through a requirements process with proposals from computing on suggested implementation technology. I think. o Stable Snapshot. For details see: New stable available. 18.625. http://aips2.nrao.edu/docs/reference/updates.html o Project Office The address is: http://projectoffice.aips2.nrao.edu o Key Targets: garwood, bob complete multi-bank MS->single MS 60 inc 5/23/2003 garwood, bob flat table->ms converter design 60 inc 5/23/2003 braatz, jim handle new m&c events for iards 60 inc 5/23/2003 mcmullin, joe implement revised multi-feed cal 40 inc 5/23/2003 These are the remaining targeted commitments from the former-AIPS++ group. Following these, the GBT filler will be returned to the GBT M&C group (as agreed with Nicole for the 6/1 deadline). At this point, project requirements will guide further development. The resources for further development will be nearly tripled at this stage with the new hire for data reduction in GB and Bob Garwood's resumption of development in this area. Exciting! o Key Support glishd problems IARDS problems (from Jim's note to aips2-gbt) Al, Jay, Glen Summary: When a user's account is configured with a certain set of parameters, there has been a problem running the spectral display portion of IARDS. The problem is now fixed at its root level, and GOspec should run cleanly for all observer accounts whether or not the toolmanager GUI is turned on. If you see a problem now, it is a new one so please report it. Details: Up to now IARDS has been run primarily from the gbtops account or from a limited number of user accounts, and it has been made to work in those accounts. However we have identified some account configurations which have caused problems in starting up the GOspec portion of IARDS. As we are transitioning to the use of user accounts for operating the GBT, it has become important to solve the problem. I could not identify a problem in the IARDS code itself, but I found that if the user turns off the toolmanager GUI, IARDS runs cleanly. So the workaround I have offered up to now has been to turn off the toolmanager in the user's .aipsrc file. Darrell and Bob have become involved in looking for the root of the problem. Yesterday Darrell identified the root as a glish defect, and he fixed it. That fix will be in place with the next stable. Meanwhile, Bob also found a better workaround, invisible to the user, and he has put that fix in the current stable installation in GB. So, whether or not the toolmanager GUI is turned on, IARDS should work for all accounts as of now. o Other Added option to calibrate to Jy Enabled SDRecord to be used as input to SRcal procedure for more flexibility Calendar: June 13-14: User's Committee August 10-17: Single Dish Summer School -- JMcM Discussion. Phil noted that it was extremely helpful having George Moellenbrock out for a couple of weeks as he made significant contributions to the OOF holography effort. Joe noted that Bob G.'s return to GBT application support and development and a potential new hire would also greatly help. 7. Project scheduling April 28th Planning Meeting Minutes 0) Observer comments GBT02A-065 L. Greenhill The focusing and pointing information that L. Greenhill provided will be taken under advisement of the PTCS project. The software problems reported are already known bugs. GBT02B_002 Pihlstroem & Murphy The software problems this project experienced are also known, and on our list to fix. They observed over the weekend that the spectrometer was down for the computer failures, and so they used the spectral processor at greatly reduced efficiency. GBT03A-020 J. Hibbard Problems John experienced were also known, and will be corrected. 1) This week's schedule A tour was set up for the afternoon. This week's maintenance schedule was approved. Bob Anderson added a track segment remove and inspect to one day. 2) Next week's schedule Nothing but generic maintenance time scheduled thus far. 3) May Observing Schedule discussions The May observing schedule is finished. 4) June Observing Schedule discussions The first 2 weeks of June are set aside for the track welding experiment. 5) July Observing Schedule discussions The first 2 weeks of July are set aside as a backup date for the welding, in case the contractor cannot get going quickly enough to meet the June time. 6) PTCS Temperature Sensor Installation Overview The plan was presented for installation of the 25 sensors. It amounted to about 440 man-hours of ironworker time, and a similar amount of electronics tech. time. [ The plans have since changed, with a fewer number of more conveniently located sensor sto be installed. Plans are being modified as of this writing] 7) 6 month plan review This will be conducted next week. Ford to put together a time, place, and agenda. 8) AOB 1) Karen O'Neil will be asked to become more formally involved in directing the spectrometer improvements efforts. 2) The PTCS team wanted the K band receiver left up until mid-june, as they can use it on bright sources for tests in spite of the opacity of the atmosphere. 3) The gbtops account will no longer be used by general users and observers after May 6th. Plans are being made, and people advised of the new locations for all the files that formerly lived under that account. Frank Ghigo is doing much of this work. L. Greenhill's comments: Report 1: Under conditions of variable (in direction) winds < 10 mph, focus at 21-22 GHz has shown shifts of several cm w/o large changes in elevation or temperature. Fitted and eyeballed values for consecutive scans can differ by 3 cm conservatively and 6 cm in apparent worst cases. It would seem the PCTS is needed for K-band even under reasonable weather conditions. It is night, it is not windy. It is cloudy. The tel. structure is pretty thermally stable at this time (22 Mar 9 UT +-). Scan 252 f=-8 9.4 C 52.3 elv Wind 5+- 273 f=19 9.4 C 44.3 elv Wind =<10 289 f>~40! 9.0 C 47.3 elv Wind < 10 294 f=50+ conditions not significantly changed Time=08:50 UT 296 f=49 ditto 297 f=60 ditto. Ripple distorts fit. Eyeball peak at 50. wind W@3 mph 298 f=60 ditto. ditto. 299 f=37 wind N @ 5-9 300 f=17 wind NNW @ 4-6 301 f=48 wind S/SW 3-8 326 f=7 +8.5 C 43.8 wind SW 0-5 327 f=15 +8.5 C 43.4 wind W/SW 7-10 WX Report 11:10 +8.1 C 918 mb 78% W6 mph. Cloudy. NB: When wind blows arm off dish (i.e. away from the surface) focus=15-20. When wind blows arm toward the dish the focus is 50. I wonder if the sign of this effect could be predicted. Software Report 2: 23mar03 10:05 UT (est) At the start of the 5th repeat on source ugc7017, I pressed Start in GO window and the antenna moved 10 deg off in Az. Skyview continued to show the correct target az/el. Do not know where ant got new az/el from. Ops console showed correct RA/DEC. Nothing had changed in the GO window. Mysterious. Report 3: Filling data on titania and thales is slow. 30 minutes of data, acquired in our 4 beam, 2IF, dual polz, 200 MHz setup, with 30s dumps, requires 9 to 20 minutes to fill. I have attempted to do this from the currently open \"file,\" carefully avoiding scans at/near the end of the file. I have used titania and thales as test machines. As this is between 11 PM and 4 AM, I presume the network and machines are lightly or not loaded. It is nearly impractical to perform off-line reduction during observing, source by source. (9 and 20 minutes are < 30 minutes, but one cannot devote 50-100% of ones time to reduction and keep pace with planning and bookeeping for an extended run.) An expanded iards really is needed, as is a better filler. Y. Pihlstroem & M. Murphy Our project intended to search for HCO+ absorption in DLAs. Since the spectrometer was broken, we did not get the backend we proposed for. Instead we used the spectral processor, with its narrower bandwidth we could not cover the complete intended frequency range needed to cover our redshift uncertainties. Due to the spectrometer failure we got more primary observing time than we initially were allocated. Therefore we could to some extent make up for the lack of bandwidth by observing overlapping bands. However, the bandwidth problem combined with the Az track limitations lead to that a few sources were not observed at all. The observations got much more complicated by these restrictions. Despite the limitations, we could get some useful data, but we feel we could not fully make use of the capabilities and advantages of the GBT. For example, with the spectrometer we would have been able to achieve the double integration time. AIPS++ & IARDS failed several times. Data processing in real time failed of reasons I am not clear of. Errors in AIPS++ seem to have transferred into IARDS, thus affecting our observations. The problem was that I reduced data under the same username as we observed, and apparently some info was entered into some logfile used by both GBDISH and IARDS. After this we couldn\'t reduce data as it came in, since we had to use Michaels account for GO/Cleo and Ylvas for IARDS. Another fault occured when IARDS did not reduce our pointing data from the DCR. Jim Braatz found out that we were using old scan numbers. This is a serious problem. Since we observed with both K-band receivers we had to switch setup every now and then, which reset the scan numbers. We suggest scan nrs to be created from the UT, similar to the name of the FITS files in order to prevent overwriting old scans. Retreiving reduced data in AIPS++: After calibrating and averaging scans, and saving this to disk, we can\'t reload the data and add to later reduced data. That should be possible so that you don\'t have to do the same calibration over and over again. GO: A few times the antenna did not take the commanded positions and started to move to a position not commanded. This was done using a source catalog and RUN. When this happened, the GO screen changed the coordinates from RA & Dec to Az/El. GO/Cleo: During the startup of our second run, Cleo did not allow us to control the system. We had to unlock every single control window. After that was fixed, GO did not start. At this time Cleo did not load the configuration, it kept hanging. It took 1.5 hours before we could get started observing, after resetting of the converter rack (operator called in help). Services We got a lot of good support from telescope friends, from operators , and with AIPS++! J. Hibbard We successfully observed for 13/23 alloted hours. 4 hours we gave over to Tyler Bourke; the other time we lost due to a RPC timeout error on the DCR. Once Ron M. told us to use the \"Quick Switch\" option to switch between our pointing observations and spectral observations, we no longer got that error. But the error was screwy - we used the same procedure in our run two weeks earlier, without the DCR errors, and we did manage to successfully switch between the DCR and SP at other times. Once we got the DCR timeout error while we were in the middle of four 10min S.P. observations, when the DCR should not have been getting any signals. Anyways, as I said - using the quick switch mode worked very well, and should be recommended to people. The only other problem I had was with my observe script. I defined some blocks to help switch between the DCR and sp (e.g., change the sc.obs_mode, lo1.refframe, etc). I was using Ch5 of the GBT manual (www.gb.nrao.edu/gbt/GBTMANUAL/GBTMANUAL.pdf), especially the example GO scripts and keyword definitions. When I tried to use these in GO, I got lots of errors. Turns out the keywords are out of date (e.g., lo1.define_velocity; dcr.switch_mode). These should be kept up to date. -- JF 8. Observing schedule Last week ======= Observing for: GBT02B-021, GBT02A-046, GBT02B-018, GBT02B-019, GBT01A-014, GBT02B-003, GBT02A-063, GBT02A-030, Projects completed: GBT02B-021, GBT02B-018, GBT02B-003 May === Astronomy ~ 341 hours Mainentance ~ 134 hours Tests & commissioning ~ 270 hours June === Shutdown ~ 260 hours Astronomy ~ 162 hours Maintenance ~ 157 hous Tests ~ 72 hours Un assigned ~ 70 hours July === Astronomy ~ 190 hours Maintenanc ~ 218 hours Tests ~ 30 hours Un-assigned ~ 80 hours Backup for track work ~221 hours -- RCB Discussion. Carl noted that he had turned on the new visitor reservation web page. Phil encouraged astronomers supporting specific programs to be sure they have had contact with the PIs at least a week prior to the run to find out who is coming and when, and to see that they have made reservations. This will help us to be prepared for their arrival and to avoid last minute surprises. 9. Any other business Richard noted that the report from the PTCS review panel has been received, is quite positive, and has been posted on the PTCS project page at www.gb.nrao.edu/ptcs or by going to www.gb.nrao.edu -> GBT -> GBT Projects -> PTCS Rick noted that the IPG had done tests to measure RFI from the feedarm servo at L-band and had found none. Tests at 340 MHz are planned. Thus, the project to suppress RFI from the feedarm servo is showing very good results. PRJ 02 May 2003 Amended 8 May 2003