GBTIDL getchunk and putchunk

Wednesday 21 May 2025
This documentation is courtesy of Amanda Kepley.

The combination of wide bandwidths, high time resolution, and high
spectral resolution makes VEGAS data sets very large, requiring more
efficient ways of dealing with the data. The GBTIDL getchunk and
putchunk routines significantly increase the processing speed for
large data sets by loading many spectra (i.e., data chunks) into
memory at once rather than continually reading and writing from
disk. This speed increase comes at the expense of having to manually
free the memory used by the data chunks via the routine data_free to
avoid memory leaks.

As elementary example of how getchunk and putchunk can be used, we
show a small IDL procedure that can be used to smooth a large number
of spectra. This routine shows how to use set_data_container to place
a spectrum from the data chunk into a default data container,
manipulate it, and then put it back into the original data container.

----------------------------------------------------------------------

filein, myfilein
fileout, myfileout

mynchan=10

scans = get_scan_numbers(/unique)
for i = 0, n_elements(scans) - 1 do begin
   chunk = getchunk(scan=scans[i],count=nchunk)
   for j = 0, nchunk - 1 do begin
      set_data_container, chunk[j]
      gsmooth, mynchan,/decimate
      data_copy, !g.s[0],chunk[j]
   endfor
   putchunk, chunk
   data_free, chunk
endfor

----------------------------------------------------------------------

For a complete list of parameters that getchunk can select on, get a
single record from the relevant data set and use the listcols
procedure. Note that getchunk can only retrieve one scan at a time.

A more complex example uses getchunk and putchunk to generate an OFF
spectrum for an on-the-fly map using the ends of the rows. It shows
how to use the accum buffer with the data chunk rather than the
default data container.

----------------------------------------------------------------------

filein, infile
fileout, outfile

; Get which scans are in the map.
mapscans = get_scan_numbers(count, procedure='RALongMap')
plnum = 0
ifnum = 0
fdnum = 0

; Get information on the data. Assume that the first scan is
; representative of the entire map
myinfo = scan_info(mapscans[0])
map_nint = myinfo.n_integrations

for i = 0, n_elements(mapscans) - 1 do begin

   ;; get offs
   offlist = indgen(2*noffs)
   offlist[noffs:2*noffs-1] =  map_nint - reverse(indgen(noffs) ) - 1
   offchunk = getchunk(scan=mapscans[i], $
                       plnum=plnum, $
                       ifnum=ifnum, $
                       fdnum=fdnum, $
                       int=offlist, count=noffchunks)
   sclear
   for j = 0, noffchunks -1 do accum, dc=offchunk[j]
   ave

   data_free, offchunk

   ;; copy the off to a dc
   copy, 0, 1
   off = getdata(0)

   tsysvec=gain*off             ;; probably want to specify the gain per beam
   vtsys = median(tsysvec)

   ; get all the integrations for a row.
   rowchunk = getchunk(scan=mapscans[i], $
                       plnum=plnum, $
                       ifnum=ifnum, $
                       fdnum=fdnum, $
                       count=nrowchunks)

   ;; now go through the integrations and calibrate the ons.
   for j = 0, nrowchunks - 1 do begin

      set_data_container, rowchunk[j]

      ;; calculate the Ta
      copy, 0, 2
      subtract, 2, 1
      divide, 0, 1

      ;; put calibrated data back in main dc
      vec = getdata(0)
      vec1 = vtsys * vec
      setdata,vec1

      ; copy the final data back into the row
      data_copy, !g.s[0], rowchunk[j]
      rowchunk[j].units='Ta'
      rowchunk[j].tsys=vtsys

   endfor

   putchunk, rowchunk
   data_free, rowchunk

endfor