VARRAY, memory & extracting subarrays

[Kristine Hensel <kristine(at)esands.com>]

I'm processing large (105 MB) arrays of images, and I've been running
into memory problems.  (Not surprisingly, right?)  I've started using
Eric Korpela's VARRAY routine, which has helped, but I still can't
manage to extract a subarray without using all of the available memory. 
Theoretically I have 1 GB of memory, and we've tried maximizing every
system variable that we can, but I'm still crashing ("Unable to allocate
memory to create array") when I try to run my image processing program. 

I can declare the main array and subarray using VARRAY, and I can read
images into the main array.   However, once I try to put anything from
main array in the subarray, e.g. subarray=main_array[i:j,k:l,*], or even
subarray=main_array, I start chewing up memory.  

Two not-very-good workarounds are:
 - using subarray=temporary(main_array), but then I lose main_array. I
suppose I could read the main array into main_array and then into some
temporary variable, so that I could do subarray=temporary(tmp_array),
but that involves more I/O
 - writing the subsection of main_array to a file, then reading it
directly into subarray - this is slow

I've attached sample code below, along with the results of
memory(/current) along the way.

Any assistance with VARRAY would be greatly appreciated.  

Thanks,
 Kristine

-- 
Kristine Hensel                              
Environmental Systems & Services      Phone: +61-3-9864-5300
405 Toorongo Rd                         FAX: +61-3-9822-8028    
Hawthorn East, VIC 3123 Australia    e-mail: kristine@esands.com

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
pro sector_noaa
  
  filename = '/home/meteor/mapped_data/200105171839N14.map'

  openr, lun, filename, /get_lun, error=err
  nlon=0L
  nlat=0L
  nchannel=0L
  readu, lun, nlon, nlat, nchannel
  lons=fltarr(nlon)
  lats=fltarr(nlat)
  readu, lun, lons, lats
                                ; declare main images array:
  file_delete, '/home/meteor/mapped_data/images.dat', /quiet

*** Memory in use: 71 kB

  images = varray("/home/meteor/mapped_data/images.dat",byte(0), $
                  nlon,nlat,nchannel,/writable)

*** Memory in use: 71 kB
                                ; read in images array:
  readu, lun, images

*** Memory in use: 71 kB

  free_lun, lun

                                ; declare subarray:
  file_delete, '/home/meteor/mapped_data/sector_images.dat', /quiet

*** Memory in use: 70 kB

  sector_images = varray("/home/meteor/mapped_data/sector_images.dat", $
                         byte(0),nlon,nlat,nchannel,/writable)

*** Memory in use: 70 kB
                                ; write out subarray:
  openw, lun, filename+'_sector', /get_lun, error=err
  writeu, lun, images[0:nlon-1,0:nlat-1,*]
  free_lun, lun
                                ; read in sector images:
  openr, lun, filename+'_sector', /get_lun, error=err
  readu, lun, sector_images
*** Memory in use: 70 kB
  free_lun, lun

                                ; try to copy directly:
  sector_images = images
*** Memory in use: 10531 kB

end