Re: lost data?

["Liam E. Gumley" <Liam.Gumley(at)ssec.wisc.edu>]

src wrote:
> Is there a bug in IDL's Save/Restore command?  I've just spent the last 18
> days running a Monte Carlo simulation to seemingly lose all my data.  The
> problem occured when our license manager stopped responding (network
> problem) hence the IDL session running the simulation crashed.  The MC
> code is designed to save results periodically as it runs (just in case
> this sort of thing happens).  I've just tried:
> 
> "restore, 'mc_file.sav', /Verbose"
> 
> only to get:
> 
> % RESTORE: IDL version 5.3 (linux, x86).
> % RESTORE: Truncated save file, restored as much as possible:
> 
> That "resored as much as possible:" is in fact 0 (zero).  Despite the file
> itself being 17 Mb!  Some of my .sav files are a lot bigger than this, yet
> don't seem to have any problems.  Is there anyway to recover this file, or
> prevent this happening again in the future?  I'm going to very upset to
> lose 18 days work...

I ran into the same problem a long time ago when I used to run long
FORTRAN jobs on a VAX/VMS system. My batch job wrote data to a netCDF
file as it ran, but if the batch job died, the file would be incomplete
and therefore unusable.

Fortunately, netCDF offers a mechanism that allows you to synchronize an
open file to disk. In IDL, it's done as follows:

ncdf_control, cdfid, /sync

where cdfid is the identifier of an open netCDF file. You will need to
create a netCDF file which has an unlimited dimension; see the example
for ncdf_vardef in the online help for an example.

Another option, which I have not tested, is to write the results from
your simulation to a binary output file (not a SAVE file), and then
periodically execute the flush procedure, e.g.,

flush, lun

where lun is the logical unit number of the file to be synchronized to
disk.

Cheers,
Liam.
http://cimss.ssec.wisc.edu/~gumley/