Re: Input line too long???

[Pavel Romashkin <promashkin(at)cmdl.noaa.gov>]

Hi David,
I played with your example and I think I have a solution. The input
buffer limitation is only in effect if you read formatted. I made a file
by repeating your example line 2000 times, obtaining 10000 element line.
Each record in your line is 8 bytes long, including a space. Try reading
unformatted and it works perfectly:

; Make fake data file with 1 line
; with 10000 data entries and open it.
openr, unit, 'test.txt', /get_lun
; Since you know the number of data points, make X:
x = strarr(10000)
; Fill X with 8 byte strings.
x[*] = '12345678'
readu, unit, x

Now you can convert X into float and use it.
Cheers,
Pavel

David Bowman wrote:
> 
> Greetings all,
> 
> I have a data file that essentially contains a very long vector of
> floating point numbers.  The file (written by a trusty old FORTRAN
> program) gives the number of elements in the record on the first line.
> The second line contains all the data, with each element separated by
> two spaces.  For example:
> 
>   .123348  .126664  .130130  .133735  .137486   and so on.
> 
> Now in the past I've read this data with no problem simply using
> 
> x=fltarr(n)
> readf,1,x,format='(nF0)'
> 
> where n is the number of elements in the vector.  However, I've
> recently tried to read in a much larger vector, and IDL chokes, saying:
> 
> % Input line is too long for input buffer of 32767 characters.
> 
> The datafile in question contains 5856 elements (five of which are in
> my little example above). All told, there are 55278 characters in the
> line in question.
> 
> Nothing I've tried works...  Reading in elements by element, with
> formatting, without formatting...  I've even tried using the BUFSIZE
> keyword to OPEN, with no result.  The IDL help says that I'm limited to
> reading in files smaller than 2^31 bytes, but I'm certainly well below
> that.
> 
> I'd love to simply rewrite the I/O section of the program that made the
> file to begin with, but I didn't write it and I don't have the source
> code.
> 
> Any Ideas?  I'm using version 5.3 on a Mac G3 with 128 Mb RAM.
> 
> Thanks in advance...
> David
> 
> --
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> David Bowman         "Smash forehead on keyboard to continue."
> Laboratoire Tectonique
> Institut de Physique du Globe - Paris
> 4 Place Jussieu
> 75252 Paris Cedex 05
> France