Re: Roundoff error in SMOOTH

[thompson(at)orpheus.nascom.nasa.gov (William Thompson)]

landsman@stars.gsfc.nasa.gov (Wayne Landsman) writes:

>I was recently surprised to discover that applying the SMOOTH function to a
>non-negative array could  yield an array with negative numbers.  I give an
>example below.  This problem is evidently due to some sort of roundoff error,
>since it does not occur when using double precision.   But it is not obvious to
>me how averaging 9 non-negative numbers (for a 3x3 box smooth) could yield a
>negative number, even allowing for roundoff error.

>Although not obvious from my simple example, this has nothing to do with edge
>effects -- I originally found the problem when 3x3 smoothing a 1024 x 1024 
>array. 

	(rest deleted)

My theory is that the SMOOTH function probably uses Fourier transforms to speed
up the convolution, and this is where the round-off error is sneaking in.

Bill Thompson