In response to my query (below) I received three answers. Thanks very
kindly to those who responded.
I asked:
I am performing a spectral analysis of a temperature series (> ten
thousand records), but I have a few hundred scattered missing values. I
also know that the missing values are averages of adjacent values.
1. How does spec.pgram deal with missing values? I was thinking I might
interpolate the missing values in order to run spec.pgram. How? I was
thinking about checking each record one at a time with an if statement
inside a for statement, but didn't like where that was headed.
2. Also, I would like to test whether the series differs from random
white noise. Any suggestions?
Simon Rosenfeld wrote:
In my view, simple schemes like linear interpolation substantially distort
reality because of multi-spectral nature of temperature variations. Here is
a brief outline of the scheme I
finally ended up with. It is a four-steps procedure.
1. Extract trends and periodic variations. This step does not require all
the data, missings are OK.
2. Compute variance of detrended data and generate random gaussian noise
with this variance. Fill your gaps (NAs) with this noise.
3. Compute autocorrelation function (this step requires all the data
points,
NAs are not acceptable)
4. OPTIMALLY interpolate your data using ACF into your gaps
This method is satisfactory until you don't try to study probabilistic
properties of your series. Spectral properties
are preserved in most cases if number of your gaps is reasonably small (up
to 15% of all data) and if they themselves do not
follow a certain spectral pattern (for example, missings are more frequent
in winter than in summer, etc)
If you want a more careful analysis and want to preserve also the
probabilistic structure (for substantially non-gaussian series)
you need couple more steps with non-linear transformations
Matthew Zack wrote:
With respect to your second question, two statistical tests have been used
to test for white noise in spectral analysis: Fisher's kappa statistic and
Bartlett's Kolmogorov-Smirnov test (Fuller WA. Introduction to statistical
time series. New York: John Wiley & Sons, Inc., 1976).
If you define the amplitude periodogram, J(k), as
J(k) = 0.5*n*(a(k)**2 + b(k)**2)
where n=the number of observations in the time series,
a(k)=the cosine coefficient at k
b(k)=the sine coefficients at k
then Fisher's kappa statistic tests whether the largest, J(k), differs
statistically significantly from the mean of the J(k).
Bartlett's Kolmogorov-Smirnov test compares the normalized cumulative
periodogram,
F(j), of the series
F(j) = [sum of (k=1 to k=j of J(k)] / [sum of (k=1 to k=m of J(k)],
where m=n/2 if n is even, (n-1)/2 if n is odd, and j=1,2,3,. . .m-1.
to the cumulative distribution function of a uniform(0,1) random variable
using the Kolmogorov-Smirnov test. I don't know whether an S-PLUS
implementation of these tests is available.
Derek N. Eder sent a section of a manuscript that I didn?t take the liberty
of poasting his email is derek.eder@neuro.gu.se
--
Dr. M. Henry H. Stevens
Postdoctoral Associate
Department of Ecology, Evolution, & Natural Resources
14 College Farm Road
Cook College, Rutgers University
New Brunswick, NJ 08901-8551
email: hstevens@rci.rutgers.edu
phone: 732-932-9631
fax: 732-932-8746
|