After calling Insightful support, I learned that this is simply
a characteristic of S-Plus and how it uses memory and that data large
objects are a problem in explicit loops, even For loops.
I got around this with a complete hack/kludge: I wrote a FORTRAN program
(I'm most comfortable with that, though C would probably be a better
choice) that ingests all the data I want into arrays of the appropriate
dimension. I then write out the arrays in a file that emulates an S-Plus
dump file, which is all ASCII. Ingesting these data into S-Plus is then
trivially easy.
I also want to thank everyone with suggestion for the use of assign() and
eval(parse(...)). These are very powerful, but I couldn't get them to work
for me because the left hand side was addressing parts of arrays. I found
that S-Plus was trying to stuff these into objects named things like
array[, jday[j], khr[j]], which was very strange, to say the least.
However, I'm now past my initial problem and now have a way in for all
18,000+ data sets.
Cheers!
Kim Elmore
At 02:32 PM 6/10/2004, jose Bartolomei wrote:
Dear all,
I have more or less the same concern.
My S-Plus too halt with small data sets in many tasks.
For example I am tried just to merge to 2MB data.frame
Or I to import 1GB data base and it compleate freeze.
The CPU and FP Memory graph present the same huge picks.
jose
From: "Kim Elmore" <Kim.Elmore@noaa.gov>
To: S-News <s-news@wubios.wustl.edu>
Subject: [S] Arrays and dyanmic memory
Date: Wed, 09 Jun 2004 13:22:47 -0500
I'm dealing with a fair bit of data from various sources -- 31 different
weather forecast models for 1892 locations over 55 days at 15 different
times, and 5 variables. I decided to put everything for each variable
into a single 1892 x 55 x 15 x 31 array. Assuming double precision, this
comes out to a single object that's about 370 MB. This is well below the
2 GB limit and so I thought I'd have plenty of room to manipulate the data.
I can declare the arrays without a problem, but when I try do much with
the arrays (I'm using a For loop in a Batch job), such as:
For (i = 1:n,
{
my.array1[, i, j, k] <- a.matrix[, 1]
my.array2[, i, j, k] <- b.matrix[, 2]
my.array3[, i, j, k] <- c.matrix[, 3]
my.array4[, i, j, k] <- d.matrix[, 4]
my.array5[, i, j, k] <- e.matrix[, 5]
})
S-plus runs out of dynamic memory and halts. I watch this in the Task
Manager, and I never see a huge spike in memory usage, so I assume that
some request for memory beyond 2 GB is being generated. Can anyone yield
some insight into why this is happening? The matrices aren't very big,
so even if all five arrays and matrices were resident, I'm still well
under the 2 GB limit.
Kim Elmore
Kim Elmore, Ph.D.
University of Oklahoma
Cooperative Institute for Mesoscale Meteorological Studies
"All of weather is divided into three parts: Yes, No, and Maybe. The
greatest of these is Maybe" The original Latin appears to be garbled.
--------------------------------------------------------------------
This message was distributed by s-news@lists.biostat.wustl.edu. To
unsubscribe send e-mail to s-news-request@lists.biostat.wustl.edu with
the BODY of the message: unsubscribe s-news
_________________________________________________________________
MSN Toolbar provides one-click access to Hotmail from any Web page FREE
download! http://toolbar.msn.click-url.com/go/onm00200413ave/direct/01/
Kim Elmore, Ph.D.
University of Oklahoma
Cooperative Institute for Mesoscale Meteorological Studies
"All of weather is divided into three parts: Yes, No, and Maybe. The
greatest of these is Maybe" The original Latin appears to be garbled.
|