An idea for the brave-of-heart:
There is a boot-time switch on some versions of Windows
/3gb
It isn't completely clear which OS the flag is available for. The help files
suggest Windows NT and later servers, and Windows XP Pro.
http://support.microsoft.com/default.aspx?scid=kb;en-us;319043
By default the OS kernel grabs 2gb of RAM, leaving 2gb for applications.
According to MS documentation, adding the switch reduces the OS to 1gb and
leaves 3gb for applications.
Is the 2gb limit you report innate to S+ or is it this limit?
Donald C.E. Robinson
Sr. Systems Ecologist, Forestry Team
ESSA Technologies Ltd
Vancouver, Canada
http://www.essa.com <http://www.essa.com>
+1 (604) 535 1997
-----Original Message-----
From: s-news-owner@lists.biostat.wustl.edu
[mailto:s-news-owner@lists.biostat.wustl.edu]On Behalf Of Tony Plate
Sent: December 15, 2004 9:28 AM
To: Bos, Roger; s-news@lists.biostat.wustl.edu
Subject: Re: [S] help importing really large files
scan() can import larger text files than can read.table()
Read the docs for scan, and then supply as much info as you can about the
file contents. in particular, pay attention to the "DETAILS" section in
the help page for "scan", especially the paragraph where it talks about
preallocating vectors via the 'what' argument.
350 megs is right at the borderline of what I can successfully read with
scan() in S-PLUS 6.1 on my Windows2000 machine.
Even though you have 4gig of memory on your machine, S-PLUS cannot use more
than 2gig, and in practice, I've never seen my S-PLUS get more than 1.7gig,
and it often seems to start getting the "unable to obtain requested dynamic
memory" at somewhere around 1.4gig. Closing and restarting the S-PLUS
application can help with reading very large files, as can closing other
applications.
-- Tony Plate
At Tuesday 08:45 AM 12/14/2004, Bos, Roger wrote:
>Has anyone found a trick to importing really large txt files into S+ 6.2
>under XP? I sent the question to Insightful and their only recommendation
>was to break it up into smaller files. The file is 350 megs, which is
large
>I grant, but my machine has 4 gigs of memory. If I did want to break it
up,
>what utility could I use to do so? Excel is not going to read it either.
>See below for my full question and support's answer. Thanks in advance.
>
>
>I get the "unable to obtain requested dynamic memory" error when I try to
>read in a large file into S+ 6.2 using the following command:
>
>data <-
>read.table("M:\\tina\\R2000V10SPLS29m.TXT",header=TRUE,sep=",",as.is=TRUE,n
a
>strings="NA")
>dim(data)
>
>The text file is 347,456 KB big. My windows XP machine has 4 Gigs of
>memory, which I believe is the max it can handle. I also believe that my
>virtual memory is maxed out. I read the FAQ on this topic, but it mostly
>said to optimize the code and I am just trying to read it in. I understand
>that the operating system steals half of this. Do I need to change any
>setting to make sure S+ is fully utilizing my memory capabilities?
Anything
>else I can try?
>
>---------------------------------------------------------------------------
-
>-----------
>Solution:
>
>The file you are trying to import is a very large file. The calculation we
>use to calculate the size of the data you are trying to import is:
>
>(rows)*(columns)*8*4.5
>
>You should import the file by breaking it into smaller files. Then import
>these smaller files into S-Plus and finally, recombine them inside S-Plus.
>
>
>
>Please let me know if you have any questions.
>
>Sincerely,
>
>Jacob Geballe
>
>===========================================================================
> Jacob Geballe email: support@insightful.com
> Technical Support Engineer FAX: (206) 283-8691
> Insightful Corporation Phone: (206) 283-8802 ext.235
> www.insightful.com 1-800-569-0123 ext.235
>===========================================================================
>
>Roger J. Bos, CFA
>Rothschild Asset Management
>1251 Avenue of the Americas
>New York, NY 10020
>212-403-5471
>
>
>********************************************************************** *
>This message is for the named person's use only. It may
>contain confidential, proprietary or legally privileged
>information. No right to confidential or privileged treatment
>of this message is waived or lost by any error in
>transmission. If you have received this message in error,
>please immediately notify the sender by e-mail,
>delete the message and all copies from your system and destroy
>any hard copies. You must not, directly or indirectly, use,
>disclose, distribute, print or copy any part of this message
>if you are not the intended recipient.
>**********************************************************************
>--------------------------------------------------------------------
>This message was distributed by s-news@lists.biostat.wustl.edu. To
>unsubscribe send e-mail to s-news-request@lists.biostat.wustl.edu with
>the BODY of the message: unsubscribe s-news
--------------------------------------------------------------------
This message was distributed by s-news@lists.biostat.wustl.edu. To
unsubscribe send e-mail to s-news-request@lists.biostat.wustl.edu with
the BODY of the message: unsubscribe s-news
|