scan() can import larger text files than can read.table()
Read the docs for scan, and then supply as much info as you can about the
file contents. in particular, pay attention to the "DETAILS" section in
the help page for "scan", especially the paragraph where it talks about
preallocating vectors via the 'what' argument.
350 megs is right at the borderline of what I can successfully read with
scan() in S-PLUS 6.1 on my Windows2000 machine.
Even though you have 4gig of memory on your machine, S-PLUS cannot use more
than 2gig, and in practice, I've never seen my S-PLUS get more than 1.7gig,
and it often seems to start getting the "unable to obtain requested dynamic
memory" at somewhere around 1.4gig. Closing and restarting the S-PLUS
application can help with reading very large files, as can closing other
applications.
-- Tony Plate
At Tuesday 08:45 AM 12/14/2004, Bos, Roger wrote:
Has anyone found a trick to importing really large txt files into S+ 6.2
under XP? I sent the question to Insightful and their only recommendation
was to break it up into smaller files. The file is 350 megs, which is large
I grant, but my machine has 4 gigs of memory. If I did want to break it up,
what utility could I use to do so? Excel is not going to read it either.
See below for my full question and support's answer. Thanks in advance.
I get the "unable to obtain requested dynamic memory" error when I try to
read in a large file into S+ 6.2 using the following command:
data <-
read.table("M:\\tina\\R2000V10SPLS29m.TXT",header=TRUE,sep=",",as.is=TRUE,na
strings="NA")
dim(data)
The text file is 347,456 KB big. My windows XP machine has 4 Gigs of
memory, which I believe is the max it can handle. I also believe that my
virtual memory is maxed out. I read the FAQ on this topic, but it mostly
said to optimize the code and I am just trying to read it in. I understand
that the operating system steals half of this. Do I need to change any
setting to make sure S+ is fully utilizing my memory capabilities? Anything
else I can try?
----------------------------------------------------------------------------
-----------
Solution:
The file you are trying to import is a very large file. The calculation we
use to calculate the size of the data you are trying to import is:
(rows)*(columns)*8*4.5
You should import the file by breaking it into smaller files. Then import
these smaller files into S-Plus and finally, recombine them inside S-Plus.
Please let me know if you have any questions.
Sincerely,
Jacob Geballe
===========================================================================
Jacob Geballe email: support@insightful.com
Technical Support Engineer FAX: (206) 283-8691
Insightful Corporation Phone: (206) 283-8802 ext.235
www.insightful.com 1-800-569-0123 ext.235
===========================================================================
Roger J. Bos, CFA
Rothschild Asset Management
1251 Avenue of the Americas
New York, NY 10020
212-403-5471
********************************************************************** *
This message is for the named person's use only. It may
contain confidential, proprietary or legally privileged
information. No right to confidential or privileged treatment
of this message is waived or lost by any error in
transmission. If you have received this message in error,
please immediately notify the sender by e-mail,
delete the message and all copies from your system and destroy
any hard copies. You must not, directly or indirectly, use,
disclose, distribute, print or copy any part of this message
if you are not the intended recipient.
**********************************************************************
--------------------------------------------------------------------
This message was distributed by s-news@lists.biostat.wustl.edu. To
unsubscribe send e-mail to s-news-request@lists.biostat.wustl.edu with
the BODY of the message: unsubscribe s-news
|