Are you sure that you are going to use all the columns? I works with large
data (telecommunication) set too. But usually I know I am not going to use
all the columns. In this case, I use UNIX to cut out those columns I am not
going to use. This can reduce a lot of memory burden for Splus.
Good luck,
Max Zhao
----- Original Message -----
From: "Bos, Roger" <BosR@ny.rothinc.com>
To: <s-news@lists.biostat.wustl.edu>
Sent: Tuesday, December 14, 2004 10:45 AM
Subject: [S] help importing really large files
> Has anyone found a trick to importing really large txt files into S+ 6.2
> under XP? I sent the question to Insightful and their only recommendation
> was to break it up into smaller files. The file is 350 megs, which is
large
> I grant, but my machine has 4 gigs of memory. If I did want to break it
up,
> what utility could I use to do so? Excel is not going to read it either.
> See below for my full question and support's answer. Thanks in advance.
>
>
> I get the "unable to obtain requested dynamic memory" error when I try to
> read in a large file into S+ 6.2 using the following command:
>
> data <-
>
read.table("M:\\tina\\R2000V10SPLS29m.TXT",header=TRUE,sep=",",as.is=TRUE,na
> strings="NA")
> dim(data)
>
> The text file is 347,456 KB big. My windows XP machine has 4 Gigs of
> memory, which I believe is the max it can handle. I also believe that my
> virtual memory is maxed out. I read the FAQ on this topic, but it mostly
> said to optimize the code and I am just trying to read it in. I
understand
> that the operating system steals half of this. Do I need to change any
> setting to make sure S+ is fully utilizing my memory capabilities?
Anything
> else I can try?
>
> --------------------------------------------------------------------------
--
> -----------
> Solution:
>
> The file you are trying to import is a very large file. The calculation we
> use to calculate the size of the data you are trying to import is:
>
> (rows)*(columns)*8*4.5
>
> You should import the file by breaking it into smaller files. Then import
> these smaller files into S-Plus and finally, recombine them inside S-Plus.
>
>
>
> Please let me know if you have any questions.
>
> Sincerely,
>
> Jacob Geballe
>
>
===========================================================================
> Jacob Geballe email: support@insightful.com
> Technical Support Engineer FAX: (206) 283-8691
> Insightful Corporation Phone: (206) 283-8802 ext.235
> www.insightful.com 1-800-569-0123 ext.235
>
===========================================================================
>
> Roger J. Bos, CFA
> Rothschild Asset Management
> 1251 Avenue of the Americas
> New York, NY 10020
> 212-403-5471
>
>
> ********************************************************************** *
This message is for the named person's use only. It may
> contain confidential, proprietary or legally privileged
> information. No right to confidential or privileged treatment
> of this message is waived or lost by any error in
> transmission. If you have received this message in error,
> please immediately notify the sender by e-mail,
> delete the message and all copies from your system and destroy
> any hard copies. You must not, directly or indirectly, use,
> disclose, distribute, print or copy any part of this message
> if you are not the intended recipient.
> **********************************************************************
> --------------------------------------------------------------------
> This message was distributed by s-news@lists.biostat.wustl.edu. To
> unsubscribe send e-mail to s-news-request@lists.biostat.wustl.edu with
> the BODY of the message: unsubscribe s-news
>
|