My reccomendation:
Using SAS to seperate the files. But you need to know how large the file
Splus can handle (How many records in source file).
Import all the files as SAS data sets.When all files imported, rbind will
finish the work.
I used to process a large file in this way.
Code in SAS:
Assume only 10000 can be imported in Splus.
data temp1;
infile 'c:\largefile' firstobs=1 obs=10000;
input a b $ c;
run;
data temp2;
infile 'c:\largefile' firstobs=10001 obs=20000;
input a b $ c;
run;
..
If you are an SAS expert, a simple macro would do the trick.
Then you got all the data sets temp1, temp2, ...
Import them directly by Splus, the file name for these dataset maybe
temp1.sas7bdat,temp2.sas7bdat ...
Hope this helps,
Chushu Gu
----- Original Message -----
From: "Bos, Roger" <BosR@ny.rothinc.com>
To: <s-news@lists.biostat.wustl.edu>
Sent: Tuesday, December 14, 2004 10:45 AM
Subject: [S] help importing really large files
> Has anyone found a trick to importing really large txt files into S+ 6.2
> under XP? I sent the question to Insightful and their only recommendation
> was to break it up into smaller files. The file is 350 megs, which is
large
> I grant, but my machine has 4 gigs of memory. If I did want to break it
up,
> what utility could I use to do so? Excel is not going to read it either.
> See below for my full question and support's answer. Thanks in advance.
>
>
> I get the "unable to obtain requested dynamic memory" error when I try to
> read in a large file into S+ 6.2 using the following command:
>
> data <-
>
read.table("M:\\tina\\R2000V10SPLS29m.TXT",header=TRUE,sep=",",as.is=TRUE,na
> strings="NA")
> dim(data)
>
> The text file is 347,456 KB big. My windows XP machine has 4 Gigs of
> memory, which I believe is the max it can handle. I also believe that my
> virtual memory is maxed out. I read the FAQ on this topic, but it mostly
> said to optimize the code and I am just trying to read it in. I
understand
> that the operating system steals half of this. Do I need to change any
> setting to make sure S+ is fully utilizing my memory capabilities?
Anything
> else I can try?
>
> --------------------------------------------------------------------------
--
> -----------
> Solution:
>
> The file you are trying to import is a very large file. The calculation we
> use to calculate the size of the data you are trying to import is:
>
> (rows)*(columns)*8*4.5
>
> You should import the file by breaking it into smaller files. Then import
> these smaller files into S-Plus and finally, recombine them inside S-Plus.
>
>
>
> Please let me know if you have any questions.
>
> Sincerely,
>
> Jacob Geballe
>
>
===========================================================================
> Jacob Geballe email: support@insightful.com
> Technical Support Engineer FAX: (206) 283-8691
> Insightful Corporation Phone: (206) 283-8802 ext.235
> www.insightful.com 1-800-569-0123 ext.235
>
===========================================================================
>
> Roger J. Bos, CFA
> Rothschild Asset Management
> 1251 Avenue of the Americas
> New York, NY 10020
> 212-403-5471
>
>
> ********************************************************************** *
This message is for the named person's use only. It may
> contain confidential, proprietary or legally privileged
> information. No right to confidential or privileged treatment
> of this message is waived or lost by any error in
> transmission. If you have received this message in error,
> please immediately notify the sender by e-mail,
> delete the message and all copies from your system and destroy
> any hard copies. You must not, directly or indirectly, use,
> disclose, distribute, print or copy any part of this message
> if you are not the intended recipient.
> **********************************************************************
> --------------------------------------------------------------------
> This message was distributed by s-news@lists.biostat.wustl.edu. To
> unsubscribe send e-mail to s-news-request@lists.biostat.wustl.edu with
> the BODY of the message: unsubscribe s-news
>
|