Re: Importing large data sets?

[Stephen Ban <ban@zoology.ubc.ca>]

I've tried importing it from a few different formats - there's the raw text 
file (tab delimited), and a couple of different SPSS v.10 formats. Oddly, 
in the past I've found that S-PLUS imports large files faster from SPSS 
formats than it does from raw ASCII text; perhaps because the SPSS files 
are in binary format?
I'm using the GUI File->Import function to bring the file in, so I'm not 
sure what method it is using when it attempts to open it.
Thanks,
Stephen


At 06:06 PM 24/04/2004, james.holtman@convergys.com wrote:




> What is the format of the file?  Are you using 'read.table', 'scan', etc.
> to read it in?
>
> 'read.table' will require that you read the entire file in and then convert
> is. This would take most of you memory assuming that you file is ASCII and
> you are converting it.
>
> Is all the data in the same format (e.g., numeric)?  If it is, then
> consider converting it to binary with PERL, for example, and then using
> 'read.binary'.  This will only require as much space as the object.
>
> You will probably have problems with doing any processing since you may
> have to make one, or more, copies of the data while processing.
> __________________________________________________________
> James Holtman        "What is the problem you are trying to solve?"
> Executive Technical Consultant  --  Office of Technology, Convergys
> james.holtman@convergys.com
> +1 (513) 723-2929
>
>
>
>
>                       Stephen 
> Ban 
>
>                       <ban@zoology.ubc.ca>             To: 
> s-news@lists.biostat.wustl.edu
>                       Sent 
> by:                         cc: 
>
>                       s-news-owner@lists.biosta        Subject:  [S] 
> Importing large data sets?
>                       t.wustl.edu 
>
>
>
>
>                       04/23/2004 
> 16:56 
>
>
>
>
>
>
>
>
> I'm trying to open/import a large (229MB) data file into S-PLUS 6.1. My
> machine has 1GB of RAM, and oodles of hard drive space. Every time I try to
>
> import the file, I get the error "Problem in rep.int(1:n, times): Unable to
>
> obtain requested dynamic memory."
>
> Any ideas as to what I can do to get around this? SPSS v10 has no problems
> opening the same file.
>
> Thanks,
> Stephen
>
> --------------------------------------------------------------------
> This message was distributed by s-news@lists.biostat.wustl.edu.  To
> unsubscribe send e-mail to s-news-request@lists.biostat.wustl.edu with
> the BODY of the message:  unsubscribe s-news