Re: Multiple Imputations within a loop

[Spencer Graves <spencer.graves@PDF.COM>]

Dear Ken:

	  Have you considered using "synchronize()" to store progress up to the 
point of failure?
	  Do you think it might be a "garbage collection" problem?  I don't 
know about S-Plus 6.1, but I know that previous releases had memory 
leaks.  If you store progress as you go, you can then just restart from 
the point of failure.  I had multiple computers running for months with 
an extensive set of computer simulations, reporting regularly the time 
taken in the last iteration.  I had to kill it and restart every day or 
two, because otherwise, the time per iteration would go to Inf.
	  Also, have you tried the same thing in R?  There are a few things 
that don't work in one of R and S-Plus but do work in the other.
hope this helps.  spencer graves

Ken Nip wrote:
> Dear Spencer,
>
> Many-thanks for responding to my question.   
>
> Below, I have set out an example to illustrate my question.  
>
> Example, consider the jth data set called 'EXAMPLE.j' as follows:
>
> group family    y1      y2
> 1       1               2.3     3.4
> 1       1               3.4     --
> 1       1               1.5     2.3
> 1       2               --      2.4
> 1       2               2.4     3.4
> 1       2               --      1.2
> 0       1               2.1     --
> 0       1               --      4.5
> 0       1               3.5     2.5
> 0       2               3.4     --
> 0       2               4.4     3.4
> 0       2               2.2     2.1
>
> The standard s-codes for Multiple Imputation are as follows:
>
> 	start <- list() 
> 	for(i in 1:5) 
> 		start[[i]] <- paramIter(emGauss(EXAMPLE.j,
> subset=sample(1:12,6,T), prior="ml")) # draw 5 imputations from 5 parallel
> chains 
> 	EXAMPLE.imp <- impGauss(EXAMPLE.j, prior="non", start=start,
> control=list(niter=50)) # Run for 50 iterations 
>
> Now imagine I have 1000 of similar data sets in a loop as follows:
>
> For (j in 1:1000){
> 	start <- list() 
> 	for(i in 1:5) 
> 		start[[i]] <- paramIter(emGauss(EXAMPLE.j,
> subset=sample(1:12,6,T), prior="ml")) # draw 5 imputations from 5 parallel
> chains 
> 	EXAMPLE.imp <- impGauss(EXAMPLE.j, prior="non", start=start,
> control=list(niter=50)) # Run for 50 iterations 
> }
> The problem lies in creating starting values/imputation sets for each j data
> set.  I would eventually get non-convergence for a set of imputations i.e.
> the imputation would go forever and the loop would not stop. I have tried
> several sub-loops but that didn't help. 
>
> I hope the above example helps.
>
> Many-thanks,
>
> Ken Nip   
>
> -----Original Message-----
> From: Spencer Graves [mailto:spencer.graves@PDF.COM] 
> Sent: 22 April 2003 21:38
> To: Ken Nip
> Cc: 's-news@lists.biostat.wustl.edu'
> Subject: Re: [S] Multiple Imputations within a loop
>
>
> 1.  Have you considered "try"?
>
> 2.  If you provide a simple example that illustrates the problem, it is 
> easier for others to help.
> spencer graves
>
> Ken Nip wrote:
>
> > The data set that I am looking at consists of thousands
> > of n.samples-by-p.variables multivariate data sets containing missing 
> > data.  For each data set I would like to impute missing data using DA 
> > method, so I've decided to explore and use a loop.  Unfortunately, I am 
> > having not much of a success with DA within a large 'for' loop.  I am 
> > having convergence problem for the starting values (created by either 
> > bootstrap or parallel chains) at some point within the loop.  I have 
> > checked that it wasn't the case when I manually run DA on the data sets 
> > which caused the non-convergence. I wonder whether I shouldn't use DA 
> > within a loop at all or is there something else I should be made aware 
> > off.  Has anyone came across with a similar problem before? Any 
> > recommendations would be great.
> > Thanks in advance.
> >
> > Ken Nip
> >
> >
> > ********************************************************************
> > The information transmitted by this email is private and confidential 
> > and is intended for the use of the intended recipients specified 
> > therein. If you are neither an intended recipient nor an employee
> > or agent responsible for delivery to an intended recipient,
> > you should be aware that any dissemination, distribution
> > or copying of this communication is strictly prohibited.
> > If you received this communication in error, please
> > notify us immediately.
> > ********************************************************************