MP version and multiple Iterations

Submitted by Yotam on Fri, 2013-03-08 14:25

Forums

Frealign

Hi,

I have been trying to run the multiprocessor reconstruction using the "_mp" version of the frealign executable. Am I right to say that the control script between the non _mp version and the _mp version is almost the same? I have seen discussion at http://emlab.rose2.brandeis.edu/node/1230, and learnt that the IFLAG should be set to 0 and the NCPUS parameter must be set. Leaving these aside, is there any other change that needs to be made to how the parameters are passed to the _mp version? Spider was used to do a coarse refinement for the angles prior to the run.

I ran the same reconstruction using the "_mp" version and the non mp version, and results look significantly different. And, from my understanding of how the structure might look, the mp version reconstruction is worse that the non mp. Is there any reason for why they might be different. I do get tremendous improvements in speed using the _mp version, but the results do not match the non mp versions.

Also, I have used the TMV scripts as a base to create my own scripts, and had a very hard time setting up multiple Iteration run. Even on the TMV script increasing the "end_process" number in the mparameters file to 10, allows the iteration to run, but the program returns a segmentation_fault just as it enters the 4th iteration. Is this related to the architecture of my processors? Am running them on 64bit Inter xeon processors. How do I deal with setting up robust iterations?

niko Fri, 2013-03-08 15:03

Are you using the latest version?

The _mp version only parallelizes the reconstruction step, not the refinement. For parallelized refinement, it is more efficient to run several instances of Frealign. Therefore, the _mp version will only make a difference when you calculate a reconstruction, for example when you run with IFLAG = 0. You also have to set the environmental variable NCPUS to the desired number of CPUs to be used by the _mp version. However, I am not sure if an arbitrary number of CPUs can be used. We usually run it with 8 or even 12 CPUs. There should be no difference in the resulting reconstruction when using the _mp version.

Occasionally, Frealign crashes. So far I was not able to determine why this happens. It appears to happen in certain specific situations for certain data. I was not aware of a problem with the TMV data. However, normally, Frealign runs quite stably over many hundreds of cycles.

Yotam Sat, 2013-03-09 04:09

Thank you for your reply.
I am using the latest version. I think I understand why the reconstruction using _mp looks worse, and just to be clear.. am I right to say that setting the IFLAG = 0 skips refinement and jumps straight into reconstruction? So the right thing to do, is run multiple instances of frealign to do the refinement, compile the .par files into 1 and run the _mp version with IFLAG = 0. Is there a way to disable the reconstruction step in the frealign_vX.exe so that it stops after the refinement. May be this is naive, is it advisable to run a refinement after each iteration(refinement+reconstruction)? Also I would like to point out that even though I set NCPUS to 8, frealign ended up using what I think is 11 CPUs as the usage consistently touched 1080%.

I it seems to be the case with the current data set that multiple iterations is an issue, for some reason there are constant crashes that specifically lead the "non-creation" of an output .par file, and this effectively stalls all the remaining runs. This sometimes occurs as early as in the first iteration.

niko Sat, 2013-03-09 14:36

Make sure you set NCPUs inside the batch file that runs Frealign. The NCPUS variable works on our machines but I am not sure if all machines are set up the same way.

To avoid calculating 3D reconstructions after refinement (IFLAG = 1), you need to set RELMAG = -100.0 (first parameter on CARD 6). Please also see the README.txt file that comes with Frealign.