How to skip global searching for good particles?

Submitted by xzhang2005 on Mon, 2013-09-02 20:05

Forums

Frealign

Hi,

In frealign, to save time, how can I only do global search (mode=3) for the particles with worse phase residual? for example, how to only search angles for particles with phase residuals > 75.0, and skip the particles with phase residual <= 75.0? Thanks.

Xing

Alexis Tue, 2013-09-03 10:15

Hi Xing,

I guess the easiest way would be to write a shell/awk/... script to generate a parameter file with only the "bad" particles. You would then run mode 3 on that parameter file, with no 3D reconstruction.
You could then merge the output parameter file back with the "good" particles, do a mode 0 to get a new reconstruction.
You'd then be ready for the next round.

Hope this helps
Alexis

Axel Tue, 2013-09-03 10:32

You may have to also generate a new stack with only the images you want to run mode 3 on if you try this approach. I am not sure how well Frealign deals with the particle numbers if they are not identical to the line number that they are on in the parameter file.

Axel

Alexis Tue, 2013-09-03 10:50

Yes, Axel is right - I should have thought of that. You'd also have to create a new stack. IMOD's newstack command or e2proc2d.py maybe good ways to script this.

niko Tue, 2013-09-03 12:49

I have not tested this recently, but in Mode = 4 the threshold parameter in Frealign would do what you are describing: Particles that have a phase residual better than the threshold will not be subjected to the global search and only undergo a single round of refinement.

xzhang2005 Tue, 2013-09-03 21:26

I tested Mode=4 as suggested, but didn't work. frealign still searches all particles including these with better phase residual than the threshold.

Another way by Alexis & Axel should work, but is too complicated to practice.

Xing

Axel Wed, 2013-09-04 09:36

If you don't mind submitting a lot of jobs, you could write an awk script which submits jobs for only the particles which have a higher phase residual using the first and last particle parameters.
You would then have to recompile the parameter file for the next round, using the new parameters for the particles which had a high phase residual.

xzhang2005 Thu, 2013-09-05 10:05

Hi, Axel,

there would be too mnay jobs to be submitted. For example, in a stack of 30k particles, if 10% are bad and sparsed in stack, then most likely there would be 3000 jobs and thus need to hold 3000 models in memory!

I would suggest the easiest way is to add a new mode in frealign to skip good particles, which would beneifit all users to save computing time. what do you think?

Axel Thu, 2013-09-05 10:34

Hi Xing,
I suppose one way to do it would be to convince Niko to modify the source code, or you could contribute code to do this by modifying the source code yourself. I'm not sure that this is the "easiest" way to do this (unless you already have code to do this, of course).
I wouldn't worry too much about 10% bad particles though, these particles might never align, due to their being damaged, or otherwise disrupted. If you remember the 2008 rotavirus DLP paper, 40% of particles were excluded from the final reconstruction to generate the best result. If 90% of the particles are already aligned, it suggests that some sort search procedure was already performed which aligned the vast majority of particles. I am not sure that running mode 3 would help at all.
However, I'm not sure that I'm the best person to answer this question. Perhaps you should speak with Niko?

Axel

xzhang2005 Fri, 2013-09-06 13:16

Hi, Axel,

thanks for the suggestion. Actually, 10% is just an example. at begining of global seaerch, frealign already finds angle&centers for many particles, and I am confident that these particles could skipped in the next cycle of global search, which could save a lot of time as global search is the most time-consuming step, especially for large dataset.

Xing