Big stacks
Forums
Hi everyone!
I am looking for help creating very large stacks for frealign. My current project has 200k particles at 420x420 pixels.
Background:
I am trying to migrate a project from relion to mrc. Using relion_stack_create I am in principal able to make a stack of three times decimated particles (~15 gb) for this project. I got nice results from this and would now like to refine with undecimated particles. When I try to make this stack (420x420x200k) I get an out of memory error on relion (I have ~230 gb). I also tried using em2em to combine all the single frames together, this gave me a file of ~ the right size (130 gb). Yet Frealign tells me that the stack only has 1 frame. I assume em2em does not set the header information correctly. Does anybody have experience with such large stacks? How do you handel them? Do you define several datasets splitting up the data in different stacks?
Thanks in advance!
Frealign should have no
Frealign should have no problem processing such a large stack. I suspect that when you used em2em something went wrong. I assume you are using MRC format. Have you checked the MRC header (using the header command from the MRC program suite)? If the header values are wrong you can try imedit (another MRC program) to correct the header.
Yes, I was also suspecting
In reply to Frealign should have no by niko
Yes, I was also suspecting the mistake on the side of em2em. Using a hex-editor to manipulate the corresponding header bytes was my only idea. But I will rather check out the software you mentioned.
Follow up question: is there a known upper limit concerning stack size? Do you know of cases where people have done classification with particle numbers ranging in lets say 10^6?
Thanks for all the help!
There is a theoretical limit
In reply to Yes, I was also suspecting by lukater
There is a theoretical limit of 2^64 bytes (for 64-bit machines).
I worked with a
In reply to There is a theoretical limit by niko
I worked with a 360*360*1.35E6 stack with no problems.
out of memory error
This is a relion error due to the implementation of relion-stack-create, which requires 2X the memory out the output stack. You may want to let the developer know about it.
There are workarounds, I generate multiple small stacks using relion stack create and append them to each other using imod's addtostack command.
Thanks for the quick reply. I
In reply to out of memory error by Axel
Thanks for the quick reply. I will try this first thing tomorrow morning.
-Lukas