Converting xmipp angles / shifts to frealign

Submitted by mjb on Fri, 2011-08-26 09:44

Forums

Frealign

Hi there!

Has anyone ever tried converting xmipp angles/shifts from a doc file to frealign.
In my case, I would get the other necessary information (micrograph id, defocus, astigmatism, ...) and the images themselves by imagic-to-frealign.

doc file:

; Headerinfo columns: rot (1), tilt (2), psi (3), Xoff (4), Yoff (5) , Ref (6), Flip (7), maxCC (8)
; prot/p000001.xmp
1 8 65.15070 29.55220 29.52542 -0.81344 6.31747 5968.00000 1.00000 0.30295
; prot/p000002.xmp
2 8 52.38140 36.61310 40.64407 -1.97147 4.19404 19032.0000 1.00000 0.29440
; prot/p000003.xmp
3 8 67.14420 40.91780 75.15254 7.42509 0.37411 19124.00000 0.00000 0.32246

rot, tilt, psi, Xoff, Yoff and Flip would be relevant...

Any hints will be highly appreciated.

Best
Matthias Brunner

dlyumkis Fri, 2011-08-26 14:19

Hi Matthias,

we recently had to do this, and I will summarize what we did below:

1) Xmipp contains mirroring operations in its doc files, which need to be taken into account and converted into a different set of Euler angles. I'm attaching a python script to do the basic calculations, though you will have to manipulate it a bit to loop over all Euler angles, create files, etc.
2) The Euler angles from this output need to all be multiplied by -1, i.e. phi(Frealign) = phi(Xmipp)*-1, ..., etc.
3) Xmipp Eulers are saved as rot (phi), theta (theta), and psi (psi) in the headers, whereas Frealign request the order to be reversed, i.e. psi, theta, phi.
4) Xmipp Xoff needs to be multiplied by -1, but the Yoff remains unchanged.

These operations should change your rot (1), tilt (2), psi (3), Xoff (4), Yoff (5) , and Flip (7) values in the Xmipp headers to a format that Frealign can read, that is PSI, THETA, PHI, SHX, SHY. In summary: After the mirroring operation has been accounted for by the attached script, the Xmipp parameters are converted as:

PSI = -psi
THETA = -tilt
PHI = -rot
SHX = -Xoff
SHY = Yoff

Hope this helps,
Dmitry

mjb Tue, 2011-08-30 06:07

Thank you for your generous help!
I will try this ASAP.

Regarding the mirror operation - do you have any idea why the xmipp equivalent angles script returns different results?


$ removeMirroringOperationFromXmippOutput.py
old Euler angles: phi=80.000000, theta=45.000000, psi=20.000000
new Euler angles: phi=-100.000000, theta=134.999999, psi=160.000001


$ equivalentAngles 80 45 20
(80,45,20) Initial
(260,135,160) Mirror Y; Another set
(260,135,340) Mirror X; Another set
(260,-45,20) Mirror XY; Another set
(80,-135,340) Mirror Y
(80,-135,160) Mirror X
(80,45,200) Mirror XY
(260,-45,200) Initial; Another set

dlyumkis Tue, 2011-08-30 15:10

Hi Matthias,

There is some degeneracy in Euler transformations, i.e. more than one way to get to the same orientation. In fact, all 3 combinations of Euler angles (rot,tilt,psi = -100,135,160 ; 260,135,160 ; and 80,-135,340) will create an identical rotation. You can check this with the xmipp_project program, all 3 combinations of Euler angles will result in an identical projection. You can also double check that they are the same using the "xmipp_angular_distance" program. Any mirroring operation should result in an angular distance that is ~120 degrees between the two Euler combinations, whereas the degenerate Euler angles should return a value close to 0 (e.g. 5.69182e-07 in the case of comparing -100,135,160 with 260,135,160).

What I don't really understand is that in the equivalentAngles script, the Eulers above are referring to a Mirroring operation about the Y axis, whereas if you look at the transformed projection image, it is mirrored along the horizontal axis (which I believe is X in their coordinate system). I double checked this previously and found that, if there is a mirroring operation specified in an Xmipp .doc file, then to get the image into the proper orientation, it needs to be shifted first, then mirrored about the X axis, and only then the Euler angles in the doc file apply. If you apply the shift parameters to the 2D image, then use "xmipp_mirror -i [inimage] -o [outimage] -flipX", it should produce an equivalent projection as the reference.

That brings me to the last point, that is the shifts. You're right that if you flip the particle, you're affecting the shift, but that also depends on the order of operations. If you (1) mirror, (2) shift, then the sign will have to be switched ; if you (1) shift, (2) mirror, then the sign is not switched. What Xmipp does is shift first to center the particle, then mirror, then apply the Euler angles, so the sign is unaffected. The reason that you have to change the sign of the Xoff to -X, whereas the Yoff remains unchanged has to do with the coordinate system convention differences between Xmipp and Frealign. We checked this empirically and found that the combination of operations above provides the correct volume.

Hope this helps,
Dmitry

mjb Wed, 2011-08-31 11:55

First of all, your removeMirroringOperationFromXmippOutput.py seems to be correct - I also get correct results.

However, by trial/error and checking projections with FMATCH=T, I had to came up with a different conversion scheme from xmipp to frealign (xmipp values in lower case, differences bold):


PSI = +psi
THETA = -tilt
PHI = +rot
SHX = -Xoff
SHY = -Yoff

This seems to work, I get nice phase residuals and a correct volume...but how can this be?
Could our different experiences be caused by the handedness/coordinate system of the input 3D or program versions?

I use the following input scheme to frealign:
PSI,THETA,PHI,SHX,SHY - from xmipp doc file (xmipp particle images generated by: em2em IMAGIC->SPIDER)
particle stack for frealign: imagic-to-frealign
3D volume: em2em IMAGIC->MRC (Consider different coordinate systems: NO)

This 3D volume conversion has the effect that the handedness of the particle changes and is wrong, at least when opening the frealign-generated MRC file directly in chimera. Is this to be expected and ok?

All this makes me wonder how much time the EM community actually has to spend converting between different formats...Thanks again for taking the time!

Versions: xmipp 2.4, EM2EM (vs. 28-Mar-2011), IMAGIC_FREALIGN (vs. 18-May-2011), frealign v8.09_110505

dlyumkis Fri, 2011-09-02 22:09

Hi Matthias,

There are some differences in how we're doing the conversions of the data itself. I'm keeping the actual images and volumes unchanged when converting from Xmipp directly to Frealign format. In other words, there is no difference in the 2D / 3D array when you open up the stack / volume in any viewer, whether in Xmipp or any other format. Is there a reason why you have your volume in IMAGIC format as opposed to SPIDER (Xmipp)? My guess is that the reason for the different conversion scheme that you have has to do with the fact that either the image or volume array has physically changed.

After running some tests with Niko, I have to make a few corrections to the original conversion scheme that was posted - the previously posted conversions created a volume in Frealign that, despite being correct, was rotated about several axes when compared to the Xmipp output. To convert Xmipp to Frealign and generate identical volumes (while directly reading the Xmipp parameters in the doc file), the following needs to be done:

- the data itself for the stack and volume should remain unchanged.
- remove the mirroring operation in the Xmipp .doc file using the python script above, or something that provides equivalent angles. The angles really need to be the same due to the degeneracy of Euler rotations, otherwise the conversions below may not apply.
- convert according to the conventions below:

psi = -psi + 180
theta = tilt + 180
phi = rot
shx = -Xoff
shy = Yoff

... OR alternatively, the 2nd degenerate conversion, which should be identical to the first in terms of the final rotations and generated volume, is:

psi = -psi
theta = -tilt + 180
phi = rot + 180
shx = -Xoff
shy = Yoff

Let me know if this works for you.

Best,
Dmitry

dlyumkis Wed, 2014-01-01 14:55

wanted to add a correction to this old post, in case other people stumble upon this at some point. In fact, when I wrote the above post, my stack data had in fact been flipped between the Xmipp and Frealign formats (for example, this can be achieved by answering "yes" to the question "Consider different coordinate systems" in EM2EM). The correct final conversions are summarized below.

If the stack data is not physically altered, then the real conversion that we're now using is:

psi = psi
theta = tilt
phi = rot
shx = -Xoff
shy = -Yoff

If the stack data is physically flipped, then the conversion is as stated in the above post:

psi = -psi + 180
theta = tilt + 180
phi = rot
shx = -Xoff
shy = Yoff

... OR alternatively, the 2nd degenerate conversion:

psi = -psi
theta = -tilt + 180
phi = rot + 180
shx = -Xoff
shy = Yoff