The map does not refine
Forums
Hello,
Not sure this goes well in the forum as I don't have a specific question. The problem that I am encountering is that all maps I am getting with Frealing are extremely low-pass filtered versions of the map that I am using as reference. In the first instance I would like to see Frealing producing the same map that I am giving as a reference and then refine the model to higher resolution. The model was obtain by ML (Xmipp).
My data had already been CTF-corrected so the density of my data is white. Other than that, the dataset had not been filtered nor masked. The map also is white. I have tried changing different parameters with no impact on the output. I am guessing there is something wrong with my data, although the same data refines better than this with Xmipp or SPIDER.
I am pasting below the different parameter files.
maparameters
Parameter file for mrefine_sge.com data_input stkgg raw_images1 stkgg3_stack thresh_reconst 90.0 thresh_refine 10.0 pixel_size 1.95 dstep 11.7 WGH -1 CS 2.2 kV1 300.0 radius 100.0 PBC 100.0 BOFF 60.0 DANG 0.0 ITMAX 200 MODE 1 XSTD 0.0 RBFACT 0.0 FMAG F FDEF F FASTIG F FPART F dfsig 200.0 IEWALD 0 res_reconstruction 10.0 res_low_refinement 100.0 res_refinement 35.0 start_process 3 end_process 10 first_particle 1 last_particle 11935 increment 100
mrefine_n.com
time frealign_v8.exe & ${data_input}_mrefine_n.log_${1}_${2}
M,${mode},${fmag},${fdef},${fastig},${fpart},${iewald},F,F,F,0,F,4 !CFORM,IFLAG,FMAG,FDEF,FASTIG,FPART,IEWALD,FBEAUT,FCREF,FMATCH,IFSC,FSTAT,IBLOW
${radius},0.,${pixel_s},${w_gh},${xstd},${pbc},0.0,${dang},${itmax},10 !RO,RI,PSIZE,WGH,XSTD,PBC,BOFF,DANG,ITMAX,IPMAX
1 1 1 1 1 !MASK
${1},${2} !IFIRST,ILAST
0
1.0, ${dstep}, ${target}, ${thresh}, ${cs}, ${kV1}, 0., 0. !RELMAG,DSTEP,TARGET,THRESH,CS,AKV,TX,TY
${rrec},${rlowref},${rref},${dfsig},${rbfact} !RREC,RMAX1,RMAX2,DFSIG,RBFACT
${working_directory}/../${raw_images1}.mrc
${data_input}_match.mrc_${1}_${2}
${working_directory}/${data_input}_${prev}.par
${data_input}.par_${1}_${2}
${data_input}.shft_${1}_${2}
-100., 0., 0., 0., 0., 0., 0., 0. !terminator with RELMAG=-100.0 to skip 3D reconstruction
${working_directory}/${data_input}_${prev}.mrc
${data_input}_weights_${start}_${1}_${2}
${data_input}_map1_${1}_${2}
${data_input}_map2_${1}_${2}
${data_input}_phasediffs_${1}_${2}
${data_input}_pointspread_${1}_${2}
eot
mreconstruct.com
time frealign_v8.exe ${data_input}_mreconstruct.log
M,0,F,F,F,F,${iewald},T,T,F,0,F,4 !CFORM,IFLAG,FMAG,FDEF,FASTIG,FPART,IEWALD,FBEAUT,FCREF,FMATCH,IFSC,FSTAT,IBLOW
${radius},0.,${pixel_s},${w_gh},${xstd},${pbc},0.0,${dang},${itmax},10 !RO,RI,PSIZE,WGH,XSTD,PBC,BOFF,DANG,ITMAX,IPMAX
1 1 1 1 1 !MASK
${1},${2}, !IFIRST,ILAST
D9 !ASYM symmetry card
1., ${dstep}, ${target}, ${thresh}, ${cs}, ${kV1}, 0., 0. !RELMAG,DSTEP,TARGET,THRESH,CS,AKV,TX,TY
${rrec}, ${rlowref}, ${rref}, ${dfsig}, ${rbfact}, !RREC,RMAX1,RMAX2,DFSIG,RBFACT
${working_directory}/../${raw_images1}.mrc
/dev/null
${working_directory}/${data_input}_$start.par
${data_input}.res
${data_input}_dummy.shft
0., 0., 0., 0., 0., 0., 0., 0. !terminator with RELMAG=0.0
${data_input}_$start.mrc
${data_input}_weights_$start
${data_input}_map1
${data_input}_map2
${data_input}_phasediffs
${data_input}_pointspread
eot
#
Latest parameters file
C Date and time 22-05-2012, 08:41 Frealign V8.09 - 01.10.11
C Image format . . . . . . . . . . . . M
C Mode . . . . . . . . . . . . . . . . 1
C Magnification refinement . . . . . . F
C Defocus refinement . . . . . . . . . F
C Astigmatism refinement . . . . . . . F
C Defocus ref. of individual particles F
C Ewald sphere correction. . . . . . . 0
C Beautify the final real space map. . F
C Apply SNR filter final map . . . . . F
C Write out matching projections . . . F
C Calculate FSPR and FSC curves. . . . 0
C Calculate more statistics. . . . . . F
C Padding factor for reference volume. 4
C Outer Radius of object [Angstroms] . 100.00
C Inner Radius of object [Angstroms] . 0.00
C Pixel size [Angstroms] . . . . . . . 1.95000
C % Amplitude contrast . . . . . . . . -1.00
C STD level for 3D mask. . . . . . . . 0.00
C Phase res. / B factor constant . . . 100.00
C Average phase residual for weighting 0.00
C Symmetry card as input . . . . . . . C1
C Number of symmetry operators . . . . 1
C 1 1.0000 0.0000 0.0000
C 1 0.0000 1.0000 0.0000
C 1 0.0000 0.0000 1.0000
C First particle . . . . . . . . . . . 1
C Last particle. . . . . . . . . . . . 100
C Relative magnification . . . . . . . 1.0000
C Densitometer step size (microns) . . 11.7
C Phase residual target. . . . . . . . 10.00
C Phase residual threshold . . . . . . 90.00
C Cs [mm]. . . . . . . . . . . . . . . 2.20
C Voltage [kV] . . . . . . . . . . . . 300.00
C Beam tilt Tx, Ty [mrad]. . . . . . . 0.00 0.00
C Resolution of reconstruction . . . . 10.000
C Low resol. limit refinement. . . . . 100.000
C High resol. limit refinement . . . . 35.000
C Defocus uncertainty. . . . . . . . . 200.000
C B-factor for parameter refinement. . 0.000
C Input image stack /a/people/bebeacua/Data/AUT03/cryoLong/Frealign/mu
C Input parameter file /a/people/bebeacua/Data/AUT03/cryoLong/Frealign/mu
C Output parameter file stkgg.par_1_100
C Output shifts file stkgg.shft_1_100
C 3D reconstruction file /a/people/bebeacua/Data/AUT03/cryoLong/Frealign/mu
C 3D weights file stkgg_weights_10_1_100
C 3D reconstruction halfset 1 stkgg_map1_1_100
C 3D reconstruction halfset 2 stkgg_map2_1_100
C 3D ave phase residual file stkgg_phasediffs_1_100
C 3D point spread function stkgg_pointspread_1_100
C
C PSI THETA PHI SHX SHY MAG FILM DF1 DF2 ANGAST PRESA DPRES
1 294.06 2.70 59.37 -2.79 -0.73 60000. 1 0.0 0.0 0.00 47.51 -0.02
2 12.64 42.99 295.25 -0.11 1.08 60000. 1 0.0 0.0 0.00 44.88 0.03
3 10.69 22.84 89.29 -4.23 3.96 60000. 1 0.0 0.0 0.00 55.17 -0.01
4 227.99 141.01 152.02 3.65 -1.50 60000. 1 0.0 0.0 0.00 48.15 -0.04
5 280.00 3.21 4.78 -2.77 -0.85 60000. 1 0.0 0.0 0.00 47.16 -0.02
6 278.64 143.09 182.54 -0.29 -1.23 60000. 1 0.0 0.0 0.00 61.89 0.01
7 93.63 139.25 129.38 1.12 2.78 60000. 1 0.0 0.0 0.00 57.33 -0.01
8 183.86 69.15 69.07 1.45 -1.21 60000. 1 0.0 0.0 0.00 47.65 0.00
9 26.39 92.80 113.32 -1.88 -0.69 60000. 1 0.0 0.0 0.00 48.00 -0.01
10 66.14 99.44 146.71 0.52 1.72 60000. 1 0.0 0.0 0.00 45.39 0.07
11 10.71 118.17 359.49 1.37 0.21 60000. 1 0.0 0.0 0.00 59.32 -0.02
12 359.69 358.80 157.97 0.42 2.10 60000. 1 0.0 0.0 0.00 53.23 -0.03
13 358.81 52.62 311.56 -0.90 -1.17 60000. 1 0.0 0.0 0.00 47.75 -0.01
input stack should not be CTF-corrected
Hi Cecilia,
The first thing is that you need to give Frealign a stack which has not been CTF corrected, and where the protein density is darker than the background density. See for example the file examples/pdh_stack.mrc distributed with Frealign.
My guess is that because you have inverted contrast relative to what Frealign expects you are getting non-sensical alignments, leading to very low resolution reconstructions.
Hope this helps,
Alexis
images are CTF-corrected
In reply to input stack should not be CTF-corrected by Alexis
Hi Alexis!
I have seen in the Forum that people use Frealing for stacks that have been CTF-corrected. Niko had said answering to somebody's question that the protein had to be white if we asked Frealign not to do the CTF-correction. This is what Niko has said before:
"If you turn off CTF correction your protein density on input should be positive with respect to the background. With CTF correction switched on, it should be negative with respect to the background. You can use Spider to do the inversion. I always use pixel arithmetic to do this where I multiply by -1.0."
I read that in the entry called "Refining initial model using negative-stain data" which I guess is something similar to what I am trying to do.
I hope you are doing great!
Cecilia
Oops..
In reply to images are CTF-corrected by bebeacua
Sorry I missed your WGH -1.0!
How about trying just a reconstruction in Frealign (mode 0), to check that the angles & shifts you are using follow the Frealign convention? If the output from mode 0 looks good, then you know there's a problem during refinement. If it looks bad, I guess you could investigate how you converted the angles from xmipp to frealign.
HTH,
Alexis
Mode -3
In reply to Oops.. by Alexis
Hi Alexis,
Thank you for your reply.
I actually was not converting the alignment parameters from Xmipp/Spider but I would like to refine a model from scratch with Frealign. To do that I was using MODE -3 before the one that I posted before to obtain rough parameters for alignment and angles from the unligned dataset and create a parameters file from scratch. At this point the model obtained was already very low-pass filtered (same problem that I had when trying to refine it as I posted before).
Basically I was trying to follow the examples that came with the package where first a "search" is done and then a "refine". I wanted to do that with my data.
Below is the parameters file used at that point:
Hi Cecilia, I haven't used
In reply to Mode -3 by bebeacua
Hi Cecilia,
I haven't used mode -3 much, apart from using it with viruses (which it doesn't look like your particles are, judging from the outer radius), so I don't know how well it performs on asymmetric assemblies.
If you plot the phase residuals, do you find that a significant number of your particles are poorly aligned?
You can play with the low and high resolution refinement cutoff parameters, to see if you get better alignments.
You may also find that using the B-factor and wgh parameters as low and high pass filter parameters may significantly improve your initial alignments using mode -3.
In general, I use initial alignments generated by Imagic (I think your xmipp alignments would work fine as well) and simply using mode 1 to refine those alignment parameters. You may find it easiest to do this, and your result should not be overly affected, especially if you spend lower your high resolution refinement parameter to 20 or 30 Angstroms for a few rounds, then generate a new reconstruction.
My understanding of frealign is that you will also get much better performance when using its built-in CTF correction.
Finally, if your XMIPP map is at a high resolution (e.g. better than 8 Angstroms), then it may be normal to have a model that looks like a low-pass filtered version of it, using your current parameters. You may have to spend a few rounds refining the model.
Finally, it is also possible the XMIPP model has been over-refined, so your low-pass filtered model is more reflective of your data.
HTH,
Axel
Hi Axel, I have no plot the
In reply to Hi Cecilia, I haven't used by Axel
Hi Axel,
I have no plot the phase residuals. I can do that and check.
I really have played with all the parameters you mentioned: the low and high resolution curoffs and The B-factor. The wgh I have in -1 since I do not want to CTF-correct.
I can try to use my old alignments but I liked the idea to start from scratch with Frealign. It is in the examples and I would not understand why that would not be possible to do.
I understand that Frealign is thought for CTF-correction built-in, but I would like then to get the confirmation that Frealign definitely should not be run with images that had been CTF-corrected. In the site I always got the impression that it was possible that is why I kept trying.
My map is not a high resolution map (around 20) so no problem with over-fitting. When I say that Frealign is not refining I mean that is giving me a map that is much much worse than what I gave in and even after many cycles of refinement I can not get to the 20 A map that I started with. Something is definitely off but I do not know what and it has started to be frustrating.
Cecilia
Hi Cecilia, I have had some
In reply to Hi Axel, I have no plot the by bebeacua
Hi Cecilia,
I have had some experience with Xmipp --> Frealign refinements, so I thought I would chime in. The general approach that I take is to run an iterative projection-matching with Xmipp, then use those final Euler angles as input to Frealign. I don't use mode -3 for the rough search, and my (not vigorously tested) assumption is that, using an iterative projection-matching search, Xmipp should provide a better starting point than a rough search with Frealign. Several things to keep in mind:
1. Xmipp stores a mirroring operation for the Euler angles, which needs to be removed before proceeding with Frealign. To do this, you can use the following script (http://emlab.rose2.brandeis.edu/node/2590). After removing the mirroring operation, the conversions are specified in the thread, and you would need to write the actual loop yourself (although I think I have a few examples somewhere, which I can certainly send along if you decide to go this route).
2. The Xmipp projection-matching refinement protocol (if unmodified by the user) contains a step where all the raw particles are grouped into classes before the actual reconstruction. I found that this is detrimental when you have preferred orientation, as a lot of the views become underrepresented (even though technically, it should be taken into account by the weighting). For some particles, I found that it was better to remove that step and just use the Euler angles found by projection-matching.
Finally, for low-resolution reconstructions, you should probably apply a high B-factor to your map (e.g. -3000). That may be the reason why your map looks much "worse". What is the resolution of the Frealign map, by the way?
Dmitry
I will try to do that, that
In reply to Hi Cecilia, I have had some by dlyumkis
I will try to do that, that is take the alignment parameters from Xmipp or SPIDER.
But then, I still do not fully understand the concept. Is it not enough to give Frealign the data and the 3D and have it do the refinement? If Frealign does the refinement at later stages I do not understand how it cannot generate the initial parameters file without problems.
I will try what you suggested and post the results here.
I don't think that the
In reply to I will try to do that, that by bebeacua
I don't think that the concept is that Frealign cannot generate initial parameters without problems. In fact, it's perfectly capable of generating initial parameters, e.g. for icosahedral viruses, that will be enough to then refine and reach near-atomic resolution. It's just that with some particles (more so with small and asymmetric ones), the rough search that Frealign does to generate a ~30Å(???) map may not be good enough for the individual particles to lock in their angular orientations. This is probably more so the case when your particles are not centered. Therefore the general idea is that you're better off properly converting between alignment parameters that have been determined (probably in an iterative fashion) by a different package than starting with a brute force search in Frealign, although obviously this is data set dependent.
In your case, I'm not sure that the alignment itself is the reason for the poor results. Before doing any parameter conversions, I would also play around with several Frealign parameters:
- try setting resolution of the refinement to, e.g. between 200Å and 20Å or even 15Å for later cycles.
- your target phase residual seems pretty low, which means that you might be throwing away a lot of particles. Have you tried setting it to a higher value to make sure that all of the particles are being kept?
If you don't get improvement in the results, then I would start playing around with alignment parameters and make sure that everything is being converted properly between Xmipp and Frealign.
Good luck,
Dmitry
sorry, I missed that you're
In reply to I don't think that the by dlyumkis
sorry, I missed that you're setting the "target" phase residual to 10 as opposed to "threshold", so that shouldn't actually make a difference in the mode that you're working with.