frealix usage

Forums

Hi Alexis,

sorry if this is kind of a beginner's question, you've probably posted the directions on how to use frealix but for the life of me I can't figure out what I'm doing wrong.

Basically, I would like to try to refine frealign parameters (obtained for microtubules) using frealix. I already have pretty good estimates, but I'm wondering whether frealix may be able to refine the parameters further, it's worth a try I think!

I've tried using the ruby script you recommended (flx_wrap.rb), but I always get errors because I don't quite understand how to specify the input in a way frealix understands.

First of all, and very simply, I don't understand how I'm supposed to specify micrograph names, and I don't understand what the structure of the other inputs (i.e. ctffind input, is it supposed to be all ctf files or a list of the ctf-files or what? same goes with boxes). When I use wildcards (i.e. *) frealix complains with this message:

**ERROR(filename_wild_find_qmarks): cannot find question mark in filename my_micrographs/14oct28a_*en-a.mrc

Fatal error in FILENAME_WILD_FIND_QMARKS

but then when I try using question marks, frealix tries to assume a naming convention for my micrograph which doesn't necessarily correspond to how the micrographs are named, so it complains that the micrographs don't exist.

i.e. here is the top of my frealix file:

FILM_SELECTION 1 # just for testing
FILM_FILES my_micrographs/14oct28a_0????sq_v01_0????hl_0????en-a.mrc

CTFFIND_FILES ctf/14oct28a_0????sq_v01_0????hl_0????en-a_diag.txt
CTFTILT_FILES none
FILAMENT_TRACE_FILES boxes/14oct28a_0????sq_v01_0????hl_0????en-a.box

TWIST_PER_SUBUNIT -27.688 #0.769 #degrees
RISE_PER_SUBUNIT 9.52 #4.7 #in Angstroms
SYMMETRY_AXIAL 1
SYMMETRY_PERP 1
FAL_WPS_IN_SCORE T # Whether the first and last segments of the filament should be included in the scoring function

and then the output:

**ERROR(mode_filament_startup): file does not exist: my_micrographs/14oct28a_00001sq_v01_0????hl_0????en-a.mrc

Fatal error in MODE_FILAMENT_STARTUP

==========================
am i supposed to follow some sort of naming convention for my input files?

is there a better way to convert frealign parameters into frealix input? sorry if i missed this information somewhere else.

thanks,

Liz

Hi Liz,

Thanks for your interest in Frealix. Indeed in at least some cases, Frealix can refine Frealign alignment parameters for filaments and improve the final resolution, so definitely worth a try.

Your goal initially will be to create a first waypoints.ali file, which is where Frealix stores alignment parameters for each waypoint of each filament. As things stand, the conversion from Frealign to Frealix is not handled by flx_wrap.rb, unfortunately (it'll only do startup from eman boxer or IMOD coordinates for now).

Instead, I'd recommend either you write an awk-based script to do things manually, or you use frealix mode frealign_stack_to_frealix_ali. In Frealix 1.1, this should handle conversion from frealign v8 correctly, though it will take a long time to run. If you used Frealign v9, I will need to send you my development version of Frealix 1.2. Let me know.

To use this mode, first create a text file (conversion.txt, say) with the following content edited appropriately:

frealix_mode 			frealign_stack_to_frealix_ali
frealign_par_filename		/path/to/frealign.par			# Frealign parameter file.
stack_filename			/path/to/stack/of/binned/particles.mrc	# Stack of particles that were processed by frealign
film_filenames			/path/to/your/micrographs/mic_???.mrc	# The question marks represent [0-9] digits and indicate the micrograph number.
film_selection			1-100,103-120,124,126			# Which film numbers to look for. Comma-separated list, where - indicates contiguous range.
list_of_film_numbers		film_number_for_each_particle.txt	# A text file with number of lines = number of particles in input stack. On each line, give the number of the micrograph from which the particle was extracted.
ali_filename			waypoints_from_frealign.ali		# Filename for output waypoints file
pixel_size			5.43					# In Angstroms, the pixel size of the stack of particle images and of the micrographs
check_180_rotation		f					# Set this to t if some of your particle images were rotated by 180 degrees

You will then run the following command to run the conversion:

/path/to/frealix conversion.txt

The issue when converting from frealign to frealix is that the frealign parameter file does not keep track of which micrograph each particle was extracted from nor what the particle coordinates are within that micrograph. To recover this information, mode frealign_stack_to_frealix_ali reads in each particle image at a time and literally looks for it in the micrographs (doing something like automatic particle picking). If you give it the "list_of_film_numbers" parameter described above, you save it a lot of time, because you're telling it in which micrograph it should expect to find each particle. Unfortunately, this feature is not available in Frealix 1.1. If this information is not available (or if you're running Frealix 1.1), it will look for each particle in each micrograph which can take a long time if you're dealing with a large dataset.

Regardless of which version of Frealix you use, you should run this with a binned / resampled stack of particles and micrographs, with a pixel size of, say, 5 to 10 Angstroms. (Slight errors in X,Y do not matter much anyway, and can usually be recovered in the first round of refinement).

Once it has found each particle and it now knows micrograph it belongs to and its coordinates, it writes out a waypoints.ali file which you can use as a starting point for frealix refinement.

You would then use flx_wrap.rb to start a Frealix refinement, by pointing it to this waypoints.ali file and to your existing reconstruction (or a low-pass version thereof).

Regarding naming conventions: the waypoints.ali file specifies on which micrograph each filament is located. When Frealix needs to access the image of that filament, it will look for the micrograph image by taking the filename pattern you supply, and substituting the micrograph id number into the part of the pattern that contains question marks. If you supplied the filename pattern given in the above example, and it's looking for a filament on micrograph 12, it would expect to find a file named /path/to/your/micrographs/mic_012.mrc. Hope this makes sense.

In the example you supplied (my_micrographs/14oct28a_0????sq_v01_0????hl_0????en-a.mrc), Frealix substituted the micrograph number into the first stretch of qmarks it found and left the other qmarks alone. So the solution in your case, it to simplify the naming scheme of your micrographs. Perhaps you could create a new directory with symbolic links to your micrographs, and the symbolic links could have simpler names with just one variable part containing the micrograph id.

If you'd like to try the current (development) version of Frealix 1.2, I'm happy to share that with you, just let me know.

I should point out I've not played with microtubules for a few years and while in theory Frealix handles microtubule geometries well (even doing the real-space reconstruction trick a la Sindelar), this hasn't been tested or used in a long, long time. In any case, I'm happy to answer questions and update the code as required.

Hope this helps,
Alexis

Thanks Alexis for the quick response! your explanation clears up a lot of confusion.

i don't mind doing some work to get all the relevant information together in a waypoints.ali file required by frealix.
I can easily map the particle image coordinates to the frealign input parameters. I just need to know the format of the waypoints file frealix expects. can you possibly share an example waypoints file with header information? I think if I can see that then I can assemble a proper input file (hopefully!) if it gets to be too much of a hassle I'll try your frealix conversion script.

thanks a bunch! I'm sure you'll be hearing more from me as I encounter more issues.

Liz

In reply to by lizkellogg

Sounds like a good plan.

Here's an example of a waypoints file (the first few lines of):

!Fil.#   WP # Film #           X           Y           Z  PathLength          Phi        Theta          Psi   Subunit #    Score Refine? Units Angstr   par_vers. 4
     1      1      2    1167.430    3762.582   -9288.093       0.000     -282.017       90.041      -93.431       1.000    0.204       T
     1      2      2    1170.901    3705.066   -9286.175      57.658      191.558       89.498      -93.440       5.000    0.224       T
     1      3      2    1173.516    3661.617   -9285.466     101.191      546.425       89.690      -93.406       8.000    0.229       T
     2      1      2    1150.923    1906.029   -9272.312       0.000      194.269      112.023      271.864       1.000    0.104       T
     2      2      2    1151.174    1854.291   -9285.710      53.626      675.366       95.953      271.609       5.000    0.143       T
     2      3      2    1142.836    1788.397   -9285.988     120.278     1239.178       82.337      280.744      10.000    0.154       T
     3      1      5    4228.687    2114.584  -13879.171       0.000     -259.251       86.739     -334.067       1.000    0.260       T
     3      2      5    4189.330    2133.726  -13876.073      43.876       96.574       86.694     -333.863       4.000    0.260       T
     3      3      5    4149.983    2152.957  -13872.983      87.779      451.186       86.379     -333.866       7.000    0.241       T

You'll need to make sure the header line is copied exactly.
You should leave the Z, PathLength, Subunit # and Score columns filled with 0.0.
The X and Y columns are Angstroem measurements from the first pixel of the micrograph. In other words, if a waypoint is exactly on the first pixel, it will have coordinate 0.0, 0.0 in the waypoints file. To be sure you know which is the first pixel, I use IMOD's display program as a reference. If you click on a point, it shows you X,Y coordinates of that pixel.

Let me know if you need more info
Alexis

In reply to by Alexis

Just a few more clarifying questions...

I assume the first column is the filament number (unique for each independent filament), the WP# corresponds to the particles within each filament (which would be my particles I have boxed out of a microtubule), and the film number is just the micrograph the particle came from. Phi, Theta, and Psi are straightforward and I assume they correspond directly to frealign convention.

I'm a little confused because the X and Y look like box-file coordinates, but then there's no x,y shifts. does that mean the X and Y in the waypoint file correspond to the center of the particle? (which would be the x-origin + x-shift?)
where x-origin is the coordinates of the boxes, and the x-shift is the shift required to align the particle to a reference?

thanks for your help

Liz

In reply to by lizkellogg

Hi Liz,

You've got it - X and Y are the non-integer coordinates of the center of each of your microtubule segments, and the other parameters are straightforward.

Cheers
Alexis

In reply to by lizkellogg

ok, so now i have a waypoints file, but i can't get frealix to run refinement

I tried your suggestion from a previous post (maybe I'm misunderstanding it) to execute the wrapper script: flx_wrap.rb with the waypoints.ali specified under the keyword INPUT_PAR_FILENAME

i.e.

head frealix_parameters.flx
FILM_SELECTION 1-123
INPUT_PAR_FILENAME waypoints.ali
FILM_FILES my_micrographs/????.mrc

CTFFIND_FILES ctf/????.txt
CTFTILT_FILES none

TWIST_PER_SUBUNIT -27.688 #0.769 #degrees
RISE_PER_SUBUNIT 9.52 #4.7 #in Angstroms
SYMMETRY_AXIAL 1

where my command is:

flx_wrap.rb -r 1 -s 6 -t 4 -n 1 frealix_parameters.flx

but now frealix complains:

**ERROR(frealix_par_to_filaments_noinit): last way point appears to be alone

Fatal error in FREALIX_PAR_TO_FILAMENTS_NOINIT

**info(terminate): Frealix is terminating in error state, see error message(s) above for details.
**info(terminate): will now attempt to generate a back trace by trigerring a floating point exception

2015-12-11 21:20:33: frealix says sorry (total execution time: 0 seconds)

Runtime Error: *** Arithmetic exception: Floating invalid operation - aborting

am i using frealix refinement correctly?

In reply to by lizkellogg

Hi Liz,

Sorry the error message isn't clearer. The problem is that Frealix 1.1 does not allow single-waypoint filaments. In Frealix 1.2 (I'm going to try to release this this week), single-waypoint filaments are allowed as long as one uses frealix in single-particle mode (i.e. almost no restraints on alignment).

So, for now, you have two options: remove from the waypoints file those filaments that have only one waypoint; or wait a little for frealix 1.2.

I think it would make sense for you to try a small subset of your data with frealix 1.1. That way, you can get used to Frealix a little and check it's not doing anything crazy with your microtubules. For example, select only 25 filaments, try to get frealix to compute a reconstruction (without refinement at first) and compare that to the reconstruction you get with frealign. This would provide a useful check on your conversion procedure etc.

HTH
Alexis