failed sanity check in fmc_maximise

Submitted by matthias on Mon, 2015-03-30 04:51

Forums

Frealix

Hi.

Frealix sometimes exits with a message like:

failed sanity check in fmc_maximise

Image PC Routine Line Source
frealix 00000000004EDBBA runtime_parameter 205 runtime_parameters.f90
frealix 0000000000B5381B fmcs_MP_fmc_maxim 298 fmcs.f90
frealix 0000000000DFA00F mode_filament_ref 1937 mode_filament_refine.f90
frealix 0000000000584597 modes_MP_mode_run 63 modes.f90
frealix 0000000000D94225 mode_filament_ref 1007 mode_filament_refine.f90
frealix 0000000000F68504 frealix_IP_main_ 167 frealix.f90
frealix 0000000000F66662 MAIN__ 21 frealix.f90
frealix 00000000004249D6 Unknown Unknown Unknown
libc.so.6 00000030A5C1ED5D Unknown Unknown Unknown
frealix 00000000004248C9 Unknown Unknown Unknown

forrtl: warning (528): IEEE_UNDERFLOW is signaling

The error is not really reproducible but appears at different stages in the run even if I start two runs with the same configuration and input data.

I started frealix via:
flx_wrap.rb -r 3 -f 20 -s 20 -t 20 -n 16 --threads 1 parameter_refine.flx

Versions:
Intel Compiler 14.0
MKL 11.1.4
GSL 1.16
Frealix: 1.1.0
System: RHEL 6.6

The error occurs with various compiler flags also with the default flags.

All tests passed except "apply CTF test (Frealix v EMAN2)" and "synthetic filament test (segments v repeats)" but in another forum post it says that is not important.

Please tell me if you need any further information.

Thanks
Matthias

Alexis Mon, 2015-03-30 08:46

Hello Matthias,

Sorry to hear this is happening in your projects. I also see this happen occasionally in my projects and I hope future releases of Frealix will be more robust in that respect.

Before the termination and the final error message ("failed sanity check"), Frealix should print out a message like "**error(fmc_maximise): on start X of Y, score at start > score at finish: XXXX YYYY". In many cases, the two scores are quite close to each other, is this the case for you also?

In any case, it should be safe to turn off this sanity check. To do this, you'll need to comment out lines 288 to 300 (inclusive) of file src/fmcs.f90 (the Fortran way to comment is to add an exclamation mark at the beginning of the line). Then you will need to rebuild Frealix.

I may turn this sanity check off altogether in future releases.

Alexis

matthias Tue, 2015-03-31 16:33

Hi Alexis,

yes, frealix does print such a message but the values are not that close I think (depends on what is "close"):
**error(fmc_maximise): on start 1 of 1, score at start > score at finish: -2.3291 -3.2152

But in a different run I got
**error(fmc_maximise): score on exit less than on entry: -.0034532 .0075683

I commented out the lines you said but the program still crashed with the message
**error(fmc_maximise): best final score less than starting score: .03032 .11571

So I started to change a little bit more:
In fmcs.f90 I set the variable sanity_checks to false and I commented out a few if cases where the return value of the function score_got_worse is checked.

But is it save to remove all checks? Can it happen that the algorithm does not converge or that it produces wrong results?

I have done two test runs and both of them were fine.

Thanks a lot
Matthias

Alexis Wed, 2015-04-01 09:14

Hi Matthias,

The short answer is yes, it is safe to remove these checks. I'll try to explain why some of these "sanity checks" fail sometimes.

**error(fmc_maximise): on start 1 of 1, score at start > score at finish: -2.3291 -3.2152

Scores are normalised to -1.0 - 1.0. The fact that you are getting scores lower than -1.0 suggests to me that you might have curvature and/or other restraints activated, because these restraints can impose a penalty on the score, making it less than -1.0. Assuming this is the case, it could be that

  • this particular filament is not being traced well by Frealix and/or that it's actually very deformed. So it gets a terrible score. What happens then is that the splines that trace the filament are hard to solve accurately enough (specifically, the arc-length parametrisation can fail) and so on two different tries, the algorithm doesn't get exactly the same spline solutions, therefore not exactly the same curvature estimate and not exactly the same curvature penalty.
  • Another possibility is that the scoring function returns -1.0 before penalties are applied. This can happen when the filament goes over the edge of a micrograph, or when some of the splines that are supposed to be monotonic (the phi and subunit_num splines) are not monotonic. When this happens, the scoring function outputs a warning like this: **WARNING(fmc_eval_slave): non-monotonic phi or subunit_num splines. Will return -1.0. If the latter is the case (non-monotonic splines), it can also cause the penalty computation to be unreliable, for reasons similar to those above. This can be solved by working out which waypoint is causing the splines to not be monotonic and editing the waypoints file manually to fix that point. Filaments can be rescued that way. 
**error(fmc_maximise): score on exit less than on entry: -.0034532 .0075683

Both of these scores are pretty terrible anyway. This is definitely safe to ignore, though I wouldn't be surprised if that filament went bad somehow.

**error(fmc_maximise): best final score less than starting score: .03032 .11571

I don't know why this happened. Could you please re-run Frealix with the same inputs exactly, with a re-compiled version of Frealix with debug = .true. on line 953 of src/fmcs.f90? If the error is reproduced, could you please upload the Frealix output to this forum (you may have to create a new forum thread to be able to upload)?

Thanks for your patience & feedback!

Alexis