use ctffind4 in relion 1.3

Hi, I am trying to use ctffind4 in relion1.3 by adding parameters in GUI. I tried both ctffind-4.0.12-linux64-compat.tar.gz and ctffind-4.0.12-linux64.tar.gz.
I got some errors as below.
Any suggestion will be appreciated!
Thanks,
Lei

Executing: mpirun -n 12 `which relion_run_ctffind_mpi` --i "selected_micrograph.star" --o "selected_micrographs_ctf4.star" --ctfWin -1 --CS 2.7 --HT 300 --AmpCnst 0.1 --XMAG 30488 --DStep 5 --Box 512 --ResMin 100 --ResMax 7 --dFMin 5000 --dFMax 50000 --FStep 500 --dAst 0 --ctffind3_exe "/soft/ctffind4-compat/ctffind --omp-num-threads 1 --old-school-input" &
+ Slave 3 runs on host = medusa.ohsu.edu
+ Slave 4 runs on host = medusa.ohsu.edu
+ Slave 5 runs on host = medusa.ohsu.edu
+ Slave 6 runs on host = medusa.ohsu.edu
+ Slave 7 runs on host = medusa.ohsu.edu
+ Slave 8 runs on host = medusa.ohsu.edu
+ Slave 9 runs on host = medusa.ohsu.edu
=== RELION MPI setup ===
+ Number of MPI processes = 12
+ Master (0) runs on host = medusa.ohsu.edu
+ Slave 1 runs on host = medusa.ohsu.edu
+ Slave 2 runs on host = medusa.ohsu.edu
=================
+ Slave 10 runs on host = medusa.ohsu.edu
+ Slave 11 runs on host = medusa.ohsu.edu
Using CTFFINDs executable in: /t1/lei/soft/ctffind4-compat/ctffind --omp-num-threads 1 --old-school-input
to estimate CTF parameters for the following micrographs:
* images/Feb23_14.33.37.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb23_14.48.40.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb23_14.54.29.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb23_23.16.29.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb23_23.19.17.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb24_01.18.32.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb24_02.39.20.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb24_05.34.00.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb24_06.08.05.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb24_08.01.12.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb24_08.51.55.tif_gaincorr_bin2_SumCorr.mrc
* images/Feb24_09.28.31.tif_gaincorr_bin2_SumCorr.mrc
Estimating CTF parameters using Niko Grigorieff's CTFFIND3 ...
000/??? sec ~~(,_,"> [oo]OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
forrtl: No such file or directory
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
forrtl: No such file or directory
forrtl: No such file or directory
forrtl: No such file or directory
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
forrtl: No such file or directory
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
forrtl: No such file or directory
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
forrtl: No such file or directory
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
forrtl: No such file or directory
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
forrtl: No such file or directory
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
forrtl: No such file or directory
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
0/ 0 sec ............................................................~~(,_,">
+++ RELION: command line arguments (with defaults for optional ones between parantheses) +++
====== General options =====
====== CTF estimation =====
--i : STAR file with all input micrographs, or a unix wildcard to all micrograph files, e.g. "mics/*.mrc"
--o (micrographs_ctf.star) : Name for the STAR file with CTF params for each micrograph
--only_make_star (false) : Don't run CTFFIND3, only join the logfile results in a STAR file
--only_do_unfinished (false) : Only run CTFFIND3 for those micrographs for which there is not yet a logfile with Final values.
--ctfWin (-1) : Size (in pixels) of a centered, squared window to use for CTF-estimation
--ctffind3_exe () : Location of ctffind3 executable (or through RLN_CTFFIND3_EXECUTABLE environment variable)
--CS (2.0) : Spherical Aberration (mm)
--HT (300) : Voltage (kV)
--AmpCnst (0.1) : Amplitude constrast
--XMAG (60000) : Magnification
--DStep (14) : Detector pixel size (um)
--Box (512) : Size of the boxes to calculate FFTs
--ResMin (100) : Minimum resolution (in A) to include in calculations
--ResMax (7) : Maximum resolution (in A) to include in calculations
--dFMin (10000) : Minimum defocus value (in A) to search
--dFMax (50000) : Maximum defocus value (in A) to search
--FStep (250) : defocus step size (in A) for search
--dAst (0) : amount of astigmatism (in A)
ERROR: cannot find line with Final values in images/Feb23_14.33.37.tif_gaincorr_bin2_SumCorr_ctffind3.log
File: src/ctffind_runner.cpp line: 299
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[65165,1],0]
Exit code: 1
--------------------------------------------------------------------------

Hi there,

I can't tell from this what happened. Could you:

  • post the contents of images/Feb23_14.33.37.tif_gaincorr_bin2_SumCorr_ctffind3.log
  • try to reproduce this with just one micrograph, without mpi and ideally in a manner that let's us see exactly what was input into ctffind

The first error message ("forrtl: No such file or directory") in the output you show above suggests that perhaps the filenames are wrong... so perhaps also double-check that the filenames are exactly correct, that they are accessible from the nodes on which ctffind is being run etc.

Alexis

In reply to by Alexis

the log file looks like:

** Welcome to CTFFind **
Version - 4.0.12

Input Mode: Batch
Date & Time: 2015-02-25 19:18:11

Copyright 2015 Howard Hughes Medical Institute. All rights reserved.
Use is subject to Janelia Farm Research Campus Software Copyright 1.1
license terms ( http://license.janelia.org/license/jfrc_copyright_1_1.html )

Overriding minimum resolution for fitting to 50.0 A

CS[mm], HT[kV], AmpCnst, XMAG, DStep[um]
2.7 300.0 0.10 30488.0 5.000

The .com file looks like:

#!/usr/bin/env csh
/t1/lei/soft/ctffind4-compat/ctffind --omp-num-threads 1 --old-school-input > images/Feb23_14.48.40.tif_gaincorr_bin2_SumCorr_ctffind3.log << EOF
images/Feb23_14.48.40.tif_gaincorr_bin2_SumCorr.mrc
images/Feb23_14.48.40.tif_gaincorr_bin2_SumCorr.ctf
2.7, 300, 0.1, 30488, 5
512, 100, 7, 5000, 50000, 500, 0
EOF

and If I just execute the .com file, the ctffind can work fine. It just can't be execute from relion GUI.
I don't know why.

In reply to by rainfieldcn

I'm afraid this is beginning to sound more like a relion / system-configuration issue than a ctffind issue, and I am not familiar enough with how relion calls ctffind to help very much, but... What happens if you get Relion to use ctffind3 instead of ctffind4? Do you still get the problem?

In reply to by rainfieldcn

Do you mean that if you run exactly the same mpirun command as in your original post, but with the ctffind command substituted, ctffind3 runs fine 12x in parallel?

In reply to by Alexis

Hi Alexis,
I met Richard, he told me to set the cpu numbers in the parameters in relion GUI, instead of in relion MPI numbers.
It seems working now!
Thanks,
Lei

In reply to by rainfieldcn

Hi Rainfieldcn:

I got similar errors as you did previously. Could you please give a detailed command you used to solve this problem?

Many thanks,

Dee

In reply to by dee123

Dee,

I'm not sure exactly what rainfieldcn's problem & solutions were, but it may have involved the command-line option "--omp-num-threads N" where N is the number of threads you want ctffind to run.

If this doesn't solve your problem, feel free to start another thread on this forum to describe the symptoms.

Cheers,
Alexis

In reply to by Alexis

Hello Alexis,

I came up with the similar error using ctffind-4.0.13. At the same time everything works fine in ctffind-4.0.7.

*** The command is:
`which relion_run_ctffind_mpi` --i "micrographs.star" --o "micrographs_ctf.star" --ctfWin -1 --CS 0.01 --HT 300 --AmpCnst 0.1 --XMAG 99291 --DStep 14 --Box 512 --ResMin 30 --ResMax 7 --dFMin 5000 --dFMax 50000 --FStep 500 --dAst 100 --ctffind3_exe "/soft/ctffind-4.0.7/ctffind --omp-num-threads 1 --old-school-input"
Executing: sbatch -o ctf407.log run_ctffind_submit.script &

Using CTFFINDs executable in: /soft/ctffind-4.0.7/ctffind --omp-num-threads 1 --old-school-input
to estimate CTF parameters for the following micrographs:
* FoilHole_6295680_Data_6289448_6289449_20150108_0438.mrc
* FoilHole_6295680_Data_6290638_6290639_20150108_0439.mrc
* FoilHole_6295680_Data_6290644_6290645_20150108_0436.mrc
* FoilHole_6295680_Data_6290646_6290647_20150108_0437.mrc
Estimating CTF parameters using Niko Grigorieff's CTFFIND3 ...
29/ 29 sec ............................................................~~(,_,">

In this case output looks OK.

But when I launch same job with 4.0.7 it crashes immediately:

*** The command is:
`which relion_run_ctffind_mpi` --i "micrographs.star" --o "micrographs_ctf4013.star" --ctfWin -1 --CS 0.01 --HT 300 --AmpCnst 0.1 --XMAG 99291 --DStep 14 --Box 512 --ResMin 30 --ResMax 7 --dFMin 5000 --dFMax 50000 --FStep 500 --dAst 100 --ctffind3_exe "/soft/ctffind-4.0.13/ctffind --omp-num-threads 1 --old-school-input"
Executing: sbatch -o ctf4013.log run_ctffind_submit.script &
Using CTFFINDs executable in: /soft/ctffind-4.0.13/ctffind --omp-num-threads 1 --old-school-input
to estimate CTF parameters for the following micrographs:
* FoilHole_6295680_Data_6289448_6289449_20150108_0438.mrc
* FoilHole_6295680_Data_6290638_6290639_20150108_0439.mrc
* FoilHole_6295680_Data_6290644_6290645_20150108_0436.mrc
* FoilHole_6295680_Data_6290646_6290647_20150108_0437.mrc
Estimating CTF parameters using Niko Grigorieff's CTFFIND3 ...
000/??? sec ~~(,_,"> [oo]forrtl: No such file or directory
forrtl: No such file or directory
forrtl: No such file or directory
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
Image PC Routine Line Source
ctffind 000000000156F4FF Unknown Unknown Unknown
ctffind 00000000017CA31F Unknown Unknown Unknown
ctffind 0000000001746992 MAIN__ 372 ctffind.f90
ctffind 0000000000400566 Unknown Unknown Unknown
ctffind 00000000016C1DFB Unknown Unknown Unknown
ctffind 0000000000400429 Unknown Unknown Unknown
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
Image PC Routine Line Source
ctffind 000000000156F4FF Unknown Unknown Unknown
ctffind 00000000017CA31F Unknown Unknown Unknown
ctffind 0000000001746992 MAIN__ 372 ctffind.f90
ctffind 0000000000400566 Unknown Unknown Unknown
ctffind 00000000016C1DFB Unknown Unknown Unknown
ctffind 0000000000400429 Unknown Unknown Unknown
forrtl: severe (28): CLOSE error, unit 20, file "Unknown"
Image PC Routine Line Source
ctffind 000000000156F4FF Unknown Unknown Unknown
ctffind 00000000017CA31F Unknown Unknown Unknown
ctffind 0000000001746992 MAIN__ 372 ctffind.f90
ctffind 0000000000400566 Unknown Unknown Unknown
ctffind 00000000016C1DFB Unknown Unknown Unknown
ctffind 0000000000400429 Unknown Unknown Unknown
^M 0/ 0 sec ............................................................~~(,_,">

This would mean it cannot be relion-related. At the same time both versions run from command line without any problems (/home/igbmc/gsharov/soft/ctffind-4.0.X/ctffind --omp-num-threads 1)

In reply to by Grigory

Grigory,

The error you are getting is:

forrtl: No such file or directory

This suggests to me that the problem might be that when/where you are running ctffind, the location of the input micrograph or of the output diagnostic images/text are not available to the process.
In addition, the fact that everything runs fine when run from the command line is strongly suggestive of some system setup issue.

I would begin by double-checking that all disks are correctly cross-mounted on all the hosts which will be running ctffind.

Alexis

In reply to by Alexis

Hi Alexis,

I experience the exact same issue with Relion and ctffind4 as rainfieldcn described, but I think I found the source of the problem.
In interactive mode ctffind4 writes two defaults files (.CTFFind.dff, .currentCTFFind.dff). When Relion tries to run several ctffind4 instances in parallel, multiple instances try to write to the same file, which probably leads to these crashes with the error message: severe (28): CLOSE error, unit 20, file "Unknown"

Is there a way to switch of writing the defaults files with a command line option or similar?

Best,
Andreas

In reply to by Andreas

Hi Andreas,

Many thanks for figuring this out! Your explanation makes sense.

I will look into this more next week, but I'm afraid until then the only workarounds would be to process each micrograph in its own directory or to ensure that no two instances of ctffind are started simultaneously. I believe as long as the no two instances have the user interaction block running concurrently, this problem should not occur.

Hopefully I will have a bugfix release ready at some point next week. I'm not sure yet what the solution will be, so it's hard to be sure when I'll get it done.

Thanks again,
Alexis

In reply to by Alexis

Hello all,

I have uploaded a new version, 4.0.14. Unfortunately I do not have an easy way to reproduce the error you are seeing, so I'd appreciate if you could test this new version and let me know the outcome.

All the best,
Alexis

In reply to by Alexis

Dear all,

Version 4.0.15 has been verified by one user to fix this issue.

Alexis

In reply to by Alexis

Hi Alexis,

thank you for the updated version. I just tried it on my machine and the issue is fixed for me as well.

Best,
Andreas