ctffind plot results script
Forums
Hello,
I tried using the script on my last ctf files generated with ctffind4 within relion, but I get the following error message and no file is written out.
We have on our cluster gnuplot and pdftk running. Am I doing something wrong or am I just missing something.
Cheers
Marcus
sh ../../ctffind/ctffind-4.0.7-linux64/ctffind_plot_results.sh align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
Error: Unable to find file.
Error: Failed to open PDF file:
/tmp/tmp.pdf
Errors encountered. No output created.
Done. Input errors, so no output created.
Generated plots of 1D profiles and fits: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf
Hi Marcus, Could you add "-x"
Hi Marcus,
Could you add "-x" to the first line of the script, and re-run it? This will help pinpoint where the problem is occuring.
I.e. the first line would now read
#!/bin/bash -xThis will put bash in "verbose" mode, so we should be able to know where the error occurs.
Could you also check whether the node where you are running this has a /tmp partition, and whether you have permission to write there?
Cheers,
Alexis
Dear Alexis, yes, there is a
In reply to Hi Marcus, Could you add "-x" by Alexis
Dear Alexis,
yes, there is a /tmp partition existing and yes, I have writing permission.
Adding a -x (#!/bin/bash -x) did not yield to a different output from the command line than I posted earlier.
However I realized, that I created at least in the /tmp folder three files
1)tmp.txt
# Output from CTFFind version 4.0.7, run on 2014-11-21 10:33:44
# Input file: mics/align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames.mrc ; Number of micrographs: 1
# Pixel size: 1.260; acceleration voltage: 300.0; spherical aberration: 2.3; amplitude contrast: .10
# Box size: 128; min res: 50.0; max. res.: 8.0; min. def.: 5000.0; max. def.: 60000.0
# 6 lines per micrograph: #1 - spatial frequency (1/pixels); #2 - 1D rotational average of spectrum (assuming no astigmatism); #3 - 1D rotational average of spectrum; #4 - CTF fit; #5 - cross-correlation between spectrum and CTF fit; #6 - 2sigma of expected cross correla
tion of noise
0.000000 0.000000 0.3374503 0.1000000E-01 1.000000 0.000000
etc..
2) and empty file called tmp.pdf
and 3) pdftemp.txt
InfoKey: Title
InfoValue: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
gnuplot problem?
In reply to Dear Alexis, yes, there is a by Marcus
Perhaps gnuplot didn't output anything, possibly because you're missing the pdfcairo terminal or have an older gnuplot than I use, or something along those lines.
To debug further, please change the gnuplot line in the script from
gnuplot > /dev/null 2>&1 <
to
gnuplot <That way, when you run the script you should see the output from gnuplot, which may be throwing an error.
Let me know
Alexis
PS. with -x, bash should output lots more. Perhaps if you called the script like this:
../../ctffind/ctffind-4.0.7-linux64/ctffind_plot_results.shyou might get the output I expected. But we may not need this if gnuplot is the source of the problem.
Probably really a gnuplot problem
In reply to gnuplot problem? by Alexis
Hi Alexis,
thanks already for all the help.
We run gnuplot 4.4 patchlevel 2
In case it is a problem with the gnuplot version, is there something that we can change on the code to make it compatible?
This is the current terminal output:
sh ../../ctffind/ctffind-4.0.7-linux64/ctffind_plot_results.sh align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
gnuplot> set style line 2 lc rgb 'red' lt 1 lw 1 pt 7 fill transparent solid 0.5 # --- red
^
line 0: Extraneous arguments to set style line
gnuplot> do for [current_micrograph=1:1] {
^
line 0: invalid complex constant
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
gnuplot> }
^
line 0: invalid character }
Error: Unable to find file.
Error: Failed to open PDF file:
/tmp/tmp.pdf
Errors encountered. No output created.
Done. Input errors, so no output created.
If I run the program using:
../../ctffind/ctffind-4.0.7-linux64/ctffind_plot_results.sh align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
i get f+ '[' 1 -ne 1 ']'
+ input_fn=align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ output_fn=align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf
++ ls align_14nov03a_tXXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames.txt
+ input_summary_fn=align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames.txt
+ hash gnuplot
+ transpose_ignore_comments align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ gawk '
{
if ($1 ~ /^#/) {
print $0
} else {
for (i=1; i<=NF; i++) {
a[NR,i] = $i
}
}
}
NF>p { p = NF }
END {
for(j=1; j<=p; j++) {
str=a[1,j]
for(i=2; i<=NR; i++){
str=str" "a[i,j];
}
print str
}
}' align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
++ gawk 'match($0,/Pixel size: ([0-9.]*)/,a) {print a[1]}' align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ pixel_size=1.260
++ gawk 'match($0,/Number of micrographs: ([0-9]*)/,a) {print a[1]}' align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ number_of_micrographs=1
+ lines_per_micrograph=6
+ i=0
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ 1.000000 != \#* ]]
+ df_one[++i]=38160.07
+ df_two[i]=36721.14
+ angast[i]=0.9210151
+ score[i]=0.1782483
+ maxres='[i]=16.12781'
+ read -a myArray
+ echo
+ gnuplot
gnuplot> set style line 2 lc rgb 'red' lt 1 lw 1 pt 7 fill transparent solid 0.5 # --- red
^
line 0: Extraneous arguments to set style line
gnuplot> do for [current_micrograph=1:1] {
^
line 0: invalid complex constant
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
gnuplot> }
^
line 0: invalid character }
+ hash pdftk
+ echo 'InfoKey: Title'
+ echo 'InfoValue: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt'
+ mv align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf /tmp/tmp.pdf
+ pdftk /tmp/tmp.pdf update_info /tmp/pdftemp.txt output align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf
Error: Unable to find file.
Error: Failed to open PDF file:
/tmp/tmp.pdf
Errors encountered. No output created.
Done. Input errors, so no output created.
+ echo ' '
+ echo 'Generated plots of 1D profiles and fits: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf'
Generated plots of 1D profiles and fits: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf
+ echo ' 'ollowing longer output
script requires gnuplot >= 4.6
In reply to Probably really a gnuplot problem by Marcus
Hi Marcus,
The script requires gnuplot 4.6. I will try to amend it now so that it detects the gnuplot version and throws a helpful error message if needed.
To confirm that this is indeed the problem here, you could just run the script on a machine with a more modern version gnuplot (>=4.6). You may need to copy the results file (.txt) to a disk which is visible from that machine, but otherwise it should be a quick test.
If that fixes the problem, I'm afraid I can't help you much more - it would take me too long to rewrite the script to be compatible with older gnuplot versions. You could ask your sysadmin to install a more recent version of gnuplot somewhere or do it yourself.
Let me know if a more recent version fixes things.
Thanks for the detailled error reporting!
Alexis
Gnuplot error gone, but...
In reply to script requires gnuplot >= 4.6 by Alexis
Hi Alexis,
we got the 4.6 version running right now, but it seems that I still have another problem and maybe its now pdftk.
+ '[' 1 -ne 1 ']'
+ input_fn=align_14nov03a_XXX_XXX00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ output_fn=align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf
++ ls align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames.txt
+ input_summary_fn=align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames.txt
+ hash gnuplot
+ transpose_ignore_comments align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ gawk '
{
if ($1 ~ /^#/) {
print $0
} else {
for (i=1; i<=NF; i++) {
a[NR,i] = $i
}
}
}
NF>p { p = NF }
END {
for(j=1; j<=p; j++) {
str=a[1,j]
for(i=2; i<=NR; i++){
str=str" "a[i,j];
}
print str
}
}' align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
++ gawk 'match($0,/Pixel size: ([0-9.]*)/,a) {print a[1]}' align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ pixel_size=1.260
++ gawk 'match($0,/Number of micrographs: ([0-9]*)/,a) {print a[1]}' align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ number_of_micrographs=1
+ lines_per_micrograph=6
+ i=0
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ 1.000000 != \#* ]]
+ df_one[++i]=38160.07
+ df_two[i]=36721.14
+ angast[i]=0.9210151
+ score[i]=0.1782483
+ maxres='[i]=16.12781'
+ read -a myArray
+ echo
+ gnuplot
+ hash pdftk
+ echo 'InfoKey: Title'
+ echo 'InfoValue: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt'
+ mv align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf /tmp/tmp.pdf
+ pdftk /tmp/tmp.pdf update_info /tmp/pdftemp.txt output align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf
Error: Unable to find file.
Error: Failed to open PDF file:
/tmp/tmp.pdf
Errors encountered. No output created.
Done. Input errors, so no output created.
+ echo ' '
+ echo 'Generated plots of 1D profiles and fits: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf'
Generated plots of 1D profiles and fits: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf
+ echo ' '
Actually from what you're
In reply to Gnuplot error gone, but... by Marcus
Actually from what you're showing, it's not clear that gnuplot was successful at creating the pdf file... Pdftk basically just says that /tmp/tmp.pdf could not be opened. /tmp/tmp.pdf is the output of gnuplot, moved & renamed. So it seems that gnuplot didn't create the pdf file in the first place, perhaps.
Did you check the output from gnuplot as I suggested in reply #3 above?
Sorry my bad
In reply to Actually from what you're by Alexis
I used again the unchanged script. If I run it again as told in #3 it seems that I am back to the old error.
Checking the /tmp folder shows that the tmp.pdf file is still having a zero filesize.
Now I am not sure if the script is really using the 4.6 version and not still the 4.4 version. I aliased gnuplot to the 4.6 version but the 4.4 version is still installed on this cluster. Can I introduce a hard coded location of gnuplot in your script somewhere to be sure?
And if this all turns out to be too complicated we (somebody with more programming knowledge than me) might just go ahead and write a new plotting script in our lab.
../../ctffind/ctffind-4.0.7-linux64/ctffind_plot_results.sh align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ '[' 1 -ne 1 ']'
+ input_fn=align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ output_fn=align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf
++ ls align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames.txt
+ input_summary_fn=align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames.txt
+ hash gnuplot
+ transpose_ignore_comments align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ gawk '
{
if ($1 ~ /^#/) {
print $0
} else {
for (i=1; i<=NF; i++) {
a[NR,i] = $i
}
}
}
NF>p { p = NF }
END {
for(j=1; j<=p; j++) {
str=a[1,j]
for(i=2; i<=NR; i++){
str=str" "a[i,j];
}
print str
}
}' align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
++ gawk 'match($0,/Pixel size: ([0-9.]*)/,a) {print a[1]}' align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ pixel_size=1.260
++ gawk 'match($0,/Number of micrographs: ([0-9]*)/,a) {print a[1]}' align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt
+ number_of_micrographs=1
+ lines_per_micrograph=6
+ i=0
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ # != \#* ]]
+ read -a myArray
+ [[ 1.000000 != \#* ]]
+ df_one[++i]=38160.07
+ df_two[i]=36721.14
+ angast[i]=0.9210151
+ score[i]=0.1782483
+ maxres='[i]=16.12781'
+ read -a myArray
+ echo
+ gnuplot
gnuplot> set style line 2 lc rgb 'red' lt 1 lw 1 pt 7 fill transparent solid 0.5 # --- red
^
line 0: Extraneous arguments to set style line
gnuplot> do for [current_micrograph=1:1] {
^
line 0: invalid complex constant
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
line 0: undefined variable: current_micrograph
gnuplot> }
^
line 0: invalid character }
+ hash pdftk
+ echo 'InfoKey: Title'
+ echo 'InfoValue: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.txt'
+ mv align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf /tmp/tmp.pdf
+ pdftk /tmp/tmp.pdf update_info /tmp/pdftemp.txt output align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf
Error: Unable to find file.
Error: Failed to open PDF file:
/tmp/tmp.pdf
Errors encountered. No output created.
Done. Input errors, so no output created.
+ echo ' '
+ echo 'Generated plots of 1D profiles and fits: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf'
Generated plots of 1D profiles and fits: align_14nov03a_XXX_XXX_00058gr_00055sq_v02_00019hl_00001en.frames_avrot.pdf
+ echo ' '
gnuplot versions
In reply to Sorry my bad by Marcus
It should be easy to force the script to use a particular gnuplot binary.
Near the top of the script add a line like this:
gnuplot_exec="/path/to/recent/gnuplot"Then comment out the following lines:
if ! hash gnuplot 2>/dev/null; thenecho "Gnuplot was not found on your system. Please make sure it is installed."
exit -1
fi
And replace this line:
gnuplot > /dev/null 2>&1 <
with:
$gnuplot_exec > /dev/null 2>&1 <Perfect
In reply to gnuplot versions by Alexis
Thanks a lot, now it's running fine. The script was actually still calling the older gnuplot version.
Therefore hardcoding the location of the gnuplot executable made the trick.
Cheers
Marcus