Error : Cannot close file

Submitted by erichard on Wed, 2010-05-12 10:57

Forums

Frealign

Hi,

I try to launch frealign 8.08 on my cluster with this attribute :
1 stack dal_stack.mrc size : 530 Mo and more than 13000 particles
I set the first, last and increment : 1,1000,100 to use 10 core with small stack
I use Torque system (PBS and MAUI) and I had modified your examples script.

The script PBS run but the third job always stop
I have a error message in the Third log file : dal_mrefine_n.log_201_300

Opening MRC/CCP4 file for READ...
112 File : dal_2.mrc_201_300
113 NX, NY, NZ: 100 100 100
114 MODE : real
115 Min, max : -0.1096198 0.1119255
116 Mean, RMS : -0.3023889E-05 0.0000000
117 TITLE 1: TRUE3D "exact" filtered back projection // Created by IMAGIC: IMAGIC image = 3d-
118 TITLE 2: Scy3-1f30 02-05-20 16:10:38 //
119
120 ***WARNING*** Circumference STD
121 significant compared with volume STD, could indicate
122 noisy reference or density gradient !!!
123
124 Mean density of 3D volume: -0.00000302
125 STD of 3D volume: 0.01251854
126 Mean density of circumference: 0.00031847
127 STD of circumference: 0.00429400
128
129 3D WEIGHTS FILE FOR OUTPUT?
130 3D RECONSTRUCTION HALFSET 1 FOR OUTPUT ?
131 3D RECONSTRUCTION HALFSET 2 FOR OUTPUT ?
132 3D PHASE RESIDUAL FILE FOR OUTPUT ?
133 3D POINT SPREAD FUNCTION FOR OUTPUT ?
134 Cannot close file ...

Have you one idea about this error ?
Is it possible to have a problem with the time of copy files ?
The stack file is on the local disk (500 Mo)
The access right are OK
I tried to modify increment to modify the stack size

Thank

niko Wed, 2010-05-12 23:31

The error message comes from Frealign’s file handling routine ioc.c. It looks as if Frealign is trying to close a file that either has never been opened or that was opened and closed already. From the log file, it looks like the problem file is the stack, dal_stack.mrc.

To be honest, I have never seen this problem before. According to your description, Frealign runs fine if you use only two cores. I am not familiar with the PBS/MAUI system. Could it be that there is some kind of limit on your system for the files that can be open at the same time? When you look through your log file, is there another error related to opening a file?

If you send me all your scripts and log files I might be able to say more.

erichard Wed, 2010-06-30 11:31

Hello Niko,

I was occupied by an other problem and I come back.

I tried to install the GPU's version with the last cuda's driver versions 256.35 on my Mandriva's workstation
All was fine except the end of make
Do you know if it exist a forum on GPU version of Frealign, because I have also some trouble "multiple definition" with the link step of make whith this version.

../..
nvcc -arch sm_13 -lpthread -O3 -lg2c -lcufft -DPROTOTYPE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE frealign_v8.cu frealign_v8.a -o ../bin/frealign_v8.exe_ref
frealign_v8.a(rlft3.o): In function `cPc(float2, float2)':
tmpxft_00004cda_00000000-1_rlft3.cudafe1.cpp:(.text+0x1390): multiple definition of `cPc(float2, float2)'
frealign_v8.a(presb_gpu.o):tmpxft_00004c7e_00000000-1_presb_gpu.cudafe1.cpp:(.text+0x1120): first defined here
frealign_v8.a(rlft3.o): In function `cXc(float2, float2)':
tmpxft_00004cda_00000000-1_rlft3.cudafe1.cpp:(.text+0x13b0): multiple definition of `cXc(float2, float2)'
../..

If you have suggestion on this problem of multiple definition

For the mp versions you can find above my script for PBS : msearch_pbs.com, my parameters file and the end of log stack_msearch_n.log_1_200 and 201_400

Many thanks

#################################
FILE msearch_pbs.com
################################"
!/bin/csh
###########################################################
# msearch_pbs.com
##########################################################
#
# Control script to submit multiple jobs on a cluster using
# the Sun Grid Engine. Each job processes N particles. N is
# specified in the 'mparameters' file as 'increment'.
#################################################################
# Partie a modifier si necessaire
# Adresse Mail a pr�venir quand c'est fini
set addr_mail = erichard@cbs.cnrs.fr
# Nom du Job
set date_lancer = `date +%D`
set name_job = `echo Frealign_$date_lancer`
#################################################################

if (! -e ../scratch) mkdir ../scratch

cp mparameters mparameters_run
set start = `grep start_process mparameters_run | awk '{print $2}'`
set end = `grep end_process mparameters_run | awk '{print $2}'`
set first = `grep first_particle mparameters_run | awk '{print $2}'`
set last = `grep last_particle mparameters_run | awk '{print $2}'`
set incr = `grep increment mparameters_run | awk '{print $2}'`
set data_input = `grep data_input mparameters_run | awk '{print $2}'`
set working_directory = `pwd`
set SCRATCH = ../scratch
set output = $working_directory/Time_Info
mainloop:

cp mparameters mparameters_run
\rm ${data_input}_$start.par >& /dev/null

# queue submission loop

set firstn = $first
@ lastn = $first + $incr - 1
while ( $lastn <= $last )

echo "Submitted particles from" $firstn to $lastn on `date`>$output
# qsub -l mem_free=1G walltime=00:30:00 nodes=1:ppn=1 -c n -r y -V -e ~/stderr -o ~/stderr -N a${firstn}_$lastn \
# $working_directory/msearch_n.com $firstn $lastn $start
qsub -l walltime=00:30:00,nodes=1:ppn=1 -c n -r y -V -N $name_job -e $working_directory/stderr -o $working_directory/stderr <= $last ) set firstn = $last
if ( $lastn >= $last ) set lastn = $last
end

checkdone:

sleep 5
#\rm $working_directory/stderr
while ( $lastn <= $last )
set continue = 0

grep "overall phase residual" $SCRATCH/${data_input}.par_${firstn}_$lastn >& /dev/null
if (! $status) then
echo "Particles $firstn to $lastn finished.... "`date`
if ($firstn == $first ) head -50 $SCRATCH/${data_input}.par_${firstn}_${lastn} | grep C >> ${data_input}_$start.par
grep -v C $SCRATCH/${data_input}.par_${firstn}_${lastn} >> ${data_input}_$start.par
\rm $SCRATCH/${data_input}.par_${firstn}_${lastn} >& /dev/null
if ( $lastn == $last ) goto reconstruct
set continue = 1
endif
if (! $continue ) goto checkdone
@ firstn = $firstn + $incr
@ lastn = $lastn + $incr
if ( $firstn >= $last ) set firstn = $last
if ( $lastn >= $last ) set lastn = $last
end

reconstruct:
echo "Calculating 3D structure...."
set job_reconstruct = 'recon_$name_job'
qsub -l walltime=00:30:00,nodes=1:ppn=1 -c n -r y -V -N $job_reconstruct -e $working_directory/stderr -o $working_directory/stderr <> ${data_input}_$start.par

if ($start < $end ) then
@ start = $start + 1
goto mainloop
endif
dlltime=00:30:00,nodes=1:ppn=1 -c n -r y -V -N $job_reconstruct -e $working_directory/stderr -o $working_directory/stderr <>$output
mailto -s "$name_job termine" $addr_mail
################ END of FILE ################################################

########## FILE mparameters_run ####################################
arameter file for mrefine_sge.com

dir_exec /home/erichard/FreAlignP23/msearch
data_input dal
raw_images1 dal_stack
thresh_reconst 90.0
thresh_refine 25.0
pixel_size 4.38
dstep 21.9
WGH 0.07
CS 2.0
kV1 200.0
radius 175.0
PBC 100.0
BOFF 35.0
DANG 200.0
ITMAX 50
ISYM D3
MODE -4
XSTD 0.0
RBFACT 0.0
FLIP F
FMAG F
FDEF F
FASTIG F
IEWALD 0
res_reconstruction 15.0
res_low_refinement 100.0
res_refinement 25.0
start_process 2
end_process 2
first_particle 1
last_particle 2000
increment 200
####################### END of FILE ############################

##########################################################
FILE stack_msearch_n.log_1_200
########################################################
MODE : real
Min, max : -0.1096198 0.1119255
Mean, RMS : -0.3023889E-05 0.0000000
TITLE 1: TRUE3D "exact" filtered back projection // Created by IMAGIC: IMAGIC image = 3d-
TITLE 2: Scy3-1f30 02-05-20 16:10:38 //

***WARNING*** Circumference STD
significant compared with volume STD, could indicate
noisy reference or density gradient !!!

Mean density of 3D volume: -0.00000302
STD of 3D volume: 0.01251854
Mean density of circumference: 0.00031847
STD of circumference: 0.00429400

3D WEIGHTS FILE FOR OUTPUT?
3D RECONSTRUCTION HALFSET 1 FOR OUTPUT ?
3D RECONSTRUCTION HALFSET 2 FOR OUTPUT ?
3D PHASE RESIDUAL FILE FOR OUTPUT ?
3D POINT SPREAD FUNCTION FOR OUTPUT ?

Opening MRC/CCP4 file for READ...
File : /home/erichard/FreAlignP23/msearch/../dal_stack.mrc
NX, NY, NZ: 100 100 13414
MODE : real
Min, max : -9.589561 11.14859
Mean, RMS : -0.7896812E-01 0.0000000
TITLE 1: 4530,1387 : /disk2-1/p2324Cryo-neg/STOCK/EM0645.tif (BAND) // Cr
TITLE 2: eated by IMAGIC: IMAGIC image = redS 02-05-20 15:54:19 //

Time before particle 1 was 10:53:35
Best phase residual for particle 1 at Rmin/Rmax 100.0 25.0: 68.741
Time before particle 2 was 10:53:56
Best phase residual for particle 2 at Rmin/Rmax 100.0 25.0: 67.177
Best phase residual for particle 3 at Rmin/Rmax 100.0 25.0: 62.351
Best phase residual for particle 4 at Rmin/Rmax 100.0 25.0: 67.776
Best phase residual for particle 5 at Rmin/Rmax 100.0 25.0: 67.535
Best phase residual for particle 6 at Rmin/Rmax 100.0 25.0: 66.684
Best phase residual for particle 7 at Rmin/Rmax 100.0 25.0: 66.223
Best phase residual for particle 8 at Rmin/Rmax 100.0 25.0: 67.259
Best phase residual for particle 9 at Rmin/Rmax 100.0 25.0: 69.246
Best phase residual for particle 10 at Rmin/Rmax 100.0 25.0: 67.432
Time before particle 11 was 10:57:01
Best phase residual for particle 11 at Rmin/Rmax 100.0 25.0: 66.274
Best phase residual for particle 12 at Rmin/Rmax 100.0 25.0: 65.450
########################## END OF FILE ##########################################

##################################################################################
FILE stack_msearch_n.log_201_400
################################################################################"
Outer Radius of object [Angstroms] . 175.00
C Inner Radius of object [Angstroms] . 0.00
C Pixel size [Angstroms] . . . . . . . 4.38000
C % Amplitude contrast . . . . . . . . 0.07
C STD level for 3D mask. . . . . . . . 0.00
C Phase res. / B factor constant . . . 100.00
C Average phase residual for weighting 0.00
C Symmetry card as input . . . . . . . 0
C Number of symmetry operators . . . . 0
C No. cycles for randomised search . . 50
C No. of search peaks to refine. . . . 10
MAGNIFICATION, STEPSIZE, TARGET & THRESH PHASE
RESIDUAL LIMIT, CS, KV, BEAM TILT X,Y ?
-100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
End of input datasets, number of datasets = 1
Average phase residual used 0.0000
C Search angle incr. 360.00 No. of search angles: 2
3D MAP FILE FOR INPUT?
/home/erichard/FreAlignP23/msearch/../dal_1.mrc

Opening MRC/CCP4 file for READ...
File : /home/erichard/FreAlignP23/msearch/../dal_1.mrc
NX, NY, NZ: 100 100 100
MODE : real
Min, max : -0.1096198 0.1119255
Mean, RMS : -0.3023889E-05 0.0000000
TITLE 1: TRUE3D "exact" filtered back projection // Created by IMAGIC: IMAGIC image = 3d-
TITLE 2: Scy3-1f30 02-05-20 16:10:38 //

***WARNING*** Circumference STD
significant compared with volume STD, could indicate
noisy reference or density gradient !!!

Mean density of 3D volume: -0.00000302
STD of 3D volume: 0.01251854
Mean density of circumference: 0.00031847
STD of circumference: 0.00429400

3D WEIGHTS FILE FOR OUTPUT?
3D RECONSTRUCTION HALFSET 1 FOR OUTPUT ?
3D RECONSTRUCTION HALFSET 2 FOR OUTPUT ?
3D PHASE RESIDUAL FILE FOR OUTPUT ?
3D POINT SPREAD FUNCTION FOR OUTPUT ?
Cannot close file ...

###################### END OF FILE #################################

niko Wed, 2010-06-30 13:13

I am still not sure if I understand completely when the error occurs. In your first post you mentioned that the third job crashes but in your latest post it looks like your second job crashes. Based on your latest post, I wonder if your system does not allow two processes to read the same file at the same time. In your script (which is based on the script included with the Frealign distribution) each msearch_n job will open the image stack for read. If only one process can read a file at a time, the second process (and all following processes) will fail to open the stack.

Maybe you can do an experiment and run one msearch_n job and try to open the stack with another program while the msearch_n job is running. This might help narrow down what the problem is.

Jianhao Wed, 2017-02-15 03:14

Hi Niko,

I ran my project on a single node, but also had this error problem("Cannot close file...").
I used frealign v9.11.
Could you provide some suggestions?

Thank you!

Regards,
Jianhao

niko Wed, 2017-02-15 12:55

I am very sorry that the problem still exists! However, I was unable to resolve it previously and am therefore not able to help. All I can recommend is to try and run it using different disks. Maybe it has to do with the file system (see above).

xueming Wed, 2010-06-30 14:15

Hi erichard,
For the GPU version, try to delete frealign_v8.a before make. Maybe some sub-library were inserted into frealign_v8.a twice.
Besides, the GPU version of FREALIGN you downloaded might have some problems when running with the latest CUDA 3 and Fermi GPU. I will send an updated version to niko and yifan, in which I fix many bugs and modify some functions. The new version will be more stable and faster than the old one.