model stopped with strange results

[fanzhongya]

Hi, Dear all!
I ran a simulation of tide of a real case,then a confused question was arisen.

MY time configuration is
! Time-Stepping parameters.

NTIMES == 172800
DT == 60.0d0
NDTFAST == 10

But my model stopped at 90 days with this result:
129595 89 23:55:00 2.954877E-02 2.500259E+02 2.500554E+02 6.934261E+12
129596 89 23:56:00 2.943912E-02 2.500138E+02 2.500433E+02 6.934152E+12
129597 89 23:57:00 2.932872E-02 2.500017E+02 2.500311E+02 6.934038E+12
129598 89 23:58:00 2.921762E-02 2.499896E+02 2.500188E+02 6.933921E+12
129599 89 23:59:00 2.910586E-02 2.499774E+02 2.500065E+02 6.933799E+12
rank 10 in job 1 c0128_33749 caused collective abort of all ranks
exit status of rank 10: killed by signal 9
rank 21 in job 1 c0128_33749 caused collective abort of all ranks
exit status of rank 21: killed by signal 9
rank 19 in job 1 c0128_33749 caused collective abort of all ranks
exit status of rank 19: killed by signal 9
rank 7 in job 1 c0128_33749 caused collective abort of all ranks
exit status of rank 7: killed by signal 9
rank 3 in job 1 c0128_33749 caused collective abort of all ranks
exit status of rank 3: killed by signal 9
rank 2 in job 1 c0128_33749 caused collective abort of all ranks
exit status of rank 2: killed by signal 9
rank 1 in job 1 c0128_33749 caused collective abort of all ranks
exit status of rank 1: killed by signal 9
rank 0 in job 1 c0128_33749 caused collective abort of all ranks
exit status of rank 0: killed by signal 9

I examined his.nc,but no strange velocity and zeta at the end of time.

THANKS ALL!

[feroda]

1. How about your setup for the following two parameters:
NHIS == ??
NDEFHIS == ??

2. What's the version of your ROMS code?

[fanzhongya]

Hi,feroda


I run using HPC and my configuration is
! Output history, average, diagnostic files parameters.

LDEFOUT == T
NHIS == 60
NDEFHIS == 0
NTSAVG == 1
sometimes it continued to work at the time of the 90 days,but sometimes it failed.

My teacher suggested some problems about core storage may cause this result.About you?

thanks for your advance

[fanzhongya]

Model Input Parameters: ROMS/TOMS version 3.2