= Regression Test v1.1 (beta) =

== Introduction ==
Regression test package provides a '''''test framework for Community sediment transport model'''''. User can automate model runs as well as various tests on
model output. The package is written in '''''Perl'''''. It has been tested to run on Linux (2.6.15) and Windows under 'Cygwin'. 
Supported version of '''''ROMS: 3.0.''''' (Successfully tested SVN version 843).

== Configure ==
Go to the directory where you have checked out the beta version of the regression package. Type
{{{
$ ./configure <ROMS root path>

ROMS root path: Complete directory path where ROMS source code has been installed.
}}}

Configuration script checks for availability of various utilities (perl, ncotools, etc) and updates path and relevant information in various files. 
You must run 'Configure' before proceeding further.

== How to Run? ==
Go to the directory where you have checked out the beta version of the regression package. Type
{{{
$ ./bin/runRegression 
}}}
If you see the output as shown below means there is future ahead. The default run is 'serial mode'. The MPI run is executed if ROMS 'build.sh' macro {{{'USE_MPI'}}}or {{{'USE_OPENMP'}}} is set to 'on'.
{{{

 runRegression: Run various regression tests for ROMS-SED ocean model.

 Usage: runRegression [options]
 Options:

   -all                :   Run all applications set for regression test.
   -app <application>  :   Run only specific application for regression test.
   -d, -display        :   Display list of applications set for regression test.
   -h, -help           :   Display options.
   -noclean            :   Do not perform clean on existing compiled code.

   -m, -mpi            :   Compile and run using MPI.
 MPI options. (should only be used along with '-m'/'-mpi').

   -nodes <number>     :   Number of Nodes.
   -ppn  <number>      :   Processes Per Node.

 Test options. (Should only be used along with '-all' or '-app').

   -nck                :   Run 'ncks' test on model output.
   -ncd                :   Run 'ncdiff' test on model output.

}}}

== Understanding the Directory Structure ==
It is important to understand the directory structure before you proceed. It provides a quick overview of where, what and how the data and test results are located.

 * '''/bin:''' Regression test executable.
 * '''/lib:''' Required modules and libraries. It contains the properties file [wiki:RegressionProps regression.props]. 
 * '''/log:''' Test run log 'regression.log'. It logs the build and run activities for the model.
 * '''/data:''' Data and files in support of running the tests. 
 * '''/data/regression.tests:''' List of applications defined for regression test. New applications needs to be added to this file before running the test.  See example file below:
{{{
#List applications for regression test.
#Use '-coupled' with application name, to enable SWAN coupling.

BENCHMARK
ESTUARY_TEST
GRAV_ADJ
LAKE_SIGNELL
MIXED_LAYER
SED_TEST
SED_TOY
SHOREFACE
TEST_CHAN
TEST_HEAD
UPWELLING
INLET_TEST -coupled
BEVANO -coupled
}}}

 * '''/data/Master:''' Master {{{NetCDF}}} output data files for various model applications (e.g. {{{SED_TOY, SHOREFACE}}} etc.). These datasets are used in [http://nco.sourceforge.net/ NCO-tools] (e.g. ncdiff) tests. Here, the master file refers to a ''true'' data file against which a user would compare the model output. User must populate this directory with their master data files before proceeding to run any tests. 'ncdiff' would result in error without these files. The files could be of type 'history','averages','diagnostics' or 'restart'. This directory needs to be populated only in case you are running any nco-tools tests. 
 * '''/data/inputList:''' Contains list of input files associated with each model application. If there is an application listed in '/data/regression.tests' file, then user must include the application name and associated input file name in 'inputList' file. List file names without any directory path. Example file:
{{{
ESTUARY_TEST = ocean_estuary_test.in
INLET_TEST   = ocean_inlet_test.in
UPWELLING    = ocean_upwelling.in
}}}

 * '''/test-run:''' Test results. Each time test is run, it creates a time-stamped directory in '/test-run' where it stores all test results for each model application. User can have the test results be saved to another location by modifying the ''$TEST_RUN'' variable in [wiki:RegressionProps regression.props] file. The 'STDERR' and 'STDOUT' from model compilation are also saved in this directory.
== Understanding Results ==
Test results, which are stored in '''/test-run/<time-stamped-dir/<application>''' can contain following data files. 
 * '''ocean_<file_type>.nc:''' Model output data files. ''<file_type>'' can be ''his(history)'', ''dia(diagnostics)'', ''rst(restart)'', ''avg(averages)'' etc.
 * '''output_<application>.out:''' STDOUT from model run. ''<application>'' can be e.g.'SED_TOY', 'SHOREFACE', etc.
 * '''build.err:''' STDERR from building/compiling ROMS model.
 * '''build.out:''' STDOUT from building/compiling ROMS model.
 * '''ncdiff_<file_type>.nc:''' [http://nco.sourceforge.net/ NCO-tools] 'ncdiff' output. ''<file_type>'' can be 'history', 'average', etc.
 * '''ncks<file_type>.nc:''' [http://nco.sourceforge.net/ NCO-tools] 'ncks' output. ''<file_type>'' can be 'history', 'average', etc.
== Understanding Logs ==
Log 'regression.log' contains the time line of test execution. It reports both success and failure of test. Example log shown below:
{{{
>>>> New Log: Fri Jul 13 10:54:50 2007

[SUCCESS][Fri Jul 13 10:54:50 2007][Read /cygdrive/c/skbhate/dev/nopp/regression/data/regression.tests.]

Running Application: INLET_TEST .

[ERROR][Fri Jul 13 10:56:07 2007][./runRegression,run_application][ROMS build failed. ROMS executables not created. 
Check 'build.err' for more info.]
[ERROR][Fri Jul 13 10:56:07 2007][./runRegression,][Model run for INLET_TEST could not be completed.]


 >>>> New Log: Fri Jul 13 12:28:49 2007

[SUCCESS][Fri Jul 13 12:28:49 2007][Read /cygdrive/c/skbhate/dev/nopp/regression/data/regression.tests.]

Running Application: UPWELLING .

[SUCCESS][Fri Jul 13 12:28:50 2007][Compiled ROMS successfully.]
[SUCCESS][Fri Jul 13 12:28:50 2007][Starting ROMS-SED in serial mode...]
[SUCCESS][Fri Jul 13 12:28:57 2007][ROMS run successful. History or Restart file created.]

Running NCO tools tests.......

File type: his
[SUCCESS][Fri Jul 13 12:29:01 2007][Created ncdiff output.]
[SUCCESS][Fri Jul 13 12:29:01 2007][Created ncks output.]

File type: avg
[ERROR][Fri Jul 13 12:29:02 2007][./bin/runRegression,run_application][Master file for file type 'avg' not found 
in /cygdrive/c/skbhate/dev/nopp/regression/data/Master/UPWELLING. 'ncdiff' or 'ncks' output won't be created.]
}}}



== Things to Remember ==
Please make appropriate changes or updates based on the following points before you proceed to run the tests. 
 * Update the variables ''$ROMS_BUILD_FILE'' and ''$MY_CPUS'' in [wiki:RegressionProps regression.props] file.
 * The environmental variable ''$MY_PROJECT_DIR'' in build.sh should always be relative to variable ''$MY_ROOT_DIR''. 
 * The project space for each model application should be declared with ''$MY_PROJECT_DIR'' following this convention.
        $MY_PROJECT_DIR=${MY_ROOT_DIR}/Projects/<application_name>    (e.g. application_name can be 'upwelling', 'benchmark' etc.)

 * If $MY_PROJECT_DIR is defined, then HEADER_DIR and ANALYTICAL DIR will be defined to MY_PROJECT_DIR unless these variables are commented out.
 * Define model applications in file ''regression.tests'', which is located in /data directory.
 * Define input file associated with each model application in file ''inputList'', which is located in /data directory.
 * You can always send an email to [mailto:skbhate@ngi.msstate.edu skbhate@ngi.msstate.edu]in case of any problems/issues with testing package.