Regression Test v1.1 (beta)

Introduction

Regression test package provides a test framework for Community sediment transport model. User can automate model runs as well as various tests on
model output. The package is written in Perl and XML. XML serves as a tool to automate the ROMS run as well as NCO tools tests on output.
It has been tested to run on Linux (2.6.15) and Windows under 'Cygwin'. Supported version of ROMS: 3.2..
(Successfully tested with Revision: 1553 of URL: ​https://www.myroms.org/svn/cstm/trunk).

Configure

Go to the directory where you have checked out the beta version of the regression package. Type

$ ./configure <ROMS root path>

ROMS root path: Complete directory path where ROMS source code has been installed.

Configuration script checks for availability of various utilities (perl, ncotools, libxml, qsub etc) and updates path and relevant information in various files.
Provide a new/existing workspace name, when prompted to create a workspace for test results.
You must run 'Configure before proceeding further.

How to Run?

Go to the directory where you have checked out the beta version of the regression package. Type

$ ./bin/runRegression 

If you see the output as shown below means there is future ahead.

------------------- Regression Test Environment for ROMS ----------------------                                                        
                                 Version 1.1


 runRegression: Automate model run and apply regression tests to ROMS-SED ocean model.

 Usage: runRegression -xml <file>
 Where:

   -xml <file>     :   XML file which defines the regression test scenario/task.

................................................................................
Questions/Queries:
Sachin Kumar Bhate (skbhate@ngi.msstate.edu

In order to automate the model run(s), you will have to specify the compile and run parameters,
in an XML document. Follow the instructions below on how to configure XML.

Configuring XML?

XML provides an easy way to configure model runs and tests. User can configure various ROMS applications (e.g. UPWELLING, CBLAST etc) using XML.
Check XML document 'regressionCSTM.xml' in '/data/metadata' on how to configure model-run. Examples are also provided in '/data/metadata/example' to try.
Examples 'example_simple.xml' which runs application 'UPWELLING' can be tried as it is without any further editing. Other examples may need some editing for directory path, etc.

Few good XML editors/validators are 'oXygen' and 'Architag's Xray'. [Detailed configuration help coming soon]

Understanding the Directory Structure

It is important to understand the directory structure before you proceed. It provides a quick overview of where, what and how the data and test results are located.

• /bin: Regression test executables.
• /lib: Required modules and libraries. It contains the properties file regression.props.
• /log: Test run log 'regression.log'. It logs the build and run activities for the model.
• /data: Data and files in support of running the tests.
• /data/Master: Master NetCDF output data files for various model applications (e.g. SED_TOY, SHOREFACE etc.). These datasets are used in ​NCO-tools (e.g. ncdiff) tests.
  Here, the master file refers to a true data file against which a user would compare the model output. User must populate this directory with their master data files before proceeding
  to run any tests. 'ncdiff' would result in error without these files. The files could be of type 'history','averages','diagnostics' or 'restart'. This directory needs to be populated only
  in case you are running any nco-tools tests.
• /data/metadata: Contains XML files for environment setup. Use 'regressionCSTM.xml' as a guide to create new xml configuration files.
• /data/metadata/schema: Schema for XML document validation. Please do not edit files in this directory.
• /data/metadata/example: Example XML files to be used for environment setup. User should edit these files before running the regression application.
• /test-run/<workspace>: Test results. Each time test is run, it creates a time-stamped directory in '/test-run/<workspace>' where it stores all test results for each model application.
  The 'STDERR' and 'STDOUT' from model compilation are also saved in this directory.

Understanding Results

Test results, which are stored in /test-run/<workspace>/<time-stamped-dir/<application>/<application_instance>/<compiler> may contain following data files.

• ocean_<file_type>.nc: Model output data files. <file_type> can be his(history), dia(diagnostics), rst(restart), avg(averages) etc.
• output_<application>.out: STDOUT from model run. <application> can be e.g.'SED_TOY', 'SHOREFACE', etc.
• build.err: STDERR from building/compiling ROMS model.
• build.out: STDOUT from building/compiling ROMS model.
• ncdiff_<file_type>.nc: ​NCO-tools 'ncdiff' output. <file_type> can be 'history', 'average', etc.
• ncks<file_type>.nc: ​NCO-tools 'ncks' output. <file_type> can be 'history', 'average', etc.
• roms_pbs.sh: PBS script auto-generated for the run.

Understanding Logs

Log 'regression.log' contains the time line of test execution. It reports both success and failure of test. Example log shown below:

>>>> New Log: Tue Jul 15 15:22:48 2008

ROMS PATH: /home/skbhate/cstm/current_roms/roms3.2_070808

[SUCCESS][Tue Jul 15 15:22:48 2008][XML document data/metadata/example/example_complex.xml parsed succesfully.]

Application: WAKE:wake_1:ifort.

[ERROR][Tue Jul 15 15:22:48 2008][./bin/runRegression,][ROMS WAKE:wake_1:ifort build failed. Check 'build.err' for more info.]

>>>> New Log: Tue Jul 15 16:18:55 2008

ROMS PATH: /home/skbhate/cstm/current_roms/roms3.2_070808

[SUCCESS][Tue Jul 15 16:18:55 2008][XML document data/metadata/example/example_complex2.xml parsed succesfully.]

Application: UPWELLING:upwelling_1:ifort.

[SUCCESS][Tue Jul 15 16:18:55 2008][Project directory does not exist:  . Setting MY_PROJECT_DIR to ROMS ROOT DIR.]
[SUCCESS][Tue Jul 15 16:27:17 2008][Compiled application instance UPWELLING:upwelling_1:ifort successfully.]
[SUCCESS][Tue Jul 15 16:27:17 2008][Created PBS script: /home/skbhate/cstm/current_roms/roms3.2_070808/roms_pbs.sh]
[SUCCESS][Tue Jul 15 16:27:17 2008][Starting ROMS-SED MPI version using PBS script /home/skbhate/cstm/current_roms/roms3.2_070808/roms_pbs.sh]
[SUCCESS][Tue Jul 15 16:27:17 2008][PBS JOB ID:  14467.pikmin.cl.whoi.edu]
[SUCCESS][Tue Jul 15 16:39:17 2008][PBS job 14467.pikmin.cl.whoi.edu finished. Check output.]
[SUCCESS][Tue Jul 15 16:39:17 2008][ROMS run successful. History or Restart file created.]

Running NCO tools tests.......

File type: his
[SUCCESS][Tue Jul 15 16:39:18 2008][Created ncdiff output.]

Things to Remember

Please make appropriate changes or updates based on the following points before you proceed to run the tests.

• Do not edit/Customize variables in regression.props file unless it is mentioned in the file.
• Make sure you run 'validation tool' on your xml document frequently while editing the xml document. For example:

    ./bin/validate   /data/metadata/example/example_simple.xml
  

• You can always send an email to skbhate@…in case of any problems/issues with testing package.