= Regression Test v1.1 (beta) = == Introduction == Regression test package provides a '''''test framework for Community sediment transport model'''''. User can automate model runs as well as various tests on[[BR]] model output. The package is written in '''''Perl''''' and '''''XML'''''. XML serves as a tool to automate the ROMS run as well as NCO tools tests on output.[[BR]] It has been tested to run on Linux (2.6.15) and Windows under 'Cygwin'. Supported version of '''''ROMS: 3.2.'''''.[[BR]] (Successfully tested with Revision: 1553 of URL: https://www.myroms.org/svn/cstm/trunk). == Configure == Go to the directory where you have checked out the beta version of the regression package. Type {{{ $ ./configure ROMS root path: Complete directory path where ROMS source code has been installed. }}} Configuration script checks for availability of various utilities (perl, ncotools, libxml, qsub etc) and updates path and relevant information in various files.[[BR]] '''You must run 'Configure before proceeding further.''' == How to Run? == Go to the directory where you have checked out the beta version of the regression package. Type {{{ $ ./bin/runRegression }}} If you see the output as shown below means there is future ahead. {{{ ------------------- Regression Test Environment for ROMS ---------------------- Version 1.1 runRegression: Automate model run and apply regression tests to ROMS-SED ocean model. Usage: runRegression -xml Where: -xml : XML file which defines the regression test scenario/task. ................................................................................ Questions/Queries: Sachin Kumar Bhate (skbhate@ngi.msstate.edu }}} In order to automate the model run(s), you will have to specify the compile and run parameters,[[BR]] in an XML document. Follow the instructions below on how to configure XML. == Configuring XML? == XML provides an easy way to configure model runs and tests. User can configure various ROMS applications (e.g. UPWELLING, CBLAST etc) using XML.[[BR]] Check XML document ''''regressionCSTM.xml'''' in '/data/metadata' on how to configure model-run. Examples are also provided in '/data/metadata/example' to try.[[BR]] Examples 'example_simple.xml' which runs application 'UPWELLING' can be tried as it is without any further editing. Other examples may need some editing for directory path, etc.[[BR]] Few good XML editors/validators are 'oXygen' and 'Architag's Xray'. [Detailed configuration help coming soon] == Understanding the Directory Structure == It is important to understand the directory structure before you proceed. It provides a quick overview of where, what and how the data and test results are located. * '''/bin:''' Regression test executables. * '''/lib:''' Required modules and libraries. It contains the properties file [wiki:RegressionProps regression.props]. * '''/log:''' Test run log 'regression.log'. It logs the build and run activities for the model. * '''/data:''' Data and files in support of running the tests. * '''/data/Master:''' Master {{{NetCDF}}} output data files for various model applications (e.g. {{{SED_TOY, SHOREFACE}}} etc.). These datasets are used in [http://nco.sourceforge.net/ NCO-tools] (e.g. ncdiff) tests.[[BR]] Here, the master file refers to a ''true'' data file against which a user would compare the model output. User must populate this directory with their master data files before proceeding[[BR]]to run any tests. 'ncdiff' would result in error without these files. The files could be of type 'history','averages','diagnostics' or 'restart'. This directory needs to be populated only[[BR]]in case you are running any nco-tools tests. * '''/data/metadata:''' Contains XML files for environment setup. Use 'regressionCSTM.xml' as a guide to create new xml configuration files. * '''/data/metadata/schema:''' Schema for XML document validation. Please do not edit files in this directory. * '''/data/metadata/example:''' Example XML files to be used for environment setup. User should edit these files before running the regression application. * '''/test-run:''' Test results. Each time test is run, it creates a time-stamped directory in '/test-run' where it stores all test results for each model application. User can have the test[[BR]] results be saved to another location by modifying the ''$TEST_RUN'' variable in [wiki:RegressionProps regression.props] file. The 'STDERR' and 'STDOUT' from model compilation are also saved in this directory. == Understanding Results == Test results, which are stored in '''/test-run///''' may contain following data files. * '''ocean_.nc:''' Model output data files. '''' can be ''his(history)'', ''dia(diagnostics)'', ''rst(restart)'', ''avg(averages)'' etc. * '''output_.out:''' STDOUT from model run. '''' can be e.g.'SED_TOY', 'SHOREFACE', etc. * '''build.err:''' STDERR from building/compiling ROMS model. * '''build.out:''' STDOUT from building/compiling ROMS model. * '''ncdiff_.nc:''' [http://nco.sourceforge.net/ NCO-tools] 'ncdiff' output. '''' can be 'history', 'average', etc. * '''ncks.nc:''' [http://nco.sourceforge.net/ NCO-tools] 'ncks' output. '''' can be 'history', 'average', etc. * '''roms_pbs.sh:''' PBS script auto-generated for the run. == Understanding Logs == Log 'regression.log' contains the time line of test execution. It reports both success and failure of test. Example log shown below: {{{ >>>> New Log: Tue Jul 15 15:22:48 2008 ROMS PATH: /home/skbhate/cstm/current_roms/roms3.2_070808 [SUCCESS][Tue Jul 15 15:22:48 2008][XML document data/metadata/example/example_complex.xml parsed succesfully.] Application: WAKE:wake_1:ifort. [ERROR][Tue Jul 15 15:22:48 2008][./bin/runRegression,][ROMS WAKE:wake_1:ifort build failed. Check 'build.err' for more info.] >>>> New Log: Tue Jul 15 16:18:55 2008 ROMS PATH: /home/skbhate/cstm/current_roms/roms3.2_070808 [SUCCESS][Tue Jul 15 16:18:55 2008][XML document data/metadata/example/example_complex2.xml parsed succesfully.] Application: UPWELLING:upwelling_1:ifort. [SUCCESS][Tue Jul 15 16:18:55 2008][Project directory does not exist: . Setting MY_PROJECT_DIR to ROMS ROOT DIR.] [SUCCESS][Tue Jul 15 16:27:17 2008][Compiled application instance UPWELLING:upwelling_1:ifort successfully.] [SUCCESS][Tue Jul 15 16:27:17 2008][Created PBS script: /home/skbhate/cstm/current_roms/roms3.2_070808/roms_pbs.sh] [SUCCESS][Tue Jul 15 16:27:17 2008][Starting ROMS-SED MPI version using PBS script /home/skbhate/cstm/current_roms/roms3.2_070808/roms_pbs.sh] [SUCCESS][Tue Jul 15 16:27:17 2008][PBS JOB ID: 14467.pikmin.cl.whoi.edu] [SUCCESS][Tue Jul 15 16:39:17 2008][PBS job 14467.pikmin.cl.whoi.edu finished. Check output.] [SUCCESS][Tue Jul 15 16:39:17 2008][ROMS run successful. History or Restart file created.] Running NCO tools tests....... File type: his [SUCCESS][Tue Jul 15 16:39:18 2008][Created ncdiff output.] }}} == Things to Remember == Please make appropriate changes or updates based on the following points before you proceed to run the tests. * Do not edit/Customize variables in [wiki:RegressionProps regression.props] file unless it is mentioned in the file. * Make sure you run 'validation tool' on your xml document frequently while editing the xml document. For example: {{{ ./bin/validate /data/metadata/example/example_simple.xml }}} * You can always send an email to [mailto:skbhate@ngi.msstate.edu skbhate@ngi.msstate.edu]in case of any problems/issues with testing package.