= Regression Test v1.1 (beta) = == Introduction == Regression test package provides a '''''test framework for Community sediment transport model'''''. User can automate model runs as well as various tests on[[BR]] model output. The package is written in '''''Perl''''' and '''''XML'''''. XML serves as a tool to automate the ROMS run as well as NCO tools tests on output.[[BR]] It has been tested to run on Linux, Unix and Cygwin. Supported version of '''''ROMS: 3.2.'''''.[[BR]] (Successfully tested with Revision: 1647 of URL: https://www.myroms.org/svn/cstm/trunk). == Getting the Package == The beta version of the test package can be checked out from the svn repository shown below. This is a development version and[[BR]] being updated and tested frequently. [coming soon] == Configure == Configuration is ''required'' and the following steps will guide you to properly configure the test environment. 1. '''Perl''' is the base development language for the test interface. Make sure version 5.8(current stable version) has been installed [[BR]] on your system. In order to install the required stable version 5.8, please go to [http://www.cpan.org/src/README.html Perl CPAN page.][[BR]] {{{ $ perl --version This is perl, v5.8.8 built for cygwin-thread-multi-64int Copyright 1987-2006, Larry Wall }}} 2. Compile and install the required XML libraries.[Required][[BR]] The XML interface for the test framework requires Libxml2 (XML C parser and toolkit) as well as Perl bindings for libxml2.[[BR]] The complete step-by-step installation instructions for libxml2 and Perl modules, please go to [wiki:Libxml2-install Libxml2-installation].[[BR]] [Make sure you install xml libraries before proceeding further]. 3. Checks for availability of various compilers/utilities, creates workspace etc. * '''Perl'''[Required]: Version 5.8.8. * '''PBS utilities'''[Recommended]: Utilities such as qsub, qdel etc. are required to perform job scheduling. * '''mpirun'''[Recommended]: ROMS-CSTM parallel runs. * '''NCO tools'''[Optional]: The netCDF Operators, or NCO tools, such as ncdiff, ncks etc. * '''Workspace'''[Required]: Workspace to store test results. Go to the directory where you have checked out the beta version of the regression package. In subsequent documentation, we will refer the[[BR]] regression package home directory as '$REGRESSION_HOME. Type {{{ $ ./install_it -romspath ROMS root path: Complete directory path where ROMS source code has been installed. }}} It is possible that your system may not have some optional/recommended softwares (as mentioned above), and the[[BR]] script may report that as warnings. It is highly recommended to install the optional softwares in order[[BR]] to take advantage of all the functionalities of regression test package. == Test Installation == Go to $REGRESSION_HOME. Type {{{ $ ./bin/runRegression }}} If everything has been installed correctly, you should see the output as shown below. If you don't[[BR]] then consult the [wiki:RegressionFAQ FAQ] page to correct the problem. {{{ ------------------- Regression Test Environment for ROMS ---------------------- Version 1.1 runRegression: Automate model run and apply regression tests to ROMS-SED ocean model. Usage: runRegression -xml Where: -xml : XML file which defines the regression test scenario/task. ................................................................................ Questions/Queries: Sachin Kumar Bhate (skbhate@ngi.msstate.edu }}} In order to automate the model run(s), you will have to specify the compile and run parameters,[[BR]] in an XML document. Follow the instructions below on how to configure XML. == Configuring XML? == XML provides an easy way to configure model runs and tests. User can configure various ROMS applications (e.g. UPWELLING, CBLAST etc) using XML.[[BR]] See a [wiki:sampleXML sample XML configuration] file. Check detailed documentation on [wiki:XmlDocumentation Configure XML] page. The document is also available in [[BR]]'$REGRESSION_HOME/data/metadata/regressionCSTM.xml'. Examples are also provided in '$REGRESSION_HOME/data/metadata/example' to try. The XML schema can be found in [[BR]] '$REGRESSION_HOME/data/metadata/schema/regressionCSTMSchema.xsd'. For a detailed schematic view of the schema, see [http://www2.msstate.edu/~skb12/nopp/regression/regressionCSTMSchema.html regressionCSTMSchema.xsd] == Try a Simple Test Example == 1. Open file '$REGRESSION_HOME/data/metadata/example/example_simple.xml' in any XML editor (oXygen or Xray). It defines a compilation and model[[BR]] run for application 'UPWELLING'. 2. Locate the tag ''. Edit the path information for various environmental variables and executables with reference to each compiler. 3. Locate the tag '' nested within a '' tag. There will be two '' tags, one is set to 'ifort' and other is set to 'pgi'. [[BR]]This will compile and run test case 'UPWELLING' for 'ifort' and 'pgi' respectively. 4. Locate the tag ''. This ensures whether to carry out the model run or not. It will run the model (i.e. oceanS) from command line. [[BR]] If you would like to submit it via queuing system (i.e. pbs), then remove or comment the tag ''. This will create a pbs script with [[BR]]specified processors and nodes and will submit a job to the pbs queue. The tags '','' and '' are all set to 1, for serial application 'oceanS'. 5. Save the file in the same directory. All xml documents has schema definition/document associated with them and needs to be placed in[[BR]] '$REGRESSION_HOME/data/metadata/example' directory for successful validation. Changing the schema reference path will be discussed later. 6. Go to '$REGRESSION_HOME'. We will now validate the conformity of edited XML document with the schema. Type {{{ $ ./bin/validate $REGRESSION_HOME/data/metadata/example/example_simple.xml }}} If there are no syntax errors and xml document conforms to the schema, then it will result in no errors. In case of any errors, check the error[[BR]] messages, edit the document, and repeat steps 5-6 till no errors. 7. Now, we will run the regression executable. Type (assuming you are in $REGRESSION_HOME) {{{ $ ./bin/runRegression -xml $REGRESSION_HOME/data/metadata/example/example_simple.xml }}} Based on the configuration defined in the 'example_simple.xml', the model will compile and run. 8. Once the test is finished, the test results will be placed in respective timestamped directory within the workspace [[BR]] which was created during the installation. See 'Understanding Test Results' below for more information. == Understanding the Directory Structure == It is important to understand the directory structure before you proceed. It provides a quick overview of where, what and how the data and test results are located. * '''/bin:''' Regression test executables. * '''/lib:''' Required modules and libraries. It contains the properties file [wiki:RegressionProps regression.props]. * '''/log:''' Test run log 'regression.log'. It logs the build and run activities for the model. * '''/data:''' Data and files in support of running the tests. * '''/data/Master:''' Master {{{NetCDF}}} output data files for various model applications (e.g. {{{SED_TOY, SHOREFACE}}} etc.). These datasets are used in [http://nco.sourceforge.net/ NCO-tools] (e.g. ncdiff) tests.[[BR]] Here, the master file refers to a ''true'' data file against which a user would compare the model output. User must populate this directory with their master data files before proceeding[[BR]]to run any tests. 'ncdiff' would result in error without these files. The files could be of type 'history','averages','diagnostics' or 'restart'. This directory needs to be populated only[[BR]]in case you are running any nco-tools tests. * '''/data/metadata:''' Contains XML files for environment setup. Use 'regressionCSTM.xml' as a guide to create new xml configuration files. * '''/data/metadata/schema:''' Schema for XML document validation. Please do not edit files in this directory. * '''/data/metadata/example:''' Example XML files to be used for environment setup. User should edit these files before running the regression application. * '''/test-run/:''' Test results. Each time test is run, it creates a time-stamped directory in '/test-run/' where it stores all test results for each model application.[[BR]] The 'STDERR' and 'STDOUT' from model compilation are also saved in this directory. == Understanding Results == Test results, which are stored in '''/test-run////''' may contain following data files. * '''ocean_.nc:''' Model output data files. '''' can be ''his(history)'', ''dia(diagnostics)'', ''rst(restart)'', ''avg(averages)'' etc. * '''output_.out:''' STDOUT from model run. '''' can be e.g.'SED_TOY', 'SHOREFACE', etc. * '''build.err:''' STDERR from building/compiling ROMS model. * '''build.out:''' STDOUT from building/compiling ROMS model. * '''ncdiff_.nc:''' [http://nco.sourceforge.net/ NCO-tools] 'ncdiff' output. '''' can be 'history', 'average', etc. * '''ncks.nc:''' [http://nco.sourceforge.net/ NCO-tools] 'ncks' output. '''' can be 'history', 'average', etc. * '''roms_pbs.sh:''' PBS script auto-generated for the run. == Understanding Logs == Log 'regression.log' contains the time line of test execution. It reports both success and failure of test. Example log shown below: {{{ >>>> New Log: Tue Jul 15 15:22:48 2008 ROMS PATH: /home/skbhate/cstm/current_roms/roms3.2_070808 [SUCCESS][Tue Jul 15 15:22:48 2008][XML document data/metadata/example/example_complex.xml parsed succesfully.] Application: WAKE:wake_1:ifort. [ERROR][Tue Jul 15 15:22:48 2008][./bin/runRegression,][ROMS WAKE:wake_1:ifort build failed. Check 'build.err' for more info.] >>>> New Log: Tue Jul 15 16:18:55 2008 ROMS PATH: /home/skbhate/cstm/current_roms/roms3.2_070808 [SUCCESS][Tue Jul 15 16:18:55 2008][XML document data/metadata/example/example_complex2.xml parsed succesfully.] Application: UPWELLING:upwelling_1:ifort. [SUCCESS][Tue Jul 15 16:18:55 2008][Project directory does not exist: . Setting MY_PROJECT_DIR to ROMS ROOT DIR.] [SUCCESS][Tue Jul 15 16:27:17 2008][Compiled application instance UPWELLING:upwelling_1:ifort successfully.] [SUCCESS][Tue Jul 15 16:27:17 2008][Created PBS script: /home/skbhate/cstm/current_roms/roms3.2_070808/roms_pbs.sh] [SUCCESS][Tue Jul 15 16:27:17 2008][Starting ROMS-SED MPI version using PBS script /home/skbhate/cstm/current_roms/roms3.2_070808/roms_pbs.sh] [SUCCESS][Tue Jul 15 16:27:17 2008][PBS JOB ID: 14467.pikmin.cl.whoi.edu] [SUCCESS][Tue Jul 15 16:39:17 2008][PBS job 14467.pikmin.cl.whoi.edu finished. Check output.] [SUCCESS][Tue Jul 15 16:39:17 2008][ROMS run successful. History or Restart file created.] Running NCO tools tests....... File type: his [SUCCESS][Tue Jul 15 16:39:18 2008][Created ncdiff output.] }}} == Things to Remember == Please make appropriate changes or updates based on the following points before you proceed to run the tests. * Do not edit/Customize variables in [wiki:RegressionProps regression.props] file unless it is mentioned in the file. * Make sure you run 'validation tool' on your xml document frequently while editing the xml document. For example: {{{ ./bin/validate $REGRESION_HOME/data/metadata/example/example_simple.xml }}} * You can always send an email to [mailto:skbhate@ngi.msstate.edu skbhate@ngi.msstate.edu] in case of any problems/issues with testing package.