unit testing in R

there are a couple of different unit testing packages for R. these include:

  • RUnit
  • svUnit
  • testthat

this page describes RUnit and svUnit.

RUnit and R CMD check

RUnit: R Unit test framework provides R functions implementing a standard Unit Testing framework, with additional code inspection and report generation tools. This page will not describe usage of RUnit but how to fuse it with R CMD check utility. Additional information on RUnit usage is available in help pages, vignette and on SourceForge site.

Aim

RUnit can be used as additional testing tool in package development and maintenance. It is very convenient if unit tests can be performed:

  1. during R CMD check phase,
  2. at anytime with only some keystrokes,
  3. additionally R CMD check should produce error if unit tests fail, and
  4. all this should be portable

The approach presented here works for Linux and Windows. (This scheme should work on Mac OS X, but it would be nice to get confirmation – any Mac OS X testers around?)

Implementation

Note that package gdata on CRAN is an actually available implementation.

(Note that from R 2.6.0 on, it should be possible to simplify this approach by making the directory unitTests a subdirectory of tests – this is not possible with earlier versions because they do not copy subdirectories of tests to the test installation. However this will not install the directory, hence the current proposal is often preferable.)

Say, you have a package PKG in a directory (aka “folder” or “map”) PKG with the following structure:

 
PKG
 |- R
 |- ...
 |- inst
 |   |- doc
 |   `- unitTests
 |- ...
 `- tests

Then the following files (file content is shown bellow) are used:

PKG/tests/doRUnit.R
PKG/inst/unitTests/Makefile
PKG/inst/unitTests/runit.*.R

If you launch in a terminal:

cd /some/path/PKG/..
R CMD check PKG 

doRUnit.R in map tests is sourced during R CMD check phase and all unit tests in PKG/inst/unitTests are issued, which fulfils aim 1. Output of unit testing can be seen in either:

PKG.Rcheck/tests/doRUnit.Rout
PKG.Rcheck/PKG/unitTests/report.txt
PKG.Rcheck/PKG/unitTests/report.html

R CMD check will return error and stop when some unit tests fails (aim 3). Above outputs should be checked to see the results of unit testing.

If you only want to launch unit tests (aim 2) use the following to test them against currently installed code:

cd /some/path/PKG/inst/unitTests
make test

or the following if the package should be first re-installed:

cd /some/path/PKG/inst/unitTests
make

In case you want any specific R version, say R-devel, you can also pass variable to make:

cd /some/path/PKG/inst/unitTests
make R=R-devel

PKG/tests/doRUnit.R

Content of doRUnit.R file. Note that you have to change content of pkg variable at the top of the script i.e. pkg <- “PKG” to pkg <- “myPackageName”. It is not possible or at least I do not know how to get package name in general - it is possible to get it if package folder is the same as package name, but not otherwise, at least for running unit tests during R CMD check.

## unit tests will not be done if RUnit is not available
if(require("RUnit", quietly=TRUE)) {
 
  ## --- Setup ---
 
  pkg <- "PKG" # <-- Change to package name!
  if(Sys.getenv("RCMDCHECK") == "FALSE") {
    ## Path to unit tests for standalone running under Makefile (not R CMD check)
    ## PKG/tests/../inst/unitTests
    path <- file.path(getwd(), "..", "inst", "unitTests")
  } else {
    ## Path to unit tests for R CMD check
    ## PKG.Rcheck/tests/../PKG/unitTests
    path <- system.file(package=pkg, "unitTests")
  }
  cat("\nRunning unit tests\n")
  print(list(pkg=pkg, getwd=getwd(), pathToUnitTests=path))
 
  library(package=pkg, character.only=TRUE)
 
  ## If desired, load the name space to allow testing of private functions
  ## if (is.element(pkg, loadedNamespaces()))
  ##     attach(loadNamespace(pkg), name=paste("namespace", pkg, sep=":"), pos=3)
  ##
  ## or simply call PKG:::myPrivateFunction() in tests
 
  ## --- Testing ---
 
  ## Define tests
  testSuite <- defineTestSuite(name=paste(pkg, "unit testing"),
                                          dirs=path)
  ## Run
  tests <- runTestSuite(testSuite)
 
  ## Default report name
  pathReport <- file.path(path, "report")
 
  ## Report to stdout and text files
  cat("------------------- UNIT TEST SUMMARY ---------------------\n\n")
  printTextProtocol(tests, showDetails=FALSE)
  printTextProtocol(tests, showDetails=FALSE,
                    fileName=paste(pathReport, "Summary.txt", sep=""))
  printTextProtocol(tests, showDetails=TRUE,
                    fileName=paste(pathReport, ".txt", sep=""))
 
  ## Report to HTML file
  printHTMLProtocol(tests, fileName=paste(pathReport, ".html", sep=""))
 
  ## Return stop() to cause R CMD check stop in case of
  ##  - failures i.e. FALSE to unit tests or
  ##  - errors i.e. R errors
  tmp <- getErrors(tests)
  if(tmp$nFail > 0 | tmp$nErr > 0) {
    stop(paste("\n\nunit testing failed (#test failures: ", tmp$nFail,
               ", #R errors: ",  tmp$nErr, ")\n\n", sep=""))
  }
} else {
  warning("cannot run unit tests -- package RUnit is not available")
}

PKG/inst/unitTests/Makefile

Here is the content of Makefile file. It is general and should work out of the box.

TOP=../..
PKG=${shell cd ${TOP};pwd}
SUITE=doRUnit.R
R=R

all: inst test

inst: # Install package
        cd ${TOP}/..;\
        ${R} CMD INSTALL ${PKG}

test: # Run unit tests
        export RCMDCHECK=FALSE;\
        cd ${TOP}/tests;\
        ${R} --vanilla --slave < ${SUITE}

PKG/inst/unitTests/runit.*.R

Content of runit.*.R files is completely free, well it certainly should conform to RUnit. Here is just a simple example that can be used as a template:

### --- Test setup ---
 
if(FALSE) {
  ## Not really needed, but can be handy when writing tests
  library("RUnit")
  library("changeToNameOfThePackage")
}
 
a <- 1
b <- 2
 
### --- Test functions ---
 
test.simple <- function()
{
  checkTrue(a < b)
}

Another approach using svUnit

Philippe Grosjean 2010/09/27

svUnit uses a slightly different approach to integrate unit tests in R CMD check, and a radically different one in interactive sessions. svUnit is available on CRAN and on R-Forge.

With svUnit you do batch run of your test units (fully compatible with RUnit, and located at the same place in the inst/unitTests/ subdirectory of the package sources). For instance, for a package named mypkg you only need a little script like this one to run all test and output results in a text file:

#! /usr/bin/Rscript --vanilla --default-packages=utils,stats,methods,svUnit
require(svUnit)  # Needed if run from R CMD BATCH
require(mypkg)  # Needed if run from R CMD BATCH
unlink("mypkgTest.txt")  # Make sure we generate a new report
mypkgSuite <- svSuiteList("mypkg", excludeList = NULL)  # List all our test suites
runTest(mypkgSuite, name = "svUnit")  # Run them...
protocol(Log(), type = "text", file = "mypkgTest.txt")  # ... and write report

The report will be located in pkg.Check/tests or next to the script, depending on how you started the check.

svUnit is fully compatible with assertion functions (checkTrue() and friends) of RUnit. It is also fully compatible with the runit*.R files, including use of .setUp() and .tearDown() function. It supports the same organisation of the unit tests for R packages (unitTests subdirectory with runit*.R files). So, the same unit tests code should be usable with both RUnit (as for version 0.4-17, at least) and svUnit, but look hereunder for an even nicer integration with R CMD check than just batch run of the tests!

Integration with R CMD check

Integration with R CMD check is done here via examples instead of tests. Several reasons for that:

  1. Running example in R CMD check does not need make, while running tests requires it. There is, at least, one situation where this is an advantage: on Mac OS X, make is not installed by default, and it is not necessary for compiling/checking package that contain pure R code. The solution using /tests impose to install make, while a solution using examples does not.
  2. Execution of examples offers much more control on when they are executed, than /tests. Thanks to the \dontrun{}, \donttest{}, \dontshow{} instructions, it is possible to define tests that are run either during R CMD check, during an interactive session, or both.
  3. Examples are also executable in an interactive session, at least, much more easily than /tests. See, for instance, ?unitTests.svUnit in the svUnit package.
  4. On the contrary to RUnit, which stops running tests in a test function at the first failure or error, all tests of the test function are run in svUnit. This give a more exhaustive report, especially in cases where one of the first tests fails!
  5. Integration mechanism with R CMD check in svUnit is compatible with all platforms where R can run. The mechanism proposed here above for RUnit still needs some adaptation for, let’s say, Windows.
  6. Finally, tests made during R CMD check with svUnit are as less invasive as possible. In case all tests succeed, there is no extra output besides * checking examples ... OK. Of course, a detailed report on the tests that failed and those that raised on error is printed in case of problem(s), and the R CMD check mechanism is interrupted (like with the RUnit integration solution proposed at the beginning of this page).

To see a working example of this implementation, run R CMD check on the svUnit package itself, or run example(unitTests.svUnit) in an interactive session, and examine code in /unitTests and its subdirectories.

Bonuses in svUnit

There are a couple of tools in svUnit that ease its use in interactive mode. The first one is the ability to define a test function simply in memory, outside of the rigid runit*.R files on disk. Here is a simple example:

## This is a very simple function
foo <- function(x, y = 2) return(x * y)
 
## Create and associate test cases for this function
test(foo) <- function () {
    checkEqualsNumeric(4, foo(2))
    checkEqualsNumeric(6, foo(2, 3))
    checkTrue(is.test(foo))
    checkTrue(is.test(test(foo)))
    checkIdentical(attr(foo, "test"), test(foo))
    checkException(foo(2, "aa"))
    checkException(foo("bb"))
}

Once this is done, the test cases are transported with the object to be tested. The tests are run simply with:

res <- runTest(foo)
res   # Print results, or ...
summary(res)  # Make a condensed report

Of course, at any time, you can transfer tests associated with objects into runit*.R files on disk for full compatibility with RUnit. This is as simple as that:

unit <- makeUnit(foo)
file.show(unit, delete.file = TRUE)  # In 'real' situation, you won't delete it!

Also with svUnit, it is much easier to list available test functions and test units, and to create test suites with them. The svSuiteList() makes all the work for you (see first svUnit example too):

## Exclusion list is used (regular expression filtering!). It contains:
(oex <- getOption("svUnit.excludeList"))
## Clear it, and relist available test units
options(svUnit.excludeList = NULL)
svSuiteList()
 
## Create the test unit file for all objects with tests in .GlobalEnv
myunit <- makeUnit(svSuiteList(), name = "AllTests")
file.show(myunit, delete.file = TRUE)
 
## Run all the tests currently available
(runTest(svSuiteList(), name = "AllTests"))
summary(Log())
## End(Not run)
 
## Restore previous exclusion list, and clean up the environment
options(svUnit.excludeList = oex)

As one can see, svUnit is mainly designed to integrate unit tests in interactive sessions, although it performs as well as RUnit to run tests in batch mode or through R CMD check. svUnit goes even a lot further in interaction: together with SciViews-K and SciViews-K Unit, it offers a GUI with an interactive tree reporting of the test runs. See SciViews-K for more on this. Here is a screenshot of this GUI on top of svUnit (the ‘R Unit’ tab at the right):

 
developers/runit.txt · Last modified: 2010/10/01 by mfrasca
 
Recent changes RSS feed R Wiki powered by Driven by DokuWiki and optimized for Firefox Creative Commons License