The test suite is designed to ensure stability of Org mode code base as it evolves, being modified by numerous contributors. The tests are usually designed aiming to ensure the expected Org mode behavior. Thus, we should not add the tests that just mechanically check what the current implementation does. In particular, we should avoid (or, at least, explicitly mark) the tests that simply check for some arbitrary behavior related to the details of the implementation; not to the original design goals. The only exception is when users get used to certain ways the code behaves by muscle memory.

The only dependency is ERT the Emacs testing library which ships with Emacs24. If you are running an older version of Emacs and don’t already have ERT installed it can be installed from its old git repository.

The simplest way to run the Org mode test suite is from the command line with the following invocation. Note that the paths below are relative to the base of the Org mode directory.

Also note that many of the current tests uses babel evaluation…

# For Emacs earlier than 24, add -L /path/to/ert
emacs -Q --batch \
      -L lisp/ -L testing/ -L testing/lisp -l lisp/org.el \
      -l lisp/org-id.el -l testing/org-test.el \
      --eval "(progn (org-reload) (setq org-confirm-babel-evaluate nil) \
      (org-babel-do-load-languages 'org-babel-load-languages \
      '((emacs-lisp . t) (shell . t) (org . t))))" \
      -f org-test-run-batch-tests

The options in the above command are explained below.

Target test can be used to trigger a test run. The tests start after cleaning up and recompilation.

make test

See ../mk/default.mk for details.

The ‘dirty’ targets are for recompiling without cleaning and rebuilding everything. This usually speeds up the recompilation considerably. Note that this speed up comes to the price of possibly weird errors due to the unclean build.

The dirty target for testing is called test-dirty.

make test-dirty

Variable BTEST_RE can be set to limit the tests which are performed. BTEST_RE is interpreted as regexp.

Example:

make BTEST_RE='test-.*-inlinetask' test-dirty

yields

...
selected tests: test-.*-inlinetask
Running 2 tests (2017-12-28 15:04:45+0100)
   passed  1/2  test-org-export/handle-inlinetasks
   passed  2/2  test-org-inlinetask/goto-end

Ran 2 tests, 2 results as expected (2017-12-28 15:04:45+0100)
...

Set the TEST_NO_AUTOCLEAN variable to leave the test directory around for inspection.

make TEST_NO_AUTOCLEAN=1 test-dirty

To run the Org mode test suite from a current Emacs instance simply load and run the test suite with the following commands.

1. First load the test suite.

   (add-to-list 'load-path (file-name-directory here))
   (require 'org-test)
       

2. Load required Babel languages

   (org-babel-do-load-languages
    'org-babel-load-languages
    (and (mapc (lambda (lang)
                 (add-to-list 'org-babel-load-languages (cons lang t)))
               '(emacs-lisp shell org))
         org-babel-load-languages))
       

3. Then run the test suite. Babel evaluation confirmation is disabled and C-c C-c is enabled while running the tests.

   (let (org-babel-no-eval-on-ctrl-c-ctrl-c
         org-confirm-babel-evaluate)
     (org-test-run-all-tests))
       

   When a test fails, run it interactively and investigate the problem in the ERT results buffer.

   To run one test: Use this as a demo example of a failing test

   (ert-deftest test-demo/example-of-a-failing-test ()
     (should (string= "AB" (concat "A" "B")))
     (should (string= "CD" (concat "C" "C" "D"))))
       

   or evaluate the ert-deftest form of the test you want to run. Then M-x ert RET test-demo/example-of-a-failing-test RET. When not visible yet switch to the ERT results buffer named *ert*. When a test failed the ERT results buffer shows the details of the first should that failed. See <a href=”info:ert::Running Tests Interactively”>info:ert::Running Tests Interactively for how to re-run, start the debugger etc.

   To run several tests: M-x ert RET "<your regexp here>" RET.

   To run all tests of a single test file: M-x ert-delete-all-tests RET and confirm. M-x load-file RET testing/lisp/<file>.el RET M-x ert RET t RET.

   Consider to set

   (setq pp-escape-newlines nil)
       

   before running the test when looking at should in the ERT results buffer. Especially when using l to look at passed test results and possibly missing an appropriate setting of pp-escape-newlines made only temporarily for the running time of the test as e. g. tests using org-test-table-target-expect-tblfm do.

• If the variable org-babel-no-eval-on-ctrl-c-ctrl-c is non-nil then it will result in some test failure, as there are tests which rely on this behavior.