Adam Retter

adam@evolvedbinary.com
 

XML Prague
@ University of Economics, Prague
2026-06-05


@adamretter

XTH
(X Test Harness)

About Me

  • Director of Evolved Binary

    • UK - Software, Consultancy, Training, and R&D

  • Co-founder and Co-owner of eXist Solutions

    • Germany - TEI Publisher software

  • Software Engineer / Prolific Open Source contributor

    • Elemental (NoSQL/XMLDB) - Founder

    • eXist-db - 20 Years as Core contributor (last 9 as main dev.)

    • RocksDB - 8 Years as RocksJava main developer

  • W3C XQuery Working Group - Invited Expert

  • Founder of EXQuery, and creator of RESTXQ

  • Enjoys Research and Development, and Snowboarding

www.adamretter.org.uk

What is XTH?

  • X Test Harness

    • X = One of the XQuery or XSLT WG Test Suites

    • X = Any XQuery or XSLT Processor

  • Research Paper

  • Freely Available Software Implementation

WG Test Suites

  • XQuery and XSLT Working Groups

  • Published > 25 Recommended Standards

    • Accompanied by 10 Test Suites

  • The Test Suites are expressed in XML using a variety of Grammars

What constitutes a Test Suite?

  • Test Suites consists of Test Sets and/or Test Cases

  • Each Test Case is a description of:

    • How to setup a Computation Environment

    • Dependencies required for Test Case

    • A computation or action(s) to be performed

    • One or more assertions about either:

      • Results of the computation

      • State of the computation environment (post computation)

  • Each Test Suite requires a Test Harness to execute it

    • Historically: per-vendor, per-product, per-standard

Where can we get a Test Harness?

“Implementers are expected to write their own test harness that implements the following tasks: [a] Read test cases from the catalog, apply customization if applicable [...], [b] Execute tests, using source files specified in the catalog, [c] Use appropriate comparator to match result, [and d,] Produce categorization of test result (pass, fail etc., see below. Ideally, the test harness produces an XML file containing all test results in the format shown below, that can be sent to the working group.”

XQTS 1.0.3: Guidelines for Running the XML Query Test Suite.html

Do we really have to write a Test Harness?

“As there is no de jure or de facto API for implementations of XQuery, we are not able to provide a test harness to execute these tests. You will have to write your own test harnesses.”

XQUTS 1.0.1: release_notes.html

Existing Test Harnesses

  • 34 Vendors executed test suites and reported their results

  • Only 6 Vendors published their Test Harness Source Code:

    • BaseX (2x)

    • Elemental (2x)

    • eXist-db (3x)

    • RumbleDB

    • Saxon (2x)

    • Zorba

Existing Test Harnesses Had/Have Issues

  • Code history reveals bugs

  • Revisions of published results show there were likely:

    • False Positives

    • False Negatives

  • Vendors took very different approaches

  • Some vendors developed multiple Test Harnesses

Building a Test Harness is
Hard!

XTH Provides a Framework
for
Building Test Harnesses

What Problems does XTH Solve?

  • Vendors no longer:

    • Duplicate Effort

    • Duplicate Code

    • Struggle with Correctness

  • Developers are motivated:

    • Easier to prototype and test new featurs/fixes

  • Users can more easily:

    • Identify and choose standards compliant products

  • Researches can now:

    • Experiment and reproduce results

  • Working Groups gain:

    • Faster feedback on implementation reports from vendors

XTH Research Analysis of Test Suites

  • Categorises the 10 Test Suites from the WGs as 3 model types:

  1. XSLT Test Suite 04

  2. XQTS like:

    1. XQTS 1.0.3

    2. XQUTS 1.0.1

    3. XQFTTS 1.04

    4. XSLT 2.0 Test Suite?

  1. QT3 like:

    1. QT3 1.0

    2. XSLT 3.0 Test Suite

    3. QT3 Tests

    4. QT4 Tests

    5. XSLT 4.0 Tests

XTH (Software) Provides:

  • "Implementers are expected to write their own test harness that implements the following tasks:"

    • "[a] Read test cases from the catalog, apply customization if applicable" ✅

    • "[c] Use appropriate comparator to match result" ✅

    • "[and d,] Produce categorization of test result (pass, fail etc." ✅

  • Abstract General Model that can hold all 3 Test Suite model types

  • 1 Parser for Each Model Type (or variant)

  • Multi-threaded scheduler for executing Test Cases

  • Reporters for each format of Test Suite results

The XTH Software

  • Implemented in Java 25

  • Minimal External Dependencies

  • Java Service Loader - Easily add your own Modules

  • Command Line Interface / Embed as a Library

  • Engineered for:

    1. Correctness

    2. Extensibility

    3. Speed - Structured Concurrency and Virtual Threads

  • Available on GitHub - https://github.com/evolvedbinary/xth

Architecture of the XTH Software

XTH Modular Software Architecture

Anatomy of an XTH Parser

public interface TestSuiteParser {

    /**
     * Get the name of the parser.
     *
     * Should be a short but humane name.
     *
     * @return the name of the parser.
     */
    String getParserName();

    /**
     * Parses the Test Suite.
     *
     * @throws IOException if an I/O error occurs whilst reading the test suite files.
     * @throws ParserException if an error occurs during parsing of the test suite files.
     */
    void parse() throws IOException, ParserException;
    
    
    // NOTE(AR) you can just extend AbstractTestSuiteParser to handle this for you!
    void addEventListener(ParserEventListener eventListener);
    void removeEventListener(ParserEventListener eventListener);
}

Anatomy of an XTH Connector 1/2

@ThreadSafe
public interface Connector<T extends TestCaseExecutionContext> {

    String getConnectorName();
    
    String getImplementationName() throws ConnectorException;
    
    String getImplementationVersion() throws ConnectorException;

    /**
     * Checks whether the implementation supports the required dependencies.
     *
     * @param dependencies the required dependencies to check for.
     *
     * @return An empty list if all dependencies are supported, otherwise a list of the unsupported dependencies
     */
    List<Dependency<?>> supports(final List<Dependency<?>> dependencies);


	// ... continues on next slide...

Anatomy of an XTH Connector 2/2


	/**
     * Performs any upfront preparation for a test case before it is executed.
     *
     * For example, compilation could be performed at this stage.
     *
     * @param testSet the Test Set.
     * @param testCase the Test Case.
     *
     * @return An Execution Context for the Test Case.
     *
     * @throws ConnectorException if an error that is outside the bound of the test case occurs.
     */
    T prepareTestCaseForExecution(TestSet testSet, TestCase testCase) throws ConnectorException;

    /**
     * Execute a test case.
     *
     * @param testCaseExecutionContext the execution context produced by {@link #prepareTestCaseForExecution(TestSet, TestCase)}.
     *
     * @return The result of executing the test case.
     */
    TestCaseResult executeTestCase(T testCaseExecutionContext);
}

Anatomy of an XTH Reporter

public interface Reporter {

    /**
     * Get the name of the reporter.
     *
     * Should be a short but humane name.
     *
     * @return the name of the reporter.
     */
    default String getReporterName() {
        return getClass().getSimpleName();
    }

    /**
     * Open a new listener for the reporter.
     *
     * @return a listener that listens for test results.
     *
     * @throws ReporterException if an error occurs when opening a new listener.
     */
    TestResultsListener open() throws ReporterException;
}

Anatomy of an XTH Reporter - Listener

@ThreadSafe
public interface TestResultsListener extends AutoCloseable {


    void testSuiteStarted(TestSuite testSuite, Instant timestamp);
    void testSuiteFinished(TestSuite testSuite, Instant timestamp);

    void testSetExcluded(TestSuite testSuite, UUID testSetId, Path testSetFile, String testSetName, Instant timestamp);
    void testSetSkipped(TestSuite testSuite, UUID testSetId, Path testSetFile, String testSetName, Instant timestamp);

    void testSetStarted(TestSuite testSuite, TestSet testSet, Instant timestamp);
    void testSetFinished(TestSuite testSuite, TestSet testSet, Instant timestamp);

    void testCaseExcluded(TestSuite testSuite, TestSet testSet, UUID testCaseId, String testCaseName, Instant timestamp);
    void testCaseSkipped(TestSuite testSuite, TestSet testSet, UUID testCaseId, String testCaseName, Instant timestamp);

    void testCaseStarted(TestSuite testSuite, TestSet testSet, TestCase testCase, Instant timestamp);
    void testCaseFinished(TestSuite testSuite, TestSet testSet, TestCase testCase, Instant timestamp, TestCaseResult testCaseResult);

    @Override
    void close();
}

DEMO

Conclusions

  • Reduced work for vendors

  • Empowered Users, Developers, and Researchers

  • Software still needs work!

  • We have:

    • CLI

    • QT3 Tests Parser

    • Saxon Connector

    • Console Reporter

    • OTR Reporter - Events XML and HTML Report

Future Work

  • We want - collaboration

    • Linus's Law - "given enough eyeballs, all bugs are shallow"

  • Elemental Connector(s) are under development

  • XML Coniguration for XTH - Shareable Reproducible Runs

  • Publish a matrix of results online with history and trends

  • Reporter to create WG Test Suite submission XML output format

  • Donate to neutral organisation - EXPath / EXQuery / QT4 WG?

Thanks
and
Questions