BoxLang 🚀 A New JVM Dynamic Language Learn More...

TestBox

v7.2.2+23 Testing

Total Downloads Latest Stable Version Apache2 License

Copyright Since 2005 TestBox by Luis Majano and Ortus Solutions, Corp
www.testbox.run | www.ortussolutions.com

TestBox - BDD/TDD Testing Framework for BoxLang & CFML

Professional BDD (Behavior-Driven Development) and TDD (Test-Driven Development) testing framework for BoxLang and CFML applications. TestBox provides a comprehensive testing ecosystem with integrated mocking capabilities, multiple output formats, and both CLI and web-based test runners.

🌟 Features

  • Dual Testing Approaches: Full support for both BDD (describe(), it()) and xUnit (setup(), test*()) styles
  • BoxLang First: Native support for BoxLang with dedicated CLI runner, plus full CFML compatibility
  • Integrated Mocking: Built-in MockBox framework with spies, stubs, and verification capabilities
  • Rich Data Generation: CBMockData integration for realistic test data (names, addresses, dates)
  • Multiple Reporters: 15+ output formats including HTML, JSON, XML, TAP, JUnit, and Console
  • CLI & Web Runners: Execute tests via CommandBox CLI or web-based interfaces
  • File Watching: Automatic test re-execution on code changes during development
  • Code Coverage: Built-in coverage analysis and reporting
  • Flexible Discovery: Automatic test bundle detection with customizable patterns
  • Thread-Safe: Concurrent test execution with proper isolation
  • Browser Testing: Drive a real browser from any spec with the @browser annotation and bx-playwright (BoxLang)
  • Run Failed: Rerun only what failed, from the CLI (--failed) or the web runners

🎬 Browser Testing

Drive a real browser (Chromium, Firefox or WebKit) right from your specs. Add the @browser annotation (or @browserProfile / @baseURL) to any spec, or extend BrowserSpec, call browse(), and assert on what your users actually see. Powered by bx-playwright and BoxLang.

A BrowserSpec signing in to a demo shop and checking the dashboard

Deploy bx-playwright to the same BoxLang web server that runs TestBox. TestBox installs the profile's browser through that deployed module the first time it is used; a separate OS-level BoxLang installation is not required. The first run needs network access and may take longer. Set @browserAutoInstall( false ) to require a pre-installed browser, or call ensureBrowserInstalled() explicitly from beforeAll(). On Linux, install required system libraries in the host or container; TestBox only downloads browser files.

@baseURL( "http://localhost:8090" )
@browserProfile( "ci" )
class extends="testbox.system.BaseSpec" {

	function run() {
		describe( "Ortus Shop", () => {
			it( "signs in and shows the dashboard", () => {
				browse( ( page ) => {
					page.visit( "/" )
						.fill( "Email", "[email protected]" )
						.fill( "Password", "secret" )
						.click( "Sign in" )
					expect( page ).toHavePath( "/dashboard.html" )
					expect( page ).toSee( "Welcome, Luis" )
				} )
			} )
		} )
	}

}

When a browser spec fails, TestBox attaches a screenshot, a Playwright trace and a video right on the failing spec in your report:

A failing browser spec with its screenshot, trace and video attached

Open the trace with bxplaywright show-trace and step through every action, the DOM, the console and the network:

The Playwright trace viewer showing the failed assertion

Fix it, then rerun only what failed with --failed or the Run Failed button. Learn more in the Browser Testing guide.

💻 Requirements

  • BoxLang: 1.0+
  • Lucee: 6.0+
  • Adobe ColdFusion: 2023+

⚡ Quick Start

1. Installation

# Install TestBox and CLI tools via CommandBox
box install testbox testbox-cli

# Or install bleeding edge
box install testbox@be testbox-cli

2. Create Your First Test

// tests/specs/UserServiceTest.cfc
class extends="testbox.system.BaseSpec" {

    function run() {
        describe( "UserService", () => {
            beforeEach( () => {
                userService = new models.UserService()
            } )

            it( "should create a new user", () => {
                var user = userService.createUser( "[email protected]", "John Doe" )
                expect( user.getEmail() ).toBe( "[email protected]" )
                expect( user.getName() ).toBe( "John Doe" )
            } )
        } )
    }
}

🎭 MockBox Integration & Test Data

Advanced Mocking with MockBox

class extends="testbox.system.BaseSpec" {

    function run() {
        describe( "Payment Service", () => {
            beforeEach( () => {
                // Create mocks and spies
                mockGateway = createMock( "services.PaymentGateway" )
                mockLogger = createEmptyMock( "cblogger.models.Logger" )

                // Setup mock behavior
                mockGateway.$( "processPayment" ).$results( {
                    success: true,
                    transactionId: "TXN-12345"
                } )

                paymentService = new models.PaymentService(
                    gateway = mockGateway,
                    logger = mockLogger
                )
            } )

            it( "should process payment and log success", () => {
                var result = paymentService.charge( 100.00, "USD" )

                // Verify method calls
                expect( mockGateway.$times( 1, "processPayment" ) ).toBeTrue()
                expect( mockLogger.$times( 1, "info" ) ).toBeTrue()

                // Verify results
                expect( result.success ).toBeTrue()
                expect( result.transactionId ).toBe( "TXN-12345" )
            } )
        } )
    }
}

Realistic Test Data with CBMockData

class extends="testbox.system.BaseSpec" {

    function run() {
        describe("User Profile Tests", () => {
            it("should handle various user data scenarios", () => {
                // Generate realistic test data
                var testUsers = mockData(
					firstName = "fname",
					lastName = "lname",
					email = "email",
					age = "age",
					address = {
						street = "streetaddress",
						city = "city",
						state = "state",
						zipCode = "zipcode"
					},
					registrationDate = "datetime"
				)

                // Test with realistic data
                for ( var user in testUsers ) {
                    var profile = userService.createProfile( user )
                    expect( profile.isValid() ).toBeTrue()
                    expect( profile.getEmail() ).toMatch( "^[\w\.-]+@[\w\.-]+\.[A-Za-z]{2,}$" )
                }
            })
        })
    }
}

3. Run Your Tests

# Via CommandBox CLI
box testbox run

# Via BoxLang CLI Runner (fastest execution)
./testbox/run                           # Run default tests.specs
./testbox/run --directory=my.tests      # Specific directory
./testbox/run --bundles=my.bundle       # Specific bundles
./testbox/run --reporter=json           # Custom reporter

# Via web browser
# Navigate to: http://localhost/testbox/system/runners/HTMLRunner.cfm

🧪 Testing Approaches

BDD Style (Behavior-Driven Development)

class extends="testbox.system.BaseSpec" {

    function run() {
        describe( "User Registration", () => {
            beforeEach(() => {
                userService = createMock("models.UserService")
                variables.sut = new handlers.Users()
            })

            describe("When registering a new user", () => {
                it("should validate email format", () => {
                    expect(() => {
                        userService.register("invalid-email", "password")
                    }).toThrow("ValidationException")
                })

                it("should create user with valid data", () => {
                    var result = userService.register("[email protected]", "securePass")
                    expect(result.success).toBeTrue()
                    expect(result.user.email).toBe("[email protected]")
                })
            })
        })
    }
}

xUnit Style (Traditional Unit Testing)

class extends="testbox.system.BaseSpec" {

    function setup() {
        // Runs before each test
        userService = new models.UserService()
        testData = {
            email: "[email protected]",
            name: "Test User"
        }
    }

    function testUserCreation() {
        var user = userService.createUser( testData.email, testData.name )
        $assert.isEqual( testData.email, user.getEmail() )
        $assert.isEqual( testData.name, user.getName() )
    }

    function testEmailValidation() {
        $assert.throws( () => {
            userService.createUser( "invalid-email", "Test User" )
        }, "ValidationException" )
    }

    function tearDown() {
        // Cleanup after each test
        structDelete( variables, "userService" )
        structDelete( variables, "testData" )
}

⚙️ Configuration & CLI Usage

CLI Test Execution

# Run all tests with default settings
box testbox run

# Run specific test bundles
box testbox run bundles=tests.specs.UserServiceTest

# Run tests with custom reporter
box testbox run reporter=json

# Run tests with labels ( focused testing )
box testbox run labels=unit --excludes=integration

# Watch mode for continuous testing
box testbox watch

# Generate test templates
box testbox create bdd MyNewTest
box testbox create unit MyNewTest

Application Setup

Create an Application.bx|cfc in your test directory:

class {
	// Test application name
    this.name = "MyApp-Tests"

    // TestBox mappings
    this.mappings[ "/testbox" ] = expandPath( "/testbox" );
    this.mappings[ "/tests" ] = getDirectoryFromPath( getCurrentTemplatePath() );
    this.mappings[ "/models" ] = expandPath( "/models" );

}

Web Runner Configuration

Create tests/runner.bxm|cfm for web-based test execution:

<!DOCTYPE html>
<html>
<head>
    <title>My Application Test Suite</title>
</head>
<body>
    <bx:script>
        // Create TestBox instance
        testbox = new testbox.system.TestBox(
            options = {
                // Test bundle directories
                bundles = [
                    "tests.specs"
                ],
                // Directories to include/exclude
                directory = {
                    mapping = "tests.specs",
                    recurse = true
                },
                // Test labels
                labels = url.labels ?: "",
                excludes = url.excludes ?: "",
                // Reporter
                reporter = url.reporter ?: "simple",
                // Coverage settings
                coverage = {
                    enabled = true,
                    pathToCapture = expandPath("/models"),
                    whitelist = "*.cfc",
                    blacklist = "*Test*.cfc"
                }
            }
        )

        // Run tests and output results
        writeOutput( testbox.run() )
    </bx:script>
</body>
</html>

📄 License

Apache License, Version 2.0.

📋 Versioning

TestBox is maintained under the Semantic Versioning guidelines as much as possible.

Releases will be numbered with the following format:

<major>.<minor>.<patch>

And constructed with the following guidelines:

  • Breaking backward compatibility bumps the major (and resets the minor and patch)
  • New additions without breaking backward compatibility bumps the minor (and resets the patch)
  • Bug fixes and misc changes bumps the patch

📚 Documentation & Resources

Source Code

Bug Tracking

Community & Support

Official Documentation

Community and Support

Join us in our Ortus Community and become a valuable member of this project TestBox BDD. We are looking forward to hearing from you!!

Official Sites

📄 License

Apache License, Version 2.0. See LICENSE file for details.

About Ortus Solutions

TestBox is a professional open-source project by Ortus Solutions.

Copyright Since 2005 TestBox by Luis Majano and Ortus Solutions, Corp

www.testbox.run | www.ortussolutions.com

✝️ HONOR GOES TO GOD ABOVE ALL

Because of His grace, this project exists. If you don't like this, then don't read it, its not for you.

"Therefore being justified by faith, we have peace with God through our Lord Jesus Christ: By whom also we have access by faith into this grace wherein we stand, and rejoice in hope of the glory of God. And not only so, but we glory in tribulations also: knowing that tribulation worketh patience; And patience, experience; and experience, hope: And hope maketh not ashamed; because the love of God is shed abroad in our hearts by the Holy Ghost which is given unto us." Romans 5:5

🍞 THE DAILY BREAD

"I am the way, and the truth, and the life; no one comes to the Father, but by me (JESUS)" John 14:1-12

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

7.2.2 - 2026-10-16

Fixed

  • resolveRetries() and a handful of reporter and runner helpers no longer use the elvis operator on struct members that may be missing, which left the variable undefined on Adobe 2023 without null support (Variable SPECRETRIES is undefined). They use structKeyExists() instead.

7.2.1 - 2026-10-16

Fixed

  • The BoxLang runner starts a named application with session management for the run, named after the project's Application (this.name) or TestBoxRunner, so specs that use the application or session scopes run from the CLI. BoxLang only starts an application for an Application.bx next to the running script, which the runner never had.
  • The reporters and the status headers no longer call getPageContext() outside of a web request: with the bx-web-support module installed it is defined in the BoxLang CLI runner too, where it throws.
  • The HTML and Agent reports point a failure at the spec that called a ColdBox testing helper such as assertRouteIs(), not at coldbox/system/testing/BaseTestCase.cfc: frames under /coldbox/system/testing/ are treated like TestBox internals when choosing the origin, the code snippet and Open in editor.

7.2.0 - 2026-10-13

Added

  • TESTBOX-471 Ask AI on failures: copy a ready-made prompt, preview it, open it in ChatGPT or Claude, or copy it as JSON for a coding agent. It is on by default with a first-use notice and is controlled by the aiAssist, aiProviders, aiContextLines, aiStackFrames and aiPrompt reporter options.
  • TESTBOX-471 The urlParams reporter option sets request params such as editor and aiAssist when a report is produced from code and there is no url scope.
  • TESTBOX-471 build/vendor rebuilds the inlined front-end libraries with one command (box run-script assets:update) and CI keeps them honest.
  • TESTBOX-469 New AgentReporter (reporter=agent): a compact, token-efficient JSON reporter for AI agents and automation. It emits totals plus only failed/errored specs, with options detail, maxFailures, maxMessageLength, includeStack, stackDepth, includeSkipped and includeDebug.
  • Browser testing on BoxLang with the bx-playwright module, turned on by class annotations: any spec whose class, or a class it extends, has @browser, @browserProfile or @baseURL gets browse( callback, options ), browserAvailable(), ensureBrowserInstalled(), getBrowserSupport(), this.playwright() and the browser matchers, mixed in by the runner (public methods the spec declares itself are kept, private ones with the same name are replaced). @browser( false ) turns it off for a class whose parent turns it on. testbox.system.BrowserSpec is an optional base class that carries @browser. One browser per bundle, closed by the runner after the bundle even when an afterAll() throws (when a request aborts, the next run that opens a browser closes the one left behind), and the kept screenshots, trace and videos of a failed browse() attached to the spec. Specs skip with an install hint when bx-playwright is missing or the engine is not BoxLang. The logic lives in testbox.system.browser.BrowserSupport.
  • Browser matchers in testbox.system.browser.BrowserMatchers, registered for every browser bundle and usable from any BoxLang spec with addMatchers(): toHaveTitle(), toHaveURL(), toHavePath() and toSee() for pages, toHaveText(), toBeVisible(), toBeHidden(), toHaveCount() and toHaveValue() for locators, plus their not forms. They delegate to the retrying bx-playwright assertions, also when negated, and fail with the bx-playwright message.
  • attach( path, type, name ) in every spec to attach files to the running spec, kept in the new attachments array of the spec stats for passed and failed specs. The JSON report includes them, the Simple report links them, the JUnit and ANT JUnit reports add a <system-out> with one [[ATTACHMENT|path]] line per file, and the text, console and stream outputs list them under failed specs.
  • Spec retries: a retries argument on it(), fit() and xit(), a retries bundle annotation, a retries method annotation for xUnit tests, and a global retries runner option (BoxLang runner --retries=N). The spec value wins over the bundle annotation, which wins over the global option. A failing or erroring spec reruns its beforeEach(), body and afterEach() (or setup(), test and teardown()) up to N more times and records the final attempt. Skipped specs are never retried. The new attempts spec stat counts the runs and the text, console, Simple and stream outputs show "(passed after N attempts)".
  • Run only what failed: TestResult.getFailedTargets() returns the bundles and spec ids that failed or errored in a run (bundles that failed outside of a spec go to bundleErrors and are not rerun). The Simple, Min, Dot and Doc reports show a Run Failed button next to Run All, a link built from the report itself, so web runners keep no state. The BoxLang runner writes {reportpath}/.testbox-failed.json on every run and ./run --failed reruns only those; when the file is missing or lists nothing, it prints a message and runs nothing.
  • The Simple, Min, Dot and Doc reports show image attachments (failure screenshots) as thumbnails that open full size, embedded in the page so they work over http and in saved reports. Images up to the new inlineImageMaxKB reporter option (default 2048, 0 turns it off) are embedded, larger ones stay links. A trace attachment offers a Copy show-trace command button. A failed spec's screenshots are embedded once, in its failure card.
  • BoxLang runner --web-server, --web-server-url and --web-server-timeout: start a web server command before the tests, wait until its URL answers, stop it and its child processes after the tests, and exit with code 1 when it does not answer in time. The URL becomes the default baseURL of browser bundles through server.testbox.webServerURL.

Changed

  • TESTBOX-471 The HTML reporters (Simple, Min, Dot, Doc) are rebuilt on Bootstrap 5.3, Bootstrap Icons, Alpine and Prism, with light and dark themes and the new TestBox logos. Everything is still inlined so reports run airgapped, and a page went from about 1.4 MB to about 0.45 MB.
  • TESTBOX-471 The verdict is the first thing in every HTML report: a sticky green or red banner with the counts, a proportion bar and status filters, and bundle exceptions get their own alert.
  • TESTBOX-471 Re-run links (run bundle, suite or spec) are plain links to the runner that carry only the requested testBundles, testSuites or testSpecs, so a refresh runs it again. Every other runner option falls back to its default.
  • TESTBOX-471 Dot opens a spec's details in a drawer instead of alert(), Doc is a styled page with a bundle navigation, and the status chips of Min and Dot now filter.
  • TESTBOX-471 Report templates keep only markup. Their logic moved to system/reports/ReportHelper.cfc and BaseHTMLReporter.cfc.
  • Playwright.AssertionFailed exceptions now count as spec failures, like TestBox.AssertionFailed, in BDD specs and xUnit tests, keeping their message and detail. Other Playwright.* errors still count as errors.
  • xUnit failures now also record the failure detail in the spec stats.

Fixed

  • TESTBOX-471 The bundle badge of the HTML reporters showed a negative error count (-1 error) when a bundle threw outside a spec, for example in beforeAll(). It now shows none.
  • Specs that do not extend testbox.system.BaseSpec errored: the virtual BaseSpec missed its private state ($specStatsHolder), three of its methods declared a BaseSpec return type and expect() resolved Expectation relative to the spec. They run again.
  • ensureBrowserInstalled() downloaded Chrome for Testing for a profile with a branded channel (chrome, msedge). It now skips the install, as the installed browser is used.
  • A bundle browser that failed to close left the ledger, so no later run closed it. It now stays in the ledger and the error goes to testbox.log.
  • The run script now quotes its arguments, so runner options with spaces reach the BoxLang runner intact.
  • The JUnit and ANTJunit reporters failed on BoxLang without the bx-esapi module (encodeForXMLAttribute()). They now fall back to xmlFormat(), also on Adobe with full null support.

Removed

  • TESTBOX-471 The url.fullPage switch of the HTML reporters. A report is always a complete page.

7.1.0 - 2026-09-11

https://testbox.ortusbooks.com/readme/release-history/whats-new-with-7.1.0

Added

  • TESTBOX-451 BDD class skip annotation support for skipping entire test classes.
  • TESTBOX-457 Expectation context support via expect( value ).withContext( message ) that prepends semantic context to all failure messages including negated matchers and custom matchers.
  • TESTBOX-458 Collection expectation modes: expectAny(), expectSome(), and expectNone() alongside existing expectAll() with detailed failure summaries including element index/key and pass count reporting.
  • TESTBOX-459 Grouped assertions via $assert.all(), assertAll() that run multiple assertion closures and report every failure at once instead of stopping at the first.
  • TESTBOX-460 New matchers: toBeTruthy(), toBeFalsy(), toBeSameInstanceAs(), toHaveSize(), toThrowMatching(), toIncludeAll(), toIncludeAny(), and toIncludeNone().
  • TESTBOX-461 Set expectations: toBeASet(), toEqualSet(), toBeSubsetOf(), toBeSupersetOf(), toBeDisjointFrom(), toHaveUnion(), toHaveIntersection(), toHaveDifference(), and toHaveSymmetricDifference() for working with BoxLang Set objects.
  • TESTBOX-462 Range expectations: toBeRange(), toContainValue(), toContainRange(), toBeInRange(), toBeBeforeRange(), toBeAfterRange(), toBeBounded(), toBeUnbounded(), toBeHalfBounded(), toBeIterable(), toBeAscending(), toBeDescending(), toHaveStep(), and toClampTo() for BoxLang Range objects.
  • TESTBOX-463 Data navigator expectations: toHavePath(), toHavePathValue(), toHavePathType(), toHavePathSatisfying(), path(), and queryPath() for navigating and asserting against nested BoxLang data structures using dot-notation, array indexes, wildcards, filters, and recursive descent.
  • TESTBOX-464 New assertion BIFs: $assert.isTruthy(), $assert.isFalsy(), $assert.includesAll(), $assert.includesAny(), and $assert.includesNone().

Changed

  • TESTBOX-466 The coverageEnabled URL parameter in the CFML test runner now defaults to false instead of true. Code coverage requires FusionReactor and is now opt-in. Pass ?coverageEnabled=true to restore the previous behavior.

Improvements

  • TESTBOX-455 Expand the BoxLang CLI url-scope guard so it skips only when the scope truly isn't there, instead of assuming it is always absent in CLI mode.
  • Improve matcher failure messages with optional contextual prefix for distinguishing chained expectations.
  • Improve expectAll() failure messages to include pass/fail counts and per-element failure details with index/key context.
  • Add the What's New With 7.1.0 release page documenting all new assertion and expectation features.

Fixed

  • TESTBOX-448 Fix MockBox $args() struct-order fragility and add Set/Range support to argument matching.
  • TESTBOX-449 Encode HTML for bundle and spec names in the Simple reporter so markup in test names no longer breaks the report.
  • TESTBOX-450 Equalize assertions now handle different types of date and date/time objects for equality, instead of blindly calling actual.equals().
  • TESTBOX-452 Fix GetPageContextResponse() error while running BoxLang in Adobe compatibility mode.
  • TESTBOX-453 Fix the BoxLang CLI runner misreading its own script path as a positional bundle argument.
  • TESTBOX-454 Fix KeyNotFoundException [url] crashing every CLI run on BoxLang 1.17+.
  • TESTBOX-456 Simplify the BoxLang CLI url scope guard to a plain param.
  • TESTBOX-465 Fix isLucee() returning true on BoxLang, which broke engine detection helpers and engine-conditional skips.
  • TESTBOX-467 Support engines running with full null support enabled.
  • TESTBOX-468 Fix custom matcher failure messages not routing through the expectation's internal fail method.

7.0.0 - 2026-03-17

6.5.0 - 2026-01-25

6.4.0 - 2025-09-18

6.3.2 - 2025-04-29

Fixed

  • Update the run runners so they use the calculated location paths.

6.3.1 - 2025-04-01

Fixed

  • Fixed a typo in BaseReporter

6.3.0 - 2025-02-25

6.2.1 - 2025-02-06

6.2.0 - 2025-01-31

6.1.0 - 2025-01-28

6.0.1 - 2024-12-05

6.0.0 - 2024-09-27

New Features

  • TESTBOX-391 MockBox converted to script
  • TESTBOX-392 BoxLang classes support
  • TESTBOX-393 New environment helpers to do skip detections or anything you see fit: isAdobe, isLucee, isBoxLang, isWindows, isMac, isLinux
  • TESTBOX-394 new test(), xtest(), ftest() alias for more natuarl testing
  • TESTBOX-397 debug() get's two new arguments: label and showUDFs
  • TESTBOX-398 DisplayName on a bundle now shows up in the reports
  • TESTBOX-399 xUnit new annotation for @DisplayName so it can show instead of the function name
  • TESTBOX-401 BoxLang CLI mode and Runner
  • TESTBOX-402 New matcher: toHaveKeyWithCase()
  • TESTBOX-403 Assertions: key() and notKey() now have a CaseSensitive boolean argument

Improvements

  • TESTBOX-289 showUDFs = false option with debug()
  • TESTBOX-331 TextReporter doesn't correctly support testBundles URL param
  • TESTBOX-395 adding missing focused argument to spec methods
  • TESTBOX-396 Generating a repeatable id for specs to track them better in future UIs

Bugs

  • TESTBOX-123 If test spec descriptor contains a comma, it can not be drilled down to run that one spec directly
  • TESTBOX-338 describe handler in non-called test classes being executed

Tasks

  • TESTBOX-400 Drop Adobe 2018 support

Use It

$ box install testbox

How do I do this?

Collaborators 0

No collaborators yet.

Links

     

User Rating

5.00 / 2

Stats

  • {{ getFullDate("2014-05-29T11:22:57-05:00") }}
  • {{ getFullDate("2026-10-16T08:26:18-05:00") }}
  • 40,841
  • 2,707,958
Report Abuse