PREVIEW: Two runners, one executable

The unit tests used to need a window. Now the same Unittests program — Delphi’s and Free Pascal’s alike — runs headless the moment you hand it a parameter, and still comes up in its GUI runner when you don’t.

One set of test sources, compiled by two toolchains: Delphi runs them under DUnit, Free Pascal under FPCUnit. Both had exactly one way in — open the project, build, run, press Run all tests in a window. That is fine at a desk. It is no use in a build script, on a machine nobody is logged into, or when you just want to know whether the STOMP decoder still passes before you commit.

So both programs now carry both runners and decide which one to use at startup, from the command line. One program, one project file, one build, on either toolchain — two ways to run it.

The rule

No parameters, GUI. Double-click the executable, or press F9 in the IDE, and the runner appears with the familiar tree of suites — DUnit’s GUITestRunner under Delphi, the LCL one under Lazarus. Nothing about that changed.

Any parameter, console. The run goes to the console runner instead, which prints its report and ends. There is no switch to remember for the mode itself: --help is a parameter, so it already puts you on the console side and lists everything else.

Registration is identical on both paths — the same TestSuiteList.pas builds the same suites in the same order — so a console run and a GUI run see the same tests. Adding a test still means adding it in demo/common-tests and registering it once.

What is behind the two console runners differs, though. Free Pascal has one in the box: consoletestrunner from fcl-fpcunit, complete with option parsing and four report formats. DUnit has no console runner worth the name — TextTestRunner is a listener and two functions, with no command line at all — so the Delphi side gets DUnitConsoleRunner.pas, which supplies the switches around it. They were kept as close together as DUnit allows.

What you can pass

Switch	DUnit	FPCUnit
-a, –all	✓	✓
-l, –list	✓	✓
–suite=NAME[,NAME]	✓	✓
-p, –progress	accepted; always on	✓
–file=NAME	✓	✓
–format=FMT	plain only	plain, plainnotiming, xml, latex
–loglevel=LEVEL	✓	✓
–sparse, –skiptiming, –no-addresses, –stylesheet=	—	✓
-h, –help	✓	✓

Given any switch other than --list or --suite=, both run everything. -p is the progress line: one character per test as it goes, . passed, F failed, E error — the same alphabet in both frameworks, except that DUnit always prints it.

Two defaults we moved

A stock consoletestrunner writes XML, and prints its usage screen when it cannot work out what to run. Both defaults are set differently in Unittests.lpr, because a suite you run by hand should answer in words:

• DefaultFormat is fPlain. Ask for --format=xml when a machine is going to read it. DUnit has nothing to move here — plain text is all it writes, and --format= anything else is refused rather than quietly ignored.
• DefaultRunAllTests is True, so Unittests --progress runs the whole suite rather than printing help. --list and --suite= still override it.

What a run looks like

> Unittests --suite=TDecoderTests --progress --format=plain

........SuiteList Time:00.002 N:8 E:0 F:0 I:0
  TDecoderTests Time:00.002 N:8 E:0 F:0 I:0
    00.001  TestHeaderPropertyValueMissing
    00.000  TestEqualSignInHeader
    00.000  TestCrLfInHeader
    00.000  TestColonInHeader
    00.001  TestBackslashInCONNECTEDHeader
    00.000  TestEscapeInCONNECTEDHeader
    00.000  TestBackslashInMESSAGEHeader
    00.000  TestUnescapedColonInHeader

Number of run tests: 8
Number of errors:    0
Number of failures:  0

Eight dots from –progress, then the plain report.

DUnit is terser, and --file= keeps the report out of the console entirely:

> Unittests --suite=TDecoderTests --file=report.txt

DUnit / Testing
...........
Report written to report.txt

> type report.txt

OK: 8 tests

DUnit: progress on the console, report in the file.

The whole suite, both toolchains, takes a few seconds when no broker is listening: 43 tests in 4.7 seconds under DUnit on this machine, unit tests and stub-server tests across all three communication adapters.

What the exit code means

Both runners set it from the result, on the same scheme — which is the point of the whole exercise: a build step can now fail on a red test. Under FPCUnit the code comes out of consoletestrunner‘s progress writer, under DUnit out of the TTestResult.

Code	Meaning
0	Everything that ran, passed.
1	Failures — an assertion did not hold (bit 0).
2	Errors — a test raised something unexpected (bit 1).
3	Both.
203	Free Pascal only, and not from FPCUnit: heaptrc found a corrupt heap on the way out. Worth failing on.

A mistake is only fatal on one side

Give the Delphi runner a switch it does not know, a --format= it cannot write, or a --suite= that matches nothing, and it says so and exits 1 without running anything.

FPCUnit prints a comparable message — Invalid option at position 1: "frmat", No tests selected. — and exits 0. That is consoletestrunner‘s own behaviour, so a typo in a Free Pascal test job passes silently. Until it is overridden, keep FPC jobs on --all, or check that the report holds the number of tests it should.

The log and the report share one pipe

Compiled with HABARI_LOGGING, the library logs through slf4p to standard output — and so does the report. Left at the GUI runner’s trace level that would bury the result under thousands of lines, so console mode configures the level as error instead. --loglevel is there for when a test misbehaves and you want to watch the frames go by:

> Unittests --suite=TDecoderTests --loglevel=trace

22 TRACE BTAbstractTransport.TBTAbstractTransport Set default communication adapter
22 INFO  BTAbstractCommAdapter.TBTAbstractCommAdapter Logging with slf4p v1.0.8
24 DEBUG BTCommAdapterBaseIndy.TBTCommAdapterBaseIndy CommAdapter using: Indy 10.6.3.14
24 TRACE BTCommAdapterIndy.TBTCommAdapterIndy Created a communication adapter: TBTCommAdapterIndy
        ...

–loglevel=trace, the level the GUI runner has always used.

Even at error the broker probe logs a few lines before the report starts, so whenever the output is going to be parsed, keep the two apart:

Unittests --all --format=xml --file=results.xml

The report then lands in the file and the log stays on the console, where a build server can archive it next to the result.

Running it from the IDE

Unittests.lpi has a second run mode called console runner, set to --all --progress. Pick it under Run › Run parameters and F9 gives you the console run inside Lazarus; switch back to default for the GUI. Delphi takes the same parameters under Run › Parameters. Nothing needs rebuilding in between — it is the same executable either way.

Inside the program

The whole decision is four lines at the bottom of the program file. Free Pascal:

// Without parameters the GUI runner comes up as before, any parameter selects
// the console runner - "Unittests --help" lists what it understands.
if ParamCount = 0 then
  RunGuiTestRunner
else
  RunConsoleTestRunner;

… and Delphi, where the console half lives in its own unit:

if DUnitConsoleRunner.RunsOnConsole then
  // Run the tests on the console
  DUnitConsoleRunner.RunRegisteredTestsOnConsole
else
  // Run Test GUI
  GUITestRunner.TGUITestRunner.RunRegisteredTests;

That unit is not called ConsoleTestRunner, tempting as it was. demo/common-tests sits on the Free Pascal project’s unit search path, and a unit of that name there would shadow FPC’s own consoletestrunner — unit names are case-insensitive, and the project path is searched first. The Lazarus build would have broken the moment the Delphi unit was added.

On the FPCUnit side, the extra switch is a small descendant of the stock runner. FPCUnit checks the command line against a list of known options, so a new one has to be registered before it can be used: AppendLongOpts is where that happens, and WriteCustomHelp is where it gets documented.

  THabariConsoleTestRunner = class(consoletestrunner.TTestRunner)
  protected
    procedure AppendLongOpts; override;
    procedure WriteCustomHelp; override;
  public
    function LogLevel: string;
  end;

procedure THabariConsoleTestRunner.AppendLongOpts;
begin
  inherited AppendLongOpts;

  LongOpts.Add('loglevel:');
end;

One more line matters on Windows. The Free Pascal program is now built as a console subsystem application, the way the Delphi Unittests.dpr always has been, so that standard output exists at all:

{$IFDEF MSWINDOWS}
// console subsystem: the console runner writes its report to standard output
{$APPTYPE CONSOLE}
{$ENDIF MSWINDOWS}

The visible consequence is that the GUI runner now has a console window behind it. That is the price of one executable instead of two, and it is a cheap one: no second project file, and no second set of search paths and conditional defines to keep in step with the first.

Before you wire it into a build

• An empty run is a green run. The broker suites are only registered when a broker answers on stomp://127.0.0.1:61613. Without one, the run quietly shrinks to the unit tests and the stub-server tests and reports success. The title line says which it was — look for broker="(not connected)".
• The integration suite runs three times, once per communication adapter, as Integration tests using TBTCommAdapterIndy and its Synapse and mORMot2 counterparts. Those names work with --suite=, quoted, when you want one networking library at a time.
• --suite= selects differently on the two sides. FPCUnit lifts the matching tests out of their suite into a decorator and runs that, so an integration test picked by name runs without its suite having chosen the communication adapter — whatever adapter is default, the first one registered, gets used. DUnit instead disables everything else and runs the tree, so the enclosing suite still runs its SetUp and the adapter is the one the suite name promises.
• The unit-test suite has a long name. It is titled with the run’s banner — product, broker, compiler — so reach for a test class name with --suite=, or run --list and copy what it prints.
• The Free Pascal build still needs Lazarus. That program links the LCL for its GUI half, so the LCL unit paths have to be on the compiler command line; the plain fpc call in the Ant script does not have them. Build it in the IDE, run it anywhere. The Delphi side has no such constraint — ant compile_d2009 produces an executable a build step can run straight away.

Where it lives

demo/common-tests/The test units themselves, shared by both toolchains and all four products.demo/common-tests/Unittests.dprThe Delphi program: picks the runner, then hands over.demo/common-tests/DUnitConsoleRunner.pasThe command line DUnit does not have: options, --list, selection, --file, exit code.demo/common-tests-fpc/Unittests.lprThe Free Pascal program: both runners, the switch, the --loglevel descendant.demo/common-tests-fpc/Unittests.lpiThe Lazarus project, with the console runner run mode.RegisterDUnitTests.pas, RegisterFPCUnitTests.pasSuite registration, unchanged — still one place each, still order-sensitive.

The demo folder is shared by Habari ActiveMQ, Artemis, OpenMQ and RabbitMQ, so all four get both console runners at once.