Moved (23 October 2026). This repository now lives in
openIE-dev/rceasconformance/, with its full history, so that it is checked from the same commit as the specification and the reference checker and can no longer drift from them. Changes go there; this repository is kept only as a pointer.
An implementation-independent conformance suite for RCE. Every test is a plain
.rce file with a metadata block. The harness talks to any implementation
through a line-oriented JSON adapter protocol and reports, per normative
clause of the RCE book, which tests exercise it and whether they pass.
The guiding rule: tests encode what the laws of computing require (IEEE 754,
two's complement, injectivity of the log digest, replay equals reopen), not
whatever the spec text or the reference happens to say. Where the two differ,
the test follows the law and the difference is filed in FINDINGS.md with a
derived fix. No test encodes a known bug.
clauses.json: every clause extracted fromspec/book/*.md(rule ids such asW-PAR-1andHC-15, theorem rows, MUST/SHALL sentences asCHnn-M-<hash>), each with a stable id and a profile (language,handler-spec,tooling,proposed,editorial).tests/<area>/: the tests, one area per directory.harness/:rcetest.py(file format) andrun.py(runner).adapters/: the reference interpreter, the Rust, Python, JS and Scala emitters, and the wasm playground.tools/: clause extraction, harvesting, generators, tagging, coverage, emission parity, tree-sitter cross-check, mutation testing.findings/andFINDINGS.md: every place an implementation or the spec disagrees with the law.reports/: coverage, per-adapter results, parity, mutation score.
See INTERPRETING.md for the file format and ADAPTERS.md for the protocol.
Current limit. The adapter binary rce-conform is built from public rce (cd spec/checker && cargo build --release --bin rce-conform, source src/bin/conform.rs, present since openIE-dev/rce cdae5cf). Tests that use $RCE_ENGINE (the handler-spec profile, appendix D) need the engine's rce, which is not public, so those cannot run against public rce. The lexing-parsing area was run against public rce main on 17 October 2026: 133 of 133 pass. The reference and the three emitter reports were rerun against public rce 4a3dc1f (next section); the other reports were produced against the engine repository, whose history is not public. The clause paths below use the public layout (spec/book).
export RCE_CONFORM=/path/to/rce-conform # cargo build -p rce-spec-check --bin rce-conform
python3 harness/run.py --adapter adapters/reference.json --jobs 8
python3 tools/coverage.py
python3 tools/parity.py
Generated from reports/ by tools/readme_status.py (--check verifies it). Every figure below is counted from the report files.
- Tests: 1979 test files under
tests/(1983.rcefiles, of which 4 named_FIXTUREare support files, not tests); the reference run has 1979 results; the coverage report counts 1979. The three counts agree. - Coverage: language profile 205 of 205 normative clauses covered (100.0%). Other profiles: tooling 5 of 7, retired 1 of 2, handler-spec 8 of 14, editorial 0 of 1, proposed 3 of 6.
- Reference: 1979 pass, 0 fail of 1979.
- Emitters: python 1459 pass / 0 fail / 520 not applicable or unsupported; js 1459 pass / 0 fail / 520 not applicable or unsupported; rust 1459 pass / 0 fail / 520 not applicable or unsupported. Parity with the reference: python 1460 agree, 0 disagree; js 1460 agree, 0 disagree; rust 1460 agree, 0 disagree.
- Scala emitter: 52 pass, 1189 fail, 591 unsupported of 1832 results (146 tests have no Scala result). Scala does not yet agree with the reference.
- Wasm playground: 419 pass, 16 fail, 1504 unsupported of 1939 results.
- Lean core model: 210 pass, 1 fail, 1766 unsupported of 1977 results.
- Tree-sitter grammar
00b639a: 1773 of 1773 programs agree with the reference parser (0 disagreements). - Mutation: 200 of 712 schema sites sampled (seed 1); 185 killed (92.5%) on the first run. After survivor-targeted tests: 94.5%; after round 3: 99.0%; round 4 reports 99.5% counting mutant 133 as equivalent. Survivors are in
reports/mutation.json. The mutation baseline (the unmutated mutation build) fails 20 tests; that is not the reference run, which has 0 failures.
The reference, emit-python, emit-js and emit-rust results in reports/ were rerun against public rce main 4a3dc1f, with rce-conform and rce built from that commit ($RCE_ENGINE set to the same rce). Before the tests were converted to the one-language rules the reference run had 1,575 pass and 404 fail; the converted suite has 1,965 pass and 14 fail.
- The 14 not re-run: the 14
handler-spectests (appendix D) need the engine'srce, which is not public; their committed results (pass, engine run) are kept, and they fail here because publicrcehas nospecdeclarations. - Not re-run, results kept as committed (minus the removed
emit06-reject): emit-scala (needs a Scala toolchain run that was not made), wasm-playground (needs a playground build of the new checker) and lean-core (needs the Lean build). Their results predate the conversion of 371 programs and the rename ofemit06-rejecttoemit06-accept, so for those tests they describe the earlier program text.reports/tree-sitter.jsonwas not regenerated either; it still lists the earlier program text. - Conversions: programs were changed only as the one-language rules require (
_namefor unused names,let _ =for dropped results,loop bound N,len_bytes, renamed reserved wordsctandlabel, externs taking&World). Expectations that encode the retired&mutwrite-back (T33) were re-derived under F.4.4 and checked by hand (write-back-methods, with its two generated fuel tests).s4-std-modulesnow expects W-UNUSED-1, because an unuseduseis refused.emit06-rejectbecameemit06-acceptbecauseparis part of the language. The 12 version mutants were removed, and clauses W-VER-1 and the 2.7 version sentence are in theretiredprofile.
The committed reports/ were produced against the engine repository, not public rce; the reference adapter (rce-conform) is not in public rce, so no reference report can be regenerated from public sources. Two partial runs were made against public rce main 4304048 with a check-only adapter and the emit-python adapter (the counts are from those runs, not from reports/):
- Check tests: 436 pass, 12 fail of 448 (1,543 further tests need an operation the check-only adapter does not offer). Causes of the 12: six
W-SES-4tests (session use after move, send, close or in a loop) are reported underW-SES-2orW-OWN-1, or accepted, so publicrcedoes not enforceW-SES-4as written; threeW-TY-3tests (int and float mixing, narrowing, equality) are reported asW-TY-1;w-mod-2-cycle(module cycle) is accepted;w-state-3-borrow-fieldis reported asW-TY-2andW-TY-1;w-trait-1-impl-mismatchis accepted. - emit-python: 1,304 pass, 156 fail, 531 not applicable. Causes: (1) 103 tests (102 in
forms-logandf6-signed-zero-state) expect the emitted program to take--traceand printtrace HEAD; emitted Python from publicrcehas no--traceline, so the adapter sees the genesis head. (2) 27concurrency/generatedtests are float reductions infor ... parwhose oracle is the fixed balanced tree of spec C.8; publicrceevaluates them as the left fold (tree-prod-006gives 1.857142857142857 where the tree gives 1.8571428571428568). Float multiplication and addition are not associative, so the two orders differ in the last digits; the specification sentence that the tree equals the left fold holds only for associative operations. (3) 13 tests expect-0where publicrceprints0(ceilof a tiny negative, and-0state). (4) 11 overflow tests agree on the trap but print a different out-of-range value in the message. The other two aren9-size-limit(publicrcefinishes where the test expects a size-limit trap) andr5-generic-call(the emitter refuses a generic call with unknown typeT).
None of these is a suite error; each is a place where public rce and the engine behind the committed reports differ. They are not fixed in public rce here because the -0 text and several of these paths are being changed in a pending export.
The suite converges on ideas already established elsewhere. These notes cite only material read while building it.
- Test262 (
INTERPRETING.mdin tc39/test262): metadata lives in the test file, negative tests state the phase and the error type they expect, flags change how a file is run, and files named with_FIXTUREare support files rather than tests. Hereexpect: rejectplus a diagnostic code plays the role of the negative phase and type, andfixtures:names support modules. - The WebAssembly spec testsuite (README of WebAssembly/testsuite and
test/core/README): assertions about traps, invalid modules and exhaustion are first-class alongside assertions about results. Here a run statesoutcome: trap:<kind>,refused:<error>,doneordeadlock, and fuel exhaustion is a trap kind. - ACATS (the ACATS overview page and the User's Guide, section 3.1): tests are organised by test objective tied to rules of the reference manual. Here every test lists the clause ids it exercises and the coverage report lists every clause with no test.
Licensed under either of Apache License, Version 2.0 (LICENSE-APACHE) or the MIT license (LICENSE-MIT), at your option. Copyright (c) 2026 Open Interface Engineering, Inc. See COPYRIGHT.