Skip to content

[dv] Repair the single-step test environment - #2499

Draft
kulan-pal wants to merge 9 commits into
lowRISC:masterfrom
kulan-pal:dv/single-step-env
Draft

kulan-pal wants to merge 9 commits into
lowRISC:masterfrom
kulan-pal:dv/single-step-env

Conversation

@kulan-pal

@kulan-pal kulan-pal commented Sep 14, 2026

Copy link
Copy Markdown

Depends on #2497 (directed-test schema and scheduling fixes). Review the last six
commits only until it lands. Prerequisite for the dummy-instruction fixes in #2500.

On baseline 34b07057, riscv_debug_single_step_test fails 1 seed in 15 on the
opentitan config, and the seeds that pass do not step through the program:
stepping silently stops at the first exception taken in debug mode. This PR fixes
the six things in the DV environment that stand in the way, one commit each, in the
order they were uncovered. With them, stepping runs through whole programs; the one
seed that still fails does so after enabling dummy-instruction insertion, in a way
consistent with #2186, and passes with #2500 on top of this PR. Five are defects in the DV environment (cosim, the Ibex riscv-dv
extension, the vendored riscv-dv, the testbench). One restricts the stock test's CSR
writes and instruction count. No RTL is touched.

The six fixes

  1. Cosim: sample debug mode before the step (dv/cosim/spike_cosim.cc).
    handle_cpuctrl_exception_entry() looked at Spike's debug-mode state after
    processor->step(). A trap on a stepped instruction enters debug mode inside that
    same step, so the check saw debug mode and skipped the sync_exc_seen /
    double_fault_seen updates the RTL had already made (DUT cpuctrlsts 0x141,
    model 0x101). Capture the state before the step and pass it in.

  2. ECALL handler pops its trap frame (riscv_dv_extension/ibex_asm_program_gen.sv).
    Ibex's ECALL handler override advanced mepc and executed mret without popping
    the 132-byte kernel-stack frame the trap dispatcher pushed, leaking one frame per
    ECALL. Once enough leaked, a subprogram epilogue reloaded its return address from
    a slot holding unrelated saved state. Spike did the same, so the cosim was silent.
    Pop the frame before the mret.

  3. riscv-dv debug ROM pops the debug frame on in-debug exceptions
    (vendor/patches/google_riscv-dv/0006). The debug exception handler ended in a
    bare dret, leaking one frame per exception taken in debug mode (one seed leaked
    825 frames, 108 KB). Route the handler to <hart>debug_end, which pops and then
    drets, when a debug section exists. This is the change that makes stepping run
    through the whole program; simulator runtime per seed grows accordingly.

  4. riscv-dv kernel-stack claim made atomic (vendor/patches/google_riscv-dv/0007).
    push_gpr_to_kernel_stack published its claim in tp only after the stores, so a
    debug entry between them, which real stepping produces, allocated its frame on top
    of the half-built one. On RV32 with bare translation and a scratch CSR (Ibex's
    case) claim in one addi, keep the frame layout identical, release in one
    instruction. Other configurations keep the original sequence.

  5. Restrict the single-step test's CSR writes (riscv_dv_extension/testlist.yaml).
    With real stepping the random debug body runs between every instruction of live
    trap handlers, and its randomized csrw mepc clobbered the handler's return
    address (trace: csrrw x1, mepc, x15 with x15 = 0, mret, walk from address 0).
    The test now writes only cpuctrlsts and secureseed, the CSRs it is about, and
    instr_cnt drops from 10000 to 2000 to keep simulator runtime manageable. It
    still arms hundreds of steps per seed.

  6. Hold debug_req until the hart reports halted (tests/core_ibex_seq_lib.sv).
    debug_seq pulsed the request for 75 cycles. A wake-up from WFI needs longer, so
    the request was gone before the debug ROM's first instruction retired; RVFI never
    reported it and the cosim stepped Spike past the WFI ("DUT retired 80000000 but
    the ISS retired 800035f4"). A debug module holds haltreq until the hart halts;
    the sequence now holds the request until an RVFI retirement carries it, with a
    5000-cycle error timeout. debug_new_seq is unchanged (its only user runs with
    +no_wfi=1).

Items 3 and 4 are carried as vendor patches so util/vendor.py re-applies them; the
patched files are included. Both apply cleanly to the pristine vendored source.

Verification

Baseline 34b07057. All on the opentitan config, VCS U-2023.03-SP2, Spike
ibex_cosim at the pinned version, SEED=1 with consecutive seeds.

Test set Tree Result
riscv_debug_single_step_test, 15 seeds baseline 14 / 15; seed 12 fails at 1.24 ms simulated time: the cosim expects a synchronous trap the DUT does not report
riscv_debug_single_step_test, 15 seeds this PR alone 14 / 15; seed 12 enables dummy insertion with csrrsi x19, cpuctrlsts, 5 at 6.39 ms and fails at 9.80 ms immediately after a dret: the DUT re-enters debug without executing the store Spike expects. Consistent with #2186
riscv_debug_single_step_test, 15 seeds this PR + #2500 15 / 15; seed 12 runs to completion (104 ms simulated time) with 7 main-core SingleStepDummyWhileArmed cover matches
riscv_debug_wfi_test, 10 seeds this PR alone 10 / 10
riscv_debug_wfi_test, 10 seeds this PR + #2500 10 / 10
10 other +enable_debug_seq tests, 2 seeds each baseline 18 / 20
10 other +enable_debug_seq tests, 2 seeds each this PR alone 18 / 20, same two (riscv_debug_triggers_test)
10 other +enable_debug_seq tests, 2 seeds each this PR + #2500 18 / 20, same two
14 non-debug random tests, 2 seeds each (arithmetic_basic, machine_mode_rand, rand_instr, ebreak, dret, illegal_instr, nested_interrupt, interrupt_wfi, csr, user_mode_rand, umode_tw, pmp_full_random, mmu_stress, unaligned_load_store) this PR alone 28 / 28
11 fault-injection and misc tests, 2 seeds each (mem_error, mem_intg_error, reset, pc_intg, rf_intg, rf_addr_intg, ram_intg, icache_intg, rv32im_instr, invalid_csr, hint_instr) this PR alone 22 / 22

The stock single-step test meets dummy instructions whenever its random CSR writes
enable them, so its last failing seed needs the RTL fix in #2500; this PR takes
that seed from an environment failure at 1.24 ms to a failure at 9.80 ms that is
consistent with #2186 and disappears with #2500. The
non-debug rows matter because items 2 and 4 change every generated program, not
only the debug tests. Directed riscv-tests and arch-tests do not use the riscv-dv
dispatcher and are unaffected. Not run: the full 57-test random regression and
coverage.

Known, not addressed here: riscv_debug_triggers_test fails on the baseline and on
this branch alike. On the baseline riscv_assorted_traps_interrupts_debug_test
passed 2 of 10 seeds; the other eight failed with synchronous-trap cosim mismatches.
A separate run of that test also showed riscv-dv emitting a legal Zcmp halfword
(0xbefe) as an "illegal" instruction. Neither is addressed here.

Disclosure

Per CLA section 9: the analysis and code in this PR were produced with the
assistance of an AI coding tool (Claude Code) and independently reviewed with a
second tool (OpenAI Codex). I have reviewed and understood every change and take
full responsibility for it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Rn6C7Eyq1LchFeP9w6up9P

run_rtl.py reads 'sim_opts' out of the test options for both riscv-dv
and directed tests, but the pydantic schema that validates a directed
test's configuration has no such field, so pydantic's default
extra="ignore" dropped it. A directed test therefore could not pass a
plusarg through its own testlist entry.

Add the field to DConfig, optional so every existing entry stays valid,
and on DConfig rather than DTest so a config can supply a default that
an individual test still overrides after the merge.

Signed-off-by: Kulan Palanichamy <kulan.palanichamy@opentitan.org>
The append that accepts a test sat inside the loop over its rtl_params,
so a test listing N parameters was scheduled N times: make warned that
its targets were given more than once and regr.log counted every seed N
times. Every existing entry lists a single parameter, which hid it.
Append in the loop's else clause instead, once all parameters matched.

Signed-off-by: Kulan Palanichamy <kulan.palanichamy@opentitan.org>
gen_testlist.py enumerated the vendored test directories with ls, whose
order follows the collation of the current locale: under C/POSIX
sh-misaligned sorts before shamt, under en_US after it. Regenerating
directed_testlist.yaml therefore produced a different file depending on
the environment, and the committed file could not be checked against
its generator.

List the directories with os.listdir() and sort the names in code point
order, which does not depend on the locale. Regenerate
directed_testlist.yaml: no entry changes, only the order of the
vendored tests.

Signed-off-by: Kulan Palanichamy <kulan.palanichamy@opentitan.org>
handle_cpuctrl_exception_entry() tested Spike's debug-mode state after
processor->step(), but a trap taken on a stepped instruction enters
debug mode inside that same step (the catch block in execute.cc), so
the post-step check saw debug mode and skipped the sync_exc_seen and
double_fault_seen updates. The RTL sets those flags at trap take, which
is outside debug mode, so the models diverged: riscv_debug_single_step
seeds read cpuctrlsts as 0x141 on the DUT against 0x101 in the model.

Capture debug mode before the step and pass it in. The RTL behaviour is
architecturally correct; only the model's sampling point was wrong.

Signed-off-by: Kulan Palanichamy <kulan.palanichamy@opentitan.org>
The Ibex ECALL handler override advanced MEPC and executed mret without
popping the kernel-stack frame the trap dispatcher had pushed on entry,
so every ECALL leaked one 132-byte frame. Once enough frames had leaked,
a generated subprogram epilogue reloaded its return address from a slot
that now held unrelated saved state and jumped into uninitialized
memory, after which the program walked mepc+4 through zeroed memory
until the wall-clock timeout. The cosimulator stayed silent because
Spike executed the same architecturally consistent corruption.

Pop the frame before the mret, restoring every interrupted GPR
including the randomized SP and TP roles. MEPC is rewritten before the
pop, so restoring the scratch register cannot lose the new return
address, and the register-dump stores keep their memory side effects.

The leak only becomes visible once single stepping runs through the
whole program (see the debug ROM fix in the vendored riscv-dv), which
is why the single-step tests were the first to hit it.

Signed-off-by: Kulan Palanichamy <kulan.palanichamy@opentitan.org>
The debug exception handler ended in a bare dret, so every exception
taken inside debug mode returned without popping the frame the debug
ROM had pushed. Each in-debug exception leaked one 132-byte frame; one
seed leaked 825 frames and walked 107.9 KB down from kernel_stack_end
into the program image.

Route the handler to <hart>debug_end (pop then dret) whenever a debug
section was generated; dpc is untouched by an in-debug exception, so
the canonical epilogue returns to the right place. Keep the bare dret
when there is no debug section and therefore no frame to release.

This restores whole-program single stepping: before it, stepping
silently stopped at the first in-debug exception, so the single-step
tests passed without stepping through the program.

Carried as vendor/patches/google_riscv-dv/0006 so that the next vendor
update re-applies it.

Signed-off-by: Kulan Palanichamy <kulan.palanichamy@opentitan.org>
push_gpr_to_kernel_stack published its claim in TP only at the end of
the push. With dcsr.step armed, stepping a trap handler lets each debug
entry allocate a frame from the stale TP, four bytes into the handler's
half-built frame: the debug frame's slot 8 lands on the handler's saved
SP and slot 28 on its frame pointer, so the handler's pop reloads its
base mid-sequence and the register file unwinds to garbage.

On the configuration Ibex runs (RV32, BARE translation, scratch CSR
implemented) claim the frame in a single instruction (addi tp, tp, -132)
so TP publishes before any store, keep the USP slot at the top of the
frame (offset 128, so the frame layout and every address are unchanged),
restore USP while the claim is still held, and release in one
instruction. A debug entry interrupting any point of the sequence now
allocates below the active frame. Every other configuration (RV64,
address translation, no scratch CSR) keeps the original sequence
untouched.

Carried as vendor/patches/google_riscv-dv/0007 so that the next vendor
update re-applies it.

Signed-off-by: Kulan Palanichamy <kulan.palanichamy@opentitan.org>
The test randomized writes to MSTATUS, MEPC, MCAUSE and MTVAL. Under
real stepping the random debug body executes interleaved between every
instruction of a live mtvec handler, so a random 'csrw mepc, <garbage>'
from the body clobbers the return address the handler underneath still
needs, and the program mrets into zeroed memory. Trace-proven: csrrw
x1, mepc, x15 with x15 = 0, then mret, then a walk from address 0. The
defect was masked because the bare dret in the debug ROM kept the debug
body degenerate; with the debug frame fix stepping runs through the
whole program.

Trim add_csr_write to cpuctrlsts (0x7c0) and secureseed (0x7c1), the
CSRs the test is actually about, and reduce instr_cnt from 10000 to
2000 to keep simulator runtime manageable now that stepping runs
through the whole program; the test still arms hundreds of steps per
seed.

Signed-off-by: Kulan Palanichamy <kulan.palanichamy@opentitan.org>
debug_seq pulsed debug_req for a fixed 75 cycles. A request that wakes
the core from WFI needs longer than that to reach the debug ROM: the
clock restarts, the instruction after the WFI is refetched into ID,
debug entry flushes it and the halt address is fetched. RVFI attaches a
captured request to the first instruction entering ID, which is the
flushed one, and the debug ROM's first instruction then reports the
live debug_req, already low. The cosim never learns of the request and
steps Spike past the WFI: riscv_debug_single_step_test seed 12 with
real stepping failed with "DUT retired 80000000 but the ISS retired
800035f4".

A debug module keeps haltreq asserted until the hart reports halted,
which the debug ROM does with its first instructions. Do the same:
after the pulse, keep debug_req high until RVFI has reported a
retirement that carries the request (debug_mode alone rises at debug
entry, before the halt address is fetched, so it is too early). Not
seeing one within 5000 cycles is an error. debug_new_seq is left
alone: its only user, riscv_assorted_traps_interrupts_debug_test, runs
with +no_wfi=1 and pulses for 3000-5000 cycles.

Signed-off-by: Kulan Palanichamy <kulan.palanichamy@opentitan.org>
@github-actions

Copy link
Copy Markdown

Thank you for your submission, we really appreciate it.

Like many open-source projects, we ask that you sign the Contributor License Agreement before we can accept your contribution.

The CLA ensures that all users of the project are granted rights to use the submission.

Before signing the CLA, please ensure that you have the authority from your organisation to grant these rights.

You will be asked to sign the CLA when you first contribute to each lowRISC repository, and will be asked to re-sign if the CLA changes.

Each individual who has committed in this Pull Request should sign the CLA by posting a Pull Request Comment containing the text below.


I have read the CLA Document. By submitting this pull request comment, I am hereby confirming my acceptance of the terms of the CLA Document and my agreement to be legally bound by its terms.


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant