Skip to content

Skip tearDown when setUp fails, and never let tearDown crash the worker - #257

Merged
kei-nan merged 1 commit into
RedisLabsModules:masterfrom
kei-nan:jk-fix-teardown-after-failed-setup
Sep 27, 2026
Merged

kei-nan merged 1 commit into
RedisLabsModules:masterfrom
kei-nan:jk-fix-teardown-after-failed-setup

Conversation

@kei-nan

@kei-nan kei-nan commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • _runTest called after_func() (tearDown) unconditionally in a bare finally, even when before_func() (setUp) raised. A tearDown written against a successful setUp (e.g. one that reads self.env) then raises its own exception -- typically AttributeError, since the attribute setUp would have assigned was never set. That second exception escaped _runTest uncaught: unlike the original setUp/test failure, it bypasses handleFailure entirely.
  • In the parallel coordinator this kills the whole worker process instead of failing just the one test. Because RLTest schedules different methods of the same test class across different workers, a single flaky setUp (e.g. a transient env-startup connection refusal) can take down every worker in the pool near-simultaneously.
  • Track whether setUp completed; skip tearDown if it didn't (mirroring the existing exit_on_failure branch's intent, generalized to the default path). If tearDown itself raises after a successful setUp, route it through handleFailure like any other test failure instead of letting it propagate.

Found while diagnosing a RediSearch nightly run where this exact pattern (TestBM25NormMax in test_scorers.py) hit all 4 parallel coordinator workers within ~2 seconds of each other and took the whole "Flow tests (coordinator)" CI step down for the full 120-minute timeout: https://github.com/RediSearch/RediSearch/actions/runs/36269638134/job/108481212637

Test plan

  • Added tests/unit/test_teardown_after_failed_setup.py reproducing both: (1) setUp fails -> tearDown must be skipped, not crash; (2) setUp/test succeed but tearDown itself raises -> must be reported via handleFailure, not propagated.
  • Verified both new tests fail with an unhandled exception against the unpatched code, and pass with the fix.
  • Full unit suite (pytest tests/unit/) passes: 161 passed, 5 skipped.

🤖 Generated with Claude Code

_runTest called after_func() (tearDown) unconditionally in a bare
finally, even when before_func() (setUp) raised. A tearDown written
against a successful setUp -- e.g. one that reads self.env -- then
raises its own exception (typically AttributeError, since the
attribute setUp would have assigned was never set). That second
exception escaped _runTest uncaught: unlike the original setUp/test
failure, it bypasses handleFailure entirely.

In the parallel coordinator this kills the whole worker process
instead of failing just the one test. Because RLTest schedules
different methods of the same test class across different workers, a
single flaky setUp (e.g. a transient env-startup connection refusal)
can take down every worker in the pool near-simultaneously, and the
coordinator has no way to tell a genuine "no more processors are
alive" condition apart from one that will resolve once a worker
finishes flushing its queued output.

Track whether setUp completed, skip tearDown if it didn't, and route
any exception tearDown itself raises through handleFailure like any
other test failure instead of letting it propagate.
@kei-nan kei-nan self-assigned this Sep 27, 2026
@kei-nan
kei-nan requested a review from GuyAv46 September 27, 2026 11:43
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 90.90909% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 46.60%. Comparing base (02cfbb9) to head (7d69fcc).
⚠️ Report is 24 commits behind head on master.

Files with missing lines Patch % Lines
RLTest/__main__.py 90.90% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master     #257       +/-   ##
===========================================
+ Coverage   32.46%   46.60%   +14.14%     
===========================================
  Files          17       18        +1     
  Lines        2597     2755      +158     
===========================================
+ Hits          843     1284      +441     
+ Misses       1754     1471      -283     
Flag Coverage Δ
unittests 46.60% <90.90%> (+14.14%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kei-nan
kei-nan merged commit 1ceb398 into RedisLabsModules:master Sep 27, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants