Skip to content

chore: keep what the pylint evaluation found, without keeping pylint - #399

Draft
thodson-usgs wants to merge 1 commit into
DOI-USGS:mainfrom
thodson-usgs:chore/pylint-findings
Draft

chore: keep what the pylint evaluation found, without keeping pylint#399
thodson-usgs wants to merge 1 commit into
DOI-USGS:mainfrom
thodson-usgs:chore/pylint-findings

Conversation

@thodson-usgs

Copy link
Copy Markdown
Collaborator

The leftovers from the pylint evaluation behind #398. Recommendation: do not adopt pylint. This PR carries the four things it found that are worth keeping, none of which need the tool, plus a note in CONTRIBUTING.md so nobody re-derives the measurement.

Independent of #398 — no overlapping hunks — though both touch pyproject.toml in different sections.

Why not pylint

Measured at pylint 4.0.8 over dataretrieval: 635 findings, of which 419 are unused-argument — the check cannot see through the locals() forwarding the collection getters use, so it calls 419 documented, tested keyword arguments unused where ruff's ARG001 finds ten. Fourteen findings have no ruff rule at all (too-few-public-methods ×5, too-many-instance-attributes ×2, too-many-lines ×3, duplicate-code ×3, unused-wildcard-import ×1); duplication is already pyscn's job, and the rest are thresholds this package has no reason to want. It runs ~100× slower than ruff and has no per-file message control.

The one capability ruff lacks is import resolution. That argument does not survive contact with the existing CI: the package-artifact job already installs the wheel with only its declared dependencies, outside the checkout. Strengthening that job — below — reaches more modules than pylint's import-error would, and proves runtime importability rather than static resolution.

One check genuinely has no other home: an except tuple whose members shadow one another. Ruff's B014 misses it because it knows alias pairs, not subclass relationships. CONTRIBUTING.md now records it as a one-line command rather than a dependency:

uvx --with pandas --with httpx --with anyio pylint -j0 --disable=all \
  --enable=bad-except-order,overlapping-except \
  --load-plugins=pylint.extensions.overlapping_exceptions dataretrieval tests

I ran it against this branch: 10.00/10.

What it found that is worth keeping

  1. nwis._parse_json_or_raise caught (ValueError, JSONDecodeError). JSONDecodeError subclasses ValueError, so the tuple advertised a distinction that cannot exist. Behavior-identical; the now-orphaned import goes with it.
  2. A dead async def parse_response in tests/waterdata_progress_test.py, left from an earlier revision — the test passes parse_sync. Ruff's F841 misses it because it covers unused variables, not nested function definitions.
  3. mypy's possibly-undefined. Catches a name bound on only some paths — a runtime NameError on the branch nobody tested. mypy ships the check but leaves it off even under strict, and neither ruff nor the suite models it. Clean on all 61 modules today, so it lands as a ratchet in a tool already in CI and pre-commit.
  4. A stronger wheel smoke test. The heredoc named six imports and reached most of the tree by luck of what those names pull in. It now walks every module the installed wheel ships and asserts the walk saw all of them — walk_packages skips the subtree under a package it cannot import and says nothing when it does, so without that assertion the loop can pass vacuously. It also pins the optional-dependency guard: without geopandas, importing nldi must fail with the message that says how to fix it, not a bare ModuleNotFoundError.

Honest limits

  • The walk buys three modules today (_version, ogc.interruptions, ogc.retry), and none of them imports a third-party package — so it does not advance the stated dependency-catching purpose right now. Its value is that it cannot go stale: a future module with a new import is covered without anyone remembering to edit the heredoc. Judge it on that, not on today's count.
  • It does not catch a subpackage missing from the wheel (the NEWS 08/02/2026 bug), because both sides of the comparison derive from the installed wheel. Comparing against the checkout instead would guard that too — deliberately left as a follow-up rather than shipped untested.
  • ADR 0005 tension, flagged rather than assumed: the ADR freezes nwis.py to "compatibility, security, and correctness fixes", and narrowing an except clause is a cleanup, none of the three. In practice the module has absorbed several refactor commits recently, so the freeze reads as "add no new capabilities" — but it deserves your explicit nod. The CONTRIBUTING one-liner also depends on this fix; without it the documented command reports W0714.
  • No NEWS.md entry: nothing here is user-visible.

Noticed, not touched (pre-existing, adjacent)

  • [project.optional-dependencies].type-check pins bare "mypy" while .pre-commit-config.yaml's comment claims it is "pinned to the same major as CI's mypy<2" — there is no <2 anywhere. A one-token change would make that comment true. Relevant here because an opt-in error code is exactly the sort whose scope widens between releases.
  • The [[tool.mypy.overrides]] block for anyio justifies follow_imports = "skip" with "Under our python_version = "3.9" target"; the setting three lines above is 3.10. I checked — it is clean without the override.

Verification

The CI smoke script was extracted verbatim, run against a freshly built wheel in a throwaway venv with only declared dependencies: passed, 60 modules walked, nldi guard verified. ruff check + format --check clean · mypy --strict + possibly-undefined 62 files clean · 995 tests pass · import-linter 8/8. The commit passed the full pre-commit suite.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T6MVcko4gh68LieGWxUnrR

Pylint was evaluated as a periodic second opinion and not adopted: 87% of
what it reports are checks ruff also implements, and its `unused-argument`
check cannot see through the `locals()` forwarding the collection getters
use. CONTRIBUTING records the measurement so nobody re-derives it, and the
one check with no other home -- an `except` tuple whose members shadow one
another, which ruff's B014 misses because it knows alias pairs rather than
subclass relationships -- is written down as a one-line command instead of
a dependency.

Four things it found that are worth keeping:

- `nwis._parse_json_or_raise` caught `(ValueError, JSONDecodeError)`, and
  `JSONDecodeError` subclasses `ValueError`, so the tuple advertised a
  distinction that cannot exist.
- `tests/waterdata_progress_test.py` held an `async def parse_response`
  left from an earlier revision; the test passes `parse_sync`.
- mypy's `possibly-undefined` catches a name bound on only some paths --
  a runtime `NameError` on the branch nobody tested. mypy ships it but
  leaves it off even under `strict`, and nothing else in the stack models
  it. The package is clean today, so it lands as a ratchet.
- The wheel smoke test named six imports and reached most of the tree by
  luck of what those names pull in. It now walks every module the installed
  wheel ships, and asserts the walk saw all of them -- `walk_packages` skips
  the subtree under a package it cannot import and says nothing when it
  does. It also pins the optional-dependency guard: without geopandas,
  importing `nldi` must fail with the message that says how to fix it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6MVcko4gh68LieGWxUnrR
@thodson-usgs
thodson-usgs force-pushed the chore/pylint-findings branch from b42e12e to e1901c3 Compare August 30, 2026 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant