fix(nwis): name the peak date format instead of making pandas guess - #403
Open
thodson-usgs wants to merge 1 commit into
Open
fix(nwis): name the peak date format instead of making pandas guess#403thodson-usgs wants to merge 1 commit into
thodson-usgs wants to merge 1 commit into
Conversation
`preformat_peaks_response` called `pd.to_datetime(df["peak_dt"],
errors="coerce")`. On a column mixing zero-filled and parseable dates pandas
cannot find one format that fits both, so it falls back to per-element
`dateutil` parsing and says so on stderr:
UserWarning: Could not infer format, so each element will be parsed
individually, falling back to `dateutil`.
That mix was rare until DOI-USGS#395, which stopped dropping peaks whose date NWIS
only partly knows. Keeping them made the mix the normal case for any long
historical record, so the warning now fires on ordinary
`get_discharge_peaks` calls -- library noise on stderr for behaviour the
library intends.
The format is not actually ambiguous. The RDB header types `peak_dt` as
`10d`, a ten-character date, and any time is carried separately in `peak_tm`,
so `%Y-%m-%d` is the only shape the column takes.
Verified identical, not assumed. Against the real
`tests/data/waterservices_peaks.txt` column plus zero-filled, blank,
malformed, and time-bearing values, `errors="coerce"` with and without
`format="%Y-%m-%d"` produce the same result on every row; only the warning
differs. A value carrying a time coerces to `NaT` either way.
The regression test asserts silence via `simplefilter("error")`, and was
confirmed to fail against the unfixed function before being kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JAEQqs7XzQHGQQi2KakuXD
This was referenced Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
preformat_peaks_responsecallspd.to_datetime(df["peak_dt"], errors="coerce").On a column mixing zero-filled and parseable dates pandas cannot find one format
that fits both, falls back to per-element
dateutilparsing, and says so:That mix was rare until #395, which stopped dropping peaks whose date NWIS only
partly knows. Keeping them made the mix the normal case for any long
historical record — so the warning now fires on ordinary
get_discharge_peakscalls. Library noise on stderr, for behaviour the library intends.
Found while reviewing #395 against the glossary; it is a separate concern from
that review, so it is a separate PR.
The fix
The format was never actually ambiguous. The RDB header types
peak_dtas10d— a ten-character date — and any time is carried separately inpeak_tm:So
%Y-%m-%dis the only shape the column takes, and naming it stops pandasguessing.
Verified identical, not assumed
Against the real
tests/data/waterservices_peaks.txtcolumn plus zero-filled,blank, malformed and time-bearing values:
YYYY-MM-DDThe warning fires only on the case #395 made normal. A value carrying a time
coerces to
NaTeither way, so nothing that parsed before stops parsing.The test can fail
test_preformat_peaks_response_is_quiet_on_partly_dated_peaksasserts silencewith
warnings.simplefilter("error"). It was run against the unfixedfunction first and confirmed to fail there:
1140 tests pass,
mypy --strictclean, all hooks green.Scope
Independent of #400 and #402. Touches
nwis.py, one new test, and a NEWS entry.🤖 Generated with Claude Code
https://claude.ai/code/session_01JAEQqs7XzQHGQQi2KakuXD