Fix variable uprating when the uprating parameter is undefined, and pick the latest earlier period

[MaxGhenis]

Problem

When a variable with uprating has no value for the requested period, Simulation._calculate takes its latest known earlier period and multiplies that value by uprating_parameter(this_period.start) / uprating_parameter(latest_known_period.start). That step had two bugs.

1. TypeError when the uprating parameter has no value at the earlier instant. A parameter returns None before its first value. If a caller supplies an uprated input only for a year the index doesn't reach, any later year that reads it divides by None. The value_in_last_period == 0 guard doesn't catch that. policyengine-us backdates parameters only to 2015-01-01, so this household fails on main:

from policyengine_us import Simulation
sim = Simulation(situation={"people": {"p": {"age": {2015: 40}, "employment_income": {2015: 30000}, "tax_exempt_interest_income": {2013: 5000}}}, "households": {"h": {"members": ["p"], "state_code": {2015: "TX"}}}})
sim.calculate("income_tax", 2015)
# TypeError: unsupported operand type(s) for /: 'float' and 'NoneType'

2. The wrong base period. The code picked known_periods[np.argmax(start_instants)], where start_instants is a filtered list: earlier periods of the variable's unit only. Whenever a later period came first in known_periods, that position pointed at the wrong period, and the result was silently wrong:

• Inputs {2020: 999, 2015: 5000} requested for 2018 return 999: the 2020 value deflated to 2018. The correct answer is 5000 uprated from 2015.
• With an input only for 2018, computing 2016 first caches the default there. A 2017 request then returns 4,545, the 2018 input deflated. Requesting 2017 directly returns 0, so the answer depended on evaluation order. A lagged-year lookup in a formula is enough to set this up.

Fix

• Undefined index. Where the uprating parameter has no value, it is held flat: before its first value it takes that first value, and after an explicit null it keeps the last value before the null (_uprating_index_value). A value known for a period the index doesn't reach is carried over unchanged until the index starts, then uprated with it. This matches extending the parameter's earliest value backward, which is what policyengine-us's backdate_parameters already does down to 2015. The factor falls back to 1 only if the parameter has no non-null value at all, alongside the existing zero-base guard.
• Base period. latest_known_period = max(earlier_known_periods, key=lambda p: p.start), taken from the filtered list itself.
• Variable.uprating now documents the rule.

Why hold the index flat rather than the alternatives

• Factor 1 whenever either value is missing would freeze a 2014 input at its 2014 level forever while a 2015 input is fully uprated. That is a jump at the index's start date. Holding the index flat gives the same result whether the index is defined back to the known year or not.
• Falling back to the default would silently turn a value the caller supplied into zero.
• Raising an error would leave these households uncomputable, even though a defined rule exists.
• Fixing it in the country package (defining uprating parameters further back) moves the floor without removing it: any earlier input year hits the same crash. It would also change the parameter history the API serves; policyengine-us#9626 argues against backdating to 2013 for that reason. Core is the only place that sees every uprated variable, and other country packages have the same exposure.

Invariants

For every known period K before a requested period R, with the known value v:

1. No crash, finite result. Uprating never raises when the uprating parameter exists and has any non-null value.
2. Unchanged where the index is defined. If the parameter has a nonzero value at both K and R, the result is exactly v × p(R) / p(K), as before.
3. Carry-over where the index is undefined. If the parameter has no value at K or at R (for instance, both precede its first value), the result equals v held with the flat index: v × p̃(R) / p̃(K), where p̃ holds the index flat.
4. Agreement with explicit backdating (differential). The result equals what you get when the index's first value is explicitly extended back over K with Parameter.update and uprating runs through the ordinary defined-index path.
5. Path independence. Computing every intermediate year first (each then known) gives the same final value as uprating straight from K.
6. Storage-order independence. For any insertion order of the known periods, R uprates from the latest known period before R.

Tests

tests/core/variables/test_variable_uprating.py (new, 233 cases):

• Regression cases for each failure above: pre-index known year, both years pre-index, monthly variable, later period stored first, cached default before the input.
• An explicit null gap from Parameter.update on a pre-series period, an index with no values, and the helper directly.
• Invariants 2–6, checked exhaustively on a grid:
  • every known/requested pair over 2008–2022 (105 pairs, against an index that starts in 2015);
  • path independence from every known year 2008–2015;
  • all 6 storage orders of three known years inside the index's range, at three requested years. Keeping the years inside the range makes this case test only the base-period choice. I used an exhaustive grid rather than Hypothesis, which isn't a dependency here; the domain is small enough to enumerate.

The module runs directly against master's simulation.py; the helper test imports the helper locally. There, 174 of 233 cases fail:

• the pre-index pairs (TypeError);
• 5 of 6 storage orders, plus the stored-first and cached-default cases (wrong values, e.g. 999 and 4,545.45);
• the helper test.

The 59 that pass on master are the ascending storage order and the pairs where the index is defined at both years; they confirm invariant 2.

Run locally:

• pytest tests: 945 passed, 4 skipped, 1 xfailed.
• Country-template YAML: 39 passed.
• ruff format --check: clean.

Downstream

• policyengine-us pins policyengine-core>=3.30.1, and its CI resolves the newest core before testing (uv lock --upgrade-package policyengine-core). This change reaches pe-us CI and pip installs on release.
• Require policyengine-core 3.32.8 and test uprated inputs supplied only before 2015 policyengine-us#9650 (draft) pins core to this branch's first two commits (753cc4e) through [tool.uv.sources], so pe-us CI runs its full suite against the fix. Later commits here change only tests and docstrings. The draft's new test fails 39 of 40 on core 3.32.7 with the reported TypeError and passes 40 of 40 with this fix. After release it switches the pin to a version floor.
• In pe-us, 312 of 365 uprated input variables use an index that starts at the 2015 floor. That includes every default-uprated dollar input. A pre-2015-only input to any of them hits bug 1. The other 53 use CPI-U, which starts in 1913.

Review

An independent Opus 5.5 review (Subfleet, code reading only) returned approve with nits. I applied the nits in b95c86b: an isolated storage-order test, an open-ended null in the helper test, a module that runs against master, and docstring and changelog wording.

It traced dataset loading. h5py iterates years in name order, so standard multi-year h5 datasets load ascending, and bug 2 never touched their microsim outputs. Old and new code differ only where master was wrong or order-dependent: unsorted flat-file columns, disk-spilled periods, cached defaults, and stray year periods on monthly variables.

It also flagged an existing issue, the same on master. get_known_periods strips branch names, so a period stored only on a non-ancestor branch could still yield None * factor. That's tracked as a follow-up, not changed here.

axiom: n/a: engine fix to variable uprating in policyengine-core; no policy rule changes

🤖 Generated with Claude Code