Repository navigation
Street address filter detects fewer address forms than Phileas (Java) #119
Description
Summary
The streetAddress filter in Python detects a much narrower set of addresses than the Phileas (Java) filter, so the same policy finds fewer addresses depending on which language runs it. Some of the gaps leave PII in redacted output: a PO box is not detected at all, and a unit such as Apt 4B is left out of the span and stays in the text.
#116 (fixing #115) brought the two in line on one point, that a match stays on one line. The rest of the pattern still differs.
Start after #116 merges: both change the same pattern, and this work keeps #116's tests.
Differences from Java
The Java filter is phileas-core/src/main/java/ai/philterd/phileas/services/filters/regex/StreetAddressFilter.java in philterd/phileas. Compared with phileas/filters/street_address_filter.py:
| Python | Java | |
|---|---|---|
PO boxes (PO Box 1234, P.O. Box 56, Post Office Box 789) |
Not detected | Detected by a second pattern |
| Street types | 23 forms | 69 forms, adding Highway/Hwy, Parkway/Pkwy, Square/Sq, Turnpike/Tpke, Expressway/Expy, Freeway/Fwy, Crossing/Xing, Crescent/Cres, Plaza/Plz, Landing/Lndg, Gardens/Garden/Gdns, Commons, Manor/Mnr, Ridge/Rdg, Point/Pt, Grove/Grv, Alley/Aly, Cove/Cv, Bend/Bnd, Loop, Pike, Path, Mews, Row, Run, Walk, Close |
Unit (Apt 4B, Suite 200, Ste. 5, # 12, Bldg C, Fl 3, Rm 101) |
Not part of the span | Folded into the span when present |
Directionals (123 N Main St, 123 Main St NW) |
Leading one counts as a name word; trailing one is not in the span | Optional leading and trailing directional |
| House number | 1 to 5 digits | 1 to 6 digits, an optional range (100-102), an optional trailing letter (123A) |
| Street name words | 1 to 4, from [A-Za-z0-9#.,\-] |
1 to 5, from [A-Za-z0-9'.-] (apostrophes allowed, commas not) |
Period after the street type (St.) |
Not in the span | Included |
| Confidence | 1.0 (the _detect_patterns default) |
0.85 for both patterns |
Suggested approach
Port the Java patterns (the address pattern built from DIRECTIONAL, STREET_TYPE, and UNIT, plus the PO box pattern) into street_address_filter.py, keeping the parts joined by [ \t]+ so matches stay on one line. Port the Java filter's tests where they apply, so the two suites check the same cases.
Acceptance Criteria
- The address pattern in
street_address_filter.pymatches the Java pattern: the same street types, directionals, unit designators, house-number forms, and name-word count and character set. -
PO Box 1234,P.O. Box 56, andPost Office Box 789are detected. - A unit is included in the span:
123 Main St Apt 4Byields one span,123 Main St Apt 4B. -
123 N Main St,123 Main St NW,100-102 Main St, and123A Main Stare each detected in full. - A street type added in Java is detected, with tests covering at least Highway, Parkway, Plaza, and Loop.
- A street name with an apostrophe (
12 O'Connor Way) is detected in full. - Matches still do not cross a line break, and the tests added for Street address filter matches across line breaks and reads "Dr." in a greeting as a street suffix #115 still pass.
- Spans carry the same confidence as Java (0.85), or the issue records why Python differs.
- The tests from the Java
StreetAddressFilterTestthat apply are ported, and the full suite passes. - The
streetAddresssection ofdocs/filters.mdlists what is detected (addresses with units and directionals, and PO boxes). -
RELEASE_NOTES.mddescribes the change under Unreleased as a parity port of the Phileas (Java) filter.
Activity