Skip to content

feat(redact): remove the text in a region, and refuse what it cannot - #68

Merged
tannevaled merged 2 commits into
mainfrom
feat/redact-the-text-not-the-view
Nov 7, 2026
Merged

tannevaled merged 2 commits into
mainfrom
feat/redact-the-text-not-the-view

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Closes #67.

What

Doc.Redact takes the text inside a region out of the page's content
stream
, and paints the region over.

d.Redact(ops.Redaction{Page: 1, Quads: []ops.Quad{ops.QuadOf(15, 95, 85, 115)}})

⛔ The only test that decides whether this is a redaction

ISO 32000-1 §12.5.6.23, on applying one:

Conforming readers that support redaction annotations shall provide a
mechanism for applying content removal, and they shall remove all traces
of the specified content. If a portion of an image is contained in a
redaction region, that portion of the image data shall be destroyed;
clipping or image masks shall not be used to hide that data.

A black rectangle over a paragraph is the standard data leak: the text is still
in the stream, so any copy and paste — or pdfops text — reads it back from
under the box. So the test that matters is not about how the page looks, it is
that the redacted string does not appear in the bytes that come out, checked
both in the decoded content stream and through extract.Text.

With the blanking disabled, that test fails and prints the leak itself:

page 1 still holds the redacted string: "q\n\nBT /F1 10 Tf 20 150 Td (public) Tj
ET BT /F1 10 Tf 20 100 Td (secret) Tj ET\nQ\n q 0 0 0 rg 15 95 70 20 re f Q"

— the fill painted over text that is still there, which is the document this
feature exists to prevent.

How

The operator that drew the text is overwritten with spaces of its own
length
. That does two jobs at once: the string is gone from the stream, and
every offset after it is unmoved, so the text that stays does not shift
(TestWhatWasKeptDidNotMove compares the kept run's x, y and width before and
after). It is the same technique recolourContent uses a file away, and for one
of the same reasons.

Which bytes to overwrite comes from extract.Run.Op
(extract#31, v0.10.0). Deciding
which glyphs are inside a region is work extract already does — the text
matrix, the font metrics, all of it — and a second implementation here would cut
the document somewhere other than where the caller was told the text was.

⛔ It refuses rather than half-redacts

A caller who is not told believes the document safe to send. So Redact
returns an error, naming the page and the reason, when the region holds:

what why
text inside a Form XObject the bytes are not the page's, and an XObject is shared — cleaning one page would change every page that draws it. RecolourText stops at the same wall
an image the spec is explicit that the pixels must be destroyed, not covered; re-encoding an image with them overwritten is not in this release
an annotation its own /Contents can hold the very words being redacted
an operator only part of whose text is in the region blanking it would remove text nobody asked about, and the survivors cannot be re-emitted: the stream holds font codes, and Unicode does not round-trip back through a CID font's map

And it refuses when Redact is called, not at Bytes — by then the caller
has been promised a redacted document by everything in between.

Text in a form that the region does not reach is fine, and tested: the page is
redacted and the form left alone.

Two write-path traps, both now controlled

  1. decorate threw away the rewritten stream. It took the source's content
    parts whenever it stamped a page. The file already carried a comment about
    that costing a recolouring its colour; for a redaction it would put the
    removed text back and draw the fill over it — a document that looks redacted.
    TestARedactionSurvivesAStamp redacts and watermarks the same page and then
    looks for the string. Without the fix both that test and the plain case
    fail, because the fill always routes through decorate.
  2. A silent fallback would have leaked. redactedSpans returning "no spans"
    when a page cannot be read would blank nothing and still paint the fill. It
    returns the error, and Doc.Bytes fails the whole write rather than hand
    back such a file — TestARedactionThatCannotBeAppliedFailsTheWrite checks
    that no bytes come back with the error, and that the next write recovers.

The fill goes through borrowedExtras rather than being appended to the
rewritten bytes, because a page's content may end inside a clipping path — which
would clip the fill away without a trace — or with a colour or transform still
set. decorate wraps the page in a save and a restore.

A page that cannot be read is also where a recolouring and a redaction part
company: for the first it is a lost colour and the page is left alone, for the
second the write fails. Both are tested through the same seam.

Checks

go vet and the full suite clean; 392 functions at 100.0% by
go tool cover -func, which is the instrument the gate reads.

gofmt -l reports addfields_test.go and cmd/pdfops/addfields.go, which are
unformatted on main and not touched here — ops' CI has no gofmt step, so
nothing is gated on them, but they are worth someone's commit.

Not in this release

RO overlays and OverlayText layout, redacting inside Form XObjects, image
pixel destruction, and a pdfops redact verb — each worth its own issue now
that the text path is proven.

🤖 Generated with Claude Code

tannevaled and others added 2 commits November 7, 2026 22:55
Doc.Redact takes the text inside a region OUT of the page's content
stream. Closes #67.

ISO 32000-1 §12.5.6.23 requires a conforming reader applying a
redaction to "remove all traces of the specified content", and says a
mask or a clip will not do. A black rectangle over a paragraph is the
standard data leak: the text is still in the stream for any copy and
paste — or `pdfops text` — to read back from under the box. So the test
that decides whether this is a redaction is not about how the page
looks: it is that the redacted string does not appear in the bytes that
come out, and a drawn rectangle fails it.

The operator that drew the text is overwritten with spaces of its own
length, which does two jobs at once: the string is gone from the stream,
and every offset after it is unmoved, so the text that stays does not
shift. extract.Run.Op (v0.10.0) says which bytes to overwrite — deciding
which glyphs are in a region is work extract already does, and a second
implementation of it here would cut the document somewhere other than
where the caller was told the text was.

⛔ It refuses rather than half-redacts, naming the page and the reason: a
caller who is not told believes the document safe to send. It refuses a
region holding text inside a Form XObject (shared bytes), an image (the
pixels must be destroyed, which is not in this release), an annotation
(its /Contents can hold the words), or an operator only part of whose
text is inside — blanking that would remove text nobody asked about, and
the survivors cannot be re-emitted because the stream holds font codes
and Unicode does not round-trip back through a CID map.

Two write-path traps, both now controlled by tests:

  - decorate() took the source's content parts whenever it stamped a
    page, which threw away a rewritten stream. The file already carried a
    comment about it costing a recolouring its colour; for a redaction it
    would have put the text back and drawn the fill over it. Both the
    plain and the watermarked case fail without the fix.
  - redactedSpans returning "no spans" on an unreadable page would blank
    nothing and still paint the fill. It returns the error instead, and
    Doc.Bytes fails the whole write rather than hand back a document
    that looks redacted.

The fill goes through borrowedExtras rather than being appended to the
rewritten bytes, because a page's content may end inside a clipping path
— which would clip the fill away without a trace — or with a colour or
transform still set.

392 functions at 100.0% by `go tool cover -func`, vet clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A run is the unit that can be removed, so a run that only overlapped the
region was being blanked entire. A producer that draws a whole line with
one Tj makes that line the run — so redacting one word of such a line
took the line, and nothing said so.

That is not a leak, which is why it was tempting to allow. It is the
other way a redaction betrays its caller: by removing what they did not
ask about.

runInside replaces runTouches in both the refusal and the collection of
spans to blank, and the comment on each says the other must agree: a run
the check passed over and the collection blanked would remove text
nobody asked about, and a run the check allowed and the collection
skipped would be left on the page under the fill.

The slack is two points, and it is a measurement rather than a fudge: a
caller mapping a lit rectangle back into page space arrives through
screen pixels and is a fraction of a point out in each direction, more
below 1:1. Two points is under half a line of ten-point text, so a run
from the line above cannot be swallowed by it.

With the test reverted to mere overlap, the new refusal is accepted and
a region well inside a run's own box takes the whole run — both fail.

It also moved the mixed-operator fixture, which was wrong and passing
for the wrong reason: the quad clipped "drop" at x 60..80 instead of
covering it, so that subtest was exercising this rule and not the one it
names.

393 functions at 100.0%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled

Copy link
Copy Markdown
Contributor Author

One more rule, pushed as ce16440: a run must be wholly inside the region, not merely touch it.

A run is the unit that can be removed, so a run that only overlapped the region was being blanked entire — and a producer drawing a whole line with one Tj makes that line the run. Redacting one word of such a line took the line, and nothing said so. Not a leak, which is why it was tempting to allow; the other way a redaction betrays its caller.

runInside now decides both the refusal and the set of spans to blank, and the comment on each says the other must agree: a run the check passed over and the collection blanked would remove text nobody asked about, and a run the check allowed and the collection skipped would be left on the page under the fill.

The slack is two points — a measurement, not a fudge. A caller mapping a lit rectangle back into page space arrives through screen pixels and is a fraction of a point out each way, more below 1:1; two points is under half a line of ten-point text, so a run from the line above cannot be swallowed. Both halves are tested: a region off by a point and a half still redacts, one well inside the run's own box is refused.

It also moved the mixed-operator fixture, which was passing for the wrong reason: its quad clipped drop at x 60..80 rather than covering it, so that subtest was exercising this new rule and not the one it names. Reverting to mere overlap now fails both the new refusal and the slack test.

@tannevaled
tannevaled merged commit c94407d into main Nov 7, 2026
1 check passed
@tannevaled
tannevaled deleted the feat/redact-the-text-not-the-view branch November 7, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redaction: remove the content, not the view of it

1 participant