From 772f920676091cda7c23eff214b354ae9ac520a2 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sat, 7 Nov 2026 22:55:17 +0100 Subject: [PATCH 1/2] feat(redact): remove the text in a region, and refuse what it cannot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Doc.Redact takes the text inside a region OUT of the page's content stream. Closes #67. ISO 32000-1 §12.5.6.23 requires a conforming reader applying a redaction to "remove all traces of the specified content", and says a mask or a clip will not do. A black rectangle over a paragraph is the standard data leak: the text is still in the stream for any copy and paste — or `pdfops text` — to read back from under the box. So the test that decides whether this is a redaction is not about how the page looks: it is that the redacted string does not appear in the bytes that come out, and a drawn rectangle fails it. The operator that drew the text is overwritten with spaces of its own length, which does two jobs at once: the string is gone from the stream, and every offset after it is unmoved, so the text that stays does not shift. extract.Run.Op (v0.10.0) says which bytes to overwrite — deciding which glyphs are in a region is work extract already does, and a second implementation of it here would cut the document somewhere other than where the caller was told the text was. ⛔ It refuses rather than half-redacts, naming the page and the reason: a caller who is not told believes the document safe to send. It refuses a region holding text inside a Form XObject (shared bytes), an image (the pixels must be destroyed, which is not in this release), an annotation (its /Contents can hold the words), or an operator only part of whose text is inside — blanking that would remove text nobody asked about, and the survivors cannot be re-emitted because the stream holds font codes and Unicode does not round-trip back through a CID map. Two write-path traps, both now controlled by tests: - decorate() took the source's content parts whenever it stamped a page, which threw away a rewritten stream. The file already carried a comment about it costing a recolouring its colour; for a redaction it would have put the text back and drawn the fill over it. Both the plain and the watermarked case fail without the fix. - redactedSpans returning "no spans" on an unreadable page would blank nothing and still paint the fill. It returns the error instead, and Doc.Bytes fails the whole write rather than hand back a document that looks redacted. The fill goes through borrowedExtras rather than being appended to the rewritten bytes, because a page's content may end inside a clipping path — which would clip the fill away without a trace — or with a colour or transform still set. 392 functions at 100.0% by `go tool cover -func`, vet clean. Co-Authored-By: Claude Opus 5 --- README.md | 52 ++++ doc.go | 11 + go.mod | 5 +- go.sum | 12 + redact.go | 363 +++++++++++++++++++++++++++ redact_test.go | 666 +++++++++++++++++++++++++++++++++++++++++++++++++ write.go | 44 +++- 7 files changed, 1145 insertions(+), 8 deletions(-) create mode 100644 redact.go create mode 100644 redact_test.go diff --git a/README.md b/README.md index 7e7249d..09bda2f 100644 --- a/README.md +++ b/README.md @@ -892,6 +892,58 @@ renders the page and looks at the pixels: And the marks are flagged Print, for the same reason a widget is: without it a highlight vanishes the moment somebody prints what they marked up. +## Redacting text: removing it, not covering it + +`Doc.Redact` takes the text inside a region **out of the page's content +stream**. ISO 32000-1 §12.5.6.23, on applying a redaction: + +> Conforming readers that support redaction annotations shall provide a +> mechanism for applying content removal, and they shall **remove all traces** +> of the specified content. If a portion of an image is contained in a +> redaction region, that portion of the image data **shall be destroyed**; +> clipping or image masks shall not be used to hide that data. + +⛔ **A black rectangle is not a redaction.** It is the standard data leak: the +text is still in the content stream, so anyone's copy and paste — or +`pdfops text` — reads it straight back out from under the box. So the test that +decides whether this is a redaction at all is not about how the page looks, it +is that **the redacted string does not appear in the bytes that come out**, and +a drawn rectangle fails it. + +The operator that drew the text is overwritten with spaces of its own length, +which does two things at once: the string is gone from the stream, and every +offset after it is still where it was, so the text that stays **does not move**. +`extract.Run.Op` says which bytes to overwrite — deciding which glyphs are +inside a region is work `extract` already does, and a second implementation of +it here would cut the document somewhere other than where the caller was told +the text was. + +The region is then painted over, because an area that was emptied and not +marked cannot be told from one that was always blank. That paint is a +*marking*, not the redaction: the text is already gone when it is drawn. + +⛔ **And it refuses rather than half-redacts**, naming the page and the reason. +A caller who is told can choose another region or another tool; a caller who is +not told believes the document safe to send. It refuses a region holding: + +| what | why | +| --- | --- | +| text inside a Form XObject | the bytes are not the page's, and an XObject is **shared** — cleaning one page would change every page that draws it. `RecolourText` stops at the same wall | +| an image | the spec is explicit that a mask or a clip will not do and the pixels must be destroyed; re-encoding an image with them overwritten is not in this release | +| an annotation | its own `/Contents` can hold the very words being redacted | +| an operator only **part** of whose text is inside the region | blanking it would remove text nobody asked about, and the surviving words cannot simply be re-emitted: the bytes in the stream are font codes, and Unicode does not round-trip back to them through a CID font's map. Keeping the survivors' original bytes and replacing the removed ones with an equivalent kern is the way, and it is next | + +The refusal happens when `Redact` is **called**, not when the document is +written — by the time of `Bytes` the caller has already been promised a +redacted document by everything in between. + +A redaction also survives the rest of the verbs. That is worth saying because +the write path once dropped a rewritten content stream whenever it stamped a +page: for a recolouring that loses a colour, but for a redaction it would put +the removed text back and draw the fill over it, so the document would **look +redacted and not be**. There is a test that redacts and watermarks the same +page and then looks for the string. + A document assembled from several files can still be given a form: two sources' forms cannot be merged — their fields may be named the same — but fields the caller *asked for* come from neither source and so are in no conflict. diff --git a/doc.go b/doc.go index 654d081..ccc1206 100644 --- a/doc.go +++ b/doc.go @@ -76,6 +76,11 @@ type Doc struct { // [Doc.Place]; zero means codec.DefaultMaxPixels. See [Doc.SetMaxPixels]. maxPixels int + // redactErr is set during a write when a page that was to be redacted + // could not be, and makes [Doc.Bytes] fail rather than hand back a + // document that looks redacted. Cleared at the start of every write. + redactErr error + // forms are the page-forms already written during the write in progress, // so a page drawn many times is embedded once. Made in [Doc.Bytes] and // dropped with it: a reader.Ref belongs to the writer that issued it, and @@ -113,6 +118,12 @@ type Page struct { // in. A pointer rather than a value with a flag beside it, because black // is a colour somebody asks for. recolour *[3]float64 + + // redactions are the regions of this page whose content is to be REMOVED + // from its stream, and the colour painted over each once it is gone. They + // are checked for what cannot be removed when Redact is called rather than + // when the document is written: see [Doc.Redact]. + redactions []Redaction } // Open reads a PDF file held in memory, with the empty password. diff --git a/go.mod b/go.mod index d8de2cb..318d3af 100644 --- a/go.mod +++ b/go.mod @@ -8,7 +8,7 @@ require ( github.com/go-ooxml/ooxml v0.7.0 github.com/go-ooxml/opc v0.2.1 github.com/go-pdfkit/document v0.2.0 - github.com/go-pdfkit/extract v0.9.0 + github.com/go-pdfkit/extract v0.10.0 github.com/go-pdfkit/forms v0.6.0 github.com/go-pdfkit/reader v0.10.0 github.com/go-pdfkit/render v0.69.1 @@ -22,6 +22,7 @@ require ( github.com/breml/rootcerts v0.3.7 // indirect github.com/dlclark/regexp2/v2 v2.5.2 // indirect github.com/dop251/goja v0.0.0-20260926152631-39ec2650adc9 // indirect + github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 // indirect github.com/evanw/esbuild v0.28.2 // indirect github.com/go-browserhttp/browserhttp v0.2.0 // indirect github.com/go-gfx/oksvg v0.2.0 // indirect @@ -30,7 +31,7 @@ require ( github.com/go-images/jpeg2000 v0.14.0 // indirect github.com/go-images/png v0.2.0 // indirect github.com/go-opentype/fonts v0.14.0 // indirect - github.com/go-opentype/opentype v0.16.1 // indirect + github.com/go-opentype/opentype v0.17.0 // indirect github.com/go-pdfkit/html2pdf v0.4.1 // indirect github.com/go-pdfkit/pdffont v0.4.0 // indirect github.com/go-pdfkit/pdfkit v0.20.0 // indirect diff --git a/go.sum b/go.sum index 3a7ab5c..41c37bc 100644 --- a/go.sum +++ b/go.sum @@ -8,6 +8,9 @@ github.com/dlclark/regexp2/v2 v2.5.2 h1:HAsucWRhsqcDzl6Ua9aR8JwYOTzrZyPrF0/FNxJV github.com/dlclark/regexp2/v2 v2.5.2/go.mod h1:avUrQvPaLz2DrFNHJF0taWAFFX2C1GMSSoeiqFjcBmU= github.com/dop251/goja v0.0.0-20260926152631-39ec2650adc9 h1:Ksd3Vil12b/ykT5NjGzd3qNc688cJB9OUXJZZChEktI= github.com/dop251/goja v0.0.0-20260926152631-39ec2650adc9/go.mod h1:u8yZRUavu+N4EnFFy6J5fVtjE7lEcZ2YyV2GcBXY9c8= +github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 h1:2tV76y6Q9BB+NEBasnqvs7e49aEBFI8ejC89PSnWH+4= +github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707/go.mod h1:qssHWj60/X5sZFNxpG4HBPDHVqxNm4DfnCKgrbZOT+s= +github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY= github.com/evanw/esbuild v0.28.2 h1:A2uETn4jrQTcXaT/shwTDTYBxDjl7fV7nXmUrJxfA2w= github.com/evanw/esbuild v0.28.2/go.mod h1:D2vIQZqV/vIf/VRHtViaUtViZmG7o+kKmlBfVQuRi48= github.com/go-browserhttp/browserhttp v0.2.0 h1:3RcjOQVHgqhSOmI5cfsURpexw/vetoDUoLXKzgTCSac= @@ -34,10 +37,14 @@ github.com/go-opentype/fonts v0.14.0 h1:WHw7k9gkHT1pu9B2yyvd0FWyQqn/4NKFQoKhvMYt github.com/go-opentype/fonts v0.14.0/go.mod h1:od2QcmdpKo7QCqo6Jb16y4GULk+h2Hhfq6x3xfvn62Q= github.com/go-opentype/opentype v0.16.1 h1:Hu2RKinqbd/WVSpRz6VrdKUPZUwPTZ4F9yx2HjZ5oCI= github.com/go-opentype/opentype v0.16.1/go.mod h1:aSxaPkjDzEnSAYB47s0oaHHa/sDViwW+aPlOz1A3dEg= +github.com/go-opentype/opentype v0.17.0 h1:yDLYLg/nUdfSSzm1Yg8t67racyFsamDEtS1gSR7xk1E= +github.com/go-opentype/opentype v0.17.0/go.mod h1:RujaDgR2h/ErgMkc5y/KYbX8NJtWw3yLRtVxbCfdvFQ= github.com/go-pdfkit/document v0.2.0 h1:4cdQLl4bcF0CxEw5eeZtkSGVimwgkR8jTMg5TKUEy6s= github.com/go-pdfkit/document v0.2.0/go.mod h1:Lgxm8Na/Nn2EI8Gmh+lpHhcPLwddOTG32GyL4GLki/c= github.com/go-pdfkit/extract v0.9.0 h1:gmyIFqZBxBBx6ccpUbF0Sw67jUmAp7q9vQ8mEsTyuFA= github.com/go-pdfkit/extract v0.9.0/go.mod h1:pUzGzNMx0WTsBOzfZ2XKT5ewqgv8cP/I6WpU4isPsmc= +github.com/go-pdfkit/extract v0.10.0 h1:9dezD5WqsMnQ9amqRaxADrjpf2pbe7TTMFUZwHXCTHU= +github.com/go-pdfkit/extract v0.10.0/go.mod h1:MNOx6Y0g//XSSx0lztuLUPDy4y5yubaNVsxJkVq3F6k= github.com/go-pdfkit/forms v0.6.0 h1:FJ5oHoIMMdbtVYWsyEEbZ8WFoJtSCdlYrjHE9hTSqxY= github.com/go-pdfkit/forms v0.6.0/go.mod h1:WZD05vH5x34pL7kOp4lVf1yOFsW7wpffj/ZqnHfsVUs= github.com/go-pdfkit/html2pdf v0.4.1 h1:zZ1sEoG6c3iJoEX4itG2Lul5z0BtyZ51lutzR4MC96M= @@ -70,10 +77,13 @@ github.com/go-widgets/painter v0.16.0 h1:j8FFObisySQChY2JiXS7Di2Mo4cqufIc0PDUnoD github.com/go-widgets/painter v0.16.0/go.mod h1:3wKWjaM8ZgEhir2HX0UDt8GemePKJ7l/yIjQb/vVSWk= github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM= github.com/goccy/go-yaml v1.19.2/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA= +github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8 h1:FKHo8hFI3A+7w0aUQuYXQ+6EN5stWmeY/AZqtM8xk9k= github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo= +github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A= github.com/klauspost/compress v1.18.7 h1:aUyZsS4kH3QTKurYhAOwAHxllVPnOthb3vPfnF1Ehjw= github.com/klauspost/compress v1.18.7/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= +github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek= github.com/refraction-networking/utls v1.8.2 h1:j4Q1gJj0xngdeH+Ox/qND11aEfhpgoEvV+S9iJ2IdQo= github.com/refraction-networking/utls v1.8.2/go.mod h1:jkSOEkLqn+S/jtpEHPOsVv/4V4EVnelwbMQl4vCWXAM= github.com/sergeymakinen/go-bmp v1.0.0 h1:SdGTzp9WvCV0A1V0mBeaS7kQAwNLdVJbmHlqNWq0R+M= @@ -84,6 +94,7 @@ github.com/srwiley/rasterx v0.0.0-20220730225603-2ab79fcdd4ef h1:Ch6Q+AZUxDBCVqd github.com/srwiley/rasterx v0.0.0-20220730225603-2ab79fcdd4ef/go.mod h1:nXTWP6+gD5+LUJ8krVhhoeHjvHTutPxMYl5SvkcnJNE= github.com/tannevaled/gobig2 v0.2.0 h1:yx97/ltljALrIWDFVqHjPUIjAp3bfUUhH8Z8UROvNLc= github.com/tannevaled/gobig2 v0.2.0/go.mod h1:X0S1H+N35kg6zYgVYRqGMsGj3C35zs+iA2r1MXyPcV4= +github.com/ulikunitz/xz v0.5.8/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU= github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E= github.com/yuin/goldmark v1.8.7 h1:XT/tNqCIbabh0ln675w4x3dv04kQBKi8NwYqi9IJaxI= @@ -99,5 +110,6 @@ golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= rsc.io/pdf v0.1.1 h1:k1MczvYDUvJBe93bYd7wrZLLUEcLZAuF824/I4e5Xr4= rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4= diff --git a/redact.go b/redact.go new file mode 100644 index 0000000..9c1d9ab --- /dev/null +++ b/redact.go @@ -0,0 +1,363 @@ +// Copyright (c) 2026, the go-pdfkit/ops authors +// All rights reserved. +// +// SPDX-License-Identifier: BSD-3-Clause + +package ops + +import ( + "errors" + "fmt" + "strings" + + "github.com/go-pdfkit/extract" + "github.com/go-pdfkit/reader" +) + +// A Redaction names a region of a page whose content is to be REMOVED, and the +// colour to paint in its place. +// +// ⛔⛔ REMOVED, NOT COVERED, and that is the entire point. ISO 32000-1:2008 +// §12.5.6.23 on applying a redaction: +// +// Conforming readers that support redaction annotations shall provide a +// mechanism for applying content removal, and they shall remove all traces +// of the specified content. If a portion of an image is contained in a +// redaction region, that portion of the image data shall be destroyed; +// clipping or image masks shall not be used to hide that data. +// +// A black rectangle drawn over a paragraph is the standard data leak: the text +// is still in the content stream, so anyone's copy and paste — or `pdfops text` +// — reads it straight back out from under the box. The test that matters, and +// the one a drawn rectangle fails, is that the redacted string does not appear +// anywhere in the bytes that come out. +// +// The region is in the page's own coordinates, up from the bottom left corner, +// in points — the same space [Quad] uses for a markup annotation. +type Redaction struct { + // Page is which page, counting from one. + Page int + // Quads is where on it. A quadrilateral per piece, as a markup annotation + // has: see [QuadOf] for the upright rectangle a line of text is. + Quads []Quad + // R, G and B are the colour painted over the region once the content is + // gone, each from zero to one. The spec leaves the region transparent when + // a redaction names no colour; this paints it, because an area that was + // emptied and not marked is one a reader cannot tell from one that was + // always blank. + R, G, B float64 +} + +// Redact removes the text inside each region and paints the region over. +// +// ⛔ IT REFUSES RATHER THAN HALF-REDACTS. A redaction that removed some of +// what it was asked to remove, and said nothing about the rest, would be worse +// than no redaction at all: the caller would believe the document safe to send. +// So this returns an error, naming the page and the reason, when a region holds +// anything it cannot take out: +// +// - text inside a Form XObject. Its bytes are not this page's, and an +// XObject is SHARED — editing it to clean one page would change every page +// that draws it. [Doc.RecolourText] stops at the same wall, and counts what +// it could not reach for the same reason. +// - an image. The spec is explicit that a mask or a clip will not do and the +// pixels themselves must be destroyed; until this can re-encode an image +// with those pixels overwritten, an image in the region is a refusal. +// - an annotation. Its /Contents can hold the very words being redacted. +// - a text-showing operator only PART of whose text is inside the region. +// Blanking it would take away text the caller did not ask about, and +// rewriting it cannot be done by re-emitting the surviving words: the +// bytes in the stream are font codes, and Unicode does not round-trip +// back to them through a CID font's map. Keeping the surviving pieces' +// original bytes and replacing the removed ones with an equivalent kern is +// the way, and it is not in this release. +// +// What it does remove it removes from the bytes: the operator is overwritten +// with spaces, so the string is gone from the content stream rather than +// hidden. +func (d *Doc) Redact(rs ...Redaction) error { + for _, r := range rs { + if r.Page < 1 || r.Page > len(d.pages) { + return fmt.Errorf("ops: there is no page %d to redact", r.Page) + } + if len(r.Quads) == 0 { + return fmt.Errorf("ops: the redaction of page %d names no region", r.Page) + } + for _, v := range []float64{r.R, r.G, r.B} { + if v < 0 || v > 1 { + return fmt.Errorf("ops: a colour component is from 0 to 1, not %v", v) + } + } + p := &d.pages[r.Page-1] + if p.src == nil { + return fmt.Errorf("ops: page %d was made rather than read, so it carries no text to redact", r.Page) + } + if err := d.checkRedactable(p, r); err != nil { + return err + } + p.redactions = append(p.redactions, r) + } + return nil +} + +// checkRedactable refuses a region this release cannot empty completely. +// +// ⛔ It runs when Redact is CALLED, not when the document is written. A caller +// that is told at once can choose another region or another tool; one told at +// Save has already been promised a redacted document by everything in between. +func (d *Doc) checkRedactable(p *Page, r Redaction) error { + runs, err := extractRuns(p.src, p.number) + if err != nil { + return fmt.Errorf("ops: page %d could not be read to redact it: %w", r.Page, err) + } + if err := unremovableText(r.Page, runs, r.Quads); err != nil { + return err + } + images, err := extractImages(p.src, p.number) + if err != nil { + return fmt.Errorf("ops: the images of page %d could not be read: %w", r.Page, err) + } + for _, im := range images { + // ⛔ DrawnWidth and DrawnHeight, not Width and Height: those two are the + // picture's own SAMPLES. Comparing a sample count with a region in + // points would weigh a 2048-pixel thumbnail against a 72-point box and + // refuse every page that carried a picture at all. + if overlaps(im.X, im.Y, im.X+im.DrawnWidth, im.Y+im.DrawnHeight, r.Quads) { + return fmt.Errorf("ops: page %d places an image across the region, and a redaction "+ + "must destroy the pixels rather than cover them, which this cannot yet do", + r.Page) + } + } + page, _ := p.src.Page(p.number) + if annots, ok := resolveArray(p.src, page.Get("Annots")); ok { + for _, a := range annots { + dict, ok := resolveDict(p.src, a) + if !ok { + continue + } + box := annotRect(p.src, dict.Get("Rect")) + if box == [4]float64{} { + continue + } + if overlaps(box[0], box[1], box[2], box[3], r.Quads) { + return fmt.Errorf("ops: page %d carries an annotation across the region, and its "+ + "own contents can hold the words being redacted", r.Page) + } + } + } + return nil +} + +// The two readings a refusal is decided from, replaceable so that a test can +// see what this does when a page cannot be read at all. The same shape as +// markup.go's `var now = time.Now`. +var ( + extractRuns = extract.Runs + extractImages = extract.Images + // pageContent is wrapped rather than used directly for the same reason: + // the difference between a recolouring and a redaction of a page that + // cannot be read is the difference between a lost colour and a document + // that must not be written, and a test has to be able to ask for both. + pageContent = func(src *reader.Document, page int) ([]byte, error) { + return src.PageContent(page) + } +) + +// unremovableText is the refusals that are decided from the runs alone, so +// that each can be put to it directly: a page that happens to hold a form, an +// unplaceable run and a half-covered operator all at once is a fixture nobody +// can read, and two of the three would never be the reason. +func unremovableText(page int, runs []extract.Run, quads []Quad) error { + for _, run := range runs { + if !runTouches(run, quads) { + continue + } + if run.InForm { + return fmt.Errorf("ops: page %d draws %q from a form XObject, whose bytes are "+ + "shared with every page that draws it, so this cannot be removed for one page alone", + page, summarise(run.Text)) + } + // ⛔ NOT a dead branch, and not one to drop. A run whose operation this + // cannot locate is a run that would be SKIPPED by redactedSpans — left + // on the page, under the fill, in a document the caller was told was + // redacted. extract sets a span for every run it draws today; this is + // what happens the day one of them does not. + if run.Op.Empty() { + return fmt.Errorf("ops: page %d draws %q from an operation this cannot locate in the stream", + page, summarise(run.Text)) + } + } + // A text-showing operator whose text is partly inside the region and partly + // outside cannot be blanked without taking away what was not asked for. + for span, group := range bySpan(runs) { + in, out := 0, 0 + for _, run := range group { + if runTouches(run, quads) { + in++ + } else { + out++ + } + } + if in > 0 && out > 0 { + return fmt.Errorf("ops: on page %d one operation (bytes %d..%d) draws both text to "+ + "redact and text to keep, and blanking it would remove both", + page, span.From, span.To) + } + } + return nil +} + +// bySpan groups the runs of a page by the operation that drew them, which is +// what says whether an operator can be blanked whole. +func bySpan(runs []extract.Run) map[extract.Span][]extract.Run { + out := map[extract.Span][]extract.Run{} + for _, r := range runs { + if r.Op.Empty() { + continue + } + out[r.Op] = append(out[r.Op], r) + } + return out +} + +// runTouches says whether a run's box meets any of the quads. The box is the +// baseline and the size above it, which is what a line of text occupies. +func runTouches(r extract.Run, quads []Quad) bool { + return overlaps(r.X, r.Y, r.X+r.Width, r.Y+r.Size, quads) +} + +// overlaps says whether a box meets any of the quads, each taken as its own +// bounding box. +func overlaps(x0, y0, x1, y1 float64, quads []Quad) bool { + for _, q := range quads { + qx0 := min(q.TopLeftX, q.BottomLeftX) + qx1 := max(q.TopRightX, q.BottomRightX) + qy0 := min(q.BottomLeftY, q.BottomRightY) + qy1 := max(q.TopLeftY, q.TopRightY) + if x0 < qx1 && qx0 < x1 && y0 < qy1 && qy0 < y1 { + return true + } + } + return false +} + +// summarise shortens a run's text for a message, so that an error about a +// secret does not print the whole of it. +func summarise(s string) string { + const most = 24 + r := []rune(s) + if len(r) <= most { + return s + } + return string(r[:most]) + "…" +} + +// redactContent blanks every text-showing operator whose text lies inside the +// region, and reports how many it blanked. +// +// ⛔ BLANKED WITH SPACES OF THE SAME LENGTH, which is what recolourContent does +// a file away and for the same reason: every offset after an edit is still the +// offset the scan reported, so the edits need no ordering rule. Here it does a +// second job — a span replaced by spaces no longer holds the string, so the +// text is gone from the bytes rather than merely unreferenced. +func redactContent(data []byte, spans map[extract.Span]bool) (out []byte, blanked int) { + if len(spans) == 0 { + return data, 0 + } + edited := make([]byte, len(data)) + copy(edited, data) + for span := range spans { + if span.From < 0 || span.To > len(edited) || span.To <= span.From { + continue + } + for i := span.From; i < span.To; i++ { + edited[i] = ' ' + } + blanked++ + } + return edited, blanked +} + +// redactedSpans is the operations to blank for a page's redactions. +// +// ⛔⛔ IT RETURNS THE ERROR RATHER THAN NO SPANS. Returning none would blank +// nothing and still paint the fill, which is a document that LOOKS redacted and +// is not — the one outcome worse than refusing. Doc.Bytes turns this into a +// failed write, so a caller gets no document at all rather than a leaky one. +func redactedSpans(src *reader.Document, page int, rs []Redaction) (map[extract.Span]bool, error) { + runs, err := extractRuns(src, page) + if err != nil { + return nil, err + } + spans := map[extract.Span]bool{} + for _, r := range rs { + for _, run := range runs { + if run.Op.Empty() || run.InForm { + continue + } + if runTouches(run, r.Quads) { + spans[run.Op] = true + } + } + } + return spans, nil +} + +// rewrittenContent is a borrowed page's own content stream with the redactions +// blanked out of it and the recolouring applied, or an error that must stop the +// write. +// +// ⛔ THE REDACTION GOES FIRST, on the bytes as they were. Its spans came from a +// scan of exactly these bytes, and a recolouring INSERTS operators — "q 1 0 0 +// rg " before every BT — so every offset after the first insertion has moved. +// Blanking is the other way round: it writes spaces over the same number of +// bytes, so the offsets a later scan reports are still the ones it reported. +// +// ⛔ And the two failures are not the same failure. A page whose stream cannot +// be read is, for a recolouring, a page left in its own colours — which is what +// this did before redaction existed and is no worse than a colour. For a +// redaction it is a document that must not be written at all. +func (d *Doc) rewrittenContent(p Page) (out []byte, rewritten bool, err error) { + if p.recolour == nil && len(p.redactions) == 0 { + return nil, false, nil + } + content, cerr := pageContent(p.src, p.number) + if len(p.redactions) > 0 { + spans, serr := redactedSpans(p.src, p.number, p.redactions) + if cerr != nil || serr != nil { + return nil, false, fmt.Errorf("ops: page %d was to be redacted and its own content "+ + "stream could not be read, so the redaction cannot be applied: %w", + p.number, errors.Join(cerr, serr)) + } + content, _ = redactContent(content, spans) + } + if cerr != nil { + return nil, false, nil // a recolouring alone: leave the page as it was + } + if p.recolour != nil { + var unreached int + content, unreached = recolourContent(content, *p.recolour) + d.unreached += unreached + } + return content, true, nil +} + +// redactionFill is the content that paints the emptied regions over, so a +// reader can see that something was taken out. +func redactionFill(rs []Redaction) string { + var sb strings.Builder + for _, r := range rs { + for _, q := range r.Quads { + x0 := min(q.TopLeftX, q.BottomLeftX) + x1 := max(q.TopRightX, q.BottomRightX) + y0 := min(q.BottomLeftY, q.BottomRightY) + y1 := max(q.TopLeftY, q.TopRightY) + if x1 <= x0 || y1 <= y0 { + continue + } + fmt.Fprintf(&sb, " q %g %g %g rg %g %g %g %g re f Q", + r.R, r.G, r.B, x0, y0, x1-x0, y1-y0) + } + } + return sb.String() +} diff --git a/redact_test.go b/redact_test.go new file mode 100644 index 0000000..88878d0 --- /dev/null +++ b/redact_test.go @@ -0,0 +1,666 @@ +package ops + +import ( + "errors" + "strings" + "testing" + + "github.com/go-pdfkit/extract" + "github.com/go-pdfkit/reader" +) + +// secretPage is a page saying two things, one of which is to be removed: a +// line at y 150 to keep and a line at y 100 to redact. +func secretPage(t *testing.T, body string, extra func(w *reader.Writer, res reader.Dict)) []byte { + t.Helper() + w := reader.NewWriter("1.7") + pagesRef := w.Reserve() + widths := make(reader.Array, 0, 224) + for i := 32; i < 256; i++ { + widths = append(widths, reader.Integer(500)) + } + res := reader.Dict{"Font": reader.Dict{"F1": w.Add(reader.Dict{ + "Type": reader.Name("Font"), "Subtype": reader.Name("Type1"), + "BaseFont": reader.Name("Helvetica"), "FirstChar": reader.Integer(32), + "LastChar": reader.Integer(255), "Widths": widths, + "Encoding": reader.Name("WinAnsiEncoding"), + })}} + page := reader.Dict{ + "Type": reader.Name("Page"), "Parent": pagesRef, + "Contents": w.Add(&reader.Stream{Dict: reader.Dict{}, Raw: []byte(body)}), + "Resources": res, + } + if extra != nil { + extra(w, res) + page["Resources"] = res + } + added := w.Add(page) + w.Put(pagesRef, reader.Dict{ + "Type": reader.Name("Pages"), "Kids": reader.Array{added}, "Count": reader.Integer(1), + "MediaBox": reader.Array{reader.Integer(0), reader.Integer(0), + reader.Integer(100), reader.Integer(200)}, + }) + root := w.Add(reader.Dict{"Type": reader.Name("Catalog"), "Pages": pagesRef}) + out, err := w.Finish(reader.Dict{"Root": root}) + if err != nil { + t.Fatal(err) + } + return out +} + +// twoLines is the ordinary fixture: one line to keep, one to remove. +const twoLines = "BT /F1 10 Tf 20 150 Td (public) Tj ET BT /F1 10 Tf 20 100 Td (secret) Tj ET" + +// overSecret is the region the second line occupies, with paper around it. +func overSecret() Redaction { + return Redaction{Page: 1, Quads: []Quad{QuadOf(15, 95, 85, 115)}} +} + +// redacted applies a redaction to the fixture and returns the file. +func redacted(t *testing.T, body string, r Redaction) []byte { + t.Helper() + d, err := Open(secretPage(t, body, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(r); err != nil { + t.Fatal(err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + return out +} + +// ⛔⛔ THE ONLY TEST THAT DECIDES WHETHER THIS IS A REDACTION. ISO 32000-1 +// §12.5.6.23 requires a conforming reader to "remove all traces of the +// specified content". A black rectangle drawn over a paragraph passes every +// test about how the page LOOKS and fails this one, because the text is still +// in the content stream for anyone's copy and paste to read back. +func TestTheRedactedStringIsGoneFromTheBytes(t *testing.T) { + out := redacted(t, twoLines, overSecret()) + for i, content := range contentsOf(t, out) { + if strings.Contains(content, "secret") { + t.Errorf("page %d still holds the redacted string: %q", i+1, content) + } + if !strings.Contains(content, "public") { + t.Errorf("page %d lost the text that was to be kept: %q", i+1, content) + } + } + // And the document read back the way a reader would read it says nothing + // of it either. + src, err := reader.Open(out) + if err != nil { + t.Fatal(err) + } + text, err := extract.Text(src, 1) + if err != nil { + t.Fatal(err) + } + if strings.Contains(text, "secret") { + t.Errorf("the page still reads back as %q", text) + } + if !strings.Contains(text, "public") { + t.Errorf("the page no longer says what it was to keep: %q", text) + } +} + +// ⛔ AND WHAT WAS KEPT DID NOT MOVE. Blanking an operator must not shift the +// text that stays — that is the difference between a redaction and a +// re-typesetting, and it is why the whole operation is overwritten with spaces +// of its own length rather than deleted. +func TestWhatWasKeptDidNotMove(t *testing.T) { + before, err := reader.Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + was, err := extract.Runs(before, 1) + if err != nil { + t.Fatal(err) + } + after, err := reader.Open(redacted(t, twoLines, overSecret())) + if err != nil { + t.Fatal(err) + } + now, err := extract.Runs(after, 1) + if err != nil { + t.Fatal(err) + } + if len(was) != 2 { + t.Fatalf("the fixture draws %d run(s), want 2", len(was)) + } + if len(now) != 1 { + t.Fatalf("the redacted page draws %d run(s), want 1", len(now)) + } + keep := was[0] + if now[0].Text != keep.Text { + t.Fatalf("what is left says %q, want %q", now[0].Text, keep.Text) + } + if now[0].X != keep.X || now[0].Y != keep.Y || now[0].Width != keep.Width { + t.Errorf("the kept line was at (%g,%g) %g wide and is now at (%g,%g) %g wide", + keep.X, keep.Y, keep.Width, now[0].X, now[0].Y, now[0].Width) + } +} + +// ⛔ THE REGION IS PAINTED OVER, because an area that was emptied and not +// marked cannot be told from one that was always blank — and a reader has to be +// able to see that something was taken out. The paint is a MARKING, not the +// redaction: the text is gone from the stream before it is drawn. +func TestTheRegionIsPaintedOver(t *testing.T) { + r := overSecret() + r.R, r.G, r.B = 0, 0, 0 + out := redacted(t, twoLines, r) + // A PDF counts up from the foot of the page and a raster down from the top: + // the region's y 95..115 on a 200pt page is rows 85..105. + if red, green, blue := pixelAt(t, out, 50, 95); blank(red, green, blue) { + t.Errorf("the middle of the region is bare paper (%d,%d,%d)", red, green, blue) + } + // The line that was kept still has its ink, and is not painted over. + inked := false + img := renderOf(t, out) + for y := 40; y < 52; y++ { + for x := 20; x < 60; x++ { + i := (y*img.W + x) * 4 + if !blank(img.Pix[i], img.Pix[i+1], img.Pix[i+2]) { + inked = true + } + } + } + if !inked { + t.Error("the line that was kept has no ink left on the page") + } +} + +// ⛔⛔ A REDACTION SURVIVES EVERYTHING ELSE THE DOCUMENT IS ASKED FOR. This is a +// control for a defect the write path already carries a comment about: the +// decoration step used to take the source's content parts whenever it stamped a +// page, which threw away a rewritten stream. For a recolouring that loses a +// colour. For a redaction it would put the removed text back and then draw the +// fill over it, so the document would LOOK redacted and not be — the worst +// possible ending. +func TestARedactionSurvivesAStamp(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(overSecret()); err != nil { + t.Fatal(err) + } + if err := d.Watermark("all", "DRAFT"); err != nil { + t.Fatal(err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + for i, content := range contentsOf(t, out) { + if strings.Contains(content, "secret") { + t.Errorf("page %d holds the redacted string after a watermark: %q", i+1, content) + } + } +} + +// ⛔⛔ IT REFUSES RATHER THAN HALF-REDACTS. Each of these is content a region +// can hold that this release cannot take out, and in every case the caller must +// be told — a caller who is not told believes the document safe to send. +func TestRedactingRefusesWhatItCannotRemove(t *testing.T) { + // A Form XObject: its bytes are not the page's, and it is shared. + t.Run("text inside a form", func(t *testing.T) { + const body = "BT /F1 10 Tf 20 150 Td (public) Tj ET q /X1 Do Q" + const inner = "BT /F1 10 Tf 20 100 Td (secret) Tj ET" + pdf := secretPage(t, body, func(w *reader.Writer, res reader.Dict) { + res["XObject"] = reader.Dict{"X1": w.Add(&reader.Stream{Dict: reader.Dict{ + "Type": reader.Name("XObject"), "Subtype": reader.Name("Form"), + "BBox": reader.Array{reader.Integer(0), reader.Integer(0), + reader.Integer(100), reader.Integer(200)}, + "Resources": res, + }, Raw: []byte(inner)})} + }) + assertRefused(t, pdf, overSecret(), "form XObject") + }) + + // An image: the spec says the pixels must be destroyed, not covered. + t.Run("an image across the region", func(t *testing.T) { + const body = "BT /F1 10 Tf 20 150 Td (public) Tj ET q 70 0 0 20 15 95 cm /Im1 Do Q" + pdf := secretPage(t, body, func(w *reader.Writer, res reader.Dict) { + res["XObject"] = reader.Dict{"Im1": w.Add(&reader.Stream{Dict: reader.Dict{ + "Type": reader.Name("XObject"), "Subtype": reader.Name("Image"), + "Width": reader.Integer(2), "Height": reader.Integer(2), + "ColorSpace": reader.Name("DeviceGray"), "BitsPerComponent": reader.Integer(8), + }, Raw: []byte{0, 64, 128, 255}})} + }) + assertRefused(t, pdf, overSecret(), "image") + }) + + // An annotation: its own contents can hold the words being redacted. + t.Run("an annotation across the region", func(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Mark(Markup{Page: 1, Kind: MarkHighlight, + Quads: []Quad{QuadOf(20, 98, 60, 112)}, Note: "the secret, again"}); err != nil { + t.Fatal(err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + assertRefused(t, out, overSecret(), "annotation") + }) + + // One operator drawing both what goes and what stays. + t.Run("an operator it would half-erase", func(t *testing.T) { + const body = "BT /F1 10 Tf 20 100 Td [(keep) -2000 (drop)] TJ ET" + // The region covers the second word only. + r := Redaction{Page: 1, Quads: []Quad{QuadOf(63, 95, 99, 115)}} + assertRefused(t, secretPage(t, body, nil), r, "would remove both") + }) +} + +// assertRefused checks that Redact says no, and says why. +func assertRefused(t *testing.T, pdf []byte, r Redaction, because string) { + t.Helper() + d, err := Open(pdf) + if err != nil { + t.Fatal(err) + } + err = d.Redact(r) + if err == nil { + t.Fatalf("the redaction was accepted although it cannot remove the %s", because) + } + if !strings.Contains(err.Error(), because) { + t.Errorf("the refusal says %q, which does not mention %q", err, because) + } + // And nothing was recorded, so a caller that ignored the error does not + // get a document that looks redacted. + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + for _, content := range contentsOf(t, out) { + if strings.Contains(content, "re f") && strings.Contains(content, "0 0 0 rg") { + t.Error("a refused redaction still painted its region") + } + } +} + +// A redaction has to name a page and a region, and a colour that is a colour. +func TestARedactionMustMakeSense(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + for _, c := range []struct { + name string + r Redaction + says string + }{ + {"no such page", Redaction{Page: 2, Quads: []Quad{QuadOf(0, 0, 10, 10)}}, "no page 2"}, + {"page zero", Redaction{Page: 0, Quads: []Quad{QuadOf(0, 0, 10, 10)}}, "no page 0"}, + {"no region", Redaction{Page: 1}, "names no region"}, + {"a colour that is not one", Redaction{Page: 1, + Quads: []Quad{QuadOf(0, 0, 10, 10)}, R: 2}, "from 0 to 1"}, + } { + t.Run(c.name, func(t *testing.T) { + err := d.Redact(c.r) + if err == nil { + t.Fatalf("%+v was accepted", c.r) + } + if !strings.Contains(err.Error(), c.says) { + t.Errorf("the refusal says %q, want it to mention %q", err, c.says) + } + }) + } +} + +// A page this package made rather than borrowed has no text of its own, and +// saying so beats appearing to redact one. +func TestAMadePageHasNoTextToRedact(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + d.Blank(100, 200) + err = d.Redact(Redaction{Page: 2, Quads: []Quad{QuadOf(0, 0, 10, 10)}}) + if err == nil { + t.Fatal("a made page accepted a redaction") + } + if !strings.Contains(err.Error(), "made rather than read") { + t.Errorf("the refusal says %q", err) + } +} + +// ⛔⛔ THE REFUSAL THAT MATTERS MOST, AND THE ONE NO FIXTURE CAN STAGE. A run +// whose operation cannot be located would be SKIPPED when the spans to blank +// are collected — left on the page, under the fill, in a document the caller +// was told was redacted. extract gives every run it draws a span today; this is +// what happens the day one of them does not, and the only way to ask is to hand +// the rule such a run directly. +func TestTextWithNoSpanIsRefusedRatherThanSkipped(t *testing.T) { + region := []Quad{QuadOf(15, 95, 85, 115)} + inside := extract.Run{Text: "secret", X: 20, Y: 100, Width: 30, Size: 10} + t.Run("no span", func(t *testing.T) { + err := unremovableText(1, []extract.Run{inside}, region) + if err == nil { + t.Fatal("a run this cannot locate was accepted for redaction") + } + if !strings.Contains(err.Error(), "cannot locate") { + t.Errorf("the refusal says %q", err) + } + }) + t.Run("in a form", func(t *testing.T) { + run := inside + run.Op, run.InForm = extract.Span{From: 1, To: 9}, true + err := unremovableText(1, []extract.Run{run}, region) + if err == nil { + t.Fatal("a run inside a form was accepted for redaction") + } + if !strings.Contains(err.Error(), "form XObject") { + t.Errorf("the refusal says %q", err) + } + }) + // ⛔ And a long secret is not printed in full by the message that refuses + // it: an error goes to a log, and a redaction's subject is the one string + // that must not. + t.Run("a long secret is shortened", func(t *testing.T) { + run := inside + run.Text = "a very long secret indeed, every word of it confidential" + err := unremovableText(1, []extract.Run{run}, region) + if err == nil { + t.Fatal("accepted") + } + if strings.Contains(err.Error(), "confidential") { + t.Errorf("the refusal printed the whole secret: %q", err) + } + if !strings.Contains(err.Error(), "…") { + t.Errorf("the refusal did not say it had shortened the text: %q", err) + } + }) + // A run outside the region is nobody's business, span or no span. + t.Run("outside the region", func(t *testing.T) { + run := inside + run.Y = 150 + if err := unremovableText(1, []extract.Run{run}, region); err != nil { + t.Errorf("a run outside the region was refused: %v", err) + } + }) +} + +// ⛔⛔ A REDACTION THAT CANNOT BE APPLIED FAILS THE WRITE. If the page's text +// cannot be read when the document is written, no operator is blanked — and the +// fill is still painted over the region, so the file would look redacted and +// hold every word. The caller gets the failure instead of the file. +func TestARedactionThatCannotBeAppliedFailsTheWrite(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(overSecret()); err != nil { + t.Fatal(err) + } + // It was readable when Redact was called, and is not when Bytes is: a + // document handed between the two is the case this guards. + was := extractRuns + extractRuns = func(*reader.Document, int) ([]extract.Run, error) { return nil, errTest } + t.Cleanup(func() { extractRuns = was }) + + out, err := d.Bytes() + if err == nil { + t.Fatal("a document whose redaction could not be applied was written anyway") + } + if out != nil { + t.Errorf("%d bytes were handed back with the failure", len(out)) + } + if !strings.Contains(err.Error(), "redaction cannot be applied") { + t.Errorf("the failure says %q", err) + } + // And the write recovers once the page can be read again. + extractRuns = was + if _, err := d.Bytes(); err != nil { + t.Errorf("the next write failed too: %v", err) + } +} + +// A page that cannot be read is a page that cannot be redacted, and the caller +// hears which reading failed. +func TestAPageThatCannotBeReadIsNotRedacted(t *testing.T) { + for _, c := range []struct { + name string + arm func(t *testing.T) + says string + }{ + {"its text", func(t *testing.T) { + was := extractRuns + extractRuns = func(*reader.Document, int) ([]extract.Run, error) { + return nil, errTest + } + t.Cleanup(func() { extractRuns = was }) + }, "could not be read to redact it"}, + {"its images", func(t *testing.T) { + was := extractImages + extractImages = func(*reader.Document, int) ([]extract.Image, error) { + return nil, errTest + } + t.Cleanup(func() { extractImages = was }) + }, "images of page 1 could not be read"}, + } { + t.Run(c.name, func(t *testing.T) { + c.arm(t) + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + err = d.Redact(overSecret()) + if err == nil { + t.Fatal("a page that could not be read was redacted anyway") + } + if !strings.Contains(err.Error(), c.says) { + t.Errorf("the refusal says %q, want it to mention %q", err, c.says) + } + }) + } +} + +// errTest is the failure the readings above are made to report. +var errTest = errors.New("this page is unreadable") + +// An entry of Annots that is not an annotation, and one with no rectangle, are +// stepped over rather than taken as covering the region — a document is free +// to hold either, and refusing to redact because of one would be refusing for +// no reason. +func TestNonsenseInAnnotsIsSteppedOver(t *testing.T) { + w := reader.NewWriter("1.7") + pagesRef := w.Reserve() + widths := make(reader.Array, 0, 224) + for i := 32; i < 256; i++ { + widths = append(widths, reader.Integer(500)) + } + font := w.Add(reader.Dict{ + "Type": reader.Name("Font"), "Subtype": reader.Name("Type1"), + "BaseFont": reader.Name("Helvetica"), "FirstChar": reader.Integer(32), + "LastChar": reader.Integer(255), "Widths": widths, + "Encoding": reader.Name("WinAnsiEncoding"), + }) + page := w.Add(reader.Dict{ + "Type": reader.Name("Page"), "Parent": pagesRef, + "Contents": w.Add(&reader.Stream{Dict: reader.Dict{}, Raw: []byte(twoLines)}), + "Resources": reader.Dict{"Font": reader.Dict{"F1": font}}, + "Annots": reader.Array{ + reader.Integer(7), // not an annotation at all + w.Add(reader.Dict{"Subtype": reader.Name("Text")}), // no Rect + }, + }) + w.Put(pagesRef, reader.Dict{ + "Type": reader.Name("Pages"), "Kids": reader.Array{page}, "Count": reader.Integer(1), + "MediaBox": reader.Array{reader.Integer(0), reader.Integer(0), + reader.Integer(100), reader.Integer(200)}, + }) + root := w.Add(reader.Dict{"Type": reader.Name("Catalog"), "Pages": pagesRef}) + pdf, err := w.Finish(reader.Dict{"Root": root}) + if err != nil { + t.Fatal(err) + } + d, err := Open(pdf) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(overSecret()); err != nil { + t.Fatalf("nonsense in Annots stopped the redaction: %v", err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + for i, content := range contentsOf(t, out) { + if strings.Contains(content, "secret") { + t.Errorf("page %d still holds the redacted string", i+1) + } + } +} + +// A span outside the stream is ignored rather than panicking: the bytes being +// edited and the bytes the spans were read from are the same page, so this +// cannot happen — and a redaction is the wrong place to find out by crashing. +func TestASpanOutsideTheStreamIsIgnored(t *testing.T) { + data := []byte("BT (x) Tj ET") + for _, span := range []extract.Span{ + {From: -1, To: 4}, + {From: 2, To: len(data) + 10}, + {From: 5, To: 5}, + {From: 6, To: 3}, + } { + out, blanked := redactContent(data, map[extract.Span]bool{span: true}) + if blanked != 0 { + t.Errorf("%+v blanked %d operation(s)", span, blanked) + } + if string(out) != string(data) { + t.Errorf("%+v changed the stream to %q", span, out) + } + } + // And no spans at all is the stream itself, not a copy of it. + if out, blanked := redactContent(data, nil); blanked != 0 || string(out) != string(data) { + t.Errorf("no spans gave %q, %d blanked", out, blanked) + } +} + +// ⛔ A RECOLOURING IS NOT A REDACTION, and an unreadable page is the one place +// the difference shows. A page whose content stream cannot be read is, for a +// recolouring, a page left in its own colours — no worse than a colour, and +// what this did before redaction existed. For a redaction the same page is a +// document that must not be written at all, which is the test above. +func TestAnUnreadablePageIsAColourLostAndARedactionRefused(t *testing.T) { + arm := func(t *testing.T) { + was := pageContent + pageContent = func(*reader.Document, int) ([]byte, error) { return nil, errTest } + t.Cleanup(func() { pageContent = was }) + } + t.Run("a recolouring leaves it alone", func(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.RecolourText("all", 1, 0, 0); err != nil { + t.Fatal(err) + } + arm(t) + out, err := d.Bytes() + if err != nil { + t.Fatalf("a page that could not be recoloured failed the write: %v", err) + } + // Left alone means left alone: the text is still there, in whatever + // colour it was. + found := false + for _, content := range contentsOf(t, out) { + if strings.Contains(content, "public") { + found = true + } + } + if !found { + t.Error("the page came back empty rather than unchanged") + } + }) + t.Run("a redaction is refused", func(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(overSecret()); err != nil { + t.Fatal(err) + } + arm(t) + if _, err := d.Bytes(); err == nil { + t.Fatal("a page whose stream could not be read was redacted anyway") + } + }) +} + +// A region of no area paints nothing, rather than a rectangle of no width that +// some readers draw as a hairline. +func TestARegionOfNoAreaPaintsNothing(t *testing.T) { + for _, q := range []Quad{QuadOf(10, 10, 10, 20), QuadOf(10, 10, 20, 10)} { + if got := redactionFill([]Redaction{{Page: 1, Quads: []Quad{q}}}); got != "" { + t.Errorf("a quad of no area painted %q", got) + } + } +} + +// Text inside a form that the region does not reach is left alone: the page is +// redacted, and the form — whose bytes are shared — is untouched. +func TestAFormAwayFromTheRegionIsLeftAlone(t *testing.T) { + const body = "BT /F1 10 Tf 20 100 Td (secret) Tj ET q /X1 Do Q" + const inner = "BT /F1 10 Tf 20 20 Td (elsewhere) Tj ET" + pdf := secretPage(t, body, func(w *reader.Writer, res reader.Dict) { + res["XObject"] = reader.Dict{"X1": w.Add(&reader.Stream{Dict: reader.Dict{ + "Type": reader.Name("XObject"), "Subtype": reader.Name("Form"), + "BBox": reader.Array{reader.Integer(0), reader.Integer(0), + reader.Integer(100), reader.Integer(200)}, + "Resources": res, + }, Raw: []byte(inner)})} + }) + d, err := Open(pdf) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(overSecret()); err != nil { + t.Fatalf("a form away from the region was refused: %v", err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + src, err := reader.Open(out) + if err != nil { + t.Fatal(err) + } + text, err := extract.Text(src, 1) + if err != nil { + t.Fatal(err) + } + if strings.Contains(text, "secret") { + t.Errorf("the page still says %q", text) + } + if !strings.Contains(text, "elsewhere") { + t.Errorf("the form's own text was lost: %q", text) + } +} + +// Redacting nothing leaves the page as it was, rather than writing a stream +// that merely looks the same. +func TestARegionWithNoTextInItChangesNoText(t *testing.T) { + out := redacted(t, twoLines, Redaction{Page: 1, Quads: []Quad{QuadOf(0, 0, 5, 5)}}) + src, err := reader.Open(out) + if err != nil { + t.Fatal(err) + } + text, err := extract.Text(src, 1) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"public", "secret"} { + if !strings.Contains(text, want) { + t.Errorf("a redaction over bare paper lost %q: the page reads %q", want, text) + } + } +} diff --git a/write.go b/write.go index 548294b..9b850ac 100644 --- a/write.go +++ b/write.go @@ -63,9 +63,17 @@ func (d *Doc) Bytes() ([]byte, error) { } } } + d.redactErr = nil for i, p := range d.pages { dicts[i] = d.buildPage(w, p, pagesRef, where, kept) } + // ⛔⛔ BEFORE ANYTHING IS HANDED BACK. A page whose redaction could not be + // applied has been built with its text still in it and the fill painted + // over the top — a document that looks redacted. The caller gets the + // failure instead of the file. + if d.redactErr != nil { + return nil, d.redactErr + } // The catalogue is rebuilt before the pages are put down, not after: what // it carries across may have something to write onto a page — the number @@ -206,12 +214,17 @@ func (d *Doc) buildBorrowedPage(w *reader.Writer, p Page, parent reader.Ref, whe // ⛔ The recolouring happens BEFORE the decoration, and the order matters: // a stamp's own colour is the stamp's business, so recolouring the page's // text must not reach the marks this document put on top of it. - if p.recolour != nil { - if content, err := p.src.PageContent(p.number); err == nil { - out, unreached := recolourContent(content, *p.recolour) - d.unreached += unreached - copied["Contents"] = w.Add(contentStream(out, nil)) + if out, rewritten, err := d.rewrittenContent(p); err != nil { + // ⛔ RECORDED, AND THE WHOLE WRITE FAILS ON IT. A page whose redaction + // could not be applied must not be written: the fill would be painted + // over text that is still there, and the caller would be holding a + // document that looks redacted. [Doc.Bytes] checks this before it + // returns anything. + if d.redactErr == nil { + d.redactErr = err } + } else if rewritten { + copied["Contents"] = w.Add(contentStream(out, nil)) } extra, resources := d.borrowedExtras(w, p, src) @@ -249,6 +262,19 @@ func (d *Doc) borrowedExtras(w *reader.Writer, p Page, src reader.Dict) ([]byte, content = append(content, stamp...) resources = mergeResources(resources, stampResources(fonts, alpha)) } + // ⛔ THE REDACTION FILL IS LAST, OVER EVERYTHING, and it goes through here + // rather than being appended to the page's own rewritten bytes. Two reasons, + // both of which were a way to lose it: the page's content may end inside a + // clipping path, which would clip the fill away without a trace, and it may + // end having left a colour or a transform set. decorate wraps the page in a + // save and a restore, so whatever is added here starts from a clean state. + // + // It is a MARKING, not the redaction: the text is already gone from the + // stream by the time this is drawn. What it does is tell a reader that + // something was taken out, which an emptied region otherwise cannot say. + if len(p.redactions) > 0 { + content = append(content, redactionFill(p.redactions)...) + } return content, resources } @@ -264,8 +290,14 @@ func (d *Doc) decorate(w *reader.Writer, p Page, src, copied reader.Dict, extra // that away: a page both recoloured and watermarked came back in its // original colours, and nothing failed. What is here is used when it is // a rewritten stream and the source's parts otherwise. + // + // ⛔ A REDACTION REWRITES IT TOO, and leaving it out of this test would be + // the same defect with a worse ending: a page both redacted and stamped + // would come back with the redacted text still in it, and the fill drawn + // over it would make the document look redacted. A recolouring that is + // undone is a colour; a redaction that is undone is a leak. var parts reader.Array - if rewritten, ok := copied["Contents"].(reader.Ref); ok && p.recolour != nil { + if rewritten, ok := copied["Contents"].(reader.Ref); ok && (p.recolour != nil || len(p.redactions) > 0) { parts = reader.Array{rewritten} } else { parts = contentParts(w, p.src, src) From ce164407b11006945e87d02df0d2f01359a48e78 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sat, 7 Nov 2026 23:00:57 +0100 Subject: [PATCH 2/2] fix(redact): a run must be WHOLLY inside the region, not merely touch it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run is the unit that can be removed, so a run that only overlapped the region was being blanked entire. A producer that draws a whole line with one Tj makes that line the run — so redacting one word of such a line took the line, and nothing said so. That is not a leak, which is why it was tempting to allow. It is the other way a redaction betrays its caller: by removing what they did not ask about. runInside replaces runTouches in both the refusal and the collection of spans to blank, and the comment on each says the other must agree: a run the check passed over and the collection blanked would remove text nobody asked about, and a run the check allowed and the collection skipped would be left on the page under the fill. The slack is two points, and it is a measurement rather than a fudge: a caller mapping a lit rectangle back into page space arrives through screen pixels and is a fraction of a point out in each direction, more below 1:1. Two points is under half a line of ten-point text, so a run from the line above cannot be swallowed by it. With the test reverted to mere overlap, the new refusal is accepted and a region well inside a run's own box takes the whole run — both fail. It also moved the mixed-operator fixture, which was wrong and passing for the wrong reason: the quad clipped "drop" at x 60..80 instead of covering it, so that subtest was exercising this rule and not the one it names. 393 functions at 100.0%. Co-Authored-By: Claude Opus 5 --- README.md | 1 + redact.go | 47 +++++++++++++++++++++++++++++++++++++++++- redact_test.go | 56 ++++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 101 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 09bda2f..fca9ce9 100644 --- a/README.md +++ b/README.md @@ -932,6 +932,7 @@ not told believes the document safe to send. It refuses a region holding: | an image | the spec is explicit that a mask or a clip will not do and the pixels must be destroyed; re-encoding an image with them overwritten is not in this release | | an annotation | its own `/Contents` can hold the very words being redacted | | an operator only **part** of whose text is inside the region | blanking it would remove text nobody asked about, and the surviving words cannot simply be re-emitted: the bytes in the stream are font codes, and Unicode does not round-trip back to them through a CID font's map. Keeping the survivors' original bytes and replacing the removed ones with an equivalent kern is the way, and it is next | +| text that **reaches outside** the region | a run is the unit that can be removed, so a producer drawing a whole line with one `Tj` makes that line the run — and redacting one word of it would take the line. That is not a leak, which is why it is tempting to allow; it is the other way a redaction betrays its caller, by removing what they did not ask about and not saying so. A region is allowed to miss a run's own box by up to two points, because a caller mapping a lit rectangle back into page space arrives through screen pixels | The refusal happens when `Redact` is **called**, not when the document is written — by the time of `Bytes` the caller has already been promised a diff --git a/redact.go b/redact.go index 9c1d9ab..d271a75 100644 --- a/redact.go +++ b/redact.go @@ -186,6 +186,19 @@ func unremovableText(page int, runs []extract.Run, quads []Quad) error { return fmt.Errorf("ops: page %d draws %q from an operation this cannot locate in the stream", page, summarise(run.Text)) } + // ⛔⛔ AND THE RUN MUST BE WHOLLY INSIDE THE REGION. A run is the unit + // that can be removed, so a run that merely OVERLAPS the region would + // be blanked entire — and a producer that draws a whole line with one + // Tj makes that line the run. Redacting one word of such a line would + // take the line. That is not a leak, which is why it is tempting to + // allow; it is the other way a redaction betrays its caller, by + // removing what they did not ask about and not saying so. + if !runInside(run, quads) { + return fmt.Errorf("ops: on page %d the text %q reaches outside the region, and the "+ + "operation that draws it can only be removed whole, which would take text "+ + "the region does not cover", + page, summarise(run.Text)) + } } // A text-showing operator whose text is partly inside the region and partly // outside cannot be blanked without taking away what was not asked for. @@ -226,6 +239,33 @@ func runTouches(r extract.Run, quads []Quad) bool { return overlaps(r.X, r.Y, r.X+r.Width, r.Y+r.Size, quads) } +// runSlack is how far a run's box may stand outside a quad and still count as +// inside it. +// +// ⛔ IT IS A MEASUREMENT, NOT A FUDGE. A caller that asks for the region a run +// occupies — the workbench, mapping a lit rectangle back into page space — +// arrives with a quad derived from screen pixels, so it is a fraction of a +// point out in each direction, and more at a zoom below 1:1. Two points is +// under half a line of ten-point text, so a run from the line above or below +// can never be swallowed by it, and the ordinary case of "this exact run" is +// not refused for a rounding error. +const runSlack = 2.0 + +// runInside says whether a run's box lies wholly within one of the quads, which +// is what makes the run's operator safe to blank whole. +func runInside(r extract.Run, quads []Quad) bool { + for _, q := range quads { + qx0 := min(q.TopLeftX, q.BottomLeftX) - runSlack + qx1 := max(q.TopRightX, q.BottomRightX) + runSlack + qy0 := min(q.BottomLeftY, q.BottomRightY) - runSlack + qy1 := max(q.TopLeftY, q.TopRightY) + runSlack + if r.X >= qx0 && r.X+r.Width <= qx1 && r.Y >= qy0 && r.Y+r.Size <= qy1 { + return true + } + } + return false +} + // overlaps says whether a box meets any of the quads, each taken as its own // bounding box. func overlaps(x0, y0, x1, y1 float64, quads []Quad) bool { @@ -295,7 +335,12 @@ func redactedSpans(src *reader.Document, page int, rs []Redaction) (map[extract. if run.Op.Empty() || run.InForm { continue } - if runTouches(run, r.Quads) { + // ⛔ INSIDE, the same test the refusal above uses. If these two + // ever disagreed, the pair would be the defect: a run the check + // passed over and this blanked would remove text nobody asked + // about, and a run the check allowed and this skipped would be left + // on the page under the fill. + if runInside(run, r.Quads) { spans[run.Op] = true } } diff --git a/redact_test.go b/redact_test.go index 88878d0..7e9856b 100644 --- a/redact_test.go +++ b/redact_test.go @@ -253,10 +253,62 @@ func TestRedactingRefusesWhatItCannotRemove(t *testing.T) { // One operator drawing both what goes and what stays. t.Run("an operator it would half-erase", func(t *testing.T) { const body = "BT /F1 10 Tf 20 100 Td [(keep) -2000 (drop)] TJ ET" - // The region covers the second word only. - r := Redaction{Page: 1, Quads: []Quad{QuadOf(63, 95, 99, 115)}} + // ⛔ The region covers the second word WHOLLY and the first not at all: + // "keep" is drawn at x 20..40, the kern moves the pen 20 points, and + // "drop" is drawn at x 60..80. A quad that clipped "drop" instead would + // be refused for reaching outside the region — a different rule, which + // is what this fixture said until it was measured. + r := Redaction{Page: 1, Quads: []Quad{QuadOf(58, 95, 85, 115)}} assertRefused(t, secretPage(t, body, nil), r, "would remove both") }) + + // ⛔⛔ AND A RUN THAT REACHES OUTSIDE THE REGION. A producer that draws a + // whole line with one Tj makes that line the run, so redacting one word of + // it would take the line. Not a leak — the other betrayal, where more goes + // than was asked and nothing says so. + t.Run("text that reaches outside the region", func(t *testing.T) { + const body = "BT /F1 10 Tf 20 100 Td (a whole line in one operation) Tj ET" + // A region over the first few letters only. + r := Redaction{Page: 1, Quads: []Quad{QuadOf(18, 97, 40, 113)}} + assertRefused(t, secretPage(t, body, nil), r, "reaches outside the region") + }) +} + +// ⛔ AND THE SLACK IS ENOUGH FOR A REGION THAT WAS MEASURED RATHER THAN TYPED. +// The workbench maps a lit rectangle back into page space through screen +// pixels, so its quad is a fraction of a point out in each direction. A region +// that misses the run's own box by less than that must still redact it, +// otherwise the one caller this was built for is refused every time. +func TestARegionOffByLessThanAPointStillRedacts(t *testing.T) { + // The run is at x 20..50, y 100..110. This quad is inside that on every + // side by a point and a half — a measured region, not a typed one. + r := Redaction{Page: 1, Quads: []Quad{QuadOf(21.5, 101.5, 48.5, 108.5)}} + d, err := Open(secretPage(t, "BT /F1 10 Tf 20 100 Td (secret) Tj ET", nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(r); err != nil { + t.Fatalf("a region measured off by a point was refused: %v", err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + for i, content := range contentsOf(t, out) { + if strings.Contains(content, "secret") { + t.Errorf("page %d still holds the redacted string", i+1) + } + } + // And a region genuinely short of the run — by more than the slack — is + // refused rather than quietly taking the whole run. + far := Redaction{Page: 1, Quads: []Quad{QuadOf(26, 103, 44, 107)}} + d2, err := Open(secretPage(t, "BT /F1 10 Tf 20 100 Td (secret) Tj ET", nil)) + if err != nil { + t.Fatal(err) + } + if err := d2.Redact(far); err == nil { + t.Error("a region well inside the run's own box took the whole run") + } } // assertRefused checks that Redact says no, and says why.