diff --git a/README.md b/README.md index 7e7249d..fca9ce9 100644 --- a/README.md +++ b/README.md @@ -892,6 +892,59 @@ renders the page and looks at the pixels: And the marks are flagged Print, for the same reason a widget is: without it a highlight vanishes the moment somebody prints what they marked up. +## Redacting text: removing it, not covering it + +`Doc.Redact` takes the text inside a region **out of the page's content +stream**. ISO 32000-1 §12.5.6.23, on applying a redaction: + +> Conforming readers that support redaction annotations shall provide a +> mechanism for applying content removal, and they shall **remove all traces** +> of the specified content. If a portion of an image is contained in a +> redaction region, that portion of the image data **shall be destroyed**; +> clipping or image masks shall not be used to hide that data. + +⛔ **A black rectangle is not a redaction.** It is the standard data leak: the +text is still in the content stream, so anyone's copy and paste — or +`pdfops text` — reads it straight back out from under the box. So the test that +decides whether this is a redaction at all is not about how the page looks, it +is that **the redacted string does not appear in the bytes that come out**, and +a drawn rectangle fails it. + +The operator that drew the text is overwritten with spaces of its own length, +which does two things at once: the string is gone from the stream, and every +offset after it is still where it was, so the text that stays **does not move**. +`extract.Run.Op` says which bytes to overwrite — deciding which glyphs are +inside a region is work `extract` already does, and a second implementation of +it here would cut the document somewhere other than where the caller was told +the text was. + +The region is then painted over, because an area that was emptied and not +marked cannot be told from one that was always blank. That paint is a +*marking*, not the redaction: the text is already gone when it is drawn. + +⛔ **And it refuses rather than half-redacts**, naming the page and the reason. +A caller who is told can choose another region or another tool; a caller who is +not told believes the document safe to send. It refuses a region holding: + +| what | why | +| --- | --- | +| text inside a Form XObject | the bytes are not the page's, and an XObject is **shared** — cleaning one page would change every page that draws it. `RecolourText` stops at the same wall | +| an image | the spec is explicit that a mask or a clip will not do and the pixels must be destroyed; re-encoding an image with them overwritten is not in this release | +| an annotation | its own `/Contents` can hold the very words being redacted | +| an operator only **part** of whose text is inside the region | blanking it would remove text nobody asked about, and the surviving words cannot simply be re-emitted: the bytes in the stream are font codes, and Unicode does not round-trip back to them through a CID font's map. Keeping the survivors' original bytes and replacing the removed ones with an equivalent kern is the way, and it is next | +| text that **reaches outside** the region | a run is the unit that can be removed, so a producer drawing a whole line with one `Tj` makes that line the run — and redacting one word of it would take the line. That is not a leak, which is why it is tempting to allow; it is the other way a redaction betrays its caller, by removing what they did not ask about and not saying so. A region is allowed to miss a run's own box by up to two points, because a caller mapping a lit rectangle back into page space arrives through screen pixels | + +The refusal happens when `Redact` is **called**, not when the document is +written — by the time of `Bytes` the caller has already been promised a +redacted document by everything in between. + +A redaction also survives the rest of the verbs. That is worth saying because +the write path once dropped a rewritten content stream whenever it stamped a +page: for a recolouring that loses a colour, but for a redaction it would put +the removed text back and draw the fill over it, so the document would **look +redacted and not be**. There is a test that redacts and watermarks the same +page and then looks for the string. + A document assembled from several files can still be given a form: two sources' forms cannot be merged — their fields may be named the same — but fields the caller *asked for* come from neither source and so are in no conflict. diff --git a/doc.go b/doc.go index 654d081..ccc1206 100644 --- a/doc.go +++ b/doc.go @@ -76,6 +76,11 @@ type Doc struct { // [Doc.Place]; zero means codec.DefaultMaxPixels. See [Doc.SetMaxPixels]. maxPixels int + // redactErr is set during a write when a page that was to be redacted + // could not be, and makes [Doc.Bytes] fail rather than hand back a + // document that looks redacted. Cleared at the start of every write. + redactErr error + // forms are the page-forms already written during the write in progress, // so a page drawn many times is embedded once. Made in [Doc.Bytes] and // dropped with it: a reader.Ref belongs to the writer that issued it, and @@ -113,6 +118,12 @@ type Page struct { // in. A pointer rather than a value with a flag beside it, because black // is a colour somebody asks for. recolour *[3]float64 + + // redactions are the regions of this page whose content is to be REMOVED + // from its stream, and the colour painted over each once it is gone. They + // are checked for what cannot be removed when Redact is called rather than + // when the document is written: see [Doc.Redact]. + redactions []Redaction } // Open reads a PDF file held in memory, with the empty password. diff --git a/go.mod b/go.mod index d8de2cb..318d3af 100644 --- a/go.mod +++ b/go.mod @@ -8,7 +8,7 @@ require ( github.com/go-ooxml/ooxml v0.7.0 github.com/go-ooxml/opc v0.2.1 github.com/go-pdfkit/document v0.2.0 - github.com/go-pdfkit/extract v0.9.0 + github.com/go-pdfkit/extract v0.10.0 github.com/go-pdfkit/forms v0.6.0 github.com/go-pdfkit/reader v0.10.0 github.com/go-pdfkit/render v0.69.1 @@ -22,6 +22,7 @@ require ( github.com/breml/rootcerts v0.3.7 // indirect github.com/dlclark/regexp2/v2 v2.5.2 // indirect github.com/dop251/goja v0.0.0-20260926152631-39ec2650adc9 // indirect + github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 // indirect github.com/evanw/esbuild v0.28.2 // indirect github.com/go-browserhttp/browserhttp v0.2.0 // indirect github.com/go-gfx/oksvg v0.2.0 // indirect @@ -30,7 +31,7 @@ require ( github.com/go-images/jpeg2000 v0.14.0 // indirect github.com/go-images/png v0.2.0 // indirect github.com/go-opentype/fonts v0.14.0 // indirect - github.com/go-opentype/opentype v0.16.1 // indirect + github.com/go-opentype/opentype v0.17.0 // indirect github.com/go-pdfkit/html2pdf v0.4.1 // indirect github.com/go-pdfkit/pdffont v0.4.0 // indirect github.com/go-pdfkit/pdfkit v0.20.0 // indirect diff --git a/go.sum b/go.sum index 3a7ab5c..41c37bc 100644 --- a/go.sum +++ b/go.sum @@ -8,6 +8,9 @@ github.com/dlclark/regexp2/v2 v2.5.2 h1:HAsucWRhsqcDzl6Ua9aR8JwYOTzrZyPrF0/FNxJV github.com/dlclark/regexp2/v2 v2.5.2/go.mod h1:avUrQvPaLz2DrFNHJF0taWAFFX2C1GMSSoeiqFjcBmU= github.com/dop251/goja v0.0.0-20260926152631-39ec2650adc9 h1:Ksd3Vil12b/ykT5NjGzd3qNc688cJB9OUXJZZChEktI= github.com/dop251/goja v0.0.0-20260926152631-39ec2650adc9/go.mod h1:u8yZRUavu+N4EnFFy6J5fVtjE7lEcZ2YyV2GcBXY9c8= +github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 h1:2tV76y6Q9BB+NEBasnqvs7e49aEBFI8ejC89PSnWH+4= +github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707/go.mod h1:qssHWj60/X5sZFNxpG4HBPDHVqxNm4DfnCKgrbZOT+s= +github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY= github.com/evanw/esbuild v0.28.2 h1:A2uETn4jrQTcXaT/shwTDTYBxDjl7fV7nXmUrJxfA2w= github.com/evanw/esbuild v0.28.2/go.mod h1:D2vIQZqV/vIf/VRHtViaUtViZmG7o+kKmlBfVQuRi48= github.com/go-browserhttp/browserhttp v0.2.0 h1:3RcjOQVHgqhSOmI5cfsURpexw/vetoDUoLXKzgTCSac= @@ -34,10 +37,14 @@ github.com/go-opentype/fonts v0.14.0 h1:WHw7k9gkHT1pu9B2yyvd0FWyQqn/4NKFQoKhvMYt github.com/go-opentype/fonts v0.14.0/go.mod h1:od2QcmdpKo7QCqo6Jb16y4GULk+h2Hhfq6x3xfvn62Q= github.com/go-opentype/opentype v0.16.1 h1:Hu2RKinqbd/WVSpRz6VrdKUPZUwPTZ4F9yx2HjZ5oCI= github.com/go-opentype/opentype v0.16.1/go.mod h1:aSxaPkjDzEnSAYB47s0oaHHa/sDViwW+aPlOz1A3dEg= +github.com/go-opentype/opentype v0.17.0 h1:yDLYLg/nUdfSSzm1Yg8t67racyFsamDEtS1gSR7xk1E= +github.com/go-opentype/opentype v0.17.0/go.mod h1:RujaDgR2h/ErgMkc5y/KYbX8NJtWw3yLRtVxbCfdvFQ= github.com/go-pdfkit/document v0.2.0 h1:4cdQLl4bcF0CxEw5eeZtkSGVimwgkR8jTMg5TKUEy6s= github.com/go-pdfkit/document v0.2.0/go.mod h1:Lgxm8Na/Nn2EI8Gmh+lpHhcPLwddOTG32GyL4GLki/c= github.com/go-pdfkit/extract v0.9.0 h1:gmyIFqZBxBBx6ccpUbF0Sw67jUmAp7q9vQ8mEsTyuFA= github.com/go-pdfkit/extract v0.9.0/go.mod h1:pUzGzNMx0WTsBOzfZ2XKT5ewqgv8cP/I6WpU4isPsmc= +github.com/go-pdfkit/extract v0.10.0 h1:9dezD5WqsMnQ9amqRaxADrjpf2pbe7TTMFUZwHXCTHU= +github.com/go-pdfkit/extract v0.10.0/go.mod h1:MNOx6Y0g//XSSx0lztuLUPDy4y5yubaNVsxJkVq3F6k= github.com/go-pdfkit/forms v0.6.0 h1:FJ5oHoIMMdbtVYWsyEEbZ8WFoJtSCdlYrjHE9hTSqxY= github.com/go-pdfkit/forms v0.6.0/go.mod h1:WZD05vH5x34pL7kOp4lVf1yOFsW7wpffj/ZqnHfsVUs= github.com/go-pdfkit/html2pdf v0.4.1 h1:zZ1sEoG6c3iJoEX4itG2Lul5z0BtyZ51lutzR4MC96M= @@ -70,10 +77,13 @@ github.com/go-widgets/painter v0.16.0 h1:j8FFObisySQChY2JiXS7Di2Mo4cqufIc0PDUnoD github.com/go-widgets/painter v0.16.0/go.mod h1:3wKWjaM8ZgEhir2HX0UDt8GemePKJ7l/yIjQb/vVSWk= github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM= github.com/goccy/go-yaml v1.19.2/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA= +github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8 h1:FKHo8hFI3A+7w0aUQuYXQ+6EN5stWmeY/AZqtM8xk9k= github.com/google/pprof v0.0.0-20240727154555-813a5fbdbec8/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo= +github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A= github.com/klauspost/compress v1.18.7 h1:aUyZsS4kH3QTKurYhAOwAHxllVPnOthb3vPfnF1Ehjw= github.com/klauspost/compress v1.18.7/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= +github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek= github.com/refraction-networking/utls v1.8.2 h1:j4Q1gJj0xngdeH+Ox/qND11aEfhpgoEvV+S9iJ2IdQo= github.com/refraction-networking/utls v1.8.2/go.mod h1:jkSOEkLqn+S/jtpEHPOsVv/4V4EVnelwbMQl4vCWXAM= github.com/sergeymakinen/go-bmp v1.0.0 h1:SdGTzp9WvCV0A1V0mBeaS7kQAwNLdVJbmHlqNWq0R+M= @@ -84,6 +94,7 @@ github.com/srwiley/rasterx v0.0.0-20220730225603-2ab79fcdd4ef h1:Ch6Q+AZUxDBCVqd github.com/srwiley/rasterx v0.0.0-20220730225603-2ab79fcdd4ef/go.mod h1:nXTWP6+gD5+LUJ8krVhhoeHjvHTutPxMYl5SvkcnJNE= github.com/tannevaled/gobig2 v0.2.0 h1:yx97/ltljALrIWDFVqHjPUIjAp3bfUUhH8Z8UROvNLc= github.com/tannevaled/gobig2 v0.2.0/go.mod h1:X0S1H+N35kg6zYgVYRqGMsGj3C35zs+iA2r1MXyPcV4= +github.com/ulikunitz/xz v0.5.8/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU= github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E= github.com/yuin/goldmark v1.8.7 h1:XT/tNqCIbabh0ln675w4x3dv04kQBKi8NwYqi9IJaxI= @@ -99,5 +110,6 @@ golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA= golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc= golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= rsc.io/pdf v0.1.1 h1:k1MczvYDUvJBe93bYd7wrZLLUEcLZAuF824/I4e5Xr4= rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4= diff --git a/redact.go b/redact.go new file mode 100644 index 0000000..d271a75 --- /dev/null +++ b/redact.go @@ -0,0 +1,408 @@ +// Copyright (c) 2026, the go-pdfkit/ops authors +// All rights reserved. +// +// SPDX-License-Identifier: BSD-3-Clause + +package ops + +import ( + "errors" + "fmt" + "strings" + + "github.com/go-pdfkit/extract" + "github.com/go-pdfkit/reader" +) + +// A Redaction names a region of a page whose content is to be REMOVED, and the +// colour to paint in its place. +// +// ⛔⛔ REMOVED, NOT COVERED, and that is the entire point. ISO 32000-1:2008 +// §12.5.6.23 on applying a redaction: +// +// Conforming readers that support redaction annotations shall provide a +// mechanism for applying content removal, and they shall remove all traces +// of the specified content. If a portion of an image is contained in a +// redaction region, that portion of the image data shall be destroyed; +// clipping or image masks shall not be used to hide that data. +// +// A black rectangle drawn over a paragraph is the standard data leak: the text +// is still in the content stream, so anyone's copy and paste — or `pdfops text` +// — reads it straight back out from under the box. The test that matters, and +// the one a drawn rectangle fails, is that the redacted string does not appear +// anywhere in the bytes that come out. +// +// The region is in the page's own coordinates, up from the bottom left corner, +// in points — the same space [Quad] uses for a markup annotation. +type Redaction struct { + // Page is which page, counting from one. + Page int + // Quads is where on it. A quadrilateral per piece, as a markup annotation + // has: see [QuadOf] for the upright rectangle a line of text is. + Quads []Quad + // R, G and B are the colour painted over the region once the content is + // gone, each from zero to one. The spec leaves the region transparent when + // a redaction names no colour; this paints it, because an area that was + // emptied and not marked is one a reader cannot tell from one that was + // always blank. + R, G, B float64 +} + +// Redact removes the text inside each region and paints the region over. +// +// ⛔ IT REFUSES RATHER THAN HALF-REDACTS. A redaction that removed some of +// what it was asked to remove, and said nothing about the rest, would be worse +// than no redaction at all: the caller would believe the document safe to send. +// So this returns an error, naming the page and the reason, when a region holds +// anything it cannot take out: +// +// - text inside a Form XObject. Its bytes are not this page's, and an +// XObject is SHARED — editing it to clean one page would change every page +// that draws it. [Doc.RecolourText] stops at the same wall, and counts what +// it could not reach for the same reason. +// - an image. The spec is explicit that a mask or a clip will not do and the +// pixels themselves must be destroyed; until this can re-encode an image +// with those pixels overwritten, an image in the region is a refusal. +// - an annotation. Its /Contents can hold the very words being redacted. +// - a text-showing operator only PART of whose text is inside the region. +// Blanking it would take away text the caller did not ask about, and +// rewriting it cannot be done by re-emitting the surviving words: the +// bytes in the stream are font codes, and Unicode does not round-trip +// back to them through a CID font's map. Keeping the surviving pieces' +// original bytes and replacing the removed ones with an equivalent kern is +// the way, and it is not in this release. +// +// What it does remove it removes from the bytes: the operator is overwritten +// with spaces, so the string is gone from the content stream rather than +// hidden. +func (d *Doc) Redact(rs ...Redaction) error { + for _, r := range rs { + if r.Page < 1 || r.Page > len(d.pages) { + return fmt.Errorf("ops: there is no page %d to redact", r.Page) + } + if len(r.Quads) == 0 { + return fmt.Errorf("ops: the redaction of page %d names no region", r.Page) + } + for _, v := range []float64{r.R, r.G, r.B} { + if v < 0 || v > 1 { + return fmt.Errorf("ops: a colour component is from 0 to 1, not %v", v) + } + } + p := &d.pages[r.Page-1] + if p.src == nil { + return fmt.Errorf("ops: page %d was made rather than read, so it carries no text to redact", r.Page) + } + if err := d.checkRedactable(p, r); err != nil { + return err + } + p.redactions = append(p.redactions, r) + } + return nil +} + +// checkRedactable refuses a region this release cannot empty completely. +// +// ⛔ It runs when Redact is CALLED, not when the document is written. A caller +// that is told at once can choose another region or another tool; one told at +// Save has already been promised a redacted document by everything in between. +func (d *Doc) checkRedactable(p *Page, r Redaction) error { + runs, err := extractRuns(p.src, p.number) + if err != nil { + return fmt.Errorf("ops: page %d could not be read to redact it: %w", r.Page, err) + } + if err := unremovableText(r.Page, runs, r.Quads); err != nil { + return err + } + images, err := extractImages(p.src, p.number) + if err != nil { + return fmt.Errorf("ops: the images of page %d could not be read: %w", r.Page, err) + } + for _, im := range images { + // ⛔ DrawnWidth and DrawnHeight, not Width and Height: those two are the + // picture's own SAMPLES. Comparing a sample count with a region in + // points would weigh a 2048-pixel thumbnail against a 72-point box and + // refuse every page that carried a picture at all. + if overlaps(im.X, im.Y, im.X+im.DrawnWidth, im.Y+im.DrawnHeight, r.Quads) { + return fmt.Errorf("ops: page %d places an image across the region, and a redaction "+ + "must destroy the pixels rather than cover them, which this cannot yet do", + r.Page) + } + } + page, _ := p.src.Page(p.number) + if annots, ok := resolveArray(p.src, page.Get("Annots")); ok { + for _, a := range annots { + dict, ok := resolveDict(p.src, a) + if !ok { + continue + } + box := annotRect(p.src, dict.Get("Rect")) + if box == [4]float64{} { + continue + } + if overlaps(box[0], box[1], box[2], box[3], r.Quads) { + return fmt.Errorf("ops: page %d carries an annotation across the region, and its "+ + "own contents can hold the words being redacted", r.Page) + } + } + } + return nil +} + +// The two readings a refusal is decided from, replaceable so that a test can +// see what this does when a page cannot be read at all. The same shape as +// markup.go's `var now = time.Now`. +var ( + extractRuns = extract.Runs + extractImages = extract.Images + // pageContent is wrapped rather than used directly for the same reason: + // the difference between a recolouring and a redaction of a page that + // cannot be read is the difference between a lost colour and a document + // that must not be written, and a test has to be able to ask for both. + pageContent = func(src *reader.Document, page int) ([]byte, error) { + return src.PageContent(page) + } +) + +// unremovableText is the refusals that are decided from the runs alone, so +// that each can be put to it directly: a page that happens to hold a form, an +// unplaceable run and a half-covered operator all at once is a fixture nobody +// can read, and two of the three would never be the reason. +func unremovableText(page int, runs []extract.Run, quads []Quad) error { + for _, run := range runs { + if !runTouches(run, quads) { + continue + } + if run.InForm { + return fmt.Errorf("ops: page %d draws %q from a form XObject, whose bytes are "+ + "shared with every page that draws it, so this cannot be removed for one page alone", + page, summarise(run.Text)) + } + // ⛔ NOT a dead branch, and not one to drop. A run whose operation this + // cannot locate is a run that would be SKIPPED by redactedSpans — left + // on the page, under the fill, in a document the caller was told was + // redacted. extract sets a span for every run it draws today; this is + // what happens the day one of them does not. + if run.Op.Empty() { + return fmt.Errorf("ops: page %d draws %q from an operation this cannot locate in the stream", + page, summarise(run.Text)) + } + // ⛔⛔ AND THE RUN MUST BE WHOLLY INSIDE THE REGION. A run is the unit + // that can be removed, so a run that merely OVERLAPS the region would + // be blanked entire — and a producer that draws a whole line with one + // Tj makes that line the run. Redacting one word of such a line would + // take the line. That is not a leak, which is why it is tempting to + // allow; it is the other way a redaction betrays its caller, by + // removing what they did not ask about and not saying so. + if !runInside(run, quads) { + return fmt.Errorf("ops: on page %d the text %q reaches outside the region, and the "+ + "operation that draws it can only be removed whole, which would take text "+ + "the region does not cover", + page, summarise(run.Text)) + } + } + // A text-showing operator whose text is partly inside the region and partly + // outside cannot be blanked without taking away what was not asked for. + for span, group := range bySpan(runs) { + in, out := 0, 0 + for _, run := range group { + if runTouches(run, quads) { + in++ + } else { + out++ + } + } + if in > 0 && out > 0 { + return fmt.Errorf("ops: on page %d one operation (bytes %d..%d) draws both text to "+ + "redact and text to keep, and blanking it would remove both", + page, span.From, span.To) + } + } + return nil +} + +// bySpan groups the runs of a page by the operation that drew them, which is +// what says whether an operator can be blanked whole. +func bySpan(runs []extract.Run) map[extract.Span][]extract.Run { + out := map[extract.Span][]extract.Run{} + for _, r := range runs { + if r.Op.Empty() { + continue + } + out[r.Op] = append(out[r.Op], r) + } + return out +} + +// runTouches says whether a run's box meets any of the quads. The box is the +// baseline and the size above it, which is what a line of text occupies. +func runTouches(r extract.Run, quads []Quad) bool { + return overlaps(r.X, r.Y, r.X+r.Width, r.Y+r.Size, quads) +} + +// runSlack is how far a run's box may stand outside a quad and still count as +// inside it. +// +// ⛔ IT IS A MEASUREMENT, NOT A FUDGE. A caller that asks for the region a run +// occupies — the workbench, mapping a lit rectangle back into page space — +// arrives with a quad derived from screen pixels, so it is a fraction of a +// point out in each direction, and more at a zoom below 1:1. Two points is +// under half a line of ten-point text, so a run from the line above or below +// can never be swallowed by it, and the ordinary case of "this exact run" is +// not refused for a rounding error. +const runSlack = 2.0 + +// runInside says whether a run's box lies wholly within one of the quads, which +// is what makes the run's operator safe to blank whole. +func runInside(r extract.Run, quads []Quad) bool { + for _, q := range quads { + qx0 := min(q.TopLeftX, q.BottomLeftX) - runSlack + qx1 := max(q.TopRightX, q.BottomRightX) + runSlack + qy0 := min(q.BottomLeftY, q.BottomRightY) - runSlack + qy1 := max(q.TopLeftY, q.TopRightY) + runSlack + if r.X >= qx0 && r.X+r.Width <= qx1 && r.Y >= qy0 && r.Y+r.Size <= qy1 { + return true + } + } + return false +} + +// overlaps says whether a box meets any of the quads, each taken as its own +// bounding box. +func overlaps(x0, y0, x1, y1 float64, quads []Quad) bool { + for _, q := range quads { + qx0 := min(q.TopLeftX, q.BottomLeftX) + qx1 := max(q.TopRightX, q.BottomRightX) + qy0 := min(q.BottomLeftY, q.BottomRightY) + qy1 := max(q.TopLeftY, q.TopRightY) + if x0 < qx1 && qx0 < x1 && y0 < qy1 && qy0 < y1 { + return true + } + } + return false +} + +// summarise shortens a run's text for a message, so that an error about a +// secret does not print the whole of it. +func summarise(s string) string { + const most = 24 + r := []rune(s) + if len(r) <= most { + return s + } + return string(r[:most]) + "…" +} + +// redactContent blanks every text-showing operator whose text lies inside the +// region, and reports how many it blanked. +// +// ⛔ BLANKED WITH SPACES OF THE SAME LENGTH, which is what recolourContent does +// a file away and for the same reason: every offset after an edit is still the +// offset the scan reported, so the edits need no ordering rule. Here it does a +// second job — a span replaced by spaces no longer holds the string, so the +// text is gone from the bytes rather than merely unreferenced. +func redactContent(data []byte, spans map[extract.Span]bool) (out []byte, blanked int) { + if len(spans) == 0 { + return data, 0 + } + edited := make([]byte, len(data)) + copy(edited, data) + for span := range spans { + if span.From < 0 || span.To > len(edited) || span.To <= span.From { + continue + } + for i := span.From; i < span.To; i++ { + edited[i] = ' ' + } + blanked++ + } + return edited, blanked +} + +// redactedSpans is the operations to blank for a page's redactions. +// +// ⛔⛔ IT RETURNS THE ERROR RATHER THAN NO SPANS. Returning none would blank +// nothing and still paint the fill, which is a document that LOOKS redacted and +// is not — the one outcome worse than refusing. Doc.Bytes turns this into a +// failed write, so a caller gets no document at all rather than a leaky one. +func redactedSpans(src *reader.Document, page int, rs []Redaction) (map[extract.Span]bool, error) { + runs, err := extractRuns(src, page) + if err != nil { + return nil, err + } + spans := map[extract.Span]bool{} + for _, r := range rs { + for _, run := range runs { + if run.Op.Empty() || run.InForm { + continue + } + // ⛔ INSIDE, the same test the refusal above uses. If these two + // ever disagreed, the pair would be the defect: a run the check + // passed over and this blanked would remove text nobody asked + // about, and a run the check allowed and this skipped would be left + // on the page under the fill. + if runInside(run, r.Quads) { + spans[run.Op] = true + } + } + } + return spans, nil +} + +// rewrittenContent is a borrowed page's own content stream with the redactions +// blanked out of it and the recolouring applied, or an error that must stop the +// write. +// +// ⛔ THE REDACTION GOES FIRST, on the bytes as they were. Its spans came from a +// scan of exactly these bytes, and a recolouring INSERTS operators — "q 1 0 0 +// rg " before every BT — so every offset after the first insertion has moved. +// Blanking is the other way round: it writes spaces over the same number of +// bytes, so the offsets a later scan reports are still the ones it reported. +// +// ⛔ And the two failures are not the same failure. A page whose stream cannot +// be read is, for a recolouring, a page left in its own colours — which is what +// this did before redaction existed and is no worse than a colour. For a +// redaction it is a document that must not be written at all. +func (d *Doc) rewrittenContent(p Page) (out []byte, rewritten bool, err error) { + if p.recolour == nil && len(p.redactions) == 0 { + return nil, false, nil + } + content, cerr := pageContent(p.src, p.number) + if len(p.redactions) > 0 { + spans, serr := redactedSpans(p.src, p.number, p.redactions) + if cerr != nil || serr != nil { + return nil, false, fmt.Errorf("ops: page %d was to be redacted and its own content "+ + "stream could not be read, so the redaction cannot be applied: %w", + p.number, errors.Join(cerr, serr)) + } + content, _ = redactContent(content, spans) + } + if cerr != nil { + return nil, false, nil // a recolouring alone: leave the page as it was + } + if p.recolour != nil { + var unreached int + content, unreached = recolourContent(content, *p.recolour) + d.unreached += unreached + } + return content, true, nil +} + +// redactionFill is the content that paints the emptied regions over, so a +// reader can see that something was taken out. +func redactionFill(rs []Redaction) string { + var sb strings.Builder + for _, r := range rs { + for _, q := range r.Quads { + x0 := min(q.TopLeftX, q.BottomLeftX) + x1 := max(q.TopRightX, q.BottomRightX) + y0 := min(q.BottomLeftY, q.BottomRightY) + y1 := max(q.TopLeftY, q.TopRightY) + if x1 <= x0 || y1 <= y0 { + continue + } + fmt.Fprintf(&sb, " q %g %g %g rg %g %g %g %g re f Q", + r.R, r.G, r.B, x0, y0, x1-x0, y1-y0) + } + } + return sb.String() +} diff --git a/redact_test.go b/redact_test.go new file mode 100644 index 0000000..7e9856b --- /dev/null +++ b/redact_test.go @@ -0,0 +1,718 @@ +package ops + +import ( + "errors" + "strings" + "testing" + + "github.com/go-pdfkit/extract" + "github.com/go-pdfkit/reader" +) + +// secretPage is a page saying two things, one of which is to be removed: a +// line at y 150 to keep and a line at y 100 to redact. +func secretPage(t *testing.T, body string, extra func(w *reader.Writer, res reader.Dict)) []byte { + t.Helper() + w := reader.NewWriter("1.7") + pagesRef := w.Reserve() + widths := make(reader.Array, 0, 224) + for i := 32; i < 256; i++ { + widths = append(widths, reader.Integer(500)) + } + res := reader.Dict{"Font": reader.Dict{"F1": w.Add(reader.Dict{ + "Type": reader.Name("Font"), "Subtype": reader.Name("Type1"), + "BaseFont": reader.Name("Helvetica"), "FirstChar": reader.Integer(32), + "LastChar": reader.Integer(255), "Widths": widths, + "Encoding": reader.Name("WinAnsiEncoding"), + })}} + page := reader.Dict{ + "Type": reader.Name("Page"), "Parent": pagesRef, + "Contents": w.Add(&reader.Stream{Dict: reader.Dict{}, Raw: []byte(body)}), + "Resources": res, + } + if extra != nil { + extra(w, res) + page["Resources"] = res + } + added := w.Add(page) + w.Put(pagesRef, reader.Dict{ + "Type": reader.Name("Pages"), "Kids": reader.Array{added}, "Count": reader.Integer(1), + "MediaBox": reader.Array{reader.Integer(0), reader.Integer(0), + reader.Integer(100), reader.Integer(200)}, + }) + root := w.Add(reader.Dict{"Type": reader.Name("Catalog"), "Pages": pagesRef}) + out, err := w.Finish(reader.Dict{"Root": root}) + if err != nil { + t.Fatal(err) + } + return out +} + +// twoLines is the ordinary fixture: one line to keep, one to remove. +const twoLines = "BT /F1 10 Tf 20 150 Td (public) Tj ET BT /F1 10 Tf 20 100 Td (secret) Tj ET" + +// overSecret is the region the second line occupies, with paper around it. +func overSecret() Redaction { + return Redaction{Page: 1, Quads: []Quad{QuadOf(15, 95, 85, 115)}} +} + +// redacted applies a redaction to the fixture and returns the file. +func redacted(t *testing.T, body string, r Redaction) []byte { + t.Helper() + d, err := Open(secretPage(t, body, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(r); err != nil { + t.Fatal(err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + return out +} + +// ⛔⛔ THE ONLY TEST THAT DECIDES WHETHER THIS IS A REDACTION. ISO 32000-1 +// §12.5.6.23 requires a conforming reader to "remove all traces of the +// specified content". A black rectangle drawn over a paragraph passes every +// test about how the page LOOKS and fails this one, because the text is still +// in the content stream for anyone's copy and paste to read back. +func TestTheRedactedStringIsGoneFromTheBytes(t *testing.T) { + out := redacted(t, twoLines, overSecret()) + for i, content := range contentsOf(t, out) { + if strings.Contains(content, "secret") { + t.Errorf("page %d still holds the redacted string: %q", i+1, content) + } + if !strings.Contains(content, "public") { + t.Errorf("page %d lost the text that was to be kept: %q", i+1, content) + } + } + // And the document read back the way a reader would read it says nothing + // of it either. + src, err := reader.Open(out) + if err != nil { + t.Fatal(err) + } + text, err := extract.Text(src, 1) + if err != nil { + t.Fatal(err) + } + if strings.Contains(text, "secret") { + t.Errorf("the page still reads back as %q", text) + } + if !strings.Contains(text, "public") { + t.Errorf("the page no longer says what it was to keep: %q", text) + } +} + +// ⛔ AND WHAT WAS KEPT DID NOT MOVE. Blanking an operator must not shift the +// text that stays — that is the difference between a redaction and a +// re-typesetting, and it is why the whole operation is overwritten with spaces +// of its own length rather than deleted. +func TestWhatWasKeptDidNotMove(t *testing.T) { + before, err := reader.Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + was, err := extract.Runs(before, 1) + if err != nil { + t.Fatal(err) + } + after, err := reader.Open(redacted(t, twoLines, overSecret())) + if err != nil { + t.Fatal(err) + } + now, err := extract.Runs(after, 1) + if err != nil { + t.Fatal(err) + } + if len(was) != 2 { + t.Fatalf("the fixture draws %d run(s), want 2", len(was)) + } + if len(now) != 1 { + t.Fatalf("the redacted page draws %d run(s), want 1", len(now)) + } + keep := was[0] + if now[0].Text != keep.Text { + t.Fatalf("what is left says %q, want %q", now[0].Text, keep.Text) + } + if now[0].X != keep.X || now[0].Y != keep.Y || now[0].Width != keep.Width { + t.Errorf("the kept line was at (%g,%g) %g wide and is now at (%g,%g) %g wide", + keep.X, keep.Y, keep.Width, now[0].X, now[0].Y, now[0].Width) + } +} + +// ⛔ THE REGION IS PAINTED OVER, because an area that was emptied and not +// marked cannot be told from one that was always blank — and a reader has to be +// able to see that something was taken out. The paint is a MARKING, not the +// redaction: the text is gone from the stream before it is drawn. +func TestTheRegionIsPaintedOver(t *testing.T) { + r := overSecret() + r.R, r.G, r.B = 0, 0, 0 + out := redacted(t, twoLines, r) + // A PDF counts up from the foot of the page and a raster down from the top: + // the region's y 95..115 on a 200pt page is rows 85..105. + if red, green, blue := pixelAt(t, out, 50, 95); blank(red, green, blue) { + t.Errorf("the middle of the region is bare paper (%d,%d,%d)", red, green, blue) + } + // The line that was kept still has its ink, and is not painted over. + inked := false + img := renderOf(t, out) + for y := 40; y < 52; y++ { + for x := 20; x < 60; x++ { + i := (y*img.W + x) * 4 + if !blank(img.Pix[i], img.Pix[i+1], img.Pix[i+2]) { + inked = true + } + } + } + if !inked { + t.Error("the line that was kept has no ink left on the page") + } +} + +// ⛔⛔ A REDACTION SURVIVES EVERYTHING ELSE THE DOCUMENT IS ASKED FOR. This is a +// control for a defect the write path already carries a comment about: the +// decoration step used to take the source's content parts whenever it stamped a +// page, which threw away a rewritten stream. For a recolouring that loses a +// colour. For a redaction it would put the removed text back and then draw the +// fill over it, so the document would LOOK redacted and not be — the worst +// possible ending. +func TestARedactionSurvivesAStamp(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(overSecret()); err != nil { + t.Fatal(err) + } + if err := d.Watermark("all", "DRAFT"); err != nil { + t.Fatal(err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + for i, content := range contentsOf(t, out) { + if strings.Contains(content, "secret") { + t.Errorf("page %d holds the redacted string after a watermark: %q", i+1, content) + } + } +} + +// ⛔⛔ IT REFUSES RATHER THAN HALF-REDACTS. Each of these is content a region +// can hold that this release cannot take out, and in every case the caller must +// be told — a caller who is not told believes the document safe to send. +func TestRedactingRefusesWhatItCannotRemove(t *testing.T) { + // A Form XObject: its bytes are not the page's, and it is shared. + t.Run("text inside a form", func(t *testing.T) { + const body = "BT /F1 10 Tf 20 150 Td (public) Tj ET q /X1 Do Q" + const inner = "BT /F1 10 Tf 20 100 Td (secret) Tj ET" + pdf := secretPage(t, body, func(w *reader.Writer, res reader.Dict) { + res["XObject"] = reader.Dict{"X1": w.Add(&reader.Stream{Dict: reader.Dict{ + "Type": reader.Name("XObject"), "Subtype": reader.Name("Form"), + "BBox": reader.Array{reader.Integer(0), reader.Integer(0), + reader.Integer(100), reader.Integer(200)}, + "Resources": res, + }, Raw: []byte(inner)})} + }) + assertRefused(t, pdf, overSecret(), "form XObject") + }) + + // An image: the spec says the pixels must be destroyed, not covered. + t.Run("an image across the region", func(t *testing.T) { + const body = "BT /F1 10 Tf 20 150 Td (public) Tj ET q 70 0 0 20 15 95 cm /Im1 Do Q" + pdf := secretPage(t, body, func(w *reader.Writer, res reader.Dict) { + res["XObject"] = reader.Dict{"Im1": w.Add(&reader.Stream{Dict: reader.Dict{ + "Type": reader.Name("XObject"), "Subtype": reader.Name("Image"), + "Width": reader.Integer(2), "Height": reader.Integer(2), + "ColorSpace": reader.Name("DeviceGray"), "BitsPerComponent": reader.Integer(8), + }, Raw: []byte{0, 64, 128, 255}})} + }) + assertRefused(t, pdf, overSecret(), "image") + }) + + // An annotation: its own contents can hold the words being redacted. + t.Run("an annotation across the region", func(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Mark(Markup{Page: 1, Kind: MarkHighlight, + Quads: []Quad{QuadOf(20, 98, 60, 112)}, Note: "the secret, again"}); err != nil { + t.Fatal(err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + assertRefused(t, out, overSecret(), "annotation") + }) + + // One operator drawing both what goes and what stays. + t.Run("an operator it would half-erase", func(t *testing.T) { + const body = "BT /F1 10 Tf 20 100 Td [(keep) -2000 (drop)] TJ ET" + // ⛔ The region covers the second word WHOLLY and the first not at all: + // "keep" is drawn at x 20..40, the kern moves the pen 20 points, and + // "drop" is drawn at x 60..80. A quad that clipped "drop" instead would + // be refused for reaching outside the region — a different rule, which + // is what this fixture said until it was measured. + r := Redaction{Page: 1, Quads: []Quad{QuadOf(58, 95, 85, 115)}} + assertRefused(t, secretPage(t, body, nil), r, "would remove both") + }) + + // ⛔⛔ AND A RUN THAT REACHES OUTSIDE THE REGION. A producer that draws a + // whole line with one Tj makes that line the run, so redacting one word of + // it would take the line. Not a leak — the other betrayal, where more goes + // than was asked and nothing says so. + t.Run("text that reaches outside the region", func(t *testing.T) { + const body = "BT /F1 10 Tf 20 100 Td (a whole line in one operation) Tj ET" + // A region over the first few letters only. + r := Redaction{Page: 1, Quads: []Quad{QuadOf(18, 97, 40, 113)}} + assertRefused(t, secretPage(t, body, nil), r, "reaches outside the region") + }) +} + +// ⛔ AND THE SLACK IS ENOUGH FOR A REGION THAT WAS MEASURED RATHER THAN TYPED. +// The workbench maps a lit rectangle back into page space through screen +// pixels, so its quad is a fraction of a point out in each direction. A region +// that misses the run's own box by less than that must still redact it, +// otherwise the one caller this was built for is refused every time. +func TestARegionOffByLessThanAPointStillRedacts(t *testing.T) { + // The run is at x 20..50, y 100..110. This quad is inside that on every + // side by a point and a half — a measured region, not a typed one. + r := Redaction{Page: 1, Quads: []Quad{QuadOf(21.5, 101.5, 48.5, 108.5)}} + d, err := Open(secretPage(t, "BT /F1 10 Tf 20 100 Td (secret) Tj ET", nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(r); err != nil { + t.Fatalf("a region measured off by a point was refused: %v", err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + for i, content := range contentsOf(t, out) { + if strings.Contains(content, "secret") { + t.Errorf("page %d still holds the redacted string", i+1) + } + } + // And a region genuinely short of the run — by more than the slack — is + // refused rather than quietly taking the whole run. + far := Redaction{Page: 1, Quads: []Quad{QuadOf(26, 103, 44, 107)}} + d2, err := Open(secretPage(t, "BT /F1 10 Tf 20 100 Td (secret) Tj ET", nil)) + if err != nil { + t.Fatal(err) + } + if err := d2.Redact(far); err == nil { + t.Error("a region well inside the run's own box took the whole run") + } +} + +// assertRefused checks that Redact says no, and says why. +func assertRefused(t *testing.T, pdf []byte, r Redaction, because string) { + t.Helper() + d, err := Open(pdf) + if err != nil { + t.Fatal(err) + } + err = d.Redact(r) + if err == nil { + t.Fatalf("the redaction was accepted although it cannot remove the %s", because) + } + if !strings.Contains(err.Error(), because) { + t.Errorf("the refusal says %q, which does not mention %q", err, because) + } + // And nothing was recorded, so a caller that ignored the error does not + // get a document that looks redacted. + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + for _, content := range contentsOf(t, out) { + if strings.Contains(content, "re f") && strings.Contains(content, "0 0 0 rg") { + t.Error("a refused redaction still painted its region") + } + } +} + +// A redaction has to name a page and a region, and a colour that is a colour. +func TestARedactionMustMakeSense(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + for _, c := range []struct { + name string + r Redaction + says string + }{ + {"no such page", Redaction{Page: 2, Quads: []Quad{QuadOf(0, 0, 10, 10)}}, "no page 2"}, + {"page zero", Redaction{Page: 0, Quads: []Quad{QuadOf(0, 0, 10, 10)}}, "no page 0"}, + {"no region", Redaction{Page: 1}, "names no region"}, + {"a colour that is not one", Redaction{Page: 1, + Quads: []Quad{QuadOf(0, 0, 10, 10)}, R: 2}, "from 0 to 1"}, + } { + t.Run(c.name, func(t *testing.T) { + err := d.Redact(c.r) + if err == nil { + t.Fatalf("%+v was accepted", c.r) + } + if !strings.Contains(err.Error(), c.says) { + t.Errorf("the refusal says %q, want it to mention %q", err, c.says) + } + }) + } +} + +// A page this package made rather than borrowed has no text of its own, and +// saying so beats appearing to redact one. +func TestAMadePageHasNoTextToRedact(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + d.Blank(100, 200) + err = d.Redact(Redaction{Page: 2, Quads: []Quad{QuadOf(0, 0, 10, 10)}}) + if err == nil { + t.Fatal("a made page accepted a redaction") + } + if !strings.Contains(err.Error(), "made rather than read") { + t.Errorf("the refusal says %q", err) + } +} + +// ⛔⛔ THE REFUSAL THAT MATTERS MOST, AND THE ONE NO FIXTURE CAN STAGE. A run +// whose operation cannot be located would be SKIPPED when the spans to blank +// are collected — left on the page, under the fill, in a document the caller +// was told was redacted. extract gives every run it draws a span today; this is +// what happens the day one of them does not, and the only way to ask is to hand +// the rule such a run directly. +func TestTextWithNoSpanIsRefusedRatherThanSkipped(t *testing.T) { + region := []Quad{QuadOf(15, 95, 85, 115)} + inside := extract.Run{Text: "secret", X: 20, Y: 100, Width: 30, Size: 10} + t.Run("no span", func(t *testing.T) { + err := unremovableText(1, []extract.Run{inside}, region) + if err == nil { + t.Fatal("a run this cannot locate was accepted for redaction") + } + if !strings.Contains(err.Error(), "cannot locate") { + t.Errorf("the refusal says %q", err) + } + }) + t.Run("in a form", func(t *testing.T) { + run := inside + run.Op, run.InForm = extract.Span{From: 1, To: 9}, true + err := unremovableText(1, []extract.Run{run}, region) + if err == nil { + t.Fatal("a run inside a form was accepted for redaction") + } + if !strings.Contains(err.Error(), "form XObject") { + t.Errorf("the refusal says %q", err) + } + }) + // ⛔ And a long secret is not printed in full by the message that refuses + // it: an error goes to a log, and a redaction's subject is the one string + // that must not. + t.Run("a long secret is shortened", func(t *testing.T) { + run := inside + run.Text = "a very long secret indeed, every word of it confidential" + err := unremovableText(1, []extract.Run{run}, region) + if err == nil { + t.Fatal("accepted") + } + if strings.Contains(err.Error(), "confidential") { + t.Errorf("the refusal printed the whole secret: %q", err) + } + if !strings.Contains(err.Error(), "…") { + t.Errorf("the refusal did not say it had shortened the text: %q", err) + } + }) + // A run outside the region is nobody's business, span or no span. + t.Run("outside the region", func(t *testing.T) { + run := inside + run.Y = 150 + if err := unremovableText(1, []extract.Run{run}, region); err != nil { + t.Errorf("a run outside the region was refused: %v", err) + } + }) +} + +// ⛔⛔ A REDACTION THAT CANNOT BE APPLIED FAILS THE WRITE. If the page's text +// cannot be read when the document is written, no operator is blanked — and the +// fill is still painted over the region, so the file would look redacted and +// hold every word. The caller gets the failure instead of the file. +func TestARedactionThatCannotBeAppliedFailsTheWrite(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(overSecret()); err != nil { + t.Fatal(err) + } + // It was readable when Redact was called, and is not when Bytes is: a + // document handed between the two is the case this guards. + was := extractRuns + extractRuns = func(*reader.Document, int) ([]extract.Run, error) { return nil, errTest } + t.Cleanup(func() { extractRuns = was }) + + out, err := d.Bytes() + if err == nil { + t.Fatal("a document whose redaction could not be applied was written anyway") + } + if out != nil { + t.Errorf("%d bytes were handed back with the failure", len(out)) + } + if !strings.Contains(err.Error(), "redaction cannot be applied") { + t.Errorf("the failure says %q", err) + } + // And the write recovers once the page can be read again. + extractRuns = was + if _, err := d.Bytes(); err != nil { + t.Errorf("the next write failed too: %v", err) + } +} + +// A page that cannot be read is a page that cannot be redacted, and the caller +// hears which reading failed. +func TestAPageThatCannotBeReadIsNotRedacted(t *testing.T) { + for _, c := range []struct { + name string + arm func(t *testing.T) + says string + }{ + {"its text", func(t *testing.T) { + was := extractRuns + extractRuns = func(*reader.Document, int) ([]extract.Run, error) { + return nil, errTest + } + t.Cleanup(func() { extractRuns = was }) + }, "could not be read to redact it"}, + {"its images", func(t *testing.T) { + was := extractImages + extractImages = func(*reader.Document, int) ([]extract.Image, error) { + return nil, errTest + } + t.Cleanup(func() { extractImages = was }) + }, "images of page 1 could not be read"}, + } { + t.Run(c.name, func(t *testing.T) { + c.arm(t) + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + err = d.Redact(overSecret()) + if err == nil { + t.Fatal("a page that could not be read was redacted anyway") + } + if !strings.Contains(err.Error(), c.says) { + t.Errorf("the refusal says %q, want it to mention %q", err, c.says) + } + }) + } +} + +// errTest is the failure the readings above are made to report. +var errTest = errors.New("this page is unreadable") + +// An entry of Annots that is not an annotation, and one with no rectangle, are +// stepped over rather than taken as covering the region — a document is free +// to hold either, and refusing to redact because of one would be refusing for +// no reason. +func TestNonsenseInAnnotsIsSteppedOver(t *testing.T) { + w := reader.NewWriter("1.7") + pagesRef := w.Reserve() + widths := make(reader.Array, 0, 224) + for i := 32; i < 256; i++ { + widths = append(widths, reader.Integer(500)) + } + font := w.Add(reader.Dict{ + "Type": reader.Name("Font"), "Subtype": reader.Name("Type1"), + "BaseFont": reader.Name("Helvetica"), "FirstChar": reader.Integer(32), + "LastChar": reader.Integer(255), "Widths": widths, + "Encoding": reader.Name("WinAnsiEncoding"), + }) + page := w.Add(reader.Dict{ + "Type": reader.Name("Page"), "Parent": pagesRef, + "Contents": w.Add(&reader.Stream{Dict: reader.Dict{}, Raw: []byte(twoLines)}), + "Resources": reader.Dict{"Font": reader.Dict{"F1": font}}, + "Annots": reader.Array{ + reader.Integer(7), // not an annotation at all + w.Add(reader.Dict{"Subtype": reader.Name("Text")}), // no Rect + }, + }) + w.Put(pagesRef, reader.Dict{ + "Type": reader.Name("Pages"), "Kids": reader.Array{page}, "Count": reader.Integer(1), + "MediaBox": reader.Array{reader.Integer(0), reader.Integer(0), + reader.Integer(100), reader.Integer(200)}, + }) + root := w.Add(reader.Dict{"Type": reader.Name("Catalog"), "Pages": pagesRef}) + pdf, err := w.Finish(reader.Dict{"Root": root}) + if err != nil { + t.Fatal(err) + } + d, err := Open(pdf) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(overSecret()); err != nil { + t.Fatalf("nonsense in Annots stopped the redaction: %v", err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + for i, content := range contentsOf(t, out) { + if strings.Contains(content, "secret") { + t.Errorf("page %d still holds the redacted string", i+1) + } + } +} + +// A span outside the stream is ignored rather than panicking: the bytes being +// edited and the bytes the spans were read from are the same page, so this +// cannot happen — and a redaction is the wrong place to find out by crashing. +func TestASpanOutsideTheStreamIsIgnored(t *testing.T) { + data := []byte("BT (x) Tj ET") + for _, span := range []extract.Span{ + {From: -1, To: 4}, + {From: 2, To: len(data) + 10}, + {From: 5, To: 5}, + {From: 6, To: 3}, + } { + out, blanked := redactContent(data, map[extract.Span]bool{span: true}) + if blanked != 0 { + t.Errorf("%+v blanked %d operation(s)", span, blanked) + } + if string(out) != string(data) { + t.Errorf("%+v changed the stream to %q", span, out) + } + } + // And no spans at all is the stream itself, not a copy of it. + if out, blanked := redactContent(data, nil); blanked != 0 || string(out) != string(data) { + t.Errorf("no spans gave %q, %d blanked", out, blanked) + } +} + +// ⛔ A RECOLOURING IS NOT A REDACTION, and an unreadable page is the one place +// the difference shows. A page whose content stream cannot be read is, for a +// recolouring, a page left in its own colours — no worse than a colour, and +// what this did before redaction existed. For a redaction the same page is a +// document that must not be written at all, which is the test above. +func TestAnUnreadablePageIsAColourLostAndARedactionRefused(t *testing.T) { + arm := func(t *testing.T) { + was := pageContent + pageContent = func(*reader.Document, int) ([]byte, error) { return nil, errTest } + t.Cleanup(func() { pageContent = was }) + } + t.Run("a recolouring leaves it alone", func(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.RecolourText("all", 1, 0, 0); err != nil { + t.Fatal(err) + } + arm(t) + out, err := d.Bytes() + if err != nil { + t.Fatalf("a page that could not be recoloured failed the write: %v", err) + } + // Left alone means left alone: the text is still there, in whatever + // colour it was. + found := false + for _, content := range contentsOf(t, out) { + if strings.Contains(content, "public") { + found = true + } + } + if !found { + t.Error("the page came back empty rather than unchanged") + } + }) + t.Run("a redaction is refused", func(t *testing.T) { + d, err := Open(secretPage(t, twoLines, nil)) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(overSecret()); err != nil { + t.Fatal(err) + } + arm(t) + if _, err := d.Bytes(); err == nil { + t.Fatal("a page whose stream could not be read was redacted anyway") + } + }) +} + +// A region of no area paints nothing, rather than a rectangle of no width that +// some readers draw as a hairline. +func TestARegionOfNoAreaPaintsNothing(t *testing.T) { + for _, q := range []Quad{QuadOf(10, 10, 10, 20), QuadOf(10, 10, 20, 10)} { + if got := redactionFill([]Redaction{{Page: 1, Quads: []Quad{q}}}); got != "" { + t.Errorf("a quad of no area painted %q", got) + } + } +} + +// Text inside a form that the region does not reach is left alone: the page is +// redacted, and the form — whose bytes are shared — is untouched. +func TestAFormAwayFromTheRegionIsLeftAlone(t *testing.T) { + const body = "BT /F1 10 Tf 20 100 Td (secret) Tj ET q /X1 Do Q" + const inner = "BT /F1 10 Tf 20 20 Td (elsewhere) Tj ET" + pdf := secretPage(t, body, func(w *reader.Writer, res reader.Dict) { + res["XObject"] = reader.Dict{"X1": w.Add(&reader.Stream{Dict: reader.Dict{ + "Type": reader.Name("XObject"), "Subtype": reader.Name("Form"), + "BBox": reader.Array{reader.Integer(0), reader.Integer(0), + reader.Integer(100), reader.Integer(200)}, + "Resources": res, + }, Raw: []byte(inner)})} + }) + d, err := Open(pdf) + if err != nil { + t.Fatal(err) + } + if err := d.Redact(overSecret()); err != nil { + t.Fatalf("a form away from the region was refused: %v", err) + } + out, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + src, err := reader.Open(out) + if err != nil { + t.Fatal(err) + } + text, err := extract.Text(src, 1) + if err != nil { + t.Fatal(err) + } + if strings.Contains(text, "secret") { + t.Errorf("the page still says %q", text) + } + if !strings.Contains(text, "elsewhere") { + t.Errorf("the form's own text was lost: %q", text) + } +} + +// Redacting nothing leaves the page as it was, rather than writing a stream +// that merely looks the same. +func TestARegionWithNoTextInItChangesNoText(t *testing.T) { + out := redacted(t, twoLines, Redaction{Page: 1, Quads: []Quad{QuadOf(0, 0, 5, 5)}}) + src, err := reader.Open(out) + if err != nil { + t.Fatal(err) + } + text, err := extract.Text(src, 1) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"public", "secret"} { + if !strings.Contains(text, want) { + t.Errorf("a redaction over bare paper lost %q: the page reads %q", want, text) + } + } +} diff --git a/write.go b/write.go index 548294b..9b850ac 100644 --- a/write.go +++ b/write.go @@ -63,9 +63,17 @@ func (d *Doc) Bytes() ([]byte, error) { } } } + d.redactErr = nil for i, p := range d.pages { dicts[i] = d.buildPage(w, p, pagesRef, where, kept) } + // ⛔⛔ BEFORE ANYTHING IS HANDED BACK. A page whose redaction could not be + // applied has been built with its text still in it and the fill painted + // over the top — a document that looks redacted. The caller gets the + // failure instead of the file. + if d.redactErr != nil { + return nil, d.redactErr + } // The catalogue is rebuilt before the pages are put down, not after: what // it carries across may have something to write onto a page — the number @@ -206,12 +214,17 @@ func (d *Doc) buildBorrowedPage(w *reader.Writer, p Page, parent reader.Ref, whe // ⛔ The recolouring happens BEFORE the decoration, and the order matters: // a stamp's own colour is the stamp's business, so recolouring the page's // text must not reach the marks this document put on top of it. - if p.recolour != nil { - if content, err := p.src.PageContent(p.number); err == nil { - out, unreached := recolourContent(content, *p.recolour) - d.unreached += unreached - copied["Contents"] = w.Add(contentStream(out, nil)) + if out, rewritten, err := d.rewrittenContent(p); err != nil { + // ⛔ RECORDED, AND THE WHOLE WRITE FAILS ON IT. A page whose redaction + // could not be applied must not be written: the fill would be painted + // over text that is still there, and the caller would be holding a + // document that looks redacted. [Doc.Bytes] checks this before it + // returns anything. + if d.redactErr == nil { + d.redactErr = err } + } else if rewritten { + copied["Contents"] = w.Add(contentStream(out, nil)) } extra, resources := d.borrowedExtras(w, p, src) @@ -249,6 +262,19 @@ func (d *Doc) borrowedExtras(w *reader.Writer, p Page, src reader.Dict) ([]byte, content = append(content, stamp...) resources = mergeResources(resources, stampResources(fonts, alpha)) } + // ⛔ THE REDACTION FILL IS LAST, OVER EVERYTHING, and it goes through here + // rather than being appended to the page's own rewritten bytes. Two reasons, + // both of which were a way to lose it: the page's content may end inside a + // clipping path, which would clip the fill away without a trace, and it may + // end having left a colour or a transform set. decorate wraps the page in a + // save and a restore, so whatever is added here starts from a clean state. + // + // It is a MARKING, not the redaction: the text is already gone from the + // stream by the time this is drawn. What it does is tell a reader that + // something was taken out, which an emptied region otherwise cannot say. + if len(p.redactions) > 0 { + content = append(content, redactionFill(p.redactions)...) + } return content, resources } @@ -264,8 +290,14 @@ func (d *Doc) decorate(w *reader.Writer, p Page, src, copied reader.Dict, extra // that away: a page both recoloured and watermarked came back in its // original colours, and nothing failed. What is here is used when it is // a rewritten stream and the source's parts otherwise. + // + // ⛔ A REDACTION REWRITES IT TOO, and leaving it out of this test would be + // the same defect with a worse ending: a page both redacted and stamped + // would come back with the redacted text still in it, and the fill drawn + // over it would make the document look redacted. A recolouring that is + // undone is a colour; a redaction that is undone is a leak. var parts reader.Array - if rewritten, ok := copied["Contents"].(reader.Ref); ok && p.recolour != nil { + if rewritten, ok := copied["Contents"].(reader.Ref); ok && (p.recolour != nil || len(p.redactions) > 0) { parts = reader.Array{rewritten} } else { parts = contentParts(w, p.src, src)