redact: destroy the pixels of a JPEG, as plain samples

README.md

@@ -1056,20 +1056,49 @@ the region are **overwritten and the picture is written back**. A scanned page
 is one picture, so without this the verb refused everything a scanner produces.
 
 What this does it to is a picture whose samples are **plain** — one the reader
-has already unfiltered — eight bits a component, `DeviceGray` or `DeviceRGB`,
-drawn **upright** and drawn **once** on the page. Each of the others is refused
-by its own name, with the reason:
+has already unfiltered — or a **JPEG**, eight bits a component, `DeviceGray` or
+`DeviceRGB`, drawn **upright** and drawn **once** on the page. Each of the
+others is refused by its own name, with the reason:
 
 | refused | why |
 | --- | --- |
-| still `DCTDecode` or `JPXDecode` | its samples would have to be decoded and re-encoded, which is a size and a quality question of its own: re-encoding as Flate makes a scan five to twenty times larger, and re-encoding as JPEG re-compresses every pixel of the page, including the ones nobody asked about |
+| still `JPXDecode`, or any other filter this does not undo | its samples would have to be decoded and written back, and a decoder this does not have is not one to guess at |
 | drawn **turned or skewed** | the box a turned picture covers looks exactly like the box an upright one covers, so mapping the region into its samples by that box would destroy more than was asked for — which is the other way a redaction betrays its caller. `extract.Image.Matrix` exists for this: `b` and `c` are zero for an upright picture, and stay zero for a flipped one |
 | drawn **more than once** on the page | replacing the picture replaces it for every `Do` that names it, so blanking it for one placement blanks it for the others, in a different place each time |
 | an **inline** image | it is written into the content stream rather than held as a resource, so there is nothing to write back in its place |
 | `/ImageMask`, `/SMask`, `/Mask`, `/Decode` | each changes what a sample means, and a sample set to zero under one of them is not necessarily a destroyed one |
 | `/Indexed`, `/ICCBased`, `/DeviceCMYK`, anything else | in an indexed space a zero is the palette's **first** colour, which may be white; in the others the component count has to be read out of a stream. What a destroyed sample holds there is not something to guess at |
 | samples that are not all there | the stream is shorter or longer than `Width × Height × components`, so what is in it is not what it says |
 
+#### A JPEG is written back as plain samples, not as another JPEG
+
+⛔ **Re-encoding would change the pixels nobody asked about.** A JPEG is
+re-quantised block by block, so writing the blanked picture back as a JPEG
+would degrade **every** block of it — the whole photograph, to destroy a corner
+of it. What this writes instead is exactly what a reader of the original would
+have seen, with the region destroyed, and the test for it decodes the original
+itself and compares the kept samples **byte for byte** against that decode.
+Asserting a colour would be asserting what an encoder does with it, which is a
+different subject and a moving one.
+
+The picture loses its compression: a photograph as Flate is several times the
+same photograph as a JPEG, and that is the trade. It is paid only by a picture
+a region actually reaches — one nobody redacted over keeps its `DCTDecode`
+stream byte for byte.
+
+Two more refusals belong to this path:
+
+| refused | why |
+| --- | --- |
+| a **CMYK** JPEG | there is no faithful CMYK to RGB without the output profile the file does not carry, so a conversion here would change every colour on the page |
+| more than **100 megapixels** | a JPEG says how big it is in a handful of bytes and the decoder then allocates `width × height × 3`, so a file of twenty-five bytes can ask for gigabytes. The size is read from the **header** and refused before anything is decoded — checking after the decode is checking after the allocation. A 600 dpi A4 page is about 35 megapixels |
+
+The test for that ceiling is a real bomb — thirty-nine bytes declaring 20000 by
+20000 and not one sample — run against the **real** bound rather than a knob
+lowered for the occasion, and it asserts that the decoder was never reached:
+the refusal message alone reads the same whether the check runs before the
+decode or after it.
+
 ⛔ **The original picture is not left in the file.** The page's `/Resources` are
 **rebuilt** with the blanked picture in place of the original rather than copied
 — copying them would have written the original into the output beside the

redact.go

@@ -60,10 +60,26 @@ type Redaction struct {
 //     XObject is SHARED — editing it to clean one page would change every page
 //     that draws it. [Doc.RecolourText] stops at the same wall, and counts what
 //     it could not reach for the same reason.
-//   - an image. The spec is explicit that a mask or a clip will not do and the
-//     pixels themselves must be destroyed; until this can re-encode an image
-//     with those pixels overwritten, an image in the region is a refusal.
+//
+//   - an image it cannot write back. The spec is explicit that a mask or a clip
+//     will not do and the pixels themselves must be destroyed, so a picture
+//     whose samples this cannot rewrite is a refusal naming the reason: a
+//     turned or skewed placement, a picture the page draws more than once, a
+//     stencil or a soft mask, a palette, more or fewer than eight bits a
+//     sample, a colour space this does not know. Plain samples and JPEGs are
+//     blanked.
+//
+//     ⛔ A JPEG IS WRITTEN BACK AS PLAIN SAMPLES, not as another JPEG.
+//     Re-encoding re-quantises every block of the picture, including the ones
+//     nobody asked about, so a redaction would visibly degrade the rest of the
+//     page; what this writes instead is exactly what a reader of the original
+//     would have seen, with the region destroyed. The picture loses its
+//     compression and the file grows — a photograph as Flate is several times
+//     the same photograph as a JPEG — which is the trade, and the caller's to
+//     know about.
+//
 //   - an annotation. Its /Contents can hold the very words being redacted.
+//
 //   - a text-showing operator only PART of whose text is inside the region.
 //     Blanking it would take away text the caller did not ask about, and
 //     rewriting it cannot be done by re-emitting the surviving words: the

redactimage.go

@@ -1,7 +1,11 @@
 package ops
 
 import (
+	"bytes"
 	"fmt"
+	"image"
+	"image/draw"
+	"image/jpeg"
 
 	"github.com/go-pdfkit/extract"
 	"github.com/go-pdfkit/reader"
@@ -131,6 +135,24 @@ func blankedImage(page int, im extract.Image, all []extract.Image, quads []Quad)
 	// then has no width or no height, and a box with no width overlaps nothing —
 	// so such a picture never reaches this function at all. The inverse below
 	// would divide by that zero; it cannot be asked to.
+	// ⛔ A JPEG IS DECODED AND WRITTEN BACK AS PLAIN SAMPLES, which is the only
+	// way to destroy its pixels that cannot be accused of changing the ones
+	// nobody asked about. Re-encoding it as a JPEG re-quantises the WHOLE
+	// picture — every block, not just the blanked ones — so a redaction would
+	// visibly degrade the rest of the page. What this does instead is write
+	// down exactly what a reader of the original would have seen, with the
+	// region destroyed, and lose the JPEG rather than the fidelity.
+	//
+	// ⛔ IT COSTS SIZE AND THE TEST SAYS SO: a photograph as Flate is several
+	// times a photograph as JPEG. That is the trade, and it is the caller's to
+	// know about rather than ours to hide.
+	if im.Filter == "DCTDecode" {
+		plain, err := jpegSamples(im)
+		if err != nil {
+			return no("%s", err)
+		}
+		im = plain
+	}
 	if im.Filter != "" {
 		return no("its samples are still %s, and re-encoding them is not in this release", im.Filter)
 	}
@@ -170,6 +192,96 @@ func blankedImage(page int, im extract.Image, all []extract.Image, quads []Quad)
 	return plannedImage{name: reader.Name(im.Name), dict: im.Dict, data: data}, nil
 }
 
+// jpegMegapixels bounds what will be decoded. A JPEG says its size in a
+// handful of bytes and the decoder then allocates width × height × 3, so a
+// file of a few kilobytes can ask for gigabytes. A hundred megapixels is far
+// past any scan — a 600 dpi A4 page is about 35 — and the refusal names the
+// size, so somebody with a genuinely enormous picture learns why rather than
+// watching the machine swap.
+//
+// ⛔ A CONST, NOT A KNOB A TEST LOWERS. The test that proves this feeds a real
+// JPEG header claiming 20000 by 20000 in twenty-five bytes, so what runs in CI
+// is the bound callers get rather than one arranged for the occasion.
+const jpegMegapixels = 100
+
+// decodeJPEG is replaceable so that a test can hand back the shapes a real
+// encoder will not produce — image/jpeg writes Gray, YCbCr and RGBA and has no
+// way to write the CMYK the refusal below is about, so without this seam that
+// refusal would be a claim no test could reach.
+var decodeJPEG = jpeg.Decode
+
+// jpegSamples decodes a DCTDecode picture into the samples a reader would have
+// drawn, so that the rest of this file can blank them.
+//
+// ⛔ THE SIZE IS READ FROM THE HEADER BEFORE ANYTHING IS DECODED. Checking
+// after the decode is checking after the allocation, which is the whole of
+// what the bound is for.
+func jpegSamples(im extract.Image) (extract.Image, error) {
+	cfg, err := jpeg.DecodeConfig(bytes.NewReader(im.Data))
+	if err != nil {
+		return im, fmt.Errorf("its samples are a JPEG this cannot read: %w", err)
+	}
+	if px := cfg.Width * cfg.Height; px > jpegMegapixels*1e6 {
+		return im, fmt.Errorf("it is %d by %d samples, past the %d megapixels this will decode",
+			cfg.Width, cfg.Height, jpegMegapixels)
+	}
+	img, err := decodeJPEG(bytes.NewReader(im.Data))
+	if err != nil {
+		return im, fmt.Errorf("its samples are a JPEG this cannot read: %w", err)
+	}
+	b := img.Bounds()
+	if b.Dx() != im.Width || b.Dy() != im.Height {
+		// ⛔ The dictionary and the picture disagreeing is not something to
+		// average out: every box this maps into the samples comes from the
+		// dictionary's size, so blanking against the other one would destroy
+		// the wrong rows.
+		return im, fmt.Errorf("its dictionary says %d by %d and the JPEG in it is %d by %d",
+			im.Width, im.Height, b.Dx(), b.Dy())
+	}
+
+	var data []byte
+	var space reader.Name
+	switch src := img.(type) {
+	case *image.Gray:
+		space, data = "DeviceGray", make([]byte, 0, b.Dx()*b.Dy())
+		for y := b.Min.Y; y < b.Max.Y; y++ {
+			data = append(data, src.Pix[(y-b.Min.Y)*src.Stride:][:b.Dx()]...)
+		}
+	case *image.CMYK:
+		// ⛔ REFUSED RATHER THAN CONVERTED. There is no faithful CMYK to RGB
+		// without the output profile this file does not carry, so a conversion
+		// here would change every colour on the page to destroy a corner of it.
+		return im, fmt.Errorf("it is a CMYK JPEG, and turning those colours into RGB without " +
+			"the profile the file does not carry would change every one of them")
+	default:
+		space = "DeviceRGB"
+		rgba := image.NewRGBA(image.Rect(0, 0, b.Dx(), b.Dy()))
+		draw.Draw(rgba, rgba.Bounds(), img, b.Min, draw.Src)
+		data = make([]byte, 0, b.Dx()*b.Dy()*3)
+		for y := 0; y < b.Dy(); y++ {
+			row := rgba.Pix[y*rgba.Stride:][:b.Dx()*4]
+			for x := 0; x < len(row); x += 4 {
+				data = append(data, row[x], row[x+1], row[x+2])
+			}
+		}
+	}
+
+	// The dictionary now describes plain samples. Width and Height are the
+	// ones it already carried, which the check above proved are the picture's.
+	dict := reader.Dict{}
+	for k, v := range im.Dict {
+		switch k {
+		case "Filter", "DecodeParms", "DP", "F", "Length":
+			continue
+		}
+		dict[k] = v
+	}
+	dict["ColorSpace"] = space
+	dict["BitsPerComponent"] = reader.Integer(8)
+	im.Dict, im.Data, im.Filter = dict, data, ""
+	return im, nil
+}
+
 // blankSamples destroys the samples of one quad's box, mapped from the page
 // into the picture's own grid.
 //

redactimage_test.go

@@ -229,9 +229,16 @@ func TestWhatARedactionOverAPictureRefuses(t *testing.T) {
 		{"skewed", "q 100 20 0 100 50 50 cm /Im1 Do Q", nil, "turned or skewed"},
 		{"drawn twice", "q 100 0 0 100 50 50 cm /Im1 Do Q q 20 0 0 20 0 0 cm /Im1 Do Q", nil,
 			`draws "Im1" 2 times`},
-		{"a JPEG", "q 100 0 0 100 50 50 cm /Im1 Do Q", func(d reader.Dict) {
+		// ⛔ NOT "A JPEG" ANY MORE. A DCTDecode picture is decoded and written
+		// back as plain samples since #82; what stays refused here is a stream
+		// that SAYS DCTDecode and does not hold a JPEG, which is a file this
+		// cannot read rather than one it will not.
+		{"a picture that says JPEG and is not one", "q 100 0 0 100 50 50 cm /Im1 Do Q", func(d reader.Dict) {
 			d["Filter"] = reader.Name("DCTDecode")
-		}, "samples are still DCTDecode"},
+		}, "a JPEG this cannot read"},
+		{"a filter this does not undo", "q 100 0 0 100 50 50 cm /Im1 Do Q", func(d reader.Dict) {
+			d["Filter"] = reader.Name("JPXDecode")
+		}, "samples are still JPXDecode"},
 		{"an image mask", "q 100 0 0 100 50 50 cm /Im1 Do Q", func(d reader.Dict) {
 			d["ImageMask"] = reader.Bool(true)
 		}, "image mask"},