Repository navigation
A ceiling on what a PNG may ask DrawPNG to allocate - #58
Merged
Merged
Conversation
Closes #43. A PNG's HEADER decides the allocation. IHDR's width and height are four-byte unsigned integers (W3C PNG 3rd ed. §11.2.1) and nothing in the file has to back them up: seventy-five bytes claiming 20000x20000 cost 1.5 GiB before png.Decode reads a single sample. The identical shape in go-pdfkit/convert held 381.6 MiB for a 10000x10000 claim. DrawPNG now reads IHDR with png.DecodeConfig and refuses past Options.MaxPixels BEFORE decoding -- a ceiling applied to what a decoder returns is not a ceiling, it is a post-mortem. The spec asks for the same order: "PNG decoders should detect errors as early as possible" (§4.10). The product is taken in int64 because both factors are the file's and range over 0..2^31-1. Options.MaxPixels defaults to DefaultMaxPixels = 40_000_000, the number go-pdfkit/render and go-gfx/gfx's svg package already use: a hundred and sixty million bytes, 152 MiB, at the four bytes a pixel image/png returns. ⛔ WHAT IS DELIBERATELY NOT BOUNDED, because "it has a limit" otherwise reads as "everything has one": - DrawJPEG decodes nothing. The bytes become a DCTDecode stream and the geometry comes from SOF, so a hundred-megapixel scan costs its own length. A ceiling there would refuse real documents to prevent no allocation. - DrawImage is handed pixels the caller already holds; refusing them saves nothing and it copies them once more, a constant factor. The w*h*3 in samples() would overflow int on a 32-bit build only past 715 megapixels -- 2.8 GB of pixel buffer the caller could not have addressed to get here. ⛔ The post-decode check go-pdfkit/convert#4 also makes is NOT repeated here: png.DecodeConfig and png.Decode take their dimensions from the same IHDR, so a second comparison could not differ, and an unreachable branch is a comfort rather than a check. That is an assumption about image/png, not a theorem, so TestTheDecodedSizeIsTheSizeTheHeaderDeclared holds it up: if a Go release ever makes the two disagree, it goes red rather than the gap being found through a 1.5 GiB allocation. ⛔⛔ THE BOMB TEST CARRIES ITS OWN CONTROL. A child process caps its own address space at 1 GiB (RLIMIT_AS, set in the child because os/exec has no rlimit) and must refuse the bomb and exit cleanly. A clean exit proves nothing on its own, so a second child decodes the same bytes with no ceiling under the same limit and MUST die: if it ever survives, the limit is not being applied and the first child's exit meant nothing. macOS rejects RLIMIT_AS outright, so there both children skip -- loudly, naming the platform, with the heap measurement left to carry the proof. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
⛔ The first version of this test failed on CI in the words of the defect it was written to catch: "fatal error: runtime: out of memory". The stack named ceiling_test.go:29 -- claimingPNG, encoding a ONE-PIXEL fixture, long before DrawPNG was called. An absolute RLIMIT_AS of 1 GiB is not a limit on a Go program, it is a sentence. The runtime reserves a great deal of address space at startup -- arena hints and the page allocator's summaries -- so by the time the child could set a limit on itself it was already over this one, and every later mmap failed, including a 1x1 png.Encode. What a test can bound is what the program asks for FROM HERE. capAddressSpace now reads the process's current size from /proc/self/statm and allows it 512 MiB more: ample for the fixture and whatever the runtime wants next, and a third of what the bomb needs. The fixture is also built BEFORE the limit comes down, so the test cannot die making its own evidence. macOS rejects RLIMIT_AS outright (EINVAL), and the relative form needs /proc/self/statm, so the helper is now linux-only and everywhere else skips loudly -- naming the platform -- with the heap measurement left to carry the proof. The "bare" control child is unchanged and is what proves the limit bites: it decodes the same bytes with no ceiling under the same headroom and must die. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
⛔⛔ I wrote "the same ceiling go-pdfkit/render uses" in the doc comment, in the
README and in the pull request, and had not opened render.go. It says
const defaultMaxPixels = 40 << 20 (render.go:79)
which is 41 943 040, not the 40 000 000 I had copied from go-gfx/gfx's svg
package. A difference of 1 943 040 pixels, asserted three times as an equality.
gfx says it too, in two places -- svg/svg.go:79 and README.md:46 -- and
codec/codec.go:201 carries the same constant. So three repositories were
asserting a parity none of them had opened the file to confirm, and pdfkit was
about to be the fourth. Filed as go-gfx/gfx#92.
DefaultMaxPixels is now 40 << 20: render's number, read there, with the file
and line in the comment. It is also the one whose arithmetic is exact -- four
bytes a pixel is 160 MiB on the nose, where 40 000 000 is 152.6 MiB described
as "a hundred and sixty million bytes".
Neither number was more correct than the other. What was wrong was a statement
of fact with nothing behind it, which is the kind of thing that travels. The
change-detector test now also fails if the default drifts back onto
40_000_000, and says why.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI's first green run of this test proved less than it looked like. Every step passed, including "Test with coverage" -- but CI runs go test without -v, and a SKIP and a PASS are the same colour there. If capAddressSpace had returned an error on the runner, the child would have skipped, the parent would have skipped, and the lane would have been green over a bomb test that never fired a single byte at anything. On Linux there is no excuse: /proc/self/statm and RLIMIT_AS are both there. So a skip on Linux is now a FAILURE that says the test has stopped testing anything, and the only platforms permitted to skip are the ones the message names. A pass on Linux now means both halves ran: the guarded child refused the bomb and exited cleanly, AND the bare control died under the same headroom. Neither alone would be worth a green light. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #43.
#43 reported this and declined to fix it, on the grounds that
DrawPNGis awriter's API and deciding who
pdfkit's callers are belongs to themaintainers. Here is that decision, and the reasoning, because the answer is
not "add a limit to everything".
The number the defect is made of
IHDR's width and height are four-byte unsigned integers over
0..2³¹-1(W3CPNG 3rd ed. §11.2.1) and nothing in the file has to back them up.
png.Decodeallocates from them before it reads a sample.Measured here, against this branch with the check cut back out:
"decode png: not enough pixel data""a PNG of 20000x20000 is 400000000 pixels, past the 40000000 allowed"1526.0 MiB is the same figure
go-gfx/gfx#72measured for the same bomb, andgo-pdfkit/convert#4measured 381.6 MiB for a 10000×10000 claim.The check reads IHDR with
png.DecodeConfigand refuses before decoding.⛔ A ceiling applied to what a decoder returns is not a ceiling, it is a
post-mortem — and the spec asks for the same reading order: "PNG decoders
should detect errors as early as possible" (§4.10). The product is taken in
int64because both factors are the file's.Options.MaxPixelsdefaults toDefaultMaxPixels=40 << 20= 41 943 040— exactly 160 MiB at the four bytes a pixel
image/pngreturns, a little overA4 at 600 dpi.
⛔ What is deliberately left unbounded
Stating this because "it has a limit" otherwise reads as "everything has one",
and a limit added where it saves nothing only refuses real documents.
DrawJPEGdecodes nothing. The bytes become a DCTDecode stream and thegeometry comes from SOF, so a hundred-megapixel scan costs pdfkit its own
length. Nor are its dimensions a lie the consumer discovers later: the
consumer decodes the same JPEG.
DrawImageis handed pixels the caller already holds. Refusing themsaves nothing; it copies them once more, a constant factor rather than an
amplification.
samples()'sw*h*3would overflowinton a 32-bit buildonly past 715 megapixels — 2.8 GB of pixel buffer the caller could not have
addressed to get here.
⛔ And one check that is deliberately not copied
go-pdfkit/convert#4measures the size again after decoding, because ingeneral a header is a claim. Not repeated:
png.DecodeConfigandpng.Decoderead their dimensions from the same IHDR, so a second comparison could not
differ, and an unreachable branch is a comfort rather than a check.
That is an assumption about
image/png, not a theorem — soTestTheDecodedSizeIsTheSizeTheHeaderDeclaredholds it up. If a Go releaseever makes the two disagree it goes red, instead of the gap being found through
a 1.5 GiB allocation.
⛔⛔ The bomb test carries its own control
A child process caps its own address space at 1 GiB (
RLIMIT_AS, set inthe child because
os/exechas no rlimit of its own) and must refuse the bomband exit cleanly. The bomb never touches the host's memory, which matters on a
machine running two dozen sessions.
A clean exit proves nothing by itself, so a second child decodes the same
bytes with no ceiling under the same limit and must die. If it ever survives,
the limit is not being applied and the first child's exit meant nothing.
macOS rejects
RLIMIT_ASoutright (EINVAL), so there both children skip —loudly, naming the platform, leaving the heap measurement to carry the proof.
Controls
Run against this branch with the two guard statements removed — a behavioural
cut, not a syntactic one, so nothing failed for being unbuildable:
The last one is the point: a ceiling tested only from above is satisfied by
refusing everything.
No timings are quoted — this machine was at a 1-minute load of 19.2. Held-heap
figures are per-process and unaffected.
🤖 Generated with Claude Code