Repository navigation
linearize: bound /S above where it costs nothing — an unfiltered hint stream - #48
Merged
Merged
Conversation
tannevaled
force-pushed
the
feat/bound-S-in-an-unfiltered-hint-stream
branch
4 times, most recently
from
November 15, 2026 09:45
1ce0931 to
c161723
Compare
… stream ⭐ THE OPEN LIMIT WAS SMALLER THAN I SAID. /S indexes the DECODED data, so bounding it above normally means decoding an attacker's bytes in the cheap path — filter.go caps expansion at 1 GiB, which refuses a bomb but is three orders of magnitude past anything a hint stream needs. That is still true and still wants a size bound derived from /N and the object count (#40). But a stream with NO FILTER needs no decoding at all: its raw bytes ARE its data. That case is free, and it is the one that matters most while a writer is being developed, because an uncompressed hint stream is what somebody debugging one emits. Ghostscript already writes one unfiltered, and qpdf does under --compress-streams=n. ⛔⛔ AND THE FILTER CONDITION IS LOAD-BEARING, WHICH THE CORPUS PROVED. Applying the bound to filtered streams as well turns the generated sweep's TWENTY-FIVE page cases red — `as qpdf writes it, 25_pages` and `with object streams, 25_pages`. For a document that size the hint table compresses well below /S, so /S is past the RAW length while sitting comfortably inside the DECODED one. The 1, 2 and 5 page cases cannot see it: their tables are too small to compress under 43. So this is not defensive coding, it is required, and only a corpus with a large file could say so. ⛔ ASKED THROUGH filterChain, NOT BY LOOKING FOR THE KEY. /Filter may be present and Null, or an empty array, both of which mean no filtering; `_, ok := s.Dict["Filter"]` would call those filtered and skip the one check they are eligible for. ⛔⛔ AND THE FIRST VERSION OF THIS CHANGE WAS GREEN, MEASURED, AND UNPINNED AT EVERY EDGE — WHICH REVIEW CAUGHT AND MY OWN MUTATIONS DID NOT. I mutated the two things I had been thinking about (the filter condition, and the bound's existence) and both died, so I called the work done. Review mutated the bound's COMPARISON instead and found three survivors: `>`, `>= len-1` and `>= len+1` all left the whole suite GREEN. The cause is that the only values the suite exercised were /S 99 against 67 bytes and /S 43 against 64 — both far from the edge, so any comparison in the neighbourhood passes. A fourth survivor was the key lookup above: no fixture had `/Filter null` or `/Filter [ ]`, so the stated REASON for calling filterChain was the one thing nothing held. ⛔ A MUTATION ONLY MEASURES THE QUESTION IT ASKS. Mine asked "is this condition load-bearing?" and got a true answer. It could not ask "is this boundary the right one?", because I chose the mutations from what I had just been thinking about — which is exactly where I am least likely to be wrong. The lesson is not "mutate more", it is that the mutation set must come from the SHAPE of the code (a comparison has three neighbours; a documented reason needs a case) and not from the author's recollection of the risk. So four tests land with it, each pinning one survivor: /S == len-1 accepted kills `>= len-1` /S == len refused kills `>` and `>= len+1` /Filter null, /S past kills the key lookup /Filter [ ], /S past kills it the other spelling /Filter /FlateDecode kills dropping the condition, LOCALLY — it was only the 25-page sweep that held this before, which is a slow and indirect witness for one line ⛔ AND /S ABOVE THE END IS DELIBERATELY ABSENT, because it kills nothing the `== len` case does not already kill. An extra case that cannot fail reports its own index rather than a measurement. ⛔ hintFixtureData IS A CONSTANT AND THE CASES COUNT WITH IT. `/S 64` written beside a fixture that happens to hold 64 bytes stops testing the boundary the moment either number is edited alone, and the suite stays green — the shape this package has already paid for under "judge and subject must count alike". ⛔ AND GHOSTSCRIPT SPACE-PADS ITS INTEGERS, which the first version of the new test got wrong: `<</Length 67\n/S 38>>`. Anchoring on "/S 38" matched nothing and the helper's uniqueness check said so rather than letting a silent zero-replacement pass. Worth knowing beyond this test: that padding is the FIXED-WIDTH PLACEHOLDER technique a two-pass linearizing writer needs for /L, /H, /E and /T, so the corpus already holds a worked example of the fixpoint trick #40 describes. ⛔⛔ AND A CONTROL WRITTEN ONE CHANGE AGO CAUGHT THE FIXTURE, NOT THE CODE. TestEveryNonIntegerSharedTablePositionIsRefused's "an integer is accepted" subtest went red the moment this bound existed: its hand-built stream carried `/Length 0`, and /S 43 does point outside zero bytes. The bound was right and the fixture was unrealistic. Without that control the seven kind cases above it would have gone on passing for the WRONG REASON — refused by the length check rather than the kind check, which is the "refused for a reason that is not the reason" shape this file has already paid for twice. The fixture now carries 64 bytes, through a helper that says why. ⛔⛔ AND THE SECOND REVIEW FOUND A SIXTH SURVIVOR, WHOSE SUGGESTED TEST WOULD HAVE FAILED. Mutating the condition to `(ferr == nil && len(names) == 0) || ferr != nil` — applying the bound to a chain it could not read — was green: the same shape again, a documented reason with no case. But review's proposed fixture was `/Filter 9 0 R`, on the stated ground that an indirect filter makes filterChain error under a nil resolver. It does not. Resolve returns `Null{}, nil` for a Ref when the resolver is nil (object.go:162), so an indirect /Filter takes filterChain's `case Null:`, names stays empty, the stream is judged UNFILTERED, and the file is REFUSED — the opposite of the acceptance the suggested test asserts. Reading object.go rather than taking the mechanism on trust is the whole of the difference. ⛔ THE TEST FOR IT ASSERTS ITS OWN PREMISE. `/Filter 5` reaches filterChain's `default:` and is the spelling that actually makes it error, so the test calls filterChain itself and fails loudly if that ever stops erroring — the day it reads as a valid chain is the day the test stops exercising the branch it names while staying green. ⛔⛔⛔ AND THEN THE THIRD ROUND FOUND THE REAL DEFECT, WHICH WAS MINE AND WHICH I HAD WRITTEN A TEST TO BLESS. Having learned that an indirect /Filter reads as "no filter", I pinned THAT as intended behaviour and argued for it: the reader this format exists for cannot resolve anything, so treating a reference as "no filter it can use" is what it would do. Review supplied the input that kills the argument. If object 9 holds /FlateDecode, the stream really IS filtered, /S really does index decoded data longer than the raw bytes, and the refusal is for a reason the check cannot evaluate — which is WORD FOR WORD the argument I had just used, one test higher in the same file, to ACCEPT `/Filter 5`. Two cases that cannot be told apart, decided opposite ways, with a test holding each in place. ⛔⛔ A TEST THAT PINS A WRONG BEHAVIOUR IS WORSE THAN NO TEST, because it converts an oversight into a specification. The mutation set cannot find this class at all: every mutation of the old condition was correctly killed by TestAnIndirectFilterIsJudgedAsNoFilter, which is exactly the problem — the tests agreed with the code, and both were wrong. Only a reader asking "what does this REFUSE, and can the check know it?" gets there. So the predicate is now rawIsTheData, and it answers a THREE-way question as such: "filtered", "not filtered" and "cannot tell" are distinct, and only the second licenses the bound. An indirect /Filter is the third, so the bound is left alone and the file is accepted HERE. It is still non-conforming — F.3.6 requires every hint-dictionary entry beyond the stream Length to be direct — but that refusal is a NAMED property tabled on #40, and #40 should make it by name rather than this bound making it as a side effect, with a message about the shared object table that would send the next reader to the wrong place. ⛔ `/Filter [ 9 0 R ]` needs no case of its own: filterChain resolves the element to Null, ToName fails, and the error is the same middle answer. It is in the predicate's own table for the record, not as a separate claim. ⛔⛔ AND LOOKING FOR A SECOND INSTANCE OF THE CLASS FOUND ONE: /F. Rather than re-checking the case just fixed, the fourth round asked whether anything else refuses on something it cannot evaluate, and 7.3.8.2 Table 5 answers — when a stream dictionary carries /F its data lives in an EXTERNAL FILE, and the bytes between `stream` and `endstream` are not it. The parse slices them into Raw anyway, so len(Raw) measures a placeholder and bounding a decoded position against it refuses on a length belonging to different bytes. No /Filter need be involved at all, which is what makes it a case and not a variant. ⛔ The predicate is named rawIsTheData, and with /F present that name would be false. A predicate that is wrong in a case it has been told about is worth one line more than an issue describing it. It DECLINES TO BOUND rather than refusing: a hint stream with /F is wrong, and saying so is #40's job. ⛔ AND THE BARE PRESENCE TEST IS ONLY ADEQUATE FOR A STREAM DICTIONARY, which is written into the comment because it is a constraint on REUSE and not a property of the format. /F is a file specification in Table 5, but the same name is an unrelated flag field elsewhere — an annotation's /F is an integer of bit flags. Neither can reach a hint stream, so presence is enough here; a future caller passing a dictionary of another kind would need the key read for what it means there rather than merely counted. ⛔ AND THE PREDICATE IS TESTED AS A PREDICATE. Reaching it only through fixtures cannot distinguish "answered false" from "never asked", so TestRawIsTheDataAnswersOnlyWhenItKnows states all eight answers directly — absent, Null, empty array, a name, a chain, a reference, an array holding a reference, and an impossible kind. A refactor that keeps checkHintStream green while changing what the predicate MEANS now goes red. ⛔ AND A SECOND /F ROW WAS WRITTEN AND THEN DELETED, which is the same rule applied to myself. `{/F, /Filter null}` reddens, but it kills nothing the /F row alone does not — and /S above the end was excluded a few paragraphs up on exactly that ground. Keeping it because it looked thorough while excluding the other because the rule said so is the incoherence this commit was corrected for, one level up. Mutations, judged by cmd/failset on the failing NAMES, nine of nine killed. The boundary pair and the gate were re-run AFTER the refactor, because the condition moved and a mutation proved against the old shape says nothing about the new one: `>` instead of `>=` -> …MustLeaveRoomForItsOwnFirstByte/exactly_at_the_end `>= len-1` -> …MustLeaveRoomForItsOwnFirstByte/one_byte_inside `>= len+1` -> …MustLeaveRoomForItsOwnFirstByte/exactly_at_the_end the `s.Dict["Filter"]` key lookup -> …AFilterThatFiltersNothingIsStillBounded, both cases drop the condition -> …ARealFilterLeavesTheSharedTablePositionUnbounded AND the two 25_pages sweep cases bound an UNREADABLE chain as well -> …AFilterThatCannotBeReadLeavesTheBoundAlone, and no other drop rawIsTheData's Ref guard -> …AnIndirectFilterIsNotSomethingThisCheckCanRead and …AnswersOnlyWhenItKnows/an_indirect_/Filter, and no other drop the /F guard -> …AnswersOnlyWhenItKnows/an_external_file_stream, and no other invert the predicate (`len(names) != 0`) -> 16 tests, five of the predicate's own answers among them Note that the first two mutations redden DIFFERENT subtests, so the two cases hold opposite sides of the boundary rather than one of them carrying both. /S == len is refused on the claim that the shared object hint table opens with a Table F.5 header and so occupies at least one byte: a table starting AT the end of the data has nowhere to put it, which makes /S == len unrepresentable in a conforming file rather than merely unusual. ⛔ FOUR REVIEW ROUNDS, EACH FINDING SOMETHING THE PREVIOUS ONE COULD NOT. Round one: the boundary was unpinned. Round two: an unreadable chain was unpinned, and the fixture suggested for it would have failed. Round three: the behaviour I had just pinned was itself wrong. Round four, asked to look for a SECOND instance of that class rather than re-check the first: /F. The first two are test gaps a mutation finds; the third is a wrong SPECIFICATION, which no mutation can find because the tests and the code agreed; the fourth came from naming the class and hunting it. gofmt clean, go vet clean, 100.0% of statements, 0 uncovered blocks of 1820 under REQUIRE_QPDF=1 (CI's own command, replayed rather than approximated). Refs #40. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
force-pushed
the
feat/bound-S-in-an-unfiltered-hint-stream
branch
from
November 15, 2026 09:49
c161723 to
76721e3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⭐ The open limit was smaller than I said it was.
/Sindexes the decoded data, so bounding it above normally means decoding an attacker's bytes in the cheap path —filter.gocaps expansion at 1 GiB, which refuses a bomb but is three orders of magnitude past anything a hint stream needs. That is still true, and still wants a size bound derived from/Nand the object count (#40).But a stream with no filter needs no decoding at all: its raw bytes are its data. That case is free — and it is the one that matters most while a writer is being developed, because an uncompressed hint stream is what somebody debugging one emits. Ghostscript already writes one unfiltered; qpdf does under
--compress-streams=n.⛔ The filter condition is load-bearing, and the corpus proved it
Applying the bound to filtered streams as well turns the generated sweep's twenty-five page cases red —
as qpdf writes it, 25_pagesandwith object streams, 25_pages.For a document that size the hint table compresses well below
/S, so/Sis past the raw length while sitting comfortably inside the decoded one. The 1, 2 and 5 page cases cannot see this: their tables are too small to compress under 43.So the condition is not defensive coding — it is required, and only a corpus containing a large file could say so.
⛔ Asked through
filterChain, not by looking for the key:/Filtermay be present andNull, or an empty array, both of which mean no filtering, and_, ok := s.Dict["Filter"]would call those filtered and skip the one check they are eligible for.⛔⛔ The first version of this was green, measured, and unpinned at every edge
Review caught what my own mutations could not. I had mutated the two things I was thinking about — the filter condition, and the bound's existence — and both died, so I called it done. Review mutated the bound's comparison instead:
>instead of>=>= len-1>= len+1s.Dict["Filter"]key lookupThe cause is that the only values the suite exercised were
/S 99against 67 bytes and/S 43against 64 — both far from the edge, so any comparison in the neighbourhood passes. And no fixture carried/Filter nullor/Filter [ ], so the stated reason for callingfilterChainwas the one thing nothing held.⛔ A mutation only measures the question it asks. Mine asked "is this condition load-bearing?" and got a true answer; it could not ask "is this boundary the right one?", because I chose the mutations from what I had just been thinking about — which is exactly where I am least likely to be wrong. The lesson is not "mutate more": the mutation set has to come from the shape of the code (a comparison has three neighbours; a documented reason needs a case), not from the author's recollection of the risk.
What lands with it
Four tests, each pinning one survivor:
/S == len-1accepted,/S == lenrefused — two cases that redden different subtests, so they hold opposite sides of the boundary rather than one carrying both./Filter nulland/Filter [ ], each with/Spast the end, refused./Filter /FlateDecodewith the same/S, accepted — the asymmetry stated locally instead of only at 25 pages.⛔⛔ Re-review found a sixth survivor, and its suggested test would have failed
Mutating the condition to
(ferr == nil && len(names) == 0) || ferr != nil— applying the bound to a chain it could not read — was green. Same shape a third time: a documented reason with no case.But the proposed fixture was
/Filter 9 0 R, on the stated ground that an indirect filter makesfilterChainerror under a nil resolver. It does not.ResolvereturnsNull{}, nilfor aRefwhen the resolver is nil (object.go:162), so an indirect/FiltertakesfilterChain'scase Null:,namesstays empty, the stream is judged unfiltered, and the file is refused — the opposite of the acceptance that test asserts. Readingobject.gorather than taking the mechanism on trust is the whole of the difference./Filter 5is the spelling that actually reachesfilterChain'sdefault:, and it is accepted: an unreadable chain means this check cannot know whether the raw bytes are the data, and refusing on a bound it cannot evaluate would refuse for a reason it does not have.⛔ That test asserts its own premise — it calls
filterChainon/Filter 5and fails loudly if that ever stops erroring, since the day it reads as a valid chain is the day the test stops exercising the branch it names while staying green.⛔⛔⛔ And the third round found the real defect, which was mine, and which I had written a test to bless
Having learned that an indirect
/Filterreads as "no filter", I pinned that as intended behaviour and argued for it: the reader this format exists for cannot resolve anything, so treating a reference as "no filter it can use" is what it would do.Review supplied the input that kills the argument. If object 9 holds
/FlateDecode, the stream really is filtered,/Sreally does index decoded data longer than the raw bytes, and the refusal is for a reason the check cannot evaluate — which is word for word the argument I had used, one test higher in the same file, to accept/Filter 5. Two cases that cannot be told apart, decided opposite ways, with a test holding each in place.⛔⛔ A test that pins a wrong behaviour is worse than no test, because it converts an oversight into a specification. No mutation can find this class: every mutation of the old condition was correctly killed by
TestAnIndirectFilterIsJudgedAsNoFilter, and that is exactly the problem — the tests agreed with the code and both were wrong. Only a reader asking "what does this refuse, and can the check know it?" gets there.So the condition becomes a named predicate that answers a three-way question as such — "filtered", "not filtered", "cannot tell", with only the second licensing the bound:
⛔⛔ That
/Fline came from looking for a second instance of the class rather than re-checking the first. 7.3.8.2, Table 5: when a stream dictionary carries/Fits data lives in an external file, and the bytes betweenstreamandendstreamare not it. The parse slices them intoRawanyway, solen(Raw)measures a placeholder and bounding a decoded position against it refuses on a length belonging to different bytes — with no/Filterinvolved at all, which is what makes it a case and not a variant. The predicate is namedrawIsTheData, and with/Fpresent that name would be false.An indirect
/Filteris the third answer, so the bound is left alone and the file is accepted here. It is still non-conforming — F.3.6 requires every hint-dictionary entry beyond the streamLengthto be direct — but that refusal is a named property tabled on #40, and #40 should make it by name rather than this bound making it as a side effect, with a message about the shared object table that would send the next reader to the wrong place.⛔
/Filter [ 9 0 R ]needs no case of its own:filterChainresolves the element toNull,ToNamefails, and the error is the same middle answer. It is in the predicate's table for the record, not as a separate claim.⛔ And the predicate is tested as a predicate. Reaching it only through fixtures cannot distinguish "answered false" from "never asked", so
TestRawIsTheDataAnswersOnlyWhenItKnowsstates all nine answers directly — absent,Null, empty array, a name, a chain, a reference, an array holding a reference, an impossible kind, and an external-file stream.⛔
/Sabove the end is deliberately absent: it kills nothing the== lencase does not already kill, and an extra case that cannot fail reports its own index rather than a measurement.⛔
hintFixtureDatais a constant and the cases count with it./S 64written beside a fixture that happens to hold 64 bytes stops testing the boundary the moment either number is edited alone, and the suite stays green.The refusal at
/S == lenrests on the claim that the shared object hint table opens with a Table F.5 header and so occupies at least one byte: a table starting at the end of the data has nowhere to put it, which makes/S == lenunrepresentable in a conforming file rather than merely unusual.⛔ Ghostscript space-pads its integers
The first version of the new test anchored on
/S 38and matched nothing:The helper's uniqueness check said so, rather than letting a silent zero-replacement pass as a test that could not fail.
⭐ Worth knowing beyond this test: that padding is the fixed-width placeholder technique a two-pass linearizing writer needs for
/L,/H,/Eand/T. The corpus already holds a worked example of the fixpoint trick #40 describes.⛔⛔ A control written one change ago caught the fixture, not the code
TestEveryNonIntegerSharedTablePositionIsRefused's "an integer is accepted" subtest went red the moment this bound existed: its hand-built stream carried/Length 0, and/S 43does point outside zero bytes.The bound was right and the fixture was unrealistic. Without that control the seven kind cases above it would have gone on passing for the wrong reason — refused by the length check rather than the kind check, which is the "refused for a reason that is not the reason" shape this file has already paid for twice. The fixture now carries 64 bytes, through a helper that says why.
Mutations
Judged by
cmd/failseton the failing names, six of six killed:Nine of nine killed.
⛔ A second
/Frow —{/F, /Filter null}— was written and then deleted. It reddens, but kills nothing the/Frow alone does not, and/Sabove the end was excluded earlier on exactly that ground. Keeping one because it looked thorough while excluding the other because the rule said so would be the same incoherence this PR was corrected for, one level up. The first two redden different subtests, so the two boundary cases hold opposite sides rather than one carrying both. The boundary pair and the gate were re-run after the refactor, since the condition moved and a mutation proved against the old shape says nothing about the new one.⛔ Four review rounds, each finding something the previous one could not. Round one: the boundary was unpinned. Round two: an unreadable chain was unpinned, and the fixture suggested for it would have failed. Round three: the behaviour I had just pinned was itself wrong. Round four, asked to look for a second instance of that class rather than re-check the first:
/F. The first two are test gaps a mutation finds; the third is a wrong specification, which no mutation can find because the tests and the code agreed; the fourth came from naming the class and hunting it.gofmt clean,
go vetclean, 100.0% of statements, 0 uncovered blocks of 1820 — underREQUIRE_QPDF=1, replaying CI's own command rather than approximating it.Refs #40.
🤖 Generated with Claude Code