Skip to content

linearize: bound /S above where it costs nothing — an unfiltered hint stream - #48

Merged
tannevaled merged 1 commit into
mainfrom
feat/bound-S-in-an-unfiltered-hint-stream
Nov 15, 2026
Merged

tannevaled merged 1 commit into
mainfrom
feat/bound-S-in-an-unfiltered-hint-stream

Conversation

@tannevaled

@tannevaled tannevaled commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

⭐ The open limit was smaller than I said it was.

/S indexes the decoded data, so bounding it above normally means decoding an attacker's bytes in the cheap path — filter.go caps expansion at 1 GiB, which refuses a bomb but is three orders of magnitude past anything a hint stream needs. That is still true, and still wants a size bound derived from /N and the object count (#40).

But a stream with no filter needs no decoding at all: its raw bytes are its data. That case is free — and it is the one that matters most while a writer is being developed, because an uncompressed hint stream is what somebody debugging one emits. Ghostscript already writes one unfiltered; qpdf does under --compress-streams=n.

⛔ The filter condition is load-bearing, and the corpus proved it

Applying the bound to filtered streams as well turns the generated sweep's twenty-five page cases red — as qpdf writes it, 25_pages and with object streams, 25_pages.

For a document that size the hint table compresses well below /S, so /S is past the raw length while sitting comfortably inside the decoded one. The 1, 2 and 5 page cases cannot see this: their tables are too small to compress under 43.

So the condition is not defensive coding — it is required, and only a corpus containing a large file could say so.

⛔ Asked through filterChain, not by looking for the key: /Filter may be present and Null, or an empty array, both of which mean no filtering, and _, ok := s.Dict["Filter"] would call those filtered and skip the one check they are eligible for.

⛔⛔ The first version of this was green, measured, and unpinned at every edge

Review caught what my own mutations could not. I had mutated the two things I was thinking about — the filter condition, and the bound's existence — and both died, so I called it done. Review mutated the bound's comparison instead:

mutation before now
> instead of >= survived killed
>= len-1 survived killed
>= len+1 survived killed
the s.Dict["Filter"] key lookup survived killed
drop the filter condition killed (25-page sweep only) killed locally too
bound an unreadable chain too survived (found in re-review) killed

The cause is that the only values the suite exercised were /S 99 against 67 bytes and /S 43 against 64 — both far from the edge, so any comparison in the neighbourhood passes. And no fixture carried /Filter null or /Filter [ ], so the stated reason for calling filterChain was the one thing nothing held.

⛔ A mutation only measures the question it asks. Mine asked "is this condition load-bearing?" and got a true answer; it could not ask "is this boundary the right one?", because I chose the mutations from what I had just been thinking about — which is exactly where I am least likely to be wrong. The lesson is not "mutate more": the mutation set has to come from the shape of the code (a comparison has three neighbours; a documented reason needs a case), not from the author's recollection of the risk.

What lands with it

Four tests, each pinning one survivor:

  • /S == len-1 accepted, /S == len refused — two cases that redden different subtests, so they hold opposite sides of the boundary rather than one carrying both.
  • /Filter null and /Filter [ ], each with /S past the end, refused.
  • /Filter /FlateDecode with the same /S, accepted — the asymmetry stated locally instead of only at 25 pages.

⛔⛔ Re-review found a sixth survivor, and its suggested test would have failed

Mutating the condition to (ferr == nil && len(names) == 0) || ferr != nil — applying the bound to a chain it could not read — was green. Same shape a third time: a documented reason with no case.

But the proposed fixture was /Filter 9 0 R, on the stated ground that an indirect filter makes filterChain error under a nil resolver. It does not. Resolve returns Null{}, nil for a Ref when the resolver is nil (object.go:162), so an indirect /Filter takes filterChain's case Null:, names stays empty, the stream is judged unfiltered, and the file is refused — the opposite of the acceptance that test asserts. Reading object.go rather than taking the mechanism on trust is the whole of the difference.

/Filter 5 is the spelling that actually reaches filterChain's default:, and it is accepted: an unreadable chain means this check cannot know whether the raw bytes are the data, and refusing on a bound it cannot evaluate would refuse for a reason it does not have.

⛔ That test asserts its own premise — it calls filterChain on /Filter 5 and fails loudly if that ever stops erroring, since the day it reads as a valid chain is the day the test stops exercising the branch it names while staying green.

⛔⛔⛔ And the third round found the real defect, which was mine, and which I had written a test to bless

Having learned that an indirect /Filter reads as "no filter", I pinned that as intended behaviour and argued for it: the reader this format exists for cannot resolve anything, so treating a reference as "no filter it can use" is what it would do.

Review supplied the input that kills the argument. If object 9 holds /FlateDecode, the stream really is filtered, /S really does index decoded data longer than the raw bytes, and the refusal is for a reason the check cannot evaluate — which is word for word the argument I had used, one test higher in the same file, to accept /Filter 5. Two cases that cannot be told apart, decided opposite ways, with a test holding each in place.

⛔⛔ A test that pins a wrong behaviour is worse than no test, because it converts an oversight into a specification. No mutation can find this class: every mutation of the old condition was correctly killed by TestAnIndirectFilterIsJudgedAsNoFilter, and that is exactly the problem — the tests agreed with the code and both were wrong. Only a reader asking "what does this refuse, and can the check know it?" gets there.

So the condition becomes a named predicate that answers a three-way question as such — "filtered", "not filtered", "cannot tell", with only the second licensing the bound:

func rawIsTheData(d Dict) bool {
	if _, external := d["F"]; external {
		return false
	}
	if _, indirect := d["Filter"].(Ref); indirect {
		return false
	}
	names, _, err := filterChain(d, nil)
	return err == nil && len(names) == 0
}

⛔⛔ That /F line came from looking for a second instance of the class rather than re-checking the first. 7.3.8.2, Table 5: when a stream dictionary carries /F its data lives in an external file, and the bytes between stream and endstream are not it. The parse slices them into Raw anyway, so len(Raw) measures a placeholder and bounding a decoded position against it refuses on a length belonging to different bytes — with no /Filter involved at all, which is what makes it a case and not a variant. The predicate is named rawIsTheData, and with /F present that name would be false.

An indirect /Filter is the third answer, so the bound is left alone and the file is accepted here. It is still non-conforming — F.3.6 requires every hint-dictionary entry beyond the stream Length to be direct — but that refusal is a named property tabled on #40, and #40 should make it by name rather than this bound making it as a side effect, with a message about the shared object table that would send the next reader to the wrong place.

⛔ /Filter [ 9 0 R ] needs no case of its own: filterChain resolves the element to Null, ToName fails, and the error is the same middle answer. It is in the predicate's table for the record, not as a separate claim.

⛔ And the predicate is tested as a predicate. Reaching it only through fixtures cannot distinguish "answered false" from "never asked", so TestRawIsTheDataAnswersOnlyWhenItKnows states all nine answers directly — absent, Null, empty array, a name, a chain, a reference, an array holding a reference, an impossible kind, and an external-file stream.

⛔ /S above the end is deliberately absent: it kills nothing the == len case does not already kill, and an extra case that cannot fail reports its own index rather than a measurement.

⛔ hintFixtureData is a constant and the cases count with it. /S 64 written beside a fixture that happens to hold 64 bytes stops testing the boundary the moment either number is edited alone, and the suite stays green.

The refusal at /S == len rests on the claim that the shared object hint table opens with a Table F.5 header and so occupies at least one byte: a table starting at the end of the data has nowhere to put it, which makes /S == len unrepresentable in a conforming file rather than merely unusual.

⛔ Ghostscript space-pads its integers

The first version of the new test anchored on /S 38 and matched nothing:

<</Length         67
/S         38>>

The helper's uniqueness check said so, rather than letting a silent zero-replacement pass as a test that could not fail.

⭐ Worth knowing beyond this test: that padding is the fixed-width placeholder technique a two-pass linearizing writer needs for /L, /H, /E and /T. The corpus already holds a worked example of the fixpoint trick #40 describes.

⛔⛔ A control written one change ago caught the fixture, not the code

TestEveryNonIntegerSharedTablePositionIsRefused's "an integer is accepted" subtest went red the moment this bound existed: its hand-built stream carried /Length 0, and /S 43 does point outside zero bytes.

The bound was right and the fixture was unrealistic. Without that control the seven kind cases above it would have gone on passing for the wrong reason — refused by the length check rather than the kind check, which is the "refused for a reason that is not the reason" shape this file has already paid for twice. The fixture now carries 64 bytes, through a helper that says why.

Mutations

Judged by cmd/failset on the failing names, six of six killed:

>  instead of >=      →  …MustLeaveRoomForItsOwnFirstByte/exactly_at_the_end
>= len-1              →  …MustLeaveRoomForItsOwnFirstByte/one_byte_inside
>= len+1              →  …MustLeaveRoomForItsOwnFirstByte/exactly_at_the_end
s.Dict["Filter"]      →  …AFilterThatFiltersNothingIsStillBounded, both cases
drop the condition    →  …ARealFilterLeavesTheSharedTablePositionUnbounded
                         and the two 25_pages sweep cases
bound an unreadable
  chain as well       →  …AFilterThatCannotBeReadLeavesTheBoundAlone, and no other
drop the Ref guard    →  …AnIndirectFilterIsNotSomethingThisCheckCanRead
                         and …AnswersOnlyWhenItKnows/an_indirect_/Filter
drop the /F guard     →  …AnswersOnlyWhenItKnows/an_external_file_stream
invert the predicate  →  16 tests, five of the predicate's own answers among them

Nine of nine killed.

⛔ A second /F row — {/F, /Filter null} — was written and then deleted. It reddens, but kills nothing the /F row alone does not, and /S above the end was excluded earlier on exactly that ground. Keeping one because it looked thorough while excluding the other because the rule said so would be the same incoherence this PR was corrected for, one level up. The first two redden different subtests, so the two boundary cases hold opposite sides rather than one carrying both. The boundary pair and the gate were re-run after the refactor, since the condition moved and a mutation proved against the old shape says nothing about the new one.

⛔ Four review rounds, each finding something the previous one could not. Round one: the boundary was unpinned. Round two: an unreadable chain was unpinned, and the fixture suggested for it would have failed. Round three: the behaviour I had just pinned was itself wrong. Round four, asked to look for a second instance of that class rather than re-check the first: /F. The first two are test gaps a mutation finds; the third is a wrong specification, which no mutation can find because the tests and the code agreed; the fourth came from naming the class and hunting it.

gofmt clean, go vet clean, 100.0% of statements, 0 uncovered blocks of 1820 — under REQUIRE_QPDF=1, replaying CI's own command rather than approximating it.

Refs #40.

🤖 Generated with Claude Code

@tannevaled
tannevaled force-pushed the feat/bound-S-in-an-unfiltered-hint-stream branch 4 times, most recently from 1ce0931 to c161723 Compare November 15, 2026 09:45
… stream

⭐ THE OPEN LIMIT WAS SMALLER THAN I SAID. /S indexes the DECODED data, so
bounding it above normally means decoding an attacker's bytes in the cheap
path — filter.go caps expansion at 1 GiB, which refuses a bomb but is three
orders of magnitude past anything a hint stream needs. That is still true and
still wants a size bound derived from /N and the object count (#40).

But a stream with NO FILTER needs no decoding at all: its raw bytes ARE its
data. That case is free, and it is the one that matters most while a writer is
being developed, because an uncompressed hint stream is what somebody
debugging one emits. Ghostscript already writes one unfiltered, and qpdf does
under --compress-streams=n.

⛔⛔ AND THE FILTER CONDITION IS LOAD-BEARING, WHICH THE CORPUS PROVED. Applying
the bound to filtered streams as well turns the generated sweep's TWENTY-FIVE
page cases red — `as qpdf writes it, 25_pages` and `with object streams,
25_pages`. For a document that size the hint table compresses well below /S,
so /S is past the RAW length while sitting comfortably inside the DECODED one.
The 1, 2 and 5 page cases cannot see it: their tables are too small to
compress under 43. So this is not defensive coding, it is required, and only a
corpus with a large file could say so.

⛔ ASKED THROUGH filterChain, NOT BY LOOKING FOR THE KEY. /Filter may be
present and Null, or an empty array, both of which mean no filtering;
`_, ok := s.Dict["Filter"]` would call those filtered and skip the one check
they are eligible for.

⛔⛔ AND THE FIRST VERSION OF THIS CHANGE WAS GREEN, MEASURED, AND UNPINNED AT
EVERY EDGE — WHICH REVIEW CAUGHT AND MY OWN MUTATIONS DID NOT. I mutated the
two things I had been thinking about (the filter condition, and the bound's
existence) and both died, so I called the work done. Review mutated the bound's
COMPARISON instead and found three survivors: `>`, `>= len-1` and `>= len+1`
all left the whole suite GREEN. The cause is that the only values the suite
exercised were /S 99 against 67 bytes and /S 43 against 64 — both far from the
edge, so any comparison in the neighbourhood passes. A fourth survivor was the
key lookup above: no fixture had `/Filter null` or `/Filter [ ]`, so the stated
REASON for calling filterChain was the one thing nothing held.

⛔ A MUTATION ONLY MEASURES THE QUESTION IT ASKS. Mine asked "is this condition
load-bearing?" and got a true answer. It could not ask "is this boundary the
right one?", because I chose the mutations from what I had just been thinking
about — which is exactly where I am least likely to be wrong. The lesson is not
"mutate more", it is that the mutation set must come from the SHAPE of the code
(a comparison has three neighbours; a documented reason needs a case) and not
from the author's recollection of the risk.

So four tests land with it, each pinning one survivor:

  /S == len-1 accepted    kills `>= len-1`
  /S == len   refused     kills `>` and `>= len+1`
  /Filter null, /S past   kills the key lookup
  /Filter [ ], /S past    kills it the other spelling
  /Filter /FlateDecode    kills dropping the condition, LOCALLY — it was only
                          the 25-page sweep that held this before, which is a
                          slow and indirect witness for one line

⛔ AND /S ABOVE THE END IS DELIBERATELY ABSENT, because it kills nothing the
`== len` case does not already kill. An extra case that cannot fail reports its
own index rather than a measurement.

⛔ hintFixtureData IS A CONSTANT AND THE CASES COUNT WITH IT. `/S 64` written
beside a fixture that happens to hold 64 bytes stops testing the boundary the
moment either number is edited alone, and the suite stays green — the shape
this package has already paid for under "judge and subject must count alike".

⛔ AND GHOSTSCRIPT SPACE-PADS ITS INTEGERS, which the first version of the new
test got wrong: `<</Length         67\n/S         38>>`. Anchoring on "/S 38"
matched nothing and the helper's uniqueness check said so rather than letting
a silent zero-replacement pass. Worth knowing beyond this test: that padding
is the FIXED-WIDTH PLACEHOLDER technique a two-pass linearizing writer needs
for /L, /H, /E and /T, so the corpus already holds a worked example of the
fixpoint trick #40 describes.

⛔⛔ AND A CONTROL WRITTEN ONE CHANGE AGO CAUGHT THE FIXTURE, NOT THE CODE.
TestEveryNonIntegerSharedTablePositionIsRefused's "an integer is accepted"
subtest went red the moment this bound existed: its hand-built stream carried
`/Length 0`, and /S 43 does point outside zero bytes. The bound was right and
the fixture was unrealistic. Without that control the seven kind cases above
it would have gone on passing for the WRONG REASON — refused by the length
check rather than the kind check, which is the "refused for a reason that is
not the reason" shape this file has already paid for twice. The fixture now
carries 64 bytes, through a helper that says why.

⛔⛔ AND THE SECOND REVIEW FOUND A SIXTH SURVIVOR, WHOSE SUGGESTED TEST WOULD
HAVE FAILED. Mutating the condition to `(ferr == nil && len(names) == 0) ||
ferr != nil` — applying the bound to a chain it could not read — was green:
the same shape again, a documented reason with no case. But review's proposed
fixture was `/Filter 9 0 R`, on the stated ground that an indirect filter makes
filterChain error under a nil resolver. It does not. Resolve returns
`Null{}, nil` for a Ref when the resolver is nil (object.go:162), so an
indirect /Filter takes filterChain's `case Null:`, names stays empty, the
stream is judged UNFILTERED, and the file is REFUSED — the opposite of the
acceptance the suggested test asserts. Reading object.go rather than taking the
mechanism on trust is the whole of the difference.

⛔ THE TEST FOR IT ASSERTS ITS OWN PREMISE. `/Filter 5` reaches filterChain's
`default:` and is the spelling that actually makes it error, so the test calls
filterChain itself and fails loudly if that ever stops erroring — the day it
reads as a valid chain is the day the test stops exercising the branch it names
while staying green.

⛔⛔⛔ AND THEN THE THIRD ROUND FOUND THE REAL DEFECT, WHICH WAS MINE AND WHICH I
HAD WRITTEN A TEST TO BLESS. Having learned that an indirect /Filter reads as
"no filter", I pinned THAT as intended behaviour and argued for it: the reader
this format exists for cannot resolve anything, so treating a reference as "no
filter it can use" is what it would do. Review supplied the input that kills
the argument. If object 9 holds /FlateDecode, the stream really IS filtered,
/S really does index decoded data longer than the raw bytes, and the refusal
is for a reason the check cannot evaluate — which is WORD FOR WORD the argument
I had just used, one test higher in the same file, to ACCEPT `/Filter 5`. Two
cases that cannot be told apart, decided opposite ways, with a test holding
each in place.

⛔⛔ A TEST THAT PINS A WRONG BEHAVIOUR IS WORSE THAN NO TEST, because it
converts an oversight into a specification. The mutation set cannot find this
class at all: every mutation of the old condition was correctly killed by
TestAnIndirectFilterIsJudgedAsNoFilter, which is exactly the problem — the
tests agreed with the code, and both were wrong. Only a reader asking "what
does this REFUSE, and can the check know it?" gets there.

So the predicate is now rawIsTheData, and it answers a THREE-way question as
such: "filtered", "not filtered" and "cannot tell" are distinct, and only the
second licenses the bound. An indirect /Filter is the third, so the bound is
left alone and the file is accepted HERE. It is still non-conforming — F.3.6
requires every hint-dictionary entry beyond the stream Length to be direct —
but that refusal is a NAMED property tabled on #40, and #40 should make it by
name rather than this bound making it as a side effect, with a message about
the shared object table that would send the next reader to the wrong place.

⛔ `/Filter [ 9 0 R ]` needs no case of its own: filterChain resolves the
element to Null, ToName fails, and the error is the same middle answer. It is
in the predicate's own table for the record, not as a separate claim.

⛔⛔ AND LOOKING FOR A SECOND INSTANCE OF THE CLASS FOUND ONE: /F. Rather than
re-checking the case just fixed, the fourth round asked whether anything else
refuses on something it cannot evaluate, and 7.3.8.2 Table 5 answers — when a
stream dictionary carries /F its data lives in an EXTERNAL FILE, and the bytes
between `stream` and `endstream` are not it. The parse slices them into Raw
anyway, so len(Raw) measures a placeholder and bounding a decoded position
against it refuses on a length belonging to different bytes. No /Filter need
be involved at all, which is what makes it a case and not a variant.

⛔ The predicate is named rawIsTheData, and with /F present that name would be
false. A predicate that is wrong in a case it has been told about is worth one
line more than an issue describing it. It DECLINES TO BOUND rather than
refusing: a hint stream with /F is wrong, and saying so is #40's job.

⛔ AND THE BARE PRESENCE TEST IS ONLY ADEQUATE FOR A STREAM DICTIONARY, which
is written into the comment because it is a constraint on REUSE and not a
property of the format. /F is a file specification in Table 5, but the same
name is an unrelated flag field elsewhere — an annotation's /F is an integer of
bit flags. Neither can reach a hint stream, so presence is enough here; a
future caller passing a dictionary of another kind would need the key read for
what it means there rather than merely counted.

⛔ AND THE PREDICATE IS TESTED AS A PREDICATE. Reaching it only through
fixtures cannot distinguish "answered false" from "never asked", so
TestRawIsTheDataAnswersOnlyWhenItKnows states all eight answers directly —
absent, Null, empty array, a name, a chain, a reference, an array holding a
reference, and an impossible kind. A refactor that keeps checkHintStream green
while changing what the predicate MEANS now goes red.

⛔ AND A SECOND /F ROW WAS WRITTEN AND THEN DELETED, which is the same rule
applied to myself. `{/F, /Filter null}` reddens, but it kills nothing the /F
row alone does not — and /S above the end was excluded a few paragraphs up on
exactly that ground. Keeping it because it looked thorough while excluding the
other because the rule said so is the incoherence this commit was corrected
for, one level up.

Mutations, judged by cmd/failset on the failing NAMES, nine of nine killed.
The boundary pair and the gate were re-run AFTER the refactor, because the
condition moved and a mutation proved against the old shape says nothing about
the new one:

  `>` instead of `>=`      -> …MustLeaveRoomForItsOwnFirstByte/exactly_at_the_end
  `>= len-1`               -> …MustLeaveRoomForItsOwnFirstByte/one_byte_inside
  `>= len+1`               -> …MustLeaveRoomForItsOwnFirstByte/exactly_at_the_end
  the `s.Dict["Filter"]`
    key lookup             -> …AFilterThatFiltersNothingIsStillBounded, both cases
  drop the condition       -> …ARealFilterLeavesTheSharedTablePositionUnbounded
                              AND the two 25_pages sweep cases
  bound an UNREADABLE
    chain as well          -> …AFilterThatCannotBeReadLeavesTheBoundAlone,
                              and no other
  drop rawIsTheData's
    Ref guard              -> …AnIndirectFilterIsNotSomethingThisCheckCanRead
                              and …AnswersOnlyWhenItKnows/an_indirect_/Filter,
                              and no other
  drop the /F guard        -> …AnswersOnlyWhenItKnows/an_external_file_stream,
                              and no other
  invert the predicate
    (`len(names) != 0`)    -> 16 tests, five of the predicate's own answers
                              among them

Note that the first two mutations redden DIFFERENT subtests, so the two cases
hold opposite sides of the boundary rather than one of them carrying both.

/S == len is refused on the claim that the shared object hint table opens with
a Table F.5 header and so occupies at least one byte: a table starting AT the
end of the data has nowhere to put it, which makes /S == len unrepresentable
in a conforming file rather than merely unusual.

⛔ FOUR REVIEW ROUNDS, EACH FINDING SOMETHING THE PREVIOUS ONE COULD NOT.
Round one: the boundary was unpinned. Round two: an unreadable chain was
unpinned, and the fixture suggested for it would have failed. Round three: the
behaviour I had just pinned was itself wrong. Round four, asked to look for a
SECOND instance of that class rather than re-check the first: /F. The first
two are test gaps a mutation finds; the third is a wrong SPECIFICATION, which
no mutation can find because the tests and the code agreed; the fourth came
from naming the class and hunting it.

gofmt clean, go vet clean, 100.0% of statements, 0 uncovered blocks of 1820
under REQUIRE_QPDF=1 (CI's own command, replayed rather than approximated).

Refs #40.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled force-pushed the feat/bound-S-in-an-unfiltered-hint-stream branch from c161723 to 76721e3 Compare November 15, 2026 09:49
@tannevaled
tannevaled merged commit 5f1b04d into main Nov 15, 2026
9 checks passed
@tannevaled
tannevaled deleted the feat/bound-S-in-an-unfiltered-hint-stream branch November 15, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant