linearize: /E is the end of the first page, and the file already says where that is

README.md

@@ -150,7 +150,7 @@ judgement a writer needs on bytes it has just produced. It establishes that
 every parameter **points at what it claims to** — `/N` is the real page count,
 `/O` is the real first page *and* is a `/Page`, `/H` begins at a real stream
 object, `/T` reaches the main cross-reference (either shape: Table F.1 gives
-it two meanings and both are real), `/E` is inside the file.
+it two meanings and both are real), `/E` is where the first page really ends.
 
 What gates it are files `Linearized` **accepts** and
 `CheckLinearized` refuses: the committed qpdf fixture with one integer
@@ -160,13 +160,56 @@ different document. The replacement's width is asserted by the helper, because
 an edit one byte longer would be refused for the length instead and the deep
 check would never run.
 
-⛔⛔ **`/E` is NOT gated, and a writer must not read a pass here as evidence
-about it.** Table F.1 calls `/E` the offset of the end of the first page, and
-qpdf *computes* it — on the committed fixture it reports `computed = 901..902`
-and refuses a wrong value. This only bounds `/E` between 1 and the file
-length, so a writer that miscomputes it passes here and fails qpdf. Computing
-it needs the first-page object set, which is the same partition the hint
-tables describe, so it waits for that work rather than being approximated now.
+⛔ **`/E` is checked against the objects the file says are the first page's,
+not merely against the file's length.** Table F.1 calls `/E` the offset of the
+end of the first page. This used to bound it only between 1 and the file
+length, on the reasoning that the first-page partition "is the hint tables'
+job" — which was true of the hint tables and false of the **first-page
+cross-reference section**, which Annex F puts at the front and the file's own
+`startxref` points at. Its entries *are* the first page's objects; that is the
+whole trick of the format. So `/E` must equal the end of the last of them, and
+a writer that is one byte out is now told so instead of passing.
+
+⛔⛔ **The hint stream is excluded, and a second producer is why that is not a
+guess.** F.3.1 permits the primary hint stream on either side of the first
+page, so the highest-offset entry in that section is not always a first-page
+object:
+
+| producer | `/H` offset | `/E` | hint stream sits |
+|---|---|---|---|
+| qpdf | 550 | 902 | **before** the first page |
+| Ghostscript `-dFastWebView` | 829 | 829 | **after** it, at `/E` exactly |
+
+On Ghostscript's file the naive maximum *is* the hint stream, ending at 967
+against an `/E` of 829 — so a rule that took it would refuse every file that
+tool writes. Both qpdf fixtures place it before and **cannot tell the two
+rules apart**, which is why `testdata/linearized-gs.pdf` is committed beside
+them: removing the exclusion turns exactly that test red and leaves the qpdf
+ones green. A second producer measures what the first cannot.
+
+⛔⛔ **What this proves is that `/E` agrees with the first-page section — not
+that the section is right.** A writer that leaves objects page one genuinely
+needs out of that section produces a file that is internally consistent and
+passes here. Ghostscript does exactly this on a document with shared fonts,
+outlines and links: it writes `/H` = `/E` = 1941, this checker accepts it, and
+`qpdf --check` computes `5500..5501` and reports the first page as having 9
+objects where the hint table claims 4. Deciding which objects page one *needs*
+means walking its resources, which is the partition the hint tables describe —
+so that judgement is still outstanding, and a pass here is not evidence about
+it.
+
+⚠ **Stricter than qpdf, deliberately.** qpdf computes `/E` as a *range*
+(`901..902` on the committed fixture), accepting the end with or without the
+line ending after `endobj`. This requires the end including it, which is what
+both measured producers actually write. A file carrying the lower value would
+pass `qpdf --check` and be refused here; none has been observed, and the
+alternative is a window that cannot tell a correct `/E` from one a byte short.
+
+⚠ The **overflow** hint stream is excluded by the same reasoning and no file
+can reach it — it needs a four-entry `/H` whose second pair is both a real
+stream and the highest-offset object, and neither producer writes one. So the
+question is asked of a function rather than inline, where four plain cases can
+put it and a mutation can kill it.
 
 ⛔⛔ **A value this build cannot address is refused, not truncated**, and the
 same bytes therefore get the same answer on a 32-bit build as on a 64-bit one.

linearized.go

@@ -255,17 +255,142 @@ func CheckLinearized(b []byte) (*Linearization, error) {
 	if err := checkMainXref(b, lin); err != nil {
 		return nil, err
 	}
-	// Table F.1: /E is "the offset of the end of the first page". Nothing here
-	// knows which objects the first page owns — that is the hint tables' job —
-	// so what is checked is the one thing that is checkable without them: an
-	// offset past the end of the file sends a reader nowhere.
-	if lin.FirstPageEnd <= 0 || lin.FirstPageEnd > len(b) {
-		return nil, fmt.Errorf("%w: /E is %d and the file is %d bytes",
-			ErrLinearizationStale, lin.FirstPageEnd, len(b))
+	if err := checkFirstPageEnd(b, lin); err != nil {
+		return nil, err
 	}
 	return lin, nil
 }
 
+// isHintObject says whether an object at this offset is one of the hint
+// streams /H names, and so is not a first-page object whose end could be /E.
+//
+// ⛔⛔ IT IS A FUNCTION SO THAT THE SECOND HALF CAN BE TESTED AT ALL. The
+// primary hint stream is reachable from a committed file — Ghostscript puts
+// it at /E — but the OVERFLOW stream is not: it needs a four-entry /H whose
+// second pair is both a real stream and the highest-offset object in the
+// first-page section, and neither qpdf nor Ghostscript writes one. Left
+// inline, that condition would be a branch no fixture could reach and no
+// mutation could kill, which is the vacuity this file has already spent
+// three review rounds removing. Asked directly it is four plain cases.
+func isHintObject(lin *Linearization, off int64) bool {
+	if off == int64(lin.HintOffset) {
+		return true
+	}
+	return lin.HasOverflowHint && off == int64(lin.OverflowHintOffset)
+}
+
+// checkFirstPageEnd checks /E, Table F.1's "offset of the end of the first
+// page", against the objects the file itself says belong to the first page.
+//
+// ⛔ THE FIRST-PAGE CROSS-REFERENCE SECTION IS THE ANSWER, and it costs
+// nothing to read. Annex F puts it at the front (part 3) and the file's own
+// `startxref` points at it, which is the whole trick of the format: a reader
+// fetches the front of the file and has page one. So its entries ARE the
+// first page's objects, and no hint table has to be decoded to learn that.
+// This checker used to say the partition "is the hint tables' job" and bound
+// /E only to the file's length; that was true of the hint tables and not of
+// this section.
+//
+// ⛔⛔ THE HINT STREAM IS EXCLUDED, AND A SECOND PRODUCER IS WHY THAT IS NOT
+// A GUESS. F.3.1 permits the primary hint stream on either side of the first
+// page, so the highest-offset entry here is not always a first-page object:
+//
+//	qpdf          /H at 550, /E 902 — the hint stream comes BEFORE
+//	Ghostscript   /H at 829, /E 829 — it comes AFTER, AT /E exactly
+//
+// On Ghostscript's file the naive maximum is the hint stream itself, ending
+// at 967 against an /E of 829, so a rule that took it would refuse every
+// file that tool writes. Both committed qpdf fixtures place the hint stream
+// before the first page and CANNOT tell the two rules apart — which is why
+// testdata/linearized-gs.pdf is committed beside them.
+//
+// ⛔⛔ AND WHAT THIS PROVES IS THAT /E AGREES WITH THAT SECTION, NOT THAT THE
+// SECTION IS RIGHT. A writer that leaves out objects page one genuinely needs
+// produces a file that is internally consistent and passes here. Review
+// measured Ghostscript doing it on a document with shared fonts, outlines and
+// links: /H = /E = 1941, accepted here, while qpdf computes 5500..5501 and
+// reports page one as 9 objects where the hint table claims 4. Deciding what
+// page one NEEDS means walking its resources — the partition the hint tables
+// describe — and that judgement is still outstanding (#40).
+func checkFirstPageEnd(b []byte, lin *Linearization) error {
+	// ⛔ A SEPARATE Document, not the one CheckLinearized opened. Open walks
+	// /Prev and merges every section into one table, and setEntry keeps the
+	// first definition it saw, so by then the front section's entries are no
+	// longer distinguishable from the rest of the file's. Reading that one
+	// section alone is the only way to ask which objects it listed.
+	front := &Document{
+		buf:     b,
+		xref:    map[int]xrefEntry{},
+		cache:   map[int]Object{},
+		loading: map[int]bool{},
+		objStms: map[int]map[int]Object{},
+	}
+	off, err := front.findStartxref()
+	if err != nil {
+		return fmt.Errorf("%w: the first-page cross-reference section cannot be found: %v",
+			ErrLinearizationStale, err)
+	}
+	// ⛔⛔ THE SECTION startxref NAMES IS NOT ALWAYS THE FIRST-PAGE ONE, and
+	// an earlier draft of this claimed /L had already made it so. It has not:
+	// an incremental update appends its own section and points startxref at
+	// THAT, and a file whose /L was rewritten to match its new length passes
+	// every earlier check. Following it then lands on the update's entries and
+	// compares /E with an object that has nothing to do with the first page —
+	// a refusal with a reason that is not the reason, which is worse than a
+	// miss because a reader believes it.
+	//
+	// Annex F puts the first-page section at part 3 and the main one at part
+	// 10, so the front section precedes /T. checkMainXref has already
+	// established /T against the file by the time this runs, and this says
+	// nothing about /E, so it still holds when /E is the thing under test.
+	if off >= int64(lin.MainXrefOffset) {
+		return fmt.Errorf("%w: the cross-reference section at byte %d is not the first-page one — "+
+			"it is at or past the main table /T names (%d), so the file has been written to since it was linearized",
+			ErrLinearizationStale, off, lin.MainXrefOffset)
+	}
+	if _, err := front.readXrefSection(off); err != nil {
+		return fmt.Errorf("%w: the first-page cross-reference section at byte %d does not read: %v",
+			ErrLinearizationStale, off, err)
+	}
+
+	// ⛔ THE COMPARISON STAYS IN int64 UNTIL THE VALUE IS KNOWN TO FIT. These
+	// offsets come from the file, and int(e.offset) is the same silent
+	// truncation `addressable` exists to refuse everywhere else.
+	last := int64(-1)
+	for _, e := range front.xref {
+		// An object inside an object stream has no offset of its own; its
+		// container has one and is listed here too, so skipping it loses
+		// nothing.
+		if e.kind != 'n' || isHintObject(lin, e.offset) {
+			continue
+		}
+		if e.offset > last {
+			last = e.offset
+		}
+	}
+	if last < 0 {
+		return fmt.Errorf("%w: the first-page cross-reference section lists no object in the file",
+			ErrLinearizationStale)
+	}
+	if last >= int64(len(b)) {
+		return fmt.Errorf("%w: the first-page cross-reference section puts an object at byte %d and the file is %d bytes",
+			ErrLinearizationStale, last, len(b))
+	}
+	at := int(last)
+	_, _, n, err := ParseIndirectObject(b[at:], nil)
+	if err != nil {
+		return fmt.Errorf("%w: the last first-page object, at byte %d, does not parse: %v",
+			ErrLinearizationStale, at, err)
+	}
+	// The same end-of-object rule the /H length uses: Table F.1 counts the
+	// line ending after `endobj` and the parse stops at it.
+	if end := at + n + eolAfter(b, at+n); lin.FirstPageEnd != end {
+		return fmt.Errorf("%w: /E says %d and the first page ends at %d",
+			ErrLinearizationStale, lin.FirstPageEnd, end)
+	}
+	return nil
+}
+
 // checkFirstPage checks /N and /O against the document's own page tree.
 //
 // ⛔⛔ THERE IS NO "AND IS IT REALLY A PAGE?" CHECK, AND ITS ABSENCE IS THE