feat(scan): plan changelog tasks from deletion vectors

[manuzhang]

What

Plan changelog tasks for format version 3 tables that use deletion vectors:

• Add DeletedRowsScanTask, a ChangelogScanTask with added_deletes() and
  existing_deletes(), for rows deleted by a deletion vector committed against a
  data file that already existed.
• Add DeletionVectorChangelogPlanner (changelog_dv_planner_internal.{h,cc}),
  which reads the delete manifests of the changelog snapshots, indexes the
  deletion vectors each snapshot added and removed, and plans the delete side: a
  vector committed together with its data file is attached to that file's
  AddedRowsScanTask; a vector removed together with its data file becomes the
  existing delete of the DeletedDataFileScanTask; a vector for an existing data
  file locates the file in the committing snapshot's data manifests and produces a
  DeletedRowsScanTask carrying the added vector and the vector it replaced.
• IncrementalChangelogScan uses the planner only when the table format version
  is 3 or higher. Format version 1 and 2 tables keep rejecting delete manifests
  with NotSupported.
• The planner checks every delete manifest of every changelog snapshot, not only
  the manifests written in the range, and rejects position delete files and
  equality delete files with NotSupported. A delete file removed before the
  range no longer applies to any row, so it is the one case that is ignored.

This is planning only, matching the existing changelog tasks: a consumer of
DeletedRowsScanTask reads the data file and emits the positions set in the added
vector that are not set in the existing vector.

Why

IncrementalChangelogScan failed on any snapshot that carried a delete manifest,
so a table with row-level deletes could not produce a changelog at all. Format
version 3 makes this tractable: each data file has at most one live deletion
vector and a new vector replaces the previous one, so the rows a snapshot deleted
are exactly the difference between the vector it added and the vector it removed,
without reading any earlier delete files.

Position and equality delete files accumulate instead, so their rows cannot be
attributed to a snapshot without reading every earlier delete file. A format
version 3 table can still carry them, for example after an upgrade from version
2, in manifests that no snapshot in the range rewrote. Checking every delete
manifest makes that case an explicit error instead of a changelog that silently
omits the rows those files delete.

Behavior change

• Changelog scans of format version 3 tables with deletion vectors now succeed and
  may return DeletedRowsScanTask, which callers that switch on task type must
  handle.
• The error for delete files on format version 1 and 2 tables is now
  "Delete files are only supported in changelog scans of format version 3 tables".
• For every changelog snapshot that adds deletion vectors to existing data files,
  planning reads that snapshot's data manifests to find the referenced files.
• Planning reads every delete manifest of every changelog snapshot, each manifest
  once, and fails with NotSupported when one holds a position delete file or an
  equality delete file that is live or that a changelog snapshot removed. Delete
  files removed before the range are ignored.

Testing

New cases in incremental_changelog_scan_test.cc cover a deletion vector added to
an existing file, a vector replacing an earlier one (single snapshot and across
two snapshots), a vector committed with its data file, a vector removed with its
data file, partition pruning of vectors by the data filter, vectors outside the
scan range, the NotSupported errors for format version 2 delete files, position
delete files and equality delete files written in the range or carried in a
manifest untouched during the range, and a position delete file removed before
the range being ignored. scan_test passes locally.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.