Skip to content

[Flink] Support Flink batch insert into Fluss primary-key tables #4227

Description

@platinumhamburg

Search before asking

  • I searched in the issues and found nothing similar.

Description

Integrate the BulkLoad client SDK with the Flink connector so a bounded
INSERT INTO job can efficiently populate an empty Fluss primary-key table or
partition.

The Flink integration should coordinate one server transaction, build buckets
in parallel, and publish the target only after every bucket has completed.

Scope

  • Add an explicit connector option for selecting the BulkLoad path.
  • Enable BulkLoad only for bounded Flink jobs and supported primary-key tables.
  • Begin one BulkLoad transaction for the physical target.
  • Distribute the immutable build context to parallel build tasks.
  • Assign every input row to its Fluss bucket.
  • Build bucket snapshots in parallel through openBuildSession.
  • Collect one completed result for every target bucket.
  • Commit the complete manifest through one coordinated commit path.
  • Abort or report a clear failure when the job cannot complete before the
    durable Commit decision.
  • Keep temporary credentials out of Flink records, committables, checkpoints,
    and savepoints.
  • Document configuration, supported cases, failure behavior, and operational
    limitations.

Willingness to contribute

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions