Skip to content

[Client] Add the primary-key table bulk-load client SDK #4225

Description

@platinumhamburg

Search before asking

  • I searched in the issues and found nothing similar.

Description

Provide a public client SDK over the server-side bulk-load protocol.

The SDK should hide RPC messages, manifest assembly, snapshot file handling,
and Commit retry details while supporting both single-process builders and
distributed execution engines.

Scope

  • Expose BulkLoad through the Fluss connection.
  • Add a client for Begin, GetInProgress, Commit, and Abort.
  • Return an immutable and serializable build context from Begin.
  • Add openBuildSession as the entry point for local build resources.
  • Create bucket writers through the build session.
  • Route and validate rows against the frozen target metadata.
  • Deduplicate primary-key rows locally, with the last row for a key winning.
  • Build ordinary Fluss KV snapshot files for each bucket.
  • Support an explicit log end offset when required by the caller.
  • Assemble and publish the complete manifest internally.
  • Retry Commit within a caller-provided await budget when the outcome is
    retriable or unknown.
  • Close all writers still owned by a build session when the session is closed.
  • Keep credentials and process-local resources out of the serializable build
    context.

Willingness to contribute

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions