feat(rocksdb): add optional ToplingDB integration by kenssa4eedfd · Pull Request #3134 · apache/hugegraph

kenssa4eedfd · 2026-08-02T11:31:04Z

Purpose of the PR

This PR introduces ToplingDB as an optional RocksDB-compatible storage engine
for HugeGraph.

ToplingDB is explicitly enabled through configuration. Standard RocksDB remains
the default, so existing deployments and data paths are unaffected unless
rocksdb.provider=topling is configured.

Main Changes

Add explicit storage engine selection through rocksdb.provider.
Integrate ToplingDB with HugeGraph Server, PD, and Store.
Use ToplingDB Easy Migrate YAML configuration while continuing to use the
standard RocksDB Java API.
Add component-specific configurations for Server, PD, and Store, including
separate HTTP ports and JRaft column-family settings.
Add startup scripts that:
- Select and preload the ToplingDB JNI library.
- Extract required native libraries and Web UI resources.
- Configure LD_PRELOAD, LD_LIBRARY_PATH, and
  TOPLINGDB_EASY_MIGRATE_CONF.
- Validate the operating system and architecture.
- Verify downloaded dependencies using SHA-256.
- Handle jemalloc and Ubuntu 24.04 libaio compatibility.
Align the RocksDB JNI version across Server, PD, and Store.
Update CI workflows to install and test the ToplingDB runtime dependencies.
Update third-party dependency metadata, LICENSE information, operational
documentation, troubleshooting guides, and security notes.

Compatibility

Standard RocksDB remains the default provider.
Existing RocksDB data and APIs remain compatible.
ToplingDB is enabled only when rocksdb.provider=topling is configured.
No public HugeGraph API changes are introduced.
The bundled ToplingDB runtime currently targets Linux x86_64.

Verifying these changes

Trivial rework / code cleanup without any test coverage. (No Need)
Already covered by existing tests, such as (please modify tests here).

Need tests and can be verified as follows:

Build all modules:

mvn clean compile -Dmaven.javadoc.skip=true

Run HugeGraph Server tests with the RocksDB backend:

mvn test -pl hugegraph-server/hugegraph-test -am \                                                                                                                                                 
  -P core-test,rocksdb

Run PD tests:

mvn install -pl hugegraph-struct -am -DskipTests                                                                                                                                                   
mvn test -pl hugegraph-pd/hg-pd-test -am

Run Store tests:

mvn test -pl hugegraph-store/hg-store-test -am

Verify Server, PD, and Store startup in both standard RocksDB mode and
ToplingDB mode.

Does this PR potentially affect the following parts?

Documentation Status

Doc - TODO
Doc - Done
Doc - No Need

…b packages

…ckage

…th the HG default configuration

[pull] master from apache:master

…TP Server

…positioning

[pull] master from apache:master

Co-authored-by: Hervé Boutemy <herve.boutemy@free.fr>

…ple calls

* refactor: Delete the package existing in java8 * chore(format): remove custom line breaks

- package component-local helpers for Server, PD, and Store - separate standard and Topling distribution contents - fail fast on invalid or incomplete Topling runtime selection - validate clean artifacts and real provider loading in CI

- add a discoverable ToplingDB documentation entry point - document standalone and HStore configuration paths - describe build, test, CI, and native runtime workflows - define volume isolation, monitoring, and rollback safety

- close graph transactions after OLTP algorithm execution - keep failures visible while releasing traversal resources - cover success and failure paths in the unit suite

- build standard and amd64 Topling targets with provenance labels - isolate provider data paths with fail-atomic marker checks - add standalone and distributed Compose deployment paths - validate runtime packaging, lifecycle, and provider selection in CI

- forward TERM to the Server process - stop waiting after child becomes an unreaped zombie - preserve nonzero status for unexpected Server exits - cover process-group SIGTERM in the entrypoint regression

- destroy thread-local transactions after commit and rollback - clean Gremlin worker transactions before resetting context - keep multi-graph cleanup isolated and concurrency-safe - make RocksDB opened checks free of session side effects - cover cleanup failures and session state regressions

- preserve column families while truncating Topling data - balance temporary RocksDB session ownership - handle SIGTERM safely throughout container startup - keep schema transactions usable after intermediate commits

- preserve existing Ubuntu 24 required runtime checks - exercise Server, PD, and Store Topling runtimes on native Ubuntu 26 x64 - keep preview-runner or dependency failures non-blocking

- resolve Docker README and entrypoint test conflicts - retain Topling provider lifecycle and runtime guards - align standalone, HStore, dev, and HA Compose overlays - add standalone Topling overlay and render contracts - keep master Docker topology and authentication changes

- bind search filters to the original logical query - cover committed HStore IN and search regressions - keep sub-query input ordering behavior unchanged

- inject provider images, roots, and isolated volumes through env - reuse standalone, HStore, dev, and HA topology files - remove duplicated Topling-only Compose overlays - update render contracts, CI, and user/developer commands

- match the dev override image selected by the generic HStore topology - keep the Topling PD and Store parameter checks unchanged - preserve standalone and HA production image contracts

- replace obsolete overlay wording with environment injection - distinguish standalone and HStore provider ownership - keep the standard HStore Server guidance explicit

- point the Store data-path example at application-pd.yml - keep the PD application.yml example unchanged - align the quickstart prose with the component table

- document generic Compose rocksdb defaults - require provider, root, marker, and volume together - prevent image-only switches from masking data safety

- assert provider mismatch leaves the existing marker intact - assert unmarked data and files survive fail-fast validation - cover rollback safety at the entrypoint boundary

Document native Linux and macOS container targets. Explain linux/amd64 platform selection for Docker engines. Exclude native macOS execution from Topling acceptance scope. Keep standard and Topling topology guidance unchanged.

• align image defaults with standalone and HStore files • document the fixed Hubble image and pull policy • keep generic Topling parameter guidance accurate

- state that sample monitors are disabled by default\n- document explicit loopback-only enablement\n- align operations guidance with bundled YAML

- use numeric JDK version comparison in PD and Store\n- quote the PD process-count command substitution\n- avoid checksum wildcard matching in the helper

- exclude lib/topling from standard Server classpath scans\n- keep explicit preload as the only Topling classpath path\n- add fake-JVM regression for provider fallback isolation\n- document the standard-provider fallback contract

- exclude optional Topling jars from init-store scans - preserve explicit Topling preload classpath selection - extend fake-JVM regression coverage to init-store - cover all launcher jar-scan categories

- resolve library paths before excluding the Topling subtree - share canonical filtering across Server and init-store - cover direct-file and directory symlink aliases - document standard-provider classpath isolation

- retain jar-only init-store classpath scanning - stop following library directory symlink aliases - emit canonical regular-file classpath entries - cover non-jar and symlink fixtures in selector tests

- preserve classpath paths containing spaces - quote extension and plugin library scans - exercise launchers from a spaced fixture path

imbajin and others added 30 commits June 12, 2025 14:08

Create auto-pr-review.yml

3c93fb4

feat(backend): Add RocksDB Plus (ToplingDB) support

b385b9a

fix(server): Use reflection to increase compatibility with old rocksd…

c26d84d

…b packages

fix(server): fix bugs when using reflection openDB

9ae925b

feat: add some scripts to unzip and preload toplingdb from the jar pa…

448551a

…ckage

chore: fix CI bugs

8ac2fdc

chore: change env.USE_STAGE to true

90cb28f

chore: try to modify the permission to solve the 401 problem

31097ed

Merge branch 'apache:master' into master

c2dab77

fix(server): solve the bug of importing topling package

2fef639

fix(CI): eliminate pollution of environment variables

cf4dbe7

install deps in hstore-ci

ccaa2b9

add compatibility for libaio.so.1 for ubuntu-24.04

cfee29b

chore(server): modify the configuration file to keep it consistent wi…

461fee3

…th the HG default configuration

Merge pull request #19 from apache/master

26b91e4

[pull] master from apache:master

feat(server): add the configuration open_http to start the Topling HT…

7cfa2ee

…TP Server

chore(server): fix formatting issues in scripts

ac03196

chore(server): adopt review suggestions from AI robots

6c82d7a

chore(server): add third-party package verification

0151849

chore(server): add unified error capture in the script to facilitate …

d33aca3

…positioning

Merge pull request #38 from apache/master

c7a4cfd

[pull] master from apache:master

chore: improve maven Reproducible Builds (apache#2874) (#42)

d107e74

Co-authored-by: Hervé Boutemy <herve.boutemy@free.fr>

Merge branch 'apache:master' into master

4570649

chore(server): extract redundant code from scripts into common functions

0c5eb2f

chore(server): enhance script robustness

ed2d49d

chore(server): corrected resource copy logic

577430b

chore(server): avoid repeated expansion of LD_PRELOAD caused by multi…

4a591e7

…ple calls

chore(docs): add rocksdb-plus documentation

f5a0d40

docs: enhance docker instruction with auth opened graph (apache#2881)

2fd566e

refactor: remove the package existing in java8 (apache#2792)

0e564b1

* refactor: Delete the package existing in java8 * chore(format): remove custom line breaks

dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Aug 30, 2026

imbajin added 2 commits August 31, 2026 03:42

feat(toplingdb): isolate component runtimes

09677bb

- package component-local helpers for Server, PD, and Store - separate standard and Topling distribution contents - fail fast on invalid or incomplete Topling runtime selection - validate clean artifacts and real provider loading in CI

docs(toplingdb): add usage and developer guides

36959aa

- add a discoverable ToplingDB documentation entry point - document standalone and HStore configuration paths - describe build, test, CI, and native runtime workflows - define volume isolation, monitoring, and rollback safety

dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Aug 30, 2026

imbajin added 8 commits August 31, 2026 07:11

fix(core): close OLTP traversal transactions

1a46632

- close graph transactions after OLTP algorithm execution - keep failures visible while releasing traversal resources - cover success and failure paths in the unit suite

fix(server): handle container shutdown zombies

c00e320

- forward TERM to the Server process - stop waiting after child becomes an unreaped zombie - preserve nonzero status for unexpected Server exits - cover process-group SIGTERM in the entrypoint regression

fix: stabilize Topling lifecycle

7afe259

- preserve column families while truncating Topling data - balance temporary RocksDB session ownership - handle SIGTERM safely throughout container startup - keep schema transactions usable after intermediate commits

chore(ci): add Ubuntu 26 Topling coverage

1950e9b

- preserve existing Ubuntu 24 required runtime checks - exercise Server, PD, and Store Topling runtimes on native Ubuntu 26 x64 - keep preview-runner or dependency failures non-blocking

fix(core): preserve filters across flattened queries

d42ee3b

- bind search filters to the original logical query - cover committed HStore IN and search regressions - keep sub-query input ordering behavior unchanged

imbajin force-pushed the toplingdb branch from 35b931d to d42ee3b Compare August 31, 2026 09:23

imbajin added 15 commits August 31, 2026 18:18

refactor(docker): parameterize Topling compose

f09a457

- inject provider images, roots, and isolated volumes through env - reuse standalone, HStore, dev, and HA topology files - remove duplicated Topling-only Compose overlays - update render contracts, CI, and user/developer commands

fix(ci): align hstore compose assertion

9229717

- match the dev override image selected by the generic HStore topology - keep the Topling PD and Store parameter checks unchanged - preserve standalone and HA production image contracts

docs(docker): describe generic Topling parameters

47c03e6

- replace obsolete overlay wording with environment injection - distinguish standalone and HStore provider ownership - keep the standard HStore Server guidance explicit

docs(topling): correct Store config path

553fe4c

- point the Store data-path example at application-pd.yml - keep the PD application.yml example unchanged - align the quickstart prose with the component table

docs(docker): require explicit provider selection

a665e7b

- document generic Compose rocksdb defaults - require provider, root, marker, and volume together - prevent image-only switches from masking data safety

fix(toplingdb): preserve data on provider failure

e9f5213

- assert provider mismatch leaves the existing marker intact - assert unmarked data and files survive fail-fast validation - cover rollback safety at the entrypoint boundary

docs(toplingdb): clarify container support matrix

76cb94f

Document native Linux and macOS container targets. Explain linux/amd64 platform selection for Docker engines. Exclude native macOS execution from Topling acceptance scope. Keep standard and Topling topology guidance unchanged.

docs(docker): correct Compose defaults

e075dcf

• align image defaults with standalone and HStore files • document the fixed Hubble image and pull policy • keep generic Topling parameter guidance accurate

docs(toplingdb): clarify monitor defaults

28061bf

- state that sample monitors are disabled by default\n- document explicit loopback-only enablement\n- align operations guidance with bundled YAML

fix(toplingdb): validate startup versions numerically

e047cbc

- use numeric JDK version comparison in PD and Store\n- quote the PD process-count command substitution\n- avoid checksum wildcard matching in the helper

fix(server): isolate optional Topling classpath

db9c556

- exclude lib/topling from standard Server classpath scans\n- keep explicit preload as the only Topling classpath path\n- add fake-JVM regression for provider fallback isolation\n- document the standard-provider fallback contract

fix(server): isolate Topling init classpath

59d50dc

- exclude optional Topling jars from init-store scans - preserve explicit Topling preload classpath selection - extend fake-JVM regression coverage to init-store - cover all launcher jar-scan categories

fix(server): guard Topling classpath aliases

1d1575a

- resolve library paths before excluding the Topling subtree - share canonical filtering across Server and init-store - cover direct-file and directory symlink aliases - document standard-provider classpath isolation

fix(server): harden library path filtering

9284b36

- retain jar-only init-store classpath scanning - stop following library directory symlink aliases - emit canonical regular-file classpath entries - cover non-jar and symlink fixtures in selector tests

fix(server): quote runtime classpaths

18635f6

- preserve classpath paths containing spaces - quote extension and plugin library scans - exercise launchers from a spaced fixture path

Navigation Menu

Uh oh!

feat(rocksdb): add optional ToplingDB integration - #3134

feat(rocksdb): add optional ToplingDB integration#3134
kenssa4eedfd wants to merge 97 commits into
apache:masterfrom
hugegraph:toplingdb

kenssa4eedfd commented Aug 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

kenssa4eedfd commented Aug 2, 2026

Purpose of the PR

Main Changes

Compatibility

Verifying these changes

Does this PR potentially affect the following parts?

Documentation Status

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants