Stop keeping the full payment history in memory

[joostjager]

#269 tracks paginating Node::list_payments, and #959 adds that API. However, PaymentStore is still initialized with read_all_objects, and DataStore retains every PaymentDetails in a HashMap for the lifetime of the node. #960 makes the startup reads paginated, which avoids one unbounded list response, but it still reads and decodes the full payment namespace before the node is usable.

This means startup work and steady-state memory usage continue to grow with the complete payment history, even though callers retrieve that history page by page. This will become increasingly relevant for long-lived server deployments.

Related work:

• Add a paginated version of list_payments #269 tracks the caller-facing pagination requirement.
• Add paginated payment listing API #959 adds the paginated Node::list_payments API while continuing to serve values from memory.
• Read stored objects page-by-page at startup #960 reads persisted objects page by page at startup, but still materializes the full namespace in memory.
• rust-lightning #4779 tracks paginated key-and-value reads, but does not address retaining the complete payment history in memory.
• ldk-server #240 contains earlier related payment-listing work and was closed without merging.

[Jolah1]

I’m interested in working on this. A fully lazy PaymentStore seems possible without a separate active index: skip loading payment history during build, read individual records from KV for get/update/duplicate checks, and have paginated listing read only the current page’s values.
PendingPaymentStore can remain memory-backed for operational on-chain state. This would require making payment reads fallible/async internally and deciding how Node::payment() should expose storage errors; it would also build on or adjust the still-open #959/#960 work.
Is that the intended direction, and should I coordinate the implementation with those PRs?

[tnull]

Is that the intended direction, and should I coordinate the implementation with those PRs?

Not quite sure, but for now we're intentionally holding off, so we're currently not moving forward with this until we landed all related open PRs and shipped v0.8.

[Jolah1]

Is that the intended direction, and should I coordinate the implementation with those PRs?

Not quite sure, but for now we're intentionally holding off, so we're currently not moving forward with this until we landed all related open PRs and shipped v0.8.

Duly noted, Thank you.