FINERACT-2672: Run standing instructions as a partitioned batch job

[oluexpert99]

The Execute Standing Instruction job ran every due instruction inside a single transaction. The step is
wired with the JPA transaction manager and AccountTransfersWritePlatformService.transferFunds is
@Transactional(REQUIRED), so every transfer joined that one transaction. When an instruction failed —
InsufficientAccountBalanceException from the withdrawal leg being the ordinary case — Spring marked the
shared transaction rollback-only, and at step commit the whole run was reverted: every transfer that had
succeeded was undone, and no history row survived to say so. The tasklet then threw JobExecutionException
and failed the job. The defect concealed itself, because the record of the failure was rolled back along
with the run that produced it.

Following the [DISCUSS] thread on dev@ and @adamsaghy's vote there for "leveraging Spring Batch for
distributed processing", this rebuilds the job on the machinery the platform already uses for LOAN_COB
rather than hand-rolling chunking, retry and parallelism.

What the job looks like now

A manager step partitions the due set; workers take a partition each and run a chunk-oriented step.

Concern	How
Bounded memory	StandingInstructionItemReader pages through its partition; one page in memory whatever the size of the due set
Parallelism without contention	StandingInstructionPartitioner cuts partitions over distinct source accounts, not instructions
Chunk-then-per-item	Spring Batch's own fault-tolerant replay, not a hand-written fallback
Retry	Spring Batch retry, scoped to transient failures
Sizing	fineract.partitioned-job.partitioned-job-properties[1], all env-overridable

Partitioning by source account. Every instruction debiting a given account lands in one partition and
runs sequentially there, so two partitions never contend on the same m_savings_account row. The partition
key collapses the savings and loan id spaces with COALESCE; the spaces overlap, so two unrelated accounts
can share a key and share a partition, which costs a little parallelism and nothing else. The direction that
matters for correctness holds: one account is never split across two partitions.

Keyset, not offset. Executing an instruction stamps its last_run_date and so removes it from the due
set. An offset page would step over as many instructions as the previous page had committed. The reader
therefore pages by keyset over the (priority, id) sort key.

The chunk and its replay. A chunk is attempted in one transaction. If any instruction in it fails, the
chunk rolls back and the step replays it one instruction per transaction. That replay is what now delivers
what this ticket is about: a failing instruction cannot leave a sibling reverted, because the sibling is
re-executed and committed on the replay. It also means an instruction can be presented for execution twice
in one run, which is why StandingInstructionExecutionService.execute claims the instruction first, by
conditionally stamping last_run_date, and transfers nothing if the claim finds the instruction already run
for that business date.

Retry and skip. Only transient failures are retried — an account short of funds will not have more of
them a moment later, so retrying only delays the run and pads the mandate's history. It is skipped and
recorded instead, in a transaction of its own so the record outlives the rollback of the transfer. Skipping
is deliberately unlimited: a count-based limit would fail the job on a day when many accounts happen to be
short, which is the day it most needs to run.

A deployment whose instructions fail often enough that the replay costs more than it saves can set
EXECUTE_STANDING_INSTRUCTIONS_CHUNK_SIZE=1 and get per-instruction behaviour with no code change.

Also fixed here

• The retrieval query sorted ORDER BY atsi.priority DESC while the enum is URGENT(1)..LOW(4), so the job
  worked through the lowest priorities first.
• The history row was written by string-concatenated INSERT, recording the attempted amount narrowed to
  a double. History is now a JPA entity recording what actually moved (zero on failure), in BigDecimal.

Deliberately not in this PR

• Persisted next_run_date. Liked on the thread, but it is a schema change and dev@ question 1 — how it
  should react to backdated / valid_from edits — was never answered. Worth its own ticket.
• Grouping partitions by destination account too. That is a union-find over the transfer graph, and the
  credit side does not carry the balance check that makes the debit side contend. Residual contention there
  is what the retry is for.

Testing

Unit tests cover the partitioner (including a day with nothing due), the keyset reader (page cursor
advances, a short page ends the partition), the due-ness processor, the writer's contract with the chunk (a
failure must propagate, not be swallowed), the skip listener's durable failure record, and the claim.

The integration test runs the real scheduler job with one under-funded instruction among funded ones, and
asserts the successful transfer persists while the failing instruction leaves a durable failed history
row — the exact scenario that used to revert the run.

Both new queries were run against live MySQL 8.0 and PostgreSQL 16 tenant databases, and the keyset walk was
checked page by page against the full ordering for overlaps and gaps.

Checklist

• Write the commit message as per our guidelines
• Acknowledge that we will not review PRs that are not passing the build ("green") - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.
• Create/update unit or integration tests for verifying the changes made.
• Follow our coding conventions.
• Add required Swagger annotation and update API documentation at fineract-provider/src/main/resources/static/legacy-docs/apiLive.htm with details of any API changes — no API changes in this PR; it rebuilds a scheduled job.
• This PR must not be a "code dump". Large changes can be made in a branch, with assistance. Ask for help on the developer mailing list.
• If merging this PR resolves a JIRA issue, I will mark that issue as resolved and set "Fix Version/s" appropriately.

[adamsaghy]

@oluexpert99 I like the idea (based on FINERACT-2672) but i am missing a couple things:

• No paginated fetching of standing instructions, it would make sense to fetch in chunks of 100 entries
• I would consider allowing to try to execute the whole chunk in 1 transaction, if any of them fails, we can fallback to save each one in separate transactions
• I would consider process them parallel (standing instructions might need to be grouped together by from account or to account to avoid locking issues) -> I would also introduce retry mechanism (Retry4j) to automatically retry them before we mark it as failed
• We might want to consider rewriting this whole logic to rather use Spring Batch remote worker capabilities (better retry mechanism, better parallelization and easier chunk usage)
• I liked the idea of next run date.

Would you mind to move this whole idea and conversation to the Fineract DEV mail list? It might be interesting conversation and we can finalize the appropriate design over there.

[github-actions]

This pull request seems to be stale. Are you still planning to work on it? We will automatically close it in 30 days.

[adamsaghy]

@oluexpert99 Have you had the chance to raise this topic on FINERACT DEV mail list?

[adamsaghy]

@oluexpert99 Please let us know if you are happy to finish this story.

[oluexpert99]

@oluexpert99 Please let us know if you are happy to finish this story.

Hi @adamsaghy , sincere apologies for the delay in response . I am more than happy to close this out . Let me know if I cna proceed

[adamsaghy]

@oluexpert99 Please let us know if you are happy to finish this story.

Hi @adamsaghy , sincere apologies for the delay in response . I am more than happy to close this out . Let me know if I cna proceed

Thank you. Go ahead. please check out my concerns:
#6078 (comment)

[oluexpert99]

Hi @adamsaghy ,
Thank you for the support. PR has now been updated

[adamsaghy]

@oluexpert99 I am traveling at the moment, but i will try to review next week.

[adamsaghy]

@oluexpert99 Please rebase