fix(load): validate TsFile before async move - #18531
Conversation
Caideyipi
left a comment
There was a problem hiding this comment.
The async move path still has two correctness gaps. I am requesting changes because both can make the source and active-load state diverge.
Please see the two inline findings below.
| } | ||
|
|
||
| // Validate before moving the source so ordinary or malformed files remain in place. | ||
| if (!isValidTsFile(file)) { |
There was a problem hiding this comment.
[P1] Validate the complete batch before moving any file
loadTsFileAsyncToActiveDir calls this method once per *.tsfile. If a directory contains a valid *.tsfile followed by a malformed *.tsfile, the valid file has already been copied/linked into the active-load directory before this call returns false for the malformed one. doAsyncLoad then falls back to normal analysis, but the first file remains queued; with on-success='delete', its source may already be deleted as well. This can cause partial ingestion and unsafe retries. Validate all files before starting the transfer, or roll back every transfer when a later validation fails.
| return true; | ||
| } | ||
|
|
||
| private static boolean isValidTsFile(final File file) { |
There was a problem hiding this comment.
[P1] Apply this validation to the Pipe async path too
IoTDBDataNodeReceiver.loadTsFileAsync calls LoadUtil.loadFilesToActiveDir, but isValidTsFile is only invoked from loadTsFilesToActiveDir. The Pipe seal path therefore still transfers and deletes the main TsFile even when its magic is invalid, then returns SUCCESS_STATUS; the active loader fails later, after the sender has already been acknowledged. Identify the main .tsfile entry and apply the same validation in loadFilesToActiveDir before transferFilesToActiveDir (while preserving valid .resource/.mods sidecars), and add coverage for this path.
Description
This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR