Skip to content

fix(load): validate TsFile before async move #18531

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
luoluoyuyu wants to merge 1 commit into
base: master
Choose a base branch
from
+50 −1
Open

fix(load): validate TsFile before async move #18531

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBLoadTsFileAuthIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
import org.junit.runner.RunWith;

import java.io.File;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.sql.Connection;
import java.sql.ResultSet;
Expand Down Expand Up @@ -180,6 +181,34 @@ public void testAsyncLoadShouldCheckWriteDataPermissionWithStoredUser() throws E
assertCountEventually(10, TimeUnit.SECONDS.toMillis(60));
}

@Test
public void testAsyncLoadShouldNotMoveFileWithoutTsFileSuffix() throws Exception {
assertAsyncLoadDoesNotMoveInvalidTsFile(new File(tmpDir, "not-a-tsfile.txt"), "Can not find");
}

@Test
public void testAsyncLoadShouldNotMoveInvalidTsFile() throws Exception {
assertAsyncLoadDoesNotMoveInvalidTsFile(new File(tmpDir, "invalid.tsfile"), "Loading file");
}

private static void assertAsyncLoadDoesNotMoveInvalidTsFile(
final File sourceFile, final String expectedErrorMessage) throws Exception {
final String originalContent = "ordinary file content";
Files.write(sourceFile.toPath(), originalContent.getBytes(StandardCharsets.UTF_8));

assertNonQueryTestFail(
String.format(
"load \"%s\" with ('async'='true', 'on-success'='delete')",
sourceFile.getAbsolutePath()),
expectedErrorMessage);

Assert.assertTrue(
"Non-TsFile source must remain in its original location", sourceFile.isFile());
Assert.assertEquals(
originalContent,
new String(Files.readAllBytes(sourceFile.toPath()), StandardCharsets.UTF_8));
}

private static void prepareSchemaAndTsFile(final File tsFile) throws Exception {
prepareSchema(MEASUREMENT.getType());
generateTsFile(tsFile);
Expand Down
2 changes: 1 addition & 1 deletion ...rc/main/java/org/apache/iotdb/db/queryengine/plan/statement/crud/LoadTsFileStatement.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ private static List<File> processTsFile(
}

final List<File> tsFiles = new ArrayList<>();
if (file.isFile()) {
if (file.isFile() && file.getName().endsWith(TsFileConstant.TSFILE_SUFFIX)) {
tsFiles.add(file);
} else {
if (file.listFiles() == null) {
Expand Down
20 changes: 20 additions & 0 deletions iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/util/LoadUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,9 @@
import org.apache.iotdb.db.storageengine.load.active.ActiveLoadPathHelper;
import org.apache.iotdb.db.storageengine.load.disk.ILoadDiskSelector;

import org.apache.tsfile.common.conf.TSFileConfig;
import org.apache.tsfile.common.constant.TsFileConstant;
import org.apache.tsfile.read.TsFileSequenceReader;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand Down Expand Up @@ -118,6 +120,11 @@ private static boolean loadTsFilesToActiveDir(
return true;
}

// Validate before moving the source so ordinary or malformed files remain in place.
if (!isValidTsFile(file)) {

@Caideyipi Caideyipi Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Validate the complete batch before moving any file

loadTsFileAsyncToActiveDir calls this method once per *.tsfile. If a directory contains a valid *.tsfile followed by a malformed *.tsfile, the valid file has already been copied/linked into the active-load directory before this call returns false for the malformed one. doAsyncLoad then falls back to normal analysis, but the first file remains queued; with on-success='delete', its source may already be deleted as well. This can cause partial ingestion and unsafe retries. Validate all files before starting the transfer, or roll back every transfer when a later validation fails.

return false;
}

final File targetFilePath;
try {
targetFilePath =
Expand Down Expand Up @@ -145,6 +152,19 @@ private static boolean loadTsFilesToActiveDir(
return true;
}

private static boolean isValidTsFile(final File file) {

@Caideyipi Caideyipi Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Apply this validation to the Pipe async path too

IoTDBDataNodeReceiver.loadTsFileAsync calls LoadUtil.loadFilesToActiveDir, but isValidTsFile is only invoked from loadTsFilesToActiveDir. The Pipe seal path therefore still transfers and deletes the main TsFile even when its magic is invalid, then returns SUCCESS_STATUS; the active loader fails later, after the sender has already been acknowledged. Identify the main .tsfile entry and apply the same validation in loadFilesToActiveDir before transferFilesToActiveDir (while preserving valid .resource/.mods sidecars), and add coverage for this path.

if (!file.isFile() || !file.getName().endsWith(TsFileConstant.TSFILE_SUFFIX)) {
return false;
}
try (final TsFileSequenceReader reader =
new TsFileSequenceReader(file.getAbsolutePath(), false)) {
return TSFileConfig.MAGIC_STRING.equals(reader.readHeadMagic())
&& TSFileConfig.MAGIC_STRING.equals(reader.readTailMagic());
} catch (Exception e) {
return false;
}
}

private static Map<String, String> appendCurrentUserIfAbsent(
final Map<String, String> loadAttributes) {
final Map<String, String> attributes =
Expand Down
Loading