[Pipe] Support receiver-side async takeover for TsFile conversion - #18421
Open
Caideyipi wants to merge 2 commits into
Open
[Pipe] Support receiver-side async takeover for TsFile conversion#18421Caideyipi wants to merge 2 commits into
Caideyipi wants to merge 2 commits into
Conversation
jt2594838
reviewed
Aug 7, 2026
Comment on lines
+547
to
+553
| final String conversionTaskId = | ||
| PipeTransferTsFileSealWithModReq.generateConversionTaskId( | ||
| sinkTaskId, | ||
| events, | ||
| dataBaseName, | ||
| outputIndex, | ||
| Objects.nonNull(modFile) && clientManager.supportModsIfIsDataNodeReceiver()); |
Contributor
There was a problem hiding this comment.
Generate the conversion key only when necessary (shouldAsyncLoadTsFileOnTypeMismatch == true)?
The calculation seems costly.
Comment on lines
+387
to
+389
| case PIPE_CONVERSION_TASK_ID_KEY: | ||
| if (value == null || value.isEmpty()) { | ||
| throw new SemanticException(StorageEngineMessages.USER_NAME_MUST_NOT_BE_EMPTY); |
Comment on lines
+276
to
+279
| final boolean asyncLoadOnTypeMismatch = | ||
| Boolean.parseBoolean( | ||
| attributes.getOrDefault( | ||
| ActiveLoadPathHelper.PIPE_ASYNC_LOAD_ON_TYPE_MISMATCH_KEY, "true")); |
Contributor
There was a problem hiding this comment.
Why is asyncLoadOnTypeMismatch used by ActiveLoader?
Comment on lines
+388
to
+396
| private String getConversionTaskId(final ActiveLoadPendingQueue.ActiveLoadEntry entry) { | ||
| final File tsFile = new File(entry.getFile()); | ||
| final File pendingDir = | ||
| entry.getPendingDir() == null | ||
| ? ActiveLoadPathHelper.findPendingDirectory(tsFile) | ||
| : new File(entry.getPendingDir()); | ||
| return ActiveLoadPathHelper.parseAttributes(tsFile, pendingDir) | ||
| .get(ActiveLoadPathHelper.PIPE_CONVERSION_TASK_ID_KEY); | ||
| } |
Contributor
There was a problem hiding this comment.
Is it possible to store the taskId in the entry?
| .getTsFiles() | ||
| .forEach( | ||
| tsFile -> { | ||
| org.apache.iotdb.commons.utils.FileUtils.deleteFileIfExist(tsFile); |
| }); | ||
| } | ||
|
|
||
| private static final class TableConversionContext implements AutoCloseable { |
Contributor
There was a problem hiding this comment.
TableConversionContext -> TabletConversionContext
Comment on lines
+295
to
+300
| int contextCount = 0; | ||
| for (final Task task : TASKS.values()) { | ||
| if (task.conversionContext != null) { | ||
| contextCount++; | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Is it possible to use an atomic var?
Comment on lines
+330
to
+332
| } catch (final Exception ignored) { | ||
| // Best-effort cleanup. The task state is still authoritative. | ||
| } |
Contributor
There was a problem hiding this comment.
May leave a warn log for debugging.
Comment on lines
+292
to
+301
| private static boolean isExistingTaskComplete( | ||
| final File transferDir, final List<File> sourceFiles) { | ||
| if (!transferDir.isDirectory()) { | ||
| return false; | ||
| } | ||
|
|
||
| final File[] targetFiles = transferDir.listFiles(File::isFile); | ||
| if (targetFiles == null || targetFiles.length == 0) { | ||
| return false; | ||
| } |
Contributor
There was a problem hiding this comment.
Why does a non-existent or empty dir indicate a task is not complete?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds resilient receiver-side handling for Pipe TsFile type-conversion failures.
connector.exception.data.convert-on-type-mismatch.tsfile.async-load(and thesink.*alias), defaulting totrue, and propagates the setting with a stable conversion task ID.Tests
git diff --checkpassed.