[Pipe] Add reliable DataNode completion metric

[Caideyipi]

Pipe Completion Metric Design

1. Conclusion

This implementation retains the complete end-to-end barrier protocol from Apache IoTDB PR #18280 and adds expected-DataRegion integrity checks on the current branch.

The new metric is:

pipe_datanode_completion_ready{name="<pipeName>",creation_time="<creationTime>"}

The metric has exactly two possible values, 0 and 1:

• 1: On this sender DataNode, the latest full-FLUSH completion barrier for every DataRegion currently expected by this Pipe has passed through the source, processor, and sink, and has been committed in order after a successful sink ACK. There are no pending non-heartbeat events, and no changes have been detected in task, source, assigner, committer, exception, or degraded state.
• 0: The operation is incomplete, unsupported, unknown, affected by a race, or failed a health check. The value 0 does not distinguish among these reasons.

The design deliberately allows temporary false negatives but never false positives. The existing remaining_event_count remains a progress metric: a value of 0 is necessary for completion, but it is not sufficient proof that sending has completed.

2. Why remaining_event_count cannot simply include "waiting for flush" and "captured TsFile"

A smaller implementation could add the storage engine's processors waiting to close, captured TsFiles, and Pipe queues together, but that would provide only an approximate backlog and could not reliably prove that sending had completed.

2.1 State transitions create counting gaps

The same TsFile goes through:

working/closing processor -> close callback -> assigner -> source queue -> processor queue -> sink/in-flight RPC

If these structures are read and summed separately, an object moving between two structures may temporarily belong to neither of the snapshots that have been read, producing a transient 0. Adding more containers only narrows the window; it cannot eliminate a cross-component snapshot race.

2.2 An empty queue does not mean that the sink has confirmed receipt

After an event is taken from the sink queue and before the destination returns a successful ACK, the queue may already be empty. At that point, remaining_event_count == 0 still does not prove that the destination has received or loaded the data.

2.3 FLUSH and concurrent writes have a boundary race

After a full FLUSH starts, a concurrent insert may create a new working processor that is not included in the capture set for that FLUSH. Sending all older TsFiles therefore does not mean that the current round of writes is complete. Every full FLUSH must establish a token, and a concurrent insert must invalidate that token.

2.4 Lifecycle changes can make old state report completion incorrectly

If a Pipe task, realtime source, assigner, or committer is replaced, state committed by the old instance cannot prove that the new instance is complete. If task initialization fails and a DataRegion is missing, the absence of a local task must not be interpreted as completion.

Therefore, if the goal is only an approximate backlog, a small change to remaining_event_count is sufficient. If the goal is to determine reliably that a Pipe has finished sending, an end-to-end ordered barrier is required. The main purpose of #18280 is to close the false-positive windows described above; retaining only one of its counting points would not be sufficient.

3. Completion barrier flow

stop and join writers
        |
        v
run a full FLUSH covering all relevant DataRegions
        |
        +-- invalidate the old completion token and record a new token
        +-- capture the working and closing TsFileProcessors at FLUSH entry
        +-- wait for ordinary asynchronous flushes already in progress
        +-- close and await the captured processors and their close callbacks
        +-- publish the barrier only if the token was not invalidated by a concurrent insert
        |
        v
assigner (bound to the assigner epoch and data generation)
        |
        v
realtime source -> processor (the barrier cannot be swallowed or rewritten)
        |
        v
sink queue (the barrier is not coalesced with ordinary heartbeat events)
        |
        v
successful sink ACK -> ordered commit -> onCommitted hook
        |
        v
completion operator: double-snapshot validation
        |
        v
pipe_datanode_completion_ready = 1

The key point is that the barrier is ordered after the TsFile events captured and published by this FLUSH. A DataRegion is marked complete only after the sink has successfully processed the barrier and the ordered commit has completed.

4. Fail-closed conditions

The metric returns 0 if any of the following conditions holds:

• remaining_event_count still contains a non-heartbeat event;
• the Pipe does not exist, is not a RUNNING USER Pipe, or has a Pipe/runtime/task exception;
• the source, processor, sink, or TsFile load strategy is unsupported;
• the historical source has not been fully consumed, or the realtime source has not started completely;
• the actual DataRegion task set differs from the expected set calculated from PipeMeta, the leader, and the local StorageEngine;
• a task, source, assigner, or committer instance has been replaced;
• the full-FLUSH token was invalidated by a concurrent insert;
• the barrier generation is behind the latest data generation;
• event publication, reference counting, enqueueing, or delivery fails;
• a hybrid source is waiting for TsFile recovery to restore a discarded tablet (degraded);
• any change occurs between the two task-topology snapshots or between the two state snapshots;
• a runtime exception occurs during metric calculation, or acquiring the task read lock times out.

Lost events and publication failures cannot be automatically proven to have recovered. The metric therefore remains 0; the usual recovery is to fix the problem, restart or rebuild the relevant Pipe tasks, and execute the completion protocol again.

5. Supported scope

Currently, only the following combinations can return 1:

• a RUNNING USER Pipe;
• source/extractor: iotdb-extractor or iotdb-source;
• processor: do-nothing-processor;
• sink/connector: a built-in IoTDB Thrift connector/sink, including its sync, async, SSL, and other aliases;
• TsFile load strategy: sync;
• the source has started and the historical phase has been fully consumed; and
• DataRegion/DML transmission has completed.

This metric does not currently prove that SchemaRegion/DDL operations have completed, and it provides no completion guarantee for custom processors, custom sinks, or asynchronous TsFile load strategies. These combinations conservatively return 0.

6. Correct use of the protocol

1. Stop all writers that may write to the relevant DataRegions and wait for their threads to finish. The current generation is scoped to a DataRegion; even concurrent writes outside the Pipe pattern may conservatively keep the metric at 0.
2. Run a successful full FLUSH covering every DataRegion that the Pipe may send. A global full FLUSH is recommended. Do not use a partial flush that closes only SEQ or UNSEQ processors as the completion barrier.
3. After the FLUSH returns successfully, obtain a fresh metric sample on each expected sender DataNode.
4. Conclude that this round of Pipe DML transmission is complete only when all expected series exist and every value is 1.

In a Prometheus deployment, also confirm that:

• the sample timestamp is later than this FLUSH;
• every target has up == 1;
• the number of series matches the expected number of sender DataNodes; and
• missing series, scrape failures, and stale samples are not treated as completion.

creation_time distinguishes different incarnations of a Pipe with the same name. The evaluation must be pinned to the exact name + creation_time for this incarnation; do not aggregate old series by name alone.

7. Relationship to PR #18280 and adaptation on the current branch

This implementation adopts the core commit 12bc3f3a5c00265f5c04dc28a4e6c76affe326bf from #18280 ([Pipe] Add reliable DataNode completion metric). The complete implementation includes:

• waiting during a full FLUSH for working and closing processors and overlapping ordinary asynchronous flushes;
• completion tokens, data generations, assigner epochs, and publication-failure epochs;
• ordered barrier preservation through the source, processor, and sink queue;
• an ordered-commit hook after the sink ACK;
• task/source/assigner/committer lifecycle validation; and
• a fail-closed completion operator and concurrency tests.

The current branch additionally reuses the expected-DataRegion calculation already present in master and requires:

actual DataRegion source ids == expected DataRegion ids

This prevents a false positive when a DataRegion task fails to initialize or is missing, which could otherwise make an empty or incomplete local task set look complete.

8. Validation results

• Spotless: passed;
• clean test-compile for the relevant modules: passed;
• 18 focused barrier, lifecycle, queue, and full-FLUSH concurrency tests: Failures 0, Errors 0, Skipped 0;
• English-locale all-reactor test-compile: 52/52 passed;
• Chinese-locale all-reactor test-compile: 52/52 passed; and
• git diff --check: passed.

The focused tests cover completion generation, fail-closed behavior, membership changes, source replacement, committer replacement, barriers not being merged with heartbeats, event-collection failure, inserts invalidating a flush token, and overlap between a full FLUSH and an ordinary asynchronous flush.