Add 4 KiB logical block footprint tracking to stats - #1070
Conversation
8801271 to
59ef172
Compare
811708d to
ee67473
Compare
|
@steadytao ubuntu workflow is the only one forced running rsync with --protocol=30 which is failing the tests. |
5ff8909 to
4ccfc8b
Compare
34bd1aa to
409fb36
Compare
steadytao
left a comment
There was a problem hiding this comment.
git diff --check reports trailing whitespace --
The C indentation is inconsistent, the sparse test still describes “physical” blocks/disk writes and the test file lacks a final newline. Please use the logical-region terminology consistently throughout the test as well.
This adds a 'Number of 4 KiB logical blocks touched' metric to the --stats output to decouple network delta payload from actual local file modifications. Previously, a small amount of literal data scattered across a file (especially with --inplace) could result in a massive number of local file write operations with no visibility, and large sparse files masked their true write opreations (ignoring punch holes). Technical details: - Implemented a stateful block tracker in the receiver that calculates touched 4K boundaries using file offsets and lengths, including strict lseek awareness to accurately skip sparse file holes. - Enforced strict per-file lifetime state with a reset hook inside receive_data(), successfully mitigating POSIX file descriptor (FD) recycling state leaks. - Created MSG_BLOCK_STATS multiplex message to tunnel the block footprint safely out of the isolated receiver process and relay it over the network. - Bumped PROTOCOL_VERSION to 33 and SUBPROTOCOL_VERSION to 8392 for safe PR testing. - Added test suite covering contiguous, scattered, zero-byte, sparse file (hole-skipping), multi-file (FD reuse), batch mode, and maximum-I/O boundary conditions.
073c9ed to
68e7e68
Compare
|
Yes thanks! the test name is changed also so you need to reconfigure the skiplist. |
68e7e68 to
21cef6a
Compare
316f9b0 to
4da79f3
Compare
|
I fixed the trailing spaces and reverted SUBPROTOCOL_VERSION back to 0. Sorry mate I forced push so I hope it didn't clear your skip list config. ( :( Didn't notice ) |
It happens. Make sure to force with lease next time :D |
Fixes #1066
Original feature request by @birdie-github
Currently, rsync --stats reports Literal data, which represents the unmatched file-update data sent over the network. However, network payload is often completely disconnected from the actual disk I/O footprint on the receiving end. Depending on the transfer flags (e.g., default temp-file rebuilding vs. --inplace), a tiny network payload can result in massive disk writes, or vice versa. System administrators currently have no native visibility into the true I/O cost of a sync.
This PR introduces a new metric to the --stats output: Number of modified 4K blocks. It tracks exactly how many 4,096-byte boundaries are written to the receiver's disk, completely decoupling network delta payload from disk I/O footprint.
O(1) Zero-Overhead Implementation
To ensure tracking does not bottleneck the receiver's high-speed I/O loop, the block calculation operates entirely in O(1) time with zero system calls and no loop iterations.
Stateful Tracking: It uses a static state machine (last_tracked_fd and last_touched_blk) to remember the last logical block written. Because the receiver strictly writes sequentially, we can calculate block deltas using purely mathematical boundaries.
Math over Memory: For each write, it calculates the block boundaries using (offset % 4096) and len. It diffs this against the last known state and instantly updates the global counter.
Real-World Examples (Why this matters)
Here is a demonstration of how network data and disk I/O diverge, using a 4MB file where we inject a single byte at 10 different 4K block boundaries:
Case 1: Delta transfer with --inplace
Context: Rsync sends a small amount of literal data over the network to patch the file. Because --inplace is used, the receiver seeks and updates only the 10 specific blocks on disk. Network and Disk I/O are both low.
Case 2: Delta transfer without --inplace (Default temp-file behavior)
Context: Rsync perfectly matches the unchanged data locally. The network payload is effectively zero. However, because rsync reconstructs the file into a new hidden temp file before moving it into place, the receiver's disk is forced to rewrite the entire 4MB file (1,024 blocks).