Fastest way to read a text file in C#: net10.0, async reads, benchmark re-run - #2172
Open
vladimir-pecanac-main wants to merge 1 commit into
Open
Conversation
…e-run benchmark - Both projects net7.0 -> net10.0; Nullable enabled on the sample project. - BenchmarkDotNet 0.13.7 -> 0.15.8. - Test packages to current: Microsoft.NET.Test.Sdk 18.9.0, xunit 2.9.3, xunit.runner.visualstudio 4.0.0, coverlet.collector 10.0.1. This clears NU1903 (Newtonsoft.Json 9.0.1 came in transitively through the old test SDK). - Primary constructor on WaysToReadATextFileInCsharp. - UseBufferedStreamObjectWithNoFileStreamBuffer moves to the FileStreamOptions overload. FileStreamOptions.BufferSize documents "0 or 1 means that buffering should be disabled"; the five-argument FileStream constructor documents its bufferSize as greater than zero and throws on zero. Same behaviour, but the section now stands on the API whose docs state what it teaches. - Three async reads with matching benchmarks and tests: UseFileReadAllTextAsync, UseFileReadLinesAsync, UseStreamReaderReadToEndAsync. The StreamReader one is built over a FileStream with FileOptions.Asynchronous rather than the path-taking constructor, which does not set that flag and so fakes async file access on the thread pool. - Benchmark file resolved with AppContext.BaseDirectory and copied to the output directory, replacing a six-level Directory.GetParent walk tied to one layout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This retargets the fastest way to read a text file sample to
net10.0, adds the asynchronous reads the article never covered, and re-runs the benchmark so the article's numbers describe the runtime it says it was tested on. The published numbers were measured onnet7.0with BenchmarkDotNet 0.13.7 before September 2023.What changed
net7.0->net10.0, and<Nullable>enabled on the sample project (the test project already had it).0.13.7->0.15.8.Microsoft.NET.Test.Sdk18.9.0,xunit2.9.3,xunit.runner.visualstudio4.0.0,coverlet.collector10.0.1. The sample stays on the xunit v2 line; only the runner moves, and the 13 tests pass on it.Testsproject was pullingNewtonsoft.Json9.0.1 transitively through the 2022-eraMicrosoft.NET.Test.Sdk17.3.2 pin, and that version carries a known high-severity advisory (GHSA-5crp-9r3c-p9vr). The test SDK bump drops the transitive reference.WaysToReadATextFileInCsharp. The_sampleFilePathfield is kept so every snippet the article prints is still byte-identical to the sample.UseBufferedStreamObjectWithNoFileStreamBuffer()moves to theFileStreamOptionsoverload. The section teaches that a buffer size of 0 disables buffering, which is whatFileStreamOptions.BufferSizedocuments ("0 or 1 means that buffering should be disabled"), while the five-argumentFileStreamconstructor the sample was using documents itsbufferSizeas "greater than 0" and listsArgumentOutOfRangeExceptionwhen it "is negative or zero". Both forms run without throwing on .NET 10, so this changes no behaviour; it moves the section onto the API whose documentation states what the section teaches.UseFileReadAllTextAsync(),UseFileReadLinesAsync(),UseStreamReaderReadToEndAsync().StreamReaderone is built over aFileStreamcarryingFileOptions.Asynchronous, not over the path-takingStreamReaderconstructor.dotnet/runtimeis blunt about why, inSystem/IO/File.cs: "If we use the path-taking constructors we will not have FileOptions.Asynchronous set and we will have asynchronous file access faked by the thread pool. We want the real thing."AppContext.BaseDirectoryand copied to the output directory, replacing a six-levelDirectory.GetParent(...).Parent.Parent.Parent.Parent.Parent.Parentwalk tied to one build layout. WithNullableenabled that walk was also seven CS8602 warnings.Verification
Benchmark, re-run on net10.0
The benchmark file is the folder's own
BenchmarkTextFile.txt, roughly 98 KB. Every method exceptFile.ReadAllText/StreamReader.ReadToEndand their asynchronous forms reads the file line by line and then reassembles one string through aStringBuilder, so the numbers measure "read the file and rebuild it as a string", not "read the file".Run 1, verbatim:
Run 2, verbatim, on the same machine and the same binaries:
What the two runs say
The old winner has not been beaten, but it can no longer be named on its own.
File.ReadAllText()andStreamReader.ReadToEnd()finish within half a microsecond to two microseconds of each other, well inside their own error bars, and the order between them flips between runs: run 1 putsFile.ReadAllText()first by 2.1 us, run 2 putsStreamReader.ReadToEnd()first by 0.5 us. They also allocate the same 410.83 KB in both runs. The honest reading is a tie between the two methods that read the file whole in one pass, and the article now says that instead of naming one.Two of the published article's other claims move with the re-run:
ReadBlockvariants come next, and theSpanone is both the fastest of the three and the lightest (418.71 KB against 609.89 KB and 617.92 KB).StreamReader.ReadLine()andFile.ReadLines()are the two slowest. On .NET 10StreamReader.ReadLine()sits mid-table, and the bottom of the synchronous ten isFile.ReadAllLines()andFile.ReadLines().Timings are from one desktop and are there to rank the methods, not to be quoted as absolute figures.