Counting lines of code sounds like wc -l with a few if statements. Doing it properly on the Linux kernel in under 70 milliseconds demands a lot more from an 8-core desktop CPU.

In that time the program walks ~5,900 directories holding ~88,800 files and picks out the ~63,700 source files to count. It opens and reads all 1.37 GB of them from a warm page cache, which works out to roughly 20 GB/s. It validates UTF-8 and settles ambiguous extensions like .h from the file's content. Then it lexes 36 million lines, tracking escaped quotes, raw strings and block comments, to tell code from comments and blanks.

(The tree is Linux v6.15 at commit 0ff41df1).

This post goes through what it took to bring mezura from about 0.4 s down to 67 ms on that workload:

Kernel run on LinuxWhat got it there
about 0.4 sversion 2.0.0: str::lines(), a delimiter search per line, a cleaned copy of every line
166 msstill safe Rust only: byte ranges in place of the copies, files handed over in batches
88 msone SIMD pass per 64-byte block, mmap for large files, the first unsafe
67 msplain lines classified a block at a time, the hot loop compiled per CPU tier, one lexer
41 msthe hardware floor (walk, read, and a simplistic parser)

Along the way there was hyperfixation on local minima, profiling tools that pointed the wrong way (cutting branch misses by 14% made the program 12% slower), a Windows filter driver that made one drive letter a third slower than the other, and a lexer that got faster and more accurate in the same change.

1. The hardware floor as a compass (don't get trapped in local minima)#

If you optimize only by looking at flamegraphs of your existing code, it is easy to think you are done too early. Once you fix the obvious bottlenecks and no single function dominates the profile, every change starts taking days of work for a 1% or 2% gain. That feels like the limit of the hardware, when most of the time it is a local minimum of your current architecture.

To know whether you are fast or stuck in a bad design, you need the hardware floor for your problem. For a line counter that is what the machine needs to walk directories, read files from the page cache, and scan bytes in RAM.

A floor program can have design flaws too, so I kept mine as close to the system calls as I could. It is a single C file (gcc -O3 -march=native) on opendir/readdir, open, read, mmap, pthread and <immintrin.h>, with static buffers and no heap allocation in the hot path. I measured it in four steps over the kernel tree on a Ryzen 7 9700X (8 cores, 16 hardware threads, Debian 13, ext4):

StepWhat it doesTime, 16 threads
1. Memory bandwidthThe 1.37 GB streamed through memory once, at the ~60 GB/s this machine sustains23 ms, computed from the measured rate
2. File I/O, no directory walkEvery path known in advance, each file opened, read (read under 256 KiB, mmap from there) and closed30.4 ms
3. Directory walk + file readsA concurrent readdir walk while every matching file is opened and read, with no byte inspected35.9 ms best, 36.5 median
4. Walk + read + SIMD + state machineStep 3 plus one SIMD pass per 64-byte block and a minimal state machine over the 15.8M bytes where a comment or a string can begin or end40.8 ms best, 42.3 median

Isolated costs do not add up linearly when they run together. On its own across 16 threads, the directory walk takes 2.9 ms, and open plus close on every file takes 14.0 ms. Running that 2.9 ms walk alongside the 30.4 ms file reads takes 35.9 ms rather than 33.3 ms, because walkers and readers contend for the same cores and kernel VFS structures. A realistic floor has to run everything concurrently.

Running Step 4 with every path known in advance and no directory walk at all took 38.2 ms, and a single pool where every thread both walks and reads landed within a millisecond of separate walker and reader pools (40.8 ms vs 41.1 ms). Neither the walk nor the thread layout was a bottleneck, so a line counter on this machine cannot go much below 41 to 42 ms on this tree.

I wrote this program late, when mezura already ran the kernel in 88 ms, after I had spent a lot of time profiling the code and just before I convinced myself that it ain't getting any faster. No single function stood out in the profile by then, and 88 ms looked like a natural place to stop. The floor showed that more than half of that run was still my own code.

2. How to measure 67 ms, and how profiling tools mislead#

At tens of milliseconds across 16 threads, casual wall-clock timing is noisy. The machine's speed drifts during a session, and reversing the order of two commands moved individual means by 1% to 10%.

To control for that I wrote linebench, a benchmark harness for line counters. It runs every table in both command orders and pools the two passes, checks the background CPU load and the power settings first, and times one control binary at the start and at the end of the session to show how far the machine drifted. It also provides useful performance insights like syscall breakdowns, hardware PMU counters, heap allocation counts, memory curves, core-scaling ladders, and cold-cache runs. It lives under the loc-conformance organization so that the authors of other counters can join the development, though today I maintain it by myself.

Where the hardware numbers pointed the wrong way#

A flamegraph shows where time is spent, not how much time you get back if you delete that code. Three times I removed a small operation taking 2% to 5% of flamegraph samples, and wall time did not move at all. Out-of-order cores pipeline cheap, frequent instructions alongside surrounding work, so deleting them frees zero cycles; only removing whole function calls consistently saved time.

Branch misses were a decoy. I replaced the byte loops that trim whitespace on every line with branchless word-at-a-time (SWAR) arithmetic. Branch misses fell by 14%, instructions rose by 16%, and the program got 12% slower. The byte loops were fast simply because they were short, whereas the branchless math saved 0.17 branch misses per line at the cost of roughly 75 extra instructions on every line.

Elapsed time on a preempted thread is not CPU cost#

mezura splits work between producer threads that walk directories and consumer threads that steal batches of file paths to open, read, and parse. Instrumenting both pools with per-thread phase timers (waiting for work, opening, reading, parsing) produced two readings that sent me down the wrong path.

On Linux the timers showed the directory walkers alive for 90 ms of a run of about 110 ms, so the walk looked slow. Measured on its own, the same walk finishes in 13 ms. The 4 walker threads were just getting preempted by 32 consumer threads that kept all 16 hardware threads busy.

On Windows the timers put File::open at 62% to 70% of all consumer time, which made a three-stage pipeline of walkers, openers and parsers look worthwhile. In CPU samples the same call was 17%. The difference was scheduling delay, since a thread preempted inside CreateFileW keeps its timer running.

The syscalls no profile showed#

perf stat counted 66,831 sched_yield and 18,628 clock_nanosleep calls in a single run (23% of all syscalls mezura made), all from worker threads waiting on queues. None of them showed up in CPU flamegraphs (0.00% of samples) because sleeping threads are off-CPU. At the same time, phase timers showed the first consumer finishing 25 ms before the last one, so I let idle consumers steal files from busy ones instead of sleeping.

The exit gap dropped from 25 ms to 0.3 ms, yet wall time failed to improve at any thread count and got 4% slower at the default 64-consumer setup. Idle threads were now spinning over everyone else's queues on sibling hyperthreads, stealing execution resources from the working thread on the same physical core, while the real tail was a single thread parsing a 24 MB generated AMD GPU register header that could not be split anyway.

Work stealing between consumers only paid off later, once the default consumer count dropped from 64 to 32 and stealing was restricted to run only after directory walking had finished (taking the kernel run from 93.0 ms to 88.5 ms).

3. File I/O on Linux vs Windows#

The same machine gives two different profiles under Linux (ext4) and Windows 11 (NTFS). On Linux the 67 ms run spends 0.62 s of CPU in user code and 0.28 s in the kernel, summed over its threads, so it is bound by the parser. On Windows the same run takes 175 ms and spends 1.14 s in user code and 1.30 s in the kernel. More than half of its CPU time is inside the kernel, where files are opened, read and closed and directories are listed.

Avoiding 63,934 statx calls on Linux#

Rust's standard library reads a file to completion by asking the OS for its size and then calling read twice, the second call returning 0 to confirm the end of the file.

On Windows, FindNextFileW already returns the file size inside each directory entry, so the walk has it for free. A buffer of size + 1 bytes then takes the whole file and confirms its end in one read call, with no metadata call.

On Linux, getdents64 returns the inode and the file type (d_type) and no size. Asking DirEntry::metadata() for it during the walk costs one statx per file, 63,934 of them on the kernel tree.

So on Linux the walkers pass a size of 0, and each consumer issues one 64 KiB read into a reusable thread-local buffer. 95.7% of the kernel's files are smaller than that, so the single read comes back short, giving the full content, its exact length, and proof of EOF in one syscall. Only the 2,778 files that fill the window call fstat. Files of 256 KiB and larger are mapped with mmap, which cut wall time by 9% and page faults in half. Files modified in the last five seconds fall back to read, since truncating a mapped file mid-scan kills the process with SIGBUS.

The Banshees of D:\ (the ghost filters haunting Windows I/O)#

Across 16 threads on Windows, opening and closing a single file costs roughly 19 µs (13.7 µs in CreateFileW, 5.4 µs in CloseHandle). Trying to bypass Win32 path handling with lower-level NT calls went nowhere. In a standalone benchmark over the kernel's 88,421 files, directory-relative NtCreateFile and minimal FILE_READ_DATA access masks landed within 3% of plain File::open. Opening files directly by NTFS file ID actually made open 1.9x slower (and the whole run nearly 1.5x slower), because a file opened by ID carries no path string, forcing the filesystem filter drivers on the volume to reconstruct the path from scratch. Whichever API you call on Windows, every open still walks through security checks, handle creation, and every minifilter driver attached to that drive.

That filter stack turned out to be behind a much weirder mystery on my machine. Counting the exact same warm Linux kernel checkout on D:\ was taking 36% longer than on C:\ (611 ms vs 450 ms in controlled back-to-back runs, burning 4.8 s of kernel CPU against 2.9 s). How does a different drive letter burn almost two extra seconds of kernel CPU when the entire repository is already sitting in RAM?

  • Slower SSD on D:\? Sampled across 40 warm runs, physical disk reads were zero. Neither drive was being touched.
  • Windows Defender scanning D:\? mezura.exe was already excluded by process name, and D:\dev had a full folder exclusion on top, making D:\ more exempt from Defender than C:\.
  • NTFS formatting differences? Cluster size, indexing, compression, and 8.3 name generation were identical on both volumes.

Running fltmc instances in an elevated prompt finally caught it. D:\ had two extra Microsoft minifilter drivers sitting on every file open and directory listing that C:\ did not have, wcifs (Windows Container Isolation) and FileCrypt. Months earlier, I had changed Windows's "Where new content is saved" setting so Windows Store apps and movies would install on D: instead of C:, and Windows had quietly wired container and DRM encryption filters into every I/O call on the drive! Detaching wcifs alone did nothing, but the moment I ran fltmc detach FileCrypt D:, kernel CPU time dropped in half (4.9 s to 2.5 s) and D:\ actually beat C:\ by 10%.

The best part is that you cannot evict them cleanly once they move in. Moving every Windows Store save setting back to C:, unregistering the volume with Remove-AppxVolume, and taking ownership of D:\WindowsApps just to delete it made no difference. On the very next reboot, Windows silently re-attached both wcifs and FileCrypt to D:\ anyway. To this day, the only thing keeping D:\ clean is a SYSTEM scheduled task at boot that runs fltmc detach on both filters!

Right alongside that, Windows Defender pulled a second trick during A/B benchmarking. For weeks, every time I copied mezura.exe to candidate.exe or control.exe to test an optimization, the copy ran 24% to 32% slower on both drives, sending me looking for some weird Windows penalty on freshly compiled binaries. Then I copied the file into another folder while keeping the name mezura.exe, and the slowdown vanished. Defender's process exclusions match the bare filename, so mezura.exe bypassed real-time scanning on all 63,700 file opens while candidate.exe got inspected on every single one.

4. Evolution of the parser (0.4 s -> 166 ms -> 88 ms -> 67 ms)#

Stage 1 (0.4 s to 166 ms)#

Version 2.0.0 was straightforward multithreaded Rust. It iterated over str::lines(), searched each line for comment and string delimiters with memchr and memmem, and allocated a cleaned copy of every line to search for keywords. On the kernel tree that took about 0.4 s on Linux and roughly a second on Windows.

Replacing those per-line string copies with byte-range slices, stealing files in batches, and switching to a depth-first directory walk brought the Linux run down to 166 ms (22.4G instructions), still in 100% safe Rust and still four times slower than the 41 ms floor.

Stage 2 (166 ms to 88 ms, 64-byte bitmasks and targeted unsafe)#

Up to 166 ms the entire workspace used #![forbid(unsafe_code)]. Going further required explicit SIMD intrinsics and mmap, so I kept #![forbid(unsafe_code)] on the binaries (mezura and mezura-mcp) and switched the core library to #![deny(unsafe_code)], allowing individual unsafe blocks only when they gave a clear wall-clock speedup.

Instead of splitting a file into lines first and scanning each line for delimiters afterward, the parser now runs one AVX2 (x86_64) or NEON (aarch64) pass per 64-byte block to produce two u64 bitmasks, newlines and candidates (the bytes where a comment, string, or escape can start or end). I prototyped the SIMD routine with Claude's help and tested it against a scalar implementation on random inputs. That change alone cut wall time by 15% and user CPU by 20%.

On the Linux kernel, 87.6% of lines contain neither " nor /. Whenever no multiline comment or string is open, those plain lines do not need delimiter scanning. I also added an AVX2 ASCII check so pure-ASCII files skip UTF-8 validation via str::from_utf8_unchecked (-3.2% instructions), and used const generics to compile separate copies of the file loop so files without heredocs, nested languages, or inline tests do not pay branches for those features. Together with mmap on large files, the run came down to 88 ms.

Not every unsafe optimization worked. Using from_utf8_unchecked on individual line slices cut instructions by 7% while increasing CPU cycles by 8.6% (IPC dropped from 2.25 to 1.93 because of code-layout changes in the hot loop), so I threw it out.

Stage 3 (88 ms to 67 ms, carry-ripple math, CPU tiers, and a table lexer)#

By default, mezura groups punctuation-only lines (like a lone }) with blank lines rather than counting them as code, whereas --counting region counts any non-blank line as code. Because the engine records all three counts (blank, punctuation-only, and word-bearing) in one pass, the parser still looped over every plain line to trim whitespace and look for word characters.

My first try added two more AVX2 masks per 64-byte block, words and content (non-whitespace bytes), while keeping the per-line loop. Computing those masks took 37 instructions per block, raised kernel instructions by 10% to 18%, and made the 454 MB of 107-byte-line AMD GPU register headers in the tree 34% slower.

Classifying a whole block of lines with binary addition#

Computing words and content masks only pays off if you stop looping over individual plain lines. In u64 bitmask form, !newlines has 1 at every byte inside a line and 0 at every \n. Adding words to !newlines ripples a binary carry from any word byte in a line up through those 1s and flips the 0 bit at that line's \n to 1. If a line has no word bytes, no carry reaches its \n and that bit stays 0 (with bit 63's carry-out passed into the next 64-byte block):

let not_newlines = !newlines;
let (words, words_out) = add_with_carry(not_newlines, classes.words, carried.0);
let (content, content_out) = add_with_carry(not_newlines, classes.content, carried.1);
let (word_lines, content_lines) = (words & newlines, content & newlines);

Bit i of word_lines is 1 if the line ending at byte i contained a word character, and content_lines does the same for non-whitespace characters. Three count_ones() calls (word_lines, content_lines & !word_lines, and plain & !content_lines) then count all plain lines in the 64-byte block at once. Files whose average line length exceeds 64 bytes fall back to the per-line loop so the long-line AMD headers do not regress. Compared to the scalar fallback, those SIMD classification routines save 38.6% of total wall time on the kernel.

Runtime CPU dispatch (#[target_feature])#

Rust's default x86-64 target does not enable popcnt, so every count_ones() on a u64 compiles into a 19-instruction software fallback. Compiling three versions of the hot loop with #[target_feature] and selecting one per file at runtime (baseline, AVX2 + popcnt/bmi, and AVX-512, where avx512bw byte shuffles build words and content in 8 instructions per block instead of 37) saved 5% to 9% of wall time in a single portable binary.

A single table-driven lexer#

On the remaining 12.4% of lines that contain candidate bytes, the old parser built four delimiter lists per line and ran 330 lines of index math to resolve overlaps. Replacing that with a single left-to-right cursor stepping through precompiled state tables cut total kernel instructions by 21% between 3.4.0 and 3.5.0, bringing the PGO release build's equal-work time from 80 ms (88 ms without PGO) down to 67 ms.

Because two counters pointed at the same folder rarely count the same files or do the same work out of the box, linebench runs both a default mode and an equal-work mode. Equal-work mode turns off optional features that not every tool implements (ignore files, minified and generated file filters, shebang checks, keyword and test detection, complexity and cost estimates) and sets mezura to --counting region to match the others. When a tool runs a check unconditionally with no flag to disable it, such as reading file contents to tell C, C++, and Objective-C .h headers apart, the other tools that have that check leave it on too. After each run, linebench checks the JSON outputs to make sure all file and line totals match within 1%.

ToolLinux (ext4)Windows (NTFS)
mezura 3.5.067 ms ± 2 (540.7M lines/s)175 ms ± 15 (205.6M lines/s)
scc 4.1.0208 ms ± 3 (173.1M lines/s)521 ms ± 36 (69.1M lines/s)
tokei 15.0.0411 ms ± 3 (87.7M lines/s)617 ms ± 20 (58.4M lines/s)

Each number is the mean of 30 timed runs after 3 warmups across both command orders, ± one standard deviation. At each tool's default settings on Linux (with mezura's ignore files, keyword counting, and test detection on), mezura 3.5.0 runs in 85 ms ± 2, scc 4.1.0 in 290 ms ± 3, and tokei 15.0.0 in 418 ms ± 3.

What separates 67 ms from 42 ms#

Part of the difference comes from real work that mezura does and the minimal floor program skips. mezura validates UTF-8 on every file, checks the content of ambiguous .h headers (6.0% of all kernel instructions, left on in equal-work mode because scc 4.1.0 cannot turn its own check off), and runs a general table-driven parser for 80+ languages instead of hardcoding C. Disabled features do not slow down the equal-work run because const-generic specialization leaves them out of the compiled loop.

67 ms is not the ceiling either. Several concrete optimizations are still open without changing behavior, including folding the ASCII check into the 64-byte SIMD pass so mapped files are read from RAM once instead of twice, combining the separate memmem passes on .h headers into one multi-pattern scan, and merging the producer and consumer pools into a single worker pool.

I also tested a pure 64-byte block state machine that did not split lines at all, covering 45 C- and Java-style languages while keeping the line lexer for the rest. It matched every file count and cut instructions on CPython (-14.9%) and the JDK (-5.4%), but ran 5.7% more instructions on the Linux kernel because / characters inside C block comments kept triggering per-block walks where the line lexer jumps straight to */. Skipping blocks inside open comments would fix part of that, though maintaining a second 660-line parser for 45 languages was not worth it for a win that depended on the codebase.

5. Counting fast means nothing if the counts are wrong#

How the faster lexer also fixed subtle bugs#

Because the old parser gathered all delimiters on a line before tracking state and tried to untangle overlaps afterward, tricky combinations could slip through. A small fuzzer generating random mixes of /, *, ", x, and spaces against a 20-line reference lexer found /* x /*/ in C within minutes, an overlap bug that caused the rest of the file to be counted as a block comment and had gone unnoticed across 378,000 real files.

The table-driven lexer reads left to right with explicit state, so once /* opens a comment, its * cannot be reused as the start of */, and the entire overlap-resolution step disappears. Knowing the current state while scanning also made context-sensitive rules easy to support, such as distinguishing JavaScript regex literals from division, lexing {...} code expressions inside interpolated strings, and restricting HTML quotes to tags.

All 80+ supported languages use this same engine. Each language is defined in a plain-text config file (comment pairs, string delimiters, escape rules, nesting, regex contexts, interpolation delimiters) that compiles into transition tables on first use, without per-language branches in the parser. Higher-level features like keyword counting, inline test detection, and nested languages (<script>/<style> in HTML, Vue, and Svelte, or HTML around PHP) simply read the clean code byte ranges returned for each line, so each feature is implemented once.

What accuracy costs in speed#

Supporting those edge cases costs real CPU time. A counter can always shave milliseconds by ignoring regex literals (/a/*b/), skipping {...} expressions inside interpolated strings ("a ${b /* c */} d"), treating a trailing \ as a normal character, or counting struct and class by raw word matches.

In 3.5.0, the fixes for regex literals, string interpolation holes, HTML tag quotes, and PHP page boundaries added 2.3% to the kernel's instruction count. Checking that struct or class actually starts a type definition adds another 2.6% to 2.9% on default runs, which on the kernel's .c files is the difference between 1.2 million occurrences of the word struct and the 33,589 structs actually defined.

Conformance testing with LineJudge#

Every change runs in CI against LineJudge, a conformance suite I built under loc-conformance (version 0.4.0 has 107 tricky edge-case files across 8 structural categories). Even though I wrote both tools, LineJudge does not test counters against mezura's numbers or enforce a single "correct" line count. Each test file only marks the exact byte spans of strings and comments according to the language specification, and each counter is checked against a small declaration of its own counting rules. Two counters that define blank or code lines differently can output different numbers on the same file and both pass. mezura 3.5.0 passes 107/107 cases, and a follow-up post will cover how that rule engine works.

For JavaScript regex literals, I also checked mezura against acorn 8.15.0 across 38,433 real JS files. It matched 48,070 of acorn's 48,114 regexes at the exact same byte offsets, and the remaining 44 did not change a single line count.