VoxelBench 2.6.4

Scores that hold from one run to the next, and reports that say why when they don't

VoxelBench 2.6.4 is the first release announced since 2.3.0, and it brings everything 2.4.0, 2.5.0, 2.5.1, 2.6.0, 2.6.1, 2.6.2 and 2.6.3 changed. Every full benchmark and stress limit report now says in what conditions it ran, down to each test: what the host and its neighbours were doing, why a slow tick was slow, which settings shape the tests. voxelbench.com's Single-Core score is now computed from the main thread's own time: measurements that vary by 1 to 9 % from one run to the next, where the previous score varied by 7 to 23 %. The calibration probe that tells a steady machine from a disturbed one waits for the server to calm down, even after it has generated chunks, measures for a second, and is judged against the host's own noise. The memory checks no longer cost a report its score over a heap that was only waiting to be collected, and a finished run no longer stays in memory. Folia reports the ticks of each test's region, and Minecraft 26.3 is supported.

Most of this is measured next to the existing values. How voxelbench.com scores a report changes in two places, the Single-Core sub-score and the stability multiplier of the total, and tests that the memory checks used to skip now run.

Reports that say in what conditions they ran

The same server can score 55,000 one day and 89,000 the next, and nothing in a 2.3.0 report said whether the host, its neighbours or the benchmark had changed. Full benchmark and stress limit reports, multi-run and auto-bench runs included, now carry the conditions of the run. VoxelBench reports facts; voxelbench.com reads them, flags a run it cannot trust, and compares only runs made in the same configuration.

Isolation: the CPU quota, the cores the server may use, the memory limit, the host's CPUs and memory, and the filesystem the server folder is on, with network storage flagged. The absence of a limit is stated positively, so a host without limits can be told from one whose files could not be read. When the memory figures found are the whole machine's rather than the server's own, they are no longer sent as a container's: one report had shown a 511 GB "container peak".
Contention, for the run and for each test: CPU time stolen by the hypervisor, time the CPU quota held the server back and in how many of its periods, CPU, disk and memory pressure, the machine's load, the CPU quota read again during each test, the container's memory, swap, memory-limit events and out-of-memory kills, and the server's own resident memory.
Why a tick was slow: on Paper and Spigot, each test counts its slow ticks (at least twice its median tick and at least 50 ms) and how many of them a garbage-collection pause, the CPU quota or stolen CPU time explain. A score that dropped because of a busy neighbour is not the same as one that dropped because of Java's collector.
The server when the run started: how long Java had been running, how many benchmark runs it had completed since, and what else was loaded or online outside the benchmark world (worlds, chunks, entities, other players; the player running the benchmark is not counted), plus, for most tests, the heap right after the collection they make before measuring.
A machine that changed: a fingerprint of the machine's boot, mixed with a random value created for your server and never sent, says whether the machine rebooted or the server moved since the previous report. It does not identify the machine, and two servers on one machine report different fingerprints.
The disk test's cache check says whether its cache bypass really happened, and why it has no result when it has none (no bypass on this filesystem, storage too slow, read error…).

Most figures are read between tests, off the server's main thread. Throttling and stolen time come from a small background thread that reads two Linux counters every 40 ms during each test: about 0.3 % of one core on a container limited to one core, and every report from a Linux host carries the share it took. A value that cannot be read is left out rather than guessed: on Windows and macOS, only the calibration remains. One console line sums up each run, for example Run conditions: steal 1.8 %, throttled 4.3 s, CPU pressure 22.0 %, calibration drift 3.1 %, and logging.test-verbosity: DEBUG prints every figure.

A Single-Core score that holds from one run to the next

The redstone circuit repeats every eight ticks, and TPS and MSPT sampled every 200 ms kept falling on the same phases of that cycle: on one machine holding 20 TPS in every run, the old Single-Core score ranged from 63,000 to 116,000. Since 2.6.1, VoxelBench measures the time the main thread really spends: the redstone's CPU time per cycle of eight ticks, the block-physics test's CPU time per tick, and the single-core test's operations per second of CPU time. Since voxelbench.com's Standard scoring 8.2.0, the Single-Core sub-score is computed from those three. From one run to the next they vary by 2 to 5 % (redstone), 1 to 3 % (block physics) and 5 to 9 % (CPU test), against 7 to 23 % for the previous Single-Core score.

The single-core test now counts, with a quarter of the Single-Core weight. The Scores Dashboard and Last Results screens of /bench gui list it under Single-Core; network still counts in no score.
A new scale. The Single-Core, synergy and total scores of these reports do not compare with those of older reports; Hardware and Gameplay scores do. Reports from plugins before 2.6.1, and from Folia, keep the previous scoring.
A stability multiplier that no longer reads the redstone. The total is adjusted by how steady TPS and MSPT stayed during the gameplay tests. The redstone's sampled values said nothing about the server, for the reason above, so Standard scoring 8.3.0 leaves that test out and turns the multiplier's steps into a continuous curve: a few hundredths of deviation no longer make two runs of the same fast server differ by 2 %. On 31 real runs, the change moved totals by −0.3 to +8.7 %; reports scored with 8.2.0 are recalculated automatically.
The redstone's effective TPS, the rate its circuit really ran at, is shown on the report; no score reads it. On servers whose sampled TPS read 17 to 20 during the redstone test, it reads 7 to 14.
What the main thread really got: the single-core test also reports the share of its compute slices the main thread was actually running (below 1: the CPU quota, neighbours, the hypervisor or a pause of Java; not on Windows, whose thread clock is too coarse), its throughput second by second, the garbage collections during the test and, on Linux hosts that expose it, the current CPU frequency. The redstone test counts slow ticks against the heaviest tick of a typical cycle, so the circuit's own work is no longer counted as spikes.

A calibration probe that judges itself

A small computation is timed at the start, in the middle and at the end of a run, and before every test: when the machine slows down, it shows there. voxelbench.com judges a run's stability test by test, so a run slowed during a few tests still counts for the scores those tests do not feed. Three changes make that verdict fairer.

It waits for the server to calm down. A point taken while the server's own threads were still busy with the previous test's chunks or garbage collection looked like a busy neighbour. VoxelBench now measures once two half-second windows in a row stay under a quarter of the CPU quota (after 8 s it measures anyway, and says so), and each point says how many of its samples the CPU quota held back. On a container limited to one core, the point after the chunk-loading test dropped from a median of 87 ms to 16–21 ms.
It waits longer after chunk generation. Once the chunk-loading test is over, the server keeps saving the chunks it generated and collecting the garbage that leaves, about 20 seconds of CPU time in our measurements. The point that follows now waits up to 30 s instead of 8: on two cores shared with a neighbour, it measured a calm server after 14 s, where an 8 s wait had measured a busy one.
voxelbench.com no longer judges a point measured on a busy server. When a point had to measure at the end of its wait while the server's own threads were still busy, the site leaves it out and judges the tests on either side on their other point. The report shows how long the server took to calm down after chunk loading: about 2 s on a calm host, 14 s on shared cores in our tests.
It measures for a second: 40 runs of the computation instead of 5, with the point's own spread. A tenth of a second was a snapshot of a host whose speed moves from second to second.
voxelbench.com judges each point on its median, against the host's own noise. On one shared host, the 60-second tests varied by less than 2 % from one run to the next, yet its three 2.6.1 runs were left out of every comparison: the best time of five samples moved from 22.6 to 31.6 ms between points. A host that is only noisy is no longer read as a disturbed run; a real slowdown still is.

The cost is paid between tests, never during one. In our measurements, the calibration takes 16 to 18 s of a standard benchmark, against about 2 s in 2.6.1: 14 to 16 s more, 1.5 % of a 16-minute run. The wait for calm adds 3 to 5 s under a two-core limit, about 3 s without limits and 14 to 16 s under a one-core limit. After chunk loading, a server still busy can add up to 22 s more: 6.4 s in our run on shared cores, nothing with two cores of its own.

No more tests lost to garbage

A report missing a required test gets no score, and two memory checks could cost one over a heap that was only waiting to be collected. Typically, the serial collector's old generation fills up with garbage between full collections, and a server started with -XX:+DisableExplicitGC ignores the collections VoxelBench asks for.

Before each test, a heap projected at or above 85 % no longer skips the test on the spot. VoxelBench waits up to 30 seconds, reads the heap again after a new collection, and starts the test as soon as there is room. On one Paper server with a 6 GB heap, three standard benchmarks out of five had lost two gameplay tests to this check. In our tests a wait lasted 4 to 5 s, outside the test's measurement.
During a test, the watchdog that stops a test whose heap stays above 92 % for 3 seconds now runs one full collection first, and stops the test only if the heap, or its old generation, is still that full. On a production server (6 GB heap, 600 MB young generation, explicit collections disabled), it had stopped lightingUpdate at 92.8 % and the report got no score. With the same memory settings and the threshold lowered to 75 % to trigger it, 2.6.1 stopped a test at 75.2 %; 2.6.2 found the heap at 11.4 % after the collection, and the same test completed.
When the server ignores explicit collections, both checks ask Java for one through its diagnostic command, the one jcmd <pid> GC.run uses, and the report says so.
A heap that is really full is still stopped, one collection later. A collection during a test pauses the whole server: the report records it, and voxelbench.com keeps the report's score but treats that test as disturbed.
A finished run no longer stays in memory. Since 1.0.0, the single-core test left its measurement timer running after the test, and that timer kept the whole finished run alive: its tests, its results and the entities the tests had removed, about 57 MB per run until the server restarted. On a server with a 3 GB heap, five benchmarks in a row left the live heap at 511 MB instead of 262 MB; on a production server with a 4.8 GB container, the fourth and fifth runs of a session lost tests to the memory checks. Now the live heap stays flat: 262, 283, 291, 296 and 296 MB over the same five runs.

The settings that shape the tests

Two servers on the same hardware can score differently because of one line in spigot.yml. Reports now include the settings that influence the gameplay tests, from spigot.yml, Paper's configuration and purpur.yml: entity activation and tracking ranges and when inactive entities wake up, merge radii, hoppers, explosions, redstone, collisions, auto-save, the chunk system's threads, Purpur's villager and fluid settings. They include the overrides of the world the run used (world-settings.<world>, its own paper-world.yml; the world's name is not sent) and the garbage collector really in use: without an explicit option Java picks one itself, Serial under two CPUs.

A default is given only where it is known for certain, from the files Paper 1.20.4 to 26.3, Folia and Purpur write themselves, and a value a server kept from an older default when it was upgraded is not counted as modified. The in-game configuration profile screen lists the modified settings first.

Every tick of every test

The TPS and MSPT a test reports are sampled every 200 ms and filtered, and they keep their meaning. Next to them, on Paper and Spigot, each test now carries the distribution of every tick of its measured window, unfiltered: the main thread's CPU time per tick, and the real time between the starts of two ticks (median, 90th, 95th and 99th percentiles, worst, and how many ticks started more than 50 and more than 100 ms after the previous one). It costs one stored number per tick on the main thread.

In a run under a two-core limit, the chunk-loading test's sampled median MSPT read 5.1 ms while its median tick took 55.7 ms of CPU and started 189 ms after the previous one: the server ran at about 5 TPS throughout. On the tests that held 20 TPS, the 99th percentile per tick was up to 40 % above the sampled one (block physics: 38.1 against 28.3 ms), because a 200 ms sample averages about four ticks and hides the tail. On a healthy server about half the ticks start a fraction of a millisecond past 50 ms: a late tick shows in the 100 ms count and the percentiles.

Each test also reports its median MSPT next to the 95th and 99th percentiles, and the number of ticks of its measured window with the main thread's CPU time over them: an average tick time that does not depend on sampling.

Folia

The ticks of the test's region. A test whose zones stay in one region during its measurement reports how long each tick of that region took and, for the redstone, per cycle of eight ticks. It is the real time a region thread spent on the tick, pauses and preemption included, not a CPU time. A test spread over several regions reports nothing of the kind: in our Folia run, 8 tests out of 16 kept their zones in one region, the redstone and block-physics tests among them. Folia reports keep the previous Single-Core scoring.
What depends on a single server tick is not measured on Folia: the causes of slow ticks, the per-tick distribution and the main-thread CPU time of a test's window. Chunks and entities are left out of the server's state at the start of a run.

Minecraft 26.3

Paper 26.3 and Folia 26.3 start with VoxelBench, Folia 26.3 runs tests in its benchmark dimension, and a full standard benchmark on Paper 26.3 completed all 16 tests. Folia's region monitoring reads 26.3 as it read 26.2.

Paper 26.2 and 26.3 servers used to report Minecraft 1.21.0, in reports, in the default values of the server settings and in the version checks, which turned Paper's native dialogs off there. They now report 26.2 and 26.3, and the default settings of Paper, Folia and Purpur 26.3 are recorded.

Also fixed

spawn-limit.water-ambient was read from the ambient category on Paper 26.1.2, so every fresh server reported 15 instead of 20 and counted as modified.
The console no longer writes "After GC" when only a minor collection happened.
Memory messages in reports no longer depend on the server's language settings: 87.3%, never 87,3%.

Upgrading from 2.3.0

Drop the new jar in plugins/ and restart. There is no configuration or data migration, no new permission and no new profile, and the memory thresholds (benchmark.memory-budget, benchmark.memory-watchdog) keep their values. On its first benchmark on Linux, VoxelBench creates plugins/VoxelBench/data/host-salt, the random value behind the machine fingerprint; deleting it gives a new fingerprint. The new data is technical only: no file path, world name, host name, address or player name leaves the server, and the machine's boot identifier is never sent.

Except on Folia, your first 2.6.4 benchmark is scored on the new Single-Core scale: compare its Single-Core and total scores with reports made since 2.6.1, not with those of 2.3.0.

The jar runs on Java 16 and later; your Minecraft version sets the minimum: 16 for 1.17, 17 from 1.18, 21 from 1.20.5, 25 for 26.1 to 26.3. Full benchmarks were run on Paper 26.1.2 and 26.3 and on Folia 26.1.2, in a Linux container with and without a CPU limit; VoxelBench also starts on Paper 1.17.1, 1.21 and 26.2, and on Folia 26.2 and 26.3, where it runs tests in the benchmark dimension.