Auto-bench — Auto-bench

This page covers what happens on your server and what it needs. Everything on the website side (auto-bench targets, schedules, the history of runs) is managed from your voxelbench.com dashboard.

How It Works

When a run is due, a VoxelBench bot (a Minecraft: Java Edition client with no screen) connects to your server like any player.
It types /vbautobot <code> <mode>. The code is a one-time code that voxelbench.com generated for this run.
VoxelBench asks voxelbench.com whether the code is valid for this server. Until the answer is yes, nothing happens on the server. The answer also says what to run (mode, test or profile, warm-up, number of runs), and VoxelBench uses only that, never what the bot typed.
VoxelBench switches the bot to spectator mode and runs the benchmark as it would for a player, without the pre-flight screen. It reports its progress to the bot in chat, on lines that start with [AutoBench].
The results go to voxelbench.com, and the bot disconnects.

Requirements

On voxelbench.com

The server is added to your account and verified (see Server Verification). A target cannot be created for an unverified server.
An auto-bench target is set up for it on the dashboard, with its mode and schedule.

On Your Server

VoxelBench, official and up to date, on the server the bot joins. voxelbench.com only accepts the VoxelBench jars it knows: a rebuilt or modified jar is refused (unauthorized_plugin). Download it from voxelbench.com or GitHub Releases.
Outgoing HTTPS to voxelbench.com. VoxelBench checks each code and sends the results itself. /bench ping diagnoses the connection.
The same server identity as when you verified it. A code only works on the server it was issued for, recognised by the identity stored in plugins/VoxelBench/identity.yml (see Privacy). Keep that file when you move or reinstall the server.
A linked server for test jobs (see Account Linking): the result of a single test is stored on your account, which needs the server's token. The other modes do not need it.
For custom_profile jobs, the profile on this server (see Modes).
A pinned benchmark world, strongly recommended (see Where Runs Take Place).

An auto-bench run loads the server like a benchmark you start yourself: players online will feel it. Schedule it for quiet hours.

Letting the Bot In

The bot connects from the internet, like a player. Whatever keeps a player out keeps the bot out, and the run then fails before VoxelBench is even involved:

Whitelist: if it is on, allow the account the bot connects with.
Online mode: an online-mode=true server only admits genuine Minecraft accounts, so the bot must sign in with one. How it signs in is set on voxelbench.com.
Anti-bot, captcha and login plugins, command filters: they must let the bot join and run /vbautobot, and must not hide VoxelBench's messages to it.
Full server, firewall, geo-blocking: leave the bot a slot and let its connection through.
Proxy (BungeeCord, Velocity): the bot goes through the proxy like any player. It must end up on the backend server where VoxelBench is installed and that you verified, not in a lobby.

Modes

The mode is chosen on voxelbench.com, for each target.

Mode	What runs	What is sent
`standard`	The full benchmark, as with `/bench start`	A benchmark report for each measured run
`custom_profile`	A benchmark profile from `plugins/VoxelBench/custom_benchmarks/`, named in the target's arguments by its file name without `.yml`	A custom profile report, even if the profile says `submit: false`
`stresslimit`	The full stress limit, every stress type, as with `/bench stresslimit`	A stress limit report
`test`	One test, given in the target's arguments as for `/bench test`: the test ID, then its parameters in order, for example `disk 4 8 512M 3`	The test result, on the linked account

A custom_profile job only accepts a benchmark profile, not a stress profile (see Custom Profiles). A test job accepts the parameters listed in Commands.

Warm-up and Multiple Runs

For standard and custom_profile, the target can ask for:

a warm-up: a full standard benchmark first, in the same zones, whose results are discarded and not sent. The first measured run starts 3 seconds after it ends.
several runs, in standard only: voxelbench.com refuses a target of any other mode with more than one run (up to 10 runs on Pro, 20 on Enterprise or a hosting provider account). Each measured run is sent as its own report. The runs reuse the same world and zones, with 10 seconds between two runs. In between, in a voxelbench_* world only, VoxelBench deletes the region files around the test zones so that each run starts from freshly generated terrain (see Where Runs Take Place). If a run fails, including when its report is refused, the remaining runs are cancelled, and voxelbench.com counts the job as a failure of your server (unlike a run stopped with /bench stop, see Stopping a Run).

stresslimit accepts a warm-up (a standard benchmark in the same world) but always makes a single run. test ignores both.

Where Runs Take Place

Auto-bench follows the same world rules as the commands:

standard, custom_profile and stresslimit run in the pinned world. Without one, they create a temporary flat world when benchmark.auto-temp-world allows it (a custom profile's own auto-temp-world option can turn it off, not on) and delete it at the end. If that option is off, or the world cannot be created, they run in the server's main world.
test runs a test that writes to the world (every gameplay test except worldSave) in the pinned world, otherwise in a temporary world, never in the bot's world or the main world. With neither, the job fails with benchmark_world_unavailable. The hardware and CPU tests and worldSave are not concerned (see Benchmarks).
On Folia, no world can be created while the server runs: pin a world that the server loads at startup.

A job with several runs resets the regions around its test zones between runs only in a voxelbench_* world. If it runs anywhere else (a pinned world with another name, or the main world it fell back to), it deletes nothing and the runs reuse the chunks already generated. VoxelBench 2.0.2 and earlier deleted them in whichever world the job ran in, the main world included: update before letting voxelbench.com run multi-run jobs on your server.

To keep auto-bench out of your main world, pin a world:

/bench world create bench
/bench world set voxelbench_bench

The pinned world must be loaded when the run starts; if it is not, VoxelBench warns and ignores the pin. Worlds created with /bench world create are loaded again at every startup. See Benchmark Worlds and Configuration.

What the Bot Can and Cannot Do

The bot is an ordinary player account. VoxelBench gives it no permission and no operator status: it only has what your server gives every player.

It can:

type /vbautobot, which needs no permission;
once voxelbench.com has confirmed its code, have VoxelBench run what the job asks for. VoxelBench puts it in spectator mode and moves it to the test zones, as it does for a player running /bench start. The run skips the pre-flight screen and the local cooldown, and does not start a cooldown for your own runs. A test job does not need the test's permission node.

It cannot:

start anything without a code that voxelbench.com confirms for this server;
choose what runs: the mode, test, profile, warm-up and number of runs come from voxelbench.com, not from what the bot types;
reuse a code: each code serves one run of one job;
start while another benchmark, stress run or test is in progress (test_already_running);
stop a run it did not start: /vbautobot stop only accepts a code of the job in progress.

During an auto-bench run, VoxelBench shows no test scoreboard, so it is not pushed to another player who is online.

The `/vbautobot` Command

/vbautobot is the bot's entry point. It has no permission node, so any player can type it and it may show in command suggestions. Without a valid code it only answers with an [AutoBench] refusal line, such as usage-error or validate-fail | code=invalid_nonce. /vbautobot stop <code> is the bot's way of cancelling its own run. With any other code it is ignored without a reply, and while a job is running the console logs the refusal.

You never need to type it. /bench autobot is the same entry point, for manual testing only, and requires voxelbench.use. See Commands.

Stopping a Run

From voxelbench.com: cancelling a run makes the bot send /vbautobot stop <code> and disconnect. VoxelBench stops the run at once (Auto-bench stop accepted for job … in the console).
From the server: /bench stop (permission voxelbench.stop) stops an auto-bench run like any other, whatever its mode, and tells the bot immediately. voxelbench.com records it as stopped on the server, not as a failure of your server.
If the bot leaves (kick, network loss), VoxelBench notices at the next test that needs a player, waits 30 seconds for it to come back, then abandons the run.
If the server stops, the run is lost: VoxelBench keeps nothing about the job across a restart. A temporary world left behind is deleted at the next startup.

A stopped or abandoned run sends no report, and the remaining runs of the job are not started.

Where Results Go

standard, custom_profile, stresslimit: each measured run is sent to voxelbench.com as a report attached to the job, which the dashboard links to. The warm-up sends nothing.
test: the result is stored privately on the account the server is linked to, attached to the job. reports.backend.unit-tests does not matter here.
Locally: as for runs you start yourself, a copy is written under plugins/VoxelBench/reports/ according to reports.storage (see Configuration and Reports).

Reports follow your anonymization level. Like any scored run, an auto-bench run pauses the profiler and memory inspection (see Profiling).

Troubleshooting

voxelbench.com shows why each run failed. Connection failures (connection refused, kicked on join, unknown Minecraft version) happen before VoxelBench is involved: see Letting the Bot In. If the bot joined but VoxelBench never answered, VoxelBench is not installed on the server the bot reached, or another plugin blocked /vbautobot.

Error Codes

These codes come from VoxelBench. The dashboard shows most of them under a code of its own, prefixed with plugin_: for example, unknown_test and test_build_failed appear as plugin_unknown_test, and network as plugin_network_error. Its raw message names VoxelBench's code.

Code	Meaning	What to do
`benchmark_world_unavailable`	A `test` job on a test that writes to the world, with no pinned world and no possible temporary world (`auto-temp-world: false`, creation failed, or Folia). Nothing ran.	Pin a world loaded at startup with `/bench world set <world>`, or turn `benchmark.auto-temp-world` back on (not enough on Folia).
`unauthorized_plugin`	voxelbench.com does not recognise this VoxelBench jar. With `stage=challenge`, it was refused at the very first request.	Install the latest official release, unmodified, and restart.
`server_mismatch`	The code was issued for another server identity.	Make sure the bot lands on the server you verified. If this server's identity changed, verify it again and update the target.
`network`	VoxelBench could not reach voxelbench.com, even after retrying.	Run `/bench ping` and allow outgoing HTTPS.
`rate_limited`	voxelbench.com refused the challenge that starts every signed request: it hands out at most 150 per hour to one IP address, shared by every server behind that address. Nothing ran.	Wait, then run again. If it keeps happening, look for other servers or repeated benchmarks sending from the same address.
`invalid_nonce`, `nonce_expired`, `nonce_consumed`, `target_missing`	The code is unknown, expired or already used, or the target was deleted.	Nothing to fix on the server: start a new run. If `invalid_nonce` keeps coming back, check that no plugin rewrites the bot's commands.
`test_already_running`	A benchmark, stress run or test was already running.	Avoid running your own benchmarks when an auto-bench is due; `/bench status` shows what is running.
`missing_profile_name`, `unknown_profile`	`custom_profile` job with no profile name, or with no benchmark profile of that name on this server (the line lists the known ones).	Check the file name in `plugins/VoxelBench/custom_benchmarks/` with `/bench custom list`; run `/bench custom reload` after adding a file.
`test_no_longer_known`	A test of the profile is no longer registered, usually because its extension was removed.	Reinstall the extension, or edit the profile.
`missing_test_args`, `unknown_test`, `invalid_test_args`	`test` job with no test, an unknown test ID, or a parameter out of range or not a number.	Fix the target's arguments: the test ID, then the parameters in the order `/bench test` takes them.
`test_build_failed`	The test is registered, but its extension could not create it.	Check the console and the extension.
`unknown_mode`	This version of VoxelBench does not know the mode.	Update VoxelBench.
`exception`	VoxelBench hit an error while preparing the run.	Look for the error in the console and report it.
`invalid_json`, `validation_error`, `invalid_response_body`, `http_<status>`, `nonces_mismatch`, `nonces_size_mismatch`	VoxelBench and voxelbench.com did not understand each other, or voxelbench.com kept answering with an error.	Update VoxelBench and run again; report it if it persists.
`report-fail` with `requires a linked account`	A `test` job on a server that is not linked.	Link the server with `/bench link`.
`run-fail`, then `multi-aborted` with `reason=run_failed`	A run of a multi-run job ended without its report: refused by voxelbench.com or not sent. The remaining runs are cancelled, and the dashboard records a failure (`agent_crash`, its raw message starting with `run-fail` and the reason).	Read the reason on the `run-fail` line; update VoxelBench and check the connection with `/bench ping`.

In the Console

On a release build, the [AutoBench] lines only go to the bot's chat. The console logs the main steps:

Auto-bench job <id> validated for mode=standard warmup=false runs=1
Auto-bench validate-nonce rejected: <code>
Auto-bench validate-nonce transient failure (<reason>), retrying in <ms>ms (attempt <n>/<max>)
Auto-bench stop accepted for job <id> (mode <mode>) — aborting run <k>/<n>
Auto-bench stop REFUSED for job <id>: nonce mismatch (from <player>)
Report submission failed: <reason>

The usual lines of a benchmark run follow, such as the creation and deletion of a temporary world.