Context before comparison.

Three kinds of evidence.

Measured: observed inference results imported from selected optimization exports. Runs remain available when metadata is missing; unknown revisions and runtime versions are disclosed rather than inferred. These results are not independently validated.

Simulated: synthetic functions or serving examples generated offline. These demonstrate concepts and do not establish runtime performance.

Architectural: descriptions of implemented workflows and boundaries. A workflow description is not a benchmark or live status.

Give every metric a definition.

Output throughput

Output tokens per second over the benchmark’s measurement interval. Preserve token counting and workload details.

p95 E2E latency and p95 TTFT

The imported runs report 95th-percentile request latency and time to first token, in milliseconds. Do not compare these as if they were means.

Mean E2E latency

Mean elapsed time from request initiation to completion, in milliseconds. It is distinct from token generation latency.

Mean TTFT

Mean time to first token, in milliseconds. Do not treat absent or default-zero values as successful measurements.

Compare like with like.

Start with model name, recorded draft model, seed, dataset and workload. Retain known GPU platform/count, revisions, engine versions, request fingerprints, and measurement constraints. Missing metadata does not exclude runs, but limits reproducibility and performance claims. An unreported draft model does not establish that none was used.

A point dominates another if it is no worse on every objective and strictly better on at least one. Selection constraints narrow feasible results; they do not rewrite the original measurements.

Failed and pruned trials are diagnostics, not successful recommendation candidates. A search winner remains unvalidated until an independent validation completes.

Real algorithms, synthetic objectives.

The search lab uses Optuna 4.9.0 TPESampler, GPSampler, and RandomSampler. Each receives 24 trials over x,y ∈ [0,1], with seeds 11 and 29. TPE and GP use six startup trials. GP uses deterministic_objective=true.

The Python generator runs single-threaded on CPU. No inference engine runs. Traces preserve full-precision parameter values and objective observations; JSON includes dependency versions and a checksum.

Branin uses the standard function with x mapped to [−5,10] and y to [0,15]. The second objective combines two smooth basins with sinusoidal variation. Exact formulas are in the evaluation guide and generator.

The inference explorer’s synthetic throughput and latency come from a documented formula applied identically to every engine label. It cannot support engine performance comparisons.

Publication is an explicit step.

Curate an allowlist of source records. Retain all selected runs, including crashed runs and records with missing context. Convert placeholder metadata to explicit unknown values. Dry runs, failed or pruned trials, and invalid metric values do not enter measured performance frontiers. Export only inference metrics and serving settings, preserving source fingerprints. Remove credentials, internal paths, prompts, and private endpoints.

The published run export records output throughput, p95 E2E latency, and p95 TTFT. These percentiles are distinct from the synthetic examples’ mean latency. Some records identify their source as MEASURED_ALLOCATION_WALL_PROXY; the explorer preserves that source label. Missing metadata is never filled with guessed values.