System Benchmarks¶
System-level benchmarks are general indicators of performance for hard problems. However, unlike component-level benchmarks, failure of specific operations are not reported directly.
QUOPS¶
Quantum Universal Operations Performance System (QUOPS) is a system-level benchmark developed by Sandia National Labs. It combines circuit size, accuracy, and execution rate to describe system performance through three outputs: a capability region showing the largest circuits that can be run at different qubit counts; a QUOPS score summarizing circuit-size capability in a specified region; and a QUOPS rate summarizing execution throughput.
Together, these outputs give customers a practical framework for assessing computational capability and tracking progress toward fault-tolerant, utility-scale quantum computing.
QUOPS requires the execution of multiple circuits consisting of alternating \(\mathrm{R}_{p}(\theta)\) and random \(\textrm{CX}\) gate-pairing layers. Each circuit is specified by a width (number of qubits), \(w\); and a size, \(s\), defined as the number of \(\mathrm{R}_{p}(\theta)\) gates plus twice the number of \(\textrm{CX}\) gates. Specified shapes \((w,s)\) are evaluated on machines using mirror-circuit fidelity estimation (MCFE) [1]. Multiple random circuits of the specified shape are run, and circuit mirroring is used to estimate polarization, \(\bar{\gamma}_{w,s}\), (roughly equal to circuit fidelity for large \(w\)). The polarization of a specified \((w,s)\) circuit is compared with a predefined polarization threshold,
Circuits above the threshold with 95% confidence trace out the machine’s capability region on a plot of \(w\) versus \(s\).
The QUOPS score, \(Q\), and QUOPS rate are scalar performance metrics derived from the capability region.
The score is the largest tested circuit size satisfying both of these conditions: it lies within the capability region; and it lies within the admissible cone. The admissible cone restricts which circuit shapes may define the score. Its boundaries are:
The QUOPS rate, \(\Omega\), is defined as:
where \(t\) is the average shot time for the circuit \((w,s)\) that defines the QUOPS score. Further details on the QUOPS test are available in [2].
We report the QUOPS capability region and admissible cone below for commercially available Quantinuum quantum computers. The QUOPS score is visualised here and reported in the table below.
Interpretation of results
QUOPS scores describe performance under the benchmark’s test conditions. Comparisons across machines or over time require the measurement dates, test configurations, and treatment of post-selection to be reported alongside the scores and rates.
Machine |
QUOPS score |
QUOPS Rate |
|---|---|---|
H2-1 |
1320 |
353 |
H2-2 |
1008 |
231 |
Helios-1 |
1504 |
303 |
Experimentally measured QUOPS capability region for H2-1.¶
Experimentally measured QUOPS capability region for H2-2¶
Experimentally measured QUOPS Capability Region plot for Helios-1¶
Quantum Volume¶
The quantum volume (QV) test is a full system-level benchmark for quantum computers that is sensitive to qubit number, fidelity, connectivity, and other quantities believed to be important in building useful devices [3]. QV requires time-consuming and extensive classical processing.
Quantum volume prescribes a set of test circuits that involve pairing of \(n_q\) qubits, applying a 2-qubit gate randomly selected from \(SU(4)\), and repeating this process for \(n_q\) rounds. Each circuit is run on the quantum computer and then compared to a full statevector classical simulation of the ideal circuit with a heavy output probability (HOP) test. The process is repeated with at least 100 circuits and if the average HOP over the set of circuits is above 2/3 with two-sigma confidence the test is passed and the system has quantum volume = \(2^{n_q}\).
GHZ State Fidelity¶
Greenberger–Horne–Zeilinger (GHZ) State Fidelity Estimation measures the ability of a system to create large, entangled states. We use the logarithmic-depth GHZ state preparation protocol and estimate the state fidelity by measuring populations and coherences of the state [4][5]. A measured state fidelity greater than 0.5 indicates the state has genuine n-body entanglement. In our benchmark we do not use any SPAM correction.
The data from H2-1 is plotted below for 56 qubits giving a fidelity of \(0.6156\pm0.0082\). The population measurement is ideally 50:50 split between \(|0 \rangle^{\bigotimes N}\) and \(|1 \rangle^{\bigotimes N}\) states. On the H2-1 device, the probability of measuring a different state is 0.2. The probability of measuring the all-ones state is worse than all-zeros state due to asymmetry in spam error. The parity measurement quantifies the degree of superposition and ideally oscillates from 1 to -1 with period equal to \(\frac{\pi}{N}\) for \(N\) qubits. A parity of zero implies no superposition in the GHZ state.
A discussion of the GHZ state is available here.
