iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
ABTestly reports three different p75 times for one specific event: a test variation appearing in the document object model (DOM). Its figures—2.2 ms, 19.9 ms and 529 ms—come from a single-page-app route change, a warm-cache repeat visit and a throttled, empty-cache first visit, respectively. They are not three interchangeable scores of overall runtime speed, and the benchmark is a vendor-reported test rather than an independent verification.
What the three numbers measure
In a post dated September 26, 2026, ABTestly says it measured the time from navigation start until a variation landed in the DOM. The company reports the 75th-percentile (p75) result across 200 page loads for each condition:
| Test condition | Reported p75 to DOM | What the condition means |
|---|---|---|
| Single-page app (SPA) route change | 2.2 ms | A route change within an already-running single-page app. |
| Repeat view, warm cache | 19.9 ms | A repeat view with cached resources. |
| First visit, empty cache, throttled | 529 ms | A first visit with an empty cache and a constrained network. |
All three figures are ABTestly’s reported results, not measurements replicated independently. Each is a p75 for the stated test condition; none is a universal speed score. The measured endpoint is DOM insertion or change, not the moment a visitor sees the changed content.
How the test was run—and what it leaves out
ABTestly says it tested its built runtime in headless Chromium against a local origin. The variation changed one above-the-fold heading. For the throttled first-visit condition, the network was limited to 1.6 Mbps downstream with a 150 ms round trip. The company reports 200 page loads in each condition.
#1 Best Overall
- Local origin: The test included the stated network throttling and transfer, but not the DNS lookup, TLS connection setup or edge latency that a real first visit can involve.
- Unthrottled processor: ABTestly constrained the network, not CPU speed.
- One simple change: Rewriting a heading does not establish timing for more complex variations, different page structures, browsers, networks or devices.
- No independent replication: The available figures are the vendor’s account of its own test.
ABTestly characterizes the 529 ms result as a floor rather than a field result and says a first visit on a mid-range phone would be slower. Because the test does not model real-world connection setup or constrain the processor, that figure should not be read as a prediction of what visitors will experience.
DOM arrival is not the same as what visitors see
A variation can be present in the DOM before a browser paints it, and the original content can be painted before the variation takes its place. ABTestly reports that in its throttled first-visit condition, the original heading appeared on screen first in all 200 loads, with a median of 326 ms before replacement.
Rank #2
The source reports a different pattern in the other conditions: on a cached repeat view, no painted frame showed the original heading in 199 of 200 loads; on the SPA route change, none did. These are observations about that heading and those test conditions—not proof that original content will or will not flash on other sites or experiments.
The anti-flicker tradeoff
ABTestly says its runtime loads as a dynamic script and does not block the HTML parser. The company offers an optional anti-flicker setting, unchecked by default, that hides the page with an opacity rule until variants apply or a two-second timeout expires. Its stated rationale is that hiding the whole page can delay visibility for visitors who are not assigned to an experiment and can leave the site looking empty if configuration is slow, whereas flicker affects pages actually changed by a variant. That is the vendor’s design rationale, not an independently tested comparison of outcomes.
Why these times should not be compared with LCP
Largest Contentful Paint (LCP) measures when the largest visible image, text block or video is rendered relative to navigation. It is a user-facing page-load metric, not the same endpoint as the time a variation lands in the DOM. A fast DOM change does not by itself show that the changed content was painted quickly, nor does it describe the page’s overall loading experience.
For general performance context, web.dev’s LCP guidance sets a good p75 target at 2.5 seconds or less and recommends segmenting page loads by mobile and desktop. It also explains that connection setup, redirects and time to first byte (TTFB) can materially affect field results. That guidance is context for evaluating a live page; it does not validate ABTestly’s benchmark or make its DOM times equivalent to LCP.
What the speed guardrail can—and cannot—tell you
ABTestly says its speed guardrail flags a variation when its p75 LCP is at least 400 ms above control. A row is evaluated only after passing these checks, stopping at the first failure:
Free tools Windows power users keep installed
One-click scans. No signup required.
- The capture rate is not above 100%.
- Each arm has at least 100 page loads.
- The capture-rate gap between arms is no more than 20 percentage points.
- Capture is at least 50% in each arm.
The post is explicit about the limits: “There is no confidence interval, no bootstrap, and no significance test. The panel is a descriptive guardrail, not an inferential one.” A 400 ms difference based on 100 loads per arm is not equivalent evidence to the same difference based on 100,000, the post notes. The panel displays load count beside a row but does not weight the evidence for the reader.
It also does not correct for multiple device and variation comparisons. ABTestly’s example has four variants across two devices, creating six comparisons, each assessed against its own threshold. A row that is not flagged therefore means only that no visible threshold crossing occurred at the displayed volume; it does not prove the variation is safe or that it caused no performance harm.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Questions to ask when evaluating a runtime benchmark
- What exact event is timed: script download, variation applied to the DOM, or changed content painted on screen?
- Are results separated by SPA navigation, warm-cache repeat visits and cold-cache first visits?
- Was the test run on a local origin or a deployed site? Were network and processor conditions both constrained?
- How often did visitors see the original content before a variation replaced it?
- How many page loads and what capture rates support each reported guardrail?
- Are device and variation comparisons shown separately, and is the reporting descriptive or inferential?
- Which field metrics, such as LCP, are reported alongside the runtime-specific endpoint?
Those questions help separate a narrowly defined lab result from the experience of visitors on a live site. The benchmark is useful as a description of ABTestly’s reported runtime behavior under its stated conditions; it is not enough, by itself, to rank vendors or predict performance across real-world pages and devices.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
