Probability-to-win and expected-loss language feels closer to business decisions.
Stats engine
Make decisions defensible with Bayesian, frequentist and SRM checks.
Testvoy combines probability, confidence, SRM and revenue context instead of flattening every result into one metric.
Product layerStats engine
Module controlPrimary signalBayesian decision view, classic validation and sample-ratio checks.
This module connects setup, reporting and decision output inside the Testvoy workflow.
GuardrailIf SRM is not clean, the decision should not be rushed
MethodologyDetails link to the public methodology page
StakeholderReports stay readable for non-technical teams
01BuildProbability-to-win
02ValidateFrequentist validation
03DecideMinimum sample and exposure context
Statistics engine
Do not trust one shiny score. Read the same data through three decision lenses.
Testvoy Stats Engine interprets experiment data through Bayesian decision probability, Frequentist validation and SRM quality checks. The goal is not to overstate a result; it is to clarify whether the winning variant is defensible, where decision risk rises and how the team should explain the outcome.
p-values and confidence intervals keep the validation frame many teams already use.
When traffic allocation looks unhealthy, users are warned before rushing the decision.
Lift can be read alongside funnel and revenue context.
Product capabilities
The core job Testvoy solves with this module.
This section shows what the module helps teams do, which risks it reduces and which decision it makes easier to defend.
Back to product hub01
Transparent decision language
Teams can read both the winner and the quality of the decision.
- Probability-to-win
- Frequentist validation
- Minimum sample and exposure context
02
Reduce false confidence
SRM, bot filtering and segment views are checked before a result is defended.
- Sample ratio mismatch
- Bot and anomaly signals
- Funnel/revenue impact
Where teams use it
Built for the experiment and optimization workflows teams run every week.
Defensible executive decisions
Explain results with probability, confidence interval and risk signals instead of saying only 'B won'.
Reduce early decision risk
Pause when sample, exposure or SRM signals are not clean enough for rollout.
Align CRO and data teams
Use Bayesian language for business teams and Frequentist validation for data teams in one report.
Decision path
The role this module plays in the experiment program.
Transparent decision language
Teams can read both the winner and the quality of the decision.
Probability-to-win
Frequentist validation
Reduce false confidence
SRM, bot filtering and segment views are checked before a result is defended.
Guardrail
If SRM is not clean, the decision should not be rushed
How it works
Clearer setup, cleaner signals, faster decisions.
Testvoy positions this module to help teams move from experiment idea to report with the same data, context and decision language.
Bayesian reporting makes decision language easier
Bayesian analysis can express win probability and expected loss in language business teams can discuss. Testvoy does not present it as a standalone truth; it shows it alongside other controls.
- Probability-to-win improves decision readability
- Expected loss makes early rollout risk visible
- Results are read with business impact and segment context
Frequentist validation keeps the familiar frame
Many teams still operate with p-values, confidence intervals and minimum sample logic. Testvoy keeps those signals visible next to the Bayesian decision surface.
- p-value is not turned into a standalone marketing claim
- Confidence interval shows the range of lift more clearly
- Minimum sample and exposure context stays in the report
SRM quality checks help catch false confidence
Sample Ratio Mismatch checks whether traffic allocation drifted away from what was expected. If the signal is not clean, the issue may be assignment, bots or event quality before variant performance.
- Variant allocation is compared with expected ratios
- Bias risk becomes visible before a decision
- Bot filtering and funnel reporting can be read alongside SRM
FAQ
Fast answers before the decision.
Why show Bayesian and Frequentist results together?
One improves decision probability language, the other preserves a familiar validation frame. Together they create a more defensible result story.
What does an SRM warning mean?
Traffic allocation may have drifted from the expected ratio, so assignment, bot, SDK or event quality should be checked before deciding.
Can Testvoy claim it is always more accurate than competitors?
Instead of unsupported absolute claims, Testvoy focuses on multi-method statistics and quality checks that make decision risk more visible.
Bayesian decision view, classic validation and sample-ratio checks.
Details link to the public methodology page