...
If preezie is embedded on a single page then this is straight forward:
Split traffic | Page trigger | Segment | Goal A: User conversion rate (sales per user) | Goal B: Sessions per user | Goal C: Bounce rate |
|---|---|---|---|---|---|
50% Test | myhomepage.com | 40% non-preezie users | 3.2% | 2.4 | 23% |
10% preezie users | 4.5% | 3.2 | 0% (preezie counts as a significant event) | ||
50% Control | myhomepage.com | 50% non-preezie users | 3.1% | 2.6 | 25% |
Once you can see these the buckets are gaining a good level of traffic (e.g. use a an A/A test to understand how long your website traffic needs to acheive achieve even conversion rates), you can start to compare the preezie 10% against the 50% who never saw it and the 40% who did see it but didn’t engage.
Control
1000 users / 31 sales = 3.1% conv rate
Test
1000 users / 37 sales = 3.5% (+12% against control)
non-preezie @3.2% conversion (+3% against control) = 26 sales
preezie @4.
...
5% conversion (+45% against control) = 9 sales
Even if you account for the +3% increased in non-preezie conversion, the preezie bucket although smaller shows at least a +40% increase.
Tracking user conversion will tell you the user level impact of preezie across sessions, so you can compare the influence of these preezie engaged users, e.g. sessions per user increases.
...
If you take 10% of your control bucket raw numbers (e.g. users, conversions, sessions, bounces) and your preezie bucket numbers put them into a signifance calculation tool:
OR
https://abtestguide.com/bayesian/
You can use this method these methods to understand both:
How preezie performs vs those who didn’t see it (i.e. the 10% / 50% buckets)
How preezie performs vs those who did but didn’t interact (i.e. the 10% / 40% buckets)
...
| Tip |
|---|
Tip: If your primary goal is something more reflrective reflective of impact to users who need help (e.g. nexw user bounce rate) then you should probably can analyse the 50/50% on the total results as usual. |
...
Here we’ll track those who saw it, those who clicked and compare against the inverse of both. Our main goal of exit intents are to keep users on the website, so our goals are now:
Split traffic | Page trigger | Segment | Goal A: Exit rate | Goal B: New user pages/session | Goal C: User conversion rate (sales per user) |
|---|---|---|---|---|---|
50% Test | any | 25% did not see preezie | 30% | 2.1 | 4.2% |
10% saw preezie and did not click it | 20% | 2.3 | 4.5% | ||
15% clicked on preezie | 6% | 4.3 | 6.8% | ||
50% Control | any | 50% no preezie loaded | 30% | 2.2 | 4.1% |
Here you can compare your control bucket with those who did not see it to ensure the behaviour is the same across buckets (see article on A/A testing).
...