A/B Testing Your Earn Page: What to Test First

How do you A/B test an earn page, and what should you test first?

To A/B test an earn page, split your logged-in users at random into two groups, show each group one version of a single change, and compare how many users in each group earn and how much net revenue each group brings after reversals. Test the changes with the biggest possible effect first: where the entry point to the offerwall sits and what it is called, then how your currency’s value is shown, then onboarding for the first offer, wall order, bonus messaging, and reminders. Fix the number of users per group before you start, run the test for whole weeks, and wait for late credits and reversals before you call a winner.

For ideas to test, see Earn page: how to write one that converts and Offerwall placement. This page is about proving which ideas work for your users.

Choose the metric before you change anything

If you pick the metric after you see the data, you will always find a number that went up. Decide in advance which number decides the test, and which numbers must not get worse.

Always divide by everyone in the group, not only by users who opened the wall. A label change that sends twice as many casual users to the wall can lower the completion rate of openers and still raise revenue per user. Judged on openers alone, you would throw away the better version.

What to test first, in order of impact

Small tweaks need very large samples to measure, so start where one change can move results by a third or more. The order below suits most rewards sites, games, and community apps. Skip a step only if you have already tested it properly.

1. Entry point placement and name

Placement decides how many users open the wall at all, and nothing downstream makes up for a wall nobody opens. Test one entry point at a time, for example a header link next to the balance against your current menu item. Then test the label: users respond to the reward and the action (“Earn coins by trying games”), not to the format (“Offerwall”).

2. How the currency’s value is shown

“Earn 5,000 coins” means little until users know what coins buy. Test a value line above the wall, or a progress bar toward the next cashout or item. Test the presentation, not the rate itself. Users who see different rewards for the same offer compare screenshots, and a rate that looks unfair costs more trust than any test gains. Change the rate for everyone (see How to set your virtual currency exchange rate) and read it as a cautious before and after comparison.

3. Onboarding for the first offer

The first credited offer is when a user learns that your site pays. Test a short first visit step: what offers are, that some credit in minutes and bigger ones step by step, and where to check status. It can head off several problems from Why users start offers but don’t finish. Measure it only on new users, each with the same 14 days to reach a first credit.

4. Wall order when you run several walls

If your earn page lists several walls, the first one gets the most opens. Judge an order test on total net revenue per active user across every wall, not on the wall you moved, because one wall’s gain can be mostly another wall’s loss. Tag each wall’s link with its position to see where the money moved.

5. Bonus event messaging

A bonus raises rewards for a set period, usually for everyone on the wall at once. What you can split is who hears about it: an email or push for half of your users, a banner only for the other half. Measure after the bonus ends too: a bonus that only pulls next week’s earning into this week added nothing. Rewards app retention covers bonus budgets.

6. Push and email reminders

Reminders are the easiest test to win in the short run and the easiest way to lose users. Always keep a holdout group that gets no reminder, and judge on day 30 retention and unsubscribes as well as revenue. In apps, Firebase A/B Testing can run experiments on messages sent with the Notifications composer and lets you track retention and revenue next to your goal (Firebase A/B Testing docs, checked on October 22, 2026). Push notifications for rewards apps covers what to send and how often.

Setting up a fair split

A fair test sends comparable users to each version at the same time. Version A this week and version B next week is not an A/B test, because paydays, holidays, and the offers on the wall change from week to week.

1. Split by user, not by page view. Assign each logged-in user to A or B at random on first entry, store it in your database, and show the same version on every visit and device.
2. Decide who is in the test. Every active user for entry point tests; only users who sign up after the start date for onboarding tests.
3. Tag the wall link with a sub ID. Add a value such as sub1=exp07_b for each variant, and another sub ID for the placement or wall position. Sub IDs come back in every postback, so each credit and reversal carries its variant.
4. Trust your own assignment table. On Sharklio, sub IDs are not part of the signed hash, so a user could change them. Join postbacks to your assignment table by user ID for the final numbers, and use sub ID reports as a quick check.
5. Count visitors yourself. Wall reports count offers started, not users who saw your page, so group sizes and earn page opens come from your own analytics.
6. Check the split. If you planned 50/50 and one group is noticeably smaller, something is filtering users out of one variant, such as a crash or a cache serving the wrong version. Microsoft’s Experimentation Platform team calls this a sample ratio mismatch, reports that about 6% of its A/B tests have one, and advises not to trust results until you find the cause (Diagnosing Sample Ratio Mismatch in A/B Testing).

Never cache an earn page across users: the signed wall link belongs to one person. Click IDs and sub IDs explains how sub IDs differ from the IDs advertisers use.

How many users you need and how long to run

Before you start, decide the smallest lift worth knowing about, then calculate how many users each group needs to detect it. Evan Miller’s free sample size calculator takes your baseline rate, the minimum detectable effect, the significance level, and the statistical power, and returns the users needed per variation. For a quick estimate, his rule of thumb is:

Users per group = 16 x p x (1 - p) / d², where p is your current rate and d is the absolute change you want to detect.

Low earn page rates make tests large. At 2 earners per 100 users, detecting a lift to 2.6 needs about 8,700 users per group, and a lift to 2.2 about 78,000. Without that traffic, test bigger changes or measure a more frequent step, such as the share of users who open the wall. A/B testing ads on a small budget walks through the arithmetic in more detail.

For test length, three rules cover most cases:

• Run whole weeks. Weekday and weekend users behave differently. Two full weeks is a sensible minimum even if you reach your sample sooner.
• Leave time for credits to arrive. Many offers confirm days after the user finishes, and multi-step game offers credit step by step. Keep counting credits from test users for at least 7 to 14 days after the test ends.
• Read reversals last. Reversals can arrive weeks after a credit. Look at the reversal rate again a month later before you make a winner permanent.

Reading the result without fooling yourself

Do not peek and stop

The most common mistake is stopping the moment one version pulls ahead. In How Not To Run an A/B Test, Evan Miller shows that checking a running test repeatedly and stopping at the first significant result inflates false winners: in his worst-case example, a nominal 5% error rate becomes 26.1%. Look at the dashboard if you like, but decide only at the sample size you planned. If you need to stop early by design, use a sequential or Bayesian method built for it.

Allow for the novelty effect

Users click on anything new, so a fresh badge can win in week one and fade by week four. Researchers at Microsoft describe novelty as a desire to use something new “that tends to diminish over time” (Sadeghi et al., Novelty and Primacy). Compare the test’s first week with its last, and if the gap shrinks every week, run longer.

One change at a time, no fishing in segments

If version B has a new label, position, and value line and wins, you do not know which change did it. And slicing results by country, device, and source afterwards will always turn up a segment where B “won big” by chance. If you expect a change to work differently in one segment, say so before the test and size it for that segment.

Illustrative example: renaming the entry point

Copy two details: the site checked that the extra earners did not bring a jump in reversals, and it waited for retention data before making the change permanent. To price a lift like this for your own traffic, try the offerwall earnings calculator.

Tests not worth running

Keep a log of every test: the question, the change, the planned sample, the result, and the decision. A draw is a useful answer too: this change is not where your growth is.

Frequently asked questions