A/B testing of headlines in advertising
What to test first: prioritizing hypotheses by impact/effort
Most instructions on A/B testing explain how to launch an experiment, but do not explain what to test first. This is a mistake: the order of hypotheses matters more than the mechanics of launching itself. If we distribute headline variables by their influence on CTR, the picture looks like this: changing the type of hook has the strongest effect, while editing the wording within one type is almost imperceptible. Therefore, the test should proceed by levels, not randomly.
- Level 1 (high impact, low effort) — complete change of hook type: pain vs benefit vs curiosity vs urgency. Test first and always.
- Level 2 — audience specificity: addressing a narrow role or situation versus general 'for everyone'.
- Level 3 — tone and structure: question, imperative, presence of specifics (numbers, deadlines).
- Level 4 (low impact) — micro-edits: punctuation, word order, synonyms. Test last and only when budget is free.
A typical mistake: the first test is spent on the 4 level — comparing 'Buy with a discount!' and 'A discount is waiting for you!'. The difference in CTR here is statistically indistinguishable from noise, and the conclusion 'exclamation mark doesn't work' is drawn from a test that doesn't even test the exclamation mark.
How to distinguish a result reversal from random fluctuation
Reversal ('reversal') is the moment when an early test leader yields to another variant as data accumulates. It happens in almost every test stopped before the critical threshold. The rule is strict: at least 7 days and at least 50 conversions per variant at 95% statistical confidence. Before this point, any leadership is noise, not a signal.
A specific scenario that repeats across different accounts: a specialist launches three headlines — with pain, with benefit, with intrigue. On the third day, 'benefit' leads with a noticeable lead, the test is stopped, and the budget is scaled up to the winner. A week later, it turns out that if the test had continued another four days ahead, the 'pain' headline would have come out on top — it had slower statistics due to a narrower audience response. The turnaround in such tests usually happens between 3 and 6 day: the first days drive random spikes in activity from individual users, and only by the end of the week behavior evens out.
You can verify whether the trend is real in one way: does the significance calculator (in Google Ads Experiments or third-party services) consider the difference statistically significant given the current sample size. If the service says 'not enough data' or the confidence interval crosses zero — no conclusions can be drawn, even if the difference in CTR visually looks substantial.
What the data shows: numbers, exclamation marks, verbs
Intuition here regularly fails. An analysis of a large FMCG advertising dataset (880 thousand+ ads, 1471 unique texts) yielded a result that contradicts the advice 'add numbers for specificity': on search, ads without numbers showed a higher CTR than those with numbers. However, the presence of an action verb and an exclamation mark worked in the positive — such headlines outperformed neutral formulations without imperative intonation.
| Headline element | Effect on CTR (search) | Comment |
|---|---|---|
| Numbers (price, %, duration) | Reduce CTR | In networks and social media, the effect is opposite — there numbers often increase response rates |
| Exclamation mark | Increases CTR | Works as an action signal, the effect fades with frequent repetition in the campaign |
| Action verb at the beginning | Increases CTR | ‘Get’, ‘Book’ consistently stronger than passive constructions |
| Hook type (pain/gain/curiosity/urgency) | Up to +34% difference between best and worst | The strongest individual factor — test first |
Practical conclusion is strict: the rule ‘numbers always convert’ is true for landing pages and offers, but on search results pages, specificity in the headline sometimes works against click-through. Seeing the price, a user mentally closes the question and does not click on the ad — the click becomes unnecessary.
Differences between platforms: Google Ads, Yandex Direct, Facebook
The same headline behaves differently on different platforms. Rewriting it for each platform is not perfectionism, but an essential condition for a correct test, because the audience is not the only thing that differs—rather, the auction mechanics and ad format vary.
| Platform | Minimum test duration | Sample threshold | Feature |
|---|---|---|---|
| Google Ads | from 1 week | 50 conversions per variant | Experiments isolates variables more accurately than other methods, but requires a budget for the entire duration |
| Yandex Direct | 3–4 days | 15 000 impressions per variant (not conversions) | Impression-based metrics make the test shorter, but less accurate for evaluating sales |
| Facebook / Instagram | 3–4 days (minimum one day, maximum 30) | depends on auction budget | The algorithm itself 'consumes' variants with low early-CTR, and the distribution of impressions is uneven |
From this follows a specific rule: a headline that wins in Direct due to volume of impressions is not guaranteed to win in Google Ads, where conversion rate, not just the click, determines success. In Facebook, the algorithm may exhaust a good variant's budget simply because it started slowly—conclusions cannot be drawn earlier than 3–4 full days without prematurely stopping the test.
Minimum sample size for local businesses and niche categories
If 50 conversions for a variant accumulate for a week or longer — this is a normal rate for local business, not a sign of poor advertising. It's not necessary to adjust the test to the standard of large accounts; a different sequence of actions applies here:
- Reduce the number of simultaneous variants to 2–3, not 6–8 as in large campaigns — otherwise the budget will be spread so thin that the statistics will not accumulate at all.
- Use impressions or clicks as an intermediate signal, similar to the Yandex Direct approach, and leave conversions as the final confirmation, not the sole criterion.
- Extend the test to 2–3 weeks instead of a week when traffic volume is low, and mark intermediate points to see whether the trend has stabilized.
- Test only the 1 level from the prioritization framework — the type of hook. With a small sample size, subtle differences in tone are statistically unattainable, so it makes no sense to spend traffic on them.
Multi-stage framework: how to use the results of one test for the next
A one-time test is almost useless without a system for accumulating results. The working schema is simple: each completed test is recorded in a table — which variable was tested (level 1–4 according to the framework above), which variant won, under what sample, on what platform. The next test takes the winning variable as a constant and tests the next level.
Example of a cascade: test 1 showed that 'pain' outperforms 'benefit' in the category — the variable is recorded as a fact for this category. Test 2 checks audience specificity within the 'pain' hook: addressing a specific role versus a general appeal. Test 3 checks tone within the winning combination. Such a chain saves budget: the type of hook does not need to be tested again in every new campaign, it can be transferred to adjacent categories with an industry adjustment.
Industry adjustment is not an abstraction but an observation from practice: in B2B-services, 'pain' consistently outperforms 'benefit' more strongly than in consumer goods, where impulse decides, and where 'urgency' or 'intrigue' perform better. Documenting these patterns by industry and category is what transforms isolated tests into a working knowledge base and eliminates the need to test every minor detail from scratch in every new campaign.
What to test first: prioritizing hypotheses by impact/effort
Most instructions on A/B testing explain how to launch an experiment, but do not explain what to test first. This is a mistake: the order of hypotheses matters more than the mechanics of launching itself. If we distribute headline variables by their influence on CTR, the picture looks like this: changing the type of hook has the strongest effect, while editing the wording within one type is almost imperceptible. Therefore, the test should proceed by levels, not randomly.
- Level 1 (high impact, low effort) — complete change of hook type: pain vs benefit vs curiosity vs urgency. Test first and always.
- Level 2 — audience specificity: addressing a narrow role or situation versus general 'for everyone'.
- Level 3 — tone and structure: question, imperative, presence of specifics (numbers, deadlines).
- Level 4 (low impact) — micro-edits: punctuation, word order, synonyms. Test last and only when budget is free.
A typical mistake: the first test is spent on the 4 level — comparing 'Buy with a discount!' and 'A discount is waiting for you!'. The difference in CTR here is statistically indistinguishable from noise, and the conclusion 'exclamation mark doesn't work' is drawn from a test that doesn't even test the exclamation mark.
How to distinguish a result reversal from random fluctuation
Reversal ('reversal') is the moment when an early test leader yields to another variant as data accumulates. It happens in almost every test stopped before the critical threshold. The rule is strict: at least 7 days and at least 50 conversions per variant at 95% statistical confidence. Before this point, any leadership is noise, not a signal.
A specific scenario that repeats across different accounts: a specialist launches three headlines — with pain, with benefit, with intrigue. On the third day, 'benefit' leads with a noticeable lead, the test is stopped, and the budget is scaled up to the winner. A week later, it turns out that if the test had continued another four days ahead, the 'pain' headline would have come out on top — it had slower statistics due to a narrower audience response. The turnaround in such tests usually happens between 3 and 6 day: the first days drive random spikes in activity from individual users, and only by the end of the week behavior evens out.
You can verify whether the trend is real in one way: does the significance calculator (in Google Ads Experiments or third-party services) consider the difference statistically significant given the current sample size. If the service says 'not enough data' or the confidence interval crosses zero — no conclusions can be drawn, even if the difference in CTR visually looks substantial.
What the data shows: numbers, exclamation marks, verbs
Intuition here regularly fails. An analysis of a large FMCG advertising dataset (880 thousand+ ads, 1471 unique texts) yielded a result that contradicts the advice 'add numbers for specificity': on search, ads without numbers showed a higher CTR than those with numbers. However, the presence of an action verb and an exclamation mark worked in the positive — such headlines outperformed neutral formulations without imperative intonation.
| Headline element | Effect on CTR (search) | Comment |
|---|---|---|
| Numbers (price, %, duration) | Reduce CTR | In networks and social media, the effect is opposite — there numbers often increase response rates |
| Exclamation mark | Increases CTR | Works as an action signal, the effect fades with frequent repetition in the campaign |
| Action verb at the beginning | Increases CTR | ‘Get’, ‘Book’ consistently stronger than passive constructions |
| Hook type (pain/gain/curiosity/urgency) | Up to +34% difference between best and worst | The strongest individual factor — test first |
Practical conclusion is strict: the rule ‘numbers always convert’ is true for landing pages and offers, but on search results pages, specificity in the headline sometimes works against click-through. Seeing the price, a user mentally closes the question and does not click on the ad — the click becomes unnecessary.
Differences between platforms: Google Ads, Yandex Direct, Facebook
The same headline behaves differently on different platforms. Rewriting it for each platform is not perfectionism, but an essential condition for a correct test, because the audience is not the only thing that differs—rather, the auction mechanics and ad format vary.
| Platform | Minimum test duration | Sample threshold | Feature |
|---|---|---|---|
| Google Ads | from 1 week | 50 conversions per variant | Experiments isolates variables more accurately than other methods, but requires a budget for the entire duration |
| Yandex Direct | 3–4 days | 15 000 impressions per variant (not conversions) | Impression-based metrics make the test shorter, but less accurate for evaluating sales |
| Facebook / Instagram | 3–4 days (minimum one day, maximum 30) | depends on auction budget | The algorithm itself 'consumes' variants with low early-CTR, and the distribution of impressions is uneven |
From this follows a specific rule: a headline that wins in Direct due to volume of impressions is not guaranteed to win in Google Ads, where conversion rate, not just the click, determines success. In Facebook, the algorithm may exhaust a good variant's budget simply because it started slowly—conclusions cannot be drawn earlier than 3–4 full days without prematurely stopping the test.
Minimum sample size for local businesses and niche categories
If 50 conversions for a variant accumulate for a week or longer — this is a normal rate for local business, not a sign of poor advertising. It's not necessary to adjust the test to the standard of large accounts; a different sequence of actions applies here:
- Reduce the number of simultaneous variants to 2–3, not 6–8 as in large campaigns — otherwise the budget will be spread so thin that the statistics will not accumulate at all.
- Use impressions or clicks as an intermediate signal, similar to the Yandex Direct approach, and leave conversions as the final confirmation, not the sole criterion.
- Extend the test to 2–3 weeks instead of a week when traffic volume is low, and mark intermediate points to see whether the trend has stabilized.
- Test only the 1 level from the prioritization framework — the type of hook. With a small sample size, subtle differences in tone are statistically unattainable, so it makes no sense to spend traffic on them.
Multi-stage framework: how to use the results of one test for the next
A one-time test is almost useless without a system for accumulating results. The working schema is simple: each completed test is recorded in a table — which variable was tested (level 1–4 according to the framework above), which variant won, under what sample, on what platform. The next test takes the winning variable as a constant and tests the next level.
Example of a cascade: test 1 showed that 'pain' outperforms 'benefit' in the category — the variable is recorded as a fact for this category. Test 2 checks audience specificity within the 'pain' hook: addressing a specific role versus a general appeal. Test 3 checks tone within the winning combination. Such a chain saves budget: the type of hook does not need to be tested again in every new campaign, it can be transferred to adjacent categories with an industry adjustment.
Industry adjustment is not an abstraction but an observation from practice: in B2B-services, 'pain' consistently outperforms 'benefit' more strongly than in consumer goods, where impulse decides, and where 'urgency' or 'intrigue' perform better. Documenting these patterns by industry and category is what transforms isolated tests into a working knowledge base and eliminates the need to test every minor detail from scratch in every new campaign.
Recent articles
MegaFon, BMW, VTB and more 240+ companies. See how it looks in the showcase reel:
Our branding agency is approached to order:
Branding agency services
