---
aliases: ["Newsletter A/B Testing"]
Is A: Procedure
Belongs to: "[[responsibility-content-production]]"
Cadence: Bi-weekly
Owner: "[[person-luca-rossi]]"
---
# Newsletter A/B Testing

A/B testing turns newsletter optimization from guesswork into evidence. By systematically testing subject lines, content formats, and structural choices, this procedure builds a growing body of knowledge about what resonates with the audience. Each test is small and low-risk, but the compounding effect of dozens of data points over time significantly improves [[measure-open-rate]] and overall engagement. See [[newsletter-subject-lines]] for accumulated insights from past tests.

## Steps

- [ ] Define the hypothesis for this test cycle: what specific question are you trying to answer? (e.g., "Do question-format subject lines outperform statement-format?")
- [ ] Choose the variable to test — only one variable per test to ensure clean results (subject line, preview text, CTA placement, content length, etc.)
- [ ] Create the two variants (A and B) with a single, clear difference between them
- [ ] Configure the A/B test in the email platform: set the audience split (typically 50/50 for subject lines, 20/20/60 for content tests with auto-send)
- [ ] Set the success metric before sending: open rate for subject line tests, click rate for content tests
- [ ] Send the test and wait for the full measurement window (minimum 24 hours for subject lines, 48 hours for content engagement)
- [ ] Record the results: variant, sample size, metric values, statistical significance, and winner
- [ ] Document the learning in the A/B testing log and update [[newsletter-subject-lines]] if the finding is about subject lines
- [ ] Apply the winning insight to the next newsletter issue as the new default

## Notes

- Statistical significance matters. A test with 200 opens per variant is too small to draw conclusions from a 2% difference. Aim for at least 500 per variant before declaring a winner.
- Test one thing at a time. Testing a new subject line format AND a new CTA simultaneously makes it impossible to attribute the result.
- Keep a running log of all tests and results. After 20+ tests, patterns emerge that are far more valuable than any single result. This feeds into [[topic-newsletter-growth]] strategy.
- Not every test will have a clear winner, and that is useful information too. "No significant difference" means the variable does not matter much — stop agonizing over it and focus elsewhere.
- Share interesting test results with [[person-sara-ricci]] — they inform her editorial instincts and help align the whole content operation around what actually works.
