Most people run cold email campaigns and hope for the best. They send 500 emails, get 3 replies, and either give up or keep doing the same thing expecting different results.
That is insane. Cold email is one of the most testable marketing channels that exists. Every element can be isolated, tested, and improved. Subject lines, opening hooks, body copy, CTAs, send times, follow-up cadence. All of it is measurable.
The teams that win at cold email are the ones that test constantly. They treat every campaign as an experiment, not a one-shot attempt. Related: Cold Email Metrics.
The Testing Hierarchy: What to Test First
Not all tests are equal. Some variables have massive impact on results. Others barely move the needle. Here is the priority order: Related: Cold Email Subject Lines.
| Priority | Element | Impact on Results | Test This When |
|---|---|---|---|
| 1 | Subject Line | Massive (controls open rate) | Always test first |
| 2 | Opening Line | High (determines if they keep reading) | After winning subject line |
| 3 | Call to Action | High (determines if they respond) | After winning opener |
| 4 | Email Length | Medium (too long kills replies) | After core elements are strong |
| 5 | Offer/Angle | High (but harder to isolate) | When reply rate plateaus |
| 6 | Send Time | Low to Medium | When everything else is optimized |
| 7 | Personalization Level | Medium | When testing effort vs results |
The one rule of A/B testing: Only test one variable at a time. If you change the subject line AND the body copy, you have no idea which change caused the difference. Discipline here separates people who improve from people who guess. Related: How To Write Cold Emails.
How to Structure an A/B Test
Step 1: Define Your Hypothesis
Do not just "try stuff." State what you believe and why. Example: "I believe a question-based subject line will outperform a statement-based subject line because questions create curiosity."
Step 2: Create Your Variations
Keep it to two variations (A vs B). Testing three or more versions splits your sample size and makes results less reliable.
Step 3: Split Your List Randomly
Your A and B groups must be randomly assigned. Do not put "better" leads in one group. Most sending tools handle this automatically.
Step 4: Send Simultaneously
Both versions should go out at the same time to the same type of prospects. Sending version A on Monday and version B on Thursday introduces a timing variable.
Step 5: Wait for Enough Data
This is where most people fail. They look at results after 50 emails and declare a winner. That is not enough data. You need:
| Metric Being Tested | Minimum Emails Per Variation | Ideal Emails Per Variation |
|---|---|---|
| Open Rate (subject line) | 100 | 200-300 |
| Reply Rate (body copy) | 150 | 250-400 |
| Meeting Rate (CTA) | 200 | 300-500 |
Step 6: Determine the Winner
Look for a meaningful difference. If version A gets 12% reply rate and version B gets 13%, that is not a real winner. That is noise. You want at least a 20% relative difference before calling a test (e.g., 10% vs 13% is a meaningful gap).
What to Test: Subject Lines
Subject lines are the most impactful test because they directly control your open rate. No opens = no replies = no meetings. Here are the variables to test:
Question vs Statement
- A: "Quick question about your marketing team"
- B: "Your marketing team is missing this"
Personalized vs Generic
- A: "{{company}} + ColdCraft"
- B: "Booking more meetings with cold email"
Short vs Long
- A: "Quick question"
- B: "Quick question about your lead generation strategy for Q2"
Direct vs Curiosity
- A: "We book meetings for companies like {{company}}"
- B: "Noticed something about {{company}}"
What we have seen work best: Short (2-4 words), lowercase, question-based subject lines consistently outperform everything else. "quick question" has been our top performer across dozens of campaigns. Boring? Yes. Effective? Very.
What to Test: Opening Lines
The opening line determines whether they keep reading. Test these approaches:
- Observation: "Noticed you are hiring 3 new SDRs this quarter."
- Pain point: "Most B2B companies waste 30% of their marketing budget on channels that do not convert."
- Question: "How is your team handling outbound lead generation right now?"
- Compliment: "Saw your recent post about scaling sales teams. Great insights."
- Trigger event: "Congrats on the Series B. That usually means aggressive hiring plans."
What to Test: CTAs
The CTA determines whether they take action. Common tests:
- Calendar link vs no link: Some people find links pushy. Others appreciate the convenience.
- Specific time vs open-ended: "15 minutes this Thursday?" vs "Worth a quick call?"
- Soft vs hard ask: "Open to learning more?" vs "Can I send you a case study?"
- Interest-based: "Interested?" (one word CTAs work surprisingly well)
The Weekly Testing Cadence
Here is the framework we use at ColdCraft for continuous improvement:
- Monday: Review last week's test results. Identify the winner.
- Tuesday: Design this week's test. One variable, clear hypothesis.
- Wednesday: Launch new test. Split list, send both versions.
- Thursday-Friday: Monitor early results (do not make decisions yet).
- Following Monday: Full week of data. Analyze and repeat.
This cadence means you run 4 tests per month, 48 per year. After 48 iterations, your campaigns will be unrecognizable from where you started. That is how you go from 3% reply rate to 15%.
Advanced Testing: Beyond Copy
ICP Testing
Sometimes the problem is not your email. It is who you are emailing. Test different target audiences:
- Company size: 10-50 employees vs 50-200 vs 200-1000
- Job titles: VPs vs Directors vs Managers
- Industries: SaaS vs agencies vs professional services
- Triggers: Recently funded vs actively hiring vs leadership change
Sequence Length Testing
How many follow-ups is too many? Test different sequence lengths and measure at which point diminishing returns kick in. Our data shows that email 3 and 4 generate the most replies, but email 5 and 6 still contribute meaningfully.
Send Time Testing
Tuesday through Thursday, 8-10am in the recipient's time zone typically performs best. But test it for your specific audience. Some industries (restaurants, healthcare) have different peak email times.
Common Testing Mistakes
- Testing too many things at once. One variable per test. Period.
- Declaring winners too early. Wait for 100+ sends per variation minimum.
- Ignoring the right metric. Open rate matters for subject lines. Reply rate matters for body copy. Meeting rate matters for CTAs. Match the metric to what you changed.
- Not documenting results. Keep a testing log. Date, hypothesis, variations, sample size, results, winner, and learnings. This becomes your playbook.
- Stopping when you find something that works. "Good enough" is the enemy of great. Keep testing even when things are going well.
Frequently Asked Questions
What should I A/B test first in cold email?
Test subject lines first because they determine whether your email gets opened. No opens means nothing else matters. After you find a winning subject line, test your opening line, then your CTA, then your overall email length and tone.
How many emails do I need for a valid A/B test?
You need at least 100 emails per variation to get directionally useful data. For statistical significance at a 95% confidence level, aim for 200-300 per variation. Anything under 50 per variation is just noise.
How long should I run a cold email A/B test?
Run each test for at least 5-7 business days to account for different response patterns. Some people check email on Monday mornings, others respond on Friday afternoons. A test that runs only 2 days misses these patterns.
What is a good reply rate for cold email?
Industry average is 5-8%. Good campaigns hit 10-15%. Top performers see 20%+. But reply rate alone is misleading. Track positive reply rate (interested responses) which should be 3-8% for a healthy campaign.
Want Us to Run Your Tests?
ColdCraft runs continuous A/B tests on every campaign. We optimize subject lines, copy, CTAs, and targeting weekly so your results improve every month.
Get Started Today
