Experiment Loop
Status: production on my own outbound, weekly cadence, fully automatic.
Problem
Everyone says "test your campaigns" and almost nobody defines when a test is over. So teams kill arms after 40 sends on vibes, or scale a fluke because one meeting landed on a Tuesday. I wanted the kill/scale decision taken away from mood entirely.
How it runs
Once a week, the loop scores every running campaign arm by positive reply rate (an 11-label reply taxonomy, because "unsubscribe" and "not right now" and "tell me more" are three different facts). Then it applies a two-tier statistical rule:
- Offer-hunting arms need on the order of 1,000 sends per arm before a chi-square comparison is allowed to conclude anything. Below that, the loop keeps its mouth shut and keeps sending.
- Optimization tests (subject lines, framing, list cuts) only run on arms that already cleared a 1 percent positive floor. You don't A/B the paint on a car that doesn't start.
Concluded experiments write what they proved to a knowledge base, then the loop proposes the next single-variable round, deploys it inside a fixed send budget, and launches. One variable per round, enforced. Tripwires (bounce rate over 2 percent, deliverability drift) auto-pause an arm without waiting for the weekly pass.
It runs full-auto and notifies me in detail afterward. The guardrails are the approval.
Stack
Node scripts, cron on an always-on VM, the sending platform's API for arm stats, Postgres for experiment state and conclusions.
Worth stealing
The two-tier rule. Most outbound "tests" are noise laundering. Deciding your minimum sample and your entry floor before the experiment starts is boring, and it's the entire difference between a finding and an anecdote.