Skip to main content
All systems

Experiment Loop

Status: production on my own outbound, weekly cadence, fully automatic.

Problem

Everyone says "test your campaigns" and almost nobody defines when a test is over. So teams kill arms after 40 sends on vibes, or scale a fluke because one meeting landed on a Tuesday. I wanted the kill/scale decision taken away from mood entirely.

How it runs

Once a week, the loop scores every running campaign arm by positive reply rate (an 11-label reply taxonomy, because "unsubscribe" and "not right now" and "tell me more" are three different facts). Then it applies a two-tier statistical rule:

  • Offer-hunting arms need on the order of 1,000 sends per arm before a chi-square comparison is allowed to conclude anything. Below that, the loop keeps its mouth shut and keeps sending.
  • Optimization tests (subject lines, framing, list cuts) only run on arms that already cleared a 1 percent positive floor. You don't A/B the paint on a car that doesn't start.

Concluded experiments write what they proved to a knowledge base, then the loop proposes the next single-variable round, deploys it inside a fixed send budget, and launches. One variable per round, enforced. Tripwires (bounce rate over 2 percent, deliverability drift) auto-pause an arm without waiting for the weekly pass.

It runs full-auto and notifies me in detail afterward. The guardrails are the approval.

Stack

Node scripts, cron on an always-on VM, the sending platform's API for arm stats, Postgres for experiment state and conclusions.

Worth stealing

The two-tier rule. Most outbound "tests" are noise laundering. Deciding your minimum sample and your entry floor before the experiment starts is boring, and it's the entire difference between a finding and an anecdote.

Agent-readable version: /systems/experiment-loop.md