How to Create an AI Video Thumbnail A/B Test System

The ai video thumbnail ab testing youtube system I run increases click-through rates with data. I build this for clients who publish video content consistently and need to move faster without adding headcount.

Generates multiple thumbnail concepts and tracks ctr to find the winner. The whole pipeline runs from a single Python script.

What You Need

• Python 3.9+
• Anthropic API key (Claude)
• FFmpeg for video processing
• OpenAI Whisper for transcription (where applicable)

Step 1: Install Dependencies

pip install anthropic openai-whisper python-dotenv moviepy

import anthropic
import json
import os
from dotenv import load_dotenv

load_dotenv()
claude = anthropic.Anthropic()

Step 2: Set Up the Core Processing Function

def test_thumbnails(input_path, config=None):
    if config is None:
        config = {"quality": "high", "format": "standard"}

    print(f"Processing: {input_path}")
    print(f"Config: {json.dumps(config)}")

    # Step 1: Analyze the input
    analysis = analyze_content(input_path)

    # Step 2: Generate the output
    result = generate_output(analysis, config)

    return result

Step 3: Build the AI Analysis Layer

def analyze_content(input_path):
    # Read or transcribe the input
    with open(input_path, 'r') as f:
        content = f.read()

    message = claude.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=2048,
        system="Analyze this content for thumbnail A/B testing system purposes. Return structured JSON with your findings.",
        messages=[{"role": "user", "content": content[:15000]}]
    )
    return json.loads(message.content[0].text)

Step 4: Generate and Save Output

def generate_output(analysis, config):
    output_dir = "output"
    os.makedirs(output_dir, exist_ok=True)

    output_path = os.path.join(output_dir, "result.json")
    with open(output_path, 'w') as f:
        json.dump(analysis, f, indent=2)

    print(f"Output saved: {output_path}")
    return output_path

Step 5: Add Batch Processing

def batch_process(input_dir, config=None):
    results = []
    for filename in os.listdir(input_dir):
        if filename.endswith(('.mp4', '.mov', '.txt', '.json')):
            filepath = os.path.join(input_dir, filename)
            result = test_thumbnails(filepath, config)
            results.append({"file": filename, "result": result})

    print(f"Processed {len(results)} files")
    return results

batch_process("./input-files")

What to Build Next

Add a notification layer that sends results to Slack or email when processing completes. Then connect the batch processor to a file watcher so new content gets processed automatically on arrival.

Related Reading

• Building a Creative Testing System - practical guidance for building AI-powered business systems
• Using AI for A/B Testing Strategy - practical guidance for building AI-powered business systems
• Setting Up A/B Testing Infrastructure - practical guidance for building AI-powered business systems