How to Build an AI Call Analysis System

An ai call analysis system for sales and support coaching turns every call into structured data. I build these for teams doing 20+ calls per day where managers cannot listen to everything. The system transcribes each call, scores it against your framework, identifies coaching opportunities, and flags deals at risk.

Managers get insights on every call without listening to a single one.

What You Need Before Starting

• Call recordings (from your phone system or meeting tool)
• Python 3.8+ with whisper and anthropic
• A scoring framework (what makes a good call)
• Storage for analysis data

Step 1: Transcribe and Segment Calls

import whisper

model = whisper.load_model("medium")

def transcribe_call(audio_path):
    result = model.transcribe(audio_path)
    return {
        "text": result["text"],
        "segments": result["segments"],
        "duration": result["segments"][-1]["end"] if result["segments"] else 0
    }

Step 2: Analyze Call Quality

import anthropic
import json

client = anthropic.Anthropic()

ANALYSIS_PROMPT = """Analyze this sales/support call transcript. Score each dimension 1-10.

Dimensions:
1. Opening (professional greeting, rapport building)
2. Discovery (quality of questions asked)
3. Active Listening (acknowledgment, paraphrasing)
4. Value Proposition (clarity of solution presented)
5. Objection Handling (addressed concerns effectively)
6. Next Steps (clear action items agreed)
7. Overall Tone (professional, confident, empathetic)

Also identify:
- Top 3 things done well
- Top 3 areas for improvement
- Any red flags (promises made, compliance issues)
- Customer sentiment (positive/neutral/negative)

Respond as JSON."""

def analyze_call(transcript_text):
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=800,
        messages=[{"role": "user", "content": f"{ANALYSIS_PROMPT}\n\nTranscript:\n{transcript_text[:6000]}"}]
    )
    return json.loads(response.content[0].text)

Step 3: Extract Key Moments

def extract_key_moments(transcript_text):
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=400,
        messages=[{
            "role": "user",
            "content": f"""From this call transcript, extract:
1. Customer pain points mentioned
2. Objections raised
3. Commitments made by either party
4. Pricing discussed
5. Next steps agreed

Transcript:
{transcript_text[:6000]}

Respond as JSON."""
        }]
    )
    return json.loads(response.content[0].text)

Step 4: Build the Pipeline

def process_call(audio_path, call_metadata):
    transcript = transcribe_call(audio_path)
    analysis = analyze_call(transcript["text"])
    moments = extract_key_moments(transcript["text"])

    overall_score = sum(analysis.get("scores", {}).values()) / 7

    result = {
        "call_id": call_metadata.get("id"),
        "agent": call_metadata.get("agent"),
        "duration_minutes": round(transcript["duration"] / 60, 1),
        "overall_score": round(overall_score, 1),
        "analysis": analysis,
        "key_moments": moments,
        "transcript": transcript["text"]
    }

    save_analysis(result)
    if overall_score < 5:
        alert_manager(result)

    return result

Step 5: Generate Coaching Reports

def generate_agent_report(agent_name, days=7):
    conn = sqlite3.connect("call_analysis.db")
    calls = conn.execute("""
        SELECT * FROM analyses WHERE agent = ? AND analyzed_at > datetime('now', ?)
    """, (agent_name, f"-{days} days")).fetchall()

    scores = [c["overall_score"] for c in calls]
    avg_score = sum(scores) / len(scores) if scores else 0

    all_improvements = []
    for call in calls:
        analysis = json.loads(call["analysis_json"])
        all_improvements.extend(analysis.get("areas_for_improvement", []))

    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=400,
        messages=[{
            "role": "user",
            "content": f"""Generate a coaching summary for {agent_name}.
{len(calls)} calls analyzed. Average score: {avg_score:.1f}/10.
Common improvement areas: {', '.join(all_improvements[:10])}
Provide 3 specific, actionable coaching recommendations."""
        }]
    )
    return {"agent": agent_name, "calls": len(calls), "avg_score": avg_score, "coaching": response.content[0].text}

What to Build Next

Add real-time call scoring. Provide live coaching prompts to agents during calls. When the AI detects a missed discovery question or an unhandled objection, surface a suggestion on the agent's screen.

Related Reading

• AI for Sales Pipeline Management - call analysis feeding pipeline intelligence
• The Feedback Loop That Powers Everything - call data improving sales processes
• The Measurement Framework That Actually Works - measuring call quality systematically