Building a meeting summarisation bot with Whisper and Claude

What the pipeline actually needs to do

Meeting summarisation sounds straightforward until you deal with real recordings. A 90-minute engineering planning call produces an audio file around 80 MB, contains overlapping speech, background noise, and topic shifts that are obvious to a human listener but invisible to a naive summariser. The output that people actually use is not a transcript — it is a structured document with decisions made, action items assigned to named owners, and open questions that need follow-up.

Building this well requires making concrete choices at each stage: how to transcribe, how to segment long audio into chunks the model can handle, how to extract structured data rather than prose summaries, and how to surface the output through an API that your team's tools can call. This article builds the complete pipeline from audio file to structured JSON summary.

Setting up Whisper for transcription

OpenAI Whisper runs locally or via the OpenAI API. For production use with meeting audio, the API is usually the right choice: you avoid GPU provisioning, get word-level timestamps in the response, and the whisper-1 model handles accents and background noise reliably.

Install the dependencies:

pip install openai anthropic pydub python-dotenv

The core transcription call with word timestamps enabled:

import openai
from pathlib import Path

openai_client = openai.OpenAI()

def transcribe_audio(audio_path: Path) -> dict:
    """Transcribe audio file and return full Whisper response with segments."""
    with open(audio_path, "rb") as audio_file:
        response = openai_client.audio.transcriptions.create(
            model="whisper-1",
            file=audio_file,
            response_format="verbose_json",
            timestamp_granularities=["segment"]
        )
    return response.model_dump()

The verbose_json format gives you segments with start and end timestamps, which you will use for chunking. Each segment looks like:

{
  "id": 12,
  "start": 145.2,
  "end": 152.8,
  "text": " So the decision we're making here is to delay the v2 launch by two weeks."
}

Whisper has a 25 MB file size limit per request. Most 90-minute meetings in MP4 or M4A format exceed this. You need to split the audio before sending it.

Splitting audio for the 25 MB limit

Use pydub to split audio at silence boundaries, which produces cleaner chunks than splitting at fixed byte offsets:

from pydub import AudioSegment
from pydub.silence import detect_nonsilent
import tempfile

def split_audio_to_chunks(audio_path: Path, max_size_mb: float = 20.0) -> list[Path]:
    """Split audio into chunks under max_size_mb, splitting at silence boundaries."""
    audio = AudioSegment.from_file(audio_path)
    
    # Convert to mono MP3 to reduce size (meetings don't need stereo)
    audio = audio.set_channels(1)
    
    total_duration_ms = len(audio)
    target_chunk_duration_ms = _estimate_chunk_duration(audio, max_size_mb)
    
    chunks = []
    start_ms = 0
    chunk_index = 0
    
    with tempfile.TemporaryDirectory() as tmpdir:
        while start_ms < total_duration_ms:
            end_ms = min(start_ms + target_chunk_duration_ms, total_duration_ms)
            
            # Find the nearest silence within 30 seconds before the target end
            end_ms = _find_silence_boundary(audio, end_ms, window_ms=30_000)
            
            chunk = audio[start_ms:end_ms]
            chunk_path = Path(tmpdir) / f"chunk_{chunk_index:03d}.mp3"
            chunk.export(chunk_path, format="mp3", bitrate="64k")
            chunks.append(chunk_path)
            
            start_ms = end_ms
            chunk_index += 1
    
    return chunks

def _estimate_chunk_duration(audio: AudioSegment, max_size_mb: float) -> int:
    """Estimate chunk duration in ms to stay under max_size_mb at 64k bitrate."""
    bytes_per_second = 64 * 1000 / 8  # 64 kbps in bytes/s
    max_bytes = max_size_mb * 1024 * 1024
    max_seconds = max_bytes / bytes_per_second
    return int(max_seconds * 1000 * 0.9)  # 10% safety margin

def _find_silence_boundary(audio: AudioSegment, target_ms: int, window_ms: int) -> int:
    """Find the nearest silence point before target_ms."""
    search_start = max(0, target_ms - window_ms)
    segment = audio[search_start:target_ms]
    
    nonsilent = detect_nonsilent(segment, min_silence_len=500, silence_thresh=-40)
    
    if not nonsilent:
        return target_ms
    
    # Return end of last non-silent segment + search_start offset
    last_nonsilent_end = nonsilent[-1][1]
    return search_start + last_nonsilent_end

With 64 kbps mono encoding, a 20-minute chunk is around 9 MB — well within the 25 MB limit and economical to process.

Transcribing multiple chunks and reassembling

Transcribe each chunk and offset the timestamps by the chunk's position in the original recording:

def transcribe_chunks(audio_path: Path) -> list[dict]:
    """Split, transcribe all chunks, and return segments with corrected timestamps."""
    chunks = split_audio_to_chunks(audio_path)
    all_segments = []
    cumulative_offset = 0.0
    
    for chunk_path in chunks:
        result = transcribe_audio(chunk_path)
        segments = result.get("segments", [])
        
        # Offset timestamps
        for seg in segments:
            seg["start"] += cumulative_offset
            seg["end"] += cumulative_offset
        
        all_segments.extend(segments)
        
        # The last segment's end time is the duration of this chunk
        if segments:
            cumulative_offset = segments[-1]["end"]
    
    return all_segments

Now you have a complete list of segments with accurate timestamps spanning the full recording. The next step is building the text that goes to Claude.

Structuring the prompt for reliable JSON output

Claude is good at extracting structured information from meeting transcripts, but the prompt design determines whether the output is reliably parseable or inconsistently formatted. Use a system prompt that describes the exact JSON structure you want, and provide the transcript as the user message:

import anthropic
import json

claude_client = anthropic.Anthropic()

SYSTEM_PROMPT = """You are a meeting summarisation assistant. Given a meeting transcript, extract structured information and return it as valid JSON with exactly this structure:

{
  "title": "string - inferred meeting topic (max 80 chars)",
  "duration_minutes": number,
  "attendees": ["list of names mentioned"],
  "summary": "string - 2-4 sentence executive summary",
  "decisions": [
    {
      "decision": "string - what was decided",
      "owner": "string or null - person responsible",
      "timestamp_seconds": number or null
    }
  ],
  "action_items": [
    {
      "task": "string - specific action",
      "owner": "string or null",
      "due_date": "string or null - ISO date if mentioned",
      "timestamp_seconds": number or null
    }
  ],
  "open_questions": ["list of unresolved questions"],
  "key_topics": ["list of main topics discussed"]
}

Return only valid JSON. Do not include markdown code fences or any other text."""

def summarise_transcript(segments: list[dict], duration_seconds: float) -> dict:
    transcript_text = _format_transcript(segments)
    
    response = claude_client.messages.create(
        model="claude-opus-4-5",
        max_tokens=2048,
        system=SYSTEM_PROMPT,
        messages=[{
            "role": "user",
            "content": f"Meeting duration: {duration_seconds/60:.1f} minutes\n\nTranscript:\n{transcript_text}"
        }]
    )
    
    raw = response.content[0].text.strip()
    return json.loads(raw)

def _format_transcript(segments: list[dict]) -> str:
    lines = []
    for seg in segments:
        timestamp = f"[{int(seg['start']//60):02d}:{int(seg['start']%60):02d}]"
        lines.append(f"{timestamp} {seg['text'].strip()}")
    return "\n".join(lines)

Including timestamps in the transcript gives Claude context for populating the timestamp_seconds fields, which let you link back to the relevant moment in the recording from action items and decisions.

Handling long transcripts with map-reduce summarisation

A 90-minute meeting produces a transcript too long to fit in a single context window along with a detailed extraction prompt. The solution is map-reduce: summarise each segment of the transcript independently, then combine the partial summaries into a final output.

def summarise_long_transcript(segments: list[dict], duration_seconds: float) -> dict:
    """Use map-reduce for transcripts that exceed ~30 minutes."""
    
    MINUTES_PER_CHUNK = 20
    chunk_segments = _split_segments_by_time(segments, MINUTES_PER_CHUNK * 60)
    
    if len(chunk_segments) == 1:
        return summarise_transcript(segments, duration_seconds)
    
    # Map: extract key info from each chunk
    partial_summaries = []
    for i, chunk in enumerate(chunk_segments):
        partial = _extract_partial_summary(chunk, chunk_index=i)
        partial_summaries.append(partial)
    
    # Reduce: combine into final structured output
    return _combine_partial_summaries(partial_summaries, duration_seconds)

def _extract_partial_summary(segments: list[dict], chunk_index: int) -> str:
    transcript = _format_transcript(segments)
    
    response = claude_client.messages.create(
        model="claude-haiku-4-5",  # Cheaper model for the map step
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": f"""Extract key information from this portion of a meeting transcript (part {chunk_index + 1}).
List all: decisions made, action items assigned (with owners), open questions raised, attendee names mentioned.
Be concise but complete.

Transcript:
{transcript}"""
        }]
    )
    return response.content[0].text

def _combine_partial_summaries(partials: list[str], duration_seconds: float) -> dict:
    combined = "\n\n---\n\n".join(
        f"Part {i+1}:\n{p}" for i, p in enumerate(partials)
    )
    
    response = claude_client.messages.create(
        model="claude-opus-4-5",
        max_tokens=2048,
        system=SYSTEM_PROMPT,
        messages=[{
            "role": "user",
            "content": f"""Meeting duration: {duration_seconds/60:.1f} minutes

Below are summaries of sequential parts of the meeting. Combine them into the requested JSON structure, deduplicating where the same point appears in multiple parts.

{combined}"""
        }]
    )
    
    return json.loads(response.content[0].text.strip())

def _split_segments_by_time(segments: list[dict], chunk_duration_s: float) -> list[list[dict]]:
    chunks = []
    current_chunk = []
    chunk_start = 0.0
    
    for seg in segments:
        if seg["start"] - chunk_start >= chunk_duration_s and current_chunk:
            chunks.append(current_chunk)
            current_chunk = []
            chunk_start = seg["start"]
        current_chunk.append(seg)
    
    if current_chunk:
        chunks.append(current_chunk)
    
    return chunks

Using claude-haiku-4-5 for the map step and claude-opus-4-5 only for the final reduce step cuts costs by roughly 70% for long meetings.

Building the API layer

Wrap the pipeline in a FastAPI endpoint so any tool — Slack bot, calendar integration, or CI webhook — can trigger it:

from fastapi import FastAPI, UploadFile, BackgroundTasks, HTTPException
from fastapi.responses import JSONResponse
import uuid
import asyncio

app = FastAPI()
jobs: dict[str, dict] = {}  # In production, use Redis or Firestore

@app.post("/summarise")
async def start_summarisation(
    file: UploadFile,
    background_tasks: BackgroundTasks
):
    if not file.filename.endswith((".mp3", ".mp4", ".m4a", ".wav", ".webm")):
        raise HTTPException(400, "Unsupported audio format")
    
    job_id = str(uuid.uuid4())
    jobs[job_id] = {"status": "processing", "result": None, "error": None}
    
    # Save upload to temp file
    tmp_path = Path(f"/tmp/{job_id}_{file.filename}")
    content = await file.read()
    tmp_path.write_bytes(content)
    
    background_tasks.add_task(run_summarisation_job, job_id, tmp_path)
    
    return {"job_id": job_id, "status": "processing"}

@app.get("/summarise/{job_id}")
async def get_result(job_id: str):
    job = jobs.get(job_id)
    if not job:
        raise HTTPException(404, "Job not found")
    return job

async def run_summarisation_job(job_id: str, audio_path: Path):
    try:
        segments = transcribe_chunks(audio_path)
        duration = segments[-1]["end"] if segments else 0
        
        if duration > 30 * 60:  # > 30 minutes
            result = summarise_long_transcript(segments, duration)
        else:
            result = summarise_transcript(segments, duration)
        
        jobs[job_id] = {"status": "complete", "result": result, "error": None}
    except Exception as e:
        jobs[job_id] = {"status": "failed", "result": None, "error": str(e)}
    finally:
        audio_path.unlink(missing_ok=True)

This gives you a simple async API: POST the audio file, get back a job ID, poll until status is complete, then retrieve the structured summary.

Cost estimation and controls

For a 60-minute meeting:

• Whisper transcription: roughly 60 minutes × $0.006/minute = $0.36
• Claude summarisation (map-reduce): ~2000 tokens input to Haiku × 3 chunks + ~6000 tokens to Opus = approximately $0.05
• Total per meeting: around $0.41

This is cheap enough that most teams will not need aggressive cost controls. If you are processing hundreds of meetings per day, add a duration gate:

MAX_DURATION_MINUTES = 120

def validate_audio_duration(audio_path: Path) -> float:
    audio = AudioSegment.from_file(audio_path)
    duration_minutes = len(audio) / (1000 * 60)
    
    if duration_minutes > MAX_DURATION_MINUTES:
        raise ValueError(f"Audio is {duration_minutes:.0f} minutes; maximum is {MAX_DURATION_MINUTES}")
    
    return duration_minutes

You can also cache results keyed by audio file hash to avoid re-processing if the same recording is uploaded twice.

Delivering summaries to where people actually read them

A JSON API is necessary but not sufficient. People read summaries in Slack, Notion, or email — not by polling a REST endpoint. Add a delivery layer that formats the structured output and sends it where it will be seen:

def format_slack_message(summary: dict) -> dict:
    blocks = [
        {
            "type": "header",
            "text": {"type": "plain_text", "text": summary["title"]}
        },
        {
            "type": "section",
            "text": {"type": "mrkdwn", "text": summary["summary"]}
        }
    ]
    
    if summary["decisions"]:
        decision_text = "\n".join(
            f"• {d['decision']}" + (f" _(owner: {d['owner']})_" if d.get("owner") else "")
            for d in summary["decisions"]
        )
        blocks.append({
            "type": "section",
            "text": {"type": "mrkdwn", "text": f"*Decisions made:*\n{decision_text}"}
        })
    
    if summary["action_items"]:
        action_text = "\n".join(
            f"• {a['task']}" + (f" → {a['owner']}" if a.get("owner") else "")
            for a in summary["action_items"]
        )
        blocks.append({
            "type": "section",
            "text": {"type": "mrkdwn", "text": f"*Action items:*\n{action_text}"}
        })
    
    return {"blocks": blocks}

The combination of Whisper for transcription and Claude for structured extraction is significantly better than using either alone. Whisper reliably handles the acoustic challenges; Claude handles the semantic ones. The pipeline described here processes a 60-minute meeting in under three minutes end-to-end, costs less than fifty cents, and produces output that teams actually use to track decisions and accountability.