Multimodal AI: sending images and documents to Claude via API

Introduction to Claude's Multimodal Capabilities

Modern artificial intelligence extends far beyond simple text processing. Claude's multimodal architecture allows developers to send rich media—including high-resolution photographs, scanned documents, and multi-page PDFs—directly to the model via the Anthropic API. In supply chain management, logistics operations, and enterprise software engineering, this capability transforms how we parse unstructured data. Instead of writing brittle OCR (Optical Character Recognition) pipelines combined with heuristic text parsers, developers can transmit raw visual artifacts directly to Claude, leveraging its contextual understanding to extract line items, verify shipping manifests, or analyze warehouse layout diagrams.

Before diving into specific implementation details across different programming languages, it is crucial to understand the high-level use cases that drive multimodal API integration:

1. Document Digitization and Extraction: Processing legacy paper bills of lading, customs declarations, and commercial invoices directly into structured JSON objects without intermediate preprocessing steps.
2. Visual Quality Control: Analyzing high-resolution images of cargo damage, container seals, or packaging defects on the loading dock to automate exception handling in real-time.
3. Diagram and Blueprint Interpretation: Sending warehouse floor plans, routing diagrams, or network schematics to Claude to generate optimized spatial queries or infrastructure configurations.
4. Cross-Reference Auditing: Comparing physical manifest photos against digital database records to flag discrepancies instantly.

To achieve production-grade reliability when building these systems, developers must understand payload structures, base64 encoding limits, mime-type declarations, and error handling protocols. This article provides a comprehensive, code-first guide to sending images and documents to Claude via the API using Python and TypeScript.

Understanding Payload Structures and Base64 Encoding

When interacting with the Anthropic Messages API, multimodal inputs are passed inside the content array of a message object. Unlike text-only prompts where content is simply a string, multimodal prompts require a structured array containing blocks for both text instructions and media objects.

For images (such as JPEGs, PNGs, GIFs, and WebP files), the payload must specify the type as image, provide the source object containing the media_type, and supply the image data encoded in base64. For documents like PDFs, Claude accepts application/pdf mime types directly within the same block structure, allowing the model to parse multi-page documents seamlessly.

Consider the conceptual schema of an API request payload designed to process a damaged cargo photograph alongside a text prompt:

model: claude-3-5-sonnet-20241022
max_tokens: 1024
messages:
  - role: user
  - content:
      - type: image
        source:
          type: base64
          media_type: image/jpeg
          data: "/9j/4AAQSkZJRgABAQEASABIAAD..."
      - type: text
        text: "Analyze this cargo container seal. Is it intact or has it been tampered with? Return JSON with fields status and confidence."

Getting the base64 string right is critical. If your encoding includes data URL prefixes (such as data:image/jpeg;base64,), Claude's API will reject the request with a validation error. You must strip any prefix and transmit strictly the raw base64 string. Furthermore, large payloads require careful memory management on your application servers to prevent memory spikes when reading local files or fetching remote assets from cloud storage buckets like Amazon S3 or Google Cloud Storage.

Implementing Multimodal Requests in Python

Python remains the lingua franca for data-heavy applications, machine learning orchestration, and backend supply chain services. Using the official anthropic SDK, sending an image or a PDF requires reading the file into memory, encoding it, and constructing the message payload.

Below is a production-ready Python implementation that reads a local invoice PDF, encodes it to base64, and sends it to Claude with explicit instructions to extract line items into a structured schema:

import base64
import os
from anthropic import Anthropic

def analyze_shipping_document(file_path: str) -> str:
    client = Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))
    
    # Determine mime type based on extension
    _, ext = os.path.splitext(file_path)
    if ext.lower() == ".pdf":
        media_type = "application/pdf"
    elif ext.lower() in [".jpg", ".jpeg"]:
        media_type = "image/jpeg"
    elif ext.lower() == ".png":
        media_type = "image/png"
    else:
        raise ValueError(f"Unsupported file type: {ext}")

    # Read and base64 encode the file
    try:
        with open(file_path, "rb") as f:
            file_data = f.read()
            base64_data = base64.b64encode(file_data).decode("utf-8")
    except Exception as e:
        raise RuntimeError(f"Failed to read file at {file_path}: {e}")

    # Construct the API request
    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=2048,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "document",
                        "source": {
                            "type": "base64",
                            "media_type": media_type,
                            "data": base64_data,
                        },
                    },
                    {
                        "type": "text",
                        "text": "Extract all shipment line items, tracking numbers, and total weights from this document. Output the result strictly as valid JSON."
                    }
                ],
            }
        ],
    0)

    return message.content[0].text

if __name__ == "__main__":
    result = analyze_shipping_document("sample_invoice.pdf")
    print(result)

This script handles MIME type detection cleanly and wraps file operations in explicit exception handlers. When integrating this into an asynchronous web framework like FastAPI or Celery workers, ensure that network timeouts are configured appropriately, as large multi-page PDF documents take longer for the model to process than simple text prompts.

Implementing Multimodal Requests in TypeScript

For full-stack TypeScript applications, enterprise dashboards, and Node.js microservices, interacting with Claude follows a similarly structured pattern using the @anthropic-ai/sdk package. Type safety ensures that media sources and message blocks conform to the exact expectations of the API client.

Here is an implementation in TypeScript that reads a JPEG image from disk using Node's standard fs/promises module and sends it to Claude for visual inspection:

import Anthropic from '@anthropic-ai/sdk';
import * as fs from 'fs/promises';
import * as path from 'path';

async function inspectWarehouseImage(filePath: string): Promise<string> {
  const anthropic = new Anthropic({
    apiKey: process.env.ANTHROPIC_API_KEY,
  });

  const extension = path.extname(filePath).toLowerCase();
  let mediaType: 'image/jpeg' | 'image/png' | 'image/webp' | 'image/gif';

  if (extension === '.jpg' || extension === '.jpeg') {
    mediaType = 'image/jpeg';
  } else if (extension === '.png') {
    mediaType = 'image/png';
  } else if (extension === '.webp') {
    mediaType = 'image/webp';
  } else {
    throw new Error(`Unsupported image format: ${extension}`);
  }

  // Read file into buffer and convert to base64
  const fileBuffer = await fs.readFile(filePath);
  const base64Data = fileBuffer.toString('base64');

  const response = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 1024,
    messages: [
      {
        role: 'user',
        content: [
          {
            type: 'image',
            source: {
              type: 'base64',
              media_type: mediaType,
              data: base64Data,
            },
          },
          {
            type: 'text',
            text: 'Examine this warehouse aisle photo. Identify any safety hazards, blocked fire exits, or improperly stacked pallets. Provide a bulleted summary.',
          },
        ],
      },
    ],
  });

  const textBlock = response.content.find(block => block.type === 'text');
  if (!textBlock || textBlock.type !== 'text') {
    throw new Error('Expected text response from Claude');
  }

  return textBlock.text;
}

// Execution example
inspectWarehouseImage('./warehouse_aisle_4.jpg')
  .then(console.log)
  .catch(console.error);

By leveraging TypeScript interfaces provided directly by the Anthropic SDK, developers avoid runtime payload malformations and ensure robust compilation checks before deploying service updates to production environments.

Error Handling, Rate Limits, and Best Practices

Deploying multimodal AI features into production demands rigorous attention to API rate limits, payload size constraints, and error management. Images and documents consume significantly more tokens than plain text prompts. Specifically, image token consumption scales with image dimensions; extremely large images should be resized client-side or server-side before encoding to optimize latency and reduce API costs.

When designing resilient integration layers, keep the following engineering principles in mind:

• Token Budgeting: Monitor input token counts rigorously. A multi-page PDF can easily consume thousands of tokens just for the visual representation, leaving fewer tokens available for the model's response if max_tokens is set too low.
• Graceful Retries: Implement exponential backoff for HTTP 429 (Rate Limit Exceeded) and HTTP 5xx (Server Error) responses from the Anthropic API.
• Input Sanitization: Validate file headers and extensions on the server side prior to base64 conversion to prevent malicious file uploads or unsupported binary streams from crashing downstream worker threads.
• Asynchronous Processing: For high-throughput document ingestion pipelines, offload API calls to background task queues (such as Celery, BullMQ, or AWS SQS) to keep web request threads unblocked.

Conclusion

Integrating multimodal capabilities into your software architecture via Claude's API bridges the gap between unstructured physical reality and structured digital workflows. By understanding the core payload structures, mastering base64 encoding mechanics in Python and TypeScript, and adhering to strict production best practices around rate limiting and error handling, developers can build powerful automated systems that process complex visual and document data with unprecedented accuracy.