How to use LLMs for automated test case generation

By Amina Coulibaly · 21 July 20260 views
#llm#test-generation#unit-testing#code-quality#automation

The problem with test generation that ignores context

Most engineers who have tried asking an LLM to write tests for their code have gotten something that compiles but misses the point. The model writes tests that verify trivial things, duplicates existing coverage, or constructs mocks that do not reflect how the dependency actually behaves. The failure mode is not that the LLM writes bad Python or TypeScript — it writes fine code. The failure mode is that it writes tests without understanding what matters.

Effective automated test generation requires giving the model three things it does not have by default: the code under test, the context of how that code is used, and information about what already exists. When you supply those, LLMs produce tests that are useful to keep around. When you do not, you get noise that pads coverage numbers without improving confidence.

This guide builds a practical test generation pipeline for TypeScript/Node.js projects. The patterns apply equally to Python and other typed languages.

Extracting the right context from source code

Before calling an LLM, you need to assemble the context it will use. The minimum useful context for generating tests for a function is:

  1. The function's source code
  2. Its type signatures (especially if they are richer than the implementation suggests)
  3. The types of any dependencies it uses
  4. Any existing tests for related functions (as style examples, not to duplicate)

For a TypeScript project, you can extract this programmatically using the TypeScript compiler API:

import ts from "typescript";
import { readFileSync } from "fs";

interface FunctionContext {
  name: string;
  sourceCode: string;
  signature: string;
  filePath: string;
  imports: string[];
  dependencyTypes: string[];
}

function extractFunctionContext(
  filePath: string,
  functionName: string
): FunctionContext {
  const sourceCode = readFileSync(filePath, "utf-8");
  const sourceFile = ts.createSourceFile(
    filePath,
    sourceCode,
    ts.ScriptTarget.Latest,
    true
  );

  let targetNode: ts.FunctionDeclaration | ts.ArrowFunction | null = null;
  let signature = "";

  ts.forEachChild(sourceFile, (node) => {
    if (
      ts.isFunctionDeclaration(node) &&
      node.name?.text === functionName
    ) {
      targetNode = node;
      signature = extractSignature(node, sourceCode);
    }
  });

  if (!targetNode) {
    throw new Error(`Function ${functionName} not found in ${filePath}`);
  }

  const imports = extractImports(sourceFile, sourceCode);
  const funcSource = sourceCode.slice(
    targetNode.pos,
    targetNode.end
  );

  return {
    name: functionName,
    sourceCode: funcSource,
    signature,
    filePath,
    imports,
    dependencyTypes: [],
  };
}

function extractSignature(
  node: ts.FunctionDeclaration,
  sourceCode: string
): string {
  const printer = ts.createPrinter();
  // Get just the signature (parameters and return type) without the body
  const bodyStart = node.body?.pos ?? node.end;
  return sourceCode.slice(node.pos, bodyStart).trim();
}

function extractImports(
  sourceFile: ts.SourceFile,
  sourceCode: string
): string[] {
  const imports: string[] = [];
  ts.forEachChild(sourceFile, (node) => {
    if (ts.isImportDeclaration(node)) {
      imports.push(sourceCode.slice(node.pos, node.end).trim());
    }
  });
  return imports;
}

For each dependency the function imports, resolve its type definitions. If it is a local module, include the relevant exported types. If it is an npm package, include the relevant parts of @types:

function resolveDependencyTypes(
  imports: string[],
  projectRoot: string
): string[] {
  const typeSnippets: string[] = [];

  for (const imp of imports) {
    // Extract module path from import statement
    const match = imp.match(/from ['"]([^'"]+)['"]/);
    if (!match) continue;

    const modulePath = match[1];

    if (modulePath.startsWith(".")) {
      // Local module - include exported types
      const resolved = resolveLocalModule(modulePath, projectRoot);
      if (resolved) {
        typeSnippets.push(extractExportedTypes(resolved));
      }
    }
    // For npm packages, rely on the model's training knowledge
    // but include the import so it knows what's being used
  }

  return typeSnippets;
}

This level of context extraction is more work than just copying the file, but it produces dramatically better tests because the model can reason about type constraints rather than guessing.

Designing the prompt for test generation

The prompt structure matters more than most people expect. Key elements:

  • Tell the model the testing framework and assertion library so it uses the right APIs
  • Provide the function context assembled above
  • Include 1-2 examples of existing tests in the project as style guides
  • Ask for specific test categories rather than "write some tests"
  • Request one test per message turn if you want to iterate
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

interface GeneratedTest {
  description: string;
  code: string;
  category: "happy_path" | "edge_case" | "error_case" | "boundary";
}

async function generateTests(
  context: FunctionContext,
  existingTestExample: string
): Promise<GeneratedTest[]> {
  const prompt = buildTestGenerationPrompt(context, existingTestExample);

  const response = await client.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 4096,
    system: `You are an expert TypeScript developer specialising in writing precise, 
maintainable unit tests. You write tests using Vitest and expect assertions. 
You prefer testing behaviour over implementation. You never use 'any' types in tests. 
You write descriptive test names that explain what the test verifies and under what conditions.`,
    messages: [{ role: "user", content: prompt }],
  });

  const raw = response.content[0].text;
  return parseGeneratedTests(raw);
}

function buildTestGenerationPrompt(
  context: FunctionContext,
  existingTestExample: string
): string {
  return `Generate unit tests for the following TypeScript function.

## Function to test

File: ${context.filePath}

\`\`\`typescript
${context.imports.join("\n")}

${context.sourceCode}
\`\`\`

## Existing test style (match this style exactly)

\`\`\`typescript
${existingTestExample}
\`\`\`

## Requirements

Generate tests in these four categories:
1. **Happy path** (2 tests): normal inputs that should succeed
2. **Edge cases** (2 tests): empty strings, zero values, null/undefined where applicable  
3. **Error cases** (2 tests): inputs that should throw or return error states
4. **Boundary conditions** (1 test): maximum/minimum values, exact thresholds

## Output format

Return a JSON array of test objects:
\`\`\`json
[
  {
    "description": "should return X when given Y",
    "category": "happy_path",
    "code": "it('should return X when given Y', () => { ... });"
  }
]
\`\`\`

Do not wrap tests in describe blocks — the caller will handle that. 
Do not include import statements — they will be added separately.
Return only the JSON array, no other text.`;
}

Asking for categories explicitly is better than asking for "comprehensive tests" because it gives the model a checklist to work through. Models asked for "comprehensive tests" tend to write 5 happy-path variations; models asked to cover specific categories generate more useful diversity.

Parsing and validating generated test code

LLMs generate tests that look correct but may have subtle errors: wrong assertion syntax, mock configurations that do not match the actual API, or test descriptions that do not match the test body. Validate before writing to disk.

import { parseTestCode, validateTestSyntax } from "./test-validator";

function parseGeneratedTests(raw: string): GeneratedTest[] {
  // Strip markdown code fences if present
  const cleaned = raw
    .replace(/^```json\n?/, "")
    .replace(/\n?```$/, "")
    .trim();

  const parsed = JSON.parse(cleaned) as GeneratedTest[];

  return parsed.filter((test) => {
    // Validate each test compiles
    try {
      validateTestSyntax(test.code);
      return true;
    } catch (e) {
      console.warn(`Skipping invalid test: ${test.description}`, e);
      return false;
    }
  });
}

function validateTestSyntax(code: string): void {
  // Use TypeScript compiler to check for syntax errors
  const result = ts.transpileModule(
    `import { it, expect, vi } from 'vitest';\n${code}`,
    {
      compilerOptions: {
        target: ts.ScriptTarget.ES2020,
        module: ts.ModuleKind.ESNext,
        strict: true,
      },
      reportDiagnostics: true,
    }
  );

  if (result.diagnostics && result.diagnostics.length > 0) {
    const errors = result.diagnostics
      .map((d) => ts.flattenDiagnosticMessageText(d.messageText, "\n"))
      .join("; ");
    throw new Error(`TypeScript errors: ${errors}`);
  }
}

Wrapping TypeScript compilation around the output catches maybe 15-20% of generated tests before they land in your codebase. The rest require human review, but having compilation errors caught automatically saves review time.

Assembling and writing the test file

Once you have validated tests, assemble them into a proper test file and write it next to the source:

import { writeFileSync } from "fs";
import path from "path";

function writeTestFile(
  context: FunctionContext,
  tests: GeneratedTest[],
  dryRun = false
): string {
  const testFilePath = context.filePath.replace(/\.ts$/, ".test.ts");

  const imports = buildTestImports(context);
  const describeBlock = buildDescribeBlock(context.name, tests);

  const fileContent = [
    "// Generated by test-gen — review before committing",
    "// Run: npx vitest run " + path.basename(testFilePath),
    "",
    imports,
    "",
    describeBlock,
  ].join("\n");

  if (!dryRun) {
    writeFileSync(testFilePath, fileContent, "utf-8");
    console.log(`Written: ${testFilePath}`);
  }

  return fileContent;
}

function buildTestImports(context: FunctionContext): string {
  const moduleSpecifier = "./" + path.basename(context.filePath, ".ts");
  return [
    `import { describe, it, expect, vi, beforeEach } from 'vitest';`,
    `import { ${context.name} } from '${moduleSpecifier}';`,
  ].join("\n");
}

function buildDescribeBlock(
  functionName: string,
  tests: GeneratedTest[]
): string {
  const groupedByCategory = groupBy(tests, "category");
  const sections: string[] = [];

  const categoryLabels: Record<string, string> = {
    happy_path: "happy path",
    edge_case: "edge cases",
    error_case: "error handling",
    boundary: "boundary conditions",
  };

  for (const [category, categoryTests] of Object.entries(groupedByCategory)) {
    const label = categoryLabels[category] ?? category;
    const testLines = categoryTests
      .map((t) => "  " + t.code.split("\n").join("\n  "))
      .join("\n\n");

    sections.push(`  describe('${label}', () => {\n${testLines}\n  });`);
  }

  return `describe('${functionName}', () => {\n${sections.join("\n\n")}\n});`;
}

function groupBy<T>(arr: T[], key: keyof T): Record<string, T[]> {
  return arr.reduce(
    (acc, item) => {
      const k = String(item[key]);
      (acc[k] ??= []).push(item);
      return acc;
    },
    {} as Record<string, T[]>
  );
}

The // Generated by test-gen comment at the top matters: it signals to reviewers that the test needs a read-through, and it makes it easy to find generated tests in git blame later.

Running the generator from the CLI

Wire it up as a CLI tool so developers can run it in their workflow:

// cli.ts
import { parseArgs } from "util";

const { values } = parseArgs({
  args: process.argv.slice(2),
  options: {
    file: { type: "string" },
    function: { type: "string" },
    "dry-run": { type: "boolean", default: false },
    "example-test": { type: "string" },
  },
});

async function main() {
  const filePath = values.file;
  const functionName = values.function;

  if (!filePath || !functionName) {
    console.error("Usage: tsx cli.ts --file src/utils.ts --function parseDate");
    process.exit(1);
  }

  const context = extractFunctionContext(filePath, functionName);
  const exampleTest = values["example-test"]
    ? readFileSync(values["example-test"], "utf-8")
    : getDefaultTestExample();

  console.log(`Generating tests for ${functionName} in ${filePath}...`);
  const tests = await generateTests(context, exampleTest);
  console.log(`Generated ${tests.length} tests`);

  const output = writeTestFile(context, tests, values["dry-run"]);

  if (values["dry-run"]) {
    console.log("\n--- Preview ---\n");
    console.log(output);
  }
}

main().catch(console.error);

Usage:

tsx cli.ts --file src/pricing/calculator.ts --function calculateDiscount

Integrating with CI to generate tests for uncovered functions

The highest-leverage use of this pipeline is not interactive use — it is automated generation for functions that have no tests at all. Add a CI job that finds uncovered functions and creates GitHub PRs with generated tests:

# .github/workflows/test-gen.yml
name: Generate missing tests

on:
  schedule:
    - cron: "0 9 * * 1"  # Monday mornings
  workflow_dispatch:

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Find uncovered functions
        run: |
          npx vitest run --coverage --reporter=json > coverage.json
          tsx scripts/find-uncovered-functions.ts coverage.json > uncovered.json
      
      - name: Generate tests
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          tsx scripts/generate-for-uncovered.ts uncovered.json
      
      - name: Create PR
        uses: peter-evans/create-pull-request@v6
        with:
          title: "test: generated tests for uncovered functions"
          body: "Auto-generated by test-gen. Review each test file carefully before approving."
          branch: test-gen/auto
          labels: generated,needs-review

The find-uncovered-functions.ts script parses the coverage JSON, finds functions below a threshold, and outputs them as structured data for the generation script.

Avoiding the pitfalls: what not to auto-generate

Some tests should never be auto-generated:

Integration tests against real services. Generated tests will mock things that should hit a real database or API. Integration tests need human design to decide what actually needs a real dependency versus a stub.

Tests for security-sensitive logic. Authentication, authorisation, and cryptographic code need tests that were designed by someone who understands the threat model. A generated test that passes gives false confidence.

Tests that are deleted immediately. If a function is a thin wrapper that will be refactored next sprint, generating tests for it creates churn. Limit generation to stable code.

The sweet spot is pure functions with well-defined types that perform data transformation, validation, or calculation. These are common, easy to generate good tests for, and the generated tests have a long useful life.

Automated test generation is not a replacement for test-driven development or code review. It is a tool for closing coverage gaps in the existing codebase without spending engineering hours on mechanical work. Used selectively — with proper context extraction, category-based prompting, and syntax validation — it consistently generates tests that are worth keeping.

Comments

No comments yet. Be the first!

Sign in to leave a comment.