Testing AI API Integrations

Testing AI API Integrations: Best Practices

Testing AI API integrations presents unique challenges. Responses are non-deterministic, quality is subjective, and external API dependencies make tests slow and expensive. Despite these challenges, rigorous testing is essential for shipping reliable AI-powered features. This guide covers practical strategies for testing every layer of your AI integration, from unit tests to production monitoring.

The Testing Pyramid for AI Applications

Adapt the traditional testing pyramid for AI-specific concerns:

1. Unit tests — Test prompt construction, response parsing, error handling, and business logic independently of the API.
2. Integration tests — Verify end-to-end request/response flow with mocked or recorded API responses.
3. Contract tests — Ensure your code handles all response formats the API can return.
4. Evaluation tests — Measure response quality using automated scoring on a curated dataset.
5. Production monitoring — Continuously validate response quality and performance in production.

Unit Testing: Prompt Construction and Response Parsing

The most testable parts of your AI integration are prompt construction and response parsing. Test these thoroughly without making API calls:

// tests/promptBuilder.test.js
import { buildTranslationPrompt, parseTranslationResponse } from '../src/ai/translation';

describe('Translation Prompt Builder', () => {
  test('includes source and target language', () => {
    const prompt = buildTranslationPrompt({
      text: 'Hello world',
      sourceLang: 'English',
      targetLang: 'French',
      tone: 'formal'
    });

    expect(prompt.system).toContain('English');
    expect(prompt.system).toContain('French');
    expect(prompt.system).toContain('formal');
  });

  test('handles special characters in input', () => {
    const prompt = buildTranslationPrompt({
      text: 'Price: $100 & "quoted"',
      sourceLang: 'English',
      targetLang: 'Spanish'
    });

    expect(prompt.messages[0].content).toContain('$100');
  });
});

describe('Response Parser', () => {
  test('extracts translation from valid response', () => {
    const mockResponse = {
      content: [{ type: 'text', text: 'Bonjour le monde' }],
      usage: { input_tokens: 50, output_tokens: 10 }
    };

    const result = parseTranslationResponse(mockResponse);
    expect(result.translation).toBe('Bonjour le monde');
    expect(result.tokenUsage.total).toBe(60);
  });

  test('handles empty response gracefully', () => {
    const mockResponse = { content: [], usage: { input_tokens: 50, output_tokens: 0 } };
    expect(() => parseTranslationResponse(mockResponse)).toThrow('Empty response');
  });
});

Integration Testing with Recorded Responses

Use recorded API responses to create deterministic integration tests that do not require live API calls:

// Record real responses during development
import { setupRecording, loadRecording } from './testUtils';

describe('AI Translation Service', () => {
  let mockClient;

  beforeEach(() => {
    // Load pre-recorded API responses
    mockClient = createMockClient(
      loadRecording('translation-english-to-french')
    );
  });

  test('translates simple text correctly', async () => {
    const result = await translateText(mockClient, {
      text: 'Good morning',
      targetLang: 'French'
    });

    expect(result.translation).toBeTruthy();
    expect(result.translation.length).toBeGreaterThan(0);
    expect(result.cost).toBeGreaterThan(0);
  });

  test('handles rate limit errors with retry', async () => {
    const rateLimitClient = createMockClient([
      { status: 429, headers: { 'retry-after': '1' } },
      loadRecording('translation-english-to-french')
    ]);

    const result = await translateText(rateLimitClient, {
      text: 'Good morning',
      targetLang: 'French'
    });

    expect(result.translation).toBeTruthy();
    expect(result.retryCount).toBe(1);
  });
});

Evaluation Testing: Measuring Quality

The most important and most challenging type of AI testing is evaluating response quality. Build an evaluation framework with curated test cases:

// evaluation/run-eval.js
const TEST_CASES = [
  {
    input: 'What is photosynthesis?',
    criteria: {
      containsKeyTerms: ['chlorophyll', 'sunlight', 'carbon dioxide', 'glucose'],
      minLength: 100,
      maxLength: 500,
      readabilityLevel: 'middle-school'
    }
  },
  {
    input: 'Explain quantum entanglement simply',
    criteria: {
      containsKeyTerms: ['particles', 'connected', 'measurement'],
      noJargon: ['eigenstate', 'Hilbert space', 'superposition operator'],
      minLength: 80
    }
  }
];

async function runEvaluation(model) {
  const results = [];

  for (const testCase of TEST_CASES) {
    const response = await callApi(model, testCase.input);
    const score = evaluateResponse(response, testCase.criteria);
    results.push({ input: testCase.input, score, response });
  }

  const avgScore = results.reduce((s, r) => s + r.score, 0) / results.length;
  console.log(`Model: ${model}, Average Score: ${avgScore.toFixed(2)}`);
  return results;
}

Testing Error Handling

AI APIs fail in specific ways. Test your error handling for each failure mode:

• 429 Rate Limit — Verify retry logic with exponential backoff works correctly.
• 500 Server Error — Ensure graceful degradation and user-friendly error messages.
• 529 Overloaded — Test queuing and retry behavior specific to Anthropic's overload response.
• Timeout — Verify that long-running requests are properly cancelled and resources cleaned up.
• Malformed response — Test JSON parsing failures, truncated responses, and unexpected formats.
• Network errors — Simulate connection drops, DNS failures, and TLS errors.

Testing with Multiple Models

If your application supports multiple AI providers (which it should for resilience), test across all of them:

• Maintain a compatibility matrix of prompt behavior across models.
• Run your evaluation suite against each model quarterly to catch regressions.
• Test failover scenarios where the primary provider is unavailable.
• Verify that prompt templates produce acceptable results across all supported models.

A relay service like claude4u.com simplifies multi-model testing by providing a single endpoint that routes to different providers. This lets you run your test suite against various models without managing multiple API keys and SDK configurations.

Continuous Quality Monitoring

In production, set up continuous monitoring to catch quality degradation early:

• Sample and score a percentage of production responses daily.
• Track user feedback signals (thumbs up/down, regeneration requests, conversation abandonment).
• Alert when quality scores drop below established baselines.
• Maintain a dashboard showing quality trends over time by feature and model.

Testing AI integrations requires a shift in mindset from traditional testing. Accept non-determinism, invest in evaluation frameworks, and prioritize production monitoring. The goal is not to guarantee identical outputs, but to maintain consistent quality within acceptable bounds.