mirror of https://github.com/fabro-sh/fabro.git synced 2026-09-07 08:27:12 +00:00

Bryan Helmkamp 5b7eabee8f Add twin test mode for OpenAI E2E tests

Integrate twin-openai (fake OpenAI server) into the workspace and wire
it into the e2e_test macro so OpenAI tests can run without real API
credentials. The twin server starts in-process via OnceLock on first use
and provides per-test isolation through bearer-token namespacing.

Changes:
- Add Twin as default TestMode, replacing Off (gating now via #[ignore])
- Extend #[e2e_test] macro with `twin` requirement for twin-only,
  live-only, and dual-mode (twin + live) test gating
- Add e2e_openai!() macro returning (base_url, api_key)
- Convert openai_complete and openai_gpt_5_3_codex_complete to dual-mode
- Add new openai_server_error twin-only test with scripted 500 error
- Standardize axum 0.8 as workspace dependency across all crates
- Relax twin-openai ResponsesRequest to accept unknown fields via flatten

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-31 20:36:48 -04:00

3 KiB

Raw Permalink Blame History

twin-openai

Async Rust fake OpenAI-compatible server for local black-box testing.

Endpoints

GET /healthz
POST /v1/responses
POST /v1/chat/completions
POST /__admin/scenarios
POST /__admin/reset
GET /__admin/requests

/v1/* routes require a non-empty bearer token. Scenarios, request logs, and deterministic response IDs are scoped by bearer token so concurrent test clients can share one server safely.

/__admin/* routes are unauthenticated by default, but an optional bearer token selects the same namespace as /v1/*. Admin requests with a malformed or empty Authorization header are rejected.

Run locally

cargo run

The server binds to 127.0.0.1:3000 by default.

Admin scripting

Load deterministic one-shot scenarios:

curl -X POST http://127.0.0.1:3000/__admin/scenarios \
  -H 'Authorization: Bearer suite-a' \
  -H 'content-type: application/json' \
  -d '{
    "scenarios": [
      {
        "matcher": { "endpoint": "responses", "model": "gpt-test", "stream": false },
        "script": { "kind": "success", "response_text": "scripted reply" }
      }
    ]
  }'

Inspect normalized request logs:

curl http://127.0.0.1:3000/__admin/requests \
  -H 'Authorization: Bearer suite-a'

Reset scenarios, logs, and deterministic counters:

curl -X POST http://127.0.0.1:3000/__admin/reset \
  -H 'Authorization: Bearer suite-a'

Behavior summary

Non-stream and stream success paths are driven from the same canonical response plan.
/v1/responses and /v1/chat/completions share the same deterministic fallback behavior.
Structured output supports json_object and a documented json_schema subset.
Scripted failures support OpenAI-shaped application errors, delays, hangs, partial streams, and malformed SSE.

Optional Live OpenAI Smoke Suite

Run the ignored live drift detector only when you explicitly want to compare twin-openai against the real OpenAI API:

OPENAI_API_KEY=... cargo test --test live_openai_contract -- --ignored --nocapture

Optional environment variables:

TWIN_OPENAI_LIVE_MODEL defaults to gpt-5-nano-2025-08-07
TWIN_OPENAI_LIVE_BASE_URL defaults to https://api.openai.com
OPENAI_ORGANIZATION and OPENAI_PROJECT are forwarded when present

This suite is not part of normal CI. It is intentionally a drift detector for request/response shape and SSE sequencing, so opt-in failures can represent real compatibility gaps rather than a broken local test harness.

If the supplied OpenAI credentials lack required endpoint scopes or quota, the ignored test will skip the blocked live surface instead of reporting protocol drift.

Current live coverage includes responses and chat.completions text, streaming, structured output, function tools, tool_choice: "none" behavior, image-input acceptance, and both non-stream and streamed responses continuation turns.

See docs/compatibility-matrix.md for the supported field matrix and explicit exclusions.

3 KiB Raw Permalink Blame History