feat(tests): vendor example_openai_endpoint as a local mock server

Adds a vendored copy of BerriAI/example_openai_endpoint under
tests/mock_endpoints/example_openai_endpoint/ together with infrastructure
to run it in-process during a test session, so CI no longer depends on the
public Railway deployment (which has had multiple outages, e.g. the May 19
2026 GCP-account incident that took down 8 CI jobs).

Pieces added:

- tests/mock_endpoints/example_openai_endpoint/  vendored upstream source
  (main.py, batch_and_files_api.py, requirements.txt, Dockerfile),
  intended to be re-vendored as-is from upstream when needed (see README).
- tests/mock_endpoints/__init__.py              helper module exposing
  MOCK_OPENAI_BASE_URL (reads LITELLM_MOCK_OPENAI_BASE_URL with the
  Railway URL as fallback) and start_mock_server() — boots the server as
  a subprocess on a free port, polls /chat/completions until it responds,
  returns an RAII-style handle for teardown.
- tests/mock_endpoints/conftest.py              session-scoped pytest
  fixture mock_openai_endpoint_server that any suite can opt into via
  pytest_plugins.
- tests/mock_endpoints/start_mock_server.sh     bash launcher (foreground
  or --background) for shell-driven workflows.
- tests/mock_endpoints/README.md                local + CI usage docs and
  instructions for re-vendoring from upstream.
- tests/test_litellm/mock_endpoints/            end-to-end smoke tests
  that boot the mock and exercise /chat/completions, /v1/embeddings, and
  the fine-tuning list endpoint.

Wiring:

- New mock-server dependency-group in pyproject.toml (just adds slowapi;
  the rest of the upstream deps already come in via dev/proxy-dev).
- _test-unit-base.yml now syncs --group mock-server so any reusable unit
  test job has the deps available.
- test-unit-misc.yml runs the new smoke tests as part of its path set.
- Makefile gains 'make mock-server' (and an install-mock-server helper)
  for running the server locally with the project venv.

The Railway URL is still the runtime fallback for any test that has not
been migrated, so this PR is non-breaking; follow-up PRs can migrate
hard-coded URLs to use MOCK_OPENAI_BASE_URL / the new fixture.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

.github/workflows/_test-unit-base.yml

@@ -68,7 +68,7 @@ jobs:
 
       - name: Install dependencies
         run: |
-          uv sync --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
+          uv sync --frozen --group ci --group proxy-dev --group mock-server --extra google --extra proxy --extra semantic-router
 
       - name: Generate Prisma client
         env:

.github/workflows/test-unit-misc.yml

@@ -30,6 +30,7 @@ jobs:
         tests/test_litellm/experimental_mcp_client
         tests/test_litellm/images
         tests/test_litellm/interactions
+        tests/test_litellm/mock_endpoints
         tests/test_litellm/passthrough
         tests/test_litellm/vector_stores
         tests/test_litellm/test_*.py

Makefile

@@ -5,8 +5,8 @@
 	test-unit-integrations test-unit-core-utils test-unit-other test-unit-root \
 	test-proxy-unit-a test-proxy-unit-b test-integration test-unit-helm \
 	info lint lint-dev format \
-	install-dev install-proxy-dev install-test-deps \
-	install-helm-unittest check-circular-imports check-import-safety
+	install-dev install-proxy-dev install-test-deps install-mock-server \
+	mock-server install-helm-unittest check-circular-imports check-import-safety
 
 # Default target
 help:
@@ -17,6 +17,7 @@ help:
 	@echo "  make install-proxy-dev-ci - Install proxy dev dependencies (CI-compatible)"
 	@echo "  make install-test-deps  - Install the full local test environment"
 	@echo "  make install-helm-unittest - Install helm unittest plugin"
+	@echo "  make mock-server        - Run the vendored mock OpenAI endpoint locally (PORT=8090)"
 	@echo "  make format             - Apply Black code formatting"
 	@echo "  make format-check       - Check Black code formatting (matches CI)"
 	@echo "  make lint               - Run all linting (Ruff, MyPy, Black check, circular imports, import safety)"
@@ -68,6 +69,16 @@ install-test-deps: install-proxy-dev
 install-helm-unittest:
 	helm plugin install https://github.com/helm-unittest/helm-unittest --version v0.4.4 || echo "ignore error if plugin exists"
 
+# Mock OpenAI/Anthropic/Vertex endpoint server (vendored copy of
+# BerriAI/example_openai_endpoint). Use this to run tests offline / without
+# relying on the Railway-hosted deployment.
+install-mock-server:
+	$(UV) sync --group mock-server --group proxy-dev --extra proxy
+
+mock-server: install-mock-server
+	@echo "Starting mock OpenAI endpoint on http://0.0.0.0:$${PORT:-8090} (Ctrl+C to stop)"
+	$(UV_RUN) python tests/mock_endpoints/example_openai_endpoint/main.py
+
 # Formatting
 format: install-dev
 	cd litellm && $(UV_RUN) black . && cd ..

pyproject.toml

@@ -216,6 +216,13 @@ healthcheck = [
     "httpx==0.28.1",
     "pyyaml==6.0.3",
 ]
+mock-server = [
+    # Extra deps required by the vendored mock OpenAI endpoint under
+    # tests/mock_endpoints/example_openai_endpoint/. Most of its deps (fastapi,
+    # uvicorn, pydantic, python-multipart, python-dotenv, websockets) are
+    # already pulled in by the proxy / dev groups; slowapi is the one extra.
+    "slowapi==0.1.9",
+]
 
 [build-system]
 requires = ["uv_build==0.11.8"]

tests/mock_endpoints/README.md

@@ -0,0 +1,113 @@
+# Mock LLM Endpoints (for local dev + CI)
+
+This directory contains small, self-contained mock servers used by tests to
+avoid hitting real provider APIs (and to avoid relying on third-party
+hosting services like Railway).
+
+## `example_openai_endpoint/`
+
+A vendored copy of [BerriAI/example_openai_endpoint](https://github.com/BerriAI/example_openai_endpoint),
+a FastAPI app that implements stub OpenAI- / Anthropic- / Vertex- / Bedrock-
+compatible endpoints (chat completions, embeddings, audio, batches, etc.).
+Many tests across the repo reference its public deployment at
+`https://exampleopenaiendpoint-production.up.railway.app/`.
+
+The Railway deployment has had multiple outages
+(see <https://status.railway.com/historical>) that break CI. The goal of
+vendoring it here is to make the same stub server runnable in the same
+container as CI (and locally on a developer machine), so tests do not depend
+on any external service being up.
+
+### Running locally
+
+The easiest way to start the server (installs `slowapi` + friends into the
+project venv first, then runs the server on `:8090`):
+
+```bash
+make mock-server          # foreground
+PORT=18090 make mock-server   # alt port
+```
+
+If you don't want to use the Makefile, you can run the server directly with
+any Python interpreter that has the deps installed:
+
+```bash
+python -m pip install -r tests/mock_endpoints/example_openai_endpoint/requirements.txt
+python tests/mock_endpoints/example_openai_endpoint/main.py
+```
+
+Or use the bash helper (handy for CI / shell scripts — it starts the server
+in the background, polls until it's ready, and writes logs to `/tmp`):
+
+```bash
+./tests/mock_endpoints/start_mock_server.sh                # foreground
+./tests/mock_endpoints/start_mock_server.sh --background   # background
+```
+
+### Using it from a pytest suite
+
+The recommended pattern is the session-scoped fixture defined in
+[`tests/mock_endpoints/conftest.py`](./conftest.py). Any test file (or a
+suite-level `conftest.py`) can opt in like so:
+
+```python
+import pytest
+
+pytest_plugins = ("tests.mock_endpoints.conftest",)
+
+
+def test_something(mock_openai_endpoint_server):
+    base_url = mock_openai_endpoint_server  # e.g. "http://127.0.0.1:53892"
+    ...
+```
+
+The fixture:
+
+- Picks a free port automatically (so suites can run in parallel without
+  colliding).
+- Boots the vendored server as a subprocess.
+- Waits until `/chat/completions` returns 200 before yielding the URL.
+- Exposes the URL as `LITELLM_MOCK_OPENAI_BASE_URL` in the environment so
+  any code that calls `tests.mock_endpoints.MOCK_OPENAI_BASE_URL` picks it
+  up automatically.
+- Tears the subprocess down at session end.
+
+See [`tests/test_litellm/mock_endpoints/test_mock_openai_endpoint_server.py`](../test_litellm/mock_endpoints/test_mock_openai_endpoint_server.py)
+for a working end-to-end example.
+
+### Quick sanity check
+
+```bash
+curl -s http://127.0.0.1:8090/chat/completions \
+  -H 'Authorization: Bearer sk-test' \
+  -H 'Content-Type: application/json' \
+  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'
+```
+
+### Pointing tests at the local server
+
+Tests that currently hard-code
+`https://exampleopenaiendpoint-production.up.railway.app/` can be pointed at
+the local mock by changing the URL to `http://127.0.0.1:8090/`. New tests
+should prefer this local mock over the Railway deployment.
+
+### Keeping the vendored copy in sync
+
+The upstream source of truth is still
+<https://github.com/BerriAI/example_openai_endpoint>. To pull the latest
+version into this repo:
+
+```bash
+curl -fsSL -o tests/mock_endpoints/example_openai_endpoint/main.py \
+  https://raw.githubusercontent.com/BerriAI/example_openai_endpoint/main/main.py
+curl -fsSL -o tests/mock_endpoints/example_openai_endpoint/batch_and_files_api.py \
+  https://raw.githubusercontent.com/BerriAI/example_openai_endpoint/main/batch_and_files_api.py
+curl -fsSL -o tests/mock_endpoints/example_openai_endpoint/requirements.txt \
+  https://raw.githubusercontent.com/BerriAI/example_openai_endpoint/main/requirements.txt
+curl -fsSL -o tests/mock_endpoints/example_openai_endpoint/Dockerfile \
+  https://raw.githubusercontent.com/BerriAI/example_openai_endpoint/main/Dockerfile
+```
+
+These files should be copied **as-is** from upstream so the two repos stay
+in sync; do not edit them in place. If an endpoint needs to change, change
+it upstream first, then re-vendor.

tests/mock_endpoints/__init__.py

@@ -0,0 +1,155 @@
+"""Helpers for running the vendored mock OpenAI / Anthropic / Vertex endpoint.
+
+The mock server itself lives in ``tests/mock_endpoints/example_openai_endpoint/``
+and is a vendored copy of https://github.com/BerriAI/example_openai_endpoint.
+
+Many tests in this repo currently point at the Railway-hosted version of that
+server (``https://exampleopenaiendpoint-production.up.railway.app``). Railway
+outages take those tests down with it. This module exposes:
+
+* :data:`MOCK_OPENAI_BASE_URL` — the URL tests should hit. It reads
+  ``LITELLM_MOCK_OPENAI_BASE_URL`` from the environment, falling back to the
+  public Railway URL for backwards-compatibility with tests that have not been
+  migrated yet.
+* :func:`start_mock_server` — spawn the vendored server as a subprocess on a
+  free port. Used by the pytest fixture in ``conftest.py`` but also callable
+  directly from scripts.
+
+The pytest fixture ``mock_openai_endpoint_server`` (see ``conftest.py``) is
+the recommended way for a test suite to opt in: it starts the server once per
+session, exposes the URL, and tears the process down on exit.
+"""
+
+from __future__ import annotations
+
+import os
+import socket
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Optional
+
+import urllib.error
+import urllib.request
+
+DEFAULT_REMOTE_URL = "https://exampleopenaiendpoint-production.up.railway.app"
+
+_SERVER_DIR = Path(__file__).resolve().parent / "example_openai_endpoint"
+_SERVER_MAIN = _SERVER_DIR / "main.py"
+
+
+def _resolve_base_url() -> str:
+    url = os.environ.get("LITELLM_MOCK_OPENAI_BASE_URL")
+    if url:
+        return url.rstrip("/")
+    return DEFAULT_REMOTE_URL
+
+
+MOCK_OPENAI_BASE_URL = _resolve_base_url()
+
+
+def _pick_free_port() -> int:
+    """Bind to port 0 and immediately release, returning the kernel-assigned port."""
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
+        sock.bind(("127.0.0.1", 0))
+        return sock.getsockname()[1]
+
+
+def _wait_for_ready(url: str, timeout: float = 30.0) -> None:
+    """Poll the mock until ``/chat/completions`` returns a 2xx response."""
+    deadline = time.monotonic() + timeout
+    last_err: Optional[BaseException] = None
+    payload = b'{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}'
+    while time.monotonic() < deadline:
+        try:
+            req = urllib.request.Request(
+                f"{url}/chat/completions",
+                data=payload,
+                headers={
+                    "Authorization": "Bearer sk-test",
+                    "Content-Type": "application/json",
+                },
+                method="POST",
+            )
+            with urllib.request.urlopen(req, timeout=2.0) as resp:
+                if 200 <= resp.status < 300:
+                    return
+        except (urllib.error.URLError, ConnectionError, OSError) as err:
+            last_err = err
+        time.sleep(0.2)
+    raise RuntimeError(
+        f"Mock server at {url} did not become ready within {timeout}s "
+        f"(last error: {last_err!r})"
+    )
+
+
+class MockServerHandle:
+    """Tiny RAII-style handle for a running mock server subprocess."""
+
+    def __init__(self, process: subprocess.Popen, base_url: str) -> None:
+        self.process = process
+        self.base_url = base_url
+
+    def stop(self) -> None:
+        if self.process.poll() is not None:
+            return
+        self.process.terminate()
+        try:
+            self.process.wait(timeout=10)
+        except subprocess.TimeoutExpired:
+            self.process.kill()
+            self.process.wait(timeout=5)
+
+
+def start_mock_server(
+    port: Optional[int] = None,
+    *,
+    wait: bool = True,
+    timeout: float = 30.0,
+    log_file: Optional[Path] = None,
+    python_executable: Optional[str] = None,
+) -> MockServerHandle:
+    """Start the vendored mock server as a child process.
+
+    Args:
+        port: Port to bind. ``None`` (default) picks a free port automatically.
+        wait: If ``True``, block until ``/chat/completions`` returns a 2xx response.
+        timeout: Maximum seconds to wait for readiness.
+        log_file: Optional file to capture stdout/stderr. ``None`` inherits the parent's streams.
+        python_executable: Python interpreter to launch the server with. Defaults to ``sys.executable``.
+    """
+    if port is None:
+        port = _pick_free_port()
+
+    env = {**os.environ, "PORT": str(port)}
+    stdout = open(log_file, "w") if log_file is not None else None
+    stderr = subprocess.STDOUT if stdout is not None else None
+
+    process = subprocess.Popen(
+        [python_executable or sys.executable, str(_SERVER_MAIN)],
+        env=env,
+        stdout=stdout,
+        stderr=stderr,
+        cwd=str(_SERVER_DIR),
+    )
+    base_url = f"http://127.0.0.1:{port}"
+    handle = MockServerHandle(process=process, base_url=base_url)
+
+    if wait:
+        try:
+            _wait_for_ready(base_url, timeout=timeout)
+        except Exception:
+            handle.stop()
+            if stdout is not None:
+                stdout.close()
+            raise
+    return handle
+
+
+__all__ = [
+    "DEFAULT_REMOTE_URL",
+    "MOCK_OPENAI_BASE_URL",
+    "MockServerHandle",
+    "start_mock_server",
+]

tests/mock_endpoints/conftest.py

@@ -0,0 +1,54 @@
+"""Pytest fixtures for the vendored mock OpenAI endpoint.
+
+Any test suite can opt into a per-session local mock server like so::
+
+    pytest_plugins = ("tests.mock_endpoints.conftest",)
+
+    def test_something(mock_openai_endpoint_server):
+        url = mock_openai_endpoint_server  # e.g. "http://127.0.0.1:53892"
+        ...
+
+This avoids hitting the Railway-hosted deployment of
+``BerriAI/example_openai_endpoint``, which has historically been a source of
+flaky CI when Railway has incidents.
+"""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Iterator
+
+import pytest
+
+from . import start_mock_server
+
+
+@pytest.fixture(scope="session")
+def mock_openai_endpoint_server(
+    tmp_path_factory: pytest.TempPathFactory,
+) -> Iterator[str]:
+    """Session-scoped fixture that boots the vendored mock server.
+
+    Yields the base URL (e.g. ``http://127.0.0.1:53892``). The server is
+    killed at session teardown. Server logs are written to
+    ``<pytest tmp>/mock_openai_endpoint.log`` for debugging.
+    """
+    log_dir: Path = tmp_path_factory.mktemp("mock_openai_endpoint")
+    log_file = log_dir / "mock_openai_endpoint.log"
+
+    handle = start_mock_server(log_file=log_file)
+
+    # Expose the URL to any code that reads the env var (this is what the
+    # ``MOCK_OPENAI_BASE_URL`` helper checks). Restore the previous value on
+    # teardown so we don't leak it across test sessions.
+    previous = os.environ.get("LITELLM_MOCK_OPENAI_BASE_URL")
+    os.environ["LITELLM_MOCK_OPENAI_BASE_URL"] = handle.base_url
+    try:
+        yield handle.base_url
+    finally:
+        if previous is None:
+            os.environ.pop("LITELLM_MOCK_OPENAI_BASE_URL", None)
+        else:
+            os.environ["LITELLM_MOCK_OPENAI_BASE_URL"] = previous
+        handle.stop()

tests/mock_endpoints/example_openai_endpoint/Dockerfile

@@ -0,0 +1,20 @@
+# Use the official Python image as the base image
+FROM python:3.9-slim
+
+# Set the working directory in the container
+WORKDIR /app
+
+# Copy the Python requirements file
+COPY requirements.txt .
+
+# Install the Python dependencies
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy the application code
+COPY . .
+
+# Expose the port the app will run on
+EXPOSE 8090
+
+# Start the application
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8090"]

tests/mock_endpoints/example_openai_endpoint/batch_and_files_api.py

@@ -0,0 +1,310 @@
+from fastapi import APIRouter, File, Form, HTTPException, UploadFile
+from pydantic import BaseModel
+from typing import Dict, Literal, Optional
+import uuid
+import hashlib
+from datetime import datetime
+
+router = APIRouter()
+
+
+# ============= Models =============
+
+class FileObject(BaseModel):
+    id: str
+    object: Literal["file"] = "file"
+    bytes: int
+    created_at: int
+    filename: str
+    purpose: str
+    status: str
+
+
+class BatchRequestCounts(BaseModel):
+    total: int
+    completed: int
+    failed: int
+
+
+class BatchObject(BaseModel):
+    id: str
+    object: Literal["batch"] = "batch"
+    endpoint: str
+    errors: Optional[Dict] = None
+    input_file_id: str
+    completion_window: str
+    status: Literal["validating", "failed", "in_progress", "finalizing", "completed", "expired", "cancelling", "cancelled"]
+    output_file_id: Optional[str] = None
+    error_file_id: Optional[str] = None
+    created_at: int
+    in_progress_at: Optional[int] = None
+    expires_at: Optional[int] = None
+    finalizing_at: Optional[int] = None
+    completed_at: Optional[int] = None
+    failed_at: Optional[int] = None
+    expired_at: Optional[int] = None
+    cancelling_at: Optional[int] = None
+    cancelled_at: Optional[int] = None
+    request_counts: Optional[BatchRequestCounts] = None
+    metadata: Optional[Dict[str, str]] = None
+
+
+class CreateBatchRequest(BaseModel):
+    input_file_id: str
+    endpoint: Literal["/v1/chat/completions", "/v1/embeddings", "/v1/completions"]
+    completion_window: Literal["24h"]
+    metadata: Optional[Dict[str, str]] = None
+    status: str
+
+
+# ============= Files Endpoints =============
+
+@router.post("/files", response_model=FileObject)
+async def create_file(
+    file: UploadFile = File(...),
+    purpose: str = Form(...)
+):
+    """
+    Upload a file that can be used for batch processing.
+    
+    Compatible with: https://platform.openai.com/docs/api-reference/files/create
+    """
+    content = await file.read()
+    
+    # Generate consistent file ID based on filename and content
+    content_hash = hashlib.md5(f"{file.filename}{len(content)}".encode()).hexdigest()[:8]
+    file_id = f"file-{content_hash}"
+    
+    return FileObject(
+        id=file_id,
+        bytes=len(content),
+        created_at=int(datetime.now().timestamp()),
+        filename=file.filename or "uploaded_file",
+        purpose=purpose,
+        status="completed"
+    )
+
+
+@router.get("/files/{file_id}", response_model=FileObject)
+async def retrieve_file(file_id: str):
+    """
+    Returns information about a specific file.
+    
+    Compatible with: https://platform.openai.com/docs/api-reference/files/retrieve
+    """
+    # Return stubbed file information
+    return FileObject(
+        id=file_id,
+        bytes=1024,  # Stubbed file size
+        created_at=1698768000,  # Stubbed timestamp
+        filename="example_file.jsonl",
+        purpose="batch",
+        status="completed"
+    )
+
+
+@router.get("/files/{file_id}/content")
+async def retrieve_file_content(file_id: str):
+    """
+    Returns the contents of the specified file.
+    
+    Compatible with: https://platform.openai.com/docs/api-reference/files/retrieve-contents
+    """
+    # Return stubbed file content
+    stubbed_content = '{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello world"}]}}\n{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "How are you?"}]}}'
+    return stubbed_content
+
+
+@router.delete("/files/{file_id}")
+async def delete_file(file_id: str):
+    """
+    Delete a file.
+    
+    Compatible with: https://platform.openai.com/docs/api-reference/files/delete
+    """
+    # Return stubbed deletion response
+    return {"id": file_id, "object": "file", "deleted": True}
+
+
+# ============= Batches Endpoints =============
+
+@router.post("/batches", response_model=BatchObject)
+async def create_batch(request: CreateBatchRequest):
+    """
+    Creates and executes a batch from an uploaded file of requests.
+    
+    Compatible with: https://platform.openai.com/docs/api-reference/batch/create
+    """
+    # Generate consistent batch ID based on input file ID
+    batch_hash = hashlib.md5(request.input_file_id.encode()).hexdigest()[:8]
+    batch_id = f"batch_{batch_hash}"
+    created_at = int(datetime.now().timestamp())
+    
+    # Return stubbed batch object
+    return BatchObject(
+        id=batch_id,
+        object="batch",
+        endpoint=request.endpoint,
+        errors=None,
+        input_file_id=request.input_file_id,
+        completion_window=request.completion_window,
+        status="completed",
+        output_file_id=f"file-output-{batch_hash}",
+        error_file_id=None,
+        created_at=created_at,
+        in_progress_at=created_at + 10,
+        expires_at=created_at + 86400,  # 24 hours
+        finalizing_at=created_at + 300,
+        completed_at=created_at + 600,
+        failed_at=None,
+        expired_at=None,
+        cancelling_at=None,
+        cancelled_at=None,
+        request_counts=BatchRequestCounts(total=2, completed=2, failed=0),
+        metadata=request.metadata or {}
+    )
+
+
+@router.get("/batches/{batch_id}", response_model=BatchObject)
+async def retrieve_batch(batch_id: str):
+    """
+    Retrieves a batch.
+    
+    Compatible with: https://platform.openai.com/docs/api-reference/batch/retrieve
+    """
+    # Extract hash from batch_id for consistent output file ID
+    batch_hash = batch_id.split("_")[-1] if "_" in batch_id else "stubbed"
+    created_at = 1698768000  # Stubbed timestamp
+    
+    # Return stubbed batch object
+    return BatchObject(
+        id=batch_id,
+        object="batch",
+        endpoint="/v1/chat/completions",
+        errors=None,
+        input_file_id=f"file-{batch_hash}",
+        completion_window="24h",
+        status="completed",
+        output_file_id=f"file-output-{batch_hash}",
+        error_file_id=None,
+        created_at=created_at,
+        in_progress_at=created_at + 10,
+        expires_at=created_at + 86400,
+        finalizing_at=created_at + 300,
+        completed_at=created_at + 600,
+        failed_at=None,
+        expired_at=None,
+        cancelling_at=None,
+        cancelled_at=None,
+        request_counts=BatchRequestCounts(total=2, completed=2, failed=0),
+        metadata={}
+    )
+
+
+@router.post("/batches/{batch_id}/cancel", response_model=BatchObject)
+async def cancel_batch(batch_id: str):
+    """
+    Cancels an in-progress batch.
+    
+    Compatible with: https://platform.openai.com/docs/api-reference/batch/cancel
+    """
+    # Extract hash from batch_id for consistent output file ID
+    batch_hash = batch_id.split("_")[-1] if "_" in batch_id else "stubbed"
+    created_at = 1698768000  # Stubbed timestamp
+    cancelled_at = int(datetime.now().timestamp())
+    
+    # Return stubbed cancelled batch object
+    return BatchObject(
+        id=batch_id,
+        object="batch",
+        endpoint="/v1/chat/completions",
+        errors=None,
+        input_file_id=f"file-{batch_hash}",
+        completion_window="24h",
+        status="cancelled",
+        output_file_id=None,
+        error_file_id=None,
+        created_at=created_at,
+        in_progress_at=created_at + 10,
+        expires_at=created_at + 86400,
+        finalizing_at=None,
+        completed_at=None,
+        failed_at=None,
+        expired_at=None,
+        cancelling_at=cancelled_at - 5,
+        cancelled_at=cancelled_at,
+        request_counts=BatchRequestCounts(total=2, completed=0, failed=0),
+        metadata={}
+    )
+
+
+@router.get("/batches")
+async def list_batches(limit: int = 20, after: Optional[str] = None):
+    """
+    List your organization's batches.
+    
+    Compatible with: https://platform.openai.com/docs/api-reference/batch/list
+    """
+    # Return stubbed list of batches
+    stubbed_batches = [
+        BatchObject(
+            id="batch_example1",
+            object="batch",
+            endpoint="/v1/chat/completions",
+            errors=None,
+            input_file_id="file-example1",
+            completion_window="24h",
+            status="completed",
+            output_file_id="file-output-example1",
+            error_file_id=None,
+            created_at=1698768000,
+            in_progress_at=1698768010,
+            expires_at=1698854400,
+            finalizing_at=1698768300,
+            completed_at=1698768600,
+            failed_at=None,
+            expired_at=None,
+            cancelling_at=None,
+            cancelled_at=None,
+            request_counts=BatchRequestCounts(total=5, completed=5, failed=0),
+            metadata={}
+        ),
+        BatchObject(
+            id="batch_example2",
+            object="batch",
+            endpoint="/v1/embeddings",
+            errors=None,
+            input_file_id="file-example2",
+            completion_window="24h",
+            status="in_progress",
+            output_file_id=None,
+            error_file_id=None,
+            created_at=1698767000,
+            in_progress_at=1698767010,
+            expires_at=1698853400,
+            finalizing_at=None,
+            completed_at=None,
+            failed_at=None,
+            expired_at=None,
+            cancelling_at=None,
+            cancelled_at=None,
+            request_counts=BatchRequestCounts(total=3, completed=1, failed=0),
+            metadata={"project": "test"}
+        )
+    ]
+    
+    # Apply pagination logic to stubbed data
+    if after:
+        try:
+            start_idx = next(i for i, b in enumerate(stubbed_batches) if b.id == after) + 1
+            stubbed_batches = stubbed_batches[start_idx:]
+        except StopIteration:
+            pass
+    
+    stubbed_batches = stubbed_batches[:limit]
+    
+    return {
+        "object": "list",
+        "data": stubbed_batches,
+        "has_more": False  # Stubbed - no more data
+    }

tests/mock_endpoints/example_openai_endpoint/requirements.txt

@@ -0,0 +1,8 @@
+fastapi
+uvicorn[standard]
+slowapi
+uuid
+python-multipart>=0.0.20
+pydantic
+python-dotenv>=0.2.0 # for env 
+websockets 

tests/mock_endpoints/start_mock_server.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+#
+# Start the vendored example_openai_endpoint mock server.
+#
+# Usage:
+#   ./tests/mock_endpoints/start_mock_server.sh                 # foreground
+#   ./tests/mock_endpoints/start_mock_server.sh --background    # background, prints PID
+#
+# Environment:
+#   PORT                 - port to bind (default 8090)
+#   MOCK_SERVER_LOG_FILE - log file when running in background (default /tmp/mock_openai_endpoint.log)
+#   MOCK_SERVER_PID_FILE - PID file when running in background (default /tmp/mock_openai_endpoint.pid)
+#   MOCK_SERVER_TIMEOUT  - seconds to wait for /chat/completions to respond in --background (default 30)
+#
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+APP_DIR="${SCRIPT_DIR}/example_openai_endpoint"
+
+PORT="${PORT:-8090}"
+LOG_FILE="${MOCK_SERVER_LOG_FILE:-/tmp/mock_openai_endpoint.log}"
+PID_FILE="${MOCK_SERVER_PID_FILE:-/tmp/mock_openai_endpoint.pid}"
+TIMEOUT="${MOCK_SERVER_TIMEOUT:-30}"
+
+# Prefer the project's venv if it has the deps; otherwise fall back to system python3.
+# Callers can override with PYTHON_BIN=/path/to/python.
+REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+if [[ -z "${PYTHON_BIN:-}" ]]; then
+    if [[ -x "${REPO_ROOT}/.venv/bin/python" ]]; then
+        PYTHON_BIN="${REPO_ROOT}/.venv/bin/python"
+    else
+        PYTHON_BIN="python3"
+    fi
+fi
+
+if [[ "${1:-}" == "--background" ]]; then
+    PORT="${PORT}" nohup "${PYTHON_BIN}" "${APP_DIR}/main.py" >"${LOG_FILE}" 2>&1 &
+    PID=$!
+    echo "${PID}" >"${PID_FILE}"
+    echo "Started mock server: pid=${PID} port=${PORT} log=${LOG_FILE}"
+
+    # Wait until the server responds (or until we exhaust the timeout).
+    for _ in $(seq 1 "${TIMEOUT}"); do
+        if curl -fsS -o /dev/null \
+            -X POST "http://127.0.0.1:${PORT}/chat/completions" \
+            -H 'Authorization: Bearer sk-test' \
+            -H 'Content-Type: application/json' \
+            -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'; then
+            echo "Mock server is ready on http://127.0.0.1:${PORT}"
+            exit 0
+        fi
+        sleep 1
+    done
+
+    echo "Mock server failed to become ready within ${TIMEOUT}s. Logs:" >&2
+    tail -n 100 "${LOG_FILE}" >&2 || true
+    kill "${PID}" 2>/dev/null || true
+    exit 1
+fi
+
+exec env PORT="${PORT}" "${PYTHON_BIN}" "${APP_DIR}/main.py"

tests/test_litellm/mock_endpoints/test_mock_openai_endpoint_server.py

@@ -0,0 +1,73 @@
+"""End-to-end smoke tests for the vendored mock OpenAI endpoint.
+
+These tests boot the mock server in-process (via the session-scoped fixture
+in ``tests/mock_endpoints/conftest.py``) and verify that the key endpoints
+that production tests rely on actually respond correctly.
+
+If this test passes in CI we know the local mock works as a drop-in
+replacement for the Railway deployment for chat-completions, embeddings, and
+fine-tuning routes — which is the whole point of vendoring it.
+"""
+
+from __future__ import annotations
+
+import json
+import urllib.request
+
+import pytest
+
+pytest_plugins = ("tests.mock_endpoints.conftest",)
+
+
+def _post_json(url: str, payload: dict) -> dict:
+    req = urllib.request.Request(
+        url,
+        data=json.dumps(payload).encode("utf-8"),
+        headers={
+            "Authorization": "Bearer sk-test",
+            "Content-Type": "application/json",
+        },
+        method="POST",
+    )
+    with urllib.request.urlopen(req, timeout=10) as resp:
+        return json.loads(resp.read().decode("utf-8"))
+
+
+def _get_json(url: str) -> dict:
+    req = urllib.request.Request(url, method="GET")
+    with urllib.request.urlopen(req, timeout=10) as resp:
+        return json.loads(resp.read().decode("utf-8"))
+
+
+@pytest.mark.usefixtures("mock_openai_endpoint_server")
+def test_should_serve_openai_chat_completion(mock_openai_endpoint_server: str) -> None:
+    body = _post_json(
+        f"{mock_openai_endpoint_server}/chat/completions",
+        {
+            "model": "gpt-4o-mini",
+            "messages": [{"role": "user", "content": "hi"}],
+        },
+    )
+
+    assert body["object"] == "chat.completion"
+    assert body["model"] == "gpt-4o-mini"
+    assert body["choices"][0]["message"]["role"] == "assistant"
+    assert body["choices"][0]["message"]["content"]
+
+
+def test_should_serve_embeddings(mock_openai_endpoint_server: str) -> None:
+    body = _post_json(
+        f"{mock_openai_endpoint_server}/v1/embeddings",
+        {"model": "text-embedding-3-small", "input": "hello world"},
+    )
+    assert body["object"] == "list"
+    assert body["model"] == "text-embedding-3-small"
+    assert len(body["data"][0]["embedding"]) > 0
+
+
+def test_should_serve_fine_tuning_jobs_list(
+    mock_openai_endpoint_server: str,
+) -> None:
+    body = _get_json(f"{mock_openai_endpoint_server}/openai/fine_tuning/jobs")
+    assert body["object"] == "list"
+    assert isinstance(body["data"], list)

uv.lock

@@ -3187,6 +3187,20 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/3b/e4/b2f4ca7965ca373b491cdb4bc25cdb30c1649ca81a8782056a83850292a9/librt-0.11.0-cp313-cp313-win_arm64.whl", hash = "sha256:96f044bb325fd9cf1a723015638c219e9143f0dfbc0ca54c565df2b7fc748b44", size = 103448, upload-time = "2026-05-10T18:16:29.066Z" },
 ]
 
+[[package]]
+name = "limits"
+version = "5.8.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "deprecated" },
+    { name = "packaging" },
+    { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/71/69/826a5d1f45426c68d8f6539f8d275c0e4fcaa57f0c017ec3100986558a41/limits-5.8.0.tar.gz", hash = "sha256:c9e0d74aed837e8f6f50d1fcebcf5fd8130957287206bc3799adaee5092655da", size = 226104, upload-time = "2026-02-05T07:17:35.859Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/b9/98/cb5ca20618d205a09d5bec7591fbc4130369c7e6308d9a676a28ff3ab22c/limits-5.8.0-py3-none-any.whl", hash = "sha256:ae1b008a43eb43073c3c579398bd4eb4c795de60952532dc24720ab45e1ac6b8", size = 60954, upload-time = "2026-02-05T07:17:34.425Z" },
+]
+
 [[package]]
 name = "litellm"
 version = "1.87.0"
@@ -3361,6 +3375,9 @@ healthcheck = [
     { name = "httpx" },
     { name = "pyyaml" },
 ]
+mock-server = [
+    { name = "slowapi" },
+]
 proxy-dev = [
     { name = "a2a-sdk" },
     { name = "azure-identity" },
@@ -3521,6 +3538,7 @@ healthcheck = [
     { name = "httpx", specifier = "==0.28.1" },
     { name = "pyyaml", specifier = "==6.0.3" },
 ]
+mock-server = [{ name = "slowapi", specifier = "==0.1.9" }]
 proxy-dev = [
     { name = "a2a-sdk", specifier = "==0.3.24" },
     { name = "azure-identity", specifier = "==1.25.2" },
@@ -6925,6 +6943,18 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/e7/0e/3ae19fa941522cd98e119762e7181d371c8dba0b2d72bfaf9522692e329c/skops-0.14.0-py3-none-any.whl", hash = "sha256:60a5db78a9db46ccee2139a0ba13ab5afb1c96f4749b382e75a371291bbe3e36", size = 132198, upload-time = "2026-04-20T18:23:54.018Z" },
 ]
 
+[[package]]
+name = "slowapi"
+version = "0.1.9"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "limits" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a0/99/adfc7f94ca024736f061257d39118e1542bade7a52e86415a4c4ae92d8ff/slowapi-0.1.9.tar.gz", hash = "sha256:639192d0f1ca01b1c6d95bf6c71d794c3a9ee189855337b4821f7f457dddad77", size = 14028, upload-time = "2024-02-05T12:11:52.13Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/2b/bb/f71c4b7d7e7eb3fc1e8c0458a8979b912f40b58002b9fbf37729b8cb464b/slowapi-0.1.9-py3-none-any.whl", hash = "sha256:cfad116cfb84ad9d763ee155c1e5c5cbf00b0d47399a769b227865f5df576e36", size = 14670, upload-time = "2024-02-05T12:11:50.898Z" },
+]
+
 [[package]]
 name = "smmap"
 version = "5.0.3"