scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-02 02:11:58 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 101

test(ci): repair stale tests and flaky CI infrastructure (#43983)

Browse source 
* test(ci): add used_client_oauth_token to the GCS pub/sub spend-log golden

#43063 stamps used_client_oauth_token into spend-log metadata, so
test_async_gcs_pub_sub_v1 failed on main with an extra metadata key

* test(ui): give the auto-router threshold save wait room for the availability debounce

#42625 keeps Save disabled while a 300ms-debounced availability check runs.
This test waits for Save right after the change, so the whole debounce lands
inside waitFor's 1s default and it times out under CI load. It is the
recurring UI Unit Tests failure on main since #42625 landed

* test(e2e): expect no pricing tier on bills for streamed calls OpenAI served at default

#42870 added both the rule that a served default or standard tier bills at
base pricing and records no service_tier, and streamed tests expecting the
row to record 'default'. They have failed on every scheduled litellm-e2e run
since. The tests now map the served tier to the pricing basis the bill must
record and check input is billed at that basis's rate; the messages case
registers custom rates so the rate check has something to compare against

* test(e2e-ui): wait for the call-id search before hovering the logs row

The row the spec hovers is already on the unfiltered first page, so it was
found before the search request returned. The search response then
re-rendered the table under the mouse, and the Base UI tooltip never opened.
Reproduced with Playwright against a local proxy: hovering right after the
fill never shows the tooltip, hovering after the search response shows the
call id every time

* test(e2e): run the Together structured-output case on the hybrid Qwen with reasoning off

The case picked the cheapest Together row flagged supports_response_schema.
DeepSeek-V4-Flash-0731 hit its cost-map deprecation date on 2026-09-29, so the
pick moved to GLM-5.3-Flash, a reasoning-only model that spends the 1024-token
budget thinking and returns content=None. Qwen3.5-9B is the pinned hybrid model
the reasoning_effort=none case already exercises, and Together lists it with
structured output support

* test(integration): read the agent 365 guardrail status by its own name in spend logs

The MCP shard runs under xdist against one database, and a sibling file creates a
default_on pre_mcp_call content filter there. The owned proxy reloads DB guardrails, so
that filter's 'success' entry could land first in guardrail_information and the test
read it instead of the agent 365 verdict

* test(unit): ignore asyncio's leaked-task records in the budget limiter push-failure log check

gc.collect() inside the caplog window can collect a pending task an earlier test left
on a closed loop, and asyncio logs 'Task was destroyed but it is pending' into this
test's records. The check still counts every LiteLLM logger, and unretrieved task
exceptions on this loop still go through the asserted exception handler

* test(e2e-ui): fill the create-tag fields inside the dialog

#42949 added 'Filter by tag name' and 'Filter by description' inputs to the Tag
Management page, so page-wide getByLabel('Tag Name') and getByLabel('Description')
match two elements and Playwright's strict mode fails the create step

* test(integration): run integration proxies with the CI license

Multi-worker proxies start each uvicorn worker in a fresh process, so every
worker reads the license from its environment. Forward LITELLM_LICENSE into the
proxy and test runner environments

* ci: save GitHub Actions caches only from main and bump codecov-action to 5.5.5

Every pull request saved its own uv, maturin, Rust and Prisma caches, about
4.5 GB per PR, so the repository's 10 GB cache budget evicted main's entries
within minutes. Pull request jobs then missed every cache, downloaded all
dependencies from PyPI and hit the install step timeouts. Pull requests now
restore only, and main keeps the caches warm for them. test-linting and
check-ui-api-types run only on pull requests and keep saving

codecov-action 5.5.4 imports its signing key from the deleted codecovsecurity
keybase account, so every upload failed signature verification. 5.5.5 reads it
from codecovsecops; the key ID matches the one signing the current CLI

* test(unit): join the session-minting thread before collecting the handler

asyncio.to_thread resumes the test as soon as the worker sets its result,
while the pool thread can still hold the work item and through it the
handler. gc.collect() then cannot finalize the handler and the session stays
open. A pool that shuts down before the test continues drops that reference

* test(integration): relaunch owned proxies that lose their port, expire idle gateway connections early

owned_proxy_process released its reserved port and the proxy bound it only
after full startup, so another xdist worker or an outgoing connection could
take it first and the proxy exited with 'address already in use'. The launch
now retries on a fresh port when that happens and stops every failed attempt.

uvicorn closes idle keep-alive connections after 5 seconds and httpx expired
them at the same 5 seconds, so a request sent right at that mark could reuse a
socket the server was closing and get 'Connection reset by peer'. Gateway
clients now drop idle connections after 2 seconds

* ci(circleci): give the base SDK wheel build the same 30 minute no-output window as the Windows build

The release profile builds with fat LTO and one codegen unit, so the final
link of litellm-cache-s3 runs silently for minutes. Successful builds take
711 to 749 seconds, right at the default 10 minute no-output limit, and about
30% of recent runs were killed there

* test(integration): model the budget-reset database outage as 10 seconds instead of 5 refused connections

The proxy retries the database about every 30 seconds and each retry opens
roughly one connection, so a 5-connection outage took 3 to 4 retries to clear
and recovery landed between 60 and 90 seconds, straddling the test's 80 second
reset window. A fixed 10 second outage still refuses the immediate reconnect
and recovers on the next retry

* ci: move the unit-test uv cache split into a composite action

check_workflow_startup_safety sums every setup step's timeout, so the save and
restore variants each counted 5 minutes although only one runs. One composite
step keeps the setup ceiling at 35 minutes

* test(unit): point tiktoken at the bundled cache for every unit test

The rust_bridge tokenizer tests loaded o200k_base before any test in their
xdist worker had imported default_encoding, so tiktoken fell back to the
temp cache and tried to download under pytest-socket. Move the session
fixture from litellm_core_utils/conftest.py to the root unit conftest.

* test(integration): answer model discovery probes in the hosted_vllm wire tests

The router's periodic upstream model info refresh sends GET /v1/models to
hosted_vllm deployments, so a wire server that is live during a refresh
sees an extra request. Answer the probe with an empty model list and leave
it out of the provider-call assertions, matching the responses bridge
tests.
This commit is contained in:
yuneng-jiang 2026-10-01 10:46:43 -07:00 • committed by GitHub
parent 91ff0454af
commit 6ca90b927c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
30 changed files with 363 additions and 149 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
.circleci/config.yml
View file

@ -486,6 +486,7 @@ jobs:
- install_rust
- run:
name: Build the wheel
no_output_timeout: 30m
environment:
UV_HTTP_TIMEOUT: "300"
command: |

2
.circleci/scripts/run_integration.sh
View file

@ -168,6 +168,7 @@ start_proxy() {
"${database_env[@]}" REDIS_HOST="$REDIS_HOST" REDIS_PORT="$REDIS_PORT" \
INTEGRATION_UPSTREAM_URL="$INTEGRATION_UPSTREAM_URL" \
LITELLM_MASTER_KEY="$LITELLM_MASTER_KEY" LITELLM_SALT_KEY="$LITELLM_SALT_KEY" LITELLM_UI_PATH="$LITELLM_UI_PATH" PROXY_BASE_URL="http://127.0.0.1:$port" \
LITELLM_LICENSE="${LITELLM_LICENSE:-}" \
LITELLM_MODE=PRODUCTION STORE_MODEL_IN_DB=True "${cost_map_env[@]}" \
AWS_EC2_METADATA_DISABLED=true DO_NOT_TRACK=1 COVERAGE_FILE="$coverage_data" \
"${proxy_command[@]}" --config tests/integration/proxy_config.yaml \
@ -228,6 +229,7 @@ env -i PATH="$PATH" HOME="$HOME" PYTHONPATH="$PYTHONPATH" \
INTEGRATION_UPSTREAM_URL="$INTEGRATION_UPSTREAM_URL" \
INTEGRATION_WORKERS="${INTEGRATION_WORKERS:-1}" \
INTEGRATION_MASTER_KEY="$INTEGRATION_MASTER_KEY" LITELLM_MODE=PRODUCTION \
LITELLM_LICENSE="${LITELLM_LICENSE:-}" \
INTEGRATION_SEED="$INTEGRATION_SEED" \
INTEGRATION_ORDER_SEED="$INTEGRATION_ORDER_SEED" \
LITELLM_LOCAL_MODEL_COST_MAP=True AWS_EC2_METADATA_DISABLED=true DO_NOT_TRACK=1 \

13
.github/actions/cache-cargo-build/action.yml vendored
View file

@ -25,6 +25,7 @@ runs:
using: composite
steps:
- name: Restore the Cargo registry and target directory
if: github.ref == 'refs/heads/main'
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
@ -34,3 +35,15 @@ runs:
key: ${{ runner.os }}-maturin-${{ inputs.profile }}-${{ hashFiles('litellm-rust/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-maturin-${{ inputs.profile }}-
- name: Restore the Cargo registry and target directory
if: github.ref != 'refs/heads/main'
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cargo/registry
~/.cargo/git
litellm-rust/target
key: ${{ runner.os }}-maturin-${{ inputs.profile }}-${{ hashFiles('litellm-rust/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-maturin-${{ inputs.profile }}-

10
.github/actions/cache-prisma-binaries/action.yml vendored
View file

@ -30,6 +30,7 @@ runs:
echo "version=${version}" >> "$GITHUB_OUTPUT"
- name: Restore Prisma binaries
if: github.ref == 'refs/heads/main'
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
# ~/.cache/prisma-python holds the npm install tree prisma-client-py
@ -38,3 +39,12 @@ runs:
~/.cache/prisma-python
~/.cache/prisma
key: ${{ runner.os }}-prisma-binaries-${{ steps.version.outputs.version }}
- name: Restore Prisma binaries
if: github.ref != 'refs/heads/main'
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cache/prisma-python
~/.cache/prisma
key: ${{ runner.os }}-prisma-binaries-${{ steps.version.outputs.version }}

25
.github/actions/cache-uv-downloads/action.yml vendored Normal file
View file

@ -0,0 +1,25 @@
name: "Cache uv downloads"
description: >-
Restore the uv download cache on every run and save it only from main, so pull
requests reuse main's cache instead of evicting it with their own copies.
runs:
using: composite
steps:
- name: Restore and save the uv download cache
if: github.ref == 'refs/heads/main'
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ${{ env.UV_CACHE_DIR }}
key: ${{ runner.os }}-uv-downloads-py${{ env.UV_PYTHON }}-${{ hashFiles('uv.lock') }}
restore-keys: |
${{ runner.os }}-uv-downloads-py${{ env.UV_PYTHON }}-
- name: Restore the uv download cache
if: github.ref != 'refs/heads/main'
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ${{ env.UV_CACHE_DIR }}
key: ${{ runner.os }}-uv-downloads-py${{ env.UV_PYTHON }}-${{ hashFiles('uv.lock') }}
restore-keys: |
${{ runner.os }}-uv-downloads-py${{ env.UV_PYTHON }}-

3
.github/actions/setup-uv-with-retries/action.yml vendored
View file

@ -17,6 +17,7 @@ runs:
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
version: ${{ inputs.version }}
save-cache: ${{ github.ref == 'refs/heads/main' }}
- name: Wait before attempt 2
if: steps.attempt-1.outcome == 'failure'
@ -30,6 +31,7 @@ runs:
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
version: ${{ inputs.version }}
save-cache: ${{ github.ref == 'refs/heads/main' }}
- name: Wait before attempt 3
if: steps.attempt-2.outcome == 'failure'
@ -41,3 +43,4 @@ runs:
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
version: ${{ inputs.version }}
save-cache: ${{ github.ref == 'refs/heads/main' }}

11
.github/workflows/_test-unit-base.yml vendored
View file

@ -132,12 +132,7 @@ jobs:
- name: Cache uv dependencies
if: steps.changes.outputs.decision != 'skip'
timeout-minutes: 5
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: ${{ env.UV_CACHE_DIR }}
key: ${{ runner.os }}-uv-downloads-py${{ env.UV_PYTHON }}-${{ hashFiles('uv.lock') }}
restore-keys: |
${{ runner.os }}-uv-downloads-py${{ env.UV_PYTHON }}-
uses: ./.github/actions/cache-uv-downloads
- name: Cache the Rust build
if: steps.changes.outputs.decision != 'skip'
@ -274,7 +269,7 @@ jobs:
- name: Upload to Codecov
id: codecov-upload
continue-on-error: true
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe # v5.5.4
uses: codecov/codecov-action@0fb7174895f61a3b6b78fc075e0cd60383518dac # v5.5.5
with:
use_oidc: true
directory: coverage-reports
@ -285,7 +280,7 @@ jobs:
- name: Upload to Codecov (retry)
if: steps.codecov-upload.outcome == 'failure'
continue-on-error: true
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe # v5.5.4
uses: codecov/codecov-action@0fb7174895f61a3b6b78fc075e0cd60383518dac # v5.5.5
with:
use_oidc: true
directory: coverage-reports

12
.github/workflows/mutation-test.yml vendored
View file

@ -44,6 +44,7 @@ jobs:
version: "0.10.9"
- name: Cache uv dependencies
if: github.ref == 'refs/heads/main'
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
@ -53,6 +54,17 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-
- name: Cache uv dependencies
if: github.ref != 'refs/heads/main'
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cache/uv
.venv
key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
restore-keys: |
${{ runner.os }}-uv-
- name: Cache the Rust build
uses: ./.github/actions/cache-cargo-build

12
.github/workflows/test-code-quality.yml vendored
View file

@ -44,6 +44,7 @@ jobs:
version: "0.10.9"
- name: Cache uv dependencies
if: github.ref == 'refs/heads/main'
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
@ -53,6 +54,17 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-
- name: Cache uv dependencies
if: github.ref != 'refs/heads/main'
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cache/uv
.venv
key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
restore-keys: |
${{ runner.os }}-uv-
- name: Cache the Rust build
uses: ./.github/actions/cache-cargo-build

14
.github/workflows/test-postgres.yml vendored
View file

@ -95,7 +95,7 @@ jobs:
version: "0.10.9"
- name: Cache uv dependencies
if: steps.changes.outputs.decision != 'skip'
if: steps.changes.outputs.decision != 'skip' && github.ref == 'refs/heads/main'
timeout-minutes: 5
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
@ -106,6 +106,18 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-postgres-
- name: Cache uv dependencies
if: steps.changes.outputs.decision != 'skip' && github.ref != 'refs/heads/main'
timeout-minutes: 5
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cache/uv
.venv
key: ${{ runner.os }}-uv-postgres-${{ hashFiles('uv.lock') }}
restore-keys: |
${{ runner.os }}-uv-postgres-
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
timeout-minutes: 12

2
.github/workflows/test-redis-compat.yml vendored
View file

@ -98,7 +98,7 @@ jobs:
- name: Upload Redis coverage
if: matrix.redis-version == '5.3.1'
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe # v5.5.4
uses: codecov/codecov-action@0fb7174895f61a3b6b78fc075e0cd60383518dac # v5.5.5
with:
use_oidc: true
files: coverage-redis.xml

3
.github/workflows/test-rust.yml vendored
View file

@ -83,6 +83,7 @@ jobs:
with:
workspaces: litellm-rust
cache-on-failure: true
save-if: ${{ github.ref == 'refs/heads/main' }}
- run: cargo clippy --workspace --all-targets --locked -- -D warnings
@ -121,6 +122,7 @@ jobs:
with:
workspaces: litellm-rust
cache-on-failure: true
save-if: ${{ github.ref == 'refs/heads/main' }}
- run: cargo nextest run --workspace --locked
@ -162,6 +164,7 @@ jobs:
with:
workspaces: litellm-rust
cache-on-failure: true
save-if: ${{ github.ref == 'refs/heads/main' }}
- run: uv build --wheel --out-dir dist

12
.github/workflows/test-terraform-provider.yml vendored
View file

@ -77,6 +77,7 @@ jobs:
version: "0.10.9"
- name: Cache uv dependencies
if: github.ref == 'refs/heads/main'
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
@ -86,6 +87,17 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-
- name: Cache uv dependencies
if: github.ref != 'refs/heads/main'
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cache/uv
.venv
key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
restore-keys: |
${{ runner.os }}-uv-
- name: Cache the Rust build
uses: ./.github/actions/cache-cargo-build

13
.github/workflows/test-unit-documentation.yml vendored
View file

@ -54,7 +54,7 @@ jobs:
version: "0.10.9"
- name: Cache uv dependencies
if: steps.changes.outputs.decision != 'skip'
if: steps.changes.outputs.decision != 'skip' && github.ref == 'refs/heads/main'
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
@ -64,6 +64,17 @@ jobs:
restore-keys: |
${{ runner.os }}-uv-
- name: Cache uv dependencies
if: steps.changes.outputs.decision != 'skip' && github.ref != 'refs/heads/main'
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cache/uv
.venv
key: ${{ runner.os }}-uv-${{ hashFiles('uv.lock') }}
restore-keys: |
${{ runner.os }}-uv-
- name: Cache the Rust build
if: steps.changes.outputs.decision != 'skip'
uses: ./.github/actions/cache-cargo-build

23
tests/e2e/llm_translation/test_together_ai_e2e.py
View file

@ -1,12 +1,13 @@
"""Live e2e: Together AI through the gateway on /chat/completions and /v1/messages.
The reasoning and tool-calling backend is the cheapest live ``together_ai/`` chat row
in the proxy's own cost map that carries both capability flags; the structured-output
and cache-pricing backends are likewise the cheapest rows carrying
``supports_response_schema`` and a ``cache_read_input_token_cost``. Two backends are
pinned because the registry has no flag for what they prove: ``enable_thinking`` and
the ``{"reasoning": {"enabled": false}}`` toggle that ``reasoning_effort="none"`` maps
to are Qwen hybrid-model contracts, and MiniMax-M3 is the serverless model whose
in the proxy's own cost map that carries both capability flags; the cache-pricing
backend is likewise the cheapest row carrying a ``cache_read_input_token_cost``. Two
backends are pinned because the registry has no flag for what they prove: ``enable_thinking``
and the ``{"reasoning": {"enabled": false}}`` toggle that ``reasoning_effort="none"`` maps
to are Qwen hybrid-model contracts, and the structured-output case runs on that hybrid
model with reasoning off, since a reasoning-only model can spend the whole token budget
thinking and return no content. MiniMax-M3 is the serverless model whose
template renders a replayed ``reasoning_content`` back into the prompt (Qwen and
DeepSeek silently drop it). MiniMax-M3 honors that replayed field on nearly every call, not
every call (one miss in dozens of otherwise identical calls), so the replay case asks
@ -109,7 +110,6 @@ MESSAGES_WEATHER_TOOL = AnthropicCustomTool(
class _Needs:
function_calling: bool = False
reasoning: bool = False
response_schema: bool = False
cache_read_pricing: bool = False
@ -172,7 +172,6 @@ def _cheapest_together_chat_model(registry: Mapping[str, CostMapEntry], needs: _
and (entry.output_cost_per_token or 0.0) > 0
and (not needs.function_calling or bool(entry.supports_function_calling))
and (not needs.reasoning or bool(entry.supports_reasoning))
and (not needs.response_schema or bool(entry.supports_response_schema))
and (not needs.cache_read_pricing or (entry.cache_read_input_token_cost or 0.0) > 0)
)
@ -586,10 +585,9 @@ class TestTogetherChatCompletions:
@pytest.mark.covers("llm.chat_completions.together_ai.structured_output.nonstream.works")
def test_response_format_json_schema_shapes_the_reply(
self, client: PassthroughClient, resources: ResourceManager, registry: dict[str, CostMapEntry]
self, client: PassthroughClient, resources: ResourceManager
) -> None:
backend = _cheapest_together_chat_model(registry, _Needs(response_schema=True))
model, key = _register(client, resources, backend)
model, key = _register(client, resources, HYBRID_REASONING_BACKEND)
message = _message(
unwrap(
@ -599,12 +597,13 @@ class TestTogetherChatCompletions:
model=model,
messages=[ChatMessage(role="user", content=PERSON_PROMPT)],
max_tokens=1024,
reasoning_effort="none",
response_format=PERSON_RESPONSE_FORMAT,
),
)
)
)
assert message.content, f"{backend} returned no content: {message}"
assert message.content, f"{HYBRID_REASONING_BACKEND} returned no content: {message}"
person = _Person.model_validate_json(message.content)
assert person.name, f"schema-shaped reply carries an empty name: {message.content!r}"

60
tests/e2e/quota_management/spend_tracking/test_service_tier_pricing_e2e.py
View file

@ -17,9 +17,10 @@ sets rather than on whatever the model happens to do by default.
The streaming cases pin the served-tier contract: OpenAI stamps the tier it actually
used on every stream chunk, and that echo is what the caller sees and what the bill
must be computed on. The request sets no service_tier, so the only place the tier
can come from is the provider's response. The spend row must record the served tier
and price input at that tier's rate, and every chunk the proxy relays must carry the
same service_tier the provider sent.
can come from is the provider's response. The spend row must record the tier the bill
was priced on and price input at that tier's rate, and every chunk the proxy relays must
carry the same service_tier the provider sent. A served `default` tier is base pricing,
which the bill records as no tier
"""
import json
@ -61,7 +62,8 @@ PRIORITY_OUTPUT_RATE = 1.6e-04
REASONING_EFFORT = "high"
TIER_INPUT_RATES = {"default": INPUT_RATE, "priority": PRIORITY_INPUT_RATE}
PRICING_BASIS_FOR_SERVED_TIER: dict[str, str | None] = {"default": None, "priority": "priority"}
INPUT_RATE_FOR_PRICING_BASIS: dict[str | None, float] = {None: INPUT_RATE, "priority": PRIORITY_INPUT_RATE}
class _StreamChunk(BaseModel):
@ -213,17 +215,20 @@ class TestServiceTierPricing:
)
chunks = _stream_chunks(result.stream_events)
served_tier = _served_tier(chunks)
assert served_tier in TIER_INPUT_RATES, f"no custom rate registered for served tier {served_tier!r}"
assert served_tier in PRICING_BASIS_FOR_SERVED_TIER, (
f"no custom rate registered for served tier {served_tier!r}"
)
pricing_basis = PRICING_BASIS_FOR_SERVED_TIER[served_tier]
stream_id = chunks[0].id
assert stream_id, f"first stream chunk carried no id: {result.stream_events[0][:200]}"
row = poll_cost_row(client.proxy, stream_id)
assert row is not None, f"no spend row with a cost breakdown landed for {stream_id}"
assert row.breakdown.service_tier == served_tier, (
f"the provider served tier {served_tier!r} on every chunk but the bill records "
f"pricing basis {row.breakdown.service_tier!r}"
assert row.breakdown.service_tier == pricing_basis, (
f"the provider served tier {served_tier!r} on every chunk, so the bill should record pricing "
f"basis {pricing_basis!r}, but it records {row.breakdown.service_tier!r}"
)
assert_fresh_tokens_billed_at(row, TIER_INPUT_RATES[served_tier])
assert_fresh_tokens_billed_at(row, INPUT_RATE_FOR_PRICING_BASIS[pricing_basis])
assert_total_is_sum_of_components(row)
@pytest.mark.covers("llm.chat_completions.openai.service_tier.stream.echoes_served_tier")
@ -264,7 +269,14 @@ class TestServiceTierPricing:
client.proxy,
resources,
"tier-responses-stream",
LiteLLMParamsBody(model=STREAM_BACKEND, api_key=OPENAI_API_KEY),
LiteLLMParamsBody(
model=STREAM_BACKEND,
api_key=OPENAI_API_KEY,
input_cost_per_token=INPUT_RATE,
output_cost_per_token=OUTPUT_RATE,
input_cost_per_token_priority=PRIORITY_INPUT_RATE,
output_cost_per_token_priority=PRIORITY_OUTPUT_RATE,
),
)
result = client.proxy.responses_stream(
@ -282,14 +294,18 @@ class TestServiceTierPricing:
)
served_tier = completed.response.service_tier
assert served_tier, f"response.completed carried no service_tier: {completed.response}"
assert served_tier in TIER_INPUT_RATES, f"no custom rate registered for served tier {served_tier!r}"
assert served_tier in PRICING_BASIS_FOR_SERVED_TIER, (
f"no custom rate registered for served tier {served_tier!r}"
)
pricing_basis = PRICING_BASIS_FOR_SERVED_TIER[served_tier]
row = poll_cost_row_where(client.proxy, scoped_key, lambda r: r.spend is not None and r.spend > 0)
assert row is not None, f"no spend row with a cost breakdown landed for the streamed responses call on {model}"
assert row.breakdown.service_tier == served_tier, (
f"response.completed served tier {served_tier!r} but the bill records "
f"pricing basis {row.breakdown.service_tier!r}"
assert row.breakdown.service_tier == pricing_basis, (
f"response.completed served tier {served_tier!r}, so the bill should record pricing basis "
f"{pricing_basis!r}, but it records {row.breakdown.service_tier!r}"
)
assert_fresh_tokens_billed_at(row, INPUT_RATE_FOR_PRICING_BASIS[pricing_basis])
@pytest.mark.covers("quota_management.spend_tracking.service_tier_stream.messages_records_served_tier")
def test_messages_stream_records_the_served_tier(
@ -299,7 +315,14 @@ class TestServiceTierPricing:
client.proxy,
resources,
"tier-messages-stream",
LiteLLMParamsBody(model=STREAM_BACKEND, api_key=OPENAI_API_KEY),
LiteLLMParamsBody(
model=STREAM_BACKEND,
api_key=OPENAI_API_KEY,
input_cost_per_token=INPUT_RATE,
output_cost_per_token=OUTPUT_RATE,
input_cost_per_token_priority=PRIORITY_INPUT_RATE,
output_cost_per_token_priority=PRIORITY_OUTPUT_RATE,
),
)
result = client.proxy.messages_stream(
@ -322,8 +345,9 @@ class TestServiceTierPricing:
row = poll_cost_row_where(client.proxy, scoped_key, lambda r: r.spend is not None and r.spend > 0)
assert row is not None, f"no spend row with a cost breakdown landed for the streamed messages call on {model}"
served_tier = row.breakdown.service_tier
assert served_tier in TIER_INPUT_RATES and served_tier is not None, (
pricing_basis = row.breakdown.service_tier
assert pricing_basis in INPUT_RATE_FOR_PRICING_BASIS, (
"the anthropic wire format carries no service_tier, so the bill is the only record of "
f"the tier OpenAI served; the row recorded pricing basis {served_tier!r}"
f"the tier OpenAI served; the row recorded pricing basis {pricing_basis!r}"
)
assert_fresh_tokens_billed_at(row, INPUT_RATE_FOR_PRICING_BASIS[pricing_basis])

8
tests/e2e/ui/tests/logs/logs.spec.ts
View file

@ -111,7 +111,15 @@ test.describe("Logs page", () => {
await dismissFeedbackPopup(page);
const search = visibleTestId(page, "datatable-search");
await expect(search).toBeVisible({ timeout: 20_000 });
const searched = page.waitForResponse(
(response) =>
response.url().includes("/spend/logs/ui") &&
new URL(response.url()).searchParams.get("search") === callId &&
response.status() === 200,
{ timeout: 20_000 },
);
await search.fill(callId);
await searched;
const row = requestLogsRows(page).filter({ hasText: requestId });
await expect(row, `no logs row for call id ${callId}`).toHaveCount(1, { timeout: 30_000 });

9
tests/e2e/ui/tests/tagManagement/tagManagement.spec.ts
View file

@ -22,11 +22,10 @@ test.describe("Tag management", () => {
async () => {
await navigateToPage(page, DashboardPage.TagManagement);
await page.getByRole("button", { name: "+ Create New Tag" }).click();
await expect(
page.getByRole("dialog", { name: "Create New Tag" }),
).toBeVisible();
await page.getByLabel("Tag Name").fill(tagName);
await page.getByLabel("Description").fill(description);
const createDialog = page.getByRole("dialog", { name: "Create New Tag" });
await expect(createDialog).toBeVisible();
await createDialog.getByLabel("Tag Name").fill(tagName);
await createDialog.getByLabel("Description").fill(description);
await page.getByRole("button", { name: "Create Tag" }).click();
await expect

3
tests/integration/_support/client.py
View file

@ -15,6 +15,7 @@ from pydantic import JsonValue, TypeAdapter
from tests.integration._support.database import read_rows
JSON_OBJECT: Final = TypeAdapter(dict[str, JsonValue])
GATEWAY_LIMITS: Final = httpx.Limits(keepalive_expiry=2)
T = TypeVar("T")
@ -243,7 +244,7 @@ class Scenario:
def gateway_from_environment() -> Iterator[Gateway]:
url: Final = os.environ["INTEGRATION_PROXY_URL"]
upstream: Final = os.environ["INTEGRATION_UPSTREAM_URL"]
with httpx.Client(base_url=url, timeout=15, trust_env=False) as client:
with httpx.Client(base_url=url, timeout=15, trust_env=False, limits=GATEWAY_LIMITS) as client:
yield Gateway(client, os.environ["INTEGRATION_MASTER_KEY"], upstream)

6
tests/integration/_support/database_relay.py
View file

@ -1,6 +1,7 @@
import asyncio
import socket
import threading
import time
from collections.abc import Generator
from contextlib import contextmanager
from typing import Final
@ -9,6 +10,7 @@ from urllib.parse import urlsplit, urlunsplit
from pydantic import TypeAdapter
PORT: Final = TypeAdapter(int)
OUTAGE_SECONDS: Final = 10.0
def _free_port() -> int:
@ -27,6 +29,7 @@ class DatabaseRelay:
self._armed: Final = threading.Event()
self.tripped: Final = threading.Event()
self.refused = 0
self._tripped_at = 0.0
self._writers: tuple[asyncio.StreamWriter, ...] = ()
self._ready: Final = threading.Event()
self._thread: Final = threading.Thread(target=self._run, daemon=True)
@ -54,7 +57,7 @@ class DatabaseRelay:
self._writers = ()
async def _serve(self, client_reader: asyncio.StreamReader, client_writer: asyncio.StreamWriter) -> None:
if self.tripped.is_set() and self.refused < 5:
if self.tripped.is_set() and time.monotonic() - self._tripped_at < OUTAGE_SECONDS:
self.refused += 1
client_writer.close()
return
@ -65,6 +68,7 @@ class DatabaseRelay:
try:
while chunk := await reader.read(65536):
if inspect and self._armed.is_set() and not self.tripped.is_set() and self._trigger in chunk:
self._tripped_at = time.monotonic()
self.tripped.set()
self._drop_all()
return

160
tests/integration/_support/process.py
View file

@ -14,7 +14,7 @@ from typing import Final
import httpx
import psutil
from integration._support.client import Gateway
from integration._support.client import GATEWAY_LIMITS, Gateway
def proxy_database_environment() -> Mapping[str, str]:
@ -80,6 +80,88 @@ def owned_proxy(
yield owned.gateway
def _stop(process: subprocess.Popen[bytes]) -> None:
root_stopped: Final = stop_root_process(process)
residual: Final = group_members(process.pid)
if residual:
signal_group(process.pid, signal.SIGTERM)
psutil.wait_procs(residual, timeout=5)
remaining: Final = group_members(process.pid)
if remaining:
signal_group(process.pid, signal.SIGKILL)
psutil.wait_procs(remaining, timeout=3)
process.wait(timeout=3)
survivors: Final = group_members(process.pid)
assert not survivors, "Owned proxy child survived cleanup"
assert root_stopped and not remaining, "Owned proxy required forced cleanup"
_PORT_ATTEMPTS: Final = 3
def _free_port() -> int:
with socket.socket() as reserve:
reserve.bind(("127.0.0.1", 0))
return reserve.getsockname()[1]
@dataclass(frozen=True, slots=True)
class _Launch:
process: subprocess.Popen[bytes]
port: int
log: Path
def _launch(command: tuple[str, ...], root: Path, environment: Mapping[str, str], output: Path) -> _Launch:
port: Final = _free_port()
log_path: Final = output / f"owned-proxy-{uuid.uuid4().hex}.log"
with log_path.open("w") as log:
process: Final = subprocess.Popen(
[*command, "--port", str(port)],
cwd=root,
env=environment,
stdout=log,
stderr=subprocess.STDOUT,
start_new_session=True,
)
return _Launch(process, port, log_path)
def _lost_port_race(launch: _Launch) -> bool:
return launch.process.poll() is not None and "address already in use" in launch.log.read_text()
def _wait_until_ready(launch: _Launch) -> None:
with httpx.Client(base_url=f"http://127.0.0.1:{launch.port}", timeout=15, trust_env=False) as client:
deadline: Final = time.monotonic() + 70
while launch.process.poll() is None:
try:
if client.get("/health/readiness", timeout=2).status_code == 200:
return
except httpx.TransportError:
pass
assert time.monotonic() < deadline, "Owned proxy readiness deadline exceeded"
time.sleep(0.1)
def _launch_until_bound(
command: tuple[str, ...], root: Path, environment: Mapping[str, str], output: Path, attempts: int
) -> _Launch:
launch: Final = _launch(command, root, environment, output)
try:
_wait_until_ready(launch)
assert launch.process.poll() is None or (attempts > 1 and _lost_port_race(launch)), (
"Owned proxy exited before readiness"
)
except BaseException:
_stop(launch.process)
raise
if launch.process.poll() is None:
return launch
_stop(launch.process)
return _launch_until_bound(command, root, environment, output, attempts - 1)
@contextmanager
def owned_proxy_process(
gateway: Gateway,
@ -90,9 +172,6 @@ def owned_proxy_process(
remove_environment: tuple[str, ...] = (),
workers: int = 1,
) -> Iterator[OwnedProxy]:
with socket.socket() as reserve:
reserve.bind(("127.0.0.1", 0))
port: Final = reserve.getsockname()[1]
root: Final = Path(os.environ.get("INTEGRATION_PROXY_ROOT") or Path(__file__).resolve().parents[3])
environment: Final = {
**{
@ -107,54 +186,25 @@ def owned_proxy_process(
}
output: Final = Path(os.environ.get("INTEGRATION_RESULTS_DIR", str(directory)))
output.mkdir(parents=True, exist_ok=True)
log_path: Final = output / f"owned-proxy-{uuid.uuid4().hex}.log"
with log_path.open("w") as log:
process: Final = subprocess.Popen(
[
sys.executable,
"-m",
"integration._support.proxy",
"--config",
str(config or "tests/integration/proxy_config.yaml"),
"--host",
"127.0.0.1",
"--port",
str(port),
"--num_workers",
str(workers),
"--use_prisma_db_push",
"--enforce_prisma_migration_check",
],
cwd=root,
env=environment,
stdout=log,
stderr=subprocess.STDOUT,
start_new_session=True,
)
try:
with httpx.Client(base_url=f"http://127.0.0.1:{port}", timeout=15, trust_env=False) as client:
deadline: Final = time.monotonic() + 70
while True:
assert process.poll() is None, "Owned proxy exited before readiness"
try:
if client.get("/health/readiness", timeout=2).status_code == 200:
break
except httpx.TransportError:
pass
assert time.monotonic() < deadline, "Owned proxy readiness deadline exceeded"
time.sleep(0.1)
yield OwnedProxy(Gateway(client, gateway.key, gateway.upstream_url), process, log_path)
finally:
root_stopped: Final = stop_root_process(process)
residual: Final = group_members(process.pid)
if residual:
signal_group(process.pid, signal.SIGTERM)
psutil.wait_procs(residual, timeout=5)
remaining: Final = group_members(process.pid)
if remaining:
signal_group(process.pid, signal.SIGKILL)
psutil.wait_procs(remaining, timeout=3)
process.wait(timeout=3)
survivors: Final = group_members(process.pid)
assert not survivors, "Owned proxy child survived cleanup"
assert root_stopped and not remaining, "Owned proxy required forced cleanup"
command: Final = (
sys.executable,
"-m",
"integration._support.proxy",
"--config",
str(config or "tests/integration/proxy_config.yaml"),
"--host",
"127.0.0.1",
"--num_workers",
str(workers),
"--use_prisma_db_push",
"--enforce_prisma_migration_check",
)
launch: Final = _launch_until_bound(command, root, environment, output, _PORT_ATTEMPTS)
process: Final = launch.process
try:
with httpx.Client(
base_url=f"http://127.0.0.1:{launch.port}", timeout=15, trust_env=False, limits=GATEWAY_LIMITS
) as client:
yield OwnedProxy(Gateway(client, gateway.key, gateway.upstream_url), process, launch.log)
finally:
_stop(process)

8
tests/integration/_support/proxy.py
View file

@ -1,9 +1,6 @@
"""Run the normal single-process CLI with the existing behavior-suite test entitlement."""
import signal
import sys
from types import FrameType
from unittest.mock import patch
from litellm import run_server
@ -14,10 +11,7 @@ def _exit_on_reraised_term(signum: int, frame: FrameType | None) -> None:
def main() -> None:
signal.signal(signal.SIGTERM, _exit_on_reraised_term)
with patch( # test-quality-ok: route entitlement only; license validation is outside these HTTP/DB contracts
"litellm.proxy.auth.litellm_license.LicenseCheck.is_premium", return_value=True
):
run_server()
run_server()
if __name__ == "__main__":

10
tests/integration/mcp/test_mcp_agent_365_guardrail.py
View file

@ -30,8 +30,10 @@ from integration._support.wire import Reply, Request, wire_server
TENANT: Final = "00000000-0000-4000-8000-0000000a3650"
REJECTED: Final = "Agent 365 guardrail rejected the tool call"
GUARDRAIL_ROWS: Final = (
"SELECT metadata->'guardrail_information' AS gi FROM \"LiteLLM_SpendLogs\" "
'WHERE api_key = %s AND call_type = %s ORDER BY "startTime"'
"SELECT COALESCE(jsonb_path_query_first(metadata, "
"'$.guardrail_information[*] ? (@.guardrail_name == $name).guardrail_status', "
"jsonb_build_object('name', %s::text)) #>> '{}', 'none') AS status "
'FROM "LiteLLM_SpendLogs" WHERE api_key = %s AND call_type = %s ORDER BY "startTime"'
)
FALLBACKS: Final = (None, "fail_open", "fail_closed")
@ -95,11 +97,11 @@ class Rig:
def guardrail_statuses(self, call_type: str, at_least: int) -> list[str]:
rows: Final = eventually(
lambda: read_rows(GUARDRAIL_ROWS, (sha256(self.key.encode()).hexdigest(), call_type)),
lambda: read_rows(GUARDRAIL_ROWS, (self.alias, sha256(self.key.encode()).hexdigest(), call_type)),
lambda seen: len(seen) >= at_least,
seconds=70,
)
return [row["gi"][0]["guardrail_status"] if row["gi"] else "none" for row in rows]
return [str(row["status"]) for row in rows]
@contextmanager

68
tests/integration/providers/test_hosted_vllm_reasoning_content_wire.py
View file

@ -1,6 +1,7 @@
import json
import uuid
from collections.abc import Sequence
from collections.abc import Callable, Iterator, Sequence
from contextlib import contextmanager
from typing import Final
import openai
@ -69,8 +70,27 @@ def _sent_messages(request: Request) -> list[dict[str, JsonValue]]:
return _MESSAGES.validate_python(_JSON_OBJECT.validate_json(request.body)["messages"])
_DISCOVERY_PROBE: Final = ("GET", "/v1/models")
def _is_discovery_probe(request: Request) -> bool:
return (request.method, request.target) == _DISCOVERY_PROBE
@contextmanager
def _vllm_server(respond: Callable[[Request], Reply]) -> Iterator[Wire]:
with wire_server(
lambda request: Reply(body=b'{"object":"list","data":[]}') if _is_discovery_probe(request) else respond(request)
) as wire:
yield wire
def _provider_calls(wire: Wire) -> tuple[Request, ...]:
return tuple(request for request in wire.drain() if not _is_discovery_probe(request))
def _only_request(wire: Wire) -> Request:
received: Final = wire.drain()
received: Final = _provider_calls(wire)
assert [(request.method, request.target) for request in received] == [("POST", "/v1/chat/completions")]
return received[0]
@ -139,7 +159,7 @@ def test_hosted_vllm_assistant_reasoning_content_reaches_the_wire(gateway: Gatew
], body["messages"]
return Reply(body=_completion(identity, "The totals differ by 42."))
with wire_server(respond) as wire, gateway.scenario() as scenario:
with _vllm_server(respond) as wire, gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
response: Final = gateway.request(
"POST",
@ -167,13 +187,13 @@ def test_hosted_vllm_assistant_reasoning_content_reaches_the_wire(gateway: Gatew
assert response.status_code == 200, response.text
payload: Final = _JSON_OBJECT.validate_json(response.content)
assert payload["id"] == identity
assert [(request.method, request.target) for request in wire.drain()] == [("POST", "/v1/chat/completions")]
_only_request(wire)
def test_openai_sdk_replayed_reasoning_reaches_hosted_vllm_and_is_billed_once(gateway: Gateway) -> None:
marker: Final = uuid.uuid4().hex
identity: Final = f"chatcmpl-sdk-{marker}"
with wire_server(lambda _: Reply(body=_completion(identity, "They differ by 42."))) as wire:
with _vllm_server(lambda _: Reply(body=_completion(identity, "They differ by 42."))) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
completion: Final = _openai_client(gateway).chat.completions.create(
@ -195,7 +215,7 @@ def test_openai_sdk_replayed_reasoning_reaches_hosted_vllm_and_is_billed_once(ga
async def test_async_openai_sdk_stream_forwards_replayed_reasoning_to_hosted_vllm(gateway: Gateway) -> None:
marker: Final = uuid.uuid4().hex
identity: Final = f"chatcmpl-stream-{marker}"
with wire_server(lambda _: _streamed_completion(identity, "They differ by 42.")) as wire:
with _vllm_server(lambda _: _streamed_completion(identity, "They differ by 42.")) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
stream: Final = await _async_openai_client(gateway).chat.completions.create(
@ -224,7 +244,7 @@ def test_each_replayed_turn_keeps_its_own_reasoning_in_order(gateway: Gateway) -
{"role": "assistant", "content": "Step two.", "reasoning_content": f"second thought {marker}"},
{"role": "user", "content": "Summarize."},
]
with wire_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Done."))) as wire:
with _vllm_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Done."))) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
assert _post_chat(gateway, model, conversation)["id"] == f"chatcmpl-{marker}"
@ -246,7 +266,7 @@ def test_only_string_reasoning_content_is_forwarded_to_hosted_vllm(
gateway: Gateway, reasoning: JsonValue, forwarded: str | None
) -> None:
marker: Final = uuid.uuid4().hex
with wire_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Done."))) as wire:
with _vllm_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Done."))) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
assert _post_chat(gateway, model, _replayed_conversation(reasoning, marker))["id"] == f"chatcmpl-{marker}"
@ -264,7 +284,7 @@ def test_assistant_turn_without_reasoning_gets_no_reasoning_key(gateway: Gateway
{"role": "assistant", "content": "Hi there."},
{"role": "user", "content": "Again"},
]
with wire_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Hello again."))) as wire:
with _vllm_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Hello again."))) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
_post_chat(gateway, model, conversation)
@ -281,7 +301,7 @@ def test_same_reasoning_on_two_turns_is_forwarded_on_both(gateway: Gateway) -> N
{"role": "assistant", "content": "Second.", "reasoning_content": reasoning},
{"role": "user", "content": "Three"},
]
with wire_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Third."))) as wire:
with _vllm_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Third."))) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
_post_chat(gateway, model, conversation)
@ -290,7 +310,7 @@ def test_same_reasoning_on_two_turns_is_forwarded_on_both(gateway: Gateway) -> N
def test_thinking_blocks_are_stripped_while_reasoning_content_is_kept(gateway: Gateway) -> None:
marker: Final = uuid.uuid4().hex
with wire_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Done."))) as wire:
with _vllm_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Done."))) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
_post_chat(
@ -316,7 +336,7 @@ def test_thinking_blocks_are_stripped_while_reasoning_content_is_kept(gateway: G
def test_list_content_is_flattened_while_reasoning_content_is_kept(gateway: Gateway) -> None:
marker: Final = uuid.uuid4().hex
with wire_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Done."))) as wire:
with _vllm_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Done."))) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
_post_chat(
@ -341,7 +361,7 @@ def test_list_content_is_flattened_while_reasoning_content_is_kept(gateway: Gate
def test_unauthenticated_replay_is_rejected_before_hosted_vllm(gateway: Gateway) -> None:
marker: Final = uuid.uuid4().hex
with wire_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Done."))) as wire:
with _vllm_server(lambda _: Reply(body=_completion(f"chatcmpl-{marker}", "Done."))) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
response: Final = gateway.request(
@ -351,7 +371,7 @@ def test_unauthenticated_replay_is_rejected_before_hosted_vllm(gateway: Gateway)
key=f"sk-not-a-key-{marker}",
)
assert response.status_code == 401, response.text
assert wire.drain() == ()
assert _provider_calls(wire) == ()
def test_hosted_vllm_auth_error_reaches_the_caller_after_one_attempt_with_reasoning(gateway: Gateway) -> None:
@ -361,7 +381,7 @@ def test_hosted_vllm_auth_error_reaches_the_caller_after_one_attempt_with_reason
status=401,
body=json.dumps({"error": {"message": error_message, "type": "authentication_error"}}).encode(),
)
with wire_server(lambda _: reply) as wire, gateway.scenario() as scenario:
with _vllm_server(lambda _: reply) as wire, gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
response: Final = gateway.request(
"POST",
@ -381,7 +401,7 @@ def test_fallback_attempt_replays_reasoning_to_the_second_deployment(gateway: Ga
return Reply(status=500, body=b'{"error": {"message": "primary deployment is down"}}')
return Reply(body=_completion(f"chatcmpl-fallback-{marker}", "Recovered."))
with wire_server(respond) as wire, gateway.scenario() as scenario:
with _vllm_server(respond) as wire, gateway.scenario() as scenario:
primary: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
fallback: Final = scenario.model(
model=f"hosted_vllm/{_FALLBACK_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY
@ -399,7 +419,7 @@ def test_fallback_attempt_replays_reasoning_to_the_second_deployment(gateway: Ga
)
assert response.status_code == 200, response.text
assert _JSON_OBJECT.validate_json(response.content)["id"] == f"chatcmpl-fallback-{marker}"
attempts: Final = wire.drain()
attempts: Final = _provider_calls(wire)
assert [_JSON_OBJECT.validate_json(attempt.body)["model"] for attempt in attempts] == [
_BACKEND,
_FALLBACK_BACKEND,
@ -413,13 +433,13 @@ def test_fallback_attempt_replays_reasoning_to_the_second_deployment(gateway: Ga
def test_identical_uncached_replays_are_each_forwarded_and_billed_once(gateway: Gateway) -> None:
marker: Final = uuid.uuid4().hex
identities: Final = iter((f"chatcmpl-first-{marker}", f"chatcmpl-second-{marker}"))
with wire_server(lambda _: Reply(body=_completion(next(identities), "Done."))) as wire:
with _vllm_server(lambda _: Reply(body=_completion(next(identities), "Done."))) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
first: Final = _post_chat(gateway, model, _replayed_conversation(_REASONING, marker))
second: Final = _post_chat(gateway, model, _replayed_conversation(_REASONING, marker))
assert (first["id"], second["id"]) == (f"chatcmpl-first-{marker}", f"chatcmpl-second-{marker}")
assert [_sent_messages(request) for request in wire.drain()] == [
assert [_sent_messages(request) for request in _provider_calls(wire)] == [
_replayed_conversation(_REASONING, marker),
_replayed_conversation(_REASONING, marker),
]
@ -430,7 +450,7 @@ def test_identical_uncached_replays_are_each_forwarded_and_billed_once(gateway:
def test_cached_replay_hits_only_for_the_same_reasoning(gateway: Gateway) -> None:
marker: Final = uuid.uuid4().hex
identities: Final = iter((f"chatcmpl-cached-{marker}", f"chatcmpl-other-{marker}"))
with wire_server(lambda _: Reply(body=_completion(next(identities), "Done."))) as wire:
with _vllm_server(lambda _: Reply(body=_completion(next(identities), "Done."))) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
@ -446,7 +466,7 @@ def test_cached_replay_hits_only_for_the_same_reasoning(gateway: Gateway) -> Non
assert ask(_REASONING)["id"] == f"chatcmpl-cached-{marker}"
assert ask(_REASONING)["id"] == f"chatcmpl-cached-{marker}"
assert ask(f"a different thought {marker}")["id"] == f"chatcmpl-other-{marker}"
assert [_sent_messages(request)[1].get("reasoning_content") for request in wire.drain()] == [
assert [_sent_messages(request)[1].get("reasoning_content") for request in _provider_calls(wire)] == [
_REASONING,
f"a different thought {marker}",
]
@ -514,14 +534,14 @@ def _responses_reply(identity: str, stream: bool) -> Reply:
def _only_responses_body(wire: Wire) -> dict[str, JsonValue]:
received: Final = wire.drain()
received: Final = _provider_calls(wire)
assert [(request.method, request.target) for request in received] == [("POST", "/v1/responses")]
return _JSON_OBJECT.validate_json(received[0].body)
def test_openai_sdk_responses_replay_reaches_hosted_vllm_with_its_reasoning_item(gateway: Gateway) -> None:
marker: Final = uuid.uuid4().hex
with wire_server(lambda _: _responses_reply(f"resp_upstream_{marker}", stream=False)) as wire:
with _vllm_server(lambda _: _responses_reply(f"resp_upstream_{marker}", stream=False)) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
response: Final = _openai_client(gateway).responses.create(
@ -542,7 +562,7 @@ async def test_async_openai_sdk_responses_stream_reaches_hosted_vllm_with_its_re
gateway: Gateway,
) -> None:
marker: Final = uuid.uuid4().hex
with wire_server(lambda _: _responses_reply(f"resp_upstream_{marker}", stream=True)) as wire:
with _vllm_server(lambda _: _responses_reply(f"resp_upstream_{marker}", stream=True)) as wire:
with gateway.scenario() as scenario:
model: Final = scenario.model(model=f"hosted_vllm/{_BACKEND}", api_base=wire.url + "/v1", api_key=_API_KEY)
stream: Final = await _async_openai_client(gateway).responses.create(

2
tests/logging_callback_tests/gcs_pub_sub_body/spend_logs_payload.json
View file

@ -11,7 +11,7 @@
"user": "",
"team_id": "",
"organization_id": "",
"metadata": "{\"actor_agent_id\": null, \"target_agent_id\": null, \"billing_agent_id\": null, \"agent_execution_mode\": null, \"verified_human_user_id\": null, \"applied_guardrails\": [], \"attempted_fallbacks\": null, \"original_model_group\": null, \"batch_models\": null, \"batch_successful_requests\": null, \"batch_failed_requests\": null, \"mcp_tool_call_metadata\": null, \"vector_store_request_metadata\": null, \"routing_decision\": null, \"internal_call_origin\": null, \"router_metadata\": null, \"autorouter_savings_estimate\": null, \"autorouter_baseline_observation\": null, \"azure_spillover\": null, \"guardrail_information\": null, \"compression_savings\": null, \"litellm_gateway_injected_cache\": null, \"usage_object\": {\"completion_tokens\": 20, \"prompt_tokens\": 10, \"total_tokens\": 30, \"completion_tokens_details\": null, \"prompt_tokens_details\": null}, \"model_map_information\": {\"model_map_key\": \"gpt-4o\", \"model_map_value\": {\"key\": \"gpt-4o\", \"max_tokens\": 16384, \"max_input_tokens\": 128000, \"max_output_tokens\": 16384, \"input_cost_per_token\": 2.5e-06, \"cache_creation_input_token_cost\": null, \"cache_read_input_token_cost\": 1.25e-06, \"input_cost_per_character\": null, \"input_cost_per_token_above_128k_tokens\": null, \"input_cost_per_token_above_200k_tokens\": null, \"input_cost_per_query\": null, \"input_cost_per_second\": null, \"input_cost_per_audio_token\": null, \"input_cost_per_token_batches\": 1.25e-06, \"output_cost_per_token_batches\": 5e-06, \"output_cost_per_token\": 1e-05, \"output_cost_per_audio_token\": null, \"output_cost_per_character\": null, \"output_cost_per_token_above_128k_tokens\": null, \"output_cost_per_character_above_128k_tokens\": null, \"output_cost_per_token_above_200k_tokens\": null, \"output_cost_per_second\": null, \"output_cost_per_image\": null, \"output_vector_size\": null, \"litellm_provider\": \"openai\", \"mode\": \"chat\", \"supports_system_messages\": true, \"supports_response_schema\": true, \"supports_vision\": true, \"supports_function_calling\": true, \"supports_tool_choice\": true, \"supports_assistant_prefill\": false, \"supports_prompt_caching\": true, \"supports_audio_input\": false, \"supports_audio_output\": false, \"supports_pdf_input\": false, \"supports_embedding_image_input\": false, \"supports_native_streaming\": null, \"supports_web_search\": true, \"supports_reasoning\": false, \"search_context_cost_per_query\": {\"search_context_size_low\": 0.03, \"search_context_size_medium\": 0.035, \"search_context_size_high\": 0.05}, \"tpm\": null, \"rpm\": null, \"supported_openai_params\": [\"frequency_penalty\", \"logit_bias\", \"logprobs\", \"top_logprobs\", \"max_tokens\", \"max_completion_tokens\", \"modalities\", \"prediction\", \"n\", \"presence_penalty\", \"seed\", \"stop\", \"stream\", \"stream_options\", \"temperature\", \"top_p\", \"tools\", \"tool_choice\", \"function_call\", \"functions\", \"max_retries\", \"extra_headers\", \"parallel_tool_calls\", \"audio\", \"response_format\", \"user\"]}}, \"additional_usage_values\": {\"completion_tokens_details\": null, \"prompt_tokens_details\": null}, \"user_api_key\": null, \"user_api_key_alias\": null, \"user_api_key_team_id\": null, \"user_api_key_project_id\": null, \"user_api_key_project_alias\": null, \"user_api_key_org_id\": null, \"user_api_key_user_id\": null, \"user_api_key_team_alias\": null, \"spend_logs_metadata\": null, \"requester_ip_address\": null, \"user_agent\": null, \"status\": null, \"proxy_server_request\": null, \"error_information\": null, \"attempted_retries\": null, \"max_retries\": null}",
"metadata": "{\"actor_agent_id\": null, \"target_agent_id\": null, \"billing_agent_id\": null, \"agent_execution_mode\": null, \"verified_human_user_id\": null, \"used_client_oauth_token\": null, \"applied_guardrails\": [], \"attempted_fallbacks\": null, \"original_model_group\": null, \"batch_models\": null, \"batch_successful_requests\": null, \"batch_failed_requests\": null, \"mcp_tool_call_metadata\": null, \"vector_store_request_metadata\": null, \"routing_decision\": null, \"internal_call_origin\": null, \"router_metadata\": null, \"autorouter_savings_estimate\": null, \"autorouter_baseline_observation\": null, \"azure_spillover\": null, \"guardrail_information\": null, \"compression_savings\": null, \"litellm_gateway_injected_cache\": null, \"usage_object\": {\"completion_tokens\": 20, \"prompt_tokens\": 10, \"total_tokens\": 30, \"completion_tokens_details\": null, \"prompt_tokens_details\": null}, \"model_map_information\": {\"model_map_key\": \"gpt-4o\", \"model_map_value\": {\"key\": \"gpt-4o\", \"max_tokens\": 16384, \"max_input_tokens\": 128000, \"max_output_tokens\": 16384, \"input_cost_per_token\": 2.5e-06, \"cache_creation_input_token_cost\": null, \"cache_read_input_token_cost\": 1.25e-06, \"input_cost_per_character\": null, \"input_cost_per_token_above_128k_tokens\": null, \"input_cost_per_token_above_200k_tokens\": null, \"input_cost_per_query\": null, \"input_cost_per_second\": null, \"input_cost_per_audio_token\": null, \"input_cost_per_token_batches\": 1.25e-06, \"output_cost_per_token_batches\": 5e-06, \"output_cost_per_token\": 1e-05, \"output_cost_per_audio_token\": null, \"output_cost_per_character\": null, \"output_cost_per_token_above_128k_tokens\": null, \"output_cost_per_character_above_128k_tokens\": null, \"output_cost_per_token_above_200k_tokens\": null, \"output_cost_per_second\": null, \"output_cost_per_image\": null, \"output_vector_size\": null, \"litellm_provider\": \"openai\", \"mode\": \"chat\", \"supports_system_messages\": true, \"supports_response_schema\": true, \"supports_vision\": true, \"supports_function_calling\": true, \"supports_tool_choice\": true, \"supports_assistant_prefill\": false, \"supports_prompt_caching\": true, \"supports_audio_input\": false, \"supports_audio_output\": false, \"supports_pdf_input\": false, \"supports_embedding_image_input\": false, \"supports_native_streaming\": null, \"supports_web_search\": true, \"supports_reasoning\": false, \"search_context_cost_per_query\": {\"search_context_size_low\": 0.03, \"search_context_size_medium\": 0.035, \"search_context_size_high\": 0.05}, \"tpm\": null, \"rpm\": null, \"supported_openai_params\": [\"frequency_penalty\", \"logit_bias\", \"logprobs\", \"top_logprobs\", \"max_tokens\", \"max_completion_tokens\", \"modalities\", \"prediction\", \"n\", \"presence_penalty\", \"seed\", \"stop\", \"stream\", \"stream_options\", \"temperature\", \"top_p\", \"tools\", \"tool_choice\", \"function_call\", \"functions\", \"max_retries\", \"extra_headers\", \"parallel_tool_calls\", \"audio\", \"response_format\", \"user\"]}}, \"additional_usage_values\": {\"completion_tokens_details\": null, \"prompt_tokens_details\": null}, \"user_api_key\": null, \"user_api_key_alias\": null, \"user_api_key_team_id\": null, \"user_api_key_project_id\": null, \"user_api_key_project_alias\": null, \"user_api_key_org_id\": null, \"user_api_key_user_id\": null, \"user_api_key_team_alias\": null, \"spend_logs_metadata\": null, \"requester_ip_address\": null, \"user_agent\": null, \"status\": null, \"proxy_server_request\": null, \"error_information\": null, \"attempted_retries\": null, \"max_retries\": null}",
"cache_key": "Cache OFF",
"spend": 0.00022500000000000002,
"total_tokens": 30,

5
tests/unit/conftest.py
View file

@ -182,6 +182,11 @@ def _flush_client_caches() -> None:
_reset_aws_auth_caches()
@pytest.fixture(autouse=True, scope="session")
def bundled_tiktoken_cache() -> None:
importlib.import_module("litellm.litellm_core_utils.default_encoding")
@pytest.fixture(scope="session")
def isolated_aws_config_files(tmp_path_factory: pytest.TempPathFactory) -> tuple[Path, Path]:
aws_dir: Final = tmp_path_factory.mktemp("aws-config")

7
tests/unit/litellm_core_utils/conftest.py
View file

@ -1,15 +1,8 @@
import importlib
import pytest
from tests.unit.litellm_core_utils.fake_secret_vault import FakeSecretVault
@pytest.fixture(autouse=True, scope="session")
def bundled_tiktoken_cache() -> None:
importlib.import_module("litellm.litellm_core_utils.default_encoding")
@pytest.fixture
def secret_vault_factory() -> type[FakeSecretVault]:
return FakeSecretVault

4
tests/unit/llms/custom_httpx/test_http_handler.py
View file

@ -7,6 +7,7 @@ import ssl
import threading
import weakref
from collections.abc import Callable, Mapping
from concurrent.futures import ThreadPoolExecutor
from typing import Final
from unittest.mock import MagicMock, patch
@ -1388,7 +1389,8 @@ async def test_finalizer_on_live_loop_disposes_foreign_loop_session_without_sche
another, dead loop must not schedule aclose() here — that is the cross-loop
path the transport refuses — and must still dispose the session."""
handler = AsyncHTTPHandler(timeout=61.0)
session = await asyncio.to_thread(_mint_session_on_dead_loop, handler)
with ThreadPoolExecutor(max_workers=1) as pool:
session = pool.submit(_mint_session_on_dead_loop, handler).result()
assert not session.closed
baseline_tasks = set(AsyncHTTPHandler._finalizer_close_tasks)

2
tests/unit/router_strategy/test_budget_limiter_hotpath.py
View file

@ -385,7 +385,7 @@ async def test_push_task_failure_is_logged_once_and_not_leaked(disable_budget_sy
finally:
loop.set_exception_handler(None)
assert [record.getMessage() for record in caplog.records] == [
assert [record.getMessage() for record in caplog.records if record.name != "asyncio"] == [
"Error syncing in-memory cache with Redis: Error 61 connecting to 127.0.0.1:6379"
]
unretrieved.assert_not_called()

4
ui/litellm-dashboard/src/components/edit_auto_router/edit_auto_router_modal.integration.test.tsx
View file

@ -157,7 +157,9 @@ describe("EditAutoRouterModal keyword matching", () => {
const threshold = screen.getByRole("textbox", { name: "Success threshold" });
expect(threshold).toHaveValue("0.91");
fireEvent.change(threshold, { target: { value: raw } });
await waitFor(() => expect(screen.getByRole("button", { name: "Save Changes" })).toBeEnabled());
await waitFor(() => expect(screen.getByRole("button", { name: "Save Changes" })).toBeEnabled(), {
timeout: 5000,
});
await user.click(screen.getByRole("button", { name: "Save Changes" }));
await waitFor(() => expect(modelPatchUpdateCall).toHaveBeenCalledOnce());
if (raw === "") expect(savedConfig()).not.toHaveProperty("heuristic_v2_success_threshold");