Regenerated every touched file from origin/main applying only the B1 test deletions and the unused import and helper cleanup they leave behind, without running the formatter across untouched code. CI only checks ruff format under litellm/, so the earlier reflows of test files were pure diff noise for reviewers
Also drops the tests/local_testing/test_prompt_caching.py entry from the caching-local shard in test-unit.yml since that file is deleted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The repo rule is that a test must only fail when litellm code changes, never when a vendor updates a price, renames a field, or drops a model. These tests asserted shipped catalog entries directly, comparing lookup results to literals copied from model_prices_and_context_window.json or requiring named entries to exist or be absent, so every cost map sync could break them without any litellm code changing
Tests that exercise real litellm behavior with an injected local model_cost, invariants like backup parity, and assertions on non-lookup code paths are untouched
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_router_fallbacks_with_cooldowns_and_dynamic_credentials expected a
caller-supplied credential to register its own deployment and cool it down.
#41289 stopped registering it, so cooldown logic skips that id and the
assertion can never hold. The test now asserts what the router guarantees
today: a 429 to a forwarded credential cools down none of the shared
deployments, the next credential is still served, and a 429 owned by a shared
deployment still cools it down. The final live OpenAI call becomes a mock
The auto-router template spec assumed the Add Auto Router form left room
below the Template select at 1280x900. #41315 added classifier fields above
it, so the options opened upward. The spec now scrolls the trigger to the top
of the dialog and asserts it sits in the upper half before checking placement
Every one of these fails on main's own scheduled CircleCI run with the same
assertion as on any PR, and each traces to a merged behavior change that
never updated the test that pinned the old behavior
- tests/integration/_support/client.py: #41311 made /key/info serve deleted
keys from the archive with status deleted, so the scenario teardown asserts
the live row is gone and the readback reports deleted instead of a 404.
This alone accounts for nine integration-management and one
integration-providers failure
- tests/integration/authorization/test_warmed_policy.py: #39996 made team
admins unable to edit any team field unless a proxy admin allow-lists it,
and tpm_limit is the only field it accepts today. The demotion test now
enables tpm_limit for the scenario and edits that instead of team_alias
- tests/llm_responses_api_testing/test_base_responses_api_streaming_iterator.py:
#41337 reads usage off the terminal response and copies the event when it
is missing, which a Mock(spec=ResponsesAPIResponse) cannot survive. The
four mocks now carry a usage object
- tests/test_openai_endpoints.py: #41310 lengthened the access-denied
message, and the test matched against the ExceptionInfo repr, which
saferepr truncates in the middle. It now matches the exception text
- tests/local_testing/test_text_completion.py: Together no longer serves
Qwen2-1.5B serverless, the cheapest cost-map row. The test mocks the
completions call and asserts the request litellm builds, so a vendor
catalog rotation cannot fail it again
test_router_fallbacks_with_cooldowns_and_dynamic_credentials is deliberately
untouched: it passes and fails on main with identical code, and the failing
path is a product question about whether dynamic-credential 429s cool down
CI runs these lanes as `python -m pytest` from the repo root, so the root is
already on sys.path and `tests._live_test_helpers` imports without help. The
insert only tripped the TQ003 test-quality budget.
`together_ai/openai/gpt-oss-20b` was hardcoded in two live tests and is no
longer served, so both failed on a vendor catalog change rather than on
anything litellm did.
Both call sites now resolve the cheapest non-deprecated together_ai chat
entry at runtime, filtered on the capabilities the tests actually exercise,
mirroring what tests/e2e/llm_translation/test_together_ai_e2e.py already
does. The selector lives in tests/_live_test_helpers.py so both lanes share
one implementation.
Together moved openai/gpt-oss-20b off serverless and three tests in
test_completion.py died on a live 400. None of them needed Together to be up:
streaming is already covered live by tests/e2e/llm_translation/test_together_ai_e2e.py,
which picks its model from the cost map instead of pinning one, and the other
two are request-shape questions. Delete all three and assert the two shapes in
the mapped transformation file: the provider prefix is stripped without eating
the rest of a slashed model name, and custom role wrappers never reach the
request.
Together moved openai/gpt-oss-20b off serverless, so three tests started
failing with a 400 model_not_available from the live API. None of them was
really testing Together: they cover provider-prefix parsing, prompt shaping
and streaming, all litellm side. Mock the transport and assert those, so the
tests answer to our code instead of a vendor catalog.
simple_shuffle logs the selected deployment at INFO whenever a weight set
applies, so the fallback group's selection line lands between the fallback
notice and the success notice and pushed the notice out of the tail-3 window.
Filter it the same way the neighbouring get_available_deployment noise is
already filtered.
_handle_clientside_credential registered the per-request Deployment it built for
a client-supplied api_key/api_base via upsert_deployment, which added it to
self.model_list under the shared model_name. That made a request-scoped
credential a permanent, load-balanced deployment that any later caller of the
same model group could be routed onto, reaching the provider with someone
else's forwarded credential.
The per-request Deployment still gets its own stable id for cooldown and
logging identity; it is just never registered with the router.
Resolves LIT-7811
A converted-stream request whose cache entry is a plain (non-stream) object is
replayed as that plain object, so nothing later fires the success callbacks.
Decide deferral from the replayed result's type instead of the request kwargs.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): rank streaming latency routing by raw TTFT, not TTFT per token
Latency-based routing divided time-to-first-token by completion_tokens
before storing it, so a deployment that streamed a long answer looked
faster to first token than one that answered briefly. TTFT is now stored
as plain seconds (first token time minus request start) in both the sync
and async success handlers, which is what the routing decision compares.
Non-streaming latency normalization per output token is unchanged.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
* fix(router): store streaming TTFT under a seconds-only cache key
Workers on the previous release keep writing seconds-per-token samples
under "time_to_first_token" in the shared router cache during a rolling
deploy, so mixing the new raw-seconds samples into the same list averaged
incompatible units. Raw TTFT now lives under "time_to_first_token_seconds"
and routing reads only that key.
Also fix the regression test's token counts: with 50 tokens on the fast
deployment and 500 on the slow one the old per-token formula picks the
slow deployment, so the routing assertion now catches the bug.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
* test(router): cover the TTFT sliding window from the unit-test shard
Move the TTFT list trimming checks from the CircleCI-only suite into the
mapped unit test file as one sync/async parametrized test, so the changed
lines in lowest_latency.py are exercised by the GitHub unit-test shard
that reports patch coverage.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
_handler_may_close_client withholds the finalizer's close from a client that
someone else references. A streaming response is not one of those referrers: it
holds the connection it reads from, never the client, so the refcount says
"sole referrer" for exactly the client that is busiest, and __del__ tears the
pool down mid-body. Both handlers, both transports, and it is the shape #24929
reports: a provider's streaming call returns the response and drops the
handler, and get_async_httpx_client lets a cached handler go after an hour.
A streaming send now anchors the handler to the response it returns, so the
handler is finalized once the caller is done with the body and the ordinary
close applies with nothing in flight. The anchor is a weakref.finalize, which
holds the handler in its own registry rather than on the response. That keeps
the handler out of the response's reference cycle, so it is finalized by
refcount and can still schedule an async close, rather than inside a cyclic
collection that reaps its aiohttp session in the same pass; and a handler
serving several streams is released only once all of them are done, because
each anchor holds it separately.
Only a streaming send anchors. A non-streaming response has been read in full
by the time the method returns, so pinning the handler to it would delay every
client close behind whatever the caller does with the response.
The alternative was to ask the connection pool whether a request was in flight,
reusing EvictedClientCloser's check. It reads client._transport, so it answers
"idle" for any client with a proxy configured, where httpx routes through
client._mounts; and having found the client busy it can only poll, which never
terminates for a response the caller abandons unread, since httpx leaves that
connection checked out. The response's own lifetime is the condition both were
approximating.
http_handler.py resolved to main. This branch deleted both __del__ methods;
main has since kept them and guarded them (#35981 refcount + ownership, #36670
loop-aware aiohttp session disposal), and added tests that assert a collected
handler does close an exclusively owned client. Taking this branch's deletion
would remove all of that, so the deletion is dropped here and the case main's
guard still misses -- a response in flight -- is fixed in the next commit.
Two routers in one process shared a single handler, because the callback
manager dedupes on the class name plus the handler's public attributes and the
handler had none. The second router's requests were never counted. The handler
now carries the id of the cache it was built on, so routers with different
caches both register while the two selectors one router builds for its routing
groups still collapse into one.
Clamping a negative count back to zero used SET, which drops the key's TTL, so
the next write started the hour over. It uses INCRBY by the negative amount now,
which leaves the expiry alone.
The Lua script had no test that ran it, so tests/local_testing covers both the
sync and async paths against a real Redis, and the file is wired into the
CircleCI job that provides one.
Least-busy kept one dict of in-flight counts per model group in the
router cache, which reads in-memory first, so every worker and replica
routed on its own stale copy and each write overwrote the shared value.
Counts now live in one key per deployment, incremented and read through
Redis when the router has a Redis cache, and in the process-local cache
otherwise.
Bedrock retired cohere.command-r-plus-v1:0 on 2026-08-19 and lists no
Cohere command chat model anymore, so the three local_testing cases that
pinned it fail with a 404 end-of-life error on every pipeline. Drop the
case from test_completion_bedrock_httpx_models and move the
parallel-streaming Bedrock entry to mistral.mistral-7b-instruct-v0:2,
which still takes the invoke route and is ACTIVE in the CI account.
test_router_timeout, test_timeout_streaming and test_openai_embedding_timeouts
asked api.openai.com for a response in 10 to 100 microseconds and asserted the
resulting exception was a timeout. No connect can finish in that window, so
socket.create_connection always walked the whole address list, and because it
re-raises only the LAST address's error, the assertion was decided by the order
getaddrinfo happened to return.
api.openai.com is dual-stack and the CI container has no usable IPv6, so a
trailing AAAA record made the last attempt fail with an OSError. httpcore maps
socket.timeout to ConnectTimeout but OSError to ConnectError, so the expected
APITimeoutError arrived as APIConnectionError and the job went red. The three
tests were really measuring DNS ordering, not litellm.
Point them at the fake OpenAI endpoint the suite already runs, ask for the
slow-endpoint model it already delays on, and give them a timeout comfortably
under that delay. The embeddings route did not honour slow-endpoint yet, so it
now delays the same way chat and text completions already do.
Each test also gained a failure on the success path. Without it a request that
returned instead of timing out fell out of the try block and the test passed on
a result it was written to reject.
This reverts merge commit 2b1bd20834 (#31125)
Two CircleCI jobs on the staging-to-main promotion went red the moment
that PR landed. proxy_multi_instance_tests boots two proxies against one
database, and both now race the same migration:
Error: P3018 A migration failed to apply
Database error code: 40P01, deadlock detected
Process 73 waits for ShareLock on virtual transaction 4/11;
blocked by process 75. Process 75 waits for ExclusiveLock on
advisory lock [16384,0,72707369,1]; blocked by process 73
Neither proxy comes up, so the job times out after 300s waiting on
localhost:4000. The same wait took 36.5s on the last green run
Timeline: #31125 merged at 18:46:14Z and the failing run started at
18:49:59Z. The merge commit is not an ancestor of the last green
revision (194a3cc) and is an ancestor of the first failing one
(01de2837)
The v2 resolver was meant to avoid exactly this class of contention, so
the deadlock looks like a bug in it rather than a reason to abandon it.
Putting the default back to v1 buys time to fix it without holding up
the release
The omitted-format path deliberately no longer dispatches through
embeddings.create, so four legacy tests now intercept at the transport or
client.post instead. Also adds a bypass error-path unit test, rewords a stale
comment and a README scope note, and ratchets the lint budgets down.
Consolidate the new regression tests into
test_openai_embedding_encoding_format_default.py, replacing mocks that
pinned the old float default with respx captures of the request body,
and update the stale local_testing default-float test to assert
omission
The v2 resolver skips the diff-and-force recovery that caused schema
thrashing when two LiteLLM versions contend for one database during a
rolling deploy. The standalone migration Job already defaulted to v2; this
aligns the proxy-server path.
v1 stays reachable two ways: --use_legacy_migration_resolver on the CLI, and
USE_V2_MIGRATION_RESOLVER=false for containerised deploys, where
prisma_migration.py calls run_server with a fixed argv and the env var is the
only route in. --use_v2_migration_resolver still parses, so existing commands
do not die on an unknown option.
Because v2 fails fast where v1 retried every failed deploy, a database that is
not accepting connections yet, or another instance holding the migration
advisory lock, would now kill a boot that used to ride it out. Those two
failures are retried, with Prisma's stderr logged each round, and still raise
once the attempts are spent.
Moves the resolver tests from litellm-proxy-extras/tests, which no CI job
runs, into tests/litellm-proxy-extras, and repoints the dedicated Postgres
CircleCI job at the legacy path so v1 keeps real-DB and proxy-boot coverage.
