A flat list[str] sent to voyage-context-4 is treated as independent inputs and forwarded flat with
enable_auto_chunking=True, chunk_size=32000, and input_type=document unless the caller already set
input_type=query. Caller-supplied params now override the defaults instead of being clobbered.
The voyage-4 family and voyage-context-4 cost map entries already exist on litellm_internal_staging,
and voyage-4-nano is not served by the Voyage API, so those additions and their pricing test are dropped.
Operators can now set aws_session_tags on a Bedrock or SageMaker
deployment and every AssumeRole call carries them as STS session tags,
so trust policies gated on sts:TagSession admit the session and
CloudTrail and cost reports see the tags. Tags are validated up front,
sorted into the credential cache key so tag order does not fork
sessions, stripped from invoke and embedding request bodies, and
blocked from client request bodies like the other AWS identity params.
Based on #34073.
Co-authored-by: Satya Yedida <222552552+satya-rubrik@users.noreply.github.com>
Access is confirmed on the Anthropic account, and all 11 cells pass live, so the
xfail marker would only hide real regressions.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds claude-fable-5-1 cost map entries on the Anthropic API, Bedrock converse
(base, global, and us/eu geo inference profiles at the 10% regional premium),
Vertex AI, and Azure AI. Specs match Fable 5 (1M context, 128K output, $10/$50
per MTok, adaptive thinking always on, xhigh and max effort), except cache reads
land at $0.25 per MTok, a quarter of Fable 5's price and 0.025x base input
instead of the usual 0.1x.
Registers anthropic.claude-fable-5-1 in BEDROCK_CONVERSE_MODELS, lists the model
in the setup wizard, and extends the reasoning effort e2e grid. The partner cells
carry fail_reason markers until access on the CI accounts is confirmed.
Partner entries deliberately carry no deprecation_date: Anthropic publishes
retirement no sooner than 2027-09-01 for the first-party model, and the Foundry
and Vertex dates are not published yet.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Resolves budget-ratchet conflicts by taking staging's tighter limits and reworks the embedding raw-response helpers so the branch stays net-negative on the LIT001/LIT002 ceilings staging lowered: the request methods now return the LegacyAPIResponse and each caller keeps a single dict(headers) conversion.
The omitted-format path deliberately no longer dispatches through
embeddings.create, so four legacy tests now intercept at the transport or
client.post instead. Also adds a bypass error-path unit test, rewords a stale
comment and a README scope note, and ratchets the lint budgets down.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
A test that asserts on the error inside its own except block passes when the
call stops raising, because nothing runs the handler. That is the exact case
the test exists to catch, so the regression lands green.
Rewrites all 111 such blocks into pytest.raises, which fails when the call
succeeds, and selects PT017 in ruff-tests.toml so no new one lands.
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
Add sync and async regression tests for the BaseLLMHTTPHandler streaming
path, which forwards provider response headers for the ~30 providers that
ride the generic handler and had no coverage. Also drop redundant setup
prose from the moonshot invoke test docstring.
* test: enforce PT012 so a pytest.raises block cannot hide dead assertions
`with pytest.raises(...)` stops at the first statement that raises. Anything
sequenced after it inside the block never runs, so an assertion written there is
never checked and the test still reports green.
Two sites were doing exactly that, and both assertions turned out to be wrong
once they started running. tests/llm_translation/test_prompt_factory.py asserted
the bedrock rejection names "requires at least one non-system message", which
holds. tests/proxy_unit_tests/test_proxy_server.py asserted the prisma startup
failure mentions "httpx.ConnectError", which never appears: the failure is an
httpx.ConnectError whose message is "All connection attempts failed", so that
test now asserts the type. Its DATABASE_URL override moves to monkeypatch, since
the old restore sat below the assertion and leaked the invalid URL into every
later DB test the moment the assertion started being able to fail.
The remaining 72 sites are rewritten without changing what they exercise: setup
that cannot raise moves above the block, a nested `patch` moves outside it, and
bodies with real control flow (a stream drain, an if/else on sync_mode, a
retry loop) move into a local closure the block calls.
Fixing PT012 unmasked two B017s, since ruff only reports a blind
pytest.raises(Exception) once the block holds a single statement.
tests/proxy_unit_tests/test_auth_checks.py narrows to the ProxyException
can_key_call_model actually raises. tests/local_testing/test_completion_cost.py
was asserting vertex_ai/medlm-medium has no cost entry, which stopped being true
at some point; that dead first half is gone and the rest of the test, which
checks medlm pricing resolves above zero, now runs instead of being skipped.
* chore(ci): ratchet TQ004 to 768 after the prisma test moved to monkeypatch
Every live together_ai call in CI has answered 503 Service unavailable since
2026-08-20, across two runs 2.5 hours apart, while Together's status page
reported no incident in either window. These are real calls, not replayed
cassettes: the VCR layer runs filter_non_2xx_response, so a 503 is never
written to a cassette and cannot be replayed back.
Qwen/Qwen2.5-7B-Instruct-Turbo does not appear anywhere on Together's monitored
component list, whose Qwen entries are all Qwen3.x, so a model-level outage
there would never surface as an incident. The same 503 already forced
test_basic_rerank_together_ai to be skipped on a different together_ai model,
so per-model 503s are an established failure mode here rather than a platform
outage.
openai/gpt-oss-20b is the cheapest together_ai entry that carries real pricing
and the capabilities these suites exercise, at $0.05/$0.20 per 1M tokens with
function calling, response schema and tool choice. Together monitors it as a
served component. The retired model also carries null pricing in the cost map,
which is its own liability now that unpriced models are blocked.
test_multiple_deployments.py keeps the old id: it is a router fallback list
that is green today, and busting its cassette to prove a point would trade a
passing test for a live call this change cannot vouch for.
a369cb0da7 made completion() hand the prefixed model back to responses(), so
that responses() running get_llm_provider() a second time becomes a no-op
instead of stripping a prefix the model id owns. That was deliberate, and it
shipped with its own unit test, but it left two older assertions behind still
expecting the bare id.
#37744 corrected the openai one in test_openai.py. This is its azure sibling,
which llm_translation_testing has been failing on ever since.
Only the expected value moves. The neighbouring custom_llm_provider assertion
already passes and stays as it is.
* test(lint): ban blind pytest.raises(Exception) with ruff B017
A bare pytest.raises(Exception) accepts whatever the body throws. The TypeError
a refactor introduces satisfies it exactly as well as the rejection the test was
written for, so the crash reads as a pass and the test never goes red.
All 111 existing sites are narrowed here. A runtime probe recorded the concrete
exception each one actually catches, and each site now names that type. Where
the code under test genuinely raises a bare Exception, the site pins a stable
slice of the message with match= instead.
Two sites tell on themselves. The shared responses-API cancel test raises
"custom_llm_provider is required but passed as None" rather than talking to a
provider at all, because cancel_responses takes a provider, not a model. And
test_bedrock_guardrails_with_streaming was the only test in its file still
passing without AWS credentials, because the NoCredentialsError boto3 raised
long before the guardrail ran satisfied the blind raises.
* fix(test): widen the openai batch-dispatch assertion to OpenAIError
The narrowed NotFoundError only holds where OPENAI_API_KEY is set. Without one
the SDK raises OpenAIError while building the client, long before any 404, so CI
went red. OpenAIError covers both and still rejects a TypeError from a refactor.
a369cb0da7 made the chat-to-responses bridge return the routing prefix on the
model it passes to responses(), so responses() re-resolving the provider is a
no-op instead of stripping a second prefix. It updated the bridge's own unit
tests but not this one, which still asserted the stripped id and has been
failing llm_translation_testing since that change landed.
The provider still receives gpt-5.4: responses() strips the openai/ prefix on
its own resolve, one layer later than this assertion used to sit. The stale
comment claiming the prefix is stripped before routing goes with it.
* test: point the live web search, groq and vertex image suites at models that still exist
Three CircleCI jobs on the staging-to-main promotion are red because the models
their live suites call have been retired by the providers, not because anything
in litellm changed.
openai/gpt-4o-search-preview now answers "has been deprecated" (its dated id
gpt-4o-search-preview-2025-03-11 carries deprecation_date 2026-07-23), so the
two web search conformance tests and the web search cost tracking test move to
gpt-5-search-api, the current search model. It keeps mode chat,
supports_web_search and a search_context_cost_per_query map, so the cost
assertion still resolves.
groq/llama-3.1-8b-instant reached its deprecation_date of 2026-08-16 and Groq
answers "does not exist or you do not have access to it". It follows
groq/llama-3.3-70b-versatile to groq/openai/gpt-oss-120b, the same replacement
PR #37422 already picked. The proxy config that job boots routes on a */*
wildcard, so no config change is needed.
vertex_ai/imagen-3.0-fast-generate-001 404s with "was not found or your project
does not have access to it". Google retired the whole Imagen family across
Vertex and the Gemini API, so there is no Imagen id left to point at. The class
is removed rather than repointed: Vertex image generation is already covered
live by TestVertexAIGeminiImageGeneration on vertex_ai/gemini-2.5-flash-image,
and the Imagen request and response transformations keep their offline coverage
in tests/test_litellm/llms/vertex_ai/image_generation/.
Only live call sites move. Remaining references to the old ids sit in offline
cost-map and transformation tests, where the string is a lookup key and no
request leaves the process.
* chore(lint): ratchet the TQ005 ceiling down to the count this branch reached
Removing the retired TestVertexImageGeneration class cleared one TQ005
violation, so the gate demands the limit come down with it.
make lint-budget-update only lowers a limit by the delta a branch cleared, and
this ceiling already sat 2 above the base count, so the tool landed on 2834
while the gate wants the limit at or below the 2832 this branch reached. The
remaining 2 are that stale headroom, which is exactly what the gate is asking
to reclaim.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
Python binds a name once per scope, so when a module or class defines the same
test twice only the last one exists. The earlier definitions are unreachable:
pytest never collects them, and nothing that references them can fail.
A sweep in August cleared nine of these. Five have appeared since, which is the
argument for a rule rather than another sweep.
Each survivor is the better version, so nothing is lost. The two SQS logger
twins additionally stub `asyncio.create_task`, which the shadowed copies did
not. The cost-calculator duplicate is a two-line stub that also takes a
`model_item` parameter no fixture supplies, so it could not have run even
unshadowed. The two `test_prompt_caching` bodies are both `pass`.
Collecting the four files reports 416 tests before and after.
`tests/proxy_unit_tests/conftest copy.py` goes with them. pytest only loads a
file named exactly `conftest.py`, nothing imports this one, and the space in the
name says what it was.
gemini/imagen-4.0-generate-001 now 404s with "no longer available, please
update your code to use models/gemini-3.1-flash-image", and
groq/llama-3.3-70b-versatile reached its deprecation_date of 2026-08-16, which
is the day the llm_translation and image_gen jobs went red.
Only the two live call sites move. The remaining references to the old ids sit
in offline transformation tests, where the string is just a routing key and no
request leaves the process.
Python keeps only the last binding for a name, so when a file defines the same
test twice the earlier one is unreachable. pytest cannot collect a function that
no longer exists, so nothing reports it and the file still looks like it covers
the scenario.
These ten are cases where the two definitions have different bodies, meaning a
real test was replaced rather than duplicated. Each is renamed to say what it
actually covers, which makes it reachable again:
- test_gemini_frequency_penalty: the dead copy checks the parameter is listed in
get_supported_openai_params for vertex_ai; the survivor checks get_optional_params
maps a value for gemini. Different function and different provider.
- test_async_log_success_event_adds_to_queue and the failure variant: the dead
copies run without mocking asyncio.create_task, so they exercise the real task
path the survivors mock out.
- test_async_send_batch_triggers_tasks: the dead copy asserts send is not awaited
directly; the survivor asserts create_task was called.
- test_model_id_in_required_metrics: the dead copy checks the model_id label on
twelve further metrics the survivor dropped.
- test_anthropic_messages_pt_file_block_preserves_cache_control: the dead copy
passes model and llm_provider explicitly and uses real base64 PDF content.
- test_translate_streaming_openai_chunk_to_anthropic_with_thinking: the dead copy
covers thinking_delta; the survivor covers signature_delta.
- test_client_initialization and test_client_without_api_key: the dead copies
assert the resource clients are wired with the right base URL and key; the
survivors only construct the object.
- test_client_initialization_strips_trailing_slash: the dead copy constructs
ModelsManagementClient directly rather than going through Client.
Verification: collecting the seven touched files gives 401 node IDs before and
411 after, the ten new names and nothing else, with nothing lost. All ten pass.
Running the touched files in full gives 299 passed, and test_optional_params.py
goes from 111 passed to 112.
Two further shadowed definitions were left alone rather than renamed: the dead
copies of test_prompt_caching and test_cost_calculator_with_base_model_with_router
have no assertions at all, one being a bare pass and the other a lone import, so
restoring them would add tests that cannot fail.
Three groups, all verified by running the suite rather than by inspection.
18 files whose every test function carries an unconditional @pytest.mark.skip,
39 test functions in total. They are collected on every CI run and always skip,
so they advertise coverage the suite does not have. Reasons on the marks include
"AWS Suspended Account", "lakera deprecated their v1 endpoint" and "moved to
using 'otel' for logging"; 26 of the marks predate 2025.
30 test functions with a byte-identical body and identical decorators to a
sibling in the same file and class, differing only in name. Deleting one of each
pair removes no coverage. Four further candidates were excluded because they
override an inherited test, where deleting the override un-shadows the base
class implementation instead of removing a duplicate.
9 test functions that a later definition of the same name shadows, so Python
never binds them and pytest cannot collect them.
One file that is a demo script rather than a test; its own docstring says to run
it with python.
Verification: collecting the 26 edited files gives 2,492 node IDs before and
2,462 after. The 30 duplicate deletions account for exactly 30 removals, the 9
shadowed deletions account for 0 (confirming at runtime that they were never
collectable), nothing unexplained disappeared, and nothing new appeared. No
other test or module imports any deleted symbol.
AWS no longer serves `anthropic.claude-3-sonnet-20240229-v1:0`. The streaming
path returns a plain 404, "Model with the provided id
anthropic.claude-3-sonnet-20240229-v1:0 is not found", and the non-streaming
path answers 500 for the same reason. Our own cost map has carried a
2026-07-30 deprecation date for it since #36538
That accounts for 20 failures across local_testing_part1, local_testing_part2
and llm_translation_testing. litellm maps both statuses correctly, so the
tests are what went stale, not the client
Replacement is `us.anthropic.claude-sonnet-4-5-20250929-v1:0`: a like-for-like
Sonnet, and the newest Bedrock Sonnet this repo exercises against the real API
in tests/e2e. Newer ids exist in the cost map, but nothing in the repo calls
them live, so picking one would be an unverified guess about model access on
the CI account
Scope is limited to the tests that actually issue a request. The occurrences
that assert on the model string itself, or that feed mocked transformations,
keep the old id so their assertions stay meaningful
Rework per review:
- Contextual inputs now prefer a flat list[str] and let the API auto-chunk
(enable_auto_chunking=True, chunk_size=32000, input_type=document) instead of
pre-wrapping into list[list[str]]. Matches the live API contract:
* flat list[str] + input_type=query -> kept flat
* flat list[str] otherwise / single str -> flat + auto-chunking
* list[list[str]] -> passthrough
- Fix mypy list-item error from the previous wrapping approach.
- Update tests for the new normalization; all pass.
- Add voyage-4, voyage-4-large, voyage-4-lite, voyage-4-nano (/bin/bash) and
voyage-context-4 to the cost/context map (both root and backup json).
- Contextual embeddings: normalize flat list[str] input. Voyage rejects a
flat list[str] unless input_type='query', so wrap each string as its own
single-chunk document otherwise; single str -> [[str]]; list[list[str]]
passes through. Keeps list[str] when input_type='query'.
- Add tests for input normalization and voyage-4 family pricing.
