AGENTS.md allows a vendor-owned literal only when its source and date are cited
next to it. A live realtime test cannot avoid naming a model, so record how the
id was checked, and record that a retired id fails as a hang rather than an
error so the next reader does not start by suspecting litellm.
(cherry picked from commit 9f84382a24)
AWS retired amazon.nova-sonic-v1:0. GetFoundationModel now answers
ResourceNotFoundException "This model version has reached the end of its life",
and opening a bidirectional stream against it fails with ValidationException
"The provided model identifier is invalid". amazon.nova-2-sonic-v1:0 is the
active replacement.
The test passed on builds 219 (2026-09-16) and 246 (2026-09-17) and has failed
every run since, three attempts per build, with no litellm change to the
realtime path in between. The symptom was a clean websocket close: Bedrock ends
the stream rather than erroring, the forwarder treats a None receive as a normal
stream end and closes the client socket, so the client sees ConnectionClosedOK
and the test fails waiting for response.done.
litellm already carries both models in the cost map, with
"deprecation_date": "2026-09-14" on the old one, and the promptStart
transformation already sends the audioOutputConfiguration that nova-2-sonic
requires; only the test constant was left behind.
Verified against live Bedrock with the promptStart shape the transformation
builds: amazon.nova-sonic-v1:0 raises "The provided model identifier is
invalid", amazon.nova-2-sonic-v1:0 opens a session and returns a usageEvent.
The mocked handler and provider-cache tests keep the old id: it is only a label
there, no call reaches AWS.
(cherry picked from commit ca18755b64)
Backport of #41347 to rc/1.102.0.
Cherry-picked from merge commit 4bb1ae115b (main), originally by app/devin-ai-integration.
Conflicts: main's #41349 (membership rows written through upsert) is not on this line, so add_new_member keeps its create call and still writes no membership row when no budget resolves. The picked tests are adapted to that and to this line's _check_team_member_budget, which loads the membership itself.
Backport of #41689 to rc/1.102.0.
Cherry-picked from merge commit 07b5051c0d (main), originally by app/devin-ai-integration.
Both conflicts were in test files: the rc line lacks the Codex additional_tools tests that sit next to the new test in test_handler.py on main, and its test_utils.py imports all_litellm_params on a separate line, so only this PR's own additions (the imports, the recording handler, and the two new tests) are taken.
Backport of #41495 to rc/1.102.0.
Cherry-picked from merge commit 0add8c0083 (main), originally by app/devin-ai-integration.
The only conflict was the test import header: the rc line never gained the logging_executor import that main carries next to this change, so only the MockResponseIterator import comes along.
Backport of #41086 to rc/1.102.0.
Cherry-picked from merge commit 4123b4bc2b (main), originally by app/devin-ai-integration.
rc/1.102.0 has no litellm/proxy/auth/resolvers/grants.py, so its one-line import change is dropped; auth_checks.py imports UserNotFoundError from the new types module, so every importer on this line still resolves.
/model/info fills a deployment's missing pricing in from the model cost map so the
Admin UI has a rate to display. Clients echo that whole model_info blob back on save,
and update_db_model merged it into the row, so editing an unrelated setting turned
that day's catalog price into a real per-deployment override. After that the
deployment ignored the cost map and Reload Price Data could no longer move it,
because the reload replays each deployment's stored pricing over the fresh catalog.
Drop the derived pricing from incoming model_info on the two write paths. The
drop-set is read off the same objects the read path uses, CustomPricingLiteLLMParams
plus the tiered *_above_N_tokens pattern that get_model_info passes through and no
model declares, so it cannot drift as new rates are added. output_vector_size is
exempt: it lives on the pricing model but is an embedding dimension, not a rate.
A deployment's own pricing still rides litellm_params, which is untouched, as is the
explicit-null clear, which reads the incoming model rather than the filtered dict.
The filter sits in the endpoint bodies rather than _add_model_to_db, which master-key
rotation reuses to re-serialize every stored deployment.
* fix(proxy): attribute gate-rejected requests to their endpoint in cache analytics
Requests rejected before dispatch (bad key, blocked key, budget, rate limit, malformed body) were spend-logged with an empty call_type because the synthesized logging object never reached the failure lifter. The caching dashboard rolled all of them, plus failed calls on info routes such as /model/info, into one Unknown group.
Resolve call_type from the matched route first, falling back to body shape, and keep the synthesized logging object on request_data so the lifter sees it. Log bare auth exceptions with the 401 ProxyException the client gets so error_code is never empty. Exclude info routes from the cache analytics groups and error breakdown. The dashboard explains the Unknown group when older rows still produce one.
Resolves LIT-5884
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the raw auth exception for failure callbacks
Record the client-facing status in the spend log through a separate client_exception argument so custom failure callbacks still receive the exception auth raised.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the route for multi-operation endpoints and exclude info routes from cache filter options
Routes such as /v1/files map to several operations (create, list) and the
method is not available in the failure hook, so a rejected request there is
filed under its route instead of the first mapped call type. The key alias and
model filter-option queries now apply the same info-route exclusion as the
groups and error breakdown, so every offered filter value returns data. The
info-route exclusion and Unknown grouping are now covered against a real
Postgres in tests/proxy_behavior/spend/test_cache_activity.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop client_exception, the spend log row never used it
The DB spend row for a gate rejection is written by _ProxyDBLogger from the
original exception, so the status-bearing copy only reached the in-memory
logging payload. Live runs at the tip still recorded bare auth exceptions as
Unknown/Exception, the same as the base branch. Removing the plumbing keeps
this PR to endpoint attribution and the info-route exclusion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The provider_config path skipped run_realtime_guardrails for transcription
sessions to avoid sending response.create, which also dropped every
realtime_input_transcription guardrail: no violation error reached the
client and on_violation / end_session_after_n_fails never fired. Run the
guardrail for every completed transcript and only suppress response.create
when the session has no assistant turn.
Muse partials carry no turnId and belong to the most recent speechStart,
and the docs say the model may keep post processing a turn after speechEnd
until speechComplete. Releasing the active turn on speechEnd made any
partial arriving in that window raise and get dropped in ENDPOINTING mode.
The turn now stays active until its speechComplete or final transcript.
* fix(guardrails): log mask when a guardrail adds request keys
_inputs_were_modified only compared keys present in the pre-hook baseline, so a
guardrail that injected a new key such as tools was logged as allow. Compare over
the union of both key sets, and narrow the pre_call return value to the same
prompt-bearing keys the baseline holds so passthrough stays allow.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): snapshot apply_guardrail inputs before the hook mutates them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
get_attached_policies_with_reasons rescanned the sorted matches with next() once
per distinct policy, which is quadratic and misses the one second budget past a
few thousand global attachments. Build a policy to broadest attachment map in one
pass instead, keeping the specificity sort and result order.
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
