* fix(dashboard): don't show a stale provider prompt-cache chip on a response-cache hit
The playground's non-streaming chat completion and responses paths replayed a cache hit's original usage payload verbatim, so ResponseMetrics kept rendering the provider's prompt-cache-write/read chips using token counts from the original request. Detect the hit via the x-litellm-cache-key response header and render a Response Cache indicator instead.
* fix(dashboard): expose x-litellm-cache-key through CORS for the playground cache-hit indicator
* fix(proxy): stop expected 4xx responses from saturating worker CPU on failure logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep threaded sync failure handler so CustomLogger sync callbacks still run
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: remove redundant comments per review
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The endpoint answers a probe that exceeds HEALTH_CHECK_TIMEOUT_SECONDS with
HTTP 200 and an in-body "Timeout exceeded", which the harness's status-code
rerun policy cannot see. The suite's parallel Bedrock load can push a Mantle
probe past that cap transiently, so only that exact error is retried, three
bounded attempts with visible prints; any other error verdict still fails
immediately.
LoggingWorker now carries still-queued coroutines onto the next event loop (12a34a10d8). Under xdist,
a success-logging coroutine queued by test_acompletion_mcp_respects_manual_approval ran nine seconds
later inside test_mcp_tool_call_hook on the same worker, resolved litellm.callbacks at run time and
overwrote that test's captured payload with a gpt-4o-mini completion (assert 1.35e-05 == 1.42).
Run clear_queue() in the suite's autouse teardown so every coroutine a test enqueues finishes before the
next test registers its callbacks, and add a subprocess regression test that runs the real conftest
against a stopped worker with work still queued.
The model Azure Model Router served was recovered by checking whether the text
"model_router" or "model-router" appeared in a model string. Spend logs applied that
check to the litellm model path, where the route prefix guarantees a match, but the
proxy applied it to the client's model group alias, which carries no prefix. A model
group named anything else therefore lost the selected model in both the response and
the spend row.
AzureModelRouterConfig now stamps the served model onto _hidden_params, and the spend
log payload and the proxy's response restamping read that stamp. The name heuristic
survives as a fallback for callers with no response in hand, routed through
get_azure_ai_route so it lives in one place.
Router._add_deployment called get_llm_provider without the deployment's api_base, so a config entry with a bare model plus a known OpenAI-compatible endpoint failed startup validation with LLM Provider NOT provided and the proxy returned 400 no healthy deployments for that model group. acompletion had the same gap at request time: it forwarded only base_url into its get_llm_provider call, dropping the api_base kwarg the router passes. Both now forward api_base so endpoint matching resolves the provider the same way sync completion already does
Together AI moved its canonical API host from api.together.xyz to
api.together.ai. Default the provider api_base and the rerank handler to
the new host, make rerank honor api_base and TOGETHER_AI_API_BASE like
chat already does, map both hosts to together_ai when passed as
api_base, and delete the dead models/info fetch in factory.py.
Adds live e2e coverage for the Bedrock combinations behind recent customer
incidents: llm_provider-* response-header forwarding on /chat/completions
(nonstream and stream), regional us.anthropic.* inference-profile ids over
the invoke route, and the Admin UI Test Connection probe for a
responses-mode Bedrock Mantle deployment. Registers the matching cells in
the coverage registry and publishes the provider x feature matrix table in
its README.
Backfill 21 serverless chat models, the multilingual-e5 embedding model, and
Llama-Guard-4-12B from the live Together catalog with per-token pricing and
capability flags. Mark 25 delisted together_ai entries with their documented
deprecation_date and point superseded models at a live successor via metadata.
Reprice Llama-3.3-70B-Instruct-Turbo to Together's current rate.
Mantle 400s ("Invalid 'input': value did not match any expected variant")
on the Codex history item types agent_message, context_compaction, and
local_shell_call, killing every Codex multi-agent session on the first
sub-agent turn. Rewrite agent_message into an assistant output_text message
(preserving encrypted_content slot payloads, which carry the plaintext task
through Mantle), context_compaction into Mantle's supported compaction
spelling, and local_shell_call into the function_call its recorded
function_call_output already pairs with.
* fix(proxy): reset a stuck team member's budget
A per-team-member budget check reads a cross-pod spend counter that
nothing ever invalidates. Once a member exceeds their per-member
budget, resetting the key's spend, raising the user's or the team's
own budget, or issuing a new key all leave the member stuck, because
none of them touch this counter or its cached membership object.
Add POST /team/{team_id}/member/{user_id}/reset_spend to reset a
member's tracked spend, and invalidate the same cached state from
/team/member_update when it raises a member's own budget, so that
path also takes effect immediately instead of waiting on the
membership cache's TTL. Name the entity in the check's error message
so a stuck member is diagnosable from the 429 body alone.
* fix(proxy): close reset-vs-floor-read race and surface double Redis write failure on member spend reset
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): broadcast spend reset as a SET so the handler's self-delivered message cannot erase the reset guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): omit null fields from the invalidation message so plain evictions keep the old wire format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
