Semantic cache keys omit the prompt, so every end user behind one virtual key
shares a bucket and can be served another user's semantically similar response.
Add an opt-in cache_params.semantic_cache_scope (key | end_user) that appends the
authenticated end-user id to the tenant scope, read from metadata and
litellm_metadata so /v1/chat/completions, /v1/responses and /v1/messages are all
covered, falling back to the key scope when no end-user id is present. Expose the
setting in the cache settings API and the Admin UI cache settings form
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Team and service keys often have no user_email after a successful token
join. Treating empty email as a miss hashed every verification token on
routine CloudZero and Focus exports.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Export rows already join user_email from DailyUserSpend.user_id. Recovered
key-owner email must not replace that when only the alias join missed.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: remove wildcard routes from /v1/models response
Wildcard routes like bedrock/* were leaking into the /v1/models response
because _get_wildcard_models only removed them from unique_models in the
fallback branches (no router or no deployment), but not when the router
had a matching deployment. Now wildcards are always removed from the base
list; they are only re-added to the result when return_wildcard_routes=True
is explicitly passed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model_checks): collapse wildcard expansion branches and tighten regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
* fix(azure): restrict the storage credential chain to deployment identities
The keyless Azure Storage path walks the full DefaultAzureCredential chain, so a
proxy with no storage service principal authenticates as whichever identity the
host happens to carry: an operator's az login on a workstation, or the
AZURE_CLIENT_ID/AZURE_CLIENT_SECRET service principal set for Azure OpenAI.
Neither is the identity granted Storage Blob Data Contributor.
Narrow the chain to workload identity and managed identity, the two credentials
a deployment legitimately holds. Azure OpenAI, Postgres IAM auth and the other
callers of get_azure_ad_token_provider keep the full chain.
* test(azure): read the credential chain off the mock instead of an accumulator
* chore: drop a stray launch traceback committed at the repo root
* fix(azure): let the storage chain reach a system assigned managed identity
DefaultAzureCredential keeps one managed identity link and pins it to
AZURE_CLIENT_ID, so a host that sets that variable for Azure OpenAI and runs as
a system assigned identity never got asked for a storage token. Build the chain
from the three credentials a deployment can carry instead of subtracting the
ones it cannot.
* fix(snowflake): normalize Cortex Claude request shapes
Co-authored-by: Kamron Javaherpour <kamron@kargo.com>
Co-authored-by: Oleksandr Kononov <oleks.konov@kargo.com>
* style(snowflake): format Cortex request transformations
* fix(snowflake): annotate Cortex wire payloads
* fix(snowflake): route Cortex content through the shared Anthropic converters
* fix(snowflake): surface Cortex prompt-cache usage and thinking blocks
Parse Cortex's Anthropic-dialect responses and SSE with Anthropic's own parser so cache_creation/cache_read counts, thinking blocks and signatures reach the caller. Restore thinking for every Claude model: Cortex documents extended thinking broadly and only adaptive thinking is 4.6-gated.
* fix(snowflake): echo signed thinking blocks on every assistant turn
The reference converter extends signed thinking blocks on each assistant turn, not just tool-call turns, so a replayed thinking-plus-text response keeps its signed block. Content-less thinking turns send no empty text block.
* fix(snowflake): preserve thinking list content
---------
Co-authored-by: Oleksandr Kononov <oleks.konov@kargo.com>
Resolving the model from the /bedrock path sends passthrough calls through
optimistic budget reservation, whose tokenizer cannot walk Converse content
blocks and so fell back to the model's max_input_tokens. Count those messages
as text and read inferenceConfig.maxTokens so a budgeted key reserves the
request's cost.
* fix(logging): aggregate guardrail_status by severity across guardrail entries
A pre_call guardrail that passed (e.g. hide-secrets recording a mask)
appends its entry before a later guardrail's block, and the first-wins
reader reported the blocked request as guardrail_status=success in
StandardLoggingPayload.status_fields. Take the most severe status
across all entries instead: guardrail_intervened >
guardrail_failed_to_respond > success > not_run.
* refactor(logging): express guardrail status severity as an immutable order
Replace the precedence dict and rebinding loop with a severity-ordered
tuple and a max() aggregation, per the repo's no-mutation and
mutable-collection lint gates; parametrize the severity test cases.
No behavior change.
* style(logging): apply ruff format to entries binding
* fix(router): evict stale global pattern_router entries on upsert/delete
upsert_deployment and delete_deployment cleaned team_pattern_routers but left
the outgoing deployment in the global pattern_router, so wildcard requests kept
round-robining onto the stale entry after a PATCH /model/{id}/update.
Fixes#29064
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): dedupe test_get_configured_mode_reads_deployment_model_info name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): restore global pattern_router eviction dropped by previous commit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(mcp): cache SSO identity assertion reads on the ID-JAG path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): guard sso assertion cache against stale relogin reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep sso assertion cache entries and generation markers in separate namespaces
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): rename duplicate get_configured_mode test so ruff F811 passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): use a process-wide epoch for sso assertion cache invalidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
DataTable loses the comment that narrated its sticky header classes. The
table scrolling e2e spec now types every management API response it
reads, seeds rows through an immutable reduce instead of pushing into
arrays, and deletes what it seeded in each test's finally block instead
of draining a shared mutable list in afterEach.
Refs LIT-4738
Claude-Session: https://claude.ai/code/session_018yW93iDaEMhoQUXcYjus7D
Virtual Keys, Teams, Request Logs and Tags now hand DataTable a bounded
flex chain and use fillHeight, so the app shell main stays the only page
scroller, the rows scroll under a pinned header and the pagination footer
sits at the bottom of the page. DataTable keeps the sticky header inside
its own scroller in maxBodyHeight mode too, which is what let the header
scroll away with the rows on Keys, Teams and Models. Model Hub, Vector
Stores and the team detail keys tab drop their 75vh boxes and flow with
the page scroller.
Adds an e2e spec that fails on the merge base for every one of those
pages and passes at this tip.
Refs LIT-4738
Claude-Session: https://claude.ai/code/session_018yW93iDaEMhoQUXcYjus7D
All Models now pushes the model group, access group and wildcard filters into
/v2/model/info (new optional access_group and wildcard_only params) so the
server total_count matches the rendered rows. Request Logs defaults to 25,
uses the shared page size options and counts rendered rows in the footer.
Deleted Teams gets the shared DataTable server pagination footer instead of a
hard-coded page size of 100. Per-user usage and the remaining unbounded list
tables get paginationMode so the size selector renders.
Resolves LIT-4738
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
PR #39604 renamed the Internal Users search box placeholder to "Search by email or ID…" but left searchUsers.spec.ts looking for the old "Search by email…" copy, so e2e_ui_testing has been red on litellm_internal_staging since it merged. Point the locator at the shipped placeholder
Derive the GCS batch object path from the deployment's configured model
when present, so a user-crafted JSONL body.model cannot redirect an
authorized deployment's credentials to a different endpoint; the JSONL
value remains the fallback for direct SDK calls with no deployment
config.
Route the fine-tuned endpoint resolution GET through _check_custom_proxy
so custom api_base deployments do not contact Google directly.
Prefer the publisher model path over an endpoints/ segment when parsing
GCS uris, and use the last endpoints/ occurrence, so a bucket prefix
containing endpoints/<digits> cannot shadow the real model path.
Move the custom_endpoint rejection from the batches dispatcher into the
Vertex batch handler so the provider policy lives in the provider
module.
