The redis breaker test raced the event loop: the fake call had to still be
pending when a real time.sleep stall began, which needs the loop to get from
scheduling to the stall in under 1ms. The fake now holds its answer behind an
asyncio.Event so the whole burst times out deterministically.
The pre-commit interrupt test found a real leak: lint_dashboard creates its
eslint report with mktemp and only removed it on the happy path, so an
interrupt landing during the whole-folder eslint run left the file behind.
The subshell now removes it from an EXIT trap, and the test drives the
interrupt while that eslint run is in flight.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
When OpenAI Responses returns a refusal content block, Anthropic /v1/messages
erased the refusal text into an empty content array and emitted stop_reason 'end_turn'.
Translate refusal blocks to Anthropic text blocks, map stop_reason to 'refusal',
and add 'refusal' to AnthropicFinishReason.
Fixes#39721
* feat(ui): page the public model hub table off /public/v1/model_hub
The public Model Hub page loaded every published model group in one call and
did all of its searching, sorting and filtering in the browser, so a proxy with
a few thousand groups sent megabytes to render one screen.
The models table now asks /public/v1/model_hub for one page at a time. Paging,
sorting, search and the provider and mode filters are query parameters on that
route, and the pagination footer counts from the response envelope's
total_count rather than the rows on screen. Column sortability is derived from
the fields the route declares sortable, so a header can no longer ask it for a
sort it answers with a 400.
The feature filter is dropped: supports_* are booleans and the route has no
boolean filter, so it could only ever have filtered the page in view.
* fix(ui): offer the model modes litellm actually prices in the hub filter
The mode filter listed 'moderations', which no model group's mode is ever set
to, so picking it could only ever return nothing; 'anthropic_messages' was
dead the same way. Four real modes the catalogue does use, search, ocr,
guardrail and vector_store, were missing entirely.
The list is now the mode vocabulary in model_prices_and_context_window.json,
and a test reads that file so an option that matches nothing, or a mode with
no option, fails instead of silently filtering to an empty table. A failed
page fetch logs the route's error detail again, as it did before the table
moved to the paginated route.
* test(ui): keep the model hub health rows out of the inline-object budget
frontend-lint's local/no-large-inline-object-arg budget went 555 to 556: the
health check rows became arguments to the row helper. They are plain literals
spreading a shared default again, which is what they were before, and the gate
reports 554 against a max of 555.
* feat: keep every model hub filter when the table pages
Moving the table onto /public/v1/model_hub cost it two controls the route
could not serve: the provider filter fell back to one substring because
providers only declared contains, and the feature filter went away entirely
because supports_* are booleans with no filter at all. Options for the
dropdowns went with them, since a page of rows only knows the values on that
page.
The route now declares providers in, a features field whose value is the
capability names a row has, and providers, rpm and tpm as sortable. Features
is one repeated field rather than a boolean per flag so selecting two of them
matches either, which is what the multi-select has always meant. Three facet
routes serve the distinct providers, modes and features across the published
groups, carrying the parent's filters, per section 12 of the list design.
All of it is additive: the route rejects unknown parameters, so no request
that worked before changes, and the design's stability policy calls new
filters and parameters safe within a version.
Health status stays unsortable. Health is read for the rows on the page, and
ordering the match set by it would mean reading it for every published group,
which is the cost the paging exists to avoid.
* fix(ui): put the model hub facet types where the generator emits them
The generated file lists paths in sorted order and operations in path order.
Both new blocks were spliced in one entry too late, after
/queue/chat/completions rather than before it, so the schema.d.ts sync check
regenerated the file and found them misplaced. Same blocks, byte for byte,
moved to the position the generator gives them.
* fix(proxy): type a facet payload as the sequence the framework hands it
The lint job's basedpyright gate flagged one new reportArgumentType: handle_facet
passes a tuple, and FacetListResponse declared data as list[str]. A list would
have traded that error for an LIT002 mutable construction, and both budgets are
already at their ceiling on the base.
Sequence[str] is what the framework actually produces and what the model always
accepted: pydantic emits the same array schema either way, verified against
model_json_schema, so the OpenAPI spec and schema.d.ts are unchanged, and the
existing list-passing caller in spend_logs still type checks.
* test(proxy): pin the facet route's rejection contract
handle_facet answers six ways before it ever reaches the executor, and
none of them was covered: a denied scope, a filter operator the spec does
not offer, a repeated parameter, a non-positive page or page_size, and the
where clause those last two feed. Every one is a 400 or 403 an
unauthenticated caller can reach, so each gets a test that fails when the
branch stops firing.
Semantic cache keys omit the prompt, so every end user behind one virtual key
shares a bucket and can be served another user's semantically similar response.
Add an opt-in cache_params.semantic_cache_scope (key | end_user) that appends the
authenticated end-user id to the tenant scope, read from metadata and
litellm_metadata so /v1/chat/completions, /v1/responses and /v1/messages are all
covered, falling back to the key scope when no end-user id is present. Expose the
setting in the cache settings API and the Admin UI cache settings form
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Team and service keys often have no user_email after a successful token
join. Treating empty email as a miss hashed every verification token on
routine CloudZero and Focus exports.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Export rows already join user_email from DailyUserSpend.user_id. Recovered
key-owner email must not replace that when only the alias join missed.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: remove wildcard routes from /v1/models response
Wildcard routes like bedrock/* were leaking into the /v1/models response
because _get_wildcard_models only removed them from unique_models in the
fallback branches (no router or no deployment), but not when the router
had a matching deployment. Now wildcards are always removed from the base
list; they are only re-added to the result when return_wildcard_routes=True
is explicitly passed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model_checks): collapse wildcard expansion branches and tighten regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
* fix(azure): restrict the storage credential chain to deployment identities
The keyless Azure Storage path walks the full DefaultAzureCredential chain, so a
proxy with no storage service principal authenticates as whichever identity the
host happens to carry: an operator's az login on a workstation, or the
AZURE_CLIENT_ID/AZURE_CLIENT_SECRET service principal set for Azure OpenAI.
Neither is the identity granted Storage Blob Data Contributor.
Narrow the chain to workload identity and managed identity, the two credentials
a deployment legitimately holds. Azure OpenAI, Postgres IAM auth and the other
callers of get_azure_ad_token_provider keep the full chain.
* test(azure): read the credential chain off the mock instead of an accumulator
* chore: drop a stray launch traceback committed at the repo root
* fix(azure): let the storage chain reach a system assigned managed identity
DefaultAzureCredential keeps one managed identity link and pins it to
AZURE_CLIENT_ID, so a host that sets that variable for Azure OpenAI and runs as
a system assigned identity never got asked for a storage token. Build the chain
from the three credentials a deployment can carry instead of subtracting the
ones it cannot.
* fix(snowflake): normalize Cortex Claude request shapes
Co-authored-by: Kamron Javaherpour <kamron@kargo.com>
Co-authored-by: Oleksandr Kononov <oleks.konov@kargo.com>
* style(snowflake): format Cortex request transformations
* fix(snowflake): annotate Cortex wire payloads
* fix(snowflake): route Cortex content through the shared Anthropic converters
* fix(snowflake): surface Cortex prompt-cache usage and thinking blocks
Parse Cortex's Anthropic-dialect responses and SSE with Anthropic's own parser so cache_creation/cache_read counts, thinking blocks and signatures reach the caller. Restore thinking for every Claude model: Cortex documents extended thinking broadly and only adaptive thinking is 4.6-gated.
* fix(snowflake): echo signed thinking blocks on every assistant turn
The reference converter extends signed thinking blocks on each assistant turn, not just tool-call turns, so a replayed thinking-plus-text response keeps its signed block. Content-less thinking turns send no empty text block.
* fix(snowflake): preserve thinking list content
---------
Co-authored-by: Oleksandr Kononov <oleks.konov@kargo.com>
* fix(logging): aggregate guardrail_status by severity across guardrail entries
A pre_call guardrail that passed (e.g. hide-secrets recording a mask)
appends its entry before a later guardrail's block, and the first-wins
reader reported the blocked request as guardrail_status=success in
StandardLoggingPayload.status_fields. Take the most severe status
across all entries instead: guardrail_intervened >
guardrail_failed_to_respond > success > not_run.
* refactor(logging): express guardrail status severity as an immutable order
Replace the precedence dict and rebinding loop with a severity-ordered
tuple and a max() aggregation, per the repo's no-mutation and
mutable-collection lint gates; parametrize the severity test cases.
No behavior change.
* style(logging): apply ruff format to entries binding
* fix(router): evict stale global pattern_router entries on upsert/delete
upsert_deployment and delete_deployment cleaned team_pattern_routers but left
the outgoing deployment in the global pattern_router, so wildcard requests kept
round-robining onto the stale entry after a PATCH /model/{id}/update.
Fixes#29064
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): dedupe test_get_configured_mode_reads_deployment_model_info name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): restore global pattern_router eviction dropped by previous commit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
