* feat(ui): add Delete to the guardrail detail Settings card for DB guardrails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): hoist guardrail delete test fixtures to stay within the inline-object lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): share one guardrail delete modal between the table row and the detail page
The table-row delete now has the same dangling-reference warning and typed-name
confirmation as the detail page. Tests now assert the success and error toasts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): warn that deleting a guardrail used in a policy pipeline step blocks requests with a 400
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover hiding guardrail Delete while settings are being edited
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show the budget spend limit and rate limits without a collapsed section
The create and edit budget dialogs hid max budget and reset period behind a collapsed Optional Settings section that sat 80px below the last field, and dropped both values from the request unless the section was open on submit
Both dialogs now share BudgetFormFields. Budget ID comes first, then the spend limit as one row ("$500 per month"), then the three rate limits as short fields beside their labels. A secondary Optional badge marks the spend limit and the rate limits group, so people can still tell those fields are optional. Every field is always submitted, and the edit dialog sends the existing spend limit and reset period back unchanged
* fix(ui): send only changed fields when editing a budget
Saving the edit budget dialog resent every field, so an unrelated rate
limit edit rounded a stored sub-cent spend limit to 2 decimals (0.001
became 0) and resent the reset period, which makes the proxy recompute
the next reset for custom durations. The dialog now sends the budget ID
plus the fields the user changed, and the proxy keeps everything else.
Also updates the budgets Playwright spec for the new field labels and
the removed Optional Settings section.
* fix(ui): track budget edit changes from render so a reverted field cannot hide a later one
react-hook-form's formState is a render-time snapshot. Reading dirtyFields
only inside the submit handler meant that after a field was changed and
changed back, a later edit to another field was missing from the first
save. Reading dirtyFields during render keeps the snapshot current.
* refactor(ui): rename the budget components to PascalCase
Renames budget_modal, edit_budget_modal and budget_panel (and their tests) to BudgetModal, EditBudgetModal and BudgetPanel, and drops their filename-pascal-case suppressions.
* feat(model hub): show admin-defined custom tags and metadata from model_info
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): move model hub custom field merging into router_utils
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): share the custom tag badge list between the column and details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(model hub): cover public hub, patch, null, hostile, forwarding, and killed-worker paths for custom fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model hub): drop custom_metadata entries with non-string keys or non-finite numbers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): use isinstance narrowing instead of TypeGuard for custom_metadata values
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model hub): build custom tags and metadata in the proxy instead of the router
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model hub): serve custom tags and metadata on model_group_alias rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* remove docstring from model_group_custom_fields.py
* style(model hub): fix lint and format in custom metadata tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(model hub): drop leading blank line in custom fields helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show a failed state with Retry in DataTable when the fetch errors instead of an empty table
* fix(ui): show the shared failed state with Retry on the Usage spend totals and the Caching activity chart when their fetch errors
* refactor(ui): name the settled-fetch objects so the inline object arg lint budget holds
* test(ui): find the caching chart card through the existing cardTitled helper
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Pillow 13.0.0 requires Python >= 3.11, so the ci group pin is split by
Python version and the 3.10 resolution stays on 12.3.0 with a 7-day
osv-scanner override. Dashboard vite moves from 7.3.5 to 7.3.7. The
certifi fix (2026.10.19) is still inside the 3-day uv exclude-newer
window, so its advisory is ignored until 2026-10-23
* feat(extraction): add Clairic extraction types and Cloudflare provider config
* feat(extraction): add the /v1/extraction route, router wiring, and Clairic billing
* fix(extraction): refuse or drop stream, pass unknown keys, run guardrails, keep the requested model name
Clairic does not stream, so stream=true answers 400 and is dropped and billed under drop_params. Keys LiteLLM does not know pass through to Cloudflare. Guardrails scan state, field instructions and text answers. Frozen decision and extraction responses now carry the requested model name. List fields must name their item type. /health/test_connection and the Add Model page accept the extraction mode.
* fix(extraction): keep upstream model names on decisions, scan images, keep typed values, refuse OpenAI params
Scope the frozen-response model stamp to extraction so /v1/decisions and /v1/systemone return the upstream name as on main. Pass request images to guardrails as data URLs and drop answer spans when a response guardrail rewrites state_text. Field spec values keep their JSON type, so a wrong type answers 400 instead of being coerced. OpenAI generation params answer 400 unless drop_params. The Model Hub shows a /v1/extraction snippet for extraction models.
* fix(extraction): send only Clairic's request fields, count extraction tokens, keep typed answers typed, drop spans a request guardrail made stale, and honor --max_tokens
* fix(extraction): scan number answers in response guardrails, rewrite guardrail fields in linear time, and leave --max_tokens as on main
* fix(extraction): scan float answers as whole numbers so fraction digits never trip a response guardrail
* fix(extraction): round only mean and median for response guardrails so fractional values keep their decimal text
* fix(extraction): count state and fields for budget reservations and scan estimate quantiles in response guardrails
* fix(extraction): send each repeated text to guardrails once and keep the context-window budget reservation
* test(extraction): type request_data in the guardrail test helpers
The team Model-Specific Rate Limits picker built its options from the
team's direct models only. A team that gets its models through access
groups has models ["no-default-models"], so the sentinel was the only
option, and a saved {"no-default-models": N} limit never enforces since
the limiter matches the requested model name.
Build one option list for every team per-model field from direct plus
access-group models, expand wildcards, treat an empty list as all
models, and drop sentinels and wildcard patterns, which no limit or
budget lookup ever matches. The team per-model budget field shared the
broken list and is fixed with it.
LIT-9490
* fix(cost): price deployments from their own entry and add model_info.pricing_provider
A deployment whose model_info.id equals another provider's cost-map key
billed $0: register_model merged its prices onto that catalog row, and the
cost calculator then resolved the id by name through get_model_info, whose
provider check rejected the row. completion_cost now hands the deployment's
entry straight to generic_cost_per_token on the openai and generic branches
instead of re-resolving it by name.
model_info.pricing_provider names the provider whose cost-map rows price a
deployment when it differs from custom_llm_provider (an openai-compatible
deployment pointed at a third-party host). It scopes the catalog row the
deployment's entry is merged onto at registration and the provider the
request is priced under at bill time. Unset keeps custom_llm_provider.
* fix(logging): type the pricing_provider metadata read and sync dashboard API types
* fix(logging): read pricing_provider without loop-scoped Final
* refactor(router): inline the deployment pricing provider pick
* fix(cost): keep only model_info.pricing_provider, drop the deployment-entry cost path
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): send batch jobs to a configured control-plane endpoint
Bedrock batch create, retrieve and cancel always built the regional
bedrock.{region} host, so a VPC endpoint or compatible gateway could not
be used. Add aws_bedrock_control_plane_endpoint as a deployment setting
read by the signed HTTP path and the boto3 path, carry it through the
proxy credential filter, and refuse it in request bodies like the other
endpoint overrides. api_base stays out of it because on a Bedrock
deployment it points at the runtime host.
Fixes#45056
* fix(bedrock): read the control-plane endpoint from typed kwargs in job status
The named parameter added one unknown-argument error in litellm/batches/main.py,
which unpacks untyped kwargs into this function
* docs(bedrock): explain batch control-plane endpoint configuration
* fix(bedrock): keep the batch control-plane endpoint out of chat request bodies
A deployment that sets aws_bedrock_control_plane_endpoint also serves chat,
and the unregistered key was swept into the Bedrock request body, so every
chat call on it failed with "Extra inputs are not permitted". Register it
with the other Bedrock batch connection fields so it stays a LiteLLM-owned
kwarg on every route while batches still read it from litellm_params.
* test(bedrock): verify endpoint signatures without internal mocks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds general_settings.maximum_spend_logs_size,
maximum_spend_logs_cleanup_target_size and
maximum_spend_logs_size_check_interval, the spend_log_size_check job,
maximum_spend_logs_cleanup_cron precedence, oldest-first bounded deletes
down to the target, storage fields in the run summary and the
litellm_spend_log_storage_bytes gauge, with a Postgres integration test
covering the post-VACUUM no-refire case
Resolves LIT-9831
Fixes#44589
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(embeddings): add max_batch_inputs pre-call validation
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* add unit tests for max embedding batch size
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* feat(embeddings): enforce batch limit on extra_body input override
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* chore(dashboard): update schema.d.ts for embedding batch size model field
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* fix(embeddings): close Bedrock batch bypass and satisfy ruff PT012
Count max(top-level input, extra_body.input) for batch limits so providers
that ignore extra_body cannot bypass max_embedding_batch_size. Add router
tests and refactor pytest.raises blocks for ruff-tests.
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* fix(embeddings): drop unchecked casts in router batch-size check
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
* fix(embeddings): route oversized batches to a group deployment that admits them
* fix(embeddings): count extra_body input for any top-level input and prefer admitting deployments before cooldown
* fix(embeddings): log cap rejections to failure callbacks and scope batch routing to embedding calls
* fix(embeddings): apply the batch cap after tag and pre-call filters and fall back on cap rejections
* fix(router): keep embedding cap rejections out of deployment health and hold only for routable cooling deployments
* fix(router): clear the embedding cap rejection mark before a fallback's provider call
A fallback or retry shares the request's logging object, so the mark a rejected attempt set made telemetry skip the fallback deployment's real call. Unit tests now pass a fixed logging start time instead of reading the clock.
* fix(router): log embedding batch rejections once, before a fallback runs
* refactor(router): drop docstrings that restate the code
---------
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): map baggage values to budget tags
* fix(proxy): preserve mapped tags in the parsed request cache
* fix(proxy): preserve mapper accounting and tag policy
* fix(proxy): preserve baggage tags across native transports
* fix(proxy): keep mapper rejection and passthrough logging safe
* fix(proxy): scope baggage tag mappings to billed inference and keep them out of tag routing
Mapped tags now apply only to billed LLM API routes, skipping token counting,
model listing, MCP and A2A. They classify spend and budgets but never select
deployments unless a key, team or client set the same tag. WebSocket inference
returns 403 only while a required rule exists. A tag created with /tag/new resets
its spend counter so its budget counts from creation. The General settings editor
keeps a new mapping unsaved until every required field is filled in.
* fix(proxy): keep client-tag rejection identical with baggage, map A2A calls, skip request previews
* fix(proxy): keep mapped tags out of requester metadata, answer 503 on tag lookup errors, map custom pass-through routes, check the key before the WebSocket 403
* fix(proxy): keep caller-sent tags routable, keep mapped tags out of policy and guardrail matching, honor allow_requests_on_db_unavailable, reject control characters in baggage values
* test(proxy): cover baggage tag mappings end to end
Adds integration cells for request_tag_mappings across chat, messages,
responses, A2A and custom pass-through spend, required and budgeted
rules, the WebSocket refusal, unbilled routes, reject_clientside_metadata_tags,
tag routing and policy isolation, a dropped tag read with and without
allow_requests_on_db_unavailable, and the /tag/new spend counter reset
* fix(proxy): keep base tag merges for requests that map nothing, accept JSON-string metadata tags, count litellm_metadata caller tags
Requests with no mapped baggage now reuse the cached body exactly as before, so header and key tag merges still normalize metadata for routing. The client body copy is made only when a tag maps, and the base merges are mirrored onto it. A request that maps nothing no longer gets a 400 for non-object metadata. caller_tags now counts tags sent in both metadata and litellm_metadata. Optional mappings no longer reject JSON-string metadata tags under reject_clientside_metadata_tags.
* fix(router): keep baggage-mapped tags from selecting tag-scoped pre-routing strategies
Auto, complexity, adaptive and quality router selection, the consumed-tags stamp and compression policy lookup now read request tags without baggage-mapped tags, unless the caller sent the tag or the key, team or project carries it, matching deployment tag filtering.
* feat(proxy): parse baggage headers into request tag mappings
* fix(proxy): match plus-sign baggage keys and keep deployment tags in guardrail modes
* fix(proxy): keep baggage-mapped tags behind key, team and deployment tags and serve non-object or invalid DB mapping settings like unset
Mapped tags are added after the key, team, deployment and caller tags, so a tag-mode guardrail picks the same mode with or without baggage. A non-object metadata field maps like empty metadata, and an invalid request_tag_mappings value loaded from the DB is ignored with an ERROR log instead of failing every request
* fix(router): merge deployment tags in linear time and keep malformed client tags as sent under mapped baggage
Merge serving deployment tags into the request tags in one pass, so a large client tag list no longer blocks the event loop. Leave a non-list metadata.tags untouched when baggage maps a tag, so the request gets the same answer as without baggage. Add the C17 integration cell for the team tag's guardrail mode under a mapped deployment tag.
* test(spend): cover string metadata.tags under mapped baggage
* fix(proxy): check mapped tag budgets when metadata.tags is a non-list or holds non-strings
Mapped baggage tags are applied again after the header and key tag merges, so
a header or key tag that replaces a string metadata.tags no longer hides the
mapped tag from the tag budget check. An empty or false metadata.tags is read
like a missing one, and a tags list with non-string items is kept as sent with
the mapped tags added after it, instead of failing auth with a 401
* feat(router): expose supported_endpoints on /model_group/info
* feat(router): route batch creates to mode batch deployments in mixed model groups
* fix(router): prefer batch deployments after eligibility filters and in group-name credential lookups
* fix(router): keep group-name credential lookups on the first deployment and prefer batch deployments before cooldown
Model-encoded ids carry only the group name, so preferring the batch deployment there moved ids created before the upgrade onto its credentials. Batch creates now pick batch-capable deployments before health checks and cooldown, routing and group info share one batch-capable check, the cost map's /v1/batch is dropped only for Azure, and /model/info applies the same per-deployment rule
* test(router): map _listed_endpoints to its /model/info tests in the router coverage check
* fix(router): skip only Azure chat deployments for batch creates and list batch support per caller team
Narrow the batch create filter to Azure and azure_ai deployments that are not mode batch, so OpenAI groups keep
their chat deployments during cooldown and under tag filters. Mark batch deployments by mode batch only, since
stored endpoint lists from older Admin UI saves copy the cost map. Leave /model/info unchanged so the Admin UI
never saves a computed endpoint list, and pass the caller's team into /model_group/info.
* test(proxy): let the model hub router fakes take the caller's team_id
* fix(router): gate batch-create routing on deployments the caller's team may use and list only /v1/batch for Azure batch deployments
* fix(router): skip a malformed model_info.supported_endpoints instead of failing group info and chat
* fix(router): apply the batch create filter after cooldown, managed files, and tags
The batch filter ran before health checks and cooldown, so a batch create on a mixed
Azure group answered with a router 429 or 401 when cooldown, tags, or a managed file's
targets left no batch deployment, where main sends the create on and returns Azure's
400. The filter now runs right before order filtering, after every eligibility filter,
so those cases answer as main does and creates still skip Azure chat deployments
whenever a batch deployment is left
* test(router): annotate the batch routing test assignments with Final
* feat(dashscope): price Qwen implicit context-cache reads separately
* fix(dashscope): normalize implicit cache price strings
* fix(dashscope): price explicit cache hits through an explicit_ override key
Keep cache_read_input_token_cost as the default (implicit) Qwen cache-read
rate, which the cost map already holds, and add
explicit_cache_read_input_token_cost for hits DashScope tags with
cache_type "ephemeral". Untyped and unknown cache types keep the base rate,
so older releases reading the remote cost map keep billing unchanged.
Add explicit cache-read and cache-creation rates to the qwen3.7-max rows.
* fix(types): accept explicit_cache_read_input_token_cost as a constructor keyword
* fix(dashscope): inherit the explicit cache price on custom deployments and keep tier rates all-or-nothing
Deployments that set only input and output prices now inherit
explicit_cache_read_input_token_cost from the cost map, so explicit hits bill
at the explicit rate. A pricing tier without its own explicit rate bills its
own cache_read rate instead of the model-level explicit one, matching how an
off-peak block falls back.
The DashScope calculator now delegates the explicit rate to the shared
resolver, and the register_model warning is back to main's wording.
* fix(dashscope): keep a deployment's own cache-read price for explicit hits and ignore a non-string cache_type
* fix(dashscope): keep cache_type through Responses usage and let a registered cache-read price cover explicit hits
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(key_management): apply default_key_generate_params.allowed_routes on key generation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(key_management): avoid mutating default routes payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): apply default allowed_routes to key_type presets it covers
* fix(proxy): keep management and read_only presets, skip service account keys, and show the default routes to custom_key_policy
* fix(proxy): keep Full Access keys out of the default allowed_routes, let owners re-pick a key type, and show custom_key_policy the member key's routes
* fix(proxy): keep owners from changing routes a key got from default_key_generate_params
* docs(proxy): describe key_type when default_key_generate_params.allowed_routes is set, and test custom_key_policy input with the default off
* test(proxy): type the default_key_generate_params allowed_routes tests
* fix(proxy): let key owners send unchanged allowed_routes back on /key/update
* fix(proxy): ignore a route default with a non-string entry and match resent routes exactly
A default_key_generate_params.allowed_routes list with a non-string entry made /key/generate fail at the database, so it is now ignored like a non-list. An owner's resent allowed_routes on /key/update skip the route check only when they equal the stored list in order, so reordered or duplicated lists are checked as a change. Adds integration cells for the default and the owner update paths.
* test(proxy): cover a non-string allowed_routes default in the key integration tests
* test(proxy): read key rows from the owned proxy's database in the default routes integration tests
* fix(proxy): leave allowed_routes out of a key update that resends them unchanged
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(router): let a deployment set a timeout per service_tier
Adds timeout_by_service_tier and stream_timeout_by_service_tier to a deployment's
litellm_params. A request whose service_tier matches a key uses that value in place
of the deployment's timeout or stream_timeout. A timeout the client sends still wins
and requests without a matching tier are unchanged
Co-authored-by: adc77 <amandogra2016@gmail.com>
* style(router): sort the request_timeout_resolver import
* fix(router): read the service tier maps without the untyped params get
* fix(router): look the service tier timeout up through a typed get
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: adc77 <amandogra2016@gmail.com>
* feat(proxy): opt-in self-serve budget edits for key owners on /key/update
Adds general_settings.self_serve_budget_policy (disabled by default) so a
non-admin can change max_budget or budget_limits on a key they created and
still own. lower_only allows adding or lowering limits. ceiling also allows
raising or removing them when the owner could create that key with the new
budget via /key/generate (same member permissions and delegation ceiling).
soft_budget and spend stay admin-only
* fix(proxy): sync dashboard API types and cover the UI session cap on update
* test(proxy): cover team role restriction for owner budget raises
Drop two unreachable branches from the non-raising key generation check.
* fix(proxy): close self-serve budget bypasses for linked budgets, null caps and retained limits
Keys with a linked budget_id need an admin to change max_budget, owners cannot
leave a key uncapped when an upperbound max_budget is configured, and the
ceiling check now runs on the budget the key would carry after the update.
* fix(proxy): keep open key budget windows and the admin path on self-serve budget edits
A resubmitted budget_limits window now keeps its stored reset_at while that period is still
running, so adding or editing one window no longer restarts the others. Team and org admins
answer False on a ceiling loosen so they fall through to _check_key_admin_access as before.
The self-serve tests feed the team through the cache and prisma mocks instead of patching
get_team_object, and every helper is typed
* perf(proxy): group key budget windows by duration in one pass
The per-duration minimum used to rescan the whole window list once per distinct
duration, so a request with thousands of windows pinned the event loop before any
permission check ran. Sort once and take each group's minimum instead
* refactor(proxy): take budget windows as a Sequence in the delegation ceiling checks
* test(proxy): assert allowed self-serve budget edits return no denial
Also drop the unreachable assert_never arm from the exhaustive budget change match.
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Show discovered Kubernetes pods on the model page for admins
Resolves LIT-9168
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Requests with an explicit session id stick to one discovered pod via rendezvous hashing
Requests without a session id keep round robin
Pod routing records are included in logs, spend logs, and traces
Resolves LIT-9168
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(vertex_ai): let operators drop chosen anthropic-beta values on the Vertex AI pass-through routes
* fix(vertex_ai): drop configured betas sent through x-pass-anthropic-beta and only fail beta requests on a malformed setting
The drop list now runs on the final upstream headers, after x-pass- expansion, through an
upstream_headers_filter on pass_through_request, so x-pass-anthropic-beta cannot carry a
dropped value to Vertex. A malformed setting now answers 500 only for requests that carry
anthropic-beta or x-pass-anthropic-beta, so Gemini and discovery calls keep working.
* test(vertex_ai): accept the new upstream_headers_filter argument in the existing Vertex route tests
* fix(vertex_ai): refuse a malformed drop-betas list at boot and match list entries written with spaces
* fix(vertex_ai): split drop-betas list entries on commas like the header
* fix(proxy): end a chat completions stream with an error event when a guardrail blocks mid-stream
A streaming guardrail that raised an HTTPException after the first chunk re-raised it inside the SSE generator, so the 200 body was cut off with no finish_reason, no error event and no [DONE]. Build the error frame through sse_error_payload instead, keeping the status, message and structured detail, and encode values JSON cannot represent as strings so the frame always goes out.
* fix(proxy): keep early guardrail blocks on their HTTP status and end mid-stream errors with [DONE]
A guardrail block before the first chunk is raised again, so its response stays the
400 event-stream body it is on main. A mid-stream error event now carries the failure
hook's replacement error when one is returned, is followed by data: [DONE] on
OpenAI-format streams, and leaves the Gemini route's raise unchanged. Error details
JSON cannot encode are written as strings instead of breaking the event.
* fix(proxy): keep a per-chunk guardrail block on the first chunk on its event-stream 400
* fix(proxy): treat a keepalive ping as a started stream so a later guardrail block ends in-band
* fix(proxy): encode non-JSON guardrail details in early stream error frames
An early guardrail block whose detail holds a value JSON cannot encode sent
400 headers and then cut the body. The error frame now writes such a value as
its string form, the same as the mid-stream event. The keepalive test now runs
on a virtual clock instead of real sleeps.
* fix(proxy): show an in-band stream error event as an error in the CLI, Python client, and dashboard playgrounds
ChatClient.completions_stream raises StreamError on a data frame carrying an error, so litellm-proxy chat prints the block reason and keeps the partial reply out of history. The prompt playground and the Interactions playground read the same frame through a shared streamErrorMessage helper. Adds the audit's integration cells for chat stream error events, and makes the endpoint tests' failure-hook mocks return None so they check the real error payload.
* fix(proxy): print stream error text verbatim in the CLI and read split error events in the prompt playground
The CLI printed the server's error message as Rich markup, so a closing tag like [/pii] in a guardrail message crashed lite chat. The prompt playground now buffers partial lines across reads, ChatClient raises only on a non-empty error, StreamError redacts keys like the other client exceptions, and StreamError is exported from litellm.proxy.client
* fix(proxy): print CLI stream errors without Rich highlighting or emoji codes
* test(proxy): cover the CLI's plain-text 400 error print
* test(proxy): annotate the new client stream test locals as Final
* fix(proxy): send the failure hook replacement on Responses streams and before the first chunk
* fix(proxy): end a Responses stream with response.failed when it fails on a built-in exception
* fix(proxy): keep a failure hook replacement before the first chunk in-band so bare stream routes still answer it
A non-HTTP failure whose hook returns an HTTP error was raised before the first
chunk, which create_response turned into the hook's status but which a bare
StreamingResponse route such as /queue/chat/completions answered as a 200 with
zero bytes. Only the original HTTP errors raise now. The replacement goes out as
the stream's error event, which create_response still answers with the hook's
status as a JSON error.
Adds audit round 2's integration cells for failure hook replacements on chat and
Responses streams, built-in exceptions on Responses streams, and the Python client.
* test(proxy): cover a queue stream whose failure hook replaces a failure before the first chunk
The audit cell posts to /queue/chat/completions with a callback that raises before the first chunk and a failure hook that answers 403. It checks for a 200 event stream whose only event is the replacement, then one [DONE], with the callback's own message absent
* feat(ui): show gateway URL and agent setup instructions after creating a key
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ui): make the selected agent button visibly distinct in dark mode
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ui): hide agent instructions by default and use a generic title
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ui): use one shared agent instructions card with Claude Code and Codex logos
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response
Count endpoints now say where each number came from in litellm_count_source
(provider, fallback_model, local_estimate) and name the fallback model in
litellm_count_fallback_model. When the provider answers 400 or 404 for a model
it cannot count, the proxy counts the same request with token_count_fallback_model.
token_count_framing adds a per-model constant to local estimates. Both settings
ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and
Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info.
* test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests
* test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing
* fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown
Moves the cost map lookup off the event loop
* ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty
* fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing
A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback
failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries
no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty
token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and
rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as
their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too.
* test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates
* fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment
A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock
calls and about 200 ms on every count, since Opus 5.5 cannot count there either.
The Kimi K3 e2e also runs the large system prompt shape now.
* test(count_tokens): use the shared integration port allocator in the fallback wire test
* fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today
* test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings
Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it
* fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body
A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases.
* fix(count_tokens): answer the provider's error whenever a framed estimate raises
A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error.
* fix(count_tokens): load the custom tokenizer inside the guarded framed estimate
A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors.
* fix(count_tokens): look up framing under the deployment's custom_llm_provider
* fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error
A framed deployment whose custom_tokenizer is not cached locally made every
count wait out the Hugging Face hub's whole retry cycle when the hub could not
be reached, and a burst of those loads held the default thread limiter that
every count's cost map lookup also needs. Counts on a worker now share one
download per minute, a failed download is reused for the rest of that minute,
and after a provider error a count waits at most 5 s before answering the
provider's error. The swallowed-estimate warning logs only the error type.
* fix(count_tokens): keep one tokenizer download per key running and load the local route per request
* test(count_tokens): type the scripted counter and wait on events instead of wall time where possible
* fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error
Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500
* feat(proxy): count tokens with token_count_model in its pinned region
token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable
* fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys
* fix(proxy): keep the upstream error envelope when adding the region hint
* test(proxy): annotate the token count setting echo test
* ci(cost map): require non-blank token_count_model strings and test the region rules
* fix(count_tokens): answer the token_count_model's error even when framing is set
* fix(count_tokens): keep the provider error for Google contents instead of a framed estimate
* fix(count_tokens): estimate Google contents the route translated to messages
* fix(count_tokens): keep the provider error for Google contents with non-text parts
* test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock
* fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags
* feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer
* test(bedrock): keep the Kimi K3 image check in-process
* fix(count_tokens): join the region hint to an upstream message that already ends with a period
Also pins that a blank token_count_model in config falls through to the cost map default
* fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts
The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts
The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool
A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning
* fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them
A flat Responses API function or custom tool becomes the chat tool the
Responses route sends to inference, and every other tool goes to the
Converse body as is, so hosted tools Bedrock drops are left out of the
count instead of being counted or turning it into an estimate.
* test(count_tokens): annotate new token counter test locals with Final
* fix(count_tokens): keep shipped count regions inside the deployment's geography
A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added
* fix(count_tokens): resolve count settings off the event loop
* fix(rag): use team provider credentials and allow viewer and JWT team access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rag): assert persisted managed store data excludes team credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(jwt): run real team model checks in the route denial test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rag): scope team provider credentials to openai vector stores
Bedrock and other providers kept their pre-PR credential and region
resolution: a team bedrock deployment's region no longer overrides the
caller's region on RAG query. Query tests now fake the provider at the
HTTP boundary with a real vector store registry.
* fix(rag): use only the team's own OpenAI deployment for RAG credentials
* fix(rag): ignore caller credential fields the query search never uses when picking team credentials
* fix(rag): treat blank credential fields as absent when picking team credentials
* test(rag): delete the stores the integration ingests create before the owned proxy exits
* fix(proxy): let a restricted key use the team deployment its wildcard or model covers
A key restricted to gpt-4o on a team whose own OpenAI deployment is openai/*
never resolved that deployment's credentials, because the key allowlist was
matched against the deployment's public name literally. The allowlist now
matches a wildcard on either side the way request authorization does, through
the auth layer's own helper, so /rag, /files and /batches resolve the team
deployment for every key that may call a model on it.
* fix(batches): read a restricted key's batch input through its team deployment
The batch access check downloaded the input JSONL with the proxy's OPENAI_API_KEY while the upload and the create used the team deployment, so a key restricted under a team openai/* deployment could not create a batch when the two keys belong to different OpenAI projects. The read now resolves the team deployment the way the create does when no model pins it
Adds the audit cells for keys restricted under the team wildcard on /rag, /files and /batches, and for the rag-vs-chat split of a key restricted to a global exact-name deployment
* fix(rag): load deployments oldest first, reject file_id on stores that cannot take one, and show ingest errors in the dashboard
* fix(rag): pick the team deployment with the lowest id, return 400 for caller mistakes, keep the ingest id in errors, and show partial success in the dashboard
* fix(rag): pick the team's oldest OpenAI deployment so adding one never moves the team
* fix(rag): keep config file team deployments first and try JWT teams in sorted order
A team's config file OpenAI deployments now win in file order, as on the
base, before its DB ones oldest first, so upgrading never moves a team that
has both. A JWT in several teams tries them in sorted order, so every worker
resolves the same team.
* fix(rag): return 400 for Bedrock, S3 Vectors and Gemini store config mistakes on ingest
* fix(rag): keep forwarded provider keys on /rag/query and mixed JWT team claims working
A forwarded x-api-key now counts as a caller credential, so the /rag/query
search uses it over the team deployment as at the merge base. Sorting the
JWT team claim compares entries as text, so a stray number or null no
longer turns into a 401. A Gemini store with no proxy key is a 500 again,
and a non-string vector_store_id keeps the merge base's 400 message.
* fix(rag): let a forwarded provider key win over team credentials on files, batches and ingest
* fix(rag): keep team credentials when a request carries its own key or api_base
A forwarded x-api-key or a body api_key now replaces only the key of the team
deployment, keeping its api_base, as chat completions does. A body
litellm_credential_name no longer turns team credentials off on /v1/batches.
* fix(batches): keep the team key on the team host when a request sends only api_base
A request api_base now replaces the team deployment's api_base only when the
request also brings its own api_key, and a non-string api_key or api_base is
ignored, as on the merge base.
* fix(files): ignore a blank request api_key or api_base next to team credentials
* fix(files): answer 400 for a request api_key that cannot be sent as an HTTP header
* fix(rag): honor a body drop_params for a forwarded key that cannot be an HTTP header
* fix(rag): read the request field for drop_params only on multipart ingest bodies
* fix(rag): type the ingest body read for drop_params
* fix(batches): count a model-routed batch's input with the deployment key it is created with
* fix(batches): reject a batch create whose x-litellm-model headers name different models
* fix(rag): treat a null provider and whitespace store credentials as unset when picking team credentials
* refactor(files): type the team deployment the credential resolver ranks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): count the whole minor line in the upgrade notice for patch releases
A patch release's notes only list its backports, so v1.105.1 showed
"0 new features, 1 fix, and 0 other updates" even though upgrading from
v1.104.x brings everything in v1.105.0 too. When the latest release is a
patch, also fetch the recent release list and sum the stable releases on
the same major.minor line (skipping rc/dev prereleases and drafts).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): use tuples for release lists and sync the dashboard API types
Satisfies the LIT001 type-discipline gate and regenerates the
latest_release_info description in schema.d.ts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): cover the bad release-list payload and non-semver latest tag
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): type the release-line Counters so basedpyright sees no unknowns
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): fetch earlier patches of the minor line by tag
Reading the first page of /releases could miss the line's .0 release when
prereleases push it off the page, and one unrelated release with a null
body failed validation for the whole list. Fetch each earlier patch tag
directly instead (v1.105.0 .. v1.105.N-1), skip tags with no stable
release, and treat a null release body as empty notes.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): take the dashboard's user role and identity from the server
Adds an internal GET /management/v1/users/me that returns the user id, role
and email the proxy resolved for the calling key. The dashboard now reads
identity from it instead of the session cookie's claims, so its navigation and
role gating always match what the backend authorizes. The layout waits on it
behind the existing loading screen, and a failed lookup clears the session and
returns to login. Sessions that must change their password can also read it
* fix(ui): sign out when the current user lookup fails and wait for it on chat
The dashboard layout now holds the loading screen and signs out when
/management/v1/users/me fails, instead of mounting the shell and looping back
to loading as each consumer refetched the failed query. The chat layout also
waits for the identity so its panels never query without a user id. The
lookup goes through the typed client with the regenerated schema.d.ts
* fix(ui): send the session's auth header on the first user lookup and keep chat closed until authorized
The first lookup runs before AuthProvider registers the custom header name, so it now reads auth_header_name from the session cookie. Chat also waits for isAuthorized, so a failed lookup no longer mounts the shell with an empty user id
* fix(ui): type the dashboard callers of the now typed user id
useAuthorized used to hand back the cookie's user id untyped, which hid six strict type errors in its callers. The id is now string | null, so the production build failed. Callers handle the null case, and query key filters accept the boolean blocked filter
* fix(ui): look up the current user only after the proxy URL is discovered
Pages outside the dashboard shell, like chat and connect, could start the lookup before UI config discovery set the proxy base URL. On a split-origin deployment both attempts then hit the UI origin and a valid session was signed out
* fix(ui): keep a loaded identity while UI config discovery reruns
Pausing the lookup by passing a null token switched the query key, so on the dashboard the identity the layout had already loaded vanished until the shell's own UI config request finished. The key now stays the same and only the fetch waits for discovery
* fix(spend_logs): persist user email in spend log metadata so it shows in UI request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): update payload expectations for user email metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(gcs_pubsub): add user_api_key_user_email to the spend log fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): drop untyped metadata key args from the debug log to keep the type gate flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): pass metadata keys to the debug log as a lazy %-style argument
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(spend_logs): store user email in spend logs only behind store_user_email_in_spend_logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(spend_logs): store user email in spend logs by default, drop store_user_email_in_spend_logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): store an empty user email as null, narrow the expected payload test diff
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): retry generic_guardrail_api checks lost after connect with num_retries
* fix(guardrails): close each guardrail retry connection before the next attempt and cap num_retries at 5
A failed retry's fresh client stayed open until every later retry finished, so a check that kept timing out held one idle client per retry. Each retry now closes its own client as soon as it ends, and num_retries is capped at 5 so the retry chain stays shallow
* refactor(guardrails): type the generic_guardrail_api inbound header sanitizer as a mapping
Keeps the type check gate at its merge-base count after the num_retries constructor params added unknown kwargs errors in the Qohash subclass
* chore(guardrails): regenerate OpenAPI snapshot and UI types for num_retries, allow the retry client
The guardrail retry opens its own client on purpose to get a new connection and closes it after the attempt, so the per-request async client check allowlists generic_guardrail_api.py
* fix(ui): preserve key settings drafts during background refreshes
* fix(ui): refresh untouched key settings while preserving edits
* fix(ui): keep team-change MCP clears across key refreshes
clearMcpGrants now marks the MCP fields dirty, so a background refresh with
keepDirtyValues no longer restores the grants a team change just cleared.
Also adapts the draft tests to the onSubmit(values, dirtyFields) signature
on main and asserts a refresh does not mark untouched fields dirty, since the
team admin member-key path rejects saves with non-budget dirty fields.
KeyEditView becomes a one-line arrow to stay under the max-lines limit.
* fix(ui): keep MCP tool defaults out of a key's edited fields
MCPToolPermissions writes non-delete tool defaults for unrestricted servers
on mount. Marking that write dirty made a team admin's budget-only save on a
member key get rejected for editing mcp_tool_permissions. onChange now says
whether a write is a default or a user edit, and the key editor dirties only
edits.
The edit session is now keyed by token or token_id, matching the form's own
fallback, so switching between keys that only carry token_id starts fresh.
* fix(ui): keep a new MCP server's default tool limits across refreshes
Default tool writes for a server the user just added in this edit session
are part of that edit, so they are marked dirty and survive a background
refresh. Defaults written for an already saved server stay clean, so a team
admin's budget-only save on a member key is still accepted.
* feat(ui): shareable URL state for usage views and users-to-usage deeplinks
The Usage page now keeps the view, global view tab, date range and entity filters in the
URL, so links can be bookmarked and shared. Relative presets are stored as range=7d and
keep rolling, custom ranges as UTC from/to. Invalid params are dropped, and a link to a
view or user the viewer cannot access shows a toast and falls back to the default view.
Internal Users rows get a "View usage" action that opens usage filtered to that user, and
the per-user usage views link back to the user's record. A "Copy Share Link" button sits
next to "Ask AI"
* fix(ui): clean usage links for sessions without the organizations query
The organizations query only runs for premium sessions, and a disabled query stays
pending forever, so the URL cleanup never ran for other sessions. Gate on isLoading
instead, and give the access denied toast an id so a re-run effect shows it once
* fix(ui): type-check usage entity filters and address review findings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep team usage available to internal users
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): format users table usage action
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): sync date picker preset with the current usage range
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): restore dashboard AGENTS.md from main and drop unused usage state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): share custom usage date ranges as calendar days
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep per-user agent usage page across rerenders
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Each price field now parses its per 1M text into a per token number or null inside the
schema, so the save handler gets clean values and the hand-rolled conversion goes away.
Negative or non-numeric prices now show a field error instead of being saved
* fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version
* fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider
* fix(utils): refresh provider model sets after register_model
* fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row
* test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version
* fix(utils): keep router deployment ids out of provider model sets
register_model now refreshes the provider sets, so the per-request and
router registrations keyed by a deployment id put those opaque ids into
wildcard model listings. Those callers now pass
add_to_provider_model_sets=False.
* fix(azure): carry responses_api_version through the chat to Responses bridge
completion() dropped responses_api_version from the litellm_params it hands
the Responses bridge, so azure/responses/<model> on a dated chat api_version
still built the 404ing /openai/responses URL. Also cover the auto-router
marker never forwarding the new param.
* fix(azure): stop listing a wrong set of reasoning efforts when none is rejected
The message said only low, medium and high are supported, which is wrong for
models that also take xhigh such as gpt-6.1-sol.
* test(azure): use the latest grok and gpt-5.4 rows in the new azure tests
* fix(proxy): list each wildcard-expanded model once
A provider set can hold a model both bare and provider-prefixed, which now
happens after any priced request registers its openai/<model> key, so the
openai/* listing showed that model twice.
* fix(azure): route Responses past dated GA api_versions by default
A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before
* chore(azure_ai): mark the grok chunk_parser dict as an override contract
* fix(azure): rewrite only the route's /openai segment when moving a call to v1
The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway
api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses.
Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too,
so the rewrite now only touches the last /openai path segment.
The Responses version resolver no longer duplicates the builder's default for an unset api_version.
Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite
now has one test for the dated GA default and one where only responses_api_version can make the call work
* test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none
Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params
* fix(utils): list ai21 chat and vertex code models under their provider wildcard
register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too.
* fix(main): keep caller-chosen model names out of provider wildcard listings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(guardrails): accept Gemini tools and Anthropic content blocks in generic_guardrail_api, name the error type on fail-open
GuardrailToolParam required a type key, so Gemini native tools
(function_declarations, googleSearch) failed validation inside the hook
and either returned a 500 or skipped the scan when fail_on_error was
false. The type key is now optional and left out of the forwarded tool
when absent, so an echoing guardrail server cannot write type: null into
the provider request.
structured_messages was re-validated against the chat completion message
types, which silently emptied tool_result, tool_use, image and document
content blocks to content: [] and dropped undeclared keys before the
guardrail saw them. The request model now forwards rows as opaque
mappings, which also retires the write-back shim that restored the
original rows for echoed-back entries.
The fail-open critical log line now carries error_type so a timeout, a
refused connection and an upstream HTTP status can be told apart without
the traceback.
* fix(httpx): retry aiohttp server disconnects the way httpx does
aiohttp raises ServerDisconnectedError when a kept-alive connection is
closed by the peer before a response arrives, which is what a load
balancer idle timeout shorter than AIOHTTP_KEEPALIVE_TIMEOUT produces.
The transport mapped it to httpx.ReadError, which the handler's
connection-error retry does not cover, so the request failed instead of
being resent on a fresh connection. It now maps to
httpx.RemoteProtocolError, the error httpx itself raises for the same
condition, so the existing retry fires.
* test(httpx): build the stub server reply in one bytes literal
* fix(httpx): resend a dropped connection's request as sent and keep a streamed resend readable
The connection-error retry closed its fresh client before a streamed body was read,
so an aiohttp disconnect now billed a second upstream request and still failed. It
also skipped masking on a status error from the resend, resent PATCH and DELETE as
POST, dropped multipart files and ignored the handler's ssl_verify.
The retry now resends the request the first attempt built, on a client with the
handler's TLS settings, closes that client when a streamed body is closed, and masks
a status error the same way the first attempt does.
* fix(guardrails): log HTTPStatusError, not the masking wrapper, as the fail-open error_type
* fix(guardrails): resend a generic guardrail check cut before any response
A load balancer that closes an idle keepalive connection made the guardrail POST
fail with "Server disconnected", because #46336 only resends POST on ConnectError.
AsyncHTTPHandler.post now takes a keyword-only idempotent flag, defaulting to
False, that gives a RemoteProtocolError the same single fresh-client resend PUT
and DELETE get. Only GenericGuardrailAPI.apply_guardrail opts in, since a
guardrail check is a pure verdict over the request. Every other POST keeps the
#46336 rule
The fail-open log line now puts error_type at the end so the pre-PR text stays
contiguous for existing alert matchers
test_rows_reach_the_guardrail_verbatim now reads the body off a real local
guardrail server instead of mock call args
* fix(http): keep a mid-body server disconnect a ReadError and move socket tests to integration
A ServerDisconnectedError raised while reading a response body now stays
httpx.ReadError, as it was before this PR. Only a disconnect before any
response maps to RemoteProtocolError, which is the case the resend rule is
about. Stream readers that end on ReadError, like the SAP stream iterator,
keep their old behavior
The loopback-server disconnect, TLS and guardrail resend tests move to
tests/integration/sdk/test_disconnect_resend_wire.py, since tests/unit is
in-process only. Unit keeps in-process MockTransport checks for the idempotent
POST resend, masking on the resend, streamed and buffered client closing, the
guardrail resend, and the guardrail receiving rows verbatim
* fix(guardrails): resend a reset guardrail check and keep typed echoes on texts
An idempotent POST now resends once on any network error after connect
(ReadError, WriteError, RemoteProtocolError), so a guardrail behind a load
balancer that answers a stale keepalive with a TCP reset rides it out too.
That makes the aiohttp ServerDisconnectedError remap unnecessary, so it goes
back to ReadError. This stops PUT and DELETE from stacking litellm's resend on
top of aiohttp's own idempotent retry (4 sends instead of 2) and stops Vigil
Guard from replaying its analyze POST on a disconnect
Every resend now runs inside the method's own error mapping, so a timeout on
the resend is a litellm.Timeout like the first attempt and a status error is
masked once by the caller
A guardrail server that parses rows with its own message types and echoes them
back without the keys those types lack (annotations, tool_calls[].index) is
read as an echo again, so its texts redaction reaches the LLM. A row only counts
as an echo when no value changed and every key the chat completion types
declare is kept
The wire tests for the PUT/DELETE resend, its masking and its TLS settings now
run on the httpx transport, where litellm's resend is the only retry, and the
POST and guardrail tests cover both transports and both cut styles
* test(guardrails): black box wire tests for generic guardrail shapes, fail-open error type and idle cut resend
Integration tests against an owned proxy and scripted peers. Gemini native tools reach the guardrail and Gemini as sent, the guardrail sees chat rows and an Anthropic image block as sent, an echoing guardrail leaves a /v1/messages request unchanged, the fail-open log line names ConnectError, Timeout or HTTPStatusError, and a guardrail check cut on an idle pooled connection is resent once on each worker. Each fails at the merge base and passes at the head
* fix(guardrails): check echoed rows without recursion and keep pytest.raises to one statement
* refactor(guardrails): move the dropped-connection guardrail resend out of this PR
The resend on a dropped or hung generic guardrail call, and the shared AsyncHTTPHandler retry rework behind it, move to a follow-up PR that adds a per-guardrail num_retries setting. This PR keeps the request and response shape fixes and the error_type on the fail-open log line
* test(guardrails): pick the closed guardrail port from the integration port allocator
Reading a port back from a port-0 bind trips TQ012, so the fail-open connect error case now takes its unused port from allocate_port
* feat(guardrails): forward structured_messages as sent only behind forward_messages_as_sent
generic_guardrail_api keeps main's OpenAI-format rows and echo comparison by default. Setting forward_messages_as_sent: true on the guardrail sends rows as the client sent them and treats an echo that only drops undeclared keys as unchanged
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add clean-slate playground chat prototype
* feat(ui): replace the playground chat with the redesigned composer and inspector
* fix(ui): close the playground chat gaps found in self-review
Resend only complete text exchanges, so empty replies, MCP direct calls and
transcriptions no longer leak into chat history. A stopped request can no longer
append output, attach its gateway trace to a newer reply or clear the newer
request's loading state. MCP selections are only pruned once both the server and
toolset lists load. The MCP tool picker lists toolsets again and calls the server
that backs the chosen tool, and a saved model the current key cannot use is
cleared when the model list loads, matching the old chat
Also vendors the scroll-fade-x utility the attachment primitive uses
* fix(ui): resolve playground toolset tools per server and drop late session ids
A toolset now lists only its own tools, each resolved on the server the toolset
names, so the args form and the call always target the same tool. A saved
toolset no longer asks for tools with its own id while the toolset list loads,
and a failed tool list says so instead of claiming the server has no tools
A stopped request can no longer overwrite the Responses session id, starting a
new chat drops a request that is still running, embeddings replies are no longer
resent as chat history, and a saved model missing from an already cached model
list is cleared too. A failing toolset list falls back right away instead of
retrying while the MCP pickers show loading
* fix(ui): guard playground sends and label inspector stats
Pressing Enter twice no longer sends the same message twice, an attachment the
new endpoint cannot take is dropped when the endpoint changes, a stopped
Responses request can no longer show its code output on a newer reply, and a
reply that broke mid-stream is no longer resent as history
The inspector's speed, cost and routing stats get the same icons as the reply
footer, and the footer's metrics now line up with its copy and Inspect buttons
* fix(ui): write playground key settings only through the encoded storage path
Key source, virtual key and custom headers now go through their own write loop, separate from the plain settings, so a key value can never take the plain sessionStorage branch
* fix(ui): restore playground BYOK connect and reconnect
Picking a BYOK server for a direct tool call now shows a key button next to the
server picker, so the key can be connected without leaving the MCP endpoint. A
connected BYOK server gets a Replace key action in the MCP dialog, also shown
when its tools fail to load, and saving a key reloads that server's tool list
* revert: restore the single playground settings storage loop
Reverts ca4e742115. The CodeQL finding covers the same encoded sessionStorage handling the old chat used, so it is left as is
* test(ui): type the BYOK credential mock without deep instantiation
* fix(ui): let users back out of replacing a BYOK key
The replace-key form in the MCP dialog now has Cancel, and it resets when another server is picked or the dialog closes, so a working key is never forced to be overwritten
* refactor(ui): name the MCP dialog row focus handler
* fix(ui): accept exact custom models, tags and decimal temperatures in the playground
A typed model or tag that is a prefix of a listed one could not be used,
because Enter picked the partial match and the custom action only showed
when nothing matched. The custom value is now a list option whenever no
exact match exists. Temperature gets an editable input beside the slider
so values finer than the 0.1 step can be sent again.
* fix(ui): stop claiming no guardrails ran when the gateway sent no headers
Direct MCP calls, A2A agents and any request that fails before headers
arrive built their trace from an empty header set, so the inspector said
no guardrails or policies ran. The gateway fields are now left unset in
that case and the inspector says the endpoint does not report them.
* fix(ui): leave policies unknown when the gateway headers are hidden
The proxy exposes x-litellm-applied-guardrails to other origins but not
x-litellm-applied-policies, so a dashboard on another origin read every
request as having run no policies. Policies are now unknown unless the
call id header is readable, and the inspector says so.
* feat(ui): add the 1 hour cache write cost to the add model and model edit forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): cover both cache write tiers in the add model and edit forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): hoist the expected pricing payload out of the inline assertion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): share the per-token conversion between both cache write tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): derive the touched pricing field type from the pricing field list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): collapse the duplicated pricing source keys into one map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): convert add model pricing once for all selected models and describe the 1 hour fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): type the cache write tier deployment read back and fill the 1 hour field
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(e2e): keep the cache write tier spec on the dashboard prettier config
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(ui): look up a team member's budget row once per render
TeamMemberTab resolved each row's membership with Array.find() over
team_memberships inside every extra column's render and sortValue, so a
render walked the full membership list once per cell. Build the user_id ->
membership map once per team_memberships change and read from it instead.
* perf(ui): page the member tables instead of rendering every row
MemberTable rendered one row per member, so a team or an org with a couple of
thousand members paid for every row on each render. Its DataTable now runs
client-side pagination, like the other list tables in the dashboard.
* fix(ui): keep the member table on its page when the roster is refetched
* test(ui): cover a team member's spend and budget refreshing after a refetch
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): gate Add Model's Test Connect on the form's own validation
Test Connect stays disabled until the Add Model form would pass validation,
with a tooltip saying why. Provider password, text and textarea inputs now
render empty after a successful add resets the form, and an empty prepared
request shows a plain message instead of a destructuring crash
* fix(ui): validate the Add Model form when Test Connect is clicked
Test Connect stays enabled and, on click, runs the same validation Add Model
runs. An incomplete form now shows the inline required errors instead of
opening a crashing connection test, and a valid form tests as before. This
replaces the disabled-button gate, which read a validity flag that went stale
after programmatic form updates
* test(ui): cover a blank required Azure API Base on Test Connect
* feat(proxy): add require_key_agent_access_defined for least-privilege agent access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align A2A model visibility assertions with supported routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep A2A model entries on their existing routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): point the listing model at the harness upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): coerce require_key_agent_access_defined like the settings validator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject agent access lookups instead of patching the handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename require_key_agent_access_defined to agent_access_default_deny
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions
* fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel
* feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI
Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing
early-access row, carrying the same limits and capabilities. The AI Studio row
mirrors the early-access entry; the Vertex row differs only in provider and
pricing source. Prices follow once Google publishes them.
Extend the mapped metadata test to cover all three ids, that each routes to its
provider, that its backup copy matches, and that the public rows carry the same
limits as the early-access row.
* feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops
* chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap
* fix(gemini): follow continuation hops in a loop instead of recursion
The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls
* fix(gemini): bill Google Search grounding from every continuation request
Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain
* fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI
A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream
On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued
On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports
continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch
* chore(gemini): name the type errors the dual-mode stream iterator trips
The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors
This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up
* fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens
A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request
A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate
Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk
The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment
* fix(gemini): keep one response id across a continued answer
A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream
A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key
* test(gemini): add integration cells for long decode continuation
Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs
Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once
* fix(vertex_ai): fetch the access token again before each continuation request
A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length
Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged
* fix(vertex_ai): start each continuation stream with an empty parser buffer
When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error
Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async
* fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up
A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token
Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way
* fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer
A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer
The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes
* fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata
A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex
A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not
* fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer
A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0
A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer
Adds the audit's wire tests for both, which fail 24 cells before this change
* test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index
Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed
* fix(vertex_ai): log a continued answer's raw response as one object
A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them
It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted
Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch
* test(integration): check a continued answer logs one merged raw response
A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch
* test(vertex_ai): cover a follow-up tool call and its thought signature in long decode
* test(gemini): pin that a one-request answer logs Google's body unchanged
* fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses
* fix(ui): send detail on playground Responses image parts and code snippets
* test(messages): cover image detail on the Responses bridge wire
GET /v1/mcp/server takes an optional user_id (admin, or the user themselves) and returns the union of the servers each of that user's teams reaches plus allow_all_keys servers. A user in no team gets the unscoped list, a deleted team still listed on the user is skipped, an unknown user is a 404 and a restricted virtual key is refused like for team_id. team_id keeps precedence when both are passed.
The dashboard MCPServerSelector passes the team and user it edits (key edit sends keyData.team_id and user_id, key create sends the picked team and owner, user edit sends the user) and narrows the server options to members of the access groups selected in the same control, keeping allow_all_keys servers and servers already granted so they can still be removed.
Resolves LIT-7784
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
