* fix(proxy): list models from team access groups
* 修复 PR #32599 的类型检查并补齐模型列表回退测试
* fix(proxy): keep a key's own model list narrow when adding team access group models
A key with its own model list was shown every model from its team's access
groups on /v1/models, /v1/model/info and /model_group/info, though auth
denies those calls. Team access group models now widen a key's list only when
the key inherits the team's models. /v1/model/info reuses the shared listing
helper without a catch-all, and route-level tests replace the mock-only ones
* fix(proxy): skip the list build in the unrestricted model info check and test every listing route
The unrestricted check for /v1/model/info only needs to know whether the key
and team lists are empty, so it no longer builds and expands the model list a
second time. The route test now covers /v1/models and /model_group/info next
to both model info routes
* fix(proxy): scope model info for personal and agent keys with no model list
* fix(proxy): keep config keys' team access groups and own-team team_id scope in model listing
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(cost): add tier 1 claude messages stream twins and recapture drifted usage shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): freeze the login throttle clock so the retry-after assertion is deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(docker): manage a quickstart install with status, stop, start, and uninstall
Rerunning either quickstart now opens a menu for the existing install
instead of installing again, and both scripts take an action: status,
start (up), stop (down), restart, logs, open, update, uninstall
Each install gets a litellm-gateway command (litellm-gateway.cmd on
Windows) in its folder, linked into ~/.local/bin when that is on PATH,
so stopping and starting no longer means finding the containers by hand
- stop keeps the database; uninstall asks before deleting it, keeps the
images unless asked, and checks for leftovers
- a busy port names the process holding it
- -y / -Yes answers every question for scripts and CI
* chore(docker): run the quickstart on the latest image instead of main-stable
The docs mark main-stable as deprecated, and on the registry both tags
point at the same image digest today. The quickstart is for local
evaluation, and the file already says to pin a release tag for anything
beyond that
* fix(docker): trust a container's compose project label only when it is a valid project name
When another container holds the port, the quickstart offers to stop it
with docker compose -p <project> stop, taking the project from the
container's label. Whoever started the container sets that label, and
quickstart.sh split it into words, so a crafted label could add
arguments to the stop command
- a label Compose would not accept as a project name now counts as no
project, and the container is stopped by name instead
- quickstart.sh runs the stop with quoted arguments rather than
splitting a command string
* feat(docker): show how many requests the quickstart gateway has handled
The status box, shown by litellm-gateway status and by the rerun menu,
gains a Requests row, so a quick test shows the gateway is working
- the count comes from /global/activity with the admin key from .env,
failed requests included; curl reads the key from stdin, off the
command line
- the row is left out when the gateway cannot say, such as an older
image
- help is now listed in the usage text; help, -h, and --help already
printed it
* fix(docker): move to a free port instead of stopping an unrelated container
When an install's saved port is taken, the quickstart now only offers to stop
the holder if it is another quickstart install. For anything else it offers the
next free port and saves it in .env, so it never stops someone else's app. A
port set through LITELLM_PORT is never moved.
* fix(jwt): honor end_user_id_jwt_field on the JWT-to-virtual-key mapped path
A JWT mapped to a virtual key dropped the end-user claim, so spend rows had
no end user and customer budgets, limits and region never applied. The mapped
token now carries the claim, and it takes precedence over the body user and
the x-litellm-end-user-id header, which remain the fallback when the claim is
missing or empty.
* test(jwt): budget-check the body or header end user when a mapped JWT has no end-user claim
* fix(jwt): replace the annotation offsets with a targeted pyright suppression to keep LIT001 flat
* test(cost): add Azure AI Claude Haiku 5.5 cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): declare service tier cache price fields on CostMapEntry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): drop CostMapEntry reorder now declared on main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): open faked stream items empty so tool arguments and text arrive once
* fix(responses): open faked stream items in progress with a fresh empty list, and open mid-stream guardrail blocks empty
* fix(responses): open faked refusals, reasoning summaries, custom tool calls and MCP calls empty so each value arrives once
* test(integration): cover the empty-open Responses stream items on every fake, replay and block path
Integration cells for the faked /v1/responses stream, the chat and messages bridges, cache replay, guardrail block events and the MCP gateway call. Each opened item starts empty and each value arrives once, checked with raw httpx, the OpenAI SDK and the Anthropic SDK, plus upstream errors, 5 KB values, a client drop, a dropped upstream connection and a worker SIGKILL
* feat(rust-bridge): scope catalog rules with a contextvar
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): select native integration routes with scoped rules
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): run the Rust extension suite without LITELLM_RUST
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): use scoped catalog rules in dispatch tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* refactor(rust-bridge): keep catalog rules a static global and override them once per test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(providers): add Inference APIs as OpenAI-compatible provider
Registers https://api.inferenceapis.com/v1 as a JSON-configured provider (chat completions and Responses API).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(providers): list Inference APIs in Add Model and the provider docs index, add request-level tests
- provider_create_fields.json entry so the Add Model dropdown test passes
- provider_endpoints_support.json (+ backup) entry so the provider docs check passes
- tests now capture the outgoing HTTP request (URL, auth header, body) instead of only inspecting registrations
- drop a redundant comment
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(providers): inject a recording HTTP session instead of patching httpx.Client
Request-level tests now set litellm.client_session to an httpx.Client with a MockTransport, so nothing is
monkeypatched on a class.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4
* chore(providers): drop redundant comment on the inferenceapis entry
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4
* feat(inferenceapis): add cost map rows for the 12 chat models and test the responses route
Prices and context windows come from https://inferenceapis.com/pricing and GET /v1/models.
Moves the provider test to tests/unit and adds the interactions flag the endpoint matrix now carries.
* feat(inferenceapis): price the vendor's chat model aliases and mark /v1/messages supported
* fix(inferenceapis): serve /v1/messages natively and keep audio routes off the provider
* fix(json providers): send provider-specific params in extra_body and report context overflow as ContextWindowExceededError
* fix(exception mapping): match the Inference APIs overflow phrase instead of any context_length_exceeded substring
* fix(exception mapping): map context overflow to ContextWindowExceededError for Inference APIs only
Providers without their own mapper keep reporting an overflow as BadRequestError, since the router raises a ContextWindowExceededError instead of trying generic fallbacks when context_window_fallbacks has no entry for the group. Adds the integration cells for the provider, its fallbacks, and the extra_body and overflow families.
* test(inferenceapis): hold upstream replies across the worker kill in the chaos cell
A reply the killed worker sent before SIGKILL can lose its queued spend row, which the
product never promised to keep. Holding the upstream until after the kill means only the
surviving worker answers, so every 200 must log exactly once.
* test(inferenceapis): make the chaos cell kill the quieter worker and always release held replies
The survivor now must hold at least one in-flight request, so the burst check cannot pass as 0 == 0, and the held vendor is released on exit so a failed assertion cannot stall the wire server teardown
---------
Co-authored-by: TaylorHawkes <857858+TaylorHawkes@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): keep provider headers on MidStreamFallbackError
A 429 raised while opening a stream is wrapped in MidStreamFallbackError,
which dropped the litellm_response_headers exception_type() attaches to
the wrapped error. retry-after and x-request-id reached callers on the
non-streaming path but not on the streaming one, so retry backoff,
provider request id logging and proxy error headers lost them.
* refactor(streaming): drop explanatory comment from MidStreamFallbackError header copy
* test(streaming): cover sync and async header carry on MidStreamFallbackError
Parametrizes the provider-header test over the sync and async stream
wrappers, and drops the hasattr test, which pinned how the attribute is
stored rather than what any caller observes
* test(streaming): add integration cells for stream-open provider headers
Cover the SDK stream openers (gemini, vertex_ai, codestral, predibase and the
eager providers), the Router stream retry and fallback paths, and the proxy on
both the router and the config-less serving paths, including a mixed burst, a
worker SIGKILL and a SIGTERM restart. Add a helper that boots a config-less
proxy owned by the test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): route Model Garden models through the generateContent adapter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vertex_ai): annotate generateContent route test parameters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): type litellm_params in get_vertex_ai_model_route
* fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions
Streamed Model Garden calls through the generateContent adapter now request
stream usage where the deployment supports it, keep cached-token details, and
look up the provider-relative cost key, so they bill like the non-stream call.
The adapter forwards router metadata, passes upstream errors through with their
status, countTokens falls back to local counting for non-Gemini routes, and the
native Gemini config is limited to the Gemini routes.
* fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing
Streamed generateContent through the chat adapter now ends with the upstream usage, honors
always_include_stream_usage false and the caller's stream_options, counts one failure per
failed call, and routes Vertex deployments by their base_model like chat does
* fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage
* test(google_genai): expect the trailing usage chunk on bridged OpenAI streams
The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone.
* fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk
* fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent
* test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire
* test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: align token counting with inference context
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: preserve Vertex Gemini count message shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: forward Google countTokens tools in shapes every provider counter accepts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: add package marker for Vertex count-token unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): ignore deployment system_instruction when counting tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): forward system and tools on Google AI Studio, keep strict on Anthropic tools
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(count_tokens): ruff format gemini count handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(count_tokens): return Mapping from count request body builders
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): truncate long Anthropic tool names on Vertex count, update stale strict test expectations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(count_tokens): simplify Gemini count context and tests
Put the Vertex Gemini count branch back in the same shape as main and the partner model branch, keeping only the system and tools context, the provider error mapping and the gemini tokenizer default
Share one error response helper between the Vertex and AI Studio callers, drop the unread kind fields and the extra validation of map_function output, and say Gemini instead of Vertex AI in the error that AI Studio users also see
Tighten the tests by removing a stray parametrize on a helper, sharing the SDK count args, asserting the exact forwarded system and tools, and folding the Anthropic tool shape tests into one table
* fix(count_tokens): reuse inference tool translation for Vertex Gemini count
Vertex and AI Studio count now convert tools with the same functions inference uses, translate_anthropic_tools_to_openai and the Responses tool converter, before VertexGeminiConfig.map_function. Unnamed and Anthropic hosted tools now count the same declarations inference sends instead of being dropped
Tools that cannot be converted, such as a non list tools value, a non object tool or a schema that is not a JSON object, now return a 400 before any request instead of being counted without them
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: keep model capability flags out of provider requests
A supports_* flag or another model map field set on a deployment's litellm_params was forwarded
to the provider as a request field, so Bedrock answered 400 Extra inputs are not permitted on
chat traffic and on the background health check alike. The shared owned-kwarg predicate now
treats every ProviderSpecificModelInfo field as litellm-owned.
Builds on the health-check-only approach in #40952 by leilei3167.
* fix: treat every supports_ model map flag as a capability field and warn when one sits in litellm_params
* fix(router): skip declared litellm params in the capability flag warning and warn for runtime-added deployments
* fix(router): strip model capability flags from the router's deployments instead of every request
* fix(router): keep capability flags out of health probes and read them from the loaded catalog
* perf(router): read the catalog's supports_ field set once per cost map load
Cache the loaded catalog's field names when the cost map snapshot is taken so the capability-flag check stops walking every catalog entry per deployment, and add integration cells that drive the flagged and control deployments through chat, streaming, messages, responses, embeddings, health and model management against a vendor that rejects the flags
* fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove an OpenAI-format box answer reaches the caller as a box
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): give the box wire test a deployment that advertises supports_box_questions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results
* fix(vertex_ai): use object instead of Any in schema-ref traversal
* fix(vertex_ai): use immutable tuple traversal to satisfy type-discipline gate
* test(vertex_ai): add direct contract test for _contains_json_schema_ref
* fix(vertex_ai): type the $ref walk over JsonValue and drop the private-helper test
* fix(vertex_ai): wrap a tool result only when a $ref holds a non-empty string
Gemini rejects {"$ref": "<name>"} in function_response.response but accepts null, empty,
numeric, boolean, object, and list values, so those tool results stay structured
* perf(vertex_ai): skip the $ref walk when the tool result text cannot spell the key
* test(vertex_ai): cover a $ref on the tool result's root object
* docs(vertex_ai): cite where the Gemini $ref rule was observed
* fix(vertex_ai): keep a $ref under a list of lists structured
Gemini accepts a $ref reached through a list nested directly in a list,
so those tool results keep their structured form. Adds the audit's
integration cells for the tool result wire, chaos and edge shapes
* test(vertex_ai): teach the fake Gemini servers the list of lists rule
The audit's scripted upstreams now skip lists inside a list like Gemini
does, add list of lists cells on every endpoint, and give the 8 MiB
timing calls their own client timeout
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(types): replace Any with proven types in 84 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep vector store search locals untyped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep runtime payload validation out of the sweep
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep only annotation-only Any fixes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): baseline breaker metrics from live state after gc
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): collect garbage before the services logger gauge test reads live breakers
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos
MinimaxVideoConfig maps litellm's OpenAI Videos API onto MiniMax's V2 task
API. Create, status, list, content and delete hit /v2/video_generation and
/v2/query/video_generation. Remix is not exposed: MiniMax's
/v2/video_regeneration only upscales a finished 768P task to 2K and ignores any
prompt, so it raises instead of returning a video that silently drops the edit
OpenAI params translate to MiniMax's contract: seconds becomes duration, size
reduces via gcd to a valid ratio, and text-to-video requests default to ratio
16:9, resolution 768P and duration 5 when unset. input_reference becomes a
first_frame content item, base64 encoded into a data URI for file inputs, and
multimodal reference media passes through a content array in extra_body. Caller
content is validated strictly as image, video or audio items, and the scanned
prompt is always the one text item sent, so text can't reach MiniMax past the
guardrails; anything else in content returns a 400. api_base
shares MINIMAX_API_BASE with the chat config (a trailing /v1 or /v2 is stripped
because the video API carries its own version). The list endpoint pages by
number with no cursor or sort, so after and an ascending order are rejected
with a 400 rather than ignored, and page_num passes through extra_query. Responses
validate into frozen pydantic models at the boundary, video ids are
provider-encoded so status and content route back to MiniMax, and create
reports duration_seconds and video_resolution in usage for cost tracking
End-to-end tests drive the real avideo_generation() entrypoint through an httpx
MockTransport and assert on the encoded request body, which is what catches a
mapper that drops input_reference before the request is built
Both model_prices maps gain minimax/MiniMax-H3 and minimax/MiniMax-H3-Max,
priced per output second by resolution tier from MiniMax's pay-as-you-go list
(H3 $0.08 at 768P and $0.13 at 2K, H3-Max $0.05 at 480P and $0.08 at 768P).
MiniMax also bills input images beyond a per-model free allowance (5 for H3, 2
for H3-Max), so each entry carries input_cost_per_image and
provider_specific_entry.minimax_free_input_images, and the create response
reports the full charge as provider_reported_cost_usd. Reference video input is
rejected with a 400, because MiniMax bills its duration only after the create
call that litellm bills
Co-authored-by: AaronHowell <237895480@qq.com>
* fix(minimax): fail closed on invalid video params and malformed task responses
Addresses review on #43705:
- A create, status or delete response without a task id (or a task) is a
502 instead of a queued object with an empty id
- An unmapped task status is a 502 instead of defaulting to queued
- seconds/duration must be a positive whole number and size a positive
WIDTHxHEIGHT or W:H, otherwise 400; a 0x0 size no longer divides by zero
and an unparseable duration no longer falls back to the billed default
- characters and image are rejected with UnsupportedParamsError unless
drop_params is set, instead of being dropped silently
- Downloading a task that is not ready or failed is a 400 that keeps its
message through the handler, instead of a ValueError surfaced as a
retried 500
- The video download goes through safe_get / async_safe_get
Co-authored-by: AaronHowell <237895480@qq.com>
* test(minimax): cover the video download path and invalid-param branches
The download now goes through safe_get, so the happy path and the
finished-task-without-a-url branch are asserted, alongside the non-object
parameters rejection and the remix response stub.
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
* fix(minimax): return 400 for video remix and clear new lint gate counts
* fix(minimax): send file-tuple images and keep input_reference next to content media
* fix(minimax): answer 404 when an H3 video is downloaded before it finishes
A download that lands while the task is still queued or running answered 400. That is a
client error, which was the intent, but 404 is what the OpenAI videos API documents for
content that is not ready, and it is the status the sibling DashScope and Volcengine
video paths now answer, so a client polling all three reads one shape
A failed task keeps its 400, since that is not a timing mistake the caller can retry
away. The download test drives avideo_content and asserts the caller sees NotFoundError
rather than an internal server error
* fix(minimax): answer 400 for unsupported video params, early downloads and unknown ids
A create param MiniMax cannot honor (seed, n, quality and the like) was dropped
without a word. It now answers 400 unless drop_params is on, like other providers
A download before the video is ready and a status call for an unknown task id
now answer 400. The router cools a deployment down on a 404 and retries a 5xx,
and neither is a deployment fault. DashScope on main answers 400 for the same case
Input images past the free allowance are now billed through the shared video cost
path, so a deployment with custom pricing bills them too. Status usage no longer
carries MiniMax's token counts, list sets has_more to false since MiniMax pages by
number, and the price map stops listing video as an input modality
* fix(minimax): keep the polled video id, honor deployment drop settings, and match chat's key order
Status re-encodes the id with the polled id's model when MiniMax omits it, drop_params and
additional_drop_params never reach MiniMax and listed keys are dropped, MINIMAX_API_KEY wins over
litellm.api_key, create sets created_at, and a content variant other than the video answers 400
* fix(minimax): list a task without a model under the routed group
MiniMax leaves `model` off some listed tasks. The caller polls the ids a list hands back, and the
proxy resolves a poll by the model in the id, so an id encoded with an empty model no longer routes
back to the deployment that served the task. Fall back to the group the router was asked for.
* fix(minimax): honor deployment drop keys inside parameters and bill images at the map rate when a deployment prices only seconds
A field the deployment lists in additional_drop_params no longer reaches MiniMax when it arrives inside parameters, and a deployment that sets only its per-second price now bills reference images past the free allowance at the cost map's per-image rate instead of $0
* fix(minimax): keep deployment-dropped fields out of the body when they arrive through extra_body
* fix(minimax): answer 400 for a video resolution with no price unless drop_params is on
* fix(minimax): send the default resolution when a deployment drops resolution
---------
Co-authored-by: AaronHowell <237895480@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Happy <yesreply@happy.engineering>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(images): preserve explicit provider for async generation
* test(images): prove a bare model name reaches its explicit provider on the async path
* fix(images): validate async provider hints instead of reading untyped kwargs
Read custom_llm_provider and api_base in aimage_generation through a TypeAdapter over a small TypedDict, so the provider lookup gets typed values and the basedpyright gate stays within its merge-base ceiling
* fix: answer bad provider hints with a 400 and keep the explicit provider for async speech and transcription
* fix: keep the explicit provider for realtime and cover the audio bad-hint 400
* fix: keep the explicit provider on the realtime HTTP routes and read base_url in async audio
* fix: stop async image generation from rejecting a base_url it never reads
* fix(audio): skip base_url validation when api_base is set on async speech and transcription
* test(integration): cover a declared provider with a bare model on image, audio, and realtime routes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): return 400 with file line number for malformed batch JSONL row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(vertex_ai): keep yield outside the JSON decode guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): surface BaseLLMException raised while streaming a request body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(http_handler): only follow __cause__ when unwrapping streamed body errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): avoid recursive streamed error unwrapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows
Keep main's error path for a directly raised provider error and only unwrap
one found behind a transport error. Add the parser column to invalid JSON
errors, and cover the real aiohttp chain in the _handle_error test
* fix(vertex_ai): report every row conversion error as a 400 naming its line
A message with no role, a non-string role, a bad temperature, or a litellm
BadRequestError raised while converting a row still returned a 500, and the
async upload lost the 400 for litellm's own exceptions. The stream now turns
any error from one row's conversion into a 400 naming that line.
_handle_error only unwraps an httpx.TransportError, which is what the aiohttp
transport raises around a failing request body, so other callers keep their
own status. A file that starts with a UTF-8 byte order mark now uploads, and
provider messages start with 'Batch input file line N' like the proxy check.
* fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line
A chat row whose body is missing or not an object now raises a 400 naming
its line instead of uploading a blank prompt, and a non-object body on
line 1 no longer crashes the GCS object name lookup. Rows too deeply nested
or too long to parse return a 400 from both the SDK and the proxy's own
batch file check. JSON error columns now count leading whitespace.
* fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites
The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line
* test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos
* fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact
* test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(llms): sanitize trailing slashes in provider get_complete_url
* test(llms): move trailing-slash url tests into each provider's mapped unit file
* fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI
* fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders
* fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests
* fix(gigachat): strip a trailing slash from the pass-through api_base
* test(integration): cover trailing-slash api_base across provider URL builders
Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex): report an unreadable credentials file as a read failure
Tell a path apart from inline JSON by shape, not os.path.exists().
Fixes#40166
* test(vertex): narrow pytest.raises to the expected message
* fix(vertex): log the credentials path instead of returning it to the caller
* refactor(vertex): log the credential source decision where it is made
* fix(vertex): handle paths open() rejects with a bare ValueError
* fix(vertex): redact the credentials path before it reaches the log
* test(vertex): cover the brace fallback and non-text credentials file
* test(vertex): exclude the unreachable exhaustiveness guard from coverage
* refactor(vertex): read the failure's fields off the matched object
CodeQL's py/uninitialized-local-variable does not follow keyword capture
patterns in a match statement and reported `path` and `reason` as possibly
unbound. Narrowing on the class and reading the attributes says the same thing
without a suppression.
* fix(vertex): accept a read-only mapping in extract_aws_params
* fix(vertex): keep encoded credentials out of the log and cover the legacy route
* fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust
* fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file
* fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message
* test(vertex): integration cells for credentials file and inline value failures
* test(vertex): drive the Rust messages cell through Vertex Claude
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema
`translate_anthropic_tools_to_openai` assigned the caller's `input_schema`
straight into `function_chunk["parameters"]`, then merged the tool's remaining
top-level keys into that same dict. Because the assignment aliased rather than
copied, keys such as `display_width_px` on a computer tool were written back
into the caller's schema.
That matters because callers reuse the tool list across passes: the pass-through
guardrail translates the tools, and the request is then translated again for the
real call. The first pass pollutes the source and the second forwards a schema
carrying non-schema keys.
Shallow-copying fixes it. `translate_anthropic_output_format_to_openai` in the
same file already copies its schema for this reason, so this brings the two into
line.
Rebased onto current litellm_internal_staging; the original branch had gone to
conflict on surrounding drift while the line it fixes is unchanged.
* fix(anthropic/adapter): drop the inline comment and test the strict tool shape that Anthropic rejects
* fix(anthropic/adapter): deep copy input_schema so downstream schema edits never reach the caller's tool
A shallow copy still shared nested properties with the caller, and the Gemini, vLLM and watsonx bridges strip nested additionalProperties in place, so a fallback to Claude sent a strict tool Anthropic rejects. A deep copy also keeps a null or malformed input_schema as sent, as before this PR
* fix(anthropic/adapter): deep copy the incoming tools so downstream edits never reach the caller's tools
Copying only input_schema left the extra keys merged into parameters, such as
input_examples, and OpenAI-format function tools shared with the caller, so
Gemini's strict strip still edited the caller's tool before a fallback to Claude
* fix(anthropic/adapter): keep translating tools that deepcopy cannot copy
A tool passed as a frozen mapping, or with a schema nested past Python's recursion limit, made copy.deepcopy raise, so the request failed before it reached the provider. Copy each tool on its own and keep the original object when the copy fails, which is how these tools were handled before
* fix(anthropic/adapter): copy tools through JSON so the Responses bridge and frozen tools keep the caller's schema
* test(anthropic/adapter): parse stub bodies with json and cover the tool copy fallback
* test(anthropic/adapter): close the connection after the malformed tools request
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): scope /spend/tags aggregates to the caller's own and permitted-team logs
Non-admin callers previously received the proxy-wide per-tag spend
aggregation. The endpoint now resolves the same read scope as
/spend/logs/ui and /spend/logs/session/ui, and the query honors the
start_date/end_date parameters it previously accepted but ignored.
* fix(proxy): use Annotated auth dependency on /spend/tags and sync dashboard API types
* fix(proxy): bind /spend/tags dates as UTC datetimes, not session-timezone strings
Raw YYYY-MM-DD strings cast to timestamptz are read in the database
session timezone, shifting the day window by that offset. Parse them
to UTC-aware datetimes first, like every sibling spend query in this
file, and reject malformed dates with a 400 before they reach the
database.
* fix(auth): return a generic 401 for an unknown key
The wrong-key 401 shipped the internal verification table name and the
submitted key's full SHA-256 hash to any unauthenticated caller. Both now
go to the proxy's own log and the response carries only "Invalid API key
provided."
The Admin UI detected a dead session by string-matching that message, so it
now reads the structured error type instead.
* Update ui/litellm-dashboard/src/lib/http/client.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(auth): log the missing key hash where every resolve path passes
Review catch: JWT key-mapping misses resolve through IdentityStore without
touching the main key path, so logging the hash there left those callers with
nothing but the generic 401. Log it at the store's raise site instead, which
every resolve path shares, and drop the now-duplicate log.
* test(auth): inject a key-less client instead of patching the DB fetch
The store test tripped TQ008 by patching _fetch_key_object_from_db_with_reconnect.
Hand IdentityStore a client whose get_data returns no row instead, so the miss is
driven through the real lookup rather than around it.
* fix(auth): log the unknown key's hash once on the auth failure line
The per-lookup warnings in get_key_object and IdentityStore logged whatever value
they were handed, which on the failure-logging path is the raw key a caller sent
(a provider key included), let a newline in ?key= forge log lines, and fired on
lookups that expect a miss. The hash now rides KeyNotFoundError onto the single
existing auth failure log line next to the requester IP, with control characters
escaped. Also drops the new comments and updates skipped tests that quoted the
old message.
* test(auth): add the blank lines ruff format expects before _InFlightCountingPrisma
* test(integration): expect the generic wrong-key 401 in the management lifecycle tests
* fix(auth): keep the operator wording on the unknown-key log line and drop the redundant 401 rewrite
* test(auth): cover the generic unknown-key 401 across routes, proxies, and a worker kill
* test(auth): cover the realtime upgrade refusing an unknown key without echoing it
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: inject clocks and wait on conditions in three wall-clock-flaky unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: keep generic API logger tests on 3.10 and cover automatic sending
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: pin router usage minute in shared rate-limit header test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: stabilize three wall-clock-sensitive unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): cap cold schema validation workers at the CPUs the proxy can use
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: inject the cache clock and wait on lock contention instead of 1s deadlines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): wait on listing events instead of 1s deadlines in the discovery drain test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
* fix(proxy): resolve model aliases to their target deployment for listing metadata
A model alias (a key's `aliases`, a team's `model_aliases`, or the router's
`model_group_alias`) is rewritten at request time and owns no deployment row,
so `get_model_listing_info` returns None for it and the alias name is not a
cost-map key either. Both metadata candidate sources come up empty and the
listing entry carries no `mode`, `max_input_tokens`, or `max_output_tokens`,
while the deployment it routes to reports them on the same response.
Resolve a listed alias to the deployment it routes to for the metadata lookup,
keeping the alias as the response id. Maps are applied in order, each against
the result of the last, mirroring how `litellm_pre_call_utils` applies the team
map and then the key map, so a chained alias reports the deployment the caller
actually reaches rather than an intermediate one. An alias shadowing a
same-named deployment follows the alias, because the request rewrite does too.
Fixes the model_aliases half of #40553.
* fix(proxy): look listed aliases up through the shared alias chain
Drop the separate alias resolver and retarget a listed alias in
alias_listing_entries and alias_target instead, so /v1/models and
/v1/models/{id} report the model /chat/completions rewrites it to, and
only when the caller can list that model
* fix(proxy): keep deployed listed models' own rows and resolve team alias targets at list and retrieval
* fix(proxy): resolve router_settings aliases, keep alias labels, and skip paused targets in model listing
* fix(proxy): leave an alias to an undiscoverable model bare in model listing
A team alias whose target has discoverable: false showed the hidden model's mode and limits on GET /v1/models and retrieved it by alias. Undiscoverable groups now count as hidden for alias resolution, as on main, while admins still see them.
* fix(proxy): leave an alias to a model a listing callback hides bare in model listing
* fix(proxy): keep aliases bare for models a listing callback never judged or another team owns
---------
Co-authored-by: haydster7 <haydster7@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
/public/providers/fields re-read and re-parsed the bundled 85K
provider_create_fields.json on every request, and /public/agents/fields did the
same for that file plus agent_create_fields.json. Both are unauthenticated
public endpoints, and both did the work inside an `async def`, so the read and
the parse ran on the event loop and serialized against every other in-flight
request. Measured against the bundled files, that is roughly 0.40 ms of blocking
work per request for the provider endpoint and 0.45 ms for the agent endpoint
Both now memoize with `lru_cache`, which is what `/public/endpoints` in the same
module already does through its `_cached_endpoints` global, so each file is read
once per process rather than once per request
The agent endpoint mutated the dicts it had just loaded, appending inherited
provider credential fields and popping `inherit_credentials_from_provider`. That
is only safe while the data is re-read every time, so caching it meant making the
merge pure: `_agent_with_inherited_credentials` builds a new mapping per agent,
and the cached value is the merged result rather than the raw file contents.
Responses are byte identical before and after, checked on both endpoints against
a live proxy
The bundled records are typed as Mapping[str, object] and narrowed with
isinstance at the point of use, rather than typing.Any, which ruff-strict bans
Side effects: 3 of the repo's ASYNC230 violations go away, and basedpyright
errors in the file drop from 63 to 52
Rebased onto main after /public/providers/fields started validating the bundled
JSON through a pydantic adapter on every request. That validation now runs once
and the validated records are cached; the agent endpoint keeps merging the raw
provider records, so both responses stay byte identical to main.
Co-authored-by: Alok Sharma <alok44170@gmail.com>
* test(integration): cache the two-turn guardrail replays and derive the thinking pass-through timeout
* test(integration): keep the thinking pass-through timeout as on main
The derived timeout had no failing run behind it, so this PR now only fixes the guardrail cache cells
Add live-captured cache-read fixtures and align replay usage blocks with current provider responses
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The request-side sanitizer on /v1/messages dropped every thinking block whose
text is empty, even with a valid signature. Claude Opus 4.8 / Fable 5 / Fable 5.1
return exactly that shape by default (thinking.display omitted): the signature
is the preserved reasoning. Replaying the assistant turn as returned therefore
lost the model's reasoning on every multi-turn request, silently (the provider
answers 200 without it), and left preserved-thinking enforcement
(thinking.block_binding) nothing to bind.
Anthropic accepts an empty thinking block that carries a signature and binds it
to its prefix; only an empty block with no signature (the LIT-6357 case, a turn
a non-Anthropic model served through the bridge) is rejected with "each
thinking block must contain thinking". Use the signature-aware predicate
is_empty_unsigned_thinking_block, which #38809 already applied on the emit
side, for the request side too. The unsigned case is still dropped.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): status filter + sort and provider-aware search on Models & Endpoints
The models table mixes active and paused deployments with no way to tell
them apart beyond the pause toggle, and the drawer only filters by public
model name and access group.
Backend (/v2/model/info):
- new optional `blocked` query param to filter by routing status
- new `blocked` sort field (asc = active first); the existing `status`
field keeps sorting by config-vs-DB source
- `search` now also matches litellm_params.model, so typing a provider
or upstream model id (e.g. "nvidia_nim", "openai/gpt-4") finds
deployments whose public name doesn't mention them
Frontend (Models & Endpoints):
- visible Status column with Active/Paused badges, sortable
(active first on asc)
- Status filter (All/Active/Paused) in the Filters drawer, wired to the
new server param through URL state
The routing-status filter ignores non-bool sentinels the same way
exclude_auto_routers does, so direct calls bypassing FastAPI keep their
no-filter behavior.
* fix(ui): inverted routing-status filter mapping and compact Status column
Selecting "Active" in the Filters drawer sent blocked=true to the server,
showing the paused deployments instead of the active ones (and vice
versa). The mapping now lives in routingStatusToBlocked() with unit
tests, and the Status column is narrower so the row actions stay in
view.
* fix(ui): pin the Actions column to the right on the models table
With the new Status column the table overflows on narrower screens and
the pause/resume switches (last column) scroll out of view. Pinning the
Actions column keeps them visible; uses the existing DataTable
meta.pinned support (same pattern as MemberTable).
* refactor: readable db search where-condition and ruff-format clean
* fix: apply routing-status filter inside the search flow and regen API types
Greptile flagged (P1): with search + a status filter combined, the
bounded DB fetch ran before status filtering, so matches of the other
status could consume the page budget and leave short pages / stale
totals.
- extract _matches_routing_status() (sentinel-safe) and apply it to the
router-side matches inside _apply_search_filter_to_models
- push the status condition into the DB where clause of
_fetch_db_models_for_search, so rows of the other status never count
against the bounded fetch
- endpoint passes blocked through to the search flow; tests cover the
combined search+status case and the generated where clause
- regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts for the new
query param
- update useModels.test.ts expectations for the new trailing argument
* fix(proxy): keep the flat DB where shape for single-condition searches
The upstream CI test
test_apply_search_filter_honours_exact_model_name_in_db_query pins the
Prisma where shape: exact model_name stays a top-level key, and plain
substring search keeps {model_name: {contains, mode}} at the top level.
Only genuinely multi-condition queries (status filter and/or router-id
exclusion) now wrap in AND.
* fix(proxy): avoid reassigning a Final variable in the db where builder
basedpyright's reportGeneralTypeIssues budget counts the reassignment of
the Final-annotated db_where_condition; express it as a single
conditional expression instead.
* chore(deps): bump pypdf to 6.19.0 to clear osv-scan
pypdf 6.16.2 has 8 High CVEs (GHSA-5jq2-8x83-x246 and aliases) fixed in
6.17.0-6.19.0; bump past all of them
* Revert "chore(deps): bump pypdf to 6.19.0 to clear osv-scan"
This reverts commit 0e2528123db7ab23304ae20f6a4c0961d0e2d073.
* fix(proxy): keep LIT002 budget flat in the model search where-builder
Extract the dynamic Prisma where-condition fragments into module-level
helpers so every dict starts on a short line, split the litellm_params
fallback lookup into a Final local, and annotate suppressions on the
construct start lines the checker attributes
* chore(tests): repoint unbounded-IN baseline at the extracted where-builder helper
The bounded-router-id filter moved from _fetch_db_models_for_search into
_not_in_router_ids, so the old baseline key is stale and the new helper
site is the finding to record
* fix(proxy): avoid searching encrypted deployment model names
* refactor(ui): keep model routing status as a filter only
* docs(api): regenerate model info sorting description
* fix(api): remove remaining paused status sort references
---------
Co-authored-by: thementalcoding <312162298+thementalcoding@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
LiteLLM_AuditLogs masked before_value and updated_values with
SensitiveDataMasker.mask_dict, which stringifies every non-native leaf, so a
null in the change was stored as the string "None" (or "" under a key-named
field). Walk audit values with mask_credentials_in_payload instead, injecting
the existing {"key"} masker so the masked format and scope stay the same.
* fix(proxy): gzip responses over 16 KiB off the event loop
GZipBufferedResponseMiddleware compressed every body under 1 MiB inline.
gzip.compress holds the event loop for its whole run, and its cost follows
how well the body compresses: an 824 KB base64 embeddings response (100
inputs, the OpenAI SDK default encoding) takes about 165 ms, so every other
request on the worker waits that long, and with several in flight the
compressions run back to back.
Lower OFF_LOOP_SIZE_BYTES to 16 KiB so only bodies that compress in well
under a millisecond stay on the loop. zlib releases the GIL, so the loop
keeps serving while a worker thread compresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): pin the gzip off-loop boundary at exactly OFF_LOOP_SIZE_BYTES
A body of exactly OFF_LOOP_SIZE_BYTES compresses in a worker thread, as the
`<` in _compress implies. Suggested in review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): pin an embeddings-sized gzip body off the event loop and drop the source comment
The off-loop test was relative to OFF_LOOP_SIZE_BYTES, so it still passed with the old
1 MiB line. An 824 KB body, the size of the 100-input base64 embeddings response in the
issue, now has to compress in a worker thread. The recorder no longer mutates a list, and
the threshold rationale stays in the commit history instead of a source comment.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(typesafe): support Jev 1.14 images, box questions, and long-context headers
* fix(cost): drop the typesafe/jev-1.14.0 entry main already prices
* fix(typesafe): send images only with the images header and box questions only to models that answer them
TypeSafe takes assets and input_image parts only when the request carries
TypeSafe-Danger-Zone-Images: true, so without it they follow drop_params like
any other provider that cannot take images. Box support now comes from a
supports_box_questions cost-map flag set on jev-1.14.0 and jev-preview, so
jev-1.13.0 and jev-latest refuse or drop a box question before the call. A
malformed provider_specific_header answers 400 instead of 500.
* fix(typesafe): allow null box confidence and validate asset references and base64
* fix(typesafe): gate box questions on the images header, drop unreadable assets under drop_params, forward opt-in headers on the pass-through
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: index streamed thinking_blocks so the OpenAI SDK .stream() accumulator accepts them
* fix: satisfy the type discipline and basedpyright gates for the thinking block index
* chore: sync the OpenAPI snapshot and schema.d.ts with the thinking block index field
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
