* fix: coerce server_tool_use dict to ServerToolUse in Usage.__init__ (#26153)
* fix: coerce server_tool_use to ServerToolUse in stream_chunk_builder (#26153)
* fix: dict/pydantic-tolerant access in tool_call_cost_tracking (#26153)
* fix: dict/pydantic-tolerant access in anthropic cost_calculation (#26153)
* test: assert ServerToolUse type in existing stream_chunk_builder anthropic web search test
* test: regression test for #26153 (stream_chunk_builder server_tool_use type)
* test: dict/pydantic safety for tool_call_cost_tracking helper
* test: dict/pydantic safety for anthropic web_search cost
* refactor: consolidate _get_web_search_requests into shared cost-calc utils
* test(realtime): use gpt-realtime; openai retired gpt-4o-realtime-preview
OpenAI shut down the gpt-4o-realtime-preview family (incl. the undated
alias) on 2026-05-07, causing the live realtime test to fail with a
4000 invalid_request_error.invalid_model close. gpt-realtime is the GA
successor; switch the live-call tests to it, matching the base branch.
* refactor(types): drop redundant server_tool_use coercion in Usage.__init__
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 4a3860df1f)
Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.
(cherry picked from commit f5e6012ab0)
* fix(proxy): align /v1/model/info with router deployments
Return router model_list entries (including team-scoped models) with team
access metadata instead of wildcard-expanded names from get_complete_model_list.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): gate v1 team filter and honor key allowlists
Only apply get_all_team_and_direct_access_models for admin or user-bound
keys, then intersect with key/team model restrictions to avoid empty lists
for service tokens and metadata leaks for restricted keys.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): skip v1 team filter when user row is missing
Require a DB-backed user before applying team-access filtering on
/v1/model/info, and skip the trailing filter in get_all_team_and_direct_access_models
when user context cannot be resolved.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Revert "fix(proxy): skip v1 team filter when user row is missing"
This reverts commit 74e1fbd77a.
* fix(proxy): restore legacy v1 model access filtering
Keep /v1/model/info on key/team allowlists instead of DB team-membership
filtering, while still listing router deployments for team-scoped models.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): drop A2A agent entries from public /v1/model/info list
* fix(proxy): scope team BYOK rows on /v1/model/info to caller's teams
Listing the full router model_list let any authenticated key without
explicit model restrictions enumerate other teams' BYOK deployments
(public name, team_id, api_base) via /v1/model/info. Reuse the existing
_get_caller_byok_team_scope check so non-admin callers only see global
deployments plus their own team's BYOK rows; admins keep the full view.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 6068bb7781)
* fix(proxy): authorize batch files using upload target_model_names (LIT-3593)
After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)
Restores the reverse-lookup for the JSONL body.model fallback path so that
legacy/pre-target_model_names managed files still map stripped provider IDs
back to proxy aliases before auth. Also cleans up redundant `or None`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)"
This reverts commit 30d2e96f77.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit 2cd7e87485)
* fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session
* fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss
(cherry picked from commit d8fe091938)
* fix(ui): load MCP tool configuration tools via the OBO/passthrough-aware GET path
* fix(mcp): admin-only include_disabled_tools so the settings UI shows toggled-off tools
* fix(ui): repopulate MCP server edit form when server data loads after mount (OAuth return)
* fix(ui): persist MCP OAuth token on save and return to the Settings tab after authorize
* fix(ui): scope MCP OAuth callback to the initiating form so create and edit flows don't cross-talk
* fix(ui): derive OAuth-return Settings tab via lazy state init instead of setState-in-effect
* Fix MCP OAuth edit token handling
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
(cherry picked from commit 51ba6e39cd)
Capture user_id and extra_info from metadata or litellm_metadata. The single-bag read dropped identity whenever a request carried a present litellm_metadata field (null or a user-supplied dict), since /chat/completions routes the authenticated identity into metadata while the guardrail read litellm_metadata first
(cherry picked from commit 1bbaf1c39d)
* feat(responses): add default no-op sign_request to BaseResponsesAPIConfig
* feat(responses): call sign_request after body is final, send signed bytes when signed
* feat(bedrock_mantle): add SigV4 sign_request via composed BaseAWSLLM (bearer path)
* test(bedrock_mantle): cover SigV4 access-key, AssumeRole, body bytes, region/auth consistency
* feat(bedrock_mantle): defer auth to sign_request; validate_environment no longer requires bearer
* docs(bedrock_mantle): document SigV4 + Bearer auth on Responses route
* test(responses): cover fake-stream signing order and mantle bearer arg/env precedence
* fix(bedrock_mantle): wrap all botocore credential errors with both-paths guidance
* fix(bedrock_mantle): catch specific credential errors, not all BotoCoreError, so STS transport failures are not masked
* fix(bedrock_mantle): sign the compact Responses route too, not just create
(cherry picked from commit 2c95d0b024)
* fix(ui): let non-creator users OAuth into OBO-mode MCP servers from the Tools page
* fix(ui): clear OBO Tools-tab one-shot on navigate-back and gate on credential-status errors
(cherry picked from commit 1528f43d4c)
* feat(proxy): publish /v2/model/info in Swagger OpenAPI spec
Expose the v2 model info endpoint in /docs by removing include_in_schema=False
and documenting query parameters used by the admin UI and proxy CLI consumers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(ui): regenerate schema.d.ts for /v2/model/info OpenAPI docs
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit f5b11b72a6)
* Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI
Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context,
128K output, adaptive thinking only) on the Anthropic API, Bedrock
converse (base, global, and us/eu geo inference profiles at the 10%
regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which
serves Fable 5 with the full 1M context window unlike Opus 4.8).
Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the
model in the setup wizard, and extends the reasoning effort e2e grid.
The Bedrock, Vertex, and Azure grid cells carry fail_reason markers
until the CI accounts are provisioned: Bedrock needs the provider data
sharing opt-in Fable 5 requires, and the Foundry resource needs a
claude-fable-5 deployment.
The first-party entry carries provider_specific_entry {us: 1.1} for the
inference_geo premium and deliberately no fast multiplier since Fable 5
has no fast mode.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Drop removed sampling params for Claude 4.7+ when drop_params is set
Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects
top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was
forwarding them even with drop_params enabled because the Anthropic and
Bedrock converse transformations passed temperature/top_p through
unconditionally.
Mirror the GPT-5/o-series handling: temperature=1 still passes through,
other values and any top_p are dropped when drop_params is set, and
without drop_params a clean client-side UnsupportedParamsError tells the
caller how to opt in, instead of surfacing the raw provider error.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Drive sampling param gating from the cost map and cover top_k
Greptile review follow-ups on the sampling param fix: the restriction for
Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false
on every affected cost map entry (perplexity excluded; that route is
OpenAI-compatible and maps sampling params upstream) and read back through
a tri-state map lookup, keeping the name check only as a fallback for
provider-routed ids whose hosted map entries predate the flag, the same
layering supports_adaptive_thinking uses. top_k bypasses map_openai_params
as a provider-specific kwarg, so it is gated at the shared
AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex,
Azure) and in the Bedrock converse _handle_top_k_value path, with
drop_params threaded through the converse transform helpers.
Also updates the reasoning effort grid cell count assertion for the four
Fable 5 rows added on this branch (29 x 11 cells).
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Declare supports_sampling_params in the cost map schema
The model map validation schema uses additionalProperties: false, so the
new flag must be declared for the 28 entries that carry it; this was the
one failing job (misc / Run tests) on the previous commit.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* fix(bedrock): gate top_k=0 on converse to match Anthropic boundary
Truthiness check let top_k=0 silently disappear on models that removed
sampling params, while AnthropicConfig.transform_request treats 0 as
present and raises UnsupportedParamsError (or drops when drop_params is
set). Switch to 'is not None' so converse, direct Anthropic, invoke,
Vertex, and Azure all behave the same for top_k=0.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* test(ci): extend record/replay proxy to chat, embeddings, moderations, rerank, anthropic
The record/replay proxy that took the gpt-image-1 spend E2E off the live OpenAI
path now fronts every provider, so the other real-provider E2Es stop paying for
and depending on live calls each commit. It keys per upstream and selects a
non-OpenAI provider by a /__recorder_upstream/<host>/ path prefix carried on the
model's api_base, since some litellm handlers (cohere rerank) drop custom
request headers. Wired into build_and_test (chat, embeddings, moderations,
image), the otel job (cohere rerank), and the anthropic-messages job via a
reusable start_openai_record_replay_proxy command.
Dropped the time.time()/uuid prompt cache-busters in the build_and_test chat
tests, whose config has the response cache off, so identical requests are
recordable. The image spend test now asserts a repeat call still bills spend,
failing loudly if the proxy response cache is ever turned on.
Responses, the anthropic passthrough, bedrock, and fake-endpoint tests are left
live: their lifecycles, api_base assertions, providers, or fake targets make a
stateless body-keyed cache either break them or add nothing.
* docs(ci): note the recorder command's OpenAI default upstream and prefix override
Addresses a review note: the shared start_openai_record_replay_proxy command
defaults the upstream to OpenAI, so a non-OpenAI model must carry the
/__recorder_upstream/<host>/ prefix on its api_base. Document that in the
command description so a future caller does not assume the default follows the
provider.
google_sql_user.app issues DROP ROLE on destroy, which Postgres refuses
because the role owns every table the migrations job created (75
objects). The previous deletion_policy=ABANDON on google_sql_database
keeps the DB intact through destroy, so the role still owns its
objects. Set the same policy on the user; the instance deletion takes
both the database and the role with it anyway.
* fix(terraform/gcp): prompt for image_registry in DeployStack one-click
The four litellm-* images live on GHCR and Cloud Run rejects ghcr.io URIs
at apply time, so every deploy has to point image_registry at an Artifact
Registry remote repo. The DeployStack installer didn't surface
image_registry as a prompt, so a click-through user landed on the
ghcr.io/berriai default and the apply failed ~20 min in, after Cloud SQL
had already provisioned. Add image_registry to custom_settings with a
PROJECT_ID-placeholder default and a description that flags the ghcr.io
rejection so the failure happens at the prompt, not after billing the
slow path. TUTORIAL.md is reworded to tell the user what to enter at the
new prompt instead of "edit terraform.tfvars before applying".
* fix(terraform/gcp): generalize image_registry default to any region
Per Greptile feedback on #29852, the prior default hardcoded us-central1
and would silently produce a Cloud Run-incompatible image path for any
deployment in another region. The user would substitute PROJECT_ID, miss
the region segment, and reproduce the original late-apply failure. Use
REGION as a second placeholder and tighten the prompt copy so both
substitutions are mandatory.
* fix(terraform/gcp): make destroy work without manual intervention
Three Cloud Run v2 services and the migrations Cloud Run v2 job all
default to deletion_protection=true at the provider level, which has no
data-safety value on stateless resources and blocks terraform destroy
with an error that can only be unstuck with a tfvars edit + apply
roundtrip. Wire deletion_protection=false directly on all four; the
operator-facing tripwire that matters is cloudsql_deletion_protection,
which guards the only resource that actually holds data.
The litellm Cloud SQL database also drops cleanly only if every
connection is closed first. Cloud Run services and the migrations job
hold connections open until they're torn down, so destroy races and
fails with "database is being accessed by other users". Setting
deletion_policy=ABANDON on the database resource lets terraform skip
the explicit drop; the Cloud SQL instance deletion takes the database
with it anyway.
Together these turn destroy into a single command, matching the AWS
stack's behavior.
* refactor: convert AWS and GCP Terraform stacks into reusable modules with examples/default entry point
- Remove `provider` blocks from both AWS and GCP stack roots so the modules
can be consumed with `count`, `for_each`, `depends_on`, assumed-role or
aliased providers — patterns that are forbidden when a module owns its own
provider configuration
- Add `examples/default/` thin-root wrappers for both stacks that wire the
provider (AWS) / providers (google + google-beta) and call the module with
a curated variable surface, preserving the one-command deploy experience
- Move `terraform.tfvars.example` files into `examples/default/` alongside
the new roots; update example comments to reflect the curated variable surface
- Thread `local.tags` (containing `litellm:stack`, `managed-by`, and
`var.tags`) explicitly onto every taggable AWS resource since the module no
longer controls the provider's `default_tags`; GCP resource labels already
flow through the module's `labels` input
- Add `examples/default/variables.tf` and `outputs.tf` for both stacks,
exposing the most-used knobs and re-exporting all module outputs
- Commit provider lock files for both examples so `terraform init` is
reproducible without a network fetch
- Update top-level and per-stack READMEs to document the module-first design,
the `for_each` multi-tenant pattern, and the `examples/default/` quick-start path
* docs(terraform): address review — state-migration guide, tag dedupe, for_each note
- Add 'Migrating an existing deployment' section to AWS & GCP READMEs
documenting the required terraform state mv step (resource addresses now
gain a module.litellm. prefix under the examples/default root)
- Remove redundant managed-by tag from the AWS example providers.tf;
reserve default_tags there for org-wide tags only
- Document the for_each single-provider limitation for GCP (no
configuration_aliases) in the README and example main.tf
Resolves LIT-3504
* docs(terraform/gcp): note expected SSL cert replacement in state-migration guide
The managed SSL cert is named with a hash of lb_domains, so TLS-enabled
stacks that migrated from the old un-hashed name will see one
create_before_destroy cert replacement after terraform state mv — not a
clean 'No changes'. Document that this single replacement is expected and
safe.
* docs(terraform): drop state-migration guides
The AWS/GCP stacks have never been published, so there are no existing
deployments to migrate from the old root-module layout. Remove the
'Migrating an existing deployment' sections from both READMEs.
* docs(terraform): call out image-registry override required for GCP 1-click
The GCP stack's default image_registry points at ghcr.io, which Cloud
Run won't authenticate against, so any real deploy (HCP Terraform
no-code or otherwise) must override it. Document that as a hard
requirement on the GCP README rather than a side note, and add a
top-level HCP Terraform 1-click section enumerating the required
inputs per stack and the migration-task caveat for HCP-hosted runners.
* feat(terraform/aws): mount proxy_config from S3 and wire OpenTelemetry v2
proxy_config
Drop the inline LITELLM_PROXY_CONFIG_B64 env var. Upload the YAML to S3
at config/litellm-config.yaml; gateway and backend container entrypoints
download it to /tmp/litellm-config.yaml via boto3 before exec'ing
uvicorn. The S3 object etag is wired into the task definition so a
config edit produces a new task-def revision and a rolling redeploy. The
existing s3_access policy already grants the task role s3:GetObject on
this bucket, so no IAM changes were needed for the mount itself.
OpenTelemetry v2
New variables otel_endpoint, otel_exporter, otel_service_name, and
otel_headers_secret_arn. Setting otel_endpoint to a non-empty value adds
LITELLM_OTEL_V2=true plus OTEL_EXPORTER / OTEL_ENDPOINT /
OTEL_SERVICE_NAME / OTEL_ENVIRONMENT_NAME to the shared env block; an
optional Secrets Manager ARN backs OTEL_HEADERS for collectors that need
an auth header. Execution role auto-gains GetSecretValue on that ARN.
Empty endpoint = nothing added, so existing deployments are unchanged.
* feat(terraform/gcp): add DeployStack one-click installer
Wires up a Cloud Shell "Open in Cloud Shell" badge backed by the
GoogleCloudPlatform DeployStack flow so examples/default can be
installed from a click in the README without a local terraform setup.
- examples/default/deploystack.json drives project/region collection
plus prompts for tenant, env, image_tag, and allow_plaintext_lb.
Complex inputs (proxy_config, *_extra_secrets, lb_domains) and
sensitive vars (litellm_master_key, litellm_license, ui_password)
stay tfvars / env only so they never land in a committed file.
- examples/default/TUTORIAL.md is a Cloud Shell walkthrough that
enables required APIs, creates the GHCR-passthrough Artifact
Registry repo, optionally exports the TF_VAR_* secrets, runs
`deploystack install`, and shows how to fetch the master key plus
migrate from plaintext LB to TLS.
- Renames var.project to var.project_id across the module and the
examples/default wrapper to match the variable DeployStack injects
from `collect_project: true`. Breaking rename for anyone with a
`project = ...` line in terraform.tfvars; the fix is one line.
* feat(terraform/gcp): mount proxy_config from GCS and wire OpenTelemetry v2
proxy_config
Drop the inline LITELLM_PROXY_CONFIG_B64 env var and the python-decode
startup fragment. Upload the YAML to a dedicated GCS bucket as
config.yaml, then mount it read-only into the gateway and backend at
/etc/litellm via Cloud Run v2's gcsfuse volume. CONFIG_FILE_PATH points
at the mount; an md5 of the YAML rides along as PROXY_CONFIG_HASH so a
config-only edit forces a new Cloud Run revision (gcsfuse only surfaces
new objects on container restart, so without the hash an updated
proxy_config would sit in the bucket unread).
The config bucket is separate from the data-plane bucket so the runtime
SA can hold objectViewer here (read-only at runtime) while keeping
objectAdmin on the data-plane bucket. Both bucket and IAM binding are
gated on proxy_config != {}; an empty config skips bucket creation and
mounts nothing.
OpenTelemetry v2
LITELLM_OTEL_V2=true is now wired into shared_env_kv unconditionally so
both the gateway and backend boot with the integration enabled. It's
dormant until otel_endpoint is non-empty; setting it injects
OTEL_EXPORTER / OTEL_ENDPOINT / OTEL_ENVIRONMENT_NAME plus a
per-component OTEL_SERVICE_NAME (\${tenant}-litellm-\${env}-{gateway,backend})
so spans land tagged with the right hop. otel_headers_secret takes a
Secret Manager resource ID for OTEL_HEADERS (collector auth); the
runtime SA auto-gains roles/secretmanager.secretAccessor on it.
otel_capture_message_content defaults to no_content matching the litellm
default. Any OTEL_* key set in *_extra_env wins over the defaults so
Cloud Run doesn't reject the apply on the duplicate-env-name check.
* refactor(terraform): make AWS and GCP stacks behave identically
Bring both modules to the same surface and the same runtime behavior so
swapping clouds (or reading either README) is symmetric.
Labels and tags. GCP previously stamped var.labels onto only the two GCS
buckets, leaving Cloud Run, Cloud SQL, Memorystore, Secret Manager, and
the LB resources unlabeled; the variable description claimed full
coverage. Now the module computes local.labels (litellm-stack +
managed-by + var.labels, mirroring AWS's local.tags) and threads it onto
every label-supporting resource: Cloud Run services and the migrations
job, Cloud SQL writer and reader (via user_labels), Memorystore, Secret
Manager entries (master_key, license, ui_password, db_password), both
GCS buckets, the global LB address, and the http/https forwarding rules.
GCP keys use 'litellm-stack' instead of AWS's 'litellm:stack' because
GCP label keys forbid colons; var.labels now defaults to {}.
OpenTelemetry v2 is opt-in on both stacks. AWS already gated everything
on otel_endpoint; GCP previously stamped LITELLM_OTEL_V2=true into
shared_env unconditionally and only ungated the OTEL_* block. Both
stacks now do the same thing: leave otel_endpoint empty and nothing
OTel-related lands in the container env; set it and gateway and backend
get LITELLM_OTEL_V2=true plus OTEL_EXPORTER, OTEL_ENDPOINT,
OTEL_ENVIRONMENT_NAME, OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT,
and a per-component OTEL_SERVICE_NAME (${tenant}-litellm-${env}-gateway
or -backend) so spans land tagged with the right hop. AWS picks up the
richer GCP surface: otel_environment_name (defaults to var.env),
otel_capture_message_content (defaults to no_content), and *_extra_env
override filtering so a caller-set OTEL_* key wins over the default for
that service (ECS allows duplicates, but the filter gives the same
predictable last-wins shape Cloud Run enforces). var.otel_service_name
on AWS is gone, replaced by the per-component naming.
uvicorn workers. GCP gains gateway_num_workers, matching AWS; threads
into the gateway args as --workers ${var.gateway_num_workers}.
Docs reflect the parity: each README's OTel section, the GCP 'Using as
a module' Labels paragraph, and a new feature-parity table in the
top-level README that lays out the AWS/GCP input mapping side by side.
* fix(terraform/aws): expose skip_final_snapshot through the default example
The example wrapper already exposed `s3_force_destroy` so ephemeral / CI
stacks could destroy the S3 bucket without manual cleanup, but the matching
Aurora knob (`skip_final_snapshot`) was hidden behind the module surface.
That meant a `terraform destroy` on a trial stack still produced a
`<cluster>-final-<short-sha>` snapshot, with no opt-out short of editing the
module call.
Adds `var.skip_final_snapshot` to the example (default `false`, preserving
the data-loss tripwire) and threads it through to the module input,
mirroring the existing `s3_force_destroy` pattern. Documented alongside it
in the tfvars example.
Verified by deploying the example end-to-end against a clean AWS account
(VPC + Aurora w/ IAM auth + Redis + ALB + 3 ECS services), confirming all
services reach steady state and the data plane serves traffic, then running
`terraform destroy` with `skip_final_snapshot = true` to a clean teardown
(93 destroyed, no Aurora snapshot left behind, no leftover billable
resources).
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: yassin-berriai <yassin.kortam@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(proxy): resolve vector store file list credentials from team deployments
GET /v1/vector_stores/{id}/files now uses the same router credential routing as POST, including JWT team model hints and wildcard model selectors, so list requests no longer call OpenAI with Bearer None.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): authorize model hints and fix credential routing for vector store file list
Resolves three review findings on the vector store file list path.
Authorize user-controlled model hints (?model= query param and the
x-litellm-model header) against the key's and team's allowed models via
can_key_call_model / _can_object_call_model before any deployment
credentials are resolved, closing a model access bypass where a normal
key could file-list using a restricted deployment's provider credentials.
Run the managed vector store registry resolution before the model routing
hint so the managed store sets the routing model first; the hint resolver
then selects credentials matching that model instead of a team fallback
deployment, avoiding a credential/model mismatch across deployments.
Skip team-fallback deployments whose provider cannot be determined instead
of treating them as OpenAI, so a deployment without an explicit
custom_llm_provider or "openai/" prefix no longer has its credentials
injected.
* fix(proxy): enforce vector store file model auth
Ensure vector store file listing routes authorize explicit and inferred model routing before resolving deployment credentials.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): type guard vector store model hints
Keep vector store model hint authorization typed to string-only values so static checks pass.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix managed batch cancel credential resolution
Decode unified batch IDs before cancel routing and resolve litellm_credential_name to api_key in Router._acancel_batch so JWT team-scoped deployments cancel with the same credentials used at create time
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix batch cancellation credential cleanup
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(fal_ai): add Nano Banana / Gemini 2.5 Flash Image generation support
Adds a FalAINanoBananaConfig for fal.ai's Nano Banana models, exposed under
both fal-ai/nano-banana and fal-ai/gemini-25-flash-image (identical schema).
This is the migration path for fal-ai/imagen4, which fal deprecates on
2026-06-30.
The config derives the request endpoint from the model name so both aliases
route correctly, maps OpenAI image params to the fal schema (n -> num_images,
size -> nearest supported aspect_ratio, response_format ignored since the model
returns URLs), and reuses the base fal response parser. Pricing is registered
at 0.039 per image in the cost map and backup.
* fix(fal_ai): tighten nano-banana routing and guard mapped params
Match the specific gemini-25-flash-image / gemini-2.5-flash-image
aliases instead of any model containing gemini so future fal.ai
Gemini-branded models aren't silently misrouted to the nano-banana
config. Guard the param mapping on the fal-side keys (num_images,
aspect_ratio) so a pre-set mapped value is respected and an OpenAI
key is never forwarded unmapped.
* fix(fal_ai): drop non-existent gemini-2.5-flash-image routing alias
fal.ai only serves the dotted-free fal-ai/gemini-25-flash-image and
fal-ai/nano-banana endpoints. Routing the dotted gemini-2.5-flash-image
alias built a https://fal.run/fal-ai/gemini-2.5-flash-image URL that
fal.ai 404s and had no pricing entry, so spend tracking silently fell to
zero. Match only the two real endpoint slugs.
* feat(proxy): hot-reload .env in dev when running with --reload
The --reload watcher already restarts the worker on *.py and --config YAML
edits, but .env was unwatched, so changing a key there did nothing until a
manual restart. Add .env to the uvicorn reload_includes (and to the
StatReload monkeypatch, which ignores reload_includes) so an edit triggers a
worker restart.
A reloaded worker is a fresh process that inherits the reloader's
environment, so load_dotenv(override=False) would keep serving the stale
inherited value for any key already in the environment. The CLI now exports
LITELLM_DEV_ENV_HOT_RELOAD when --reload is set, and litellm/__init__.py
reads it to load .env with override=True only on that dev path, leaving
normal startup precedence untouched.
* feat(proxy): warn that --reload makes .env override shell env vars
When --reload is active, worker processes re-read .env with override=True, so
.env values win over shell-exported environment variables. Surface this dotenv
precedence change with a startup warning so a developer who relies on a
shell-exported override is not silently surprised.
* fix(proxy): type reload helper paths as Optional[str] to satisfy mypy
* fix(proxy): watch the cwd .env in both reload backends for parity
WatchFiles only watches cwd (and the --config dir) for .env, while the
StatReload fallback used find_dotenv(usecwd=True), which walks up to a
parent-dir .env that WatchFiles never sees. Point StatReload at the same
cwd .env so the two reload backends react to the same file.
The recorder could come up pointed at a missing or unreachable cassette redis
and silently forward every request live; the health check still passed and the
process logged nothing, so a CI run looked identical whether it replayed from
the cassette or paid OpenAI for a fresh call every commit. There was no way to
tell from the logs whether the 24h caching was actually happening.
It now announces its mode at startup (REPLAY when the cassette redis is
reachable, PASSTHROUGH when CASSETTE_REDIS_URL is unset, DEGRADED when it is set
but the redis is unreachable) and logs a HIT/MISS line per request. _cache_set
returns whether the write landed so a mid-run redis failure surfaces as a
warning instead of masquerading as a successful record.
Adds unit tests covering the three startup modes and the HIT/MISS/not-recorded
request paths; both new behaviors were mutation-checked.
* refactor(ui): route callbacks/nudges calls through apiClient
* refactor(ui): route alerting + key/user/team delete calls through apiClient
* fix(ui): late-bind fetch in apiClient so global.fetch swaps take effect
createApiClient captured fetch at construction time, so reassigning
global.fetch (as tests do) had no effect and a real network call leaked.
Resolve fetch per request instead; harmless in production where fetch is
never swapped, and required for apiClient-based calls to be testable.
* refactor(ui): route behavior-preserving networking calls through apiClient
Collapse ~61 hand-rolled fetch() calls whose semantics already match the
shared apiClient (auth header, JSON body, json-error + deriveErrorMessage +
handleError) into apiClient.get/post/etc. Query-string builders and the
divergent error-handling functions (no-check, custom messages, text-error)
are intentionally left for a follow-up normalization pass, since converting
them changes wire encoding or error behavior. Prunes the now-stale
no-restricted-syntax suppressions for the removed fetch calls.
* refactor(ui): convert remaining admin/guardrail GETs, guard late-bind fetch
Routes adminspendByProvider, adminGlobalActivity, and the three guardrail
submission calls (list/approve/reject) through apiClient so they match their
already-converted siblings instead of staying on raw fetch. Adds a
client.test.ts case that swaps globalThis.fetch after createApiClient() and
asserts the swap takes effect, which fails on the pre-fix captured-fetch line
and locks in the per-call resolution
Deleting a team-scoped BYOK model left its public name in team.models, so /models
with a team key kept listing the now-deleted "ghost" model. delete_model stripped
team.models using only litellm_modeltable alias lookups, but models added via
/model/new with a team_id never create an alias row; their public name lives only
in team.models and model_info.team_public_model_name, so it was never removed. The
team cache was also left stale because the delete path skipped _refresh_cached_team.
The cleanup now keys off team_public_model_name (falling back to alias keys), runs
after the deployment row is deleted, and strips a public name only when no remaining
team deployment still backs it, so a load-balanced replica is not revoked and
concurrent deletes cannot leave a ghost. The updated team row is refreshed in cache
so /models reflects the change immediately
* feat(ui): generate dashboard API types from the proxy OpenAPI spec
Introduces the shared type foundation for the dashboard without touching any
runtime code. The proxy's FastAPI app is the source of truth; app.openapi()
emits the spec and openapi-typescript turns it into src/lib/http/schema.d.ts.
Adds an npm run gen:api script (a Python spec dump piped into openapi-typescript)
and a Check UI API Types Sync CI job that regenerates the file from the live
spec and fails if it drifts, so the committed types can never silently fall out
of step with the backend. The generated file is pinned to openapi-typescript
7.13.0 and excluded from prettier, eslint, and knip, and marked linguist-generated
so it collapses in diffs.
No openapi-fetch and no call-site changes yet; this only makes the types exist.
* chore(ui): tidy gen-api-types script per review
Write the spec dump inside a with-block and clean up the temp dir in a
finally, so repeated local runs don't leave stray ~MB JSON files behind.
* fix(jwt): attribute spend to resolved DB user_id on email/sso fuzzy match
When user_id_upsert is enabled with JWT auth and a pre-migration user row
exists whose user_email matches the JWT email but whose user_id is a UUID,
get_user_object resolves the legacy row via fuzzy lookup, but the JWT-claim
user_id (the email) still flowed into team-membership lookup,
JWTAuthBuilderResult.user_id, UserAPIKeyAuth and the spend tables. Spend was
orphaned under a phantom email id; /user/info and the Usage page showed $0
for the legacy user (GH #26789).
Treat the resolved user_object as the source of truth: add
_canonical_user_id_from_db, rebind inside get_objects, and return
effective_user_id so auth_builder unpacks it without adding statements.
Fixes#26789
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(jwt): log user_id rebind at DEBUG to avoid email PII in INFO streams
Greptile review on #29217: rebinding often logs JWT email claims at INFO.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(jwt): update passthrough allowlist mock for 5-tuple get_objects
Staging #29256 added a test that still mocked get_objects with a
4-tuple; our PR expanded the return to 5 values (effective_user_id).
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(google): add google-genai SDK proxy integration tests for Gemini and Vertex
Pin google-genai in the CI dependency group and exercise streaming/non-streaming
generate_content through the LiteLLM proxy in the existing unified_google_tests suite.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): address Greptile review for google-genai proxy SDK tests
Restore GOOGLE_APPLICATION_CREDENTIALS after the module proxy fixture tears down,
initialize temp-file tracking on the proxy SDK base class, and skip litellm reload
for proxy_genai_sdk tests so the module-scoped proxy server stays consistent.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): only load Vertex credentials when keys exist for proxy SDK tests
Avoid writing empty GOOGLE_APPLICATION_CREDENTIALS temp files so Vertex tests
skip cleanly without credentials, use a session-scoped proxy fixture, and clean up
per-test credential temp files.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(test): scope google-genai pin to unified_google_tests only
Remove google-genai from the ci dependency group and pin it in
tests/unified_google_tests/requirements.txt for local test installs.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(google): tie litellm reload skip to proxy fixture dependency
Replace the name-based reload guard with a check on whether the test
requests the google_genai_proxy_url fixture, so the skip stays correct
if the proxy SDK tests are renamed.
* fix(test): stop DatabaseURLSettings tests leaking DATABASE_URL into os.environ
The autouse env scrubber relied on monkeypatch.delenv, but apply_to_env
writes DATABASE_URL straight into os.environ, which monkeypatch never
tracks and therefore never undoes. The synthesized writer.example.com URL
leaked past the last test in this module and into proxy-infra tests that
read DATABASE_URL to decide whether to hit a real database, e.g.
test_deprecated_key_grace_period_cache_hit_path, turning an intended skip
into a ConnectError. Snapshot and restore the managed vars directly so the
original environment is reinstated regardless of how it was mutated.
* test(google): drop redundant per-test vertex credential setup
The session-scoped google_genai_proxy_url fixture already configures
GOOGLE_APPLICATION_CREDENTIALS before the proxy starts, and
_require_proxy_sdk skips when credentials are missing, so the per-test
_setup_vertex_credentials_if_needed helper and its temp-file tracking
never did any work. Remove it to keep the ABC self-contained.
* test(google): declare model_config contract on proxy SDK ABC
_skip_reason_if_credentials_missing reads self.model_config to pick the
provider, but that property was only declared on the sibling
BaseGoogleGenAITest. Make the dependency explicit by adding model_config
as an abstract property on BaseGoogleGenAIProxySDKTest so the ABC is
self-contained and a standalone subclass fails fast instead of hitting an
AttributeError.
* test(google): narrow streaming error catch to Exception
Catching BaseException in the streaming assertion swallowed
KeyboardInterrupt and SystemExit, turning a Ctrl-C into a test failure
message instead of letting pytest interrupt cleanly. Only genuine runtime
errors should be recorded as stream failures, so catch Exception.
* test(google): initialize proxy on the same loop that serves it
The proxy was initialized via asyncio.run() on the main thread, which
creates and tears down a throwaway event loop, while requests were served
on a separate loop in the worker thread. Any asyncio primitive bound to
the init loop would be unusable once serving started. Run initialize()
on the worker thread's loop right before server.serve() so setup and
request handling share a single event loop.
* test(google): drop redundant google-genai requirements pin
google-genai>=1.37.0,<2.0 is already declared in the proxy-runtime extra,
which the google_generate_content_endpoint_testing CI job installs via
uv sync --all-extras. The standalone tests/unified_google_tests/requirements.txt
duplicated that pin with a narrower ==1.37.0 specifier and was never
installed by CI, so it added a second source of truth without changing
what gets installed. Drop it and rely on the proxy-runtime extra.
* chore: revert incidental uv.lock exclude-newer bump
The google-genai ci pin was added and then dropped (it is already
provided by the proxy-runtime group), but each uv lock recomputed the
relative exclude-newer span, leaving only a timestamp bump in uv.lock.
Restore it to the base value so this test-only PR carries no lockfile
change.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
* Mark xAI models retiring on 2026-05-15 (#28788)
Per https://docs.x.ai/developers/migration/may-15-retirement, xAI is
retiring the following slugs on 2026-05-15 (auto-redirect to grok-4.3
with various reasoning efforts; callers continuing to use the old slugs
will be billed at grok-4.3 pricing):
grok-4-1-fast-reasoning{,-latest} -> grok-4.3 (low effort)
grok-4-1-fast-non-reasoning{,-latest} -> grok-4.3 (none)
grok-4-fast-reasoning -> grok-4.3 (low effort)
grok-4-fast-non-reasoning -> grok-4.3 (none)
grok-4-0709 -> grok-4.3 (low effort)
grok-code-fast-1{,-0825} -> grok-build-0.1
grok-3 -> grok-4.3 (none)
Only the direct xai/ slugs are tagged; third-party hosts (azure_ai,
oci, vercel_ai_gateway, perplexity/xai) run their own schedules. The
grok-3 retirement list explicitly names only the base grok-3 slug — the
-mini / -fast / -beta / -latest variants are not listed, so they remain
untouched.
* feat(moonshot): advertise json_schema response support on live models (#29683)
litellm.responses() already routes Moonshot through the responses->chat-completions
bridge, and Moonshot honors response_format json_schema on chat completions. The
cost-map entries left supports_response_schema unset, so discovery layers that gate
on that flag dropped Moonshot from structured-output / responses listings even though
the capability works end to end.
Set supports_response_schema on the nine models currently live on api.moonshot.ai:
kimi-k2.5, kimi-k2.6, the moonshot-v1 8k/32k/128k text and vision-preview variants,
and moonshot-v1-auto. Verified against the live API that each honors json_schema and
that litellm.responses() returns schema-valid structured output through the bridge.
* chore(moonshot): mark models retired from api.moonshot.ai as deprecated (#29685)
Thirteen Moonshot/Kimi models in the cost map no longer resolve on
api.moonshot.ai (all return 404). Stamp each with its deprecation_date from
platform.kimi.ai/docs/models rather than deleting the entries, so historical
cost calculation keeps resolving the names while tooling can surface the
retirement.
Dates: kimi-thinking-preview 2025-11-11; kimi-latest and its 8k/32k/128k context
variants 2026-01-28; the kimi-k2 preview/turbo/thinking series 2026-05-25; the
moonshot-v1 -0430 snapshots use their own 2024-04-30 snapshot date (Moonshot
publishes no discontinuation date for them).
* fix(moonshot): drop temperature for reasoning models (kimi-k2.5/k2.6) (#29687)
Kimi reasoning models reject every temperature except 1; a request with
temperature=0.2 returns "invalid temperature: only 1 is allowed for this model".
litellm only clamped temperature into [0.3, 1], so any value below 1 still 400'd.
Drop the temperature param entirely for reasoning models (gated on
supports_reasoning, the same signal transform_request already uses) so the model
default is used; the non-reasoning moonshot-v1 models keep the existing clamp.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* feat(mcp): add per-server timeout configuration (#29672)
* feat(mcp): add per-server timeout configuration
* fix(mcp): address timeout field review comments
- use is not None guard instead of or for 0.0 edge case
- copy timeout in both LiteLLM_MCPServerTable constructions (health check path + _build_mcp_server_table)
- add timeout Float? column to all three schema.prisma files
- extend round-trip test to cover _build_mcp_server_table direction
- add test for zero timeout not treated as falsy
* fix(mcp): forward timeout in _build_temporary_mcp_server_record
* fix(mcp): return 504 instead of 500 when per-server timeout fires
* test(mcp): add 504 timeout regression test; fix black formatting
* Add jp. Bedrock cross-region inference profile for claude-opus-4-7 (#28567)
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598)
Squash-merged by litellm-agent from Terrajlz's PR.
* feat(helm): support tpl rendering in podAnnotations (#28609)
Squash-merged by litellm-agent from devauxbr's PR.
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505) (#28575)
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505)
When a Chat Completions request to a GPT-5.4+ model contains both
`tools` and `reasoning_effort`, `completion()` auto-routes through
`responses_api_bridge`. The bridge handler called
`litellm.responses()` / `litellm.aresponses()` without forwarding the
already-resolved `custom_llm_provider`, so the downstream call
re-invoked `get_llm_provider()` with `custom_llm_provider=None` and
stripped a second provider prefix from a `provider/provider/model`
deployment string.
For a deployment configured as `openai/openai/openai/gpt-5.5`,
the bridge flow sent `openai/gpt-5.5` to the upstream API instead of
the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce
model-name allow-lists rejected this as `key_model_access_denied`.
Fix: pass the locally-resolved `custom_llm_provider` into both the
sync `responses()` and async `aresponses()` calls so the downstream
`_resolve_model_provider_for_responses` sees an explicit provider
and skips the second prefix-strip.
New regression test
`tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py`
pins both call sites: each must forward `custom_llm_provider`.
* fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg
Greptile flagged that the previous patch passed custom_llm_provider as an
explicit kwarg to responses()/aresponses() while request_data already
carried it via the spread of sanitized_litellm_params, which would raise
TypeError: got multiple values for keyword argument on every real bridge
call.
Switches to assigning request_data['custom_llm_provider'] before the call
so the resolved provider wins over whatever sanitized_litellm_params spread
in, without duplicating the kwarg.
Updates the regression test to seed request_data with a sentinel
custom_llm_provider so it actually exercises the overwrite path (the
previous test mocked transform_request with a minimal dict and never hit
the conflict).
* chore: trigger shin-agent re-eval on retargeted staging base
* chore: trigger shin-agent re-eval against updated Greptile state
* Add jp. Bedrock cross-region inference profile for claude-opus-4-7
AWS Bedrock documents jp.anthropic.claude-opus-4-7 alongside the
existing us./eu./au./global. profiles for Claude Opus 4.7
(ap-northeast-1 Tokyo / ap-northeast-3 Osaka), but the entry is
missing from model_prices_and_context_window.json. Tokyo-region
users currently get an "unknown model" error when routing through
the JP geo profile.
Adds the entry to both the canonical file and the bundled backup,
mirroring the recent pattern for sonnet-4-6 (#27831). Pricing matches
the other regional profiles (10% premium over base/global).
Regression test pins all six documented profiles (base, global, us, eu,
au, jp) and asserts pricing parity between jp. and au. variants.
Source: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-opus-4-7.html
---------
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* feat(soniox): add soniox audio transcription integration (#29508)
* feat(openmeter): add OPENMETER_TRUST_REQUEST_USER to prevent forged attribution (#29650)
The OpenMeter callback resolves the CloudEvent subject from kwargs["user"]
first, then falls back to the key-bound user_api_key_user_id. For
multi-tenant proxy deployments, a client can set `"user": "..."` in the
request body and cause their usage to be attributed to that arbitrary
string — a billing-attribution forgery risk.
Adds OPENMETER_TRUST_REQUEST_USER env var (default "true" for backward
compatibility). When set to "false", the request-supplied `user` field is
ignored and the subject is resolved solely from user_api_key_user_id.
Matches the existing env-var-driven config pattern in this file
(OPENMETER_API_KEY, OPENMETER_API_ENDPOINT, OPENMETER_EVENT_TYPE).
* feat(search): add you_com as a search provider (#28370)
* feat(search): add you_com as a search provider
Registers You.com Search API as a first-class `search_provider` in the
`search_tools` registry, alongside Tavily, Exa, Perplexity, etc.
- New adapter: litellm/llms/you_com/search/transformation.py
- POSTs to https://ydc-index.io/v1/search
- Auth: X-API-Key from YOUCOM_API_KEY (or explicit api_key)
- Maps Perplexity unified spec: max_results -> count,
search_domain_filter -> include_domains, country -> country
- Flattens results.web + results.news into a single SearchResult list;
snippet prefers snippets[0], falls back to description; page_age -> date
- Registry: SearchProviders.YOU_COM in litellm/types/utils.py and wired
into ProviderConfigManager.get_provider_search_config()
- Pricing entry: model_prices_and_context_window.json (placeholder $0.0;
happy to adjust to maintainers' preferred public number)
- Docs: example router config snippet and example proxy yaml updated
- Tests: tests/search_tests/test_you_com_search.py - 5 mocked tests
(payload shape, domain filter mapping, snippet fallback, news flattening,
missing-api-key error)
Refs upstream expansion signal: #15942
* review fixups: normalize api_base, lowercase country, scope env-var to test
Addresses Greptile inline review comments on #28370:
- get_complete_url: strip trailing slashes from api_base *before* the
endswith("/v1/search") check, so a custom base like ".../v1/search/"
doesn't become ".../v1/search/v1/search".
- transform_search_request: .lower() country before sending, matching
Tavily's convention so callers using the unified spec form ("US") get
consistent behavior across providers.
- Tests: replace direct os.environ writes with an autouse monkeypatch
fixture so YOUCOM_API_KEY is set per-test and removed afterwards.
The missing-key test now uses monkeypatch.delenv. New test asserts the
trailing-slash normalization above.
Reverts the ARCHITECTURE.md / example yaml edits per the reviewer note
that documentation changes belong in the litellm-docs repo.
* support keyless free tier (api.you.com/v1/agents/search) as default
You.com offers an IP-throttled keyless endpoint that returns the same
response shape as the keyed one (~100 queries/day, no signup). This is a
significant onboarding lever - mirrors the keyless DuckDuckGo/SearXNG
providers already in the search_tools registry.
Behavior:
- YOUCOM_API_KEY set -> keyed: POST https://ydc-index.io/v1/search
(X-API-Key header)
- no key -> free: POST https://api.you.com/v1/agents/search
(no auth)
- YOUCOM_API_BASE override -> honored as-is
Tests:
- New: test_you_com_search_keyless_free_tier - asserts URL + absence of
X-API-Key when no key is configured.
- New: test_you_com_search_validate_environment_keyless - asserts the
config no longer raises when the key is absent.
- Removed: test_you_com_search_raises_without_api_key (the precondition
no longer holds).
- Existing payload/domain-filter/etc tests still cover keyed mode via
the autouse YOUCOM_API_KEY fixture.
Verified both endpoints accept POST + return identical JSON shape:
results.web[] / results.news[] with title, url, snippets, description,
page_age.
* register you_com in provider_endpoints_support.json
Adding `litellm/llms/you_com/` requires a corresponding entry in
provider_endpoints_support.json or the
code-quality/check_provider_folders_documented CI check fails.
Follows the compact tavily/serper pattern - endpoints: { search: true }.
Local run of the check now reports "All 114 provider folders are documented".
* move tests under tests/test_litellm/llms/ so CI exercises them
The litellm CI workflows scope unit tests to `tests/test_litellm/...`
(see test-unit-llm-providers.yml: `tests/test_litellm/llms` path), so
tests living under `tests/search_tests/` are never run in CI - which is
why codecov reports 0% patch coverage for the new adapter even though
the unit tests exist and pass locally.
Move test_you_com_search.py into `tests/test_litellm/llms/you_com/` so
the test-unit-llm-providers job picks it up. 7/7 tests still pass at
the new location.
(Sibling search-only providers - tavily, exa_ai, brave, etc. - still
live only in `tests/search_tests/` and would benefit from the same
move, but that is out of scope for this PR.)
* fix(you_com): pin Accept-Encoding: identity to dodge keyless gzip bug
The keyless free-tier endpoint (api.you.com/v1/agents/search) advertises
Content-Encoding: gzip but returns a body that httpx's decoder rejects
with `zlib.error: Error -3 while decompressing data: incorrect header
check`, surfacing as litellm.APIConnectionError in user code. curl works
because it doesn't request compression by default.
Pin Accept-Encoding: identity in validate_environment so the upstream
server skips compression entirely. Harmless on the keyed endpoint
(ydc-index.io/v1/search) which negotiates content-encoding correctly.
The header uses setdefault so a caller-supplied Accept-Encoding still
takes precedence. (Server-side bug has been flagged to the You.com team
separately - once fixed there, this workaround can be removed.)
New unit test: test_you_com_search_pins_identity_accept_encoding.
---------
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* docs: fix README typo (#29419)
Correct clear spelling mistakes in documentation without changing behavior.
Confidence: high
Scope-risk: narrow
Tested: git diff --check; uvx codespell on changed files
Not-tested: Full docs build not run; text-only changes
* Fix(langfuse): pass httpx_client to Langfuse in langfuse_prompt_management to respect SSL_VERIFY (#29480)
* fix(langfuse): pass ssl_verify to Langfuse httpx client
* fix_langfuse_
* add unit tests
* addressed comments
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(models): add minimax/MiniMax-M3 to model cost map (#29412)
Add MiniMax's new flagship MiniMax-M3 to the native minimax provider:
512K context, 128K max output, native multimodal (supports_vision),
reasoning, prompt caching. Pricing (USD/M tokens): input 0.6 / output
2.4 / cache read 0.12. M3 has no active prompt-cache-write tier, so
cache_creation_input_token_cost is omitted.
Updated both the root model_prices_and_context_window.json (remote
source) and the bundled litellm/model_prices_and_context_window_backup.json
(local fallback), keeping them in sync.
* fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log (#29394)
* fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log
* fix(logging): extend terminal event handling to ResponseIncompleteEvent and ResponseFailedEvent; fix return type annotation
* feat(provider): Add Neosantara provider as OpenAI Compatible (#29646)
* Add Neosantara provider
* Register Neosantara provider enum
* Address Neosantara provider review feedback
* Add Neosantara packaged endpoint support
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix: address greptile and veria review feedback
- langfuse: guard httpx_client injection behind version check (>= 2.7.3)
- soniox: propagate audio_transcription_duration in _hidden_params for spend tracking
- soniox: give SONIOX_API_BASE env var priority over caller-supplied api_base
- mcp: replace CancelledError catch with asyncio.wait_for + TimeoutError
* chore(mcp): add migration for per-server timeout column
* fix(test): add tool_use_system_prompt_tokens to model prices schema validator
* fix: mcp timeout test uses real asyncio.wait_for timeout; you_com get_complete_url respects resolved api_key
* fix: forward resolved api_key into you_com endpoint selection and apply timeout to soniox polling GETs
The search flow resolves api_key in validate_environment but never passed it
into get_complete_url, so a programmatic api_key (with no YOUCOM_API_KEY in the
env) set the X-API-Key header yet still selected the keyless free-tier endpoint.
Forward api_key through both the search entrypoint and the http handler so the
keyed endpoint is chosen.
HTTPHandler.get/AsyncHTTPHandler.get had no timeout parameter, so the Soniox
poll and transcript-fetch GETs silently used the client global default instead
of the caller timeout. Add a per-request timeout to get() and forward the
configured timeout from the Soniox handler.
* fix(soniox): price stt-async-v4 per second so transcriptions are billed
The handler stores audio_transcription_duration in _hidden_params, but the
model carried only token cost fields and the response has no token usage, so
the transcription cost path fell through to cost_per_second and returned $0.
An authenticated caller could transcribe Soniox audio without decrementing
their budget. Switch the entry to output_cost_per_second at Soniox's published
$0.10/hour async rate so the stored duration produces a real charge.
* fix(langfuse): use a dedicated httpx client for the SDK injection
The httpx_client handed to the Langfuse SDK came from _get_httpx_client(),
which returns LiteLLM's globally cached HTTPHandler. If Langfuse closed that
client on teardown it would invalidate the shared client used by every other
LiteLLM HTTP call. Build a dedicated httpx.Client instead, still resolving SSL
verification and client certificate from LiteLLM's configuration.
* fix(soniox): prefer caller-supplied api_base over SONIOX_API_BASE env var
* fix(cohere): support max_completion_tokens on cohere v2 chat (default route) (#29779)
* fix(cohere): support max_completion_tokens on cohere v2 chat
The default cohere_chat route resolves to CohereV2ChatConfig, which did not
list or map max_completion_tokens, so get_optional_params raised
UnsupportedParamsError for the standard OpenAI parameter (the modern
replacement for the deprecated max_tokens). The v1 config already maps it to
cohere's max_tokens; mirror that in v2 and add v2 regression tests.
* fix(cohere): make max_completion_tokens take precedence over max_tokens on v2
When both max_tokens and max_completion_tokens are supplied, prefer
max_completion_tokens explicitly rather than relying on dict iteration order,
and cover both orderings with a regression test.
---------
Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com>
Co-authored-by: hectorc98 <hector.chamorroalvarez@adyen.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Dan Lemon <dan@danlemon.com>
Co-authored-by: Saswat <saswatds@users.noreply.github.com>
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Zhao73 <156770117+Zhao73@users.noreply.github.com>
Co-authored-by: Urain Ahmad Shah <60431964+urainshah@users.noreply.github.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: kape <168134658+kapelame@users.noreply.github.com>
Co-authored-by: danisalvaa <159898202+danisalvaa@users.noreply.github.com>
Co-authored-by: Just R <remixingmagelang@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com>
* refactor(ui): centralize proxy base URL resolution into tested resolver
The API base URL join logic was hand-rolled inside networking.tsx and
re-derived inline at hundreds of call sites, with no test coverage and a
latent double-slash bug when the base carried a trailing slash. This pulls
the join into a single pure resolveApiBase() with full unit coverage and
routes the existing resolution through it, also de-duplicating the env
precedence ladder that was copied in two places.
* test(ui): assert root-path redirect joins prefix exactly once
The existing toContain check accepts a doubled separator; tighten it to a
strict prefix match plus a no-double-slash assertion so a regression in the
resolveApiBase origin+SERVER_ROOT_PATH join is caught end-to-end.
Interactive PKCE and OBO servers were mislabeled as M2M, so passthrough never showed the Authorize gate; classify by oauth2_flow + delegate_auth_to_upstream instead.
* test(ci): record/replay OpenAI image gen so the spend E2E isn't outage-bound
The dockerized spend test test_key_info_spend_values_image_generation curls
the proxy for a gpt-image-1 image, which wildcard-routes to real api.openai.com
on every commit; an OpenAI outage then reddens unrelated PRs and each run pays
for an image.
Add an in-repo record/replay reverse proxy (tests/_openai_record_replay_proxy.py)
that sits between the proxy and OpenAI. The first run, and the first after the
recording lapses, records live; subsequent runs replay from the shared Redis
cassette store. The proxy keeps its real separate-process HTTP topology; only
the image model's api_base is pointed at the recorder in CI via
IMAGE_GEN_RECORDER_BASE_URL, which is unset elsewhere so it falls back to
api.openai.com.
Recordings lapse 24h after write and are never refreshed on read, matching the
VCR persister contract, so provider drift is still caught. Replayed responses
drop upstream framing/server headers (content-length, transfer-encoding,
content-encoding, date, server) so the re-serving layer recomputes them,
honoring the Bedrock content-length lesson.
* test(ci): close recorder http client on app shutdown
Add a Starlette lifespan that closes the self-created httpx.AsyncClient on
teardown, and leave caller-injected clients untouched so reuse across
create_app calls is not broken. Covers the unclosed-client ResourceWarning
raised in review.
The Redis cassette persister slid the 24h TTL forward on every successful
read, so any cassette replayed at least once per day never expired. With CI
running more than once a day that means a recorded response is replayed
forever and the suite never re-hits the provider, so a changed request or
response contract goes undetected indefinitely.
Drop the refresh-on-read. The TTL now counts down from the last write, so a
cassette lapses 24h after it was recorded and the next run past that point
re-records live and catches provider drift. Per-commit runs in between still
replay from cache; only the one boundary-crossing run goes live.
* fix(auth): expand all-team-models sentinel in can_key_call_model
Keys with models=["all-team-models"] were denied during batch JSONL
model validation because can_key_call_model matched the literal string
against the model name. Add _resolve_key_models_for_auth_check to
expand the sentinel to team_models before the check, consistent with
get_key_models in model_checks.py and the completion-route bypass.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(auth): document empty team_models unrestricted access behavior; add regression test
Adds a docstring note to _resolve_key_models_for_auth_check explaining that
when team_models is empty, all-team-models resolves to [] which is treated as
unrestricted access (consistent with get_key_models behavior on other auth
paths). Adds a test to lock in this behavior.
* fix(auth): deny all-team-models access when key has no team_id
A key configured with models=["all-team-models"] but no team_id could
previously resolve to an empty allowlist, which _check_model_access_helper
treats as unrestricted access. Now the sentinel is only expanded when
team_id is set; otherwise the unresolved sentinel stays in the model list
and causes a deny (no real model name matches it). Same fix applied to
get_key_models in model_checks.py for consistency across batch and
non-batch auth paths.
* style: black format model_checks.py
* Fix batch all-team-models auth
* style: black format batch_rate_limiter.py
* fix(test): add tool_use_system_prompt_tokens to model prices schema validator
* fix(batch): catch get_team_object errors to avoid 404 escaping batch auth
* fix(batch): apply per-member model scope check after team auth in batch validation
* Fail closed on batch team auth fetch errors
* test(batch): cover team_object grant and member-scope denial in batch auth
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
