The field itself landed on staging via 0c5c9c79d7; these are the regression tests from PR #31435 for the retrieval-facing half.
(cherry picked from commit a9a322d63f6d4658b1f28d1622335775e94736a4)
Bedrock batch jobs write their results to s3_output_bucket_name when it differs
from the input bucket, but the file-content retrieval path validated the file id
only against the input bucket (s3_bucket_name). A deployment that configures a
separate output bucket therefore could not retrieve its own batch outputs: the
id validated against the input bucket and was rejected as a foreign bucket.
Resolve the trusted output bucket alongside the input bucket from the immutable
credential snapshot (or AWS_S3_OUTPUT_BUCKET_NAME), and try the file id against
each configured bucket, returning the first that validates. The SSRF guard is
preserved: only server-configured buckets are tried, never a request param, and
an id outside both is still rejected.
(cherry picked from commit 1d407c2f26)
Resolves the transform_create_file_response conflict by keeping the
_uploaded_object_size handoff over the response Content-Length read,
and adds the rebind-ok justification LIT011 now requires for the
upload-size litellm_params handoff after the base budget ratcheted.
Three groups, all verified by running the suite rather than by inspection.
18 files whose every test function carries an unconditional @pytest.mark.skip,
39 test functions in total. They are collected on every CI run and always skip,
so they advertise coverage the suite does not have. Reasons on the marks include
"AWS Suspended Account", "lakera deprecated their v1 endpoint" and "moved to
using 'otel' for logging"; 26 of the marks predate 2025.
30 test functions with a byte-identical body and identical decorators to a
sibling in the same file and class, differing only in name. Deleting one of each
pair removes no coverage. Four further candidates were excluded because they
override an inherited test, where deleting the override un-shadows the base
class implementation instead of removing a duplicate.
9 test functions that a later definition of the same name shadows, so Python
never binds them and pytest cannot collect them.
One file that is a demo script rather than a test; its own docstring says to run
it with python.
Verification: collecting the 26 edited files gives 2,492 node IDs before and
2,462 after. The 30 duplicate deletions account for exactly 30 removals, the 9
shadowed deletions account for 0 (confirming at runtime that they were never
collectable), nothing unexplained disappeared, and nothing new appeared. No
other test or module imports any deleted symbol.
S3 rebuilds the canonical request from the wire path with single
percent-encoding, which botocore models as S3SigV4Auth. Generic SigV4Auth
quotes the already encoded path a second time, so an object key holding any
character that percent-encodes was signed over %2520 while the request
carried %20, and S3 answered 403 SignatureDoesNotMatch.
The S3 logger was corrected in #35726; the Bedrock managed-files upload and
retrieval paths copied that same pre-fix pattern and were left behind, so a
configured bucket prefix with a space 403s every file upload and every file
content read.
Bedrock managed-batch file upload read `messages` unconditionally, so a
JSONL record shaped for /v1/completions (`prompt`) or /v1/responses
(`input`) reached the per-provider transform with an empty message list.
Anthropic and Nova rejected it at POST /v1/files, and the passthrough
providers shipped an empty conversation to AWS.
Classify each record by its OpenAI batch `url`, then normalize the
non-embedding shapes to chat completions before the Bedrock transforms:
`prompt` wraps into user messages the way litellm.text_completion does in
real time, and `input` goes through the existing Responses-to-Chat bridge.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (#30708)
OpenAI GPT-5 models require max_completion_tokens >= 16.
Health checks were using 5 (proxy/health_check.py) and 10
(health_check_helpers.py), causing failures on GPT-5 models.
Fixes#23836
* fix: increase health check max_tokens from 5 to 16 (#23836) (#26610)
GPT-5 models enforce a minimum of 16 for max_output_tokens. The current
default of 5 still causes health checks to fail for these models. Bump
the non-wildcard default to 16 — the smallest value that satisfies all
known provider minimums while keeping health checks lightweight.
Also tightens the wildcard test assertion from a weak disjunctive check
to strict key-absence.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (#30696)
* fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema.
* fix: remove async keyword from test.
* fix: make Bedrock Mantle Responses routing data-driven per model (#30700)
* Make Bedrock Mantle Responses routing data-driven per model
Route Bedrock Mantle models to the native Responses API based on each
model's price-map capability signal instead of a hardcoded model-name
heuristic, and derive the OpenAI-compatible base path segment per model.
Responses dispatch now selects the native config when the model advertises
responses support (/v1/responses in supported_endpoints, or mode=responses),
both overridable via register_model and proxy model_info. This enables
native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping
chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing
chat-completions emulation. Capability is per-model, so gpt-oss-120b routes
natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss
substring.
The wire path is a separate concern, driven by the existing
use_openai_responses_path flag rather than a model-name match: gpt-5.x and
gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat
config now derives its base from the same flag, fixing gemma-4
chat-completions requests that previously went to /v1 instead of /openai/v1.
Cost maps: add supported_endpoints to the gpt-oss entries (responses for the
non-safeguard variants, chat-only for safeguard) and supported_endpoints +
use_openai_responses_path to all three gemma-4 entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: move capability helper into bedrock_mantle package
Move the Responses capability check out of utils.py into
litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses,
alongside its companion wire-path helper mantle_base_segment. Both are now
pure functions of (model, model_cost): the price-map mode/supported_endpoints
read replaces the get_model_info call, so the rules are unit-testable without
patching global state and the Bedrock Mantle package is self-contained.
Use str | None instead of Optional[str] on the new signatures to satisfy the
ruff UP045 strict-rule gate. Add direct unit tests for both helpers.
Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b
now legitimately supports Responses, so it can no longer be the
"None after restore" vehicle; use the chat-only safeguard variant, which
isolates the register/restore effect from the model's own capability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)
The tiered cost calculator resolved a tier's per-token cost with
`tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or`
short-circuits on any falsy value, a tier that legitimately prices a
component at 0.0 (e.g. a free-cache-read tier with
cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated
as missing and silently billed at the full fallback rate
(input_cost_per_token / output_cost_per_token).
The flat-pricing path in the same module already handles this correctly
with an `is None` guard. Resolve tier costs through a small helper that
mirrors it, so 0.0 is honored at both the in-range and overflow sites.
No shipped model currently has a 0.0 tier cost, so this is a latent
defect; the fix makes the tiered path consistent with the flat path and
prevents over-charging the first time such a tier appears. Adds unit
tests covering the in-range and overflow paths, and drops an unused
import flagged by ruff in the touched test file.
* feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)
* fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (#30618)
In the messages->chat/completions bridge, translate_anthropic_tools_to_openai
merged every non-mapped tool key into the function parameters dict. The
Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object'
-> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type).
Exclude 'type' from the passthrough. Fixes#30557.
* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)
An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:
1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
engine, spawn new one.
2. The engine-death watcher sees that kill, assumes a crash, and calls
`attempt_db_reconnect(force=True)` (a different lock,
`_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
3. In-flight queries failing during the swap are classified as transport
errors and trigger their own `attempt_db_reconnect` -> recreate again.
Fix coordinates planned restarts across the wrapper and the watcher:
- PrismaWrapper records the old engine PID in `_expected_engine_deaths`
before killing it; all four watcher death-detectors (waitpid thread,
pidfd, already-dead probe, os.kill poll) consume that PID and skip the
reconnect instead of treating it as a crash.
- `recreate_prisma_client` now serializes through `_reconnection_lock` and
bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
as an optimistic-lock token, so racing/cascading recreates collapse into a
single restart (losers no-op). This closes the two-lock gap.
- The direct reconnect path probes the writer with SELECT 1 before
recreating; a healthy connection (e.g. engine already replaced by a
refresh) skips the recreate entirely.
- `_safe_refresh_token` coalesces: it skips when the current token still has
more than the refresh buffer of runway, so stacked triggers (proactive
loop + __getattr__ fallback) don't each restart the engine. An
`on_engine_replaced` hook re-arms the watcher on the new PID.
RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.
* feat(bedrock): support file content retrieval for batch output files (#30595)
Implements transform_file_content_request and transform_file_content_response
in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch
files. The request transform resolves the file id (direct s3:// URI or base64
unified id) to its S3 object, validates bucket and key prefix against the
server-configured bucket, and SigV4-signs an S3 GetObject using the same
credential and region resolution as the existing upload path. The credential
and region params are validated into a typed model at the boundary, so the only
untyped values left are the botocore signing primitives.
Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries
s3_bucket_name (previously dropped when building deployment credentials) and
the managed-files hook passes the deployment credential snapshot when routing
afile_content, so unified-id content retrieval works with per-model bucket
config instead of only the AWS_S3_BUCKET_NAME env var.
Preserves managed-file access control: the proxy file-content endpoint now
rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the
owner/team check that only runs for unified ids and let a caller read another
tenant's batch output by its object key. Managed outputs are reachable only
through their unified file id. The afile_content "not found" error now reports
the caller's unified id rather than the resolved internal S3 URI.
Fixes#16186, #15563
* fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)
* fix(oci): map Cohere tool array/object params to lowercase builtins
OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare
"List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema
arrays. MLflow {{trace}} judges trip this: their tools (get_root_span,
get_span) take an attributes_to_fetch array. The lowercase builtins list/dict
are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but
both are lowercased for consistency).
Verified live against us-chicago-1 (cohere.command-a-03-2025 and
command-latest). Adds a unit regression on the transformed parameterDefinitions
plus a gated integration test exercising an array-param tool end to end.
* fix(oci): make Cohere agentic tool-calling continuation work
Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges
drive once a tool has been executed and its result is fed back.
Request side: litellm pulled the last user message into the top-level `message`
and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that
("cannot specify message if the last entry in chat history contains tool
results"), and an empty message alone is rejected too ("message must be at least
1 token long or tool results must be specified"). OCI carries the current turn's
results in a dedicated top-level `toolResults` field. The Cohere transform now
sends an empty message, keeps the user turn in chatHistory, and puts the results
in `toolResults`, matching the langchain-oracle reference. Tool results are no
longer represented as chatHistory entries.
Response side: tool-grounded answers come back with citations carrying
`documentIds` (camelCase) and no `document_ids`, which made the required
`CohereCitation.document_ids` field fail validation and sink the whole response
parse. Those citations are never surfaced, so the field (and CohereSearchQuery's
generation_id) is now optional.
Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest),
single and multi-round tool loops. Adds unit regressions on the transformed
request shape and on citation parsing, plus gated integration tests for the
continuation.
* feat: integrate Repelloai Argus guardrail (#30673)
* feat(guardrails): add RepelloAI Argus guardrail integration (#1)
* feat(guardrails): add RepelloAI Argus guardrail integration
Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.
* fix(guardrails): harden RepelloAI Argus guardrail
- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
block on unknown/malformed verdicts so an API change can't silently
disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
expose unreachable_fallback in the config schema, and stop leaking the
raw provider response / exception strings to the client
* fix(guardrails): address RepelloAI Argus review feedback
- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description
* docs(guardrails): add RepelloAI Argus docs page and dashboard listing
- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution
* fix(guardrails): keep RepelloAI asset_id optional in config model
A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.
* Add comment for last user turn scanning
* feat(guardrails): harden repelloai scanning
* feat(guardrails): expand repelloai scanning to include tool definitions
Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.
* refactor(guardrails): fix and harden repelloai schema text extraction
- Fix duplicate text in _iter_schema_text: previously all dict values were
re-queued onto the stack even after scalar/list keys were already extracted
explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param
* fix(guardrails): resolve pyright type errors in repelloai guardrail
- Narrow async_handler.post return from Response|None to Response with
explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
type in pyright
* fix(guardrails/repello): include Responses API instructions field in prompt scan
The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.
* feat: add api_key to config model and read prompt from data dict
* fix(guardrails/repello): plug input_text and tool-call response bypass gaps
Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.
Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.
Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.
* fix(guardrails/repello): scan Responses API function_call output arguments
Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.
* refactor(guardrails/repello): clean up typing and remove lint-any workarounds
- Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout
- Use dict[str, object] instead of bare dict in all signatures
- Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly
- Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel
- Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks
- Use TypeAdapter.validate_json() instead of response.json() + manual dict construction
- Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any
- Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check
- Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType
- Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel]
* fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning
- Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate
- Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks
* refactor: modifications for lint check
* feat: add Pinstripes as an OpenAI-compatible provider (#30567)
* feat: add Pinstripes as an OpenAI-compatible provider
Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference
provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.)
with per-token pricing and no subscriptions.
Changes:
- `litellm/llms/openai_like/providers.json`: register pinstripes with
base_url, api_key_env, and max_completion_tokens→max_tokens mapping
- `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders
- `litellm/constants.py`: add to openai_compatible_providers and
openai_compatible_endpoints lists
- `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect
provider when api_base is "https://pinstripes.io/v1"
- `provider_endpoints_support.json`: document supported endpoints
- `tests/`: 7 unit tests covering provider registration, resolution,
URL auto-detection, api_base override, and Router config
Usage:
import litellm
response = litellm.completion(
model="pinstripes/ps/glm-4.5-air",
messages=[{"role": "user", "content": "Hello"}],
api_key=os.environ["PINSTRIPES_API_KEY"],
)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): resolve Greptile P1 review comments
- Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works
- Set responses: false in provider_endpoints_support.json — not actually wired up
- Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): add api_base_env and correct responses capability
- Add api_base_env: PINSTRIPES_API_BASE to providers.json
- Set responses: false in provider_endpoints_support.json
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): wire up Responses API — add supported_endpoints
Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so
JSONProviderRegistry.supports_responses_api returns true correctly,
matching what provider_endpoints_support.json advertises.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(pinstripes): enable embeddings endpoint
Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings.
Add /v1/embeddings to supported_endpoints and set embeddings: true.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json
Matches the file's existing convention. Flagged by Greptile review.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(pinstripes): set a2a: false — A2A protocol not implemented
All comparable JSON-configured providers (tensormesh, parasail, empiriolabs,
libertai, neosantara) have a2a: false. Pinstripes does not implement the
Google A2A protocol, so this should be false to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: inference_provider <max@redactedlab.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(rag): attach existing OpenAI file ids (#30628)
* fix(rag): attach existing OpenAI file ids
* chore: use modern typing in rag ingest fix
* chore: retrigger ci
* fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)
cache_control_injection_points was only consumed by the chat/completions
prompt-management hook; on the native Anthropic /v1/messages path it was
forwarded unused, so deployment-level cache injection was silently dropped
(cache_creation_input_tokens stayed 0 for Anthropic-native clients).
Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject
cache_control at block level for system / tools / message locations (the only
forms /v1/messages accepts), wire it into the native anthropic_messages
handler, and pop the param so it does not leak upstream as an unknown field.
A {location: message, role: system} config is redirected to the top-level
system prompt so the same YAML works on both endpoints.
Injection respects Anthropic's 4-block cache_control limit shared across
system, tools, and messages: client-supplied markers count toward the cap and
are never overwritten, a slot is reserved per Bedrock tool_config point, and
injection stops once the budget is exhausted. Locations this path cannot
represent (tool_config) are forwarded downstream instead of being silently
consumed, mirroring get_chat_completion_prompt's remaining_points pass-through.
Built on litellm_internal_staging. Refs BerriAI/litellm#30293
* fix(proxy): release budget reservation when a request is cancelled mid-flight (#30522)
* fix(proxy): release budget reservation on cancel when no chunk was delivered
The pre-call budget reservation increments the cross-pod spend counter by a
request's worst-case cost, then reconciles it on success (cost callback) or
error (failure hook). A client disconnect or timeout cancels the request and
surfaces as CancelledError / GeneratorExit, which neither path catches, so the
reservation leaks. Under a retry storm the leaked holds accumulate, pin the
counter above real spend, and return spurious 429 "Budget has been exceeded" to
keys whose spend is far below budget; the counter only recovers when its TTL
lapses, so the failure is intermittent and self-healing.
Release the reservation in async_streaming_data_generator (which the Anthropic
and Google SSE generators delegate to) on the (CancelledError, GeneratorExit)
path, alongside the existing max_parallel_requests release. release_budget_
reservation_on_cancel runs under asyncio.shield so it completes despite the
in-progress cancellation, is guarded by the reservation's finalized flag, and
swallows a failing release so it cannot replace the in-flight cancellation.
The refund is gated on whether a chunk reached the client. The flag is set
immediately before the yield, after the slow-path hook await: an async generator
suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk
sees it True (keep the hold), while a cancellation during the slow-path await
leaves it False (refund, nothing sent). A non-streaming cancellation delivers
nothing and a completed non-streaming response is reconciled by the success
callback, so neither needs a release here.
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(proxy): reconcile a cancelled reservation to input cost, not zero
A streaming request cancelled before the first chunk previously reconciled its
reservation to zero and finalized it. But by the time the generator is
consuming the response the provider call was already dispatched, so the input
tokens were billed even though no chunk reached the client, and the
success/failure cost callbacks are skipped on cancellation. Refunding to zero
let a caller send an expensive request and abort pre-token to dodge the input
charge.
Compute the request's input-token cost at reservation time and reconcile the
cancelled reservation to it instead of zero. The worst-case output portion of
the reservation is still released (so a legitimate mid-flight cancellation no
longer pins the counter and 429s the key), while the input the provider already
processed is charged.
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(caching): encode object name in GCS cache GET path (#30378)
GCS cache reads always missed when gcs_path was set. The GET methods
interpolated the object name directly into the URL path, while the GCS
JSON API requires it to be URL-encoded (a "/" must be sent as %2F).
With gcs_path configured the object name is "<prefix>/<sha256>", so the
raw slash produced a malformed object path and GCS returned 404. httpx
does not raise on 4xx, so the status_code == 200 check fell through and
get/async_get returned None, silently missing on every read. Without
gcs_path the key has no slash, which is why this went unnoticed.
Wrap the object name with urllib.parse.quote(..., safe="") in get_cache
and async_get_cache. Apply the same encoding to the name= query
parameter in set_cache and async_set_cache so the key written matches
the key read back.
Adds regression tests asserting the GET path and SET query are encoded
(%2F) when gcs_path is set, for both sync and async paths; these fail on
the unpatched code.
Fixes#30377
* chore: add soniox stt-async-v5 model (#30672)
* fix(proxy): include model group aliases in v1 model info (#30626)
* Include model group aliases in v1 model info
* Fix model info alias implementation
* removed extra blank line
* chore: rerun CI
* fix(lint): remove redundant noqa directive in proxy_cli.py
* fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme
* Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme"
This reverts commit 52c7a07777.
* Revert "fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)"
This reverts commit c9e8a177bd.
* Revert "fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)"
This reverts commit 85828da695.
* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)
An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:
1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
engine, spawn new one.
2. The engine-death watcher sees that kill, assumes a crash, and calls
`attempt_db_reconnect(force=True)` (a different lock,
`_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
3. In-flight queries failing during the swap are classified as transport
errors and trigger their own `attempt_db_reconnect` -> recreate again.
Fix coordinates planned restarts across the wrapper and the watcher:
- PrismaWrapper records the old engine PID in `_expected_engine_deaths`
before killing it; all four watcher death-detectors (waitpid thread,
pidfd, already-dead probe, os.kill poll) consume that PID and skip the
reconnect instead of treating it as a crash.
- `recreate_prisma_client` now serializes through `_reconnection_lock` and
bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
as an optimistic-lock token, so racing/cascading recreates collapse into a
single restart (losers no-op). This closes the two-lock gap.
- The direct reconnect path probes the writer with SELECT 1 before
recreating; a healthy connection (e.g. engine already replaced by a
refresh) skips the recreate entirely.
- `_safe_refresh_token` coalesces: it skips when the current token still has
more than the refresh buffer of runway, so stacked triggers (proactive
loop + __getattr__ fallback) don't each restart the engine. An
`on_engine_replaced` hook re-arms the watcher on the new PID.
RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.
* fix(lint): modernize type annotations in IAM-refresh prisma client files (UP006/UP045)
* Revert "feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)"
This reverts commit f530b2237c.
* Revert "fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)"
This reverts commit 4f58bd0df5.
* Revert "fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)"
This reverts commit 50f34e0b15.
* Revert "fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)"
This reverts commit 0544eed6ea.
* fix(bedrock_mantle): restore BedrockMantleAuthMixin and constants removed by routing rewrite
* fix(key management): restore exact /key/list user_id & key_alias matching by default (#30593)
Before substring search was added (commit 33bd570d5e), /key/list matched user_id
and key_alias exactly. That change made admin-authenticated calls substring-match
by default, breaking the prior contract: a caller passing an exact user_id as an
access filter (e.g. an integration scoping to one user with an admin key) then
received other users' keys -- user_id="alice" also returned "alice2",
"alice-test", etc. This is a cross-user key disclosure.
Make substring matching opt-in via a new admin-only substring_matching=true query
param; default to exact, restoring the prior behavior. The dashboard search box
(keyListCall) passes the flag so partial search still works. Non-admins remain
exact and scoped to their own keys.
Updates the proxy-behavior key_alias test to opt in and adds an exact-by-default
guard; adds list_keys unit coverage for the opt-in gate.
---------
Co-authored-by: perseus <51974392+tcconnally@users.noreply.github.com>
Co-authored-by: Hannah Smith <64043506+hannahmadison@users.noreply.github.com>
Co-authored-by: Charlie Patterson <Pattersoncharlesl@gmail.com>
Co-authored-by: Matthew Lapointe <mlapointe@alpha-sense.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com>
Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com>
Co-authored-by: Nitish Agarwal <1592163+nitishagar@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com>
Co-authored-by: AD Mohanraj <admohanraj@gmail.com>
Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com>
Co-authored-by: Lavish Bansal <lavish.bansal619@gmail.com>
Co-authored-by: max-amos <gruffulom@gmail.com>
Co-authored-by: inference_provider <max@redactedlab.com>
Co-authored-by: NK <93352237+Nithish-Yenaganti@users.noreply.github.com>
Co-authored-by: 安妮的心动录 <74543653+anneheartrecord@users.noreply.github.com>
Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com>
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Burak Ömür <burak.omur.1998@gmail.com>
Co-authored-by: Dan Lemon <daniel.lemon@amazee.io>
Co-authored-by: Vanika Dangi <166420943+vanika02@users.noreply.github.com>
Co-authored-by: Jay Gowdy <130084966+jgowdy-godaddy@users.noreply.github.com>
* feat(bedrock_mantle): add SigV4/IAM auth to Responses API route (fixes#29665) (#29788)
* feat(responses): add default no-op sign_request to BaseResponsesAPIConfig
* feat(responses): call sign_request after body is final, send signed bytes when signed
* feat(bedrock_mantle): add SigV4 sign_request via composed BaseAWSLLM (bearer path)
* test(bedrock_mantle): cover SigV4 access-key, AssumeRole, body bytes, region/auth consistency
* feat(bedrock_mantle): defer auth to sign_request; validate_environment no longer requires bearer
* docs(bedrock_mantle): document SigV4 + Bearer auth on Responses route
* test(responses): cover fake-stream signing order and mantle bearer arg/env precedence
* fix(bedrock_mantle): wrap all botocore credential errors with both-paths guidance
* fix(bedrock_mantle): catch specific credential errors, not all BotoCoreError, so STS transport failures are not masked
* fix(bedrock_mantle): sign the compact Responses route too, not just create
* fix(github-copilot): route per-model on /v1/responses based on model info (#29747)
* feat(focus): add GCS destination for FOCUS export (#29751)
* test: add failing tests for FocusGCSDestination
* feat: add FocusGCSDestination reusing GCSBucketBase auth
* feat: register FocusGCSDestination in factory; export from __init__
* fix(focus): preserve GCS_PATH_SERVICE_ACCOUNT when service_account_json not in config
* style: apply Black formatting to gcs_destination and tests
* style: apply Black formatting to factory.py
* fix(bedrock): omit empty additionalModelRequestFields and system from Converse API payload (#29565)
Amazon Nova Pro (and other strict Bedrock models) return 400 Malformed input
request when additionalModelRequestFields: {} or system: [] are present in the
payload. Both fields are optional in CommonRequestObject (total=False) and must
be omitted rather than sent as empty structures.
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible in pass-through cost tracking (#29730)
* fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible
Azure OpenAI resources created via the newer "Azure AI Foundry" /
Cognitive Services pathway live on `*.cognitiveservices.azure.com`
subdomains, not the older `openai.azure.com`. Both are valid Azure
OpenAI surfaces in production today.
The OpenAI pass-through cost-tracking handler hard-codes only the older
hostname in five places (four `is_openai_*_route` methods on
OpenAIPassthroughLoggingHandler, plus is_openai_route on
PassThroughEndpointLogging). As a result, calls from newer Azure
deployments are silently classified as "not an OpenAI route", the
dispatch into the cost-tracking handler is skipped, and tokens/cost
never get extracted into LiteLLM_SpendLogs — the row gets written with
prompt_tokens=0, completion_tokens=0, spend=0, model='unknown'.
Reproduced 2026-06-04 against a real Azure OpenAI deployment on
`*.cognitiveservices.azure.com` proxied through LiteLLM v1.88.0.
Fix: factor the hostname check into a single helper
`_is_openai_compatible_host` listing all three recognized surfaces
(api.openai.com, openai.azure.com, cognitiveservices.azure.com), and
have all five call sites delegate to it. Purely additive — never
weakens recognition for the originally-supported hostnames.
Adds a test
`test_is_openai_route_recognizes_cognitiveservices_azure_com` that
exercises all four `is_openai_*_route` static methods against
`*.cognitiveservices.azure.com` URLs (positive cases per route + a
small cross-route negative to confirm route-specific path matching
still works on the new hostname).
Out of scope for this PR (separate followup):
- `openai_passthrough_handler` calls chat/completions
`transform_response` on Responses API payloads (`output:` not
`choices:`), which throws inside the dispatch and drops the
SpendLogs row entirely. Recognized + tracked separately.
* ci: trigger fresh run
Empty commit to re-run checks. The previous auth-and-jwt failure was
a transient HuggingFace Hub 429 rate-limit hitting tokenizer downloads
in tests/proxy_unit_tests/test_custom_tokenizer_bug.py — unrelated to
this PR's scope (hostname recognition in pass-through cost tracking).
No code change.
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(responses): preserve forced-function tool_choice name in Responses to Chat transform (#29812)
The Responses API forces a specific function with a top-level name
({"type": "function", "name": "X"}), but _transform_tool_choice only handled the
nested Chat Completions shape and fell through to returning "required" for the flat
form, silently dropping the function name and degrading a forced function call to
force-any-tool. Map the flat Responses shape to the nested Chat shape, keeping the
"required" fallback when no name is present.
* Preserve x-anthropic-billing-header system blocks for first-party Anthropic (#29584)
* Preserve x-anthropic-billing-header system blocks for first-party Anthropic
PR #20951 strips system blocks beginning with "x-anthropic-billing-header:" for
every Anthropic target. That block is how the first-party Anthropic API recognizes
Claude Code subscription (OAuth) traffic, so dropping it makes requests that carry
only that block, such as the auto-mode tool-safety classifier, fail with a
misleading 429 rate_limit_error; normal turns still work because they also carry
the "You are Claude Code" identity block.
Gate the strip behind should_strip_billing_metadata(), defaulting to False on the
first-party AnthropicConfig and AnthropicMessagesConfig so the block is kept, and
overridden to True on the providers that reach these transforms and reject the
block (Bedrock platform, Vertex, Azure for the chat path; Minimax, Azure, DeepSeek
for the messages path). Behavior for those providers is unchanged.
* Strip billing header on Bedrock invoke and Vertex messages pass-through
Two more subclasses reach the gated strip but inherited keep-by-default.
AmazonAnthropicClaudeConfig (Bedrock invoke) calls AnthropicConfig.transform_request,
which calls translate_system_message, and VertexAIPartnerModelsAnthropicMessagesConfig
(Vertex messages pass-through) calls super().transform_anthropic_messages_request.
Override should_strip_billing_metadata() to True on both.
Add a parametrized test asserting the flag for every first-party base (False) and
provider subclass (True), covering all overrides, plus a translate_system_message
regression test for the Bedrock invoke path.
* fix(cache): log hashed cache keys (#29890)
* fix(ui): save routing groups as list (#29889)
* Revert "fix(ui): save routing groups as list (#29889)" (#29928)
This reverts commit 9b1f78ffa7.
* feat(parasail): add Parasail as a JSON-configured OpenAI-compatible provider (#29842)
* feat(parasail): add Parasail as a JSON-configured OpenAI-compatible provider
Registers parasail in the openai_like JSON provider loader with both
/v1/chat/completions and /v1/responses support. Parasail's Responses API
rejects store:true and any request that omits store, so the loader gains a
force_store_false special_handling flag; the parasail entry sets it and
the generated Responses config overrides store=false on every call. This
keeps callers from hitting "State storage not supported" and matches what
Parasail's docs require.
Adds the PARASAIL enum value, listing under openai_compatible_providers,
provider documentation at docs/my-website/docs/providers/parasail.md, and
a focused unit test file under tests/test_litellm/llms/parasail/ that
covers JSON registration, chat URL construction, Responses URL
construction with PARASAIL_API_BASE override, and the force_store_false
regression in both the caller-sent-store=true and caller-omitted cases.
* fix(parasail): register in provider_endpoints_support, drop in-repo docs
Greptile review feedback. The provider doc belongs in the litellm-docs
repo, not this one's docs/my-website tree; removing it here. Adds the
parasail entry to provider_endpoints_support.json so the
check_provider_folders_documented.py CI check passes (chat_completions
and responses true; others false).
* fix: normalize Anthropic passthrough server tool usage (#29827)
* test(anthropic): cover server_tool_use dict cost tracking
* fix: normalize Anthropic server tool usage
(cherry picked from commit 982f726bed)
* fix: keep server tool usage subscriptable
(cherry picked from commit 70280b9b27)
---------
Co-authored-by: Genmin <joey@joeyroth.com>
* fix(proxy): fix typo generic_role_mappoings -> generic_role_mappings in ui_sso.py (#29753)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(proxy): add disable_budget_reservation general setting (#27639) (#29493)
* feat(proxy): add disable_budget_reservation general setting (#27639)
* feat(proxy): register disable_budget_reservation in ConfigGeneralSettings (#27639)
* docs(proxy): document disable_budget_reservation concurrency tradeoff (#27639)
* ci: re-trigger flaky docker build (prisma generate ECONNRESET)
* fix(proxy): warn and document budget enforcement tradeoff when disable_budget_reservation is set (#27639)
* feat(gemini_tts): adding support to Gemini TTS languageCode parameters (#29623)
* Adding support to Gemini TTS Language Code parameters
* Mapping Gemini TTS languageCode param in Docstring
* Use snake_case for language_code input keyMapping Gemini TTS languageCode param in Docstring
* Restoring files modified under enterprise/litellm_enterprise due to lint/formatting checks
---------
Co-authored-by: João Garrido <joaogarrido@google.com>
* feat(guardrails): capture user and model metadata in CrowdStrike AIDR (#29517)
* fix(proxy): require OpenAI path segment for shared Azure Cognitive Services domains
Address Greptile review: the `*.cognitiveservices.azure.com` /
`*.openai.azure.com` domains are shared by every Azure Cognitive Service
(Speech, Vision, Language, ...), so a hostname-only substring match
misclassified non-OpenAI Azure traffic as OpenAI routes.
- Replace the substring host test with suffix matching (rejects look-alike
domains like cognitiveservices.azure.com.attacker.example).
- Add `_is_openai_compatible_url` that requires an OpenAI-style path marker
(`/openai/` or `/v1/`) on the shared Azure domains, and use it in
PassThroughEndpointLogging.is_openai_route (previously hostname-only).
- Add negative tests for Azure Speech/Vision paths and look-alike domains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: support Responses input in Redis semantic cache (#29581)
* fix: support responses input in redis semantic cache
* test: cover redis semantic prompt extraction
* test: handle blank redis semantic text fallbacks
* chore: remove async cache dead statement
* test: cover redis semantic cache miss paths
* fix: filter sensitive cache lookup kwargs
* chore: rerun ci after huggingface rate limit
* chore(ui): regenerate dashboard API types (npm run gen:api)
Sync src/lib/http/schema.d.ts with the proxy OpenAPI spec: adds the
disable_budget_reservation general-settings field and picks up the
RateLimitError docstring reindent. Fixes the gen:api CI drift check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(bedrock): assert empty additionalModelRequestFields is omitted
The Converse transformer now drops an empty additionalModelRequestFields
block instead of sending it as `{}`. Update test_bedrock_top_k_param so
models without top_k support (llama3) assert the key is absent rather than
equal to an empty dict.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: codgician <15964984+codgician@users.noreply.github.com>
Co-authored-by: Praveen Ghuge <95286176+pghuge-cloudwiz@users.noreply.github.com>
Co-authored-by: Roi <roytev@gmail.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Liam Scott <liam@uilliam.com>
Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com>
Co-authored-by: Ceder Dens <cederdens@gmail.com>
Co-authored-by: 冯基魁 <56265583+fengjikui@users.noreply.github.com>
Co-authored-by: Kai Huang <kaihuang724@gmail.com>
Co-authored-by: rinto <54238243+ririnto@users.noreply.github.com>
Co-authored-by: Genmin <joey@joeyroth.com>
Co-authored-by: Arnav Bhilwariya <arnavbhilwariya0408@gmail.com>
Co-authored-by: Armaan Sandhu <74664101+Ar-maan05@users.noreply.github.com>
Co-authored-by: João Garrido <48538534+johngarrido@users.noreply.github.com>
Co-authored-by: João Garrido <joaogarrido@google.com>
Co-authored-by: Kenan Yildirim <kenan@kenany.me>
Co-authored-by: Dávid Balatoni <balcsida@gmail.com>
* update bedrock models in tests
* updated more tests and model_prices_and_context_window
* fix model id and pricing
* replace more sonnet models
* update tests
* git push
* update pricing
* flaky total cost
* monkey patch
* relax the cost change
* fix and revert some changes
* revert the pricing
* chore: move cost/pricing changes to bedrock-cost-fixes branch
* chore: split Bedrock file-api beta stripping to separate branch
Removes strip_unsupported_file_api_betas_for_bedrock_invoke from this branch;
see litellm_bedrock_invoke_strip_file_api_betas for that fix.
Made-with: Cursor
* fix(bedrock): respect s3_region_name for batch file uploads (#23569)
* fix(bedrock): respect s3_region_name for batch file uploads (GovCloud fix)
* fix: s3_region_name always wins over aws_region_name for S3 signing (Greptile feedback)
* fix: _filter_headers_for_aws_signature - Bedrock KB (#23571)
* fix: _filter_headers_for_aws_signature
* fix: filter None header values in all post-signing re-merge paths
Addresses Greptile feedback: None-valued headers were being filtered
during SigV4 signing but re-merged back into the final headers dict
afterward, which would cause downstream HTTP client failures.
Made-with: Cursor
* feat(router): tag_regex routing — route by User-Agent regex without per-developer tag config (#23594)
* feat(router): add tag_regex support for header-based routing
Adds a new `tag_regex` field to litellm_params that lets operators route
requests based on regex patterns matched against request headers — primarily
User-Agent — without requiring per-developer tag configuration.
Use case: route all Claude Code traffic (User-Agent: claude-code/x.y.z) to
a dedicated deployment by setting:
tag_regex:
- "^User-Agent: claude-code\\/"
in the deployment's litellm_params. Works alongside existing `tags` routing;
exact tag match takes precedence over regex match. Unmatched requests fall
through to deployments tagged `default`.
The matched deployment, pattern, and user_agent are recorded in
`metadata["tag_routing"]` so they flow through to SpendLogs automatically.
* fix(tag_regex): address backwards-compat, metadata overwrite, and warning noise
Three issues from code review:
1. Backwards-compat: `has_tag_filter` was widened to activate on any non-empty
User-Agent, which would raise ValueError for existing deployments using plain
tags without a `default` fallback. Fix: only activate header-based regex
filtering when at least one candidate deployment has `tag_regex` configured.
2. Metadata overwrite: `metadata["tag_routing"]` was overwritten for every
matching deployment in the loop, leaving inaccurate provenance when multiple
deployments match. Fix: write only for the first match.
3. Warning noise: an invalid regex pattern logged one warning per header string
rather than once per pattern. Fix: compile first (catching re.error once),
then iterate over header strings.
Also adds two new tests covering these cases, and adds docs page for
tag_regex routing with a Claude Code walk-through.
* refactor(tag_regex): remove unnecessary _healthy_list copy
* docs: merge tag_regex section into tag_routing.md, remove standalone page
- Add ## Regex-based tag routing (tag_regex) section to existing
tag_routing.md instead of a separate page
- Remove tag_regex_routing.md standalone doc (odd UX to have a separate
page for a sub-feature)
- Remove proxy/tag_regex_routing from sidebars.js
- Add match_any=False debug warning in tag_based_routing.py when regex
routing fires under strict mode (regex always uses OR semantics)
* fix(tag_regex): address greptile review - security docs, strict-mode enforcement, validation order
- Strengthen security note in tag_routing.md: explicitly state User-Agent
is client-supplied and can be set to any value; frame tag_regex as a
traffic classification hint, not an access-control mechanism
- Move tag_regex startup validation before _add_deployment() so an invalid
pattern never leaves partial router state
- Enforce match_any=False strict-tag policy: when a deployment has both
tags and tag_regex and the strict tag check fails, skip the regex fallback
rather than silently bypassing the operator's intent
- Extract per-deployment match logic into _match_deployment() helper to
keep get_deployments_for_tag() readable
- Add two new tests: strict-mode blocks regex fallback, regex-only
deployment still matches under match_any=False
* fix(ci): apply Black formatting to 14 files and stabilize flaky caplog tests
- Run Black formatter on 14 files that were failing the lint check
- Replace caplog-based assertions in TestAliasConflicts with
unittest.mock.patch on verbose_logger.warning for xdist compatibility
- The caplog fixture can produce empty text in pytest-xdist workers
in certain CI environments, causing flaky test failures
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Lint fixes (check_code_and_doc_quality job):
- Remove unused variable reasoning_effort in gpt_5_transformation.py (F841)
- Remove unused timezone imports in mcp_server rest_endpoints.py and server.py (F401)
- Remove unused ProxyBaseLLMRequestProcessing import in realtime endpoints.py (F401)
- Add BaseRealtimeHTTPConfig to TYPE_CHECKING block in utils.py (F821)
- Add PLR0915 per-file-ignore for mcp_server/rest_endpoints.py in ruff.toml
Test fixes (litellm_mapped_tests_llms job):
- Gemini video cost tests: pass explicit model_info to video_generation_cost()
instead of relying on gemini/veo-3.0-generate-preview being in model_prices JSON
- Anthropic max_tokens tests: mock get_max_tokens() to return expected values
instead of depending on claude-3-5-sonnet-20241022 being in model_prices JSON
- Vertex AI pydantic obj test: update from removed gemini-1.5-pro to gemini-2.5-flash,
update expected request body to use response_json_schema format
- Vertex AI/Bedrock file_content integration tests: update mocks to target
base_llm_http_handler.retrieve_file_content (the new code path via
ProviderConfigManager) instead of the old vertex_ai_files_instance/
bedrock_files_instance paths
Co-authored-by: yuneng-jiang <yuneng-jiang@users.noreply.github.com>
