A pre-existing logic bug in ``_check_banned_params``: when the
deployment-level ``configurable_clientside_auth_params`` permitted one
banned field, the loop ``return``-ed on the first match instead of
``continue``-ing, so any other banned param later in the same body or
metadata dict was never checked. This PR's metadata walk multiplies the
surface where that bypass matters — a body pairing an allowed
``api_base`` with an observability credential like ``langfuse_host``
would silently pass.
Proxy-wide ``allow_client_side_credentials`` keeps ``return`` (it's a
global opt-in for every banned param). The per-param branch becomes
``continue`` so only the one explicitly-permitted field is skipped.
Adds a regression test that exercises the api_base + langfuse_host pair.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related gaps in the proxy's request bouncer:
1. ``is_request_body_safe`` (auth_utils.py) walked the request-body root
and the ``litellm_embedding_config`` nested dict, but not ``metadata``
or ``litellm_metadata``. The same fields it bans at root — Langfuse /
Langsmith / Arize / PostHog / Braintrust / Phoenix / W&B Weave / GCS /
Humanloop / Lunary credentials and routing — were silently accepted
when the caller put them inside metadata, retargeting observability
callbacks to a caller-controlled host with caller-supplied creds.
Walk both metadata containers (and parse the JSON-string form sent via
multipart / ``extra_body``) through the same banned-params helper, so
the existing ``allow_client_side_credentials`` opt-in covers both
paths consistently.
2. The banned-params list was hand-maintained and lagged the canonical
``_supported_callback_params`` allow-list in
``initialize_dynamic_callback_params``. Derive the observability bans
from that allow-list (minus a small ``_SAFE_CLIENT_CALLBACK_PARAMS``
set for informational fields like ``langfuse_prompt_version`` and
``langsmith_sampling_rate``) so future integrations are covered
automatically; ``_EXTRA_BANNED_OBSERVABILITY_PARAMS`` carries the
handful of fields integrations read but the allow-list hasn't caught
up to. A guard test fails CI if a new entry is added to
``_supported_callback_params`` without an explicit safe-list decision.
Separately in ``litellm_pre_call_utils.py``: add ``callbacks``,
``service_callback``, ``logger_fn``, and ``litellm_disabled_callbacks``
to ``_UNTRUSTED_ROOT_CONTROL_FIELDS``. The first three are appended to
worker-wide ``litellm.{input,success,failure,_async_*,service}_callback``
lists / ``litellm.user_logger_fn`` from inside ``function_setup`` — one
request poisons every subsequent caller in that worker. The last is the
inverse primitive: the legitimate path reads it from key/team metadata,
the request-body version silently disables admin-configured audit /
observability for the call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Setting reasoning_effort="none" on Anthropic chat models (direct, Bedrock
Invoke, Bedrock Converse, Vertex AI Anthropic, Azure AI Anthropic) crashed
LiteLLM with:
litellm.APIConnectionError: 'NoneType' object has no attribute 'get'
Both the Anthropic chat transformation and Bedrock Converse called
``AnthropicConfig._map_reasoning_effort`` and assigned the ``None`` it returns
for ``"none"`` directly to ``optional_params["thinking"]``. Downstream
``is_thinking_enabled`` then did ``optional_params["thinking"].get("type")``
and crashed.
Pop ``thinking`` (and on Claude 4.6/4.7, ``output_config``) instead of
assigning ``None``, restoring the documented contract that
``reasoning_effort="none"`` means "do not enable thinking". This also
prevents downstream Anthropic 400s ("thinking: Input should be an object",
"output_config.effort: Input should be ...") if the bug were ever masked.
Verified end-to-end against the live Anthropic API and Bedrock Converse
on claude-opus-4-{5,6,7} and claude-sonnet-4-6, plus Bedrock Invoke for
Claude 4.5/4.6. Vertex AI Anthropic and Azure AI Anthropic inherit the
fixed ``map_openai_params`` from ``AnthropicConfig`` and need no further
changes.
Adds litellm.disable_vertex_batch_output_transformation (default False).
When True, afile_content returns raw Vertex predictions.jsonl untouched
so users that parse candidates/modelVersion directly are not broken.
The method was overwriting logging_obj.optional_params, logging_obj.model,
and logging_obj.start_time on the caller's Logging instance. When invoked
from llm_http_handler.py's generic framework path, the framework's own
logging_obj (which already went through pre_call) had its properties
clobbered, causing model and start_time to reflect the last batch line's
values rather than the original call context.
Fix: create a fresh local Logging instance for the per-line transformation
instead of mutating the incoming logging_obj. The caller's object is now
left entirely untouched regardless of whether a logging_obj was passed in
or not.
Regression tests added to verify model, start_time, and optional_params
are not mutated on the caller's logging_obj.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
- Remove unused _encode_gcp_label_value / _decode_gcp_label_value singular
helpers; only the _chunks variants are actually called.
- Use 'is not None' check for custom_id so empty-string custom_ids are
still labeled and round-trip through batch outputs.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Newer Cloudflare Workers AI models (e.g. Nemotron) emit 'response_text'
instead of 'response' on streamed chunks. The non-streaming path was
already updated to fall back to 'response_text' (#26385), but the
streaming chunk parser still only read 'response', which caused
streaming requests against those models to silently produce empty
content.
Mirror the non-streaming fallback in CloudflareChatResponseIterator.chunk_parser
and add a streaming test for the response_text shape.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
base_anthropic_messages_test.test_anthropic_messages_with_thinking and
test_anthropic_streaming_with_thinking still pinned to
claude-4-sonnet-20250514 — the same legacy alias Anthropic no longer
recognizes under freshly issued keys. The other four tests in this base
class already use claude-sonnet-4-5-20250929; these two were missed.
Bump to claude-haiku-4-5-20251001 (supports_reasoning=true, no upcoming
deprecation). Subclasses including TestAnthropicPassthroughBasic
inherit these methods.
test_anthropic_messages_streaming_cost_injection hits the proxy's
/v1/messages route, which routes via the anthropic/* wildcard to
api.anthropic.com. The 404 surfaced in the test was Anthropic's own
not_found_error propagated back through the proxy (visible from the
x-litellm-model-id hash on the response — the proxy did route).
Same root cause as the prior commit: the legacy claude-4-sonnet-20250514
alias is no longer recognized by Anthropic's main API under the new key.
Swap to claude-haiku-4-5-20251001 — same routing path, canonical model.
Three live-API tests pinned to claude-4-sonnet-20250514, which is a
non-canonical alias of claude-sonnet-4-20250514. Anthropic's main API
no longer resolves the legacy form under freshly issued keys, so the
tests fail with not_found_error. The token counter test pinned to
claude-sonnet-4-20250514 itself (deprecation_date 2026-05-14, two weeks
out) was on borrowed time too.
Bump all four to claude-haiku-4-5-20251001 — capability superset for what
these tests exercise (streaming, parallel tool calling, extended thinking,
token counting), no upcoming deprecation, cheaper per-token.
Mirrors the membership rule on /key/update so that /key/generate and
/key/{key}/regenerate apply the same `_validate_caller_can_assign_key_org`
gate when the caller specifies an `organization_id`. Proxy admins bypass.
The check no-ops when `organization_id` is not being set.
Conflict resolution for #26968 dropped the `Iterator` typing import
(NameError at module load), left a dead `fallback_models = cast(...)`
block, and the new tests called `_enforce_key_and_fallback_model_access`
without the now-required `request` kwarg.
* fix(proxy): re-validate user_id ownership after /user/info re-parses query
The route-level access check in `RouteChecks.non_proxy_admin_allowed_routes_check`
reads `request.query_params.get("user_id")`, which decodes literal `+` to
spaces. The endpoint then re-parses the raw query string with `urllib.unquote`
in `get_user_id_from_request` to preserve `+` characters (so plus-addressed
emails work as user_ids). Those two paths produce different ids: a caller
who registered a user_id containing a literal space could pass the route
check and then read another user's row by sending the encoded `+` form.
Add `_enforce_user_info_access` and call it after `_normalize_user_info_user_id`
returns the final id. Proxy admin / view-only admin still bypass; everyone
else must match the resolved user_id (or have no user_id, which falls back
to the caller's own id later in the handler).
Tests cover the admin bypass, owner-match path, and the cross-user lookup
that this change blocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(proxy): apply user_info ownership check to PROXY_ADMIN_VIEW_ONLY
`_enforce_user_info_access` was bypassing both PROXY_ADMIN and
PROXY_ADMIN_VIEW_ONLY, but the upstream route check in
`RouteChecks.non_proxy_admin_allowed_routes_check` only treats
PROXY_ADMIN as a true admin for the `/user/info` route — view-only
admins go through the `user_id == valid_token.user_id` enforcement
along with regular users. Mirroring that asymmetry left the same
encoded-`+` bypass open for view-only admins whose user_id contains a
literal space.
Drop the PROXY_ADMIN_VIEW_ONLY exemption so the post-decode re-check
matches the upstream rule. Update tests: a view-only admin must now
be blocked from cross-user lookups but still allowed to read their
own row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(batches): count non-chat tokens and validate every model in batch file
Two security control bypasses on POST /v1/batches:
1. `_get_batch_job_input_file_usage` only summed tokens for
`body.messages` (chat completions). Embedding (`input`) and text
completion (`prompt`) batches reported zero, letting massive
non-chat workloads slip past TPM rate limits. Extend the counter
to handle string and list shapes for both fields.
2. The batch input file was forwarded to the upstream provider
without inspecting the models named inside the JSONL — only the
outer `model` query parameter was checked against the caller's
allowlist. A caller restricted to gpt-3.5 could submit a batch
targeting gpt-4o and the upstream would execute it under the
proxy's shared API key.
Add `_get_models_from_batch_input_file_content` (returns the
distinct `body.model` values) and call it from
`_enforce_batch_file_model_access` in the pre-call hook, which runs
each model through `can_key_call_model` so the same allowlist
semantics (wildcards, access groups, all-proxy-models, team aliases)
the proxy enforces on `/chat/completions` apply here too. Any
unauthorized model raises a 403 before the file is forwarded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(batches): count pre-tokenized prompt/input shapes, classify 403 logs
Two follow-ups from the Greptile review on the batch validation PR:
1. P1 TPM bypass via integer token arrays. The OpenAI batch schema
accepts ``prompt`` and ``input`` as ``list[int]`` (a single
pre-tokenized prompt) or ``list[list[int]]`` (multiple) in addition
to the string and ``list[str]`` shapes. Pre-fix only the string
shapes were counted, so a caller could submit a batch with hundreds
of millions of pre-tokenized tokens and the rate limiter would
record zero. Extract the per-field logic into
``_count_prompt_or_input_tokens`` and count each int as one token.
2. P2 access-denial logs were indistinguishable from I/O failures.
``count_input_file_usage`` caught every exception under a generic
"Error counting input file usage" message, so an intentional 403
from ``_enforce_batch_file_model_access`` looked the same in the
logs as a missing file or a Prisma timeout. Catch ``HTTPException``
separately and log 403s at WARNING level with a security-relevant
message before re-raising.
Tests cover the new shapes: single ``list[int]``, ``list[list[int]]``
(the worst-case bypass vector), and embeddings ``input`` with
pre-tokenized arrays.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reset the policy ID index during policy engine test cleanup so stale policy versions cannot leak between tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
