Reverts #37725. The field existed so SDK callers that cannot read
`x-litellm-model-id` could tell which tier an auto-router picked, and the
framework that motivated it was LangChain. `@langchain/openai` builds
`additional_kwargs` and `response_metadata` from fixed key allowlists and drops
unknown fields at both the chunk top level and inside `delta`, so no
proxy-side placement of a namespaced key can reach a LangChain caller.
The complexity router's existing `return_raw_model_name` already covers that
case: it puts the resolved model in the standard `model` field, which
LangChain does propagate (`model_name` is on its metadata allowlist), and the
proxy honors it on both the streaming and non-streaming paths.
Keeps the unrelated cleanup from #37725 that dropped the redundant
function-local `ProxyBaseLLMRequestProcessing` import shadowing the
module-level one in `async_data_generator`.
`TestModelGroupAliasReachesPreRoutingStrategies` asserted on the marker as a
proof of strategy dispatch; the surviving `response.model == "gemini-flash"`
assertion already proves it.
The breakdown priced reasoning tokens at the flat standard rate while the
total billed them tier-aware, so on flex requests the reasoning sub-cost
header could exceed the whole response cost. Route the breakdown's
reasoning rate through the same tier-aware resolver as the total.
On /v1/messages the response is a TypedDict that can never carry hidden
params, yet the client wrapper still recomputed cost on it, clobbering the
already-correct breakdown with a tier-less, reasoning-less one. Skip the
metadata pass for results that cannot hold hidden params, since apply()
discarded it anyway.
* feat(langfuse): support langfuse_environment as a per-key dynamic callback param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): type the langfuse_environment constructor param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): only pass environment when the SDK client supports it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): drop the request-body metadata test for langfuse_environment
The proxy bans request-body callback params by default (derived from
_supported_callback_params in auth_utils), so the metadata channel this
test asserted is rejected with a 401 on the proxy. The supported channel
is admin-set key/team callback_vars, with LANGFUSE_TRACING_ENVIRONMENT
as the deployment-wide fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(langfuse): validate langfuse_environment, avoid redundant clients, honor it in langfuse_otel
Closes the review gaps on the langfuse_environment param:
- Validate values against Langfuse's environment pattern at save time
(/key/generate, /key/update, /team callback all 400 on e.g. 'Production'
instead of 200-then-silently-dropping every trace server-side) and at
logger init; non-string values are str()-coerced instead of crashing
the SDK's regex check per event.
- Treat empty/whitespace values and values equal to the deployment-wide
LANGFUSE_TRACING_ENVIRONMENT as non-dynamic so an environment-only
override that changes nothing no longer mints a duplicate SDK client
against MAX_LANGFUSE_INITIALIZED_CLIENTS.
- langfuse_otel now reads the per-key/team langfuse_environment from
standard_callback_dynamic_params instead of only the env var.
- Advertise the param on the discovery surfaces: callback_configs.json
(langfuse + langfuse_otel), the dashboard callback registry, and the
/team/{team_id}/callback docstring (schema.d.ts regenerated).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: ruff format langfuse files
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lint): remove duplicate test import, LIT002 dict literal, and mock-echo otel test
- drop redundant in-function import of callback_config_error (F811)
- avoid the `or {}` mutable literal in _set_langfuse_specific_attributes (LIT002)
- rewrite the dynamic-env otel test to observe span.set_attribute output
instead of patching litellm internals (TQ002/TQ008)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`AnthropicCacheControlHook` spends the configured injection points on the first
message list it is shown and drops the message points that matched nothing. That is
right when the messages it sees are the ones going upstream. It is wrong for
/v1/responses: the system prompt lives in `instructions`, which only becomes a system
message once the chat-completion bridge builds one, so a role-targeted point matched
nothing and was thrown away before the message it wanted existed. Injection silently
did nothing across the whole surface.
Hand those points back instead, stamped as judged, when the caller says its message
list is provisional. The stamp is what makes carrying them safe: without it the next
pass re-judges the points against messages this pass has already marked and stands the
whole configuration down. Callers holding the final messages -- /chat/completions and
/v1/messages -- do not raise the signal and keep dropping unmatched points as before.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
An auto_router deployment is a marker, not something a probe can contact, so
`_run_model_health_check` returns `{}` for it and it lands healthy whatever is
behind it. This derives its verdict from the models it actually resolves.
Rules and owners:
- `strategy_router_dependencies` is the single answer to "what does this router
call": tier, default, classifier and embedding names per router kind, aligned
with what init and the request path actually use.
- `_health_check_eligible` is the single probe-eligibility gate, applied to the
requested set and to the pool a router's dependencies are drawn from alike, so
an opted-out deployment cannot re-enter through a router that depends on it.
- `_resolved_deployment_ids` resolves names through `get_model_list`, the same
composition of alias, routing-group and wildcard channels a request uses.
- A dependency reds its router only when *every* deployment behind the name is
known unhealthy. A replica this run never judged, hidden from the caller or
opted out of health checks, can still serve what the dead one drops, so
partial evidence leaves the verdict green. Absent information never reds.
- Verdicts settle over rounds, because a marker never fails a probe of its own
and a parent whose tier is a red router must inherit that fault. Both sweeps
are bounded loops, so a router cycle terminates green.
- Dependency probes are added only on the targeted `/health?model_id=` path the
dashboard uses per deployment, and are dropped from the response.
Resolves LIT-6073
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Vertex Gemini 3.x models route through cost_per_character (the cost_router
token-path gate only matches gemini-2), and its token fallbacks dropped
service_tier, so ON_DEMAND_FLEX responses were still billed at the standard
rate. Pass the tier through the call site and all four fallbacks.
The GA vertex model had no cost map entry, and the realtime cost handler
accepted the router's price-less auto-registered deployment entry for the
session.created model at zero-defaulted rates, so sessions billed 0.0 even
when base_model pointed at the priced preview key. Adds the GA entry at its
published rates and makes the handler fall through zero-defaulted candidates
unless their cost map entry explicitly declares pricing.
