* feat(langfuse): migrate the sdk callback to langfuse v4
Replace the v2 trace()/generation()/span() calls with SDK v4 observations exported over OpenTelemetry, with one isolated tracer provider per Langfuse credential set, a discarding exporter for mock mode, and v4 trace and observation id normalization. Keeps the session-header trace provenance logic from main so each call under a session alias still gets its own trace
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): drop the always-true prompt client check now that v4 get_prompt is non-optional
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): isolate the e2e sync test from cached clients and log the real sdk major
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): type the slack trace-url lookup and drop dead v2 test shims
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(slack): cover the langfuse trace url built from the logger host
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(docker): pin langfuse to the locked 4.15.2 in the pip image
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): hash all-zero trace and observation ids instead of passing them through
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(langfuse): honour caller generation ids and assert v4 OTLP exports in legacy tests
v2 accepted generation(id=...). v4 derives the observation id from the OTel
span id, so the isolated tracer provider now carries an id generator that
hands out the id start_generation asked for through a context variable, and
the callback passes the resolved generation_id metadata into it.
The legacy e2e suite patched httpx.Client.post and compared v2 ingestion
batches; it now patches requests.Session.post, decodes the OTLP protobuf
and compares the exported generation against regenerated fixtures. The
local readback test replaces the removed get_generations() with
api.observations.get_many() and polls Langfuse Cloud instead of sleeping.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): read the sdk version header from package metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): propagate trace_metadata as trace-level attributes in v4
v2 wrote trace(metadata=...) onto the trace object. In v4 the trace only
carries what the observations propagate, so a continuation request with
update_trace_keys=["trace_metadata"] updated the generation's metadata
while the trace kept its stale values. Coerce each entry to the SDK's
string limit and hand it to propagate_attributes(metadata=...).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): propagate interrupts raised during deferred client teardown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): honor ssl_verify=False and SSL_VERIFY on the v4 OTLP exporter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): fall back to the default CA when the configured bundle path is missing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): renew the client when eviction lands before the callback lease
The cache can evict a logger between handing it to the callback and the callback taking its
lease. Such a lease now hands back a fresh client acquired through the same parameters, so that
callback exports through a live tracer provider instead of one teardown already shut down.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): emit litellm_call_id and response_id as generation metadata
v2 put the provider response id inside the generation id. v4 observation ids are 16 hex chars derived from that string, so the ids move to generation metadata to keep generations searchable by response id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): read the response id through a typed protocol
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): do not claim trace root when continuing an existing trace
Langfuse derives a trace's name and I/O from any observation flagged
langfuse.internal.as_root, so a request carrying existing_trace_id
renamed the trace to the generation name and replaced the trace input
and output on every continuation. v2 only updated the keys listed in
update_trace_keys. Continuations now export as plain children of the
remote parent and keep the explicit langfuse.trace.* attributes for the
fields they do want changed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): iterate lease renewal instead of recursing, monkeypatch update_trace_keys flag in tests
The recursive lease fallback tripped tests/code_coverage_tests/recursive_detector.py; the renewal
candidates are now walked with itertools.chain. The six update_trace_keys tests set the litellm
global through pytest monkeypatch so the TQ008 budget stays within its ceiling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): retry raised OTLP exports and honor LANGFUSE_TIMEOUT
The OTLP http exporter only retries 429 and 5xx; a connect or read timeout
propagates and BatchSpanProcessor drops the batch. Wrap the exporter in
RetryingSpanExporter (three backoff retries, as the v2 consumer did) and
build it on every path so the default and private-CA deployments share the
same channel, timeout and retry behaviour
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): sample on a hash of the full trace id and tolerate bad LANGFUSE_SAMPLE_RATE
TraceIdRatioBased reads the low 64 bits of the trace id. litellm trace ids are
UUIDs, whose variant bits sit at the top of that word, so every fractional rate
up to 0.5 dropped all traces. A SHA-256 of the full id gives an unbiased,
deterministic decision. Values outside [0, 1] or non numeric now warn and export
everything instead of raising during callback construction, which surfaced as a
500 on the first request of each worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): put the Langfuse trace link back into Slack alerts
The proxy registers LangfusePromptManagement for callbacks: ["langfuse"], so the alert helper never saw the literal "langfuse" string and returned before looking up the trace id, and the prompt management logger never stored the trace id it got back from log_event_on_langfuse. Recognize LangFuseLogger instances in the callback list, record the returned trace id in the shared service trace id cache, and skip the link when no trace id arrives
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): relock langfuse 4.15.2 and opentelemetry 1.33.1 on current main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(langfuse): mark the deliberate blind except in client teardown for the strict ruff gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): pass the resource attributes mapping straight to Resource.create
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): warn about ignored UPSTREAM_LANGFUSE_* on the shared client init path too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): normalise the OTLP export path so a trailing host slash never yields a double slash
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): nest guardrail and grounding spans under the generation
Langfuse v4 derives the trace name and I/O from every observation marked as_root, and the one with the latest start time wins. Guardrail and grounding spans used to claim root next to the generation, so a post_call guardrail could replace the model's request and response on the trace with its own. Only the generation claims root now; the sibling spans become its children
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): rebuild the cached bundle when mock mode or sample rate changes
The SDK keys resource bundles on the public key alone, so a bundle built with the discarding exporter for LANGFUSE_MOCK, or with an earlier LANGFUSE_SAMPLE_RATE, was handed back to a client that asked for a live exporter or a different rate. Compare both when deciding whether the cached bundle is still valid
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): keep trace_public true when a guardrail span is exported
Langfuse folds langfuse.trace.public across every observation in the trace and reads a missing attribute as false, so a guardrail child span without the flag turned a trace_public: true request private on Langfuse Cloud. Child spans now repeat the generation's value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): emit observations as plain OTel spans, keep the SDK for prompts and auth
The callback now owns an isolated TracerProvider and OTLP exporter and builds generation and child spans with public OpenTelemetry APIs plus the LangfuseOtelSpanAttributes constants. Caller trace ids, generation ids, parent observation ids and historical start and end times are honoured through the OTel id generator, remote SpanContext and explicit span timestamps, so no private Langfuse SDK tracing handle is used any more. The Langfuse client stays only for get_prompt and auth_check
This also resolves the gauntlet findings on the previous draft: fresh traces start from an empty context so caller application spans are never stamped, the Slack trace link is read from the request logging state instead of constructing a logger per alert, a truthy non-mapping trace_metadata is serialized instead of raising, trace_input and trace_output land on the root generation, discarding a cached client is done under the lock, and the prompt cache no longer leaks a task manager because the client cache no longer tears down shared providers
Fixtures under tests/logging_callback_tests lose the SDK-private langfuse.internal.as_root marker; every other exported attribute is unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): hand the SDK client a validated sample rate so an unusable LANGFUSE_SAMPLE_RATE no longer breaks the callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): gate the SDK version before importing the OTel module in prompt management
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): flush every export channel on proxy shutdown and use the callback's host in Slack trace links
The shutdown hook imported litellm.utils.langFuseLogger, a global the callback registry never assigns, so a graceful restart dropped the spans still queued in the batch processors. Shutdown now calls flush_langfuse_tracing, which force-flushes every acquired channel. The Slack alert link falls back to the registered LangFuseLogger's langfuse_host when the request carries no dynamic host, and the export endpoint tests pin that scheme-relative or absolute LANGFUSE_OTEL_TRACES_EXPORT_PATH values stay on the configured host
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): store resolved credentials on LangfusePromptManagement
The Slack alert trace link reads langfuse_host from every registered LangFuseLogger. Prompt management subclasses it without calling the parent constructor, so it never set the attribute and the alerting handler crashed before posting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): flush every export channel concurrently under one shutdown deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): flush export channels on daemon threads so a stuck channel cannot hold up interpreter exit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): own the tracer config and drop the SDK client for prompts and auth
The callback's TracerProvider now sets its sampler, span limits and id generator explicitly so unrelated OTEL_* variables no longer change what Langfuse receives, and trace metadata is written once on the trace instead of folded into the generation, which kept input and output under the attribute cap. Spans are emitted under the langfuse-sdk scope so Langfuse renders them natively, the batch processor queues 100k spans and honors LANGFUSE_FLUSH_AT, and the proxy shutdown flush runs off the event loop with a 10s deadline and logs a miss.
Prompts, auth_check and the project id now go through LangfuseAPI directly with a litellm-owned TTL cache, so no Langfuse() client is built and a host application's client on the same public key is left alone. Dead attributes, the unreachable exporter branch and the export list are cleaned up, and the client-budget eviction behavior is documented.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): export OTLP spans and fetch prompts through litellm's HTTPHandler instead of a private requests session
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): gate the SDK version before importing the tracing module and retire unheld export channels
An installed v2 SDK used to fail inside the langfuse_sdk import and surface as "Langfuse not installed"; the version check now runs first so v2 users get the upgrade message, and only PackageNotFoundError means the package is missing
Export channels are now leased per credential set: acquire adds a holder, LangFuseLogger.stop (called by DynamicLoggingCache on expiry) releases one, and a channel with no holders is flushed and shut down after a 60 s grace, so rotating key or team credentials no longer grows one batch thread per credential set for the life of the process
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): end the generation when a child span fails, take the client slot last, keep prompt cache keys structured
Generation spans now end in a finally block so a bad guardrail or provider entry cannot strand the trace. The logger acquires its export channel and REST client before counting a client slot and releases the channel synchronously if the REST client fails to build, so retries after a bad config do not exhaust the budget. LANGFUSE_TIMEOUT accepts decimals for the REST client like it already did for OTLP export. The prompt cache keys on (name, version, label) so a missing label and the literal label None stay apart
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): claim the cache entry before releasing its slot and channel hold on eviction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): coerce generation names, keep v2 release, timeout and retry defaults, refresh stale prompts off the loop
A non-string metadata generation_name reached the OTLP encoder and took the whole batch down; it is now exported as its text and the exporter drops only the span the encoder rejects. LANGFUSE_RELEASE falls back to the deploy platform's commit variable again, the export deadline is back to the v2 default of 20 s and LANGFUSE_MAX_RETRIES sizes the retry ladder. An expired prompt is served at once while one background thread refreshes it, a re-acquired export channel cancels the pending retire timer, and flush reports delivery rather than a drained queue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert the current Langfuse shutdown flush warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): keep host OTel resource out, carry big metadata ints, tolerate bad flush and TTL env, stamp trace I/O under a parent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): name a malformed prompt cache TTL before the SDK import, keep metadata ints JSON safe, retry every 5xx export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): name the auth check failure, split a 413 export, wire LANGFUSE_DEBUG, stamp error output under a parent, send the ingestion version header
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): honor LANGFUSE_DEBUG on the callbacks path, cap retry backoff, name the auth failure status and body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): cap LANGFUSE_MAX_RETRIES at 1000 so an absurd value cannot stall callback init
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): fold 413 halving into bounded rounds instead of recursion
The code-quality recursive-function gate flagged LangfuseSpanExporter.export. A batch of n spans settles within n.bit_length() halving rounds, so the split is a reduce over a frozen round state with the same posts, logs and results. The TTL gate test now asserts the gate returns without raising
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): truncate a single oversized span like v2 instead of dropping it, no retries on REST auth and project lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): write the metadata truncation marker under a flattened key so Langfuse keeps it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): patch the HTTPHandler export path and sync the metadata fixture and lease registry with the v4 callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): give the 413 split helpers a single explicit return path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): url-encode prompt names and fetch cold prompts without client retries
A cold get_prompt runs inline on the event loop; the generated v4 client's default two retries slept through
Retry-After (up to 60 s per attempt) and held the loop. The wrapper also passed the raw name into
api/public/v2/prompts/{name}, so 'what?' fetched prompt 'what' and folder names left the route. Quote the
name with safe='' like the v4 SDK's own get_prompt and pass max_retries=0 like the projects.get calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): retry a cold prompt miss once and drop upstream headers from prompt errors
A cold prompt fetch makes one immediate second attempt after a 5xx or a
transport failure, as the v2 client did, still with the generated client's
sleeping retries and Retry-After handling off so the event loop never stalls.
A failed fetch raises LangfusePromptError carrying only the status and body,
so the proxy no longer forwards Langfuse's response headers to its client
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): stub the logger in the health auth_check test instead of dialing a closed port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): integration test for OTLP v4 delivery and prompt fetch through a real proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(docker): keep the pip image's langfuse and otel pins on the v2 line its litellm 1.83.0 wheel expects
The image validates the published PyPI artifact, whose langfuse callback still
reads langfuse.version, so the 4.15.2 pin broke that callback. The pins move
together with the next LITELLM_VERSION bump. Also rewords the trace_version
precedence test docstring: v2 carried two version fields, v4 has one per span
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): match provider-prefixed fallback keys for bare model groups
* fix(router): infer the fallback key's provider the way routing does for bare model groups
A bare model group served by a wildcard deployment (claude-sonnet-4-6 routed to anthropic/*) now finds a fallback keyed <provider>/<group>. The provider is inferred through one shared helper, inferred_provider, which the pattern router already used inline, so the fallback lookup and routing agree on the prefix. The lookup only infers a provider when some fallback key ends in /<group>, so alias-style groups never hit the resolver
* fix(router): resolve context window and content policy fallback keys through the shared lookup
---------
Co-authored-by: Jason Dougherty <jasondoc3@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): let callbacks filter the model listing routes per caller
* fix(proxy): offer every listed name to the listing callback, agent groups and deployment lookups included
* fix(proxy): hide aliases of a team model by its public name and offer /model/info lookups the listed name
* fix(proxy): map a malformed model listing filter return to the proxy error contract and document legacy team names
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ssrf): point the blocked-address remediation at litellm_settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ssrf): pin the block message's named section to litellm_settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(compaction): pin litellm.max_budget so leaked proxy budget cannot 401 the child auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): time-windowed team reservation of deployments via model_info.access_windows
Deployments can declare model_info.access_windows, a list of local wall-clock windows (IANA timezone, cross-midnight allowed) that reserve the deployment for the listed team_ids. While a window is active the router drops the deployment for every other request, including no-team and admin requests, on every candidate path (model name, model id, specific_deployment, early-resolve, wildcard, litellm_params.model lookup, fallbacks). If every candidate is reserved the request fails with a 400 naming the window end instead of falling back. Outside a window routing is unchanged and reserved deployments stay visible in /model/info and /v1/models. Malformed windows (bad time, unknown timezone, empty team_ids, start equal to end, offset-aware times) fail proxy startup with a clear error since the proxy router runs with ignore_invalid_deployments=True
Resolves LIT-8308
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover _filter_reserved_deployments directly for coverage gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): keep reservation filtering immutable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): drop strategy markers before reservation filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): answer 503 no_db_connection on management routes when the caller's user read hits a database outage
Under allow_requests_on_db_unavailable, once the caller's key row lapses the
request runs as the restricted fallback identity, and its own user read fails
on the outage. /v2/team/list, /user/list, and /user/filter/ui answered a bare
500 for that; every route now answers the same 503 body auth gives, through
one shared builder consulted by the generic exception handler and by
ui_view_users' own catch-all
* fix(proxy): log the database outage before answering 503 on user search
* chore(proxy): drop the docstrings on the db outage 503 helper and its tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): add conversational matrix across chat, messages and responses
Parameterizes one behavioral contract (reply, stream, cost log, tool call,
tool round trip) across /v1/chat/completions, /v1/messages and /v1/responses,
OpenAI and Anthropic models, and env-ref vs stored-credential auth, with
record/replay fixtures.
Adds general_settings.disable_model_info_refresh so the proxy fronting a
replay fixture does not poll every OpenAI-compatible deployment's /v1/models
in the background, which otherwise leaves unconsumed interactions in the
recorded bundle.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): force the weather tool on the first turn and rename Provider to Deployment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): opt-in litellm_call_id in JSON error bodies
Add general_settings.include_call_id_in_error_body. When true, the value
already on the x-litellm-call-id response header is copied into JSON error
bodies: as error.litellm_call_id on the OpenAI-shaped routes, /v1/messages,
and streaming first-chunk errors, and as a top-level litellm_call_id on
pass-through routes. Off by default, so error bodies stay byte-identical
unless an admin opts in
* chore(proxy): drop helper docstring and restore lazy OpenAPI snapshot
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A stored null or empty list for a nested alerting field is still the value
the proxy serves when the config file leaves alerting_args alone, so the
source is db. Keying off the value rather than its presence reported those
fields as default and hid a stored setting that is genuinely in effect.
Presence in the stored row now decides, with the config file still checked
first so a config-owned key keeps reporting config. Test helpers are typed
and the router test injects a stub rather than patching a class attribute.
When the config file declares alerting_args at all, the resolver hands the
file's dict to every reader and the stored row never reaches one. Reporting
a nested field as "db" because the row happens to carry it told the admin a
value was in effect that the proxy does not serve: a live proxy answered
source "db" for outage_alert_ttl while serving the default.
A config-owned parent now reports the field's own default, and the DB is
consulted only when the file leaves the parent alone.
_nested_setting_source returned "db" whenever the stored row held a value,
without first asking whether the config file declares the same key. For a
config-owned alerting_args field that disagrees with the database, the
endpoint reported source "db" while the proxy actually serves the file's
value and rejects any write to it.
Config ownership is now checked first, matching SettingsStore.source and
the precedence the rest of the resolver applies. The source test set grows
a field that only the database sets, a field only the file sets, and a
stored empty list, so each reported source is discriminating.
The shared settings-store work this branch was stacked on landed on main,
along with source reporting for /config/field/info and /config/list.
Resolved by taking main for the store, the config routes and their tests,
and re-applying only the read surfaces main does not cover:
/alerting/settings, /get/ui_settings and /router/settings.
Peer pods gate the settings poll on general_settings.supported_db_objects,
which validates against SupportedDBObjectType. Without a member for this
name an operator could not opt in, so a configured allowlist left every
pod but the one that served the write on stale settings.
Also types the dashboard's settings payload off the generated schema
instead of Record<string, any>.
A second review pass found two more ways a write through the generic
config endpoint, which validates nothing, could strand the feature.
Dropping the enabled flag from a settings block the proxy had already
applied stopped the poller from reconciling it ever again, so the
callback served the old search tool forever. The poller now yields to
litellm_settings.callbacks only while it has applied nothing itself;
once it owns the callback it keeps reconciling.
A provider list written as a bare string was iterated one character at a
time, so interception matched no real provider - the same failure the
empty list already had. Anything that is not a non-empty list is now
dropped so the handler default applies.
The page also derives its toggle from whether the callback is registered
rather than from a stored flag, because a block can be live with no flag
in it at all, and the toggle is what an admin saves back.
Self-review found four ways the new settings page could take web search
interception down instead of configuring it.
A proxy that activates interception through litellm_settings.callbacks
stores no enabled flag, so the page reported it as off while it was
serving, and saving anything on that page persisted that answer and the
next poll removed the running callback. Reads now resolve the flag from
the callbacks list, and a stored block without an explicit flag no longer
touches the callback list at all.
An empty provider list is the page's own default, but the handler reads
it as "match no provider" rather than falling back to Bedrock, so
enabling the feature without naming a provider switched it on and
intercepted nothing. The empty list is now dropped so the handler default
applies.
The replacement logger is also built before the old one is removed, so a
loop ceiling the handler refuses no longer leaves the proxy with none and
retrying every poll, and a stored "false" string now reads as off rather
than as a truthy string.
Web search interception could only be switched on by editing config.yaml
and restarting the proxy, so an admin had no way to turn it on, choose
which providers it covers, or pick which configured search tool runs the
searches without a redeploy.
Adds GET/PATCH /get|update/websearch_interception_settings backed by a
WebSearchInterceptionSettings model, and an Admin Settings panel that
reads and writes them. Config/database precedence comes from the existing
settings store, so a key the config file declares is still refused here.
The stored settings apply to a running proxy: the DB poll rebuilds the
WebSearchInterceptionLogger, removing the old instance before adding the
new one, because two instances with different params hash differently in
the callback dedup key and the first to short-circuit would win. A proxy
that activates interception the existing way, through
litellm_settings.callbacks with no stored params, is left untouched.
Emit data: [DONE] after event: response.failed, and after a late failure
when a terminal event already went out, so OpenAI SDK clients see the
same stream end as a completed response. Restore the lazy OpenAPI
snapshot to its Python 3.12 rendering, which is what CI regenerates.
RateLimitError and InternalServerError now carry the provider body, so
the OpenAI exception mapper keeps upstream codes like cyber_policy and
the upstream message instead of a generic mapped one
The proxy's response.failed event prefers the upstream body's code,
message, and type over the mapped exception's, and numeric error codes
in an error event map to their own HTTP status
* feat(proxy): limit repeated failed Admin UI sign-in attempts
The Admin UI sign-in endpoints accept an unbounded number of password attempts.
All three call authenticate_user, and none of them keeps any record of how many
times a given caller has already been refused, so a misbehaving or misconfigured
client can retry indefinitely at full speed.
A LoginThrottle is now a required argument to authenticate_user, so the accounting
lives at the one function all three endpoints share and a fourth endpoint cannot be
added without deciding what to pass. Failures are counted per username and source
address over a fixed window and further attempts are refused with 429 and a
Retry-After header. The check runs before the database lookup and before the
password comparison, so a refused caller does no further work.
Only genuine credential rejections count. Configuration errors do not, a refused
attempt does not extend the window, and a successful sign-in clears the bucket.
The username is case folded because the user lookup is case insensitive, so casing
cannot multiply the allowance. Both credential rejections now return one identical
message. SSO is unaffected; it never calls this function.
max_failed_login_attempts (10) and failed_login_window_seconds (900) are read from
config.yaml, with LITELLM_DISABLE_LOGIN_RATE_LIMIT to turn the accounting off. They
are deliberately not database backed, so editing YAML always wins and an operator
refused by a bad value can recover.
* test(proxy): clear failed-login TTLs when resetting the throttle between tests
* feat(proxy): harden Admin UI login throttling
* docs(proxy): clarify login throttle configuration
* fix(proxy): keep the login throttle inside the type budget and fail safe on secret errors
* fix(proxy): warn about per-worker sign-in counters without a module global
* fix: honor environment login throttle settings
* fix: satisfy login setting type checks
* refactor(proxy): keep authenticate_user within the C901 budget after merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): write failed-login counters and their expiry in one Redis call
Use RedisCache.async_increment_with_floor (a single Lua INCRBY + EXPIRE) for the
shared login counters instead of the two-step INCRBYFLOAT then EXPIRE, so a
counter can never be committed to Redis without its expiry. The repair in
_remaining_window now only covers expiries stripped out of band (PERSIST, a
restore) and uses the same atomic call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve the login rate limit kill switch once per process
Reading LITELLM_DISABLE_LOGIN_RATE_LIMIT through get_secret_bool on every
unauthenticated sign-in attempt meant a hosted secret manager in read mode
was queried once per password guess, before any counter was checked
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count failed sign-ins in Redis alone while it answers
Every worker spends one shared budget and a successful sign-in clears it for all
of them. This worker's own counter is only consulted while Redis raises, so an
outage degrades to per-worker accounting instead of switching the control off
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep counting the failed sign-ins Redis missed once it answers again
A guess is recorded in exactly one place, Redis or this worker's own store when Redis
refused it, so the count is the sum of the two. Redis is read through
async_batch_get_counts, which raises on failure, instead of async_get_cache, which
swallows it into None and read as an empty counter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): read the shared sign-in counter through a tuple of keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(caching): spell out the key collections batch_get_counts accepts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): warn about per-worker login counters even without general_settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): inject settings and Redis cache into LoginThrottle.from_request
Removes the runtime import of proxy_server from login_throttle so the
throttle module no longer participates in the import cycle CodeQL
flagged (py/cyclic-import). Callers pass general_settings and
redis_usage_cache explicitly; behavior is unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop the section banner comment from the login tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): assert separate login counter stores in the spray regression test instead of a comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): throttle failed Admin UI sign-ins per source and source/username
Replace the username-global lockout with counters keyed by source address and by
source/username pair. Each has a fixed counting window (60s) and a separate
block TTL (300s). Blocks are soft: a correct password still signs in, wrong
passwords from a blocked key take one of 5 held slots per worker and are held
30s before a 429. Once a pair is blocked its failures stop counting against the
source. The source scope runs only when trusted_proxy_ranges is set, IPv6 is
grouped by /64, and per-source limits accept IP and CIDR overrides with
longest-prefix matching. Redis is authoritative through one Lua script per
failure, with bounded per-worker fallback when Redis raises.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): explain the internal patches in the login throttle tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): key held sign-in attempts on the source while the source is blocked
An active source block now takes precedence over a pair block, so every blocked username behind one blocked source shares the source's five held slots instead of getting five each
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply source overrides to IPv4-mapped IPv6 sign-in sources
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): catch only Redis failures when falling back to local login counters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type the login throttle's local store and pass frozen Redis script arguments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the sign-in hold pool from refusing a correct password
The held-attempt cap ran before the password check, so five parked wrong
guesses from a blocked source turned the soft block into a lockout for the
real user. The slot is now taken only after a wrong password, and the
pool-full refusal carries the block's remaining time as Retry-After
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): stub DATABASE_URL in the hold-pool regression test so it passes off the dev box
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): drop a comment that restated the NUM_WORKERS assignment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): match IPv4-mapped IPv6 peers against IPv4 trusted proxy ranges
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep mapped-notation trusted proxy ranges matching mapped peers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): hard-block throttled Admin UI sign-ins with no credential bypass
A blocked source, or source and username pair, is now refused with 429 before the database lookup and password check, in place of the soft block that held wrong guesses for 30 seconds and let a correct password through. The env admin credentials and the master key typed into the login form are refused like any other credential while blocked; recovery is the master key as an API bearer token, which never goes through the sign-in path
trusted_proxy_ranges: [] now means clients connect directly, so the peer address is the source and the per-source limit stays on. Only an unset or malformed value leaves the topology unknown, warns at startup and turns the per-source limit off
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): move login throttle sentinels into constants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): derive the per-username sign-in allowance from the address limit
The per-address-and-username allowance is now half the effective address allowance, rounded up, instead of a separate max_failed_login_attempts_per_user setting. A per-address override therefore raises or effectively removes both limits for that address, and no second override table is needed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): raise the sign-in block explicitly and type the empty settings mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): round the per-username sign-in allowance down and exempt an address with an override of 0
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): import LoginThrottle under TYPE_CHECKING for the throttle helper annotation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): break ties between equivalent login limit overrides deterministically
Two spellings of one network share a prefix length, so the exemption wins the tie, then the higher limit, regardless of mapping order
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): treat a malformed trusted_proxy_ranges entry as an undeclared topology
A list with an entry that is not an address or CIDR range no longer switches
the per-source Admin UI sign-in limit on against the direct peer address, so a
typo cannot make a shared ingress address the bucket for every user behind it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject blank trusted_proxy_ranges entries before they are dropped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
POST /config/update compared litellm_settings after lowercasing the
callback list, so a config file spelling a callback in mixed case refused
the same list sent back, and it stored every general_settings model default
next to the keys the request set. Both now use the request as sent; only
the stored callback list is lowercased.
Also drops config_data from the router settings reload callers the previous
commit left behind and teaches the legacy MockProxyConfig the ownership
check.
GET /model/info?litellm_model_id= went through _get_proxy_model_info, a copy of _enrich_model_info_with_litellm_data that missed the unpriced check, so the same deployment read as null in the list and as 0 on the id lookup. The id lookup now delegates to the shared helper, so the two paths cannot drift again
POST /config/update stored keys the config file owns and answered 200
while the file silently kept winning. Run the config-owned check for
general_settings, litellm_settings, and router_settings before the first
database read, the way /config/field/update already does, so a refused
request stores nothing.
The config reload re-loaded the merged settings as yaml settings, which
turned every saved router setting read-only after one tick. Read the
saved router settings row instead so database-owned values stay writable.
The integration harness seeds num_retries through /config/update instead
of the config file, which is what the effective-settings and observed
routing tests need to keep exercising a database-owned value.
A model_info blob that carries key next to pricing fields is a copy of a /model/info response (only litellm.get_model_info emits key), so those pricing fields are dropped when the row is loaded from the DB and on every Reload Price Data, and the deployment follows the current cost map again. Prices typed into litellm_params, or into model_info without key, stay as they are.
/model/info, /v1/model/info and /v2/model/info now report model_info.pricing_overrides, the pricing fields the deployment sets itself, and the Admin UI model page says whether a price follows the cost map or overrides it.
The per-address-and-username allowance is now half the effective address allowance, rounded up, instead of a separate max_failed_login_attempts_per_user setting. A per-address override therefore raises or effectively removes both limits for that address, and no second override table is needed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
_apply_general_settings_side_effects grew a fourth argument when the reload
started comparing the resolved pass-through list, and this dispatch test calls
it positionally, so it failed with a TypeError.
