proxy_admin_viewer is mapped to the effective role Admin so it gets read parity with proxy admin, and the write restriction travels separately on isViewOnly. This page derived write access from the role alone, so after that mapping landed a read-only admin saw Add Callback and the per-row Edit scope and Delete actions on admin-owned destinations.
Verified live: the viewer session now renders no Add Callback and keeps only the config-callback Test menu, while a proxy admin keeps all eleven row menus.
Adds the destinations table, the scoped-destination add flow, the Edit scope dialog and the Not active badge to Logging and Alerts.
Squashed onto the rebased disclosure branch; the previous history conflicted with staging's changes to the same dashboard files.
Adds resolved_logging_exporters to /team/info and /organization/info, naming every destination whose access scope grants that identity and which actually builds.
Squashed onto the rebased export branch; the previous history conflicted with staging's Final-annotation pass.
The router treats every entry in _known_custom_logger_compatible_callbacks as a
prompt-management provider when it appears before the / of a model string, so
adding "generic" as a callback name silently reserved the generic/ prefix
product-wide: a pre-existing deployment with model: generic/<x> stopped routing
and answered 500 'Prompt ID is not set', with the OTEL v2 flag off.
Split the two uses of that list. The callback list keeps "generic" so
callbacks: ["generic"] still initializes and exports; the router's prefix test
reads a separate set that excludes callback names which are ordinary words.
Verified base-vs-head on a live proxy with the flag unset: generic/<x> now
answers 400 'no healthy deployments', byte-identical to base, while
callbacks: ["generic"] still delivers the gen-AI span to its exporter.
Never-rebound names carry Final, deliberate rebinds carry rebind-ok with the reason, and the endpoint suffix check collapses into one endswith call. Two hook parameters and the constructor kwargs move from Any to object, which is both stricter and what the bodies actually rely on.
The remaining constructions keep a mutable-ok naming what needs a concrete dict: pydantic model_copy payloads, the OTel SDK Resource call, and the LRU caches that are mutable by design.
Fans each request's trace out to the destinations its identity resolved, applies per-request credentials only to the exporter their own backend contributed, and caches tracer providers per credential, endpoint and transport.
Squashed onto the current staging tip; the previous history conflicted with staging's Final-annotation pass over the same OTEL v2 modules.
Same ratchet as the credential-model branch: never-rebound names carry Final, the request-state mirror logs at debug instead of swallowing silently, and the empty header and resource fallbacks share one immutable mapping.
Two constructions keep a mutable-ok with a reason: the dedup accumulator consumed on the next line, and the TypedDict rows the resolver returns.
Publishing the destination ContextVar imports the OTel context module, and opentelemetry ships only in the proxy-runtime extra. That call sat outside the resolver's try, so a litellm[proxy] install raised ModuleNotFoundError on every request over a feature it never enabled; the flag being off did not help, since the gate lives inside the resolver and an empty result still fell through.
The helper now returns before publishing when the flag is off, and a missing package with the flag on warns instead of raising, matching what instrument_fastapi_app already did.
The org fallback loads the team, which on a cache miss is a Prisma read on the authentication path. It cannot change which destinations are selected unless some access names an org, so it now runs only then; the logging credentials are also parsed once instead of twice.
The type-discipline refactor inserted _destination_verdict between the
@router.get decorator and get_credentials, so FastAPI registered the helper
as the listing endpoint. It declares a CredentialItem parameter, which FastAPI
reads as a required request body, and every GET /credentials returned 422.
Add the GET routing regression test alongside the existing PATCH one; the
other tests call get_credentials directly and stay green while the route is
dead.
The stack was rebased onto a base whose ceilings had since been ratcheted down, so code that fit before no longer did. Never-rebound names carry Final, mapping payloads are built immutably through MappingProxyType, the four duplicated access-rejection bodies collapse into one helper, and the listing's optional verdict key moves into a named function.
Four constructions keep a mutable-ok with a reason: each is handed to FastAPI or an OTLP exporter, which need a concrete dict.
It had no consumer: a team is bound to a destination by credential_info.access. It was
allowlisted as a team callback var but never added to all_litellm_params, so it escaped the
absorber into the outbound provider payload. Also corrects a test that asserted
partial-PATCH credential_info loss was intended; the loss is real but pre-existing, so it
is documented rather than codified.
An `arize` credential carrying only `otel_endpoint` is built by the generic passthrough but
resolved under the `arize` name, so the router applied Arize's intrinsic gRPC transport to
the plain HTTP URL the admin typed. The destination delivered nothing while `/team/info`
kept advertising it as active, which is the invariant `resolved_logging_exporter_names`
exists to hold. The passthrough now pins `otlp_http`, which is what it has always meant;
adapter-built Arize destinations keep their gRPC default, covered by its own test.
`GET /credentials` now reports `resolves_to_destination` for logging credentials, computed
by `destination_for_credential` so it cannot drift from the resolver or the disclosure. The
dashboard's Scope column needs this verdict; recomputing the adapter rules there would drift
from them, and that drift is what let a dead destination read as live.
Both regression tests were mutation-checked by reverting their own fix in isolation.
APScheduler anchors an interval job at now + interval, so every scheduled
background job registered in one proxy startup shares a single firing instant
for the life of the process, and every replica a rollout brought up together
shares that instant too. Each tick the spend flushes, budget reset sweep,
config-in-DB reload, credential reload and cost pollers all hit Postgres at the
same moment, on every pod, competing with request-path auth and budget queries.
Shift each eligible job by a deterministic offset derived from
sha256(job_id, identity), where identity covers the pod and the worker process.
The offset lives in the trigger rather than in a one-off next_run_time, because
a cron trigger recomputes each fire from the wall clock and would otherwise snap
straight back onto the shared instant. An interval job is never offset by more
than one of its own periods.
Only schedules LiteLLM chose are shifted: interval jobs always, cron jobs only
when the id is one of the product's own defaults, so an operator-supplied
crontab keeps the instant it asks for. general_settings.scheduled_job_stagger
turns it off, widens the window, replaces the identity, or pins a job. The
applied offsets are logged once at startup and each fire logs its scheduled
instant against its actual start.
Resolves LIT-5433
The outer wrap_sse_stream_with_keepalive_pings layer duplicated the
keepalive engine that PR #34423 already runs inside async_data_generator
for chat completions and responses streams, and it kept pinging
deployments whose operator set keepalive_seconds: 0 as a hard disable.
sse_keepalive_ping_interval_seconds is now the global fallback inside
_resolve_keepalive_seconds, so deployment and request values keep
precedence, an explicit 0 still disables, the [1, 300]s clamp applies,
and router-less proxies arm the wrap when the global default is set.
The Rust messages bridge logs a parsed Anthropic response without an
httpx_response, so the fallback transform dropped the request speed and
billed fast-mode calls at the standard rate. Thread optional_params
speed into transform_parsed_response and add a regression test for the
parsed-response branch.
A stream can terminate with a response.failed event that still reports
consumed tokens; those were rebuilt as None and logged at zero spend.
Parse response.failed alongside completed and incomplete, matching the
buffered path, which prices any terminal response that reports usage.
The Anthropic batch create never persisted the creating key's hashed token or its
request tags on the managed object, so when CheckBatchCost billed the batch hours
later there was nothing to attribute it to. Key spend, key budgets and tag spend
never moved for batch usage.
Persist both from the create, the way the Vertex passthrough already does, and
register the batch only from the collection route. An id-scoped route cannot
rebuild the unified object id, because it embeds the model and the model comes
from the create's request body, so it could only claim a row it did not create or
fail the model_object_id unique constraint.
The shared metadata helpers, the route predicate and the registration-result
logging now live in batch_attribution instead of being copied per provider. The
Anthropic write previously logged success unconditionally, before the
fire-and-forget task had run.
Resolves LIT-5288
LiteLLMCompletionStreamingIterator overrides __init__ without calling
super().__init__(), so completed_response was only set once the stream
reached RESPONSE_COMPLETED. On a mid-stream provider error the router's
_extract_partial_responses_usage read source_iterator.completed_response
during fallback recovery and raised AttributeError, masking the real
provider error (e.g. Anthropic 529) and bypassing configured retries and
fallbacks. Initialize the attribute to None so recovery degrades to no
partial usage instead of crashing.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Streams that terminate with response.incomplete (e.g. max_output_tokens
reached) carry real usage in the terminal event but were rebuilt as None
and logged at zero spend, letting callers bypass budget enforcement.
Parse response.incomplete alongside response.completed when
reconstructing the streamed response.
Addresses review feedback: the ResponseCompletedEvent SSE parsing now lives
in OpenAIResponsesAPIConfig next to the other Responses stream event handling,
and the proxy logging handler calls it. Adds coverage for streams that end
without a response.completed event.
AWS no longer serves `anthropic.claude-3-sonnet-20240229-v1:0`. The streaming
path returns a plain 404, "Model with the provided id
anthropic.claude-3-sonnet-20240229-v1:0 is not found", and the non-streaming
path answers 500 for the same reason. Our own cost map has carried a
2026-07-30 deprecation date for it since #36538
That accounts for 20 failures across local_testing_part1, local_testing_part2
and llm_translation_testing. litellm maps both statuses correctly, so the
tests are what went stale, not the client
Replacement is `us.anthropic.claude-sonnet-4-5-20250929-v1:0`: a like-for-like
Sonnet, and the newest Bedrock Sonnet this repo exercises against the real API
in tests/e2e. Newer ids exist in the cost map, but nothing in the repo calls
them live, so picking one would be an unverified guess about model access on
the CI account
Scope is limited to the tests that actually issue a request. The occurrences
that assert on the model string itself, or that feed mocked transformations,
keep the old id so their assertions stay meaningful
