* fix(bedrock): price claude_platform calls from the anthropic cost map rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: avoid recursive Claude Platform lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): price claude_platform calls from their own cost map rows
Replace the claude_platform -> anthropic lookup redirect with
bedrock/claude_platform/* rows, and apply the inference_geo multiplier
in the bedrock cost path so US-only inference is priced at 1.1x
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(bedrock): exempt claude_platform rows from the Converse routing guard
The claude_platform/ route is explicit in the row key, so these rows are
not accidental non-Converse Bedrock chat rows
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(anthropic): stop injecting empty-content placeholder into tool-call-only assistant turns
_sanitize_empty_text_content replaced empty or whitespace-only text with
"[System: Empty message content sanitised to satisfy protocol]" for every user and
assistant turn. anthropic_messages_pt calls it unconditionally, independent of
modify_params, because an empty text block always makes Anthropic 400.
An assistant turn carrying tool_calls and no text is the normal shape an OpenAI-format
agent emits, and Anthropic accepts an assistant turn whose content is just tool_use, so
there was nothing to substitute there. Substituting anyway put a protocol diagnostic in
the content field, which is the assistant's utterance as far as the model and the client
are concerned, so it got persisted into conversation history and echoed back on later
turns.
Empty text is now dropped rather than replaced when the assistant turn already carries
substantive content (tool_calls, or non-text blocks such as tool_use). Turns that would
otherwise be left with no content at all keep the placeholder, since Anthropic rejects
those too.
Fixes#24498
* fix(anthropic): only drop empty text when a sibling block survives conversion
_carries_non_text_content treated any non-text block as substantive, but the assistant
list-content converter only keeps thinking (when signable), server_tool_use and
*_tool_result blocks. For an assistant turn holding empty text next to a block it
discards, such as an image_url or a signature-less thinking block, dropping the empty
text left nothing behind and the whole turn disappeared from the request.
The predicate now mirrors what the conversion actually keeps, so those turns retain the
placeholder and survive. This also improves on the previous behaviour, where such a turn
was dropped outright.
* test(anthropic): cover malformed content blocks in the survivor predicate
_survives_assistant_conversion had a separate non-dict guard that no test reached,
since _is_empty_text_block already rejects non-dict blocks before the predicate runs
on them in practice. Folding the check into the type lookup keeps the same behaviour
for bare strings and blocks without a type key, with no unreachable branch.
Extends the sibling-block test with a block missing its type key.
* test(anthropic): set modify_params through monkeypatch in sanitization tests
The test-quality gate flags direct writes to litellm module globals. Swap the
class setup/teardown for an autouse monkeypatch fixture, drop writes that just
repeat its default, and route the False cases through monkeypatch
---------
Co-authored-by: tbartik <22518863+tbartik@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(vector_stores): put azure vector store id before api-version query
Azure OpenAI's get_complete_url returns a url ending in ?api-version=...,
and the vector store handlers appended the id with an f-string, producing
.../openai/vector_stores?api-version=X/vs_abc and a 404 from Azure.
Add BaseVectorStoreConfig.get_vector_store_url, defaulting to the previous
behaviour so other providers are unchanged, and override it in
AzureOpenAIVectorStoreConfig to keep the query string last. The six
retrieve/update/delete handlers now call it.
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(vector_stores): trim comments on get_vector_store_url
Drops the inline comment in the Azure override and shortens the base docstring, per AGENTS.md
Co-authored-by: Keith-Hardaway <276775292+hardawaykwork@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vector_stores): capture the Azure retrieve request without a mutable url list
Greptile on #48224: the handler test now follows the file's received list pattern with Final bindings
Co-authored-by: Keith-Hardaway <276775292+hardawaykwork@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Keith-Hardaway <keith.hardaway@docusign.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Keith-Hardaway <276775292+hardawaykwork@users.noreply.github.com>
* fix(router): keep latency routing alive when latency samples are unusable
latency-based-routing ranks deployments by the average of the cached samples
and keeps everything within lowest_latency_buffer times the lowest. A negative
or NaN sample breaks that invariant, because the minimum no longer satisfies
its own bound. The candidate list then comes back empty and random.choice
raises IndexError, which turns every request to the group into a 500 until the
cache entry expires.
Samples come from completion_start_time - start_time, and that difference is
negative whenever the two timestamps invert. Production recorded samples
around -600s from successful calls.
Drop non-finite and negative samples before ranking, and fall back to every
candidate if the window is empty anyway, for example with a negative buffer.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* test(router): seed latency_normalized in the unusable TTFT fallback test
Main reads latency_normalized since #47727, so seeding latency left both deployments ranked at zero and the test picked at random.
Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): assert ranked latency after dropping unusable TTFT samples
Greptile on #48216: the new tests passed with the sample filter removed because the empty-window fallback alone saved them. They now assert _latency_per_deployment so unfiltered negative or NaN samples fail deterministically. Also adds Final to ttft_samples and drops a comment that restated the code
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): drop Final on a loop-local in latency ranking
basedpyright rejects Final on a variable assigned inside a loop, and the type_check_gate counts it as a new reportGeneralTypeIssues error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): type the request kwargs in the unusable TTFT tests
Greptile on #48216: bare dict annotations. The tests now hold the metadata dict in a typed local and assert on it directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(typing): replace pass-through Any with object in 13 annotations
* docs(client): say dict[str, object] in the models get docstring
* refactor(typing): drop the unused Span aliases in base_cache and levo
* refactor(typing): take a Mapping in delete_nested_value and cover the fresh copy
* fix(typing): deep-copy a read-only Mapping in delete_nested_value through a dict
* fix(proxy): buffer tool discoveries in redis and let the leader write LiteLLM_ToolTable in one statement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the redis family on the tool discovery buffer and drop the blind except and cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): satisfy LIT010 and LIT014 in the tool discovery buffer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the aggregation fold Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): requeue drained tool rows to redis when the write provably did not apply
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drain tool discoveries on their own lock and scheduler job, not the spend commit
* chore(proxy): suppress the untyped apscheduler add_job on the tool discovery job
* refactor(proxy): move the LiteLLM_ToolTable bulk upsert into its own module next to the other bulk upserts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): name the LiteLLM_ToolTable bulk write span so the prisma span check resolves it
* refactor(proxy): move the tool discovery drain out of DBSpendUpdateWriter into tool_registry_writer
* refactor(proxy): share db_write_failure_is_requeue_safe from exception_handler so the spend writer stays untouched
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add OpenCode Go model discovery, protocol routing, session affinity, and published pricing metadata. Cover Chat Completions, Responses, Messages, and Gemini through the existing provider adapters.
Fixes#31568
* feat(ui): configure the default Usage time range
* fix(ui): retain the selected calendar preset during input sync
* fix(ui): preserve fallback dates when applying across DST
* fix(router): route decisions calls only to deployments whose provider supports decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): account for indirect route helper coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): pin the chat leg so the mixed-group check is deterministic
Weights pick the Bedrock leg for chat instead of relying on simple-shuffle landing on both legs. The route filter now passes the candidate list through unchanged in type, so the pre-existing selector lines lose their pyright ignores
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1)
* refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2)
* refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts
* fix: restore pass-through behavior on paths the typing pass tightened
MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites
* fix(runwayml): keep a malformed 2xx task body on the 500 path
* fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report
EdenAI's embedding and image generation parse dropped a non-object body from the error
they raised, so the message no longer named what the provider sent. A helper now raises
with the body in the message, and both paths have a regression test.
`lite encryption migrate` exited 0 with no message when the proxy's report was not an
object or its residual_legacy was not a number, where base crashed. It now fails with a
message naming the value.
The CompactifAI and Gemini image-edit non-object body tests accept TypeError or
AttributeError, which both base and this branch raise, so they guard against a
ValidationError instead of pinning one class. RunwayML's async parse gets the same
non-object body test as the sync one.
* fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed
* test: annotate the new regression tests' locals with Final
* test: annotate the remaining new test locals with Final
* feat: add OpenAI 12-hour cache write pricing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: omit empty cache-write response fields
* style: format cache-write fields
* fix(cost): bill openai 30m and 12h cache writes from the usage split, add gpt-5.6 rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): apply the 12h long-context write rate only above the token threshold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): key cache write rates by TTL instead of growing the rate tuple
_get_token_base_cost returned a positional 6-tuple and every cache lifetime added a slot to it, to the off-peak helper and to every caller. It now returns a TokenBaseCosts dataclass whose cache write rates live in one mapping keyed by TTL. The TTLs come from the ephemeral_*_input_tokens fields on CacheCreationTokenDetails, so supporting a new lifetime only means adding that usage field and its rate key
A lifetime with no rate of its own now always falls back to the cache creation rate after off-peak pricing is applied. Before this the non-tiered 12h rate and the tiered 1h rate fell back to the rate from before off-peak, so the same request billed differently depending on the pricing path
* refactor(cost): hardcode the cache write TTLs as a Literal instead of parsing usage field names
Parsing ephemeral_*_input_tokens names saved nothing, since a new TTL still needs a usage field and provider parsing code. The TTLs are now a CacheWriteTTL Literal with an explicit rate key per TTL
* refactor(cost): resolve cache write lifetime rates to floats at lookup
Decide the 1h and 12h cost map key in the threshold loop, then resolve each
rate once after off-peak with the base write rate as the default, so no None
flows through the cost code. Add an integration case for a 12h write on a
row without a 12h rate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): name every cache write TTL, marking 5m and 30m as base rate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): list every cache write TTL in one rate key table, None meaning the base write rate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): drop base rate entries from the cache write rate table and rely on fallback
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): key cache write rates by plain string TTL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): fill missing cache write TTL rates with the base write rate after off-peak
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): default missing cache write TTL rates to the base write rate inline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): rename cache write rate key table to make clear it only covers long TTLs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cost): list every cache write TTL and bill writes of an unknown TTL at the base write rate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): resolve tiered cache write TTL rates after off-peak instead of passing None
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): carry cache write TTL rates on TokenRates and drop TokenBaseCosts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): define TokenRates before its first use to drop the string annotation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cost): build the TokenRates TTL default with default_factory and type tiers as Mapping
MappingProxyType({}) as a dataclass default is rejected on Python 3.11, where
mappingproxy is unhashable, so import litellm failed there. The tier type
moves from dict[str, object] to Mapping[str, object] (LIT001), with tier_rate
accepting a Mapping to match.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): type select_tier_for_input tiers as Mapping so tier_rate callers stay typed
The tier returned as a bare dict made every tier_rate call site a partially
unknown argument once tier_rate took Mapping[str, object]. The range bounds are
now validated with a TypeAdapter instead of indexing an unknown list.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(responses): honor per-call custom pricing on the native Responses API path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): narrow the custom pricing regression test to the type gates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(main): expose register_custom_pricing_for_request for the responses path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): mark custom pricing test locals Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): price per-call custom rates request-locally instead of registering them globally
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): bind per-call rates on the request logging object instead of inferring them in Logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep Router deployments on the cost-map pricing path
Router inherits cache rates onto the deployment's cost-map entry, so a routed
Responses call binds no request-local price and keeps billing cached tokens at
the inherited rate. Moves the deployment-id lookup to router_utils so the
Responses and completion paths share it.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): gate request-local pricing on the typed deployment model_info
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep image requests off deployments declared supports_vision false in pre-call checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): type the vision pre-call helpers so the basedpyright gate holds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep the pre-call index set typed without a mutable annotation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover the vision pre-call helpers directly for the router coverage check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): type the vision pre-call test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover the no-params and unresolvable-provider vision branches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): await the per-request rollup enqueues and report their failures
The daily and entity spend rollups ran in an unreferenced asyncio task that
nothing awaited or drained, so a shutdown after the spend log was enqueued
could drop every rollup for that request, and a malformed metadata payload
dropped the request from all daily tables at debug level. The rollups now run
inline in update_database, each helper failure is logged through
spend_log_error with the request id, and unreadable metadata only costs the
row its metadata-derived fields
Resolves LIT-6027
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): type the rollup failure helpers for the lint gates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): track the rollup enqueue task so shutdown drains it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): shutdown drain waits for an in-flight rollup enqueue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): register the rollup task without blocking the cost callback
Awaiting the rollups inline made a blocked queue put stall increment_spend_counters, so budget checks could run on stale counters. The task is now only registered via spawn_logging_task; drain_logging_tasks still waits for it on shutdown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(aws_polly): read the engine from the bare model name
speech() strips the aws_polly/ prefix before AWSPollyTextToSpeechConfig sees
the model, so _extract_engine_from_model got a bare "standard", found no
slash and returned the neural default. Every Polly request went out with
Engine: neural whatever the model named.
Take the last path segment instead, which covers both the bare name and the
prefixed one.
* test(aws_polly): move imports to the top of the file
* docs(aws_polly): update the engine comment to the bare model name
* feat(proxy): add LITELLM_OFFLINE_MODE to stop every request the proxy makes on its own
* fix(ui): name LITELLM_OFFLINE_MODE where the cost map panel says local mode is forced
* test: isolate the offline-mode tokenizer tests from the local Hugging Face cache
* test: cover the offline policy templates and the cached Kimi K3 tokenizer
* test(integration): count Kimi K3 tokens in the offline-mode egress test
* refactor(token_counter): skip non-bundled tokenizers in offline mode instead of listing downloaded ones
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(chat): port Moyai UI with Model Hub and images
* feat(chat): add Models sidebar with embedded Model Hub
* feat(chat): polish Model Hub with provider cards and logos
* refactor(chat): use a provider dropdown in Model Hub
* feat(chat): keep workspace switching visible with a Chat identity
* Add minimal opt-in Chat usage metrics and prioritize Chat in workspace selector
* fix(chat): enable usage metrics by default only when chat is used
* fix(chat): count requesting users with frontend-only metrics
* feat(chat): send default-on request metrics to litellm-tel
The router plants its deployment fields in litellm_metadata for Interactions
calls, while the proxy put the request's metadata (spend_logs_metadata, tags,
key fields) under metadata. The spend log resolves litellm_metadata and only
carries user_api_key* fields over, so spend_logs_metadata never reached
LiteLLM_SpendLogs. Route /interactions like /responses so both land in
litellm_metadata.
Fixes#47841
Co-authored-by: Igor Tavares <2163472+igorborgest@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(ui): sort every teams table data column
* fix(ui): order unlimited team limits as infinity and verify org scope
* fix(ui): validate team sort filters without recursion
* feat(ui): sort every users table data column
* fix(ui): order unlimited user budgets as infinity and verify org scope
* fix(ui): cover user sort queries and document pagination bound
* fix(ui): preserve Aria usage token breakdown and viewer model access
* fix(ui): make Aria conversations easier to read and reply to
* fix(ui): wrap long headings in assistant tables
* feat(ui): simplify Aria and add usage charts with PDF answer export
* fix(ui): retain full chart values in Aria PDF exports
* fix(deps): upgrade vulnerable Python dependencies
* fix(deps): lock patched Python dependency versions
* fix(otel): preserve JSON export with the updated SDK
* test(otel): cover JSON export with the updated SDK
* test(otel): update SDK log records and worker discovery
* fix(ui): align Lens peer dependency with patched Next.js
* fix(otel): hand the AgentOps exporter a session it owns for the JWT header
OpenTelemetry 1.45 moved the OTLP HTTP exporter's requests.Session behind a
transport, so writing the minted JWT to the exporter's private _session raised,
was swallowed, and every AgentOps batch went out unauthenticated. The exporter
now creates the session, passes it through the public session= argument and
sets the Authorization header on it
The tests now drive a real export, serving the token endpoint with respx and
capturing the OTLP POST with requests_mock, instead of reading SDK internals
* fix(otel): skip FastAPI instrumentation once the app has started
opentelemetry-instrumentation-fastapi 0.66b1 wraps build_middleware_stack instead
of calling add_middleware, so instrumenting an app whose middleware stack is
already built no longer raises. The call silently did nothing while we still
registered the global response propagator. Check the built stack up front and
skip both, which is what the existing test expects
* test(otel): assert exported requests and span flags instead of SDK internals
OpenTelemetry 1.45 dropped the exporter's private _headers and sets the W3C
random-trace-id flag on new traces, so traceresponse ends in -03. The Langtrace
and Langfuse tests now export through requests_mock and check the URL and headers
the collector receives, and the traceresponse tests expect the server span's own
trace flags
* fix(deps): bump dashboard zod to 4.6.6 and make the vendored Lens use it
zod 4.6.5 is flagged by the dependency scan and blocks osv-scan on every PR. A
root bump alone leaves the vendored Lens tarball, which pins zod 4.6.5 exactly,
on its own nested copy, so a top-level zod override points every consumer at the
root version and the lock carries a single zod 4.6.6
* test(ui): stub the tag summary fetch the client panel retry refreshes
Retrying the model breakdown refetches /tag/summary, which this integration test
left on the real network. When that request failed before the assertion, the
panel switched to its summary error and unmounted the model row, so the test
passed or failed by timing
* fix(deps): drop the duplicated Lens override from the dashboard package.json
* style(ui): format the client panel fetch stub
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* feat(ui): consolidate gateway and usage assistants as Aria Agent
* refactor(ui): give Aria an original blue sprite mascot
* feat(ui): prompt shared Aria from usage and retire duplicate usage agent
* fix(ui): keep Aria naming consistent and reachable across dashboard
* feat(ui): open Aria in a floating popup with suggested questions
* fix(ui): simplify the shared Aria Agent title
* feat(proxy): add LITELLM_DISABLE_LATEST_RELEASE_CHECK to stop the release check
The dashboard asks the proxy for the latest LiteLLM release, and the proxy fetches it from api.github.com on every cache miss. Deployments that block or audit outbound traffic had no way to turn that off. With LITELLM_DISABLE_LATEST_RELEASE_CHECK=true the endpoint returns null without making a request, which the dashboard already treats as no upgrade banner.
* fix(proxy): regenerate dashboard types for the release check flag
The endpoint docstring feeds the OpenAPI description, so the ui-api-types
check needs schema.d.ts regenerated. The docstring now says the flag skips
every outbound request, which stays true once the changelog feed is read
too
---------
Co-authored-by: Sujit Nirmal <nirmalsujit891@gmail.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
The Admin UI upgrade notice read GitHub's latest release, which the release
pipeline sets on its own, so admins were offered releases the changelog had
not announced yet and sent to the auto-generated GitHub release page. The
notice now reads the newest stable release from the docs changelog feed and
links to its changelog page. GitHub release notes are still used for the
feature and fix counts of that version's minor line.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(interactions): assert the live status enum is parsable instead of pinning it
* test(interactions): read the spec from a vendored snapshot and canary the live one daily
* test(interactions): run the canary on PRs touching its inputs, file an issue on a red run, pin the regenerate command
* test(interactions): take the live spec URL as a pytest option, assert its source, split the canary's tracking-issue job
* test(interactions): file the canary issue whenever its compliance step did not pass, match the issue title exactly, cover UrlContextResult
Update certifi, Rust trust roots, and the Docker CA bundle to remove emSign ECC Root CA - G3
Require the fixed certifi release in litellm-core and retain libssl3 in Docker runtimes for Prisma compatibility
Co-authored-by: shin-berri <shin-laptop@berri.ai>
* fix(responses): round-trip Anthropic compaction blocks as compaction items
/v1/responses on Anthropic routes now returns Claude's compaction block as a
compaction output item (and streams it as output_item.added/done), and a
compaction input item is replayed to Claude as its native compaction block,
so the next turn reuses the summary instead of compacting again. Other routes
receive the summary as assistant text. The Anthropic stream handler now keeps
the summary text that arrives in compaction_delta.
Fixes#41456
* fix(responses): replay compaction blocks natively only when the request compacts
* fix(responses): carry compaction through dict-form edits, SDK replay, affinity, and guardrails
Read Anthropic's dict-form context_management, accept SDK ResponseCompactionItem
objects on replay, strip the encrypted_content affinity wrap before decoding,
scan and rewrite compaction summaries in Responses guardrails, drop the history
a summary replay replaces, send the summary as a user message so thinking
ordering holds, and send the streamed compaction .done with the newest block.
* fix(responses): count, route, guard, and redact compaction items consistently
A compaction item now reads as its summary everywhere the proxy counts or routes
a Responses request: pre-call context checks, the TPM limiter, the input token
endpoint, the stream usage estimate, and the previous_response_id session rebuild
all drop the history the item replaces. A Claude compaction item sent to a
non-Anthropic model (including a fallback) becomes a user summary message instead
of an encrypted blob the provider rejects, and Claude on Bedrock Claude Platform
takes the summary path. Output guardrails see the summary of a returned
compaction item, and logging redaction hides it.
* fix(responses): start previous_response_id replay at a stored compaction and count reservations from it
* fix(responses): keep a stored compaction summary whole in spend logs
* fix(responses): keep stored compaction items summary-only, redact compaction_delta, and finish compaction before reasoning
A spend-log row now stores a top-level compaction input item as its summary alone, so extra keys in a
client-built item no longer reach the database whole. Message redaction also nulls the streamed
compaction_delta, and the streaming bridge sends the compaction item's done event before the reasoning
item that follows it. Adds the audit's integration tests for compaction items on the wire.
* test(responses): cover stored compaction items, compaction_delta redaction, and compaction done order
Adds integration cells for the summary-only spend row of a caller-sent compaction item and its previous_response_id replay, a redacted chat stream row that holds no streamed compaction_delta, and a compaction then thinking stream whose done events follow output order.
* fix(spend_logs): honor the redaction header on failure rows for litellm_metadata routes
A failed /v1/responses or /v1/messages call keeps its proxy request headers in
litellm_metadata, but the failure row read redaction headers from metadata only,
so x-litellm-enable-message-redaction never redacted those rows.
* fix(spend_logs): lift only the redaction opt-in header into failure rows
The failure hook on litellm_metadata routes copied every real request header into the caller's own metadata dict, so the stored request body gained the headers, a caller's metadata.headers was replaced, and a disable header loosened global redaction on failed calls. It now copies only the enable headers into a new litellm_params metadata dict, and only when the caller sent no metadata.headers, so no failure row is less redacted than on main
* test(responses): cover failure-row redaction headers and caller metadata on litellm_metadata routes
* fix(responses): keep earlier-turn instructions after compaction and count compaction items by their summary
A previous_response_id rebuild that starts at a compaction now keeps the system and developer
messages of every earlier turn. Token estimates for raw Responses input count a compaction item
as its summary text instead of the serialized item.
* fix(responses): drop the stored history a caller-sent compaction item replaces on previous_response_id
* fix(responses): count only message items with a string role as compaction instructions
* test(responses): flatten the last two-for comprehension in the compaction wire test
* fix(responses): replay from a compaction in stored Responses output and SDK compaction items
A previous_response_id whose stored row is a Responses output holding a compaction item now starts its replay at that compaction, and the handler dumps SDK compaction objects before checking for a summary. Adds tests for the uncovered compaction paths.
* fix(responses): keep the stored tool call a function_call_output answers when a compaction item replaces history
* fix(responses): keep a stored tool call on every turn after a compaction item and bound the stored summary
A previous_response_id rebuild that starts at a turn holding a compaction item now sends again each stored call a
replayed tool output answers, so a retry or a later turn no longer loses the turn-1 call. The start is picked only from
rows up to the referenced response. A spend row keeps a compaction summary up to the larger of 64K chars and
MAX_STRING_LENGTH_PROMPT_IN_DB instead of with no bound.
* fix(responses): replay a compacted session only through the referenced response
A later sibling turn that sent a compaction item put its summary between a stored tool call and its tool output, so Claude answered 400 on a turn that continued the first of two retried turns, and on a retry sent without the item
* fix(spend-logs): keep main's stored value for a prompt length limit under 2
At a MAX_STRING_LENGTH_PROMPT_IN_DB of 0 or 1 the shared truncation helper stored only the truncation marker, where
main stores the marker followed by the whole value
* fix(spend-logs): keep main's stored tail for a negative prompt length limit
At a negative MAX_STRING_LENGTH_PROMPT_IN_DB the shared truncation helper stored one more character of each value than main, since it dropped main's clamp of the end slice to the limit minus the start
* fix(responses): keep a compaction item when a guardrail rewrites its summary through structured_messages
* test(responses): run the compaction wire tests on the worker database under isolated workers
* fix(router): keep the encrypted content affinity pin out of provider requests
* fix(proxy): inspect compaction summaries in guardrails, strip a client-sent affinity pin, and price failures from the newest compaction
* fix(proxy): count a failed chat or /v1/messages call on its whole history, not from a compaction-typed message
* fix(proxy): trim a failed call's input at the newest compaction only when the client called /v1/responses
* fix(proxy): trim a failed call's input at the newest compaction when the proxy dispatched it as a Responses call
* test(proxy): run the dispatched-call-type failure test through the real request setup
* fix(proxy): count a failed Cursor stream's input from the newest compaction item
The Cursor route maps to no call type, so the stream generator's failure hook fell
back to the route and counted the whole history. Pass the processor's route_type.
* fix(proxy): count a broken Cursor stream's prompt from the newest compaction item
* fix(responses): keep a tool call a later output answers when trimming before a compaction item
* fix(responses): move a compaction-split tool call's turn to right before its output on both compaction routes
* perf(responses): look up moved turn items in a set so a long compaction-split turn trims in linear time
* test(responses): measure the compaction-split turn move as a CodSpeed benchmark instead of a clock read in unit tests
* fix(responses): move only calls still open at the compaction, and bill a disconnected stream from the newest compaction item
* fix(responses): restore only the newest stored call when a reused call_id is answered after a compaction item
* fix(spend logs): bound a Responses compaction item's summary like a chat block instead of cutting its encrypted_content
* test(proxy): price a failed dispatched call through a real ProxyLogging instead of mocked hooks
* fix(responses): finish a stream whose Claude compaction failed and group answered call ids by turn once
* fix(responses): count input_tokens after a compaction item the way inference sends it
* fix(llms): sanitize trailing slashes in provider get_complete_url
* test(llms): move trailing-slash url tests into each provider's mapped unit file
* fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI
* fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders
* fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests
* fix(gigachat): strip a trailing slash from the pass-through api_base
* fix(llms): join api_base and route with one shared helper
A deployment api_base that ends in / made several get_complete_url builders emit a double slash, which Envoy and other gateways answer with 404. Add join_api_base in url_utils and use it in those builders, with a regression test that drives every get_complete_url under litellm/llms with a trailing-slash api_base
* refactor(llms): move per-provider api_base strips onto join_api_base
Use the shared join_api_base helper in the search, chat, embedding, image variation, GigaChat and VolcEngine URL builders instead of a per-provider rstrip and suffix check, so one helper owns the single-slash join
* ci: run the trailing-slash URL builder test in the llms shard
* test(integration): cover trailing-slash api_base across provider URL builders
Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell
* fix(llms): send every provider route through join_api_base
Every place under litellm/llms and litellm/main.py that joined a caller-settable api_base with a route by concatenation or f-string now goes through join_api_base, so an api_base with a trailing slash sends the same upstream path as one without it
Routes made only of a caller-supplied id, model, or passthrough path stay on a plain join over an already normalized base, because the helper's suffix dedup would turn a lookup by id into a list or create call
The sweep test now also reaches the factory-made JSON provider builders, and a new loopback test drives the public entrypoints with an api_base ending in /gw/ and checks that every request path keeps /gw/ with no double slash
* fix(llms): keep the route skip only where a builder already had it
join_api_base is a pure join again. The new ensure_api_route skips the route only when the URL path
already ends in it, and it is used only where the merge base or the trailing-slash test PR head
already skipped. Evals cancel, Predibase, and Azure AI threads append their route again, EdenAI
model listing and Gemini RAG ingestion join through join_api_base, and Azure AI Anthropic checks the
path so a host named anthropic-* builds /anthropic/v1/messages
* fix(azure_ai): read /anthropic from the URL path when building the count_tokens URL
* test(llms): run the Gemini RAG trailing-slash case through aingest
The sync ingest wrapper needs a current event loop, which an earlier async test in the same worker clears
* test(llms): cover the Azure realtime, Bedrock rerank, and oobabooga joins with a trailing-slash api_base
* test(integration): cover api_base joins across providers, query strings and named hosts
* feat(url_utils): add join_api_base and ensure_api_route helpers
* fix(url_utils): keep the scheme slashes when a stripped suffix was the whole host
join_api_base and ensure_api_route now keep the // of a base left with only a scheme,
so http://v1 with /v1 stripped joins to http://v1/files as before. The Azure v1 client
base_url goes through ensure_api_route so a base ending in /openai/v1 no longer doubles it
* fix(azure_ai): build the count_tokens URL from the messages URL
* fix(azure_ai): match anthropic as a whole path segment in the messages URL
* fix(llms): keep main's Azure AI anthropic match and non-string Bedrock endpoints
Azure AI chat and messages go back to cutting the base at the first
/anthropic, as on main, and count_tokens keeps its path check, so a
gateway path like /anthropic-proxy builds the same URLs as main. A base
that already ends in /v1/messages still gets count_tokens next to it.
A non-string aws_bedrock_runtime_endpoint (an httpx.URL VPC endpoint) is
used as its text again in AgentCore, Bedrock responses and the RAGFlow
base, instead of falling back to the default host.
get_volcengine_base_url and GigaChat's get_api_base strip a trailing
slash again for outside callers. The cache-hit wire test is skipped as a
known proxy bug, since one cache hit writes two cache_hit spend rows.
* fix(llms): drop a slash before the api_base query and tighten the route test checks
* test(llms): annotate new test locals as Final and flatten the builder generator
* fix(url_utils): keep a scheme-only api_base hostless so it fails before sending
---------
Co-authored-by: Raushankumar0720 <raushan.singh.cg@gmail.com>
A guardrail hook plants a litellm_metadata key on a chat completions request after the proxy
stamped litellm_received_at into metadata. get_litellm_metadata_from_kwargs prefers that key
when both exist and only copied the user_api_key_* fields across, so the response timing
headers lost their receive anchor and x-litellm-overhead-duration-ms reported 2 ms on
/v1/chat/completions against 58 ms on /v1/responses for the same proxy work. The receive stamp
now rides along with the user_api_key_* fields.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
