mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-01 07:03:02 +00:00
144 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
tessellark
|
a084d13635
|
fix(passthrough): log the URL model and declared provider on Gemini and Vertex failures (#49075)
* fix(passthrough): log the URL model and declared provider on Gemini and Vertex failures Pass-through failures on Gemini and Vertex routes carry the model only in the URL, so the failure spend row and the Prometheus failure counter logged an empty model. The failure payload now falls back to the model parsed from the URL on those routes, and the Prometheus provider label honors the provider the request declares before inferring one from the model name * fix(passthrough): keep model and provider on mid-stream and Vertex failure logs Build the failure payload from the URL at both streamed relay sites so a 200 stream that dies mid-body still logs the model, derive vertex_ai for Vertex routes that never declare a provider, and have the Vertex logging handler reuse get_vertex_model_id_from_url instead of its own regex * fix(passthrough): bound the Prometheus provider label and name Vertex failures like the success handler Declared providers only reach the api_provider label when they name a registered provider, otherwise the label falls back to inference. Vertex failure rows now use the success handler's model naming (vertex_ai/ prefix on rawPredict and streamRawPredict, vertex_ai/search_api on :search) and its host-based provider detection, and the detected-stream failure sites are covered by tests. * fix(passthrough): name Google failures from the URL on untracked methods too A Gemini or Vertex pass-through failure on a method the success handler does not track (countTokens, for one) still logged model "" and no provider, although the model sits in the URL. The failure payload now falls back to the Google AI host for the provider and reads /models/<name> from the URL, so countTokens 400s name their model and provider like generateContent ones do. Non-Google hosts keep the tracked-route rule and still log nothing extra. * fix(passthrough): trust the declared Google provider and Vertex-shaped paths on custom bases The failure path only named the Google provider from the URL when the method was one the success handler tracks or the host was a Google API host, so a Gemini countTokens failure behind a custom GEMINI_API_BASE (or a Vertex one behind a custom base) still logged an empty model. A route that declares gemini or vertex_ai now wins outright, and a URL whose path carries /projects/<p>/locations/<l>/ counts as Vertex on any host, while a tracked method on a non-Google host no longer passes as Vertex. * fix(pass_through): stop the URL model parser at the query string so a configured key never lands in a spend row's model * test(pass_through): run the URL-model failure tests through the real ProxyLogging with a recording callback * fix(passthrough): scope declared provider to pass-through failures and match Google hosts only Prometheus trusts the resolved provider as before and reads the request-declared provider only on pass-through failures. Vertex attribution matches Google hosts, strips operation poll suffixes, prefixes vertex_ai/ only for rawPredict, reads the URL path only, covers Vertex Live WebSocket failures, and labels a router alias failure with the alias the client requested * fix(passthrough): attribute failures from the URL only on built-in Google routes and keep other routes' metric labels A user-defined pass-through route to a Google host keeps its empty model and provider, since its URL names an upstream model rather than a LiteLLM-managed one. Prometheus failure labels read the route's provider and model_group only for gemini and vertex_ai, so Azure, OpenRouter, Bedrock and other pass-through routes keep their earlier labels. The Gemini URL parser drops the query string, Google host matching needs a dot or dash boundary, and the Vertex Claude prefix follows alt=sse like the success handler. * fix(passthrough): log the URL model on guardrail blocks and the named model on Vertex Live failures A pre-call guardrail block on a pass-through route now builds its failure payload the same way upstream failures do, so the model and provider read from the URL reach spend logs and metrics. Vertex Live websocket failures (connect errors, upstream rejections, and the credential failure) now carry the model from the ?model= query parameter under the vertex_ai provider * fix(passthrough): mark guardrail-block and Vertex Live credential failures as pass-through calls Prometheus reads a failure's declared provider only on pass-through calls, so without call_type it fell back to guessing from the model name. That labeled a guardrail block on the Gemini route as vertex_ai and the Live credential failure as None * fix(passthrough): let the route name a failure's provider and give guardrail blocks their own spend row * fix(pass-through): keep URL model attribution off routes that name another provider A built-in route that names a non-Google provider, like /openai pointed at Gemini's OpenAI-compatible host, no longer takes its failure model from the Google URL. A guardrail block response now returns the x-litellm-call-id of its spend row. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a44e44b980
|
fix(cache_control): pick the chat-path dialect from the deployment's provider prefix (#45295)
* fix(cache_control): resolve the chat-path dialect with the deployment api_base * test(cache_control): type the chat-path resolver helpers * fix(cache_control): pick the chat-path dialect from the deployment's provider prefix Resolving with the deployment api_base followed the dispatch's azure_ai to azure flip, so chat on a non-Foundry Azure host sent Anthropic cache_control while /v1/messages, told azure_ai, sent the OpenAI breakpoint. Use the model's provider prefix when the caller does not name a provider, so chat matches the told path whatever AZURE_AI_API_BASE holds, and keep name-only resolution everywhere else. * fix(cache_control): keep litellm_proxy routing when the chat path reads the provider prefix With USE_LITELLM_PROXY or litellm.use_litellm_proxy on, every call goes to the upstream proxy, so the provider prefix no longer decides the cache format there and resolution matches main. The plain OpenAI check reads the same provider value as the hosted check, so one function no longer mixes two lookups * fix(cache_control): resolve the chat-path dialect with the deployment api_base, as the call is routed The hosted cache-format check now resolves the provider with the deployment's own api_base when the caller names none, the same provider the call dispatches to and the Responses path passes, so AZURE_AI_API_BASE no longer decides the chat format. The plain OpenAI check keeps main's name-only lookup. * fix(cache_control): send acompletion fallbacks the caller's request, not the primary's hooked kwargs acompletion handed async_completion_with_fallbacks the kwargs the cache seed and prompt hook had already rewritten for the first model, so a Claude fallback behind a Foundry GPT model received prompt_cache_options and got a 400. Fallbacks now get a copy of the caller's model, messages, tools, and kwargs taken before the hooks, as completion already does with its args. Both cache seeds also read base_url before api_base, since completion sends the call to base_url when both are set. * fix(cache_control): keep the caller's tools list intact for acompletion fallbacks The vector-store hook pops recognized file_search tools from the request's tools list in place, so the fallback request built from the pre-hook snapshot lost the retrieved context. Snapshot a copy of the list instead. * fix(cache_control): skip acompletion's prompt hooks when fallbacks will run them Each fallback attempt runs the prompt hooks for its own model, so running them first in the outer call only cost a vector-store search whose result the hand-off threw away. * fix(cache_control): run acompletion's prompt hooks for fallbacks and leave only the cache hook to each attempt --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Filip Sinitsin
|
e81dd348a4
|
fix(proxy)!: log and bill Responses-style bodies sent to /cursor/chat/completions (#30277)
* fix(proxy): restore spend logging for /cursor/chat/completions (#30126) The /cursor/chat/completions route processes requests through the Responses API pipeline, but _get_metadata_variable_name did not treat it as a litellm_metadata route. Proxy auth metadata therefore landed in the user-facing metadata param, so the spend tracking callback saw user_api_key=None and silently skipped the SpendLogs write. Add the route to LITELLM_METADATA_ROUTES and cover it with regression tests. * test(proxy): pin provider metadata and key billing on /cursor/chat/completions A Responses-format body on the Cursor route now sends OpenAI only the metadata the client sent and writes a spend row that bills the key. The chat-body branch, which this fix also moves onto litellm_metadata, still bills the key with its team and request tags. The scripted chat reply gets a unique id per test, since a shared id let only the first spend row per database land * fix(proxy): pick the Cursor route's metadata field from the body shape A Responses-style body on /cursor/chat/completions keeps proxy metadata in litellm_metadata, the same as /v1/responses, while a Chat Completions body keeps metadata, the same as /v1/chat/completions. Adding the whole route to the internal-metadata list let a client's chat-body tags replace the key's tags in spend logs and skip tag budgets. * fix(proxy): keep get_metadata_variable_name callable with only the request * fix(proxy): check client tags against tag budgets and read agent limits from litellm_metadata on Responses bodies A Responses-style body on /cursor/chat/completions now keeps its proxy fields in litellm_metadata, so the pre-call tag budget check and the v3 agent rate limiter stopped seeing the client's metadata.tags and the x-litellm-agent-id value. The pre-auth tag check now merges the caller's metadata.tags into the selected metadata field, sharing one helper with request setup, and the limiter reads agent_id from either field * fix(proxy): check client tags sent in a JSON-string metadata against tag budgets on Responses bodies * fix(proxy): answer 422, not 500, when a budget rejects a Responses body whose metadata is a JSON string * fix(proxy): keep key guardrails, metadata type errors and failure-row client details on Responses-style bodies --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Devon Krisman
|
953750ed41
|
fix(websearch-interception)!: stream native server-tool blocks to clients (#40967)
* fix(websearch-interception): stream native server-tool blocks to clients Streaming native clients (Anthropic SDK, Claude Desktop, Claude Code) never received the server_tool_use and web_search_tool_result blocks the interception layer builds for them, so citation panels stayed empty and result counts read zero. Two gaps in the same flow: FakeAnthropicMessagesStreamIterator._create_content_block_chunks only knew text, thinking, redacted_thinking and tool_use; any other block type emitted a bare content_block_stop with no start and no payload, silently dropping the block from the fake stream. server_tool_use now streams as content_block_start plus input_json_delta and any other server-generated block rides whole in content_block_start, matching Anthropic's native streaming of these types. try_short_circuit_search only recognized a native client by finding a web_search_* tool in the tools list, but the pre-request and deployment hooks convert that tool to the LiteLLM standard shape before the short-circuit runs, recording the conversion in WEBSEARCH_EMIT_NATIVE_BLOCKS_KEY. The flag now counts as native too, so the synthetic response carries the native blocks on the converted path instead of only when hook ordering happens to leave the tool untouched. Fixes #35333 * fix(anthropic): accept server_tool_use and web_search_tool_result blocks when costing a /v1/messages response The short-circuit now returns native web search blocks, and the cost path validated the served body against a content union that only allowed text, tool_use and thinking blocks, so non-streaming requests answered 500 * fix(anthropic): keep every server_tool_use field when re-streaming a /v1/messages reply * fix(anthropic): keep tool_use fields, redacted thinking data and text citations when re-streaming a /v1/messages reply * test(websearch-interception): build the cost test's logging object with a datetime start time * fix(websearch-interception): send search snippets once in the direct search answer * fix(guardrails): mask web search queries, titles and urls on /v1/messages and deliver stream rewrites past server_tool_use blocks Output guardrails now scan the native web search blocks the short-circuit emits, ahead of the reply text so guardrails that read the last text still scan the reply. A guardrail reply of a different length falls back to the reply-only write-back. Ended-stream tool call rewrites land on server_tool_use blocks too, the ended-stream scan key covers the result strings, and legacy-hook pipeline steps pass through the result strings their response view lacks. The wire test reads the streamed server_tool_use input from its input_json_delta. * fix(guardrails): deliver a legacy post-call hook's tool-call rewrites on streams instead of discarding the stream rewrite A legacy hook that masked a tool call's arguments on a buffered stream made the whole rewrite undeliverable, text included, so the original stream went out. With web search blocks in the stream that now hit any hook masking the search query. The adapter hands same-count tool-call rewrites to the translation's write-back, as a unified guardrail's already are, and still discards a rewrite that adds or drops a tool call. * fix(guardrails): show a legacy post-call hook the web search result blocks of a /v1/messages stream The legacy hook's view of a buffered Anthropic stream is rebuilt from the assembled ModelResponse, which keeps web_search_tool_result blocks only in provider-specific fields, so a hook masking result titles on a stream never saw them. The hook's view now carries those blocks, its rewrite of them goes back through the ended-stream write-back, and the adapter no longer needs to pass texts outside the hook's view through untouched. * test(guardrails): type the legacy hook stream test helpers * fix(guardrails): scan a streamed web search query as text like the non-streaming reply * test(guardrails): type the masking guardrail test helpers * fix(guardrails): keep the block id when scanning a streamed web search query --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
norkelbyte
|
e0778bc9dd
|
refactor(types): replace Any with proven types in 21 files (#48921)
* refactor(types): replace Any with proven types in 35 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): keep anthropic chat transformation json path untyped Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): accept null gemini vector store fields and load encryption TypedDicts on python 3.10 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): validate Python client user responses and type delete_user as the count it returns * fix(types): keep loose payloads working under the sweep's validators An off-schema Anthropic stream event is now skipped instead of ending the guardrail's text and stop_reason scan, and MAI image usage accepts the loose numbers it did before. New dict and list adapters hide their input from validation errors, RerankResponse indexing keeps its field types, and the delete_user test mocks the int /user/delete returns. * fix(types): make the changed DynamoAI fields ReadOnly and skip non-object A2A stream items * refactor(types): tighten round 2 parse sites and keep payloads out of errors Parse Anthropic SSE events with validate_json and ignore extra fields, validate admin team and key rows per row with hidden inputs, type rerank get and __getitem__ per key, and move the Phoenix prompt version response to a frozen BaseModel * fix(types): keep MAI image calls alive on unreadable usage counts An unreadable MAI usage count now counts as absent instead of failing a billed image call, and the provider's usage fields pass through as sent. The Anthropic stream scan warnings no longer log raw model output, and the off-schema skip has tests that fail on the old scan. * fix(types): count non-object MAI usage as absent and parse Anthropic SSE with json.loads --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
8d441ebc15
|
test(otel): give the Langfuse destination tests the otel callback's stdout exporter (#49129)
#48718 added TestLangfuseDestinationFields while #48723 made the unconfigured stdout placeholder opt-in (console_default=True, the otel callback's setting). Both merged within a minute, so main's integrations job fails six tests that read the placeholder's exporter. Build those providers with console_default=True so the operator-exporter checks keep their target. |
||
|
paveln83
|
4ce724e3df
|
fix(caching): key cached async clients by a per-loop token instead of id(loop) (#48877)
* fix(caching): key cached async clients by a per-loop token instead of id(loop) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): drop deferred closes queued on a collected event loop Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): pin that a close queued on a live loop waits for that loop * test(caching): run loop-reuse tests until a loop address repeats The collected-loop tests ran a fixed 50 loops and passed with the fix reverted whenever no loop reused an earlier address. They now stop at the first reuse and skip if none happens, through a shared helper. * test(caching): cover the cache key fallback for loops without weak references Also corrects the evicted client closer docstring: a dropped bucket's clients are left to the garbage collector. * fix(http): free short-lived event loops whose handler finalizer close finished as the loop stopped * fix(caching): share sync clients across event loops and catch a failed finalizer close * fix(agentops): stop closing the shared sync http client after the auth call --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
ffcc4f7a70
|
fix(otel)!: print OTel v2 spans to stdout only when the operator asks (#48723)
* fix(otel): stop OTel v2 vendor presets from printing every span to stdout With LITELLM_OTEL_V2 on and a vendor callback such as langfuse_otel or arize holding operator credentials, the proxy printed every span as JSON to stdout, prompts included, on the request path. Nobody set OTEL_EXPORTER, but the console placeholder that OpenTelemetryV2Config folds in for an empty exporter list stayed next to the vendor's own exporter. The provider now skips that placeholder when the config has any other exporter. A console exporter the operator names with OTEL_EXPORTER=console is kept, and so is the placeholder when it is the only exporter. * test(otel): type the console-placeholder tests and drop inherited OTEL_* env from the stdout check * fix(otel): print the console default only for the otel callback The proxy's background V2 logger (published when no callback built one, which is what key-only and team-only Langfuse destinations run under) also kept the unconfigured console exporter, so it printed every span with prompts to stdout. The console default now prints only for a listed otel callback or an exporter the operator named. * fix(otel): keep the otel callback's console output on service-name routes A key or team otel_service_name moves the call onto a provider the tenant tracer cache builds, which never got the console choice, so the plain otel callback stopped printing those calls. The no-operator-keys integration rig also drops inherited Langfuse keys. |
||
|
yucheng-berri
|
1e7da87e1e
|
fix(otel)!: give key/team Langfuse destinations user, tags and session on OTel v2 without global Langfuse keys (#48718)
* fix(otel)!: keep a credential-less langfuse_otel on OTel v2 so key destinations get the Langfuse mapper With LITELLM_OTEL_V2 on, langfuse_otel in callbacks and no operator LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY, the preset raised at startup and the callback fell back to the legacy logger. Key and team destinations were then served by a generic v2 logger without the Langfuse mapper, so their traces lost tags, user and cost, and the legacy logger added an orphaned raw_gen_ai_request span. The preset now always degrades to the credential-gated exporter when the operator has no keys, the same way arize_preset does, so the startup logger stays on v2, exports nowhere for traffic without a destination, and carries the Langfuse mapper to every key and team destination. * test(otel): type the langfuse tenant-only test helpers * fix(otel): give key/team Langfuse destinations the Langfuse fields when langfuse_otel is not a callback With langfuse_otel only in a key's or team's logging, the startup logger is the generic one and its spans reached the key's Langfuse project without user, session or tags. The emitter now adds the Langfuse mapper for a request with a Langfuse destination, and every non-Langfuse exporter strips the Langfuse request metadata, since any logger can now write it. * fix(otel): stop a key's Langfuse logger from re-sending spans to the operator's collector With langfuse_otel only in key or team logging, the proxy publishes the generic V2 logger at startup and it already exports to the operator's OTLP collector. The Langfuse logger a key's request builds kept the same base exporter, so the collector got every model span twice. A logger built after the publish now keeps only its preset's own exporters, as it already did beside a configured V2 logger, and a later call that would rebuild one exporting nowhere reuses it. * test(otel): assert the collector gets one copy instead of reading the key logger's exporters * test(otel): count the collector's spans with one comprehension clause * fix(otel): keep the legacy langfuse_otel logger and stand it down for requests a Langfuse destination carries Reverts the preset degrade and the published-provider exporter trim, so every request without a key or team Langfuse destination keeps main's legacy logger. For a request with one, the V2 fan-out already delivers the whole trace with the Langfuse fields, so the legacy logger emits nothing for it. * test(otel): pin the legacy langfuse_otel paths a tenant's project still gets without a destination * fix(otel): shed a destination's Langfuse request metadata before its other Langfuse fields At its attribute limit a span sheds the Langfuse fields only a destination asked for first, in the mapper's order, so the generation marker went before the request metadata blob. Put the request metadata first, as a Langfuse-configured logger already does. * fix(otel): keep the legacy langfuse_otel span for a destination's call rejected before any provider The OTel v2 fan-out records no call for a request the proxy rejects before a provider (rate limit, pre-call hook), so standing the legacy logger down there lost the user, session and tags main sent the tenant. Stand it down only for a call the fan-out records, using the same check the v2 logger uses. The tenant and collector integration tests now hold their exact counts for a few seconds, so a late duplicate copy fails them. * fix(otel): give a key's Langfuse destination the langfuse_environment the legacy logger wrote The destination's resource carries langfuse.environment from the key's or team's langfuse_environment, else LANGFUSE_TRACING_ENVIRONMENT, so a key's project files its traces under the environment it chose instead of default. Only that destination's export carries it. * fix(otel): keep the Langfuse fields written for a key's Langfuse destination off every other exporter * fix(otel): give a key's Langfuse destination the prompt and response the legacy logger sent, only where it ran * fix(otel): keep a key's Langfuse destination on the global capture setting when it sets none, with or without global Langfuse keys |
||
|
Nate Armstrong
|
ead221995e
|
refactor(router): move router callers onto the entry points and invert the proxy imports (#48847)
Code outside the router now reaches it only through the entry points (registry, settings, health, select, apply, observe, register_plugin, retire) or the endpoint methods; the outside-member allowlist shrinks from 139 to 45. litellm/router.py no longer imports litellm.proxy: it asks the installed RouterHost (litellm/router_host.py) for vertex pass-through credentials, member auto-router authorization and auto-router prompt compression, and the proxy installs its host on import. Call sites where a model name can be missing keep the old answer instead of reaching an entry point with None. |
||
|
Rick
|
c98641fd5a
|
feat(dashscope)!: derive protocol bases from one domain (#47812)
* feat(dashscope): derive protocol bases from one domain Accept the DashScope domain root, the OpenAI-compatible base, the Anthropic Messages base, or a native /api/v1 base and derive every surface from it Adds a native Anthropic Messages config for /apps/anthropic so a single deployment serves chat, messages, embeddings, rerank, image generation, and video against the same credential Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com> * fix(dashscope): keep Anthropic credentials off DashScope messages Require a DashScope key or a caller auth header before the Messages path runs, and opt out of workload identity, so the base class cannot fall back to ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN Also format the image generation base selection * style(dashscope): keep the messages config within the format gate * test(dashscope): cover caller Authorization on messages * fix(dashscope): keep the chat bridge for protocol-named bases and the image default for unconfigured calls * fix(dashscope): keep configured image and rerank bases, normalize blank keys, derive brand chat bases - image calls fall back to the provider's own chat base env (DASHSCOPE_API_BASE, QWENCLOUD_API_BASE, QWEN_AI_PLATFORM_API_BASE) before the image default - an explicit /compatible-api/v1/reranks URL wins over DASHSCOPE_API_BASE_RERANK again - a blank DashScope key counts as unset on /v1/messages, so it never falls back to ANTHROPIC_API_KEY - QwenCloud and Qianwen AI Platform chat and embeddings derive their compatible-mode base from a domain root - tests annotate new variables with Final and drop the config-class assertion * fix(dashscope): keep an explicit compatible-api rerank base as given * fix(dashscope): derive chat from compatible-api bases, keep the caller's rerank base, and let explicit bases beat per-endpoint env vars * fix(dashscope): keep a full DashScope image endpoint URL as given and type the brand tests * fix(dashscope): let the image env var beat any shared DashScope base * fix(dashscope): keep a full DASHSCOPE_API_BASE image URL ahead of the image env var * fix(dashscope): keep a native DashScope endpoint URL as given for chat and the api base header --------- Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tavren27
|
e7d3b24d7c
|
fix(logging): mark mirrored silent_model calls with is_silent_experiment in spend logs (#48888)
* fix(logging): mark mirrored silent_model calls with is_silent_experiment in spend logs and logging payloads * fix(router): keep the mirror flag when litellm_metadata is set and strip a client-sent is_silent_experiment Spend rows read router metadata from litellm_metadata when a call carries it, so the router now sets the flag there too. The proxy drops a client-sent is_silent_experiment with the other router-owned keys, so real traffic cannot be recorded as mirrored. * fix(logging): mark mirrored calls with a router-only key so clients cannot set is_silent_experiment Spend rows and the standard logging payload now derive is_silent_experiment from _litellm_silent_experiment, which only the router writes and the proxy and pass-through routes strip from client metadata. A caller's own is_silent_experiment key stays untouched, so it keeps its current effect on mirroring and on provider metadata, and unmirrored calls log false in both places. * test(langfuse): expect is_silent_experiment false on every logged generation * fix(logging): mark mirrored calls with a router-only enum value so no client route can set is_silent_experiment The router stamps SilentExperimentMarker.MIRRORED, which client JSON can never produce, so batches, Responses WebSocket frames and pass-through need no strip. Langfuse layers the flag after its requester-metadata dedupe. * fix(logging): set is_silent_experiment only on mirrored calls so real calls keep their payloads Real calls no longer gain an is_silent_experiment key in spend rows or the standard logging payload, so OTEL, S3, GCS and every other logger receive what they did before. The Langfuse regression test now runs the real prompt helper. * fix(logging): keep the mirror marker out of callback metadata and read the spend-row flag from the logging payload Logging now strips the router's mirror marker from the request metadata buckets before any callback or vendor sees it, without touching the router's own dicts, so retries keep it. Spend rows read is_silent_experiment from the standard logging payload, which survives the collector's JSON round trip, so collector-mode rows keep the flag too. * fix(logging): mark mirrored calls through a context variable so router usage counts them once * fix(router): leave callback-started calls unmarked and stop client flags from turning mirroring off Logging handlers now run outside the mirrored-call context, so an LLM call a success or failure callback starts while logging a mirrored call is not flagged as mirrored. The guard against mirroring a mirror reads the same context variable instead of request metadata, so a client-sent is_silent_experiment no longer skips the mirrors. The unused module-level get_standard_logging_metadata stops copying resolved keys from raw metadata. * fix(router): keep main's client flag mirroring skip and the other logging metadata keys A client's is_silent_experiment: true skips mirroring again as on main, the module-level get_standard_logging_metadata drops only the flag, and the silent experiment tests wait on a logged-call signal instead of polling a deadline * fix(logging): take the mirror flag in Prometheus labels and Datadog cost tags only from the router * test: type the new parametrized tests and keep the recorder from changing its argument * fix(datadog): keep client request tags from setting the silent experiment cost tag A request tag is_silent_experiment:true set the Datadog cost tag on a direct call. Only the router flag sets it now. Also covers the sync failure handler's mirrored-call reset with a failure function callback test. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Priyansh Nandwana
|
67ae3fb527
|
fix(websearch)!: keep the provider prefix on the agentic follow-up model (#38885)
* fix(websearch): keep the provider prefix on the agentic follow-up model The agentic follow-up rebuilt its model string with "a slash means it is already qualified". Handlers are handed the provider-stripped model, and for a provider that routes through a sub-path the remainder still holds a slash, so bedrock/mantle/anthropic.claude-sonnet-5 arrived as mantle/anthropic.claude-sonnet-5 and stayed that way. The follow-up acompletion could not resolve a provider from it and raised "LLM Provider NOT provided". Prefix unless the model is already qualified with that same provider, in one shared helper used by both the handler and the websearch patch builder, which carried copies of the same heuristic. This also fixes openrouter, whose stripped model keeps a slash for the same reason, and the old startswith test that read openai as a prefix of openai_like. Fixes #38829 * test(websearch): cover the agentic follow-up call site The helper had unit tests but the call site did not, so the two changed lines in _execute_chat_completion_agentic_plan were uncovered. Drive the real follow-up with mock_response instead of patching litellm, which keeps the test on the behaviour: against the old code the sub-path case raises "LLM Provider NOT provided", and the ordinary-model case passes either way as a guard. * refactor(websearch): trim the comments on the provider-qualify helper * refactor(websearch): cut the helper and test docstrings to one line * fix(websearch): leave a hook-qualified follow-up model on its own provider A hook that patches in its own model owns that choice, so re-qualifying it sent a cross-provider follow-up to the original request's provider instead. Only a bare patched model and the request's provider-stripped model take the prefix now, and the sibling agentic-loop path resolves the model the same way. * fix(websearch): fall back to the request model when a patch carries no model An empty patched model dispatched a provider-only name instead of falling back, and a provider-stripped sub-path copied into the patch lost its prefix. A patched model now keeps its own provider only when its leading segment names one. * fix(websearch): keep the provider prefix on the responses follow-up model too The responses patch builder still used the old "a slash means already qualified" test, so a provider-stripped sub-path model reached the follow-up unqualified on that surface. It now resolves the model the same way the chat completion builder does. * fix(websearch): read custom_llm_provider without a member call on untyped kwargs * fix(websearch): type the chat patch builder kwargs instead of working around them * test(websearch): run the real search path and flatten the qualify helper tests * fix(websearch): route a hook's model on a JSON-configured provider and keep the request provider on an echoed model * fix(websearch): leave a follow-up model alone when its prefix already routes to the request provider * fix(websearch): send the agentic follow-up to the model the caller requested completion() and responses() record the caller's model before routing strips it, and the chat and Responses follow-up builders prefer that model. This keeps bedrock/converse_like/, bedrock/claude_platform/ and openrouter/openrouter/* deployments on their own route for the follow-up. * test(websearch): cover the recorded caller model in Headroom's retrieval follow-up * test(websearch): cover the recorded caller model in Compresr's retrieval follow-up Also merges the repeated provider-prefix check in core_helpers into one helper and drops a test docstring * test(websearch): drive the real search path through a mocked Perplexity HTTP route * refactor(websearch): read the provider from kwargs inside _caller_model * test(http-handler): type the agentic follow-up provider tests * refactor(websearch): record agentic loop params through a Logging method Moving the recording out of completion() keeps it under basedpyright's code-flow complexity limit after the merge with main. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a2ef18b257
|
test(unit): keep unix socket paths in unit tests under the macOS AF_UNIX limit (#45040)
* test(proxy): keep the spend sidecar socket path under the macOS AF_UNIX limit * test(unit): share the short socket fixtures across unit tests Move socket_dir and socket_path to tests/unit/conftest.py so the datadog redis test's absent socket fails for the right reason on a Mac, rename the pgbouncer helper's directory argument, and record the rule in tests/unit/AGENTS.md * test(unit): fall back to /tmp when TMPDIR leaves no room for a socket path socket_dir now uses /tmp when TMPDIR plus the longest socket suffix would pass macOS's sun_path limit, and its teardown ignores errors so a failed test reports only its own failure * test(unit): keep socket fixtures inside tmp_path behind a short link socket_dir now links a short temp directory to tmp_path, so artifacts keep pytest's retention and a killed run leaks only an empty link. The fallback to /tmp counts the temp directory in bytes, and socket_tempdir points tempfile at it for the pooler tests whose socket directory production picks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
agustin18
|
3f526a5017
|
fix(websearch): strip Claude Code instructional prefix in short-circuit search (#43458)
* fix(websearch): strip Claude Code instructional prefix in short-circuit search Signed-off-by: agustin18 <agustinsaiz02@gmail.com> * fix(websearch): preserve quotes for unwrapped search queries Signed-off-by: agustin18 <agustinsaiz02@gmail.com> * test(websearch): replace mock with pure unit tests and cover empty queries Signed-off-by: agustin18 <agustinsaiz02@gmail.com> * style(websearch): fix ruff formatting blank line in handler Signed-off-by: agustin18 <agustinsaiz02@gmail.com> * refactor(websearch): enforce Final annotations, prevent rebinding, and retain e2e coverage Signed-off-by: agustin18 <agustinsaiz02@gmail.com> * test(websearch): assert server_tool_use block unconditionally Signed-off-by: agustin18 <agustinsaiz02@gmail.com> * fix(websearch): search only the query inside Claude Code's web search prompt Match Claude Code's exact "Perform a web search for the query:" prefix instead of a broad regex family, keep any quotes the user typed so exact-phrase searches still work, and test the search request at the HTTP boundary. --------- Signed-off-by: agustin18 <agustinsaiz02@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tin-berri
|
54a6a5fb3a
|
fix(otel): suppress spans for excluded proxy requests (#48720) | ||
|
yucheng-berri
|
74392efbd4
|
fix(otel): label configured pass-through routes litellm.passthrough and read the operation from the upstream (#48675) | ||
|
devin-ai-integration[bot]
|
1171273511
|
fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697)
* fix(langfuse_otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): name trace metadata identity keys after their request metadata fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): type metadata helper and drop null fields so they cannot clobber caller trace metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): annotate new request metadata tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): build rig config and wire batches immutably Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): skip empty trace identity values so teamless requests cannot erase attribution Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): drain wire batches per poll instead of accumulating Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): cover null stripping and empty trace identities Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): set redaction flag directly instead of monkeypatching Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): add integration audit matrix for langfuse metadata keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): tighten audit chaos and unauthenticated cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): restore contracts.json formatting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): close SDK clients, pin worker identity, and normalize response id matching Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): gate the slow sink reply instead of sleeping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): emit request metadata and identity under langfuse.observation.metadata and langfuse.trace.metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): satisfy type discipline gate for metadata export Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: drop prohibited covers markers from langfuse otel integration tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): put the mutable-ok suppression on the dict construction line the type-discipline gate reports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse otel): share request metadata export across callbacks * fix(langfuse otel): drop prompt variables, MCP tool I/O and vector store queries from exported request metadata * fix(langfuse_otel): keep span export alive when nested request metadata has non-string keys The content strip validated prompt_management_metadata, mcp_tool_call_metadata and vector_store_request_metadata items as str-keyed mappings, so an SDK caller passing one with a non-string key raised inside exported_request_metadata: OTEL v2 dropped the LLM span for every vendor and v1 langfuse_otel lost the request metadata and every attribute after it. Validate those nested mappings with object keys instead. Also hoists the langfuse_otel imports. * fix(langfuse_otel): send the request metadata blob to Langfuse destinations only - A requester_metadata that is not a mapping no longer raises. The v2 LLM span (every vendor) and the v1 Langfuse attributes survive it again. - Prompt-quoting routing_decision fields stay out of langfuse.observation.metadata. - An exporter or key/team destination for any other backend drops langfuse.observation.metadata and langfuse.trace.metadata.user_api_key_*, as on main. This covers a collector folded into the Langfuse logger. - The v2 Langfuse key-set pin in the arize family test gains the two new keys. * fix(langfuse_otel): export only caller identity under the Langfuse metadata keys - The Langfuse metadata keys carried the whole request metadata. On OTel v2 that sent request headers, caller metadata, user email, user agent and auth metadata to a Langfuse destination that never received them on main; on v1 it doubled each span's metadata payload, so big caller metadata doubled the export size - Both paths now export only the key alias, user, end user, team, team alias and org. On v2 that is the team plus the identity among the operator's `baggage_metadata_keys`, which v2 spans already carry as `litellm.metadata.*`. Everything else stays in the bare `metadata` attribute, as on main - `output_messages` returns a list again, and the arize v2 b3 key pin is back to main's set * fix(langfuse_otel): drop the caller identity first when a span hits its attribute limit v1 wrote the identity attributes after the OpenInference ones, so a span near the 128-attribute limit (about 33+ tools) evicted openinference.span.kind, metadata and llm.model_name instead. The identity now goes first on a fresh span, so the SDK evicts it before anything else, and a caller's active span that already carries attributes does not get it. v2 sheds the Langfuse identity keys before any indexed message attribute when a span is over budget, so a span keeps every attribute it carried without them. * fix(langfuse_otel): keep other spans and dropped counts as main exports them - v1 writes the caller identity only into the room the span's attribute limit leaves next to the error attributes a failed call adds, so the exported dropped-attribute count matches main - v2 strips the Langfuse request metadata only from LiteLLM's own spans; application, instrumentation and server spans keep every key they set - a stripped v2 span keeps the dropped attribute, event and link counts - a config that lists the Langfuse mapper without the langfuse_otel exporter also keeps the request metadata away from other exporters * fix(langfuse_otel): keep the caller identity off the caller's active span v1 writes LiteLLM's attributes onto the caller's active span when one is current instead of starting its own span. That span goes to the caller's own exporter, so the identity keys reached a non-Langfuse collector and could take attribute slots the caller fills later. The identity keys now go only on the span LiteLLM starts, whatever the caller's span carries. * fix(langfuse otel): keep the identity keys on the OTEL v2 Langfuse mapper only The v1 langfuse_otel callback goes back to main: it sends no langfuse.observation.metadata or langfuse.trace.metadata.user_api_key_* keys. The OTEL v2 Langfuse mapper keeps them. The integration cells now run every endpoint, client, redaction, failure and chaos case on v2. * test(langfuse otel): count only accepted batches in the sink outage cell The wire also records the batches the sink answered 503, so an exporter retry made a span look exported twice. The chaos cells now also check the spans that landed carry the caller identity. * test(langfuse otel): split stacked span comprehensions and drop constant comments * test(langfuse otel): split the metadata integration module into focused files * fix(langfuse otel): send the org id in trace metadata too The trace metadata keys now reuse IDENTITY_METADATA_FIELDS, so a key in an org gets langfuse.trace.metadata.user_api_key_org_id next to the generation's org id. --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
788868a52f
|
fix(security)!: build proxy-only failure logs from an allowlist and drop caller guardrail info (#48599)
* fix(proxy): build proxy-only failure logs from an allowlist and ignore request-time GCS credentials * fix(proxy): drop caller metadata from proxy-only failure logs metadata and litellm_metadata leave the proxy-only failure-log allowlist, so a rejected request can no longer point its failure log at its own Langsmith or Langfuse host through nested metadata. * fix(proxy): keep allowlisted metadata in proxy-only failure logs * test(proxy): isolate proxy-only failure log metadata tests * fix(logging): ignore non-list request tags when building the logging payload A guardrail-rejected request with metadata tags set to a non-list value crashed the standard logging payload, so its spend row lost model attribution. * fix(proxy): keep router attribution and drop unauthenticated tags in proxy-only failure logs A request rejected before routing now logs the configured deployment and model group even when its metadata or litellm_metadata carries its own values. Requests rejected at auth no longer write caller tags into failure logs and tag spend. * fix(proxy): always strip client guardrail info and mirror failure attribution into litellm_metadata allow_client_pricing_override no longer keeps a caller-supplied standard_logging_guardrail_information on the chat or pass-through paths, since its guardrail_cost is billed. Proxy-only failure logs now also carry key and team attribution in litellm_metadata. * refactor(proxy): keep the proxy-only failure-log fixes inside the type gates Types the picked helpers so basedpyright and the type-discipline gate stay at main's counts. No runtime change. * fix(proxy): keep the WS frame guardrail-info strip and string interaction tags as sent The guardrail-info key left the pricing strip list, so the Responses WS frame guard now drops it by name. get_request_tags no longer raises on string tags, so background interaction polling checks for them before keeping them as sent. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0657ed46cf
|
fix(langfuse_otel)!: derive trace-level name, input, output and tags on traces litellm starts (#42705)
* fix(langfuse_otel): derive trace-level name, input, output and tags on the generation span Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): keep every reasoning summary in the derived observation output Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): leave joined traces untouched by derived trace fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): restore preset cache key fallback and type the derived-field helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): keep trace tags off joined traces entirely Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): tolerate out-of-band upstream requests and add sink outage chaos cell Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): build the langfuse rig config without mutating the loaded yaml Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(langfuse_otel): flatten output items with itertools.chain to satisfy LIT014 budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): type the _RecordingSpan and _emitted unit helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): keep joined-trace tags, mask trace io, leave v2 trace io to the root Joined traces (existing_trace_id) keep the caller tags they carried before, and only the derived trace fields are skipped for them. mask_input and mask_output now redact the derived trace input and output, matching the vanilla Langfuse callback. The v2 mapper derives the trace name only, because the root observation already carries the trace input and output there. Uses the public preset cache key helper, moves imports to module top, restores the original formatting of the unit test file, and adds integration cells for the v2 llm_only scope and SDK sync/async calls. * fix(langfuse_otel): split string caller tags the way Langfuse reads them A caller tags value given as a comma list or a JSON array string (body metadata.tags or the langfuse_tags header) used to reach Langfuse raw, where ingestion split it into separate tags. Merging it with the derived tags kept it as one tag, so parse it the same way before the merge * test(otel): expect langfuse.trace.name on the v2 langfuse_otel generation span * fix(langfuse_otel): read string mask flags the way the classic callback does The langfuse_mask_input / langfuse_mask_output headers arrive as strings, so "false" redacted the derived trace io. Parse them with str_to_bool and fall back to truthiness, as the classic langfuse callback does * fix(langfuse_otel): derive trace fields only on traces litellm starts A litellm span that joins a trace an app or caller started (inbound traceparent, the app's active span, a caller-provided parent span) no longer gets a derived trace name, input, output or tags, like a request joining an existing_trace_id. Langfuse applies the later-starting span's trace update, so the derived values overwrote the app's own trace name and io. OTel v2 derives the name on the request root only when it has no parent, and the generation span keeps main's caller-only trace name. * test(langfuse_otel): keep main's OTel v2 generation key set in the arize family cell The OTel v2 generation span no longer carries a derived langfuse.trace.name, so the pinned key set goes back to main's. * fix(langfuse_otel): name the OTel v2 request root once, from the request's own call The v2 Langfuse logger re-derived langfuse.trace.name on the request root for every LLM call in the request. A post_call llm_as_a_judge guardrail's judge call replaced the caller's trace_name on the root with litellm-acompletion, and with another v2 callback (otel, arize_phoenix) listed before langfuse_otel that vendor's root span gained langfuse.trace.name on every request. The derived name now fills only an unnamed root this logger's provider started, and never comes from a call litellm makes for its own features. A caller trace_name still names the root on every call, as on main. * fix(langfuse_otel): keep the derived trace fields from pushing other attributes off a full span The derived trace name, tags, input and output were written after the OpenInference attributes with no room check, so a span near the SDK's attribute limit (about 34 tools at the default 128) evicted keys main exported, such as openinference.span.kind, metadata and llm.model_name, and raised the dropped count. They are now written last and only into the room the limit leaves next to the error attributes a failed call adds, so a full span exports what it would without them. * test(langfuse_otel): give the full-span cells' derived request a caller trace id, as the joined one's traceparent does * chore(langfuse_otel): drop the docstrings and comments the house rule disallows and type the new tests' parameters * test(langfuse_otel): set the span attribute limit the full-span tests run at and keep Final locals out of the app-trace loop --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
14193435b8
|
refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages Move the unified /v1/messages code out of a directory called pass_through, since it is not the passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner model experimental_pass_through package becomes messages. The old packages stay as import shims that warn once and alias every submodule in sys.modules, so old deep imports still resolve Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): keep Final variables out of loops in the rename shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): match the warning text in the rename shim test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): resolve old pass_through imports lazily through a meta path finder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): mark the alias loader spec restore for the type discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): format the renamed package shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): keep the rebind-ok marker on the spec restore line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): carry the new package search path on alias specs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): declare the shim test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
715e9bb08d
|
fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668)
* fix(prometheus): count provider-side 401s in LLM failure and deployment metrics The invalid virtual key skip no longer applies to an error the LLM provider returned while the deployment was called with its own credential, so a 401 for a bad deployment credential increments litellm_llm_api_failed_requests_metric, litellm_deployment_failure_responses and litellm_deployment_total_requests and marks litellm_deployment_state as a partial outage again. Proxy-gate rejections and a failing credential that belongs to the caller (a rejected per-user credential, an api_key sent with the request, a per-user GitHub Copilot session) keep the skip * fix(prometheus): keep a caller's own Anthropic OAuth token failure out of the deployment metrics A provider 401 on an Anthropic OAuth token the caller forwarded through the proxy is the caller's credential failing, so it stays out of the deployment failure and outage series like a caller-supplied api_key. Adds integration cells for an Anthropic deployment's own key, the forwarded caller OAuth token and a direct SDK call that brings its own key * test(prometheus): pin how a caller's own key or Authorization header on a deployment is counted An SDK Router call that passes its own api_key is counted against the one-off deployment the router builds for that key, so the operator deployment keeps only its own failure. A provider 401 on an Authorization header the caller sends through the proxy counts against the deployment, as on v1.80.12 * fix(prometheus): keep caller-sent provider credentials and token-exchange 401s out of deployment metrics A caller can send its own provider credential in the request body, such as azure_ad_token, azure_scope, the Azure Entra app fields, watsonx token, gigachat_access_token, OCI keys, AWS keys or Vertex credentials, at the top level or inside extra_body. A provider 401 on that credential is the caller's failure, so it stays out of the shared deployment's failure, request, LLM-failed and state series, as on main. A token-exchange deployment trades the caller's identity token on every call, so its 401s stay out as well, matching the router cooldown. * test(prometheus): Final locals, precise helper types and one for per comprehension in the new test helpers * test(prometheus): pin that a provider 401 at a caller-sent api_base counts against its one-off deployment * fix(prometheus): keep a provider 401 on caller-sent headers out of deployment metrics A caller can override the deployment's credential with headers it sends in the request body: headers or provider_specific_header, or an Authorization, api-key, x-api-key, x-goog-api-key or Ocp-Apim-Subscription-Key entry in extra_headers. A provider 401 on such a request is the caller's failure, so it stays out of the shared deployment's failure, request, LLM-failed and state series, as on main. * fix(prometheus): skip a provider 401 only when the caller's headers carry a credential The proxy now records headers, extra_headers or provider_specific_header in credential_fields only when they name a provider credential header (Authorization, api-key, x-api-key, x-goog-api-key, Ocp-Apim-Subscription-Key), the same way api_key is recorded. A caller that sends a plain header such as a trace id still has a provider 401 on the deployment's own key counted against the deployment. * fix(prometheus): stamp credential_fields after the proxy forwards the caller's headers A provider credential header the proxy forwards from the caller's HTTP request (forward_llm_provider_auth_headers + forward_client_headers_to_llm_api) lands in data["headers"] after proxy_server_request is built, so the stamp missed it and a 401 on that caller credential was counted against the deployment. Compute the stamp after the header forwarding steps and pin both forwarded cases. * fix(prometheus): skip a provider 401 when the caller sends any header or an OpenAI organization The proxy now records headers, extra_headers or provider_specific_header in credential_fields whenever the caller sends or the proxy forwards any header there, and records organization when the caller sends one in the body or as a forwarded OpenAI-Organization header. A forwarded header can be a gateway key such as x-portkey-api-key or an OpenAI-Project id, and a wrong organization gets a 401 from OpenAI, so these 401s stay out of the deployment metrics as on main instead of marking a healthy deployment as a partial outage. * fix(prometheus): ignore the user information the proxy adds when stamping caller headers With add_user_information_to_llm_headers on and header forwarding enabled, the proxy adds x-litellm-user_api_key_* headers to the forwarded headers. Those carry the proxy's own key and user details, not the caller's, so they no longer make a request look caller-scoped, and a provider 401 on the deployment's own key on such a request is counted. * fix(prometheus): skip a 401 raised before the provider is called or at the caller's own endpoint A deployment with no key of its own fails with LiteLLM's local "Missing Anthropic API Key" 401 before any provider call, and a caller who sends aws_bedrock_runtime_endpoint (or another provider endpoint field) can get a 401 from their own gateway. Both were counted against the shared deployment and marked it as a partial outage, where main skipped every 401. Count a 401 only once the provider was called (api_call_start_time is set), and treat caller-sent provider endpoint fields like caller credential fields. * fix(prometheus): count the deployment's own 401 on requests that carry only anthropic-version or anthropic-beta The Anthropic SDK sends anthropic-version on every request, and the proxy copies anthropic-version and anthropic-beta into provider_specific_header for Anthropic, Bedrock and Vertex. Both were treated as caller headers, so a 401 on an Anthropic deployment's own key from any Anthropic SDK client stayed out of the deployment metrics. These two headers only pick the API version and beta features. Anthropic answers a bad anthropic-version or an unknown anthropic-beta with a 400, never a 401, so they cannot turn the deployment's own key into a 401 and no longer mark the request as carrying a caller header. * test(prometheus): read and write the owned proxy's own database in the expired-key and MCP cells Since #47213 an owned proxy boots on a private database, so the expired-key update and the MCP guardrail row read now go through the gateway's database_url instead of the worker's. * fix(prometheus): keep credential_fields as main stamps it and give Prometheus its own caller_credential_fields credential_fields tells the Rust OCR bridge which stripped body fields the caller sent. Repurposing it for the Prometheus caller-credential check changed its value (organization and extra_headers added, anthropic-version-only headers dropped). Restore main's stamp unchanged and stamp the Prometheus classification under caller_credential_fields, which only Prometheus reads. * fix(prometheus): skip a provider 401 on a stored credential the caller picks by litellm_credential_name A request body litellm_credential_name swaps the deployment's stored credential for one the caller chose, so a 401 on it is the caller's, not the deployment's. Treat it like the other caller credential fields so it no longer marks a healthy shared deployment as a partial outage. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
38f318e736
|
fix(otel): decide per-team Arize sampling from the trace id so every pod agrees (#47468)
* fix(otel): decide per-team Arize sampling from the trace id so every pod agrees * test(otel): cover per-team sampling across pods, workers, endpoints and restarts * fix(otel): remember only a failed tree's sampling verdict so a later failed call answers to the error rate A successful tree's verdict follows from the trace id alone, so remembering it only made a later failed call of the same trace follow the success rate on the pod that served both, while other pods judged it at the error rate * test(otel): flush every worker before the restart cell kills it and cover a failed call after a dropped one * fix(otel): decide team sampling per request, not per trace Calls sharing a caller's trace id are separate requests, keyed by their root span. Each is decided by the trace hash at the rate for its own outcome, and only its own late spans follow its verdict, so no pod applies an earlier call's verdict to a later call of the trace * fix(otel): forget a span's request root when it ends, not when newer spans start The root of each open span was kept in a first-in-first-out table of 16384 entries, so a span two levels below the root that stayed open while the worker started more spans than that lost its entry and ended under its parent's id. Its request then never closed and was held from every sampled destination until shutdown. Each open span now keeps its entry until it ends, and a span starting under a parent that has already ended reads the root from the parent span itself * fix(otel): decide a request as its spans end once it falls out of the open span table A request whose span was pushed out of the table of open spans, because the worker held more than 16384 open at once, ended that span under another key and was held from every sampled destination until shutdown. Pushing a span out now drops its request's open count too, so the request is decided as each of its spans ends, the way one pushed out of the open request table already is * test(otel): annotate the team sampling test locals as Final * test(otel): build the caller's remote parent in one helper instead of rebinding it per trace * fix(otel): keep each open span's request until it ends, bounded by the open requests The span table had its own bound of 16384, so a worker holding more spans open than that pushed out the oldest, and a failure in such a span no longer reached its request. Each request now holds the ids of its open spans, and a span's entry in the lookup goes when the span ends or when its request is pushed out of the open request table, the same bound main already applies. A failure anywhere in a tracked request reaches its verdict however many spans are open --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
464932b1d1
|
refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789)
* refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1) * refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2) * refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts * fix: restore pass-through behavior on paths the typing pass tightened MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites * fix(runwayml): keep a malformed 2xx task body on the 500 path * fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report EdenAI's embedding and image generation parse dropped a non-object body from the error they raised, so the message no longer named what the provider sent. A helper now raises with the body in the message, and both paths have a regression test. `lite encryption migrate` exited 0 with no message when the proxy's report was not an object or its residual_legacy was not a number, where base crashed. It now fails with a message naming the value. The CompactifAI and Gemini image-edit non-object body tests accept TypeError or AttributeError, which both base and this branch raise, so they guard against a ValidationError instead of pinning one class. RunwayML's async parse gets the same non-object body test as the sync one. * fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed * test: annotate the new regression tests' locals with Final * test: annotate the remaining new test locals with Final |
||
|
Ishaan
|
8f9894fbd6
|
fix(ci): unblock dependency security checks (#48055)
* fix(deps): upgrade vulnerable Python dependencies * fix(deps): lock patched Python dependency versions * fix(otel): preserve JSON export with the updated SDK * test(otel): cover JSON export with the updated SDK * test(otel): update SDK log records and worker discovery * fix(ui): align Lens peer dependency with patched Next.js * fix(otel): hand the AgentOps exporter a session it owns for the JWT header OpenTelemetry 1.45 moved the OTLP HTTP exporter's requests.Session behind a transport, so writing the minted JWT to the exporter's private _session raised, was swallowed, and every AgentOps batch went out unauthenticated. The exporter now creates the session, passes it through the public session= argument and sets the Authorization header on it The tests now drive a real export, serving the token endpoint with respx and capturing the OTLP POST with requests_mock, instead of reading SDK internals * fix(otel): skip FastAPI instrumentation once the app has started opentelemetry-instrumentation-fastapi 0.66b1 wraps build_middleware_stack instead of calling add_middleware, so instrumenting an app whose middleware stack is already built no longer raises. The call silently did nothing while we still registered the global response propagator. Check the built stack up front and skip both, which is what the existing test expects * test(otel): assert exported requests and span flags instead of SDK internals OpenTelemetry 1.45 dropped the exporter's private _headers and sets the W3C random-trace-id flag on new traces, so traceresponse ends in -03. The Langtrace and Langfuse tests now export through requests_mock and check the URL and headers the collector receives, and the traceresponse tests expect the server span's own trace flags * fix(deps): bump dashboard zod to 4.6.6 and make the vendored Lens use it zod 4.6.5 is flagged by the dependency scan and blocks osv-scan on every PR. A root bump alone leaves the vendored Lens tarball, which pins zod 4.6.5 exactly, on its own nested copy, so a top-level zod override points every consumer at the root version and the lock carries a single zod 4.6.6 * test(ui): stub the tag summary fetch the client panel retry refreshes Retrying the model breakdown refetches /tag/summary, which this integration test left on the real network. When that request failed before the assertion, the panel switched to its summary error and unmounted the model row, so the test passed or failed by timing * fix(deps): drop the duplicated Lens override from the dashboard package.json * style(ui): format the client panel fetch stub --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
Kerry Lu
|
5dce13d2c2
|
fix(proxy): attribute retry attempt spend rows to their deployment and keep them out of daily request counts (#47658)
* fix(proxy): attribute retry attempt spend rows to their own deployment and keep them out of daily request counts Each <request_id>-attempt-<n> row now carries the model, provider, api_base and model group of the deployment that billed it (looked up by model_info.id through the proxy router, injected as deployment_lookup) instead of copying the final row's, and is stamped internal_call_origin=billed_retry_attempt so the daily spend tables keep its spend and tokens while counting zero api_requests and failed_requests for it * fix(proxy): declare DeploymentLookup as a TypeAlias and leave the test file's untouched lines as they were * fix(proxy): mark attempt rows with billed_retry_attempt instead of internal_call_origin so model usage keeps their spend Bugbot: an internal_call_origin stamp made build_model_usage_transaction drop the attempt rows from LiteLLM_DailyModelUsage. The rows now carry their own SpendLogsMetadata flag, which only the daily request counters read * fix(proxy): read the billed_retry_attempt flag through a typed helper so the basedpyright gate holds * fix(proxy): keep the typed litellm_params local in _deployment_provider so the basedpyright gate holds * fix(proxy): annotate the validated litellm_params as LiteLLM_Params in _deployment_provider * fix(proxy): keep billed retry attempts out of daily request errors and strip the flag from caller metadata Bugbot: _record_request_error counted every attempt row as a caller failure in LiteLLM_DailyRequestErrors. Veria: a caller could send metadata.billed_retry_attempt to zero its own request counts, so the key joins the untrusted metadata fields the proxy strips on the way in * fix(proxy): stamp billed_retry_attempt explicitly in the spend log metadata so the basedpyright gate holds --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
480ee653c5
|
fix(responses): send the billed background response cost to the creating key's and team's loggers (#47506)
* test(responses): the billed background response cost reaches the creating key's and team's loggers * fix(responses): send the billed background response cost to the creating key's and team's loggers * fix(responses): bill an unreadable or rejected creator through proxy loggers, redact and honor disabled callbacks on the billed event * fix(responses): follow the creator's message logging setting on the billed event and read team logging for non-key creators The billed event no longer forces redaction. It follows the creating key's or team's turn_off_message_logging, else the proxy's, and redacts when the creator cannot be read or its settings are rejected. A creator without a virtual key hash bills through its team's settings. function_setup keeps its original callback split, and the billed path uses a separate non-mutating helper. * fix(responses): bill rotated, deleted, and custom auth creators through their own logging and retry failed creator lookups * fix(responses): follow a key's whole grace-period rotation chain to the live key for its billed charge's logging * fix(responses): bill a background response whose creator cannot be read, without the creator's settings * docs(enterprise): say the background response row is claimed before its creator is read * fix(responses): keep the loaded creator rows' budgets and aliases when one creator lookup fails |
||
|
devin-ai-integration[bot]
|
b55024658c
|
fix(proxy)!: stop update_cache from resurrecting evicted auth cache entries (#46997)
* fix(proxy): stop update_cache from resurrecting evicted auth cache entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update update_cache pins for synchronous write-back Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): wrap long assert in update_cache tag test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover refill race and pin TTL with injected clock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): count global proxy spend for every request with a user, cached on this worker or not * fix(proxy): add global proxy spend with an atomic Redis increment that never re-creates a reset counter The global proxy spend bump read the total, awaited, and wrote old + cost back from a background task, so a budget reset that deleted the key in between got the pre-reset total written back, and workers overwrote each other's totals. RedisCache.async_increment_if_present runs INCRBYFLOAT only when the key exists, keeping its TTL, and this worker's copy is then set from the shared total with no await between its read and write. The bump now runs whenever a proxy max_budget is set, matching the DB writer, and the armed post-call read and the batch-read declaration it used are removed. * fix(proxy): report the global spend Redis increment to service metrics and request traces The global proxy spend increment now emits Redis service success and failure events under the spend_counters target and passes the request span through, so it shows as redis.incr spend_counters like the write it replaced. The unit tests run the real RedisCache increment against fakeredis instead of mocks, and a new test checks the span lands under the request span on success and on a Redis failure. * test(proxy): keep the spend counter Redis fake off the network * test(proxy): count increment_if_present as a shared-cache call in the key-family guard * fix(proxy): keep the newer global spend total when Redis replies arrive out of order --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
9e3d3a87f4
|
fix(anthropic, bedrock)!: keep a redacted_thinking content item on every Claude route (#47489)
* test(anthropic, bedrock): cover a redacted_thinking content item on every Claude route * fix(anthropic, bedrock): keep a redacted_thinking content item on every Claude route * fix(anthropic, bedrock): send each replayed thinking block once, without cache_control or blank data * fix(anthropic, bedrock): keep thinking order and gate replay key drops on drop_params A thinking block that only one of content and thinking_blocks carries now goes back in its original place instead of being prepended. cache_control on a thinking block and blank redacted data are dropped only under drop_params and otherwise reach Claude as sent. Converse drops a cachePoint after reasoning under drop_params, skips a thinking_blocks entry that is not a block, and the cache control hook no longer marks a thinking part. * fix(anthropic, snowflake): match a signed thinking block on its signature and send each block once on Snowflake Cortex A signed thinking block in both content and thinking_blocks is the same block when the signatures match, even if the thinking text differs, so it is sent once from content instead of twice. Snowflake Cortex now merges thinking_blocks into content the same way the Anthropic routes do. A redacted_thinking item with blank data no longer counts as sent for system message placement. * fix(anthropic, bedrock): send a thinking block once when consecutive assistant messages both carry it Consecutive assistant messages merge into one Claude turn, so a block one message holds in content and the next holds in thinking_blocks went out twice and Claude answered 400. The merged turn now keeps the first copy of each redacted or signed block on the Anthropic routes and both Bedrock Converse builders. Snowflake Cortex goes back to its previous message building, since its double send cannot be proven live without a Cortex account and is tracked on its own. * test(anthropic, bedrock): cover a redacted block with null data and a non-block part beside a cache breakpoint * fix(anthropic, bedrock): match only text-valued thinking items when merging replayed blocks |
||
|
Max Azatian
|
5145250d83
|
fix(langfuse_otel)!: send the completion start time (#45597)
* fix(langfuse_otel): send the completion start time langfuse_otel never set langfuse.observation.completion_start_time, so Langfuse showed no time to first token for any call, streamed or not. The legacy langfuse callback already sends kwargs["completion_start_time"]. Send the same value as an ISO timestamp with an explicit offset. * fix(langfuse_otel): type kwargs and test against a real span Type the kwargs of _set_langfuse_specific_attributes as dict[str, Any], the declared type of model_call_details, so the new completion start time read adds no unknown-type errors. The tests now check a real SDK span instead of patching safe_set_attribute, and the parameter is typed. * fix(otel): send the completion start time from the v2 Langfuse mapper With LITELLM_OTEL_V2 the langfuse_otel spans come from LangfuseMapper, which had no completion start time either. Carry the payload's completionStartTime on LLMCallSpanData for streamed calls, the same calls that get time_to_first_chunk_seconds, and map it to langfuse.observation.completion_start_time. * fix(langfuse_otel): take the logging kwargs as a read-only Mapping The new kwargs annotation tripped LIT001. Both helpers only read the logging kwargs, so type them Mapping[str, Any] instead of dict. * fix(langfuse_otel): time /v1/messages streams from the call start and send the V2 completion start time like the default emitter * fix(langfuse_otel): time Gemini, Interactions and A2A streams from their first chunk and send the start time as JSON The native Gemini, Interactions and A2A streams never recorded a first chunk, so with completion_start_time now sent both emitters showed TTFT equal to latency on them. They now record it, and Gemini and Interactions take the call's start time like /v1/messages. Both emitters send the completion start time as a JSON string, the form the Langfuse SDK and the langfuse callback send, through one serializer. * fix(langfuse_otel): time A2A streams from the call start and record a cached replay's first event A2A streams took their start time when the iterator was built, so their latency left out the work before it. They now use the call's start time like every other route. Cached /v1/messages stream replays recorded no first event, so their time to first token fell back to the end time. The replay now records it on the first event it yields. The tests use the real Interactions config and typed logger hooks. * fix(a2a): time streamed A2A calls from before the agent card fetch and make the cached replay test clock-free * test(a2a): type the card fetch test and stop rebinding the seed helper's recorder * test(otel): expect completion_start_time and cache_hit in the B3 span keys --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Max Azatian
|
77880a0653
|
fix(otel)!: leave out span cost when the price lookup failed (#44187)
* fix(otel): leave out span cost when the price lookup failed A model missing from the price map gets response_cost 0.0 in the standard logging payload, with response_cost_failure_debug_info set. The OpenInference writer (arize, phoenix, langfuse_otel) and the OTel v2 span data copied that 0.0 into llm.cost.total, litellm.cost.total and langfuse.observation.cost_details, so backends showed the call as free. Skip the cost when the lookup failed, except on cache hits, which are free. * test(otel): keep the guardrail charge on the guardrail span for unpriced calls * refactor(otel): type the Arize cost writer input and reuse the cache_hit flag * fix(otel): keep a billed guardrail fee on an unpriced call's Arize span cost Only the placeholder 0 is left out when the price lookup failed, so a guardrail fee LiteLLM billed still reaches Arize, Phoenix, Langfuse OTel and W&B Weave OTel. Adds unit cases for that fee and a malformed cost. * fix(arize): leave out the span cost on any failed price lookup, matching OTel v2 * fix(logging): clear a stale cost failure flag when a later pricing succeeds Stream-start pricing on the request model name can fail and set the cost failure flag even when the final pricing succeeds (OpenRouter usage.cost, an Azure deployment name priced from the response model). The new span gate then dropped a real cost. Each pricing now resets the flag, and uncached_price restores whatever flag it found. * test(logging): pin the start time in the stale cost flag tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
AaronHowell
|
6b387e84b5
|
fix(prometheus): preserve requested model after fallback (#45555)
* fix(prometheus): preserve requested model after fallback Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> * fix(prometheus): read fallback origin from both metadata buckets Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> |
||
|
Max Azatian
|
f43f2d8c15
|
fix(otel): put the error message on every failed span status (#44185)
* fix(otel): put the error message on the failed span status The legacy OTel logger (used by otel, langfuse_otel, arize and others) ended a failed call with Status(StatusCode.ERROR) and no description, so backends that read the span status, such as Langfuse's status message, showed an empty error. The text was only in the error.message attribute. Use the same message as that attribute, as the OTel v2 emitter already does. * fix(otel): take the failure status message from _record_exception_on_span Instead of reading the logging payload a second time in _handle_failure, _record_exception_on_span returns the error message it already records as error.message, and _handle_failure puts it on the ERROR status. The other callers ignore the return value, so their spans are unchanged. Its input is now a typed view of what it reads (exception, error_information, error_str) instead of a bare dict. * fix(otel): put the error message on every failed span status Covers the proxy SERVER span, management endpoint spans, service spans, and Arize Phoenix spans, so no ERROR span ships with an empty status description * fix(otel): cut the error status to the span attribute length limit and describe dangling server spans The status description now uses the error text as stored on the span, so OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT applies to it too. A server span the proxy closes on a 4xx or 5xx outside the request path gets the same description, and an empty error message falls back to the error class * test(otel): type the validation-handler status test * test(otel): cover the Phoenix parent span and error_str length-limit status --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
ca0ec627b6
|
fix(exa): return highlights, highlight scores, score and the response envelope from Exa search (#47423)
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
365d7342c0
|
fix(prometheus): label deployment failures with the deployment's provider instead of None (#47384)
* fix(prometheus): label deployment failures with the deployment's provider instead of None * test(prometheus): drive every deployment failure surface through the provider label Streaming chat, responses and messages, dropped and stalled upstreams, JSON-configured, custom-provider-map, prompt-management and unprefixed deployments, forged request metadata, an unauthenticated request, the OpenAI and Anthropic SDKs sync and async, and a concurrent burst counted exactly once per deployment --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
f85926a652
|
feat(router): keep a session on one kubernetes pod (#44393)
Requests with an explicit session id stick to one discovered pod via rendezvous hashing Requests without a session id keep round robin Pod routing records are included in logs, spend logs, and traces Resolves LIT-9168 Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4bebcdf757
|
fix(databricks): use the native Anthropic route for Claude /v1/messages (#46091)
* fix(databricks): use the native Anthropic route for Claude /v1/messages Claude models on a Databricks workspace were bridged through chat completions on /v1/messages, which drops the safeguards field and never returns safeguard_results. They now call the workspace's /serving-endpoints/anthropic/v1/messages, the proxy forwards the client's anthropic-beta and anthropic-version headers to Databricks deployments, and the beta header map marks dangerous-tool-use-2026-09-03 as supported there. Any route whose platform does not take that beta drops safeguards together with it instead of answering 400. Non-Claude Databricks models keep the chat completions bridge * fix(databricks): derive the dangerous-tool-use beta from safeguards instead of forwarding client betas A Databricks Claude serving endpoint answers 400 to betas Claude Code sends on every turn, prompt-caching-scope-2026-01-05 among them, so the proxy no longer forwards the client's anthropic-beta header to Databricks deployments. The beta the classifier needs is derived from the safeguards body field, the same way the Azure AI route derives it. The Rust beta-header mirror stops rejecting the Databricks dangerous-tool-use cell so its config-cell test matches the JSON again * fix(databricks): keep the billing block, build the messages route once, and pass the partner user agent * fix(databricks): route Unity Catalog Claude names to the gateway messages route and keep env credentials Unity Catalog model-service names now reach the AI Gateway's native Anthropic route instead of a serving-endpoints path that answers 404 for them, and the native route resolves the key and base through the same order the chat route uses, so environment-only deployments keep working. The native route is limited to Databricks foundation-model Claude names, an unlisted provider forwards safeguards and the beta as sent, and the ten betas the workspace rejects are marked unsupported for Databricks in the beta map and its Rust mirror. * test(rust): expect Databricks to drop the advanced-tool-use beta * fix(databricks): keep passthrough safeguards, partner user agent, and classifier fallback Only the OpenAI-compatible JSON providers drop safeguards now, so a deployment opted into /v1/messages passthrough forwards the field and the beta as sent. When a route drops safeguards and the proxy has a classifier model, the proxy answers the verdict itself, as the chat completions bridge did. The Databricks Messages route also reads the databricks_user_agent param like the chat route. * fix(databricks): drop rejected server tools under drop_params, short-circuit web search, route UC names from a full Messages URL * fix(databricks): drop container on the native Messages route, as the chat completions bridge did * fix(databricks): keep every api_base on a route the workspace serves and gate the derived beta on the map Gate the dangerous-tool-use beta derivation on the beta map the way #43955 does, so GitHub Copilot, DeepSeek, and opt-in passthroughs keep the merge base wire. Recognize /serving-endpoints/anthropic api_bases, fall back to the merge base routing for an api_base path the workspace trim does not know, strip Bedrock invocation metrics from the streamed message_stop, forward client betas to Databricks filtered by the map, and keep tool_choice when a dropped server tool has no name * test(databricks): cover the native Messages route, derived beta and JSON provider safeguards drop on the wire * test(anthropic): expect the dangerous-tool-use beta derived from safeguards next to the client's betas * fix(databricks): send client anthropic headers only on the native Messages route and keep SDK auth Databricks chat completions, called directly or through the Messages bridge, no longer receive the client's anthropic-beta and anthropic-version, as at the merge base. The native Messages route still forwards them, filtered by the beta map, without counting them as caller credentials, so a deployment with no api_key keeps falling back to Databricks SDK auth. A pathless api_base goes to the native route only for Unity Catalog names, matching the chat route. * test(databricks): cover SDK auth with client anthropic headers and a Unity Catalog workspace root on the native route * fix(databricks): keep client anthropic headers off an opted-in passthrough api_base A databricks Claude deployment whose api_base is no workspace route and that opts into /v1/messages with supported_endpoints falls back to the OpenAI-like passthrough. That config does not filter betas through the beta map, yet it received the native-only Databricks header entry, so Claude Code's prompt-caching-scope beta reached the upstream raw and got a 400 where the merge base answered 200. Each Messages config now says whether it is the provider's native route, the OpenAI-like passthrough says no, and the handler passes that answer when it reads provider-specific headers, so native-only entries reach only native configs. * test(databricks): cover client anthropic betas on an opted-in passthrough api_base A databricks Claude deployment whose api_base is no workspace route and whose model_info.supported_endpoints opts into /v1/messages takes the OpenAI-like passthrough. The cell sends Claude Code's prompt-caching-scope and context-1m betas through the proxy and asserts the caller gets the upstream's message and the gateway receives no anthropic-beta, as on the merge base. The scripted gateway answers 400 on the prompt-caching-scope beta, as the real one does * fix(anthropic): keep beta map loads off shared threads and build the safeguards evaluator only when the proxy classifies Unfiltered configs such as first-party Anthropic never read the beta map to decide safeguards support. Filtered configs await the map on its dedicated loader before reading it, on both the native and the MCP paths. The native handler decides after the upstream call and builds the evaluator only when the proxy classifies, so malformed messages keep the upstream 400. The Databricks dispatch tests assert the dispatched route instead of the config class, and the sail and Eden AI wire rows expect safeguards dropped, as JSON providers do. * test(databricks): cover a loaded beta map that marks the safeguards beta unsupported A remote beta map that nulls dangerous-tool-use for databricks while the bundled map allows it must send neither safeguards nor the beta to the serving endpoint, with or without a client beta, and the caller gets a 200 with no verdict. * test(anthropic): annotate the MCP beta map recordings as Final * fix(anthropic): drop safeguards on routes without the classifier only under drop_params A route whose upstream does not answer safeguards (a JSON provider, or a map that marks the dangerous-tool-use beta unsupported) now forwards the field and the beta as the merge base did unless drop_params is set or the proxy runs its own classifier, matching how litellm treats any unsupported param. Databricks keeps dropping both, since its merge base bridge never forwarded the field. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4273996985
|
fix(spend_logs): persist user email in spend log metadata so it shows in UI request logs (#41150)
* fix(spend_logs): persist user email in spend log metadata so it shows in UI request logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): update payload expectations for user email metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(gcs_pubsub): add user_api_key_user_email to the spend log fixture Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): drop untyped metadata key args from the debug log to keep the type gate flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): pass metadata keys to the debug log as a lazy %-style argument Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(spend_logs): store user email in spend logs only behind store_user_email_in_spend_logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(spend_logs): store user email in spend logs by default, drop store_user_email_in_spend_logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): store an empty user email as null, narrow the expected payload test diff Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
8226b2be9a
|
fix(otel): report cache hits at zero cost with a cache-hit flag in V1 and V2 spans (#43604)
* fix(litellm): report zero cost and cache-hit attrs on cache-served spans
* fix(litellm): stamp cache-hit marker on HiddenParams carriers and restore probe side effects
gauntlet-mini on
|
||
|
Mateo Wang
|
e960d96ae9
|
fix(prometheus): set the remaining rate-limit gauges from decimal headers again (#47039)
* test(integration): wait for both workers to load each metered model before sending * fix(prometheus): read remaining rate-limit headers as numbers and let the router fill unreadable ones * fix(prometheus): keep deployment metrics on non-string header keys and cover decimal and router-filled gauges Accept any mapping key when reading additional headers, so a non-string key no longer drops every success metric. The router fallback test now uses a real Router and covers inf, nan and unreadable requests values, the decimal test uses a fractional requests value, and the integration cell sends 12.5 and an unreadable tokens value to deployments with tpm set * fix(prometheus): leave the router fallback unchanged and keep only the decimal header fix * fix(prometheus): read remaining rate-limit headers with float() like every release did * test(integration): check the remaining rate-limit gauges on /v1/responses and /v1/messages too |
||
|
jesus-berri
|
7f603da9f1
|
fix(openai_like)!: surface provider response headers on non-streaming chat completions (#46928)
* fix(openai_like): surface provider response headers on non-streaming chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(openai_like): mark the header test locals Final * fix(proxy): leave out response header values HTTP cannot carry Non-streaming OpenAI-like chat now forwards every provider header, so on the httpx transport a provider header value outside latin-1 made the proxy answer 500 after the provider had served the call. get_custom_headers now leaves such a value out and keeps the other headers, which also ends the same 500 on streaming Adds integration cells for the provider request id on non-streaming chat and the Responses bridge, LiteLLM headers an upstream cannot overwrite, spend parity under an upstream cost header, the latin-1 case, and the logging callback payload * fix(cost): ignore an upstream response cost header that is negative or not a number * test(integration): cover invalid upstream cost headers and non-latin-1 headers on every endpoint * fix(prometheus): keep deployment metrics when a provider rate-limit header is not a number * test(prometheus): type the rate-limit header test and drop its clock reads * test(integration): cover deployment metrics under an unparseable provider rate-limit header --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
d76ec3a47e
|
fix(proxy)!: answer 400 for malformed tools on count_tokens instead of crashing (#46583)
* fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing A non-list tools value crashed the vector store lookup during auth and the built-in tool cost lookup in the failure logging path, so the proxy answered an unhandled 500 and closed the keep-alive connection. count_tokens now answers a 400 invalid_request_error for a malformed body, drops malformed tools when drop_params is on, and both lookups skip values of the wrong shape. * fix(proxy): skip non-list tools in the tool allowlist check Also validate the count_tokens body without recursion, which the recursion detector rejects. * test(integration): restore the int tools case on count_tokens The proxy now answers a non-list tools value with a 400, so the case is deterministic again. * fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids * fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids Check tools after request setup so the Claude Code default and the logged call type apply, read the request, global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off * fix(proxy): keep the request's call type when logging a gate rejection after request setup The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses rejection as aembedding, so it now only applies when the logging object carries no known call type * fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry * fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket * refactor(proxy): leave the responses websocket out of the request-level tool check * fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections * fix(proxy): read router default drop lists and deployment ids in the malformed tools check * fix(proxy): leave malformed tools to the router when it can pick another deployment Fallbacks, auto-routers, model strings, comma batches and team model names let the router pick a deployment after the proxy checks the request, so the proxy now forwards a malformed tools field there as main does, and only rejects it when the deployment it checks is the one that serves. Token counting never reroutes, so count_tokens and input_tokens keep reading the group they count with. additional_drop_params names are read the way core reads them, so a list with a non-string entry still names its strings. * test(integration): cover malformed tools on reroutable models, router defaults and deployment ids * fix(proxy): resolve malformed tools at the deployment the router picks * fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells. * test(integration): expect a 400 for an empty-string tools field on /v1/messages Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test * fix(router): keep web search routing for tools passed as a tuple * fix(router): keep a deployment's own tools when drop_params drops a malformed request tools * test(router): list _merged_tools as covered through the deployment tools merge tests * fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them * fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError * fix(prometheus): keep deployment health metrics clean on a malformed tools rejection * test: type the new malformed tools tests * test: mark new test locals Final and type kwargs dicts |
||
|
yucheng-berri
|
f281f77bc0
|
fix(guardrails): record every content filter match in logging_only and offer the mode in the UI (#46673)
* test(guardrails): pin content filter logging_only matches and UI modes in an integration test * fix(guardrails): record every content filter match in logging_only and offer the mode in the UI * style(tests): format the content filter logging_only integration test * refactor(guardrails): keep the logging_only predicate and scan override within the type gates * fix(guardrails): flag tool arguments a content filter would reject in logging_only and keep scanning * test(integration): expect guardrail_flagged for an observe-only content filter block in the scope config test * test(integration): pin logging_only flags for tool arguments an enforcing content filter rejects * refactor(guardrails): drop the content filter flagged branch no logging_only scan can reach * test(integration): pin OTel spans, Guardrails Monitor and a provider outage burst for a logging_only content filter * test(integration): pin MCP tool call verdicts and fail the Monitor cell when usage never rolls up * test(integration): pin category and competitor intent verdicts and the provider failure row in logging_only --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
38f3f8a1a9
|
fix(proxy): cooldown errors carry 429, Sentry starts at boot without request bodies, tracing failures log at ERROR, exporter counters exported (#46344)
* fix(router): carry a 429 status on cooldown and no-deployment errors RouterRateLimitError and RouterRateLimitErrorBasic had no status_code, so the proxy answered an all-deployments-cooled-down request as a generic 500 class error and logged the bug report notice for every cooldown. Both now carry status_code 429, which the proxy maps to a rate_limit_error and which the retry policy already treats as retryable * fix(proxy): start Sentry at config load and never attach request bodies Sentry only came up inside set_callbacks on the first LLM call, so boot and background errors were never reported, and general_settings.alerting: [sentry] always raised Missing SENTRY_DSN because the alerting path read a dead litellm.utils pin instead of the live SDK. The proxy now initializes Sentry once while loading the config whenever sentry is a configured callback or alerting target, the alerting and failure handlers read the live instance, and max_request_body_size is pinned to never so prompts and completions stay out of Sentry events * fix(tracing): log Lens failures at ERROR and export row counters to Prometheus A tracing store that fails to start, or a Lens service that rejects a batch, was only logged at WARNING, so operators running at the default ERROR level never saw tracing silently go dark. Both now log at ERROR and name the lens store. The exporter's rows_written and rows_dropped tallies were never exported, so the two counts are now Prometheus counters litellm_lens_export_rows_written_total and litellm_lens_export_rows_dropped_total, registered lazily on the default registry like the other proxy-level counters and charted on the all-metrics Grafana dashboard * test(proxy): drop the litellm import the alerting tests no longer use * fix(proxy): start Sentry once a DSN is set, from any config source Boot-time init used to lock Sentry in even with no SENTRY_DSN, so a DSN loaded later from DB environment_variables was never picked up. initialize_sentry now skips when no DSN is set and re-inits when the DSN changes. alerting_handler calls it too, so alerting: [sentry] set through the DB works. Drops the litellm_settings.callbacks check since that config fails to load with sentry in it * fix(sentry): keep stack frame locals out of events unless PII is opted in Frame locals carried prompts (messages, request bodies, provider options) into every error event. include_local_variables now follows SENTRY_SEND_DEFAULT_PII * test(router): drop the test that only pinned the 429 class attribute * fix(proxy): start Sentry at boot when it is only an input callback * fix(proxy): keep Sentry init and alerting within the strict ruff and basedpyright ceilings --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
67bfb7a148
|
fix(ollama): prevent reused response ids from dropping spend logs (#39227)
* fix(spend_tracking): key completion spend logs off the per-call id so duplicate provider response ids cannot drop rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_tracking): narrow the provider response id with isinstance instead of cast Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gcs_bucket): store request payloads under the spend log request id so the logs viewer can find them Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_tracking): validate callback fields for current type gates * test(gcs_pubsub): assert provider response id in spend metadata * fix(ollama): normalize reused response ids automatically --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Dushyant Acharya
|
06efa5de85
|
fix(proxy): honor disable_end_user_cost_tracking in spend logs and end-user tables (#40945)
* fix(proxy): strictly honor disable_end_user_cost_tracking in SpendLogs * fix(proxy): drop end-user id on failed requests and ClickHouse rows when end-user cost tracking is off * fix(clickhouse): drop the end-user id from spend log metadata when end-user cost tracking is off * test(spend): cover disable_end_user_cost_tracking across endpoints, failures, cache hits and chaos --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
62af39aaf8
|
fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it (#46591)
* fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it With modify_params on, a trailing tool search turn whose call has a kept result got a placeholder answer from the factory and went out as a marked tool_use, but the cache control hook counted marks before that placeholder existed and skipped the mark, so a request at the cap reached Anthropic with 5 marks and failed with a 400. The hook now finds client answers on the sanitized messages, the same list the factory reads. Also make the replay checks cheaper (prefix check before validation, early return without provider_specific_fields, and look up client answers only when a tool call carries cache_control), drop the two audit cells that cover old gaps outside the tool search fix, and derive the pass-through test timeout from the graceful stop budget. * fix(anthropic): keep the duplicate tool result warning out of the cache control hook The hook finds client answers with only the step of the modify_params sanitize pass that pairs tool calls with their results, so a request with a duplicate tool result logs the warning once again instead of once per mark count. Also restore the Snowflake non-stream and cached-stream tool search cells as known bugs with pytest.skip instead of deleting them, and put the Anthropic pass-through test timeout back to 240 seconds. * fix(anthropic): count cache marks next to a message the pairing step cannot read With modify_params on and a marked tool call, the cache control hook paired tool calls with their results before counting. A message without a get method (a stray string or null in extra_body.messages, or an openai SDK ChatCompletionMessage) made that pairing raise AttributeError before any provider call, where main counted the marks and let the request through. The hook now pairs only when every message can be read the way the pairing step reads it, and otherwise counts on the messages as sent, like main did. * fix(anthropic): count cache marks after placing a later system message the way the request builder does * test(anthropic): cover a leading system prompt and a system message before the answer in the cache mark census * test(anthropic): align the tool search replay cells with main and cover a system message after the search turn The malformed kept result cells now expect the unanswered search call to be dropped, as main does since #45882. The stream cut cell checks streamed tool call names, since the kept search result names get_weather as a tool reference. The streamed response cache hit cell runs again because #46104 fixed it. The modify_params cap cell expects the four client marks, since the placeholder answer sends the search call as a plain tool_use with its mark, and a new cell covers a system message after the trailing search turn in both stream modes * fix(anthropic): count marks on the raw messages when they cannot be placed, and find client answers once per request * test(anthropic): cover marked tool calls next to a malformed tool id and a later system message * perf(anthropic): look up client tool answers only when a marked tool call is a server tool call Answered ids only decide whether a marked srvtoolu_ call is rebuilt as server_tool_use, so a request whose marked tool calls are all client calls no longer places system messages or runs the modify_params pairing step to count marks * test(anthropic): reach the client answer lookup with a marked search call in the malformed tool id cell * perf(anthropic): place mid-conversation system messages in linear time for models that keep them * fix(anthropic): count cache marks with the model's own system message placement and pair tool results in linear time * test(anthropic): cover modify_params dropping a tool answer that answers no call of the latest assistant turn * fix(anthropic): count marks with in-place system placement when the model lookup cannot confirm the flag * test(anthropic): cover a bare model with custom_llm_provider filling the cache mark budget through litellm.completion and Router * fix(anthropic): give unmapped models the request builder's converting placement and survive any model lookup failure * test(anthropic): cover an unmapped model name counting cache marks with the request builder's converting placement * fix(anthropic): count server tool call cache marks the request builder's way only on routes that build Anthropic Messages requests Databricks, OpenRouter and Bedrock Converse send every marked tool call as is, so the hook keeps the census main uses there. The seeding step carries custom_llm_provider on the injection points when a server tool call is marked, so a bare model name still resolves its route. * test(anthropic): cover two answered marked server tool calls before a system message on Databricks, OpenRouter and Bedrock Converse * fix(anthropic): keep main's cache mark count on GitHub Copilot and ChatGPT routes without resolving them, so a sync call with no stored login never starts the device flow * test(anthropic): cover a marked server tool call reaching the GitHub Copilot and ChatGPT login hosts like an unmarked one |
||
|
Mateo Wang
|
835038cbbc
|
fix(websearch_interception): intercept the bare web_search tool type on chat completions (#46736)
* fix(websearch_interception): intercept the bare web_search tool type on chat completions * fix(websearch_interception): match the bare web_search type only on chat completions /v1/messages keeps its existing detector, so a non-Anthropic model there no longer short-circuits a bare web_search tool into raw search results * test(websearch_interception): cover the bare web_search type on chat and through the messages adapter |
||
|
Weihong Xu
|
b4a63d3b6a
|
fix(s3): bound object key filenames (#40825)
* fix(s3): bound object key filenames * Update litellm/constants.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(s3): pin the 1024-byte full key boundary next to the filename limit --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
30ada9d26a
|
feat(batches): store each batch line's request and response in S3 and GCS (#41691)
* feat(batches): emit per-line JSONL batch records to callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): only emit line items for final batches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): bump soupsieve to 2.9.2 for GHSA-gjv8-xp57-g29c and GHSA-j934-xhv5-fg8f
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(batches): satisfy ruff format for line item logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for store_batch_line_items_in_callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): cover line-item edge shapes and general settings propagation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep deployment credentials out of callback-visible params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(batches): declare credential transport attribute on Logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): integration contract for per-line batch callback events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve store_batch_line_items_in_callbacks through the settings store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): assert per-line events beside the aggregate and create events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop resurrected use_legacy_interactions_schema shim from merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): assert line item events carry no credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reset batch line-item flag when the setting is absent
Removing store_batch_line_items_in_callbacks from the DB general settings left the module flag at its previous value because the effect early-returned when the key was neither in the DB nor owned by YAML. The proxy owns the global, so an absent resolved value now writes False.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): map anthropic batch lines through the anthropic message transformer
Anthropic result lines carry a message-shaped body, so ModelResponse(**body) produced empty choices and a usage key that blocked the batch-rate usage attach. Extract the existing AnthropicResponse -> ModelResponse conversion used for /v1/messages logging into anthropic_message_to_model_response and reuse it for anthropic line items.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: tag the anthropic result cast with a cast-ok reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: shorten cast-ok reason so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip line-item callbacks for providers without a line reconstruction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): reconstruct bedrock and mistral batch lines through their provider transformations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: wrap widened provider literals so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): pair bedrock lines by recordId and type them from the reconstructed response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep line item events out of proxy limiters and strip credentials from child params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep cast-ok tags on the flagged cast lines for the discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip failed line item events in parallel request limiter failure hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): tolerate lifecycle spend rows when asserting no line item rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): place mutable-ok tags on the lines the discipline gate flags
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip router budget accounting for batch line item events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep anthropic line errors and skip native vertex batch rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): restore test-quality tags on line item tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move batch line-item unit test to tests/unit after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): emit batch line-item callbacks once per batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): release line-item claim when fan-out emitted nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): make the line-item claim release ownership-safe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep bodyless failed batch lines reaching callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): seed batch line hidden params in get_hidden_params defaults
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(batches): reuse fetched batch result files for line-item logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): release line-item claim when nothing was emitted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): type batch line-item callback sites to satisfy basedpyright budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: wrap batch hidden-params cast for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): reason suppressions for batch line-item lint deltas
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop stale mutable-ok suppressions after LIT002 removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters (#44486)
* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters
Batch line-item callbacks (call_type=acompletion + litellm_params.batch_parent_id)
bypassed the call-type-only batch guards:
- ClickHouseSpendLogger wrote one spend row per line on top of the aggregate
aretrieve_batch row, billing every batch twice with the flag on.
- Router deployment callbacks and the routing-strategy usage handlers charged the
historical batch's tokens/rpm to the live TPM/RPM counters, so after
Router.aretrieve_batch the next chat raised RouterRateLimitError.
Skip these events wherever the aggregate batch retrieve is already skipped.
* fix(logging): keep internal litellm_params authoritative in model_call_details
A request body key named litellm_params lands in optional_params, and
update_environment_variables merged **optional_params after the internal
entry, so a caller could overwrite the logging kwargs' litellm_params and
forge markers like batch_parent_id. Live-reproduced: a Gemini chat whose
body carried {"litellm_params": {"batch_parent_id": ...}} returned 200
with no ClickHouse spend row. Re-assert the internal litellm_params after
the merge so every sink (ClickHouse, Postgres, router quota counters)
reads the same trusted value, and assert the aggregate aretrieve_batch
row still writes in the ClickHouse line-item test (review finding).
* fix(clickhouse): skip failed batch line items in the spend sink too
A failed JSONL line is part of the aggregate aretrieve_batch row's
request counts; writing it its own spend_logs row inflated request
counts with the line-item flag on. Extend the batch line-item guard to
the failure handler (review finding) and pin both directions: a failed
line item writes no row, a genuine live failure still does.
* fix(batches): declare batch_line_items service family on claim calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(integrations): keep batch line items out of the built-in metering sinks
With store_batch_line_items_in_callbacks on, per-line batch events carry a
real response_cost and reached six metering sinks that lacked the
is_batch_line_item_event guard, so a completed batch metered aggregate +
per-line (roughly 2x true spend): prometheus (litellm_spend_metric and
request counters), openmeter, lago, datadog cost management (FOCUS
BilledCost), newrelic metrics, and the OTel v1/v2 gen_ai.usage.cost
metrics. Skip line-item events in each sink's success/failure metering
entry points, matching the guards already on the Postgres and ClickHouse
spend sinks. OTel spans for line items are still emitted; only their
cost/token metrics are skipped. Tests drive real sink instances on the
real dispatch lists through a completed 2-line batch and assert each
sink meters exactly the aggregate.
* style: format metering-sinks test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): skip synthetic line-item durations in v2 metrics; isolate prometheus test
Follow-ups from review on the sink guards: record_failure() now skips
batch line items too (a failed line's interval measures retrieval and
emission, not its model call), and the prometheus guard test snapshots
and restores the process-wide registry collectors instead of leaving
them unregistered. The OTel v2 test is now a flag on/off A/B over the
whole metric snapshot: cost and usage values must match exactly and no
metric may gain datapoints with the flag on.
* chore: finish main merge conflict resolution for batch line items
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: sort imports in litellm_logging after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): use the public hook class names main now exports
* fix(batches): only the batch writer can mark a callback event as a line item
amoderation, transcription and speech build litellm_params from every request kwarg,
so a body field named batch_parent_id reached Logging.litellm_params and every
spend, budget and rate-limit hook skipped the request. update_environment_variables
now drops an incoming batch_parent_id; the line-item writer sets it through
Logging.mark_batch_line_item. Batch line hidden params are emitted only when set.
* chore(batches): keep the lint gates within main's ceilings after the merge
Typed the helpers main now types (Mapping litellm_params, Any Span), replaced the
banned cast with a literal match, dropped isinstance checks main made redundant,
and moved test-quality reasons onto the lines the gate reads.
* fix(anthropic): read the message id off the validated message
The anthropic_messages logging path can hand over an already parsed message object,
which has no .get; main read its id through getattr, so take it from the validated
AnthropicResponse instead.
* fix(batches): an unreadable error file no longer drops the output line events
Costing counts an error file it could not fetch as no failed lines; line logging
refetched it and the raise aborted the whole fan-out, so the successful lines were
never emitted. The error file now reads as empty with a warning.
* test(batches): pin billing for batch_parent_id on every endpoint and value shape
Chat completions, embeddings, responses and messages join moderations, speech and
transcriptions, and speech is driven with an empty, int, list, object and 5 KB value.
* test(batches): bill every request of a concurrent burst mixing forged and plain bodies
* revert: keep the line-item fan-out all-or-nothing on an unreadable error file
|