Commit graph

54329 commits

Author SHA1 Message Date
yujonglee
271b4f0306
refactor(rust): one stream codec shape per API format (#45996)
Chat Completions, Messages and Responses each get a streaming module with
the same three names, event_stream, decode and encode, built on a shared
SSE helper in base_llm/sse.rs that owns the [DONE] marker. Callers use the
module path instead of vendor-prefixed names. Encoding matches what the
Python proxy writes to clients: Messages frames carry an event line, the
OpenAI formats are bare data frames.

Responses gets one event type for SSE and WebSocket. ResponsesEventType
names every event on OpenAI's streaming-events reference, with unknown
names riding in Other, and is_terminal and is_failure follow Python's
iterator. The framer adds a bulk decode_all and an SseBlockCodec that
keeps each event's wire bytes. An event the stream ends inside is dropped,
as the SSE spec and the openai SDK's decoder do.

ChatCompletionChunk no longer requires id, created or object, since
compatible hosts omit them.
2026-10-13 15:30:14 -07:00
ryan-crabbe-berri
7bd108139f
fix(openai): carry cache_control to prompt_cache_breakpoint on /v1/responses to gpt-5.6 (#46105)
* fix(openai): carry cache_control to prompt_cache_breakpoint on /v1/responses to gpt-5.6

Native /v1/responses used to strip every Anthropic cache_control mark, so a
gpt-5.6 deployment in explicit mode never wrote a cache. Marks on system,
developer and user parts and on function_call_output now become
prompt_cache_breakpoint on the same block when the deployment reads
breakpoints, and are stripped exactly as before everywhere else

* fix(openai): read str providers and base_url in the prompt cache breakpoint gate

Responses configs generated from a provider JSON file, and openai_like
itself, name their provider as a plain str, so .value crashed every native
Responses request through them. The gate also only read api_base, while
responses() keeps a caller's base_url as an extra field on the params, so an
injected mark on a custom host was rewritten to a breakpoint

* test(openai): cover the pass-through branches of the prompt cache breakpoint swap

A part type OpenAI does not cache keeps no breakpoint and an item mark skips
it, and an item or content of a shape OpenAI will reject reaches the strip
untouched instead of failing inside the proxy

* fix(prompt_caching): keep a client-written prompt_cache_breakpoint when a cache_control mark lands on the same block
2026-10-13 15:23:07 -07:00
Yassin Kortam
92faffa78d
fix(proxy): bound SIGTERM connection wait by GRACEFUL_SHUTDOWN_TIMEOUT (#46186)
Resolves LIT-9448

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 14:47:35 -07:00
Yassin Kortam
ec28acf322
fix(mcp): resolve a bare tool name on a single-server scope against that server (#46225)
A JSON-RPC tools/call on /{server_name}/mcp (or /mcp with x-mcp-servers naming one server) that uses the upstream's bare tool name answered "Tool not found" on every proxy process that had not yet listed that server, because the per-process tool-name map stays empty for a server whose upstream only lists tools with the caller's own credentials, and mcp SDK 2 no longer lists before an uncached call. call_mcp_tool now hands the single scoped server to execute_mcp_tool as requested_server_id, so the existing REST contract lists the server with the caller's headers and routes the bare name to it, and a bare name shared by two servers goes to the one the path names

Co-authored-by: aayushbaluni <73417844+aayushbaluni@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 14:39:52 -07:00
devin-ai-integration[bot]
d9123f4595
feat(realtime): proxy OpenAI Live sessions on /v1/live/sessions (#43621)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 13:59:27 -07:00
berriai-litellm-provider-info-sync[bot]
426ab45f2d
fix(cost-map): set together_ai Qwen3.7-Max to published price and restore 2026-09-15 removal dates (#46281)
Price-Sync: litellm-providers

Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
2026-10-13 13:31:37 -07:00
Yassin Kortam
581449b4d8
fix(proxy): warn when --max_requests_before_restart runs with a single worker (#46185)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 20:26:19 +00:00
berriai-litellm-provider-info-sync[bot]
eacc143dca
feat(pricing): add github_copilot/gemini-4-argon (#46280)
Price-Sync: litellm-providers

Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
2026-10-13 13:25:31 -07:00
mrinal-berri
93b29c3977
fix(proxy): broadcast team cache invalidation on team writes (#46113)
* fix(proxy): broadcast team cache invalidation on team writes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): cover cross-instance team cache invalidation

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 16:25:22 -04:00
devin-ai-integration[bot]
74e5dc9747
fix(proxy): keep provider refusal, logprobs and system_fingerprint in chat completions (#44788)
* fix(proxy): keep provider refusal, logprobs and system_fingerprint in chat completions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): handle missing chat completion roles

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(types): restore throwaway Usage skip in ModelResponse

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): handle explicit fields on custom response dumps

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: update refusal expectations for LIT-9235

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(lint): suppress type discipline violations with reasons

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style: ruff format proxy utils

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(types): keep basedpyright counts at merge-base for azure dump and ModelResponse

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(types): keep an explicit null system_fingerprint passed to ModelResponse

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 13:17:40 -07:00
Kerry Lu
d7b58126a7
feat(ui): source Home What's new from day 0 support blog posts (#46260)
* feat(ui): source Home What's new from day 0 support blog posts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(ui): Home cards read card_title and card_summary from blog tags with provider logo

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 13:11:59 -07:00
devin-ai-integration[bot]
829bd6a8cc
fix(model-prices): add TokenHub rows, fix TokenHub DeepSeek-V4-Pro price, Bedrock Claude 3.5 Sonnet and Fireworks context windows, Together dates (#46206)
* fix(model-prices): add TokenHub DeepSeek, GLM and Kimi rows, correct TokenHub DeepSeek-V4-Pro price, update Together removal dates

Co-authored-by: halewang <halewang@tencent.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(model-prices): correct bare Bedrock Claude 3.5 Sonnet context window

Align bare Sonnet entries with their regional 200K context metadata.

Co-authored-by: OrangerXu <OrangerXu@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(model-prices): correct Fireworks serverless context windows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: halewang <halewang@tencent.com>
Co-authored-by: OrangerXu <OrangerXu@users.noreply.github.com>
2026-10-13 12:52:01 -07:00
yuneng-jiang
1587faeeb9
test(unit): wait on the work itself instead of fixed sleeps (#46261)
* test(unit): wait on the work itself instead of fixed sleeps in 21 flaky files

Replace sleeps that raced background logging, cache writes, refreshes and
thread work with either awaiting that work directly (logging worker flush,
the batch logger's flush_queue, the tasks the call spawned, thread joins) or
an Event the test's own fake sets when it runs. Each wait keeps a 5s hang
guard that never decides pass or fail. Tests that need a fake clock, real
sockets, or measure latency are left unchanged, and no production code moves

* test(unit): await the tasks each completion spawns instead of yielding once

The logging tests yielded once with asyncio.sleep(0) so the task that
enqueues the success log could run before the worker flush. That relied
on the enqueue happening before the task's first await. Snapshot the
running tasks around each acompletion call and await the ones it spawned,
so the guarantee holds however many steps sit before the enqueue
2026-10-13 12:41:06 -07:00
moyai-devin-berriai[bot]
0b3fcc197d
fix(logging): redact nested callback input and provider response fields (#45923)
* fix(logging): redact nested callback input and provider response fields

* fix(logging): keep shared request body for per-callback redaction and drop hidden provider fields

The deprecated per-callback message_logging=False path ran perform_redaction on
the shared model_call_details, which now also redacted the nested
proxy_server_request body. The spend log Request column and
previous_response_id replay read that body, so one opted-out callback blanked
them for everyone. That path now keeps the shared body and hidden params, and
redact_streaming_responses_for_custom_logger hands the opted-out callback its
own redacted litellm_params.

Redacted litellm_params also clear provider_specific_fields copied into
metadata.hidden_params (Anthropic thinking text), in litellm_params and in the
nested request body, and the nested body redacts system and instructions.

* fix(logging): keep replayed instructions and router usage marker under redaction

The shared request body keeps system and instructions again, so spend logs
store them and previous_response_id follow-ups replay the real instructions.
Per-callback copies still redact them.

Provider-specific fields in metadata.hidden_params are cleared in place, so
the metadata dict keeps its identity and the router's counted-usage marker
stays visible to deployment_callback_on_success (no double tpm/rpm count).

* fix(logging): never copy shared metadata dicts during redaction

The shared redaction pass now leaves litellm_params metadata dicts as they
are, so a hidden_params write from the other success handler between the
in-place clear and the copy can no longer move the router usage marker to a
stale dict. The key-presence predicate change is reverted as unneeded.

* refactor(logging): redact nested request body and provider fields with the existing message rules

Run perform_redaction on a copy of proxy_server_request.body, the same call the
spend log already makes, and redact reasoning inside provider_specific_fields the
way the message's own reasoning_content and thinking_blocks are redacted, instead
of clearing every provider field. Opaque fields such as Gemini thought_signatures
stay, and the metadata dicts and streaming objects are handled as on main.

---------

Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
2026-10-13 12:25:56 -07:00
Krrish Dholakia
719c44db3f
fix(proxy): raise on provider count_tokens errors instead of falling back to tiktoken (#46121)
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
2026-10-13 19:08:01 +00:00
yucheng-berri
784581bace
fix(proxy)!: return provider-matching 4xx for the LIT-8062 status code follow-ups (#45537)
* fix(proxy): return provider-matching 4xx for the LIT-8062 status code follow-ups

- OCR multipart errors are 400 invalid_request_error on the file param
- a missing model is a 400 on chat, responses and messages even with a wildcard deployment
- responses max_output_tokens <= 0 reaches the provider instead of being raised to 16
- daily activity rejects end_date before start_date with a 400
- vector store file attributes are sent in full so the provider enforces its key limit
- router-wide defaults no longer satisfy a param the Router takes positionally, or a model the router does not serve
- a JSON null user_config is treated as absent

* chore: keep the type gates at their merge-base counts

* fix(proxy): keep fixed-target wildcards serving a missing model and close router-default gaps

- a missing model is rejected only when the wildcard would forward it into the target
- moderation only counts router defaults for models get_model_list lists
- user_config defaults and deployment params never satisfy a positionally bound param

* test(integration): pin the fixed-target wildcard path and no provider call on positional-param 400s

* chore: validate pattern-matched deployments instead of reading untyped dicts

* fix(proxy): keep router defaults counting for a2a agent models

* test: drop the signature-walk test and type the new test helpers

* test(integration): audit matrix for the status-code follow-ups

Drives every mode the fix branches on through a real proxy, Postgres, Redis and
the scripted upstream: positional params against router, deployment and
user_config defaults; moderation listed vs unlisted; unknown and a2a models;
every user_config shape on chat, responses and messages, streaming and SDK
clients; forwarding vs fixed-target wildcards for missing, null and empty
models; Responses max_output_tokens edges; OCR parse errors; vector store file
attributes; every daily activity route; and a provider outage mid burst plus a
killed worker on a two-worker proxy. Product gaps found are skipped as BUG.

* style: ruff format the audit test files

* test: name the auth pre-check as the source of the JSON array 500

* test: build the audit helpers without mutation and share one sentinel

* test: build every audit request body as one literal

* fix(proxy): keep router defaults and model-less dispatch for requests a fallback chain serves

* fix(proxy): count request, key and team fallbacks when an unlisted model inherits router defaults

* fix(proxy): skip key and team fallbacks on the client-credential dispatch path

* fix(proxy): keep router defaults when the router's own * fallback serves an unlisted model

* fix(proxy): count client-style fallbacks, forward an empty model, report invalid usage chat dates

- The router-default pre-check now treats client-style request, key and team fallbacks
  ([{"model": ...}]) as serving an unlisted model, as the router does
- The forwarding-wildcard gate only rejects a missing or null model; an empty model
  reaches the provider as before (vLLM serves its loaded model for it)
- The /usage/ai/chat tools hand the model the date-range error instead of an empty
  dataset or a generic fetch error

* test(usage): type the invalid-date tool test

* fix(proxy): keep rejecting an empty model behind a forwarding wildcard

Key and team model checks skip an empty model, so forwarding it let a key
restricted to another model reach a server's default model (vLLM serves
its loaded model for ""). Pin that with a restricted-key integration cell.

* fix(usage): report the query layer's date 400 to the usage chat model

Checking dates before the fetch refused formats the global usage tool's query
accepts (2026/10/01 returned spend on main). Catch the daily activity query's
own 400 instead, so each tool rejects exactly what it rejected before and only
the message changes.

* fix(proxy): restore three model-less and unlisted-model paths main served

An empty model is looked up as sent, so next to a forwarding */* wildcard it
still reaches the fixed * deployment it matches. A null model with a prompt_id
is left to the prompt manager, which may pick the model. A Claude Code
subagent turn on a session-bound auto/complexity router counts as served, so
the router default max_tokens still applies.

* fix(proxy): only skip the model check for a prompt_id when a prompt manager is registered

Without a callback prompt manager nothing can pick the model, so a null model
with a prompt_id behind a forwarding wildcard gets the same 400 as any other
null model.

* fix(proxy): leave a model-less Claude Code subagent turn to its session router

A subagent turn on a session bound to an auto, complexity, adaptive or
quality router is answered by that router whatever model it names, so a
null or missing model behind a forwarding wildcard is no longer rejected.
An empty model stays rejected.

* fix(proxy): skip the wildcard model check only for a bound Claude Code session

Read the session's router binding, as the router does, instead of only
checking that the request could resume one, so a subagent turn without a
model on an unbound session is still rejected before a forwarding
wildcard sends it upstream.

* refactor(proxy): stop predicting routing in the proxy pre-checks

- drop the wildcard missing-model gate: a model-less request behind a forwarding wildcard goes to the router and gets the provider's own status, as on main
- drop the "model the router serves" check: router defaults count for any named model again, except moderation, where the Router merges them only for listed models
- keep the positional-param, moderation and null user_config fixes

* test(ocr): cover the form, size, empty body, invalid JSON and file-document 400s

---------

Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
2026-10-13 11:42:39 -07:00
Mateo Wang
fb051f9647
chore: bump litellm-enterprise 0.1.76 -> 0.1.77, litellm-proxy-extras 0.4.108 -> 0.4.109, litellm 1.106.0 -> 1.107.0 (#46239)
* chore: bump litellm 1.106.0 -> 1.107.0

* chore: bump litellm-enterprise 0.1.76 -> 0.1.77, litellm-proxy-extras 0.4.108 -> 0.4.109

---------

Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
2026-10-13 18:12:54 +00:00
berriai-litellm-provider-info-sync[bot]
cb94567ed7
feat(vertex-ai): add gemini-4-argon-cyber-permissive (#46249)
Price-Sync: litellm-providers

Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
2026-10-13 11:03:58 -07:00
Kerry Lu
61841887f3
fix(pricing): drop duplicate and bare gemini-4-argon rows (#46246)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 17:44:26 +00:00
VimarshDwivedi
22c9e383f3
fix(mcp): let tool-argument validation calls set their own timeout budget (#46184)
test_cold_concurrent_schema_validation_accepts_valid_arguments pins the
process to a single CPU core and fans out 8 concurrent calls through
anyio.to_process's subprocess worker pool, each needing to cold-spawn a
fresh interpreter and import jsonschema. Under a slow or contended CI
runner that routinely exceeds the production 30-second budget before any
actual validation work runs, failing the test on a genuine timeout rather
than a validation bug.

_tool_argument_validation_error now takes an optional timeout_seconds,
defaulting to the existing constant, so production behavior is
unchanged. The test passes a timeout sized for its own single-core,
cold-start stress scenario, matching the margin the companion
integration test (test_schema_validation_stops_expensive_work_and_recovers)
already grants a similar concurrency=8 case.
2026-10-13 17:15:22 +00:00
yuneng-jiang
ec58b889cc
test(integration): isolate shared proxy state behind six order-dependent flakes (#46172)
Each test below passed or failed on the same commit depending on test order
or port timing in the CircleCI integration workflow

- readiness probe only polls a port the launched proxy owns, so a stolen
  port no longer sends GET /health/integration-readiness to another test's
  wire server
- team update tests seed fresh member users, since a proxy admin relocation
  adds team members to the destination org and leaked org_b membership into
  the shared INTERNAL_USER
- realtime whisper tests wait until every proxy worker serves only the new
  gpt-realtime-whisper deployment, via a per-worker deployment id route
- configured fallbacks test runs on its own proxy and database, because
  /config/update persists router settings and cannot unset fallbacks, which
  left [] and changed the no-fallback error text other tests assert
- otel excluded services tests each get a private Redis so a sibling proxy
  cannot warm the budget registry cache the tenant span depends on
- Langfuse audit upstream uses unique response ids instead of len(targets),
  which could repeat under concurrency and drop SpendLogs rows
2026-10-13 09:55:11 -07:00
Kerry Lu
59ef56f636
feat(pricing): add vertex gemini-4-argon (#46237)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 16:50:01 +00:00
berriai-litellm-provider-info-sync[bot]
d7fca3dad0
fix(gemini): take gemini-4-argon max output tokens from the model docs (#46236)
Price-Sync: litellm-providers

Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
2026-10-13 09:49:18 -07:00
berriai-litellm-provider-info-sync[bot]
3e16bf9518
feat(vertex-ai): add gemini-4-argon (#46228)
Price-Sync: litellm-providers

Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
2026-10-13 09:44:02 -07:00
berriai-litellm-provider-info-sync[bot]
9dc6d9a3ee
chore(prices): add gemini/gemini-4-argon from the Gemini pricing page (#46227)
Price-Sync: litellm-providers

Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
2026-10-13 09:28:27 -07:00
berriai-litellm-provider-info-sync[bot]
948b88bfeb
fix(cost-map): restore together_ai Kimi-K3 standard price after promo ended (#46226)
Price-Sync: litellm-providers

Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
2026-10-13 09:21:51 -07:00
Kerry Lu
5f74a79c75
test(cost): add xai grok-4.7, Bedrock runtime chat completions and Bedrock Mantle cost cases (#46154)
* test(cost): add xai grok-4.7, Bedrock runtime chat completions and Bedrock Mantle cost cases

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost): bring back known_bug and carry the xai grok-4.7 stream breakdown bug as a strict xfail case

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 09:06:19 -07:00
Kerry Lu
79b93aab2c
test(translation): add StreamingTranslationTestCase with a Claude Sonnet 5.5 chat completions case (#46151) 2026-10-13 08:18:29 -07:00
devin-ai-integration[bot]
9f511e1b25
fix(db): partition runbook moves the call-id index off the legacy spend logs table (#46131) 2026-10-13 08:17:48 -07:00
moyai-devin-berriai[bot]
0976c0889a
fix(proxy): enforce fallback model access by default (#46127) 2026-10-13 07:58:53 -07:00
devin-ai-integration[bot]
6b7d1928b8
fix(a2a): add per-agent a2a_protocol_version override for langgraph-api 0.15 cards (#44463)
* fix(a2a): add per-agent a2a_protocol_version override for langgraph-api 0.15 cards

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(a2a): cover streamed 0.3 replies through the real client

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(a2a): accept new kwargs in create_a2a_client fake; add protocol-version integration tests

The create_a2a_client fake in tests/agent_tests/test_a2a_agent.py listed a
fixed signature, so asend_message's new protocol_version= kwarg raised
TypeError in test_a2a_non_streaming and test_a2a_streaming. Accept and
ignore extra kwargs with **_, matching the new _capture fake in test_main.py.

Add tests/integration/compatibility cells for litellm_params.a2a_protocol_version
against a real 2-worker proxy with Postgres and Redis and scripted A2A peers
modeled on langgraph-api 0.15.4: pinned send/stream on all three routes, aliases,
the 1.0 opt-out, unchanged genuine v1 agents, PATCH taking effect on both
workers, persistence via /v1/agents, the localhost retry, invalid values,
upstream errors, the dialect hint, and the completion bridge stripping the key.

Merge base 5f969982fc: 33 red, 7 green. Head ba709d3de5: 40/40 green twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 09:04:36 -04:00
berriai-litellm-provider-info-sync[bot]
7dd53d6adc
feat(azure): add azure_ai/claude-fable-5-5 (#46201)
Price-Sync: litellm-providers

Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
2026-10-13 05:27:21 -07:00
berriai-litellm-provider-info-sync[bot]
e5db66df92
fix(azure): sync embedding, tts-hd and legacy gpt prices with the azure-openai pricing page (#46200)
Price-Sync: litellm-providers

Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
2026-10-13 04:25:46 -07:00
devin-ai-integration[bot]
77005080f3
refactor(types): replace Any with proven types in 6 files (#46042)
* refactor(types): replace Any with proven types in 8 files

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(types): keep azure search and google_genai seams untyped

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 10:00:52 +00:00
yucheng-berri
a5d05355b6
feat(focus): add ternary destination for FOCUS export (#46106)
* feat(focus): add ternary destination for FOCUS export

Add a `ternary` FOCUS export destination (plus a `ternary` callback) so a LiteLLM proxy can push FOCUS-formatted spend to Ternary with `callbacks: ["ternary"]` -- no separate export job

- FocusTernaryDestination: a pure sink modeled on the sibling vantage/mavvrik/gcs destinations; POSTs the FOCUS CSV to the Ternary cost-ingestion endpoint, chunking an oversized backfill (csv-module repartition, quoted-newline safe; oversized single row raises rather than silently dropping) with a stable X-Ternary-Upload-Id + 0-based index/total so the receiver stages-then-swaps. connection_id is validated + URL-encoded; base_url must be HTTPS (loopback exempt for local); the upload has a bounded timeout; it aborts on the first failed chunk so nothing partial lands
- TernaryLogger: presets the FOCUS pipeline (CSV) and enriches the FOCUS Tags JSON with the per-request token counts the shared transformer drops (Ternary weights allocation by tokens); the enrichment is confined to this path and only adds keys. Cadence is daily (default) or interval (validation loops, an explicit interval-seconds required); the export window is day-aligned and drops days before the window start so a whole-day-replace receiver never truncates
- Registration mirrors vantage/mavvrik: ternary callback literal, logging dispatch, scheduler job, and CustomLoggerRegistry entry -- no shared-code changes
- Tests: mocked unit tests only (no live calls), 100% line coverage of both new modules

* fix(focus): warn instead of silently dropping token tags on mismatch

_merge_token_tags returned the frame unchanged when the transformed data
had no `Tags` column or its row count disagreed with the source, so every
token count was dropped with no signal -- the receiver then lands NULLs
that read as "this gateway reports no tokens". Emit a warning on each
degenerate path (mirroring _drop_days_before), and cover both with caplog
tests asserting the warning and the unchanged return.

* feat(focus): upload one UTC day per Ternary signed-URL PUT

Replace the multipart POST with chunking by a per-day signed-URL upload,
matching the Mavvrik destination's pattern. The logger partitions the raw
usage rows on `date` (dropping unparseable dates with a warning), then
transforms, enriches and delivers each day separately; one day's failure
is logged and the remaining days still upload, with a single error raised
at the end. The destination requests a signed URL for the day, validates it
is an HTTPS GCS URL, and PUTs the CSV with exactly the signed headers.

Removes the 30 MiB / 100k-row chunking, the X-Ternary-* upload headers and
the .partN filenames; a re-sent day overwrites its object, so retries stay
idempotent without them.

* fix(focus): re-send the last two whole days so late-flushed spend lands

LiteLLM writes DailyUserSpend in batches, so spend dated yesterday can be
flushed after the 00:05 UTC daily export. That export only covered the
previous UTC midnight onward, and the next day's export moved past it, so
the tail of each day was never sent. Start the window at UTC midnight two
days back: the following run re-sends yesterday whole, overwriting its
per-day object with the complete day.

* fix(focus): lazy-load the Ternary engine, fetch whole days, keep ternary out of the generic factory

Every proxy imported polars at boot through the logger registry, so a proxy without polars failed to start even with no FOCUS callback. The polars-backed engine now lives in its own module and loads on the first export

A non-UTC Postgres session timezone shifted the naive updated_at filter, so the oldest day in each run lost rows and Ternary's whole-day replace made the loss permanent. The Ternary read now starts one day early with no upper bound and keeps rows by their own UTC date

callbacks: ["focus"] with FOCUS_PROVIDER=ternary uploaded a parquet blob of one interval over a whole-day CSV file. The generic factory no longer builds a Ternary destination and the Ternary engine injects its own

* fix(focus): keep only the days a Ternary export window covers

Dropping the updated_at upper bound let a bounded export_usage_data window upload every day through today. Rows are now kept by their UTC date between the window's first day and the day before its end instant

* refactor(focus): leave the shared FOCUS export engine as on main

The Ternary engine sets the base engine's fields itself and passes its own
destination, so focus/export_engine.py and its test match main again.

---------

Co-authored-by: Aaron Sproul <aaron@ternary.app>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
2026-10-12 22:52:40 -07:00
yucheng-berri
c3ca8a5140
feat(guardrails): add Wingback guardrail integration (#46136)
* feat(guardrails): add Wingback guardrail integration

Mirrored from #38685 at d73b66544d (net diff against its merge base, applied on main)

* fix(guardrails): dispatch Wingback through the unified guardrail

The proxy only runs a unified guardrail when apply_guardrail is defined on
the guardrail's own class, so WingbackGuardrail now binds the parent's
decorated method in its class body. Chat, messages and responses pre, post
and during call checks reach Wingback again, and each check logs once.

Also types the initializer and config model for the lint gates and uses
monkeypatch.setenv in the unit test

* chore(guardrails): drop the Wingback dispatch comment

The unit and integration hook tests already fail when the apply_guardrail binding is removed, so the comment is redundant under the repo comment rule

---------

Co-authored-by: bsk-wb <bharath@wingback.ai>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
2026-10-12 22:18:10 -07:00
Kerry Lu
9621a60d27
test(cost): add long context tier cost cases for gpt-6 (272k) and claude-haiku-5-5 (100k) (#46147) 2026-10-13 04:29:02 +00:00
devin-ai-integration[bot]
fdab308646
fix(vertex_ai): keep Gemini cache creation token count out of request kwargs (#46152) 2026-10-12 21:19:12 -07:00
Kerry Lu
9395932cba
test(cost): add gpt-6 chat to Responses bridge cost parity pairs (#46143) 2026-10-12 21:14:11 -07:00
Kerry Lu
a41ff05084
test(cost): add Claude 5 /v1/responses bridge cost parity pairs (#46140)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 04:01:38 +00:00
Kerry Lu
ce6fcafd04
test(cost): add azure gpt-6 /v1/messages bridge cost parity pairs (#46139)
Some checks failed
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
unit / db-and-spend (push) Blocked by required conditions
unit / endpoints-and-responses (push) Blocked by required conditions
unit / enterprise-routing (push) Blocked by required conditions
unit / guardrails-hooks (push) Blocked by required conditions
unit / rust-bridge (push) Waiting to run
unit / assert-shard-coverage (push) Waiting to run
unit / enterprise-managed-files (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Waiting to run
unit / OpenAI 3.0.0 compatibility (push) Waiting to run
unit / OpenAI 3.25.0 compatibility (push) Waiting to run
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 03:26:46 +00:00
Kerry Lu
0eff31ae4e
fix(gemini): bill context cache creation tokens from cachedContents responses (#46144)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 03:18:50 +00:00
Kerry Lu
00e25792c5
test(cost): add Bedrock Converse Claude 5 /v1/responses bridge cost parity pairs (#46141)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 03:13:33 +00:00
devin-ai-integration[bot]
ff0c6e0a51
test(cost): add BridgeCostParityTestCase with a gpt-6-sol /v1/messages to Responses pair (#46130)
* test(cost): add BridgeParityTestCase with a gpt-6-sol /v1/messages to Responses pair

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost): rename BridgeParityTestCase to BridgeCostParityTestCase

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(cost): collect BRIDGE_COST_PARITY only from modules that define one

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 02:51:55 +00:00
Kerry Lu
24bbf8f0e1
fix(cost): bill azure_ai claude web searches under deployment-owned pricing ids (#46135)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 19:40:53 -07:00
devin-ai-integration[bot]
ffec84f918
feat(ui): give the Decisions playground a clearer request hierarchy (#45981)
* feat(ui): give the Decisions playground a clearer request hierarchy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): tie Decisions setup labels to their controls and cover the new hierarchy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): assert the Decisions actions live in the editor footer toolbar

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): scope the Decisions footer assertion to the editor card

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(ui): align the Decisions playground with the Magic Patterns mockup

Response panel is always visible beside the editor with an empty state, the
footer is one row with the validation summary and the actions, and answers
render as muted rows under a Response header with latency, model and usage.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): left align the empty Response header in the Decisions playground

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): expect the compact usage badge in the Decisions response header

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): let long model names wrap in the Decisions response header

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(ui): match the Decisions playground to the Magic Patterns v18 design

Start from an empty request with one blank question, move Clear to an icon-only eraser in the editor header next to a Load example menu, and leave Send alone in the footer

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): put the blank starting question in the payload and gate Send on context

LiteReview found the empty state rendering a question card that was not in the request, so Add question kept one card. The blank question now lives in emptyPayload for both endpoints, and validation adds a state or input issue while the context is blank so Send stays off until there is text

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): type the blank input cases as a plain list

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): clear the Decisions form to a truly blank question and report missing context first

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep the blank question name quiet until it is edited

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): name the blank question in the Decisions validation footer

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): label Decisions validation issues by field instead of JSON key

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): give the Decisions eraser the same icon color as the docs link

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 02:36:10 +00:00
devin-ai-integration[bot]
7265b0e889
fix(bedrock): bill the regional row for inference-profile ARNs that embed a cross-region prefix (#46128)
* fix(bedrock): bill the regional row for inference-profile ARNs that embed a cross-region prefix

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(bedrock): keep the ARN regional-row regression tests to the touched lines

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:56:43 -07:00
tin-berri
cca06b27e4
feat(ui): add Home entry and independent LiteAdmin chat tabs (#46068) 2026-10-12 18:30:01 -07:00
berriai-litellm-provider-info-sync[bot]
c64c1c8a9a
feat(vertex-ai): add gemini-robotics-er-2-preview (#46126)
Price-Sync: litellm-providers

Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com>
2026-10-12 18:26:42 -07:00
devin-ai-integration[bot]
bfb4f4b2c3
fix(databricks): accept service_tier on requests and carry the served tier on responses (#46111)
* fix(databricks): accept service_tier on requests and carry the served tier on responses

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(databricks): mark the in-place service_tier store as rebind-ok

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(databricks): wrap the service_tier store to the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(databricks): keep the rebind-ok suppression on the service_tier line

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(databricks): only set service_tier on responses when the upstream returned one

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(databricks): name the B010 suppression on the service_tier setattr

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(databricks): shorten the service_tier suppression to the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(responses): read service_tier from the Responses request only

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:22:10 -07:00