Commit graph

16410 commits

Author SHA1 Message Date
Sean Murphy
35c021b2a3 feat(opencode): add opencode_go and opencode_zen first-class providers
Adds OpenCode Zen and OpenCode Go as first-class LiteLLM providers,
each serving three wire formats: Chat Completions, Anthropic Messages,
and OpenAI Responses.

Routing between the three arms is decided in code, not from the runtime
cost map, because a published map predating this provider would send
every Messages-native model down the wrong wire. The model sets carry
the full forward-looking grid of model names so a gateway-side addition
routes correctly without a release; names without a bundled price simply
stay unpriced until a real one is published.

Cost resolution falls back to pricing bundled with the package when the
runtime cost map carries no usable entry -- the Router registers a bare
placeholder for every deployment at startup, so the guard asks for
pricing the cost calculator can actually use, not for the key's
presence.

The cost-map JSON schema gains a `messages` mode so the new entries
validate, and the provider tests set module-level configuration through
monkeypatch rather than writing process-wide globals directly.
2026-09-12 19:06:07 -07:00
devin-ai-integration[bot]
56045503db
fix(proxy): bound tool and guardrail index create_many by the spend-log statement budgets (#40561)
* fix(proxy): bound tool and guardrail index create_many by the spend-log statement budgets

One flush drains up to MAX_LOGS_PER_INTERVAL source transactions or logs, but a
transaction fans out to one LiteLLM_SpendLogToolIndex row per tool and a log
to one LiteLLM_SpendLogGuardrailIndex row per guardrail, so the index
create_many payload was unbounded. Both index writes now go through
spend_log_write_batches(SPEND_LOG_WRITE_BATCH_MAX_BYTES, SPEND_LOG_WRITE_BATCH_MAX_ROWS).
The tool index write moves out of the rollup batch_() so the split reduces
the query-engine payload; replayed index rows are no-ops under
skip_duplicates, and the daily rollup upserts stay in one transaction

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): pin the row budget in the index fan-out tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 19:00:47 -07:00
ryan-crabbe-berri
67dd150fed
Merge pull request #36363 from joshgarnett/bugfix/responses-reasoning-object-to-effort
fix(responses): translate the reasoning object into a chat-completion reasoning effort
2026-09-12 18:52:25 -07:00
ryan-crabbe-berri
a969319fb5
Merge pull request #36222 from BerriAI/litellm_lit_5292_model_info_pricing_filter
fix(model_management): stop persisting cost map pricing as a deployment override
2026-09-12 18:16:06 -07:00
ryan-crabbe-berri
0c83e831db fix(responses): carry the reasoning summary as an alias, not inside reasoning_effort
The bridge probe asked `responses_api_bridge_check` with the summary read straight off
the Responses object, but `litellm.completion` reads it from `optional_params` via
`peek_reasoning_summary_aliases`, which the bridged request never populated. So gpt-5,
gpt-5.1 and azure/gpt-5 answered "bridging" to the probe and "not bridging" for real,
and the object still landed on Chat Completions, which only takes a string

`reasoning_effort` is now always the effort string, and `summary` rides the
`reasoning_summary` alias that main.py already reassembles into `{effort, summary}` on
the bridged path. The alias is emitted only when the probe says the model bridges, so
no chat provider ever sees it, and the probe is now asked with the exact params this
transform emits
2026-09-12 17:49:30 -07:00
Mateo Wang
9d984371fd
Merge pull request #40909 from BerriAI/litellm_databricks_reasoning_effort_thinking
fix(databricks): translate reasoning_effort to thinking for Gemini 2.5
2026-09-12 17:46:44 -07:00
Joshua Garnett
1d5ed79931 fix(responses): translate the reasoning object into a chat-completion reasoning effort
The Responses API takes reasoning as an object, {effort, summary}. Chat
Completions takes reasoning_effort as a string enum and has no equivalent of
summary, but the completion bridge forwarded the whole object whenever summary
was set, which agentic clients set on every request.

Bedrock Converse guards its mapping with isinstance(value, str) and has no else
branch, so the object fell through, thinking was never enabled, and the caller
was billed for a non-thinking turn with nothing in the response to explain it.

The object is still forwarded for the one caller that can consume it: a model
whose cost-map mode is responses, which litellm.completion bridges back onto the
Responses API and reassembles {effort, summary} there. That decision is delegated
to responses_api_bridge_check, the same check litellm.completion runs, rather
than a second copy of the rule that could drift from it. An object carrying no
effort now yields no reasoning_effort at all.
2026-09-12 17:42:05 -07:00
ryan-crabbe-berri
76cb0fec1c fix(model_management): stop persisting cost map pricing as a deployment override
/model/info fills a deployment's missing pricing in from the model cost map so the
Admin UI has a rate to display. Clients echo that whole model_info blob back on save,
and update_db_model merged it into the row, so editing an unrelated setting turned
that day's catalog price into a real per-deployment override. After that the
deployment ignored the cost map and Reload Price Data could no longer move it,
because the reload replays each deployment's stored pricing over the fresh catalog.

Drop the derived pricing from incoming model_info on the two write paths. The
drop-set is read off the same objects the read path uses, CustomPricingLiteLLMParams
plus the tiered *_above_N_tokens pattern that get_model_info passes through and no
model declares, so it cannot drift as new rates are added. output_vector_size is
exempt: it lives on the pricing model but is an embedding dimension, not a rate.

A deployment's own pricing still rides litellm_params, which is untouched, as is the
explicit-null clear, which reads the incoming model rather than the filtered dict.
The filter sits in the endpoint bodies rather than _add_model_to_db, which master-key
rotation reuses to re-serialize every stored deployment.
2026-09-12 17:29:35 -07:00
ryan-crabbe-berri
d4a72e7372
Merge pull request #40907 from BerriAI/litellm_key_alias_substring_non_admin
fix(proxy): allow key_alias substring matching on /key/list for non-admins
2026-09-12 16:48:17 -07:00
kerry-berri
d565031b9c
Merge pull request #40902 from BerriAI/litellm_openai_reasoning_fallback_rule
feat(registry): add openai reasoning-family fallback generalization
2026-09-12 16:27:16 -07:00
ryan
8ccde3879d Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_key_alias_substring_non_admin 2026-09-12 23:06:23 +00:00
devin-ai-integration[bot]
b1360efc2f
fix(proxy): attribute gate-rejected requests to their endpoint in cache analytics (#40824)
* fix(proxy): attribute gate-rejected requests to their endpoint in cache analytics

Requests rejected before dispatch (bad key, blocked key, budget, rate limit, malformed body) were spend-logged with an empty call_type because the synthesized logging object never reached the failure lifter. The caching dashboard rolled all of them, plus failed calls on info routes such as /model/info, into one Unknown group.

Resolve call_type from the matched route first, falling back to body shape, and keep the synthesized logging object on request_data so the lifter sees it. Log bare auth exceptions with the 401 ProxyException the client gets so error_code is never empty. Exclude info routes from the cache analytics groups and error breakdown. The dashboard explains the Unknown group when older rows still produce one.

Resolves LIT-5884

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): keep the raw auth exception for failure callbacks

Record the client-facing status in the spend log through a separate client_exception argument so custom failure callbacks still receive the exception auth raised.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): keep the route for multi-operation endpoints and exclude info routes from cache filter options

Routes such as /v1/files map to several operations (create, list) and the
method is not available in the failure hook, so a rejected request there is
filed under its route instead of the first mapped call type. The key alias and
model filter-option queries now apply the same info-route exclusion as the
groups and error breakdown, so every offered filter value returns data. The
info-route exclusion and Unknown grouping are now covered against a real
Postgres in tests/proxy_behavior/spend/test_cache_activity.py

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): drop client_exception, the spend log row never used it

The DB spend row for a gate rejection is written by _ProxyDBLogger from the
original exception, so the status-bearing copy only reached the in-memory
logging payload. Live runs at the tip still recorded bare auth exceptions as
Unknown/Exception, the same as the base branch. Removing the plumbing keeps
this PR to endpoint attribution and the info-route exclusion

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 16:03:38 -07:00
ryan-crabbe-berri
c134fb7a38
Merge pull request #39395 from seyeong-han/litellm_meta_muse_voice_realtime
Some checks failed
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests / caching-local (push) Waiting to run
Unit Tests / core-utils (push) Waiting to run
Unit Tests / enterprise-package (push) Waiting to run
Unit Tests / enterprise-routing (push) Waiting to run
Unit Tests / integrations (push) Waiting to run
Unit Tests / All Other Providers (push) Waiting to run
Unit Tests / Vertex AI (push) Waiting to run
Unit Tests / misc (push) Waiting to run
Unit Tests / proxy-auth (push) Waiting to run
Unit Tests / proxy-endpoints (push) Waiting to run
Unit Tests / proxy-extras (push) Waiting to run
Unit Tests / proxy-infra (push) Waiting to run
Unit Tests / proxy-server (push) Waiting to run
Unit Tests / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
feat(realtime): add Meta Muse Voice transcription
2026-09-12 15:58:24 -07:00
Mateo Wang
566f026c1c
Merge pull request #40767 from BerriAI/litellm_ocr_custom_pricing
fix(cost): honour deployment custom pricing for OCR calls
2026-09-12 15:55:44 -07:00
mateo-berri
d7158ba795 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_ocr_custom_pricing 2026-09-12 15:42:59 -07:00
Mateo Wang
426e675c49
Merge pull request #40764 from BerriAI/litellm_redis_pool_timeout_counts_as_timeout
fix(redis): count pool wait timeouts as breaker timeouts
2026-09-12 15:42:57 -07:00
ryan-crabbe-berri
0e435e4148 fix(realtime): run transcription guardrails on transcription-only sessions
The provider_config path skipped run_realtime_guardrails for transcription
sessions to avoid sending response.create, which also dropped every
realtime_input_transcription guardrail: no violation error reached the
client and on_violation / end_session_after_n_fails never fired. Run the
guardrail for every completed transcript and only suppress response.create
when the session has no assistant turn.
2026-09-12 15:32:45 -07:00
ryan
02dbff485b test(proxy): mark prisma_client patch in /key/list alias test with test-quality reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 22:32:14 +00:00
Mateo Wang
fe5ff9d3b0
Merge pull request #40771 from BerriAI/litellm_regression_coverage_followup
test: tighten regression tests added in #37974
2026-09-12 15:29:36 -07:00
mateo-berri
51abb95de0 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_redis_pool_timeout_counts_as_timeout
# Conflicts:
#	tests/test_litellm/caching/test_redis_cache.py
2026-09-12 15:26:08 -07:00
Mateo Wang
e1bff56f0b
Merge pull request #40772 from BerriAI/litellm_lit6982_bucket_config_includes
fix(proxy): resolve config include directives for bucket-hosted configs
2026-09-12 15:21:56 -07:00
Mateo Wang
c040061f15
Merge pull request #40766 from BerriAI/litellm_fix_v1_messages_disconnect_partial_cost
fix(anthropic): price recovered tokens when a /v1/messages client disconnects mid-stream
2026-09-12 15:21:43 -07:00
Mateo Wang
d72ae3b9a2
Merge pull request #39895 from BerriAI/litellm_deflake_20260905
test: deflake redis semantic cache sys.modules leak, LangSmith init loop patch, wall-clock stagger assertion, and zombie grandchild check in the fake prisma cli
2026-09-12 15:20:26 -07:00
Mateo Wang
fc053dae3e
Merge pull request #40770 from BerriAI/litellm_lit6996_agent_skills_wellknown
feat(proxy): serve registered skills as an Agent Skills well-known index
2026-09-12 15:20:20 -07:00
Mateo Wang
f531537aa3
Merge pull request #40768 from BerriAI/litellm_prisma_cli_resolve
fix(proxy): run migrations through python -m prisma when the prisma console script is not on PATH
2026-09-12 15:20:06 -07:00
ryan
df6ec79810 test(proxy): assert /key/list alias substring results end to end for non-admins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 22:15:11 +00:00
ryan-crabbe-berri
4647cd1215 fix(realtime): keep a Muse turn active for turnless partials after speechEnd
Muse partials carry no turnId and belong to the most recent speechStart,
and the docs say the model may keep post processing a turn after speechEnd
until speechComplete. Releasing the active turn on speechEnd made any
partial arriving in that window raise and get dropped in ENDPOINTING mode.
The turn now stays active until its speechComplete or final transcript.
2026-09-12 15:13:44 -07:00
ryan-crabbe-berri
ba6a0c9fc6
Merge pull request #40652 from BerriAI/litellm_key_activity_search
feat(ui): search Key Activity by key alias, key hash, user id, or email
2026-09-12 15:05:53 -07:00
kerry
df272d7e2f fix(registry): limit reasoning fallback to single-digit gpt majors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 22:01:58 +00:00
kerry
ca35119168 style(tests): wrap oversized model tuples
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 21:56:11 +00:00
kerry-berri
0f03fc2985
Merge pull request #40901 from BerriAI/litellm_remove_fireworks_price_snapshot_tests
test(fireworks): stop pinning prices in the cost-map tests
2026-09-12 14:53:58 -07:00
ryan
20787ba186 fix(proxy): allow key_alias substring matching on /key/list for non-admins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 21:52:49 +00:00
devin-ai-integration[bot]
5b36de4646
fix(guardrails): log mask when a guardrail adds request keys (#40882)
* fix(guardrails): log mask when a guardrail adds request keys

_inputs_were_modified only compared keys present in the pre-hook baseline, so a
guardrail that injected a new key such as tools was logged as allow. Compare over
the union of both key sets, and narrow the pre_call return value to the same
prompt-bearing keys the baseline holds so passthrough stays allow.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): snapshot apply_guardrail inputs before the hook mutates them

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 14:39:40 -07:00
kerry
543ed2f6da fix(registry): scope codex/deep-research/chat-latest markers to gpt bases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 21:23:24 +00:00
yuneng-jiang
883b722fd2
Merge pull request #40895 from BerriAI/litellm_budget_clear_persistence
fix(ui): persist cleared budgets and reset intervals
2026-09-12 14:22:24 -07:00
kerry
fbcc602122 test(fireworks): stop pinning prices in the cost-map tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 21:19:35 +00:00
devin-ai-integration[bot]
311d9bba37
perf(policy_engine): dedup attachments in one pass after sorting (#40883)
get_attached_policies_with_reasons rescanned the sorted matches with next() once
per distinct policy, which is quadratic and misses the one second budget past a
few thousand global attachments. Build a policy to broadest attachment map in one
pass instead, keeping the specificity sort and result order.

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 14:19:20 -07:00
kerry
db6b851884 feat(registry): add openai reasoning-family fallback generalization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 21:14:24 +00:00
kerry
09d63b259c test(fireworks): drop hardcoded price snapshot tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 21:11:02 +00:00
devin-ai-integration[bot]
261807114a
fix(proxy): accept both deferred stream logging arg shapes on native routes (#40869)
* fix(proxy): accept both deferred stream logging arg shapes on native routes

_arm_deferred_stream_dispatch armed a one-argument closure on every
anthropic_messages/aresponses stream that was not a CustomStreamWrapper or a
LiteLLMCompletionStreamingIterator. The bridged /v1/messages path returns a
plain SSE generator that shares its inner CustomStreamWrapper logging_obj, so
it stores (assembled_response, cache_hit) and _fire_deferred_stream_logging
raised TypeError, dropping spend logs and callbacks and ending the stream with
an error. The closure now dispatches on the stored args shape: a single
coroutine is enqueued, a two-tuple runs success handlers, anything else is
logged and dropped

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): assert dropped deferred payload via caplog instead of patching the logger

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 14:10:58 -07:00
Yuneng Jiang
ad966d8340
fix(projects): persist explicit budget cap clears 2026-09-12 13:43:55 -07:00
Yuneng Jiang
f5f81e973a
fix(budgets): preserve explicit reset interval clears 2026-09-12 13:43:55 -07:00
mateo-berri
c7b607c46e fix(databricks): keep the Claude fallback when gating the anthropic thinking payload
Some checks failed
ai-gateway image / ai-gateway release image (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
Gate the reasoning_effort translation on the cost-map flag or the model name containing
claude, so unmapped Claude serving endpoints keep translating. Flag the newer Claude
entries that were missing it. Expose supports_anthropic_thinking_payload as a public
helper next to the other supports_* wrappers instead of importing the private factory.
Drop the adaptive-only guard, since the adaptive flags only ever match Claude ids, and
add regression tests for an unmapped Claude endpoint and an adaptive Claude model
2026-09-12 13:13:38 -07:00
mateo-berri
10a0da7a32 Merge litellm_internal_staging into devin/1784568628-databricks-gemini-reasoning-effort 2026-09-12 13:08:22 -07:00
ryan-crabbe-berri
a2e383a1a5 fix(realtime): ignore a late speechStart for a finished Muse turn
A duplicate speechStart for a turn that already stopped used to make that
closed turn active again, so the next turnless PUSH_TO_TALK transcript was
routed to the finished item and dropped.
2026-09-12 12:53:03 -07:00
ryan-crabbe-berri
6b78438c99 fix(realtime): close every Muse turn on its own terminal signal
Turns no longer wait behind each other in a FIFO queue, so an empty
server_vad turn (speechStart then speechEnd with no transcript) cannot
stall every later turn, and a PUSH_TO_TALK speechComplete now closes its
turn without waiting for a speechEnd that never arrives. Each turn keeps
its own idempotent emit state, so late or duplicate speechEnd,
speechComplete and transcript frames are no-ops, and finished turns are
remembered in a bounded map instead of a separate tombstone deque.

The session.created ack and the sanitized error frame are now typed as
members of OpenAIRealtimeEvents, which removes the typing.cast calls
that the strict ruff budget flagged.
2026-09-12 12:39:59 -07:00
yuneng-jiang
5f73f837ec
Merge pull request #40774 from BerriAI/litellm_cache_response_semantics
test(e2e): verify cached answers and upstream request count
2026-09-12 12:11:57 -07:00
yujonglee
347b642bdd
refactor(ocr): complete native lifecycle and preserve Azure auth (#40734)
* refactor(ocr): extract call completion boundary

* fix(ocr): release completion state after dispatch

* test(ocr): prove wrapper completion handoff

* test(ocr): narrow mapped failure assertion

* fix(ocr): preserve wrapper invocation kwargs

* fix(ocr): retain completion through finalization

* fix(ocr): make completion ownership explicit

* refactor(ocr): resolve logging executor explicitly

* fix(callbacks): preserve completion lifecycle behavior

* refactor(ocr): move public OCR into native lifecycle

* refactor(ocr): remove unused rust bridge capability

* wip

* wip

* refactor

* wip

* fix(ocr): preserve reducto native compatibility

* wip

* fix(ocr): document native callable casts

* perf(ocr): bound responses and reduce native scheduling overhead

* refactor(python-bridge): organize placeholder routes

* refactor test

* fix(ocr): normalize DeepSeek document content

* perf(ocr): skip unused callback work and benchmark callback overhead

* fix(ocr): align conversion contracts

* test(ocr): cover official provider response shapes

* fix(ocr): restore Python fallback and honor Rust opt-out

* fixes and refactor

* fix(ocr): preserve Azure Document Intelligence authentication

* fix(rust): enforce OCR response limits and lint contracts

* test(rust): align native OCR contract coverage

* test(ocr): isolate Azure auth precedence coverage
2026-09-12 11:56:49 -07:00
devin-ai-integration[bot]
0c98afa780
fix(mcp): enforce end user mcp_tool_permissions on tools/list and tools/call (#40865)
* fix(mcp): apply end user mcp_tool_permissions as a tool ceiling on tools/list and tools/call

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): restore scoped session admission coverage dropped by mistake

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 10:33:09 -07:00
devin-ai-integration[bot]
e4f59a953c
feat(guardrails): add Conduct Guard integration with validated hooks and forwarded params (#40785)
* feat(guardrails): add ConductGuard integration

Adds Conduct Guard as a first-class LiteLLM guardrail. Point any
LiteLLM proxy at Conduct and every LLM call routed through it is
policy-checked before the upstream request goes out — block, warn,
audit, or trigger a human-in-the-loop approval, with the same signed
configuration + hash-chained audit log Conduct exposes on its native
enforcement surfaces.

- litellm/types/guardrails.py: add CONDUCT to SupportedGuardrailIntegrations.
- litellm/proxy/guardrails/guardrail_hooks/conduct/__init__.py: registration
  via guardrail_initializer_registry and guardrail_class_registry, picked
  up by the auto-discovery in guardrail_registry.py.
- litellm/proxy/guardrails/guardrail_hooks/conduct/conduct.py: the adapter.
  CustomGuardrail subclass, async_pre_call_hook, response envelope parser
  for the five Conduct verdicts (ok / advisory / WARNING / BLOCKED /
  PENDING approval), fail-mode logic, session-ID resolution chain
  (litellm_metadata.trace_id → X-Conduct-Session-Id → hash fallback).
- tests/test_litellm/proxy/guardrails/test_conduct_guardrail.py: envelope
  parsing, pre-call allow/block/approval, config precedence, missing-token
  construction error.

```yaml
guardrails:
  - guardrail_name: conduct-guard
    litellm_params:
      guardrail: conduct
      mode: pre_call
      api_base: https://api.conductai.ai      # optional, default
      api_key: os.environ/CONDUCT_AGENT_TOKEN # cond_agt_* token
      fail_mode: fail_closed                   # or fail_open
      tool_name: llm_call                      # scoped tool_name
```

A standalone PyPI package `conduct-litellm-guard` shipped ahead of this
PR for teams pinned to older LiteLLM versions. Once this integration
merges, the standalone README will point at the native support as the
preferred path.

- PyPI: https://pypi.org/project/conduct-litellm-guard/
- Product: https://conductai.ai/guard

Contact: sudhi@b2bsphere.com

* chore: ruff format for conduct guardrail

Fixes lint check on the upstream PR.

* chore: fix ruff lint errors

- Remove unused TYPE_CHECKING import (F401).
- Un-quote self-forward-ref type annotation (UP037).
- Suppress BLE001 on transport-fallback broad-except (intentional).

* chore: drop typing.Any to satisfy strict-rule budget

BerriAI's ruff strict-rule budget caps ANN401 (Any type annotation)
and TID251 (banned import) totals. Aligning with the CustomLogger
base signature (data: dict, cache: object, **kwargs untyped)
eliminates all Any uses in the module. Local tests still pass 15/15.

* chore: annotate **kwargs to satisfy ANN003 strict rule

Removing 'Any' in the prior commit left **kwargs untyped, which
tripped ANN003 (missing type annotation on **kwargs). Using 'object'
threads the strict-rule budget cleanly.

* refactor: slim upstream adapter — import from conduct-litellm-guard PyPI

The full adapter (response parser, session-ID chain, fail-mode logic,
HTTP client) lives in the conduct-litellm-guard package on PyPI. The
upstream tree hosts a thin re-export + the LiteLLM registration wiring.

Matches the Aporia / Lakera pattern — vendor SDK on PyPI, upstream
integration is a tiny adapter.

Benefits:
- Passes ruff-strict-budget and type-discipline-budget without new
  violations.
- Users get the same install experience as any other guardrail vendor:
    pip install conduct-litellm-guard
- Vendor keeps ownership of the parser + fail-mode semantics; upstream
  keeps a stable interface.

Tests slimmed to smoke coverage (imports work, class is a
CustomGuardrail, enum + registries wired, missing-package error path).
Full behavioural coverage stays in the PyPI package.

Local runs of both scripts/ruff_strict_gate.py and
scripts/type_discipline_gate.py against upstream/litellm_internal_staging:
both pass.

* test(conduct): skip smoke tests when conduct-litellm-guard not installed

The wrapper module imports its runtime from the conduct-litellm-guard
PyPI package. When the package is not installed in the CI environment,
the smoke tests can't verify wiring (the import raises before any test
runs). Use pytest.importorskip so BerriAI's default CI env doesn't
fail on this integration, while environments that do install the
package (via 'pip install conduct-litellm-guard[dev]' or similar)
still get the smoke coverage.

Full behavioural test coverage lives in the conduct-litellm-guard
package's own CI.

* test(conduct): cover initialize_guardrail to raise patch coverage

Codecov flagged the __init__.initialize_guardrail body as uncovered
(30% patch coverage on that file). Added a test that mocks
litellm.logging_callback_manager and calls initialize_guardrail with
a SimpleNamespace stand-in for LitellmParams — exercises the full
function body and confirms the callback is registered.

* address review findings on #38143 (yucheng-berri, cursor, veria-ai, devin)

Rename fail_mode → unreachable_fallback (typed field)
─────────────────────────────────────────────────────
The shim was reading a free-form ``fail_mode`` field; a typo silently
defaulted the plugin to fail-open behavior. Switch to the typed
``LitellmParams.unreachable_fallback`` field so Pydantic validates the
value at config load. The plugin's constructor kwarg stays as
``fail_mode`` — the initializer maps the typed field onto it.
(yucheng-berri, devin-ai-integration)

Fix timeout default (was silently discarded)
────────────────────────────────────────────
``getattr(litellm_params, "timeout", 8.0)`` only applied the default
when the attribute was missing; ``LitellmParams.timeout`` always
exists and defaults to ``None``, so the intended 8-second budget was
never used. Change to ``getattr(..., None) or 8.0`` so ``None`` (and
``0``) fall through to the default.
(cursor[bot])

Move ImportError from module-load to __init__
─────────────────────────────────────────────
Raising ImportError at module load caused the guardrail-hook
auto-discovery loop to silently drop the registration when
``conduct-litellm-guard`` was missing. Users saw configs load with
no guardrail active and no error. Import lazily; raise the friendly
``pip install`` error at ``ConductGuardrail.__init__`` when
actionable.
(cursor[bot])

Advertise only supported event hooks
────────────────────────────────────
``during_call`` mode was advertised in the guardrail config but the
class never overrode ``async_moderation_hook`` — every request in that
mode silently bypassed policy. Override ``get_supported_event_hooks``
to return only ``pre_call`` so LiteLLM validates configs against
supported modes at load time. ``during_call`` / ``post_call`` support
lands with plugin 0.3.x once the underlying response-gate is wired
through ``guard_check_response``.
(veria-ai)

Text-completion + full-turn prompt scanning
───────────────────────────────────────────
Fixed in the standalone package: ``conduct-litellm-guard 0.2.2``
(BerriAI/litellm PR #38143 companion, shipping to PyPI shortly).
Pinned in the docstring here as the minimum supported version.
(veria-ai — text_completion bypass + 4KB truncation)

Tests
─────
  * ``test_only_pre_call_event_hook_advertised`` — regression for
    ``during_call`` silent-bypass finding
  * ``test_initialize_prefers_typed_unreachable_fallback`` — regression
    for typo silent-fail-open finding
  * ``test_initialize_applies_timeout_default_when_field_is_none`` —
    regression for silently-discarded 8.0 default
  * ``test_missing_standalone_package_raises_at_construction`` —
    regression for silent-drop-on-import-failure finding (previous
    module-load raise replaced with lazy import + init-time raise)

* style: ruff format on the conduct guardrail shim + tests

Lint job on #38143 flagged three files as needing reformat. No
behavior change — just ruff-format's chosen line breaks and quoting.

* style: remove redundant noqa on re-exported GuardDecision

Ruff lint flagged this as unused because GuardDecision is re-exported
via __all__. Removing the noqa satisfies ruff without changing behavior.

* style: satisfy strict-rule budget (ANN201, ANN401, TID251)

BerriAI/litellm CI's ruff strict-rule budget check flagged four new
violations on the conduct shim. Fixes:

- __init__.py: add return type annotation on initialize_guardrail
  (ANN201)
- conduct.py: swap Any → object on __init__(*args, **kwargs) so the
  signature stays permissive without dynamically-typed Any (ANN401)
- conduct.py: drop the now-unused Any import (TID251)

Ruff --select ANN,TID passes locally.

* style: satisfy type-discipline budget (LIT008, LIT009)

BerriAI/litellm CI's type-discipline budget check flagged the
subclass __init__ shim. Fixes:

- Drop the __init__ override entirely — the subclass now inherits
  __init__ from _BaseConductGuard (when the standalone package is
  installed) or from CustomGuardrail (fallback). Removes both the
  banned **kwargs (LIT008) and all four inert # type: ignore markers
  (LIT009 x 4).
- Move the missing-package check into a dedicated
  raise_if_missing_package() helper called by
  initialize_guardrail before construction. Preserves the
  cursor[bot] fix (silent-drop-on-import-failure) without needing
  a custom __init__.
- Fallback branch aliases _BaseConductGuard = CustomGuardrail
  directly, no type-ignore comment needed.
- Test updated to exercise the helper instead of the removed
  __init__ path; new companion test asserts the helper is a no-op
  when the package IS installed.

Local: ruff --select ANN,TID passes clean. ruff format applied.

Same behavioral surface — user-visible error message unchanged.

* style: explicit assert on noop test (TQ001 zero-assert budget)

BerriAI/litellm CI's test-quality budget flagged
test_raise_if_missing_package_is_noop_when_present as a zero-assert
test (TQ001). Make the intent explicit: raise_if_missing_package()
must return None when the package IS installed.

* refactor: shim becomes a pure alias, hooks now on plugin's ConductGuard

Plugin conduct-litellm-guard 0.2.3 ships SUPPORTED_EVENT_HOOKS +
get_supported_event_hooks on ConductGuard directly. The upstream
shim's subclass wrapper is now redundant — dropping it clears every
strict-rule budget gate (ruff-strict / test-quality /
type-discipline / basedpyright) in one pass.

Changes:
- conduct.py: subclass removed; ConductGuardrail is now an alias for
  the plugin's ConductGuard (no dynamic base class, no reassignment,
  no # type: ignore). raise_if_missing_package helper unchanged.
- test file: _IMPORT_ERROR → _import_error rename to satisfy
  reportConstantRedefinition (basedpyright treats SCREAMING_CASE as
  constant). Also drops unused sys import.
- Pin bumped to conduct-litellm-guard>=0.2.3 in the module docstring.

Verified all four LiteLLM gate scripts locally against
upstream/litellm_internal_staging:
  ruff_strict_gate    OK
  test_quality_gate   OK
  type_discipline_gate OK
  type_check_gate     OK

* fix: real stub class in the missing-package fallback

Runtime regression in the previous simplification — the guardrail
registry iterates every registered class at load time and calls
get_supported_event_hooks(). Fallback of ConductGuardrail = None
crashed the whole registry with AttributeError, which cascaded into
unrelated guardrails' tests (noma_v2, repelloai, hide_secrets,
provider_specific_params, etc.).

Fallback now defines ConductGuardrail as a real subclass of
CustomGuardrail with the required class attrs (SUPPORTED_EVENT_HOOKS
+ get_supported_event_hooks). Matches the pattern the
guardrails_ai integration already uses in the same repo.
raise_if_missing_package still fires before instantiation so users
see the friendly pip install error.

All four budget gates re-verified locally against
upstream/litellm_internal_staging:
  ruff_strict_gate    OK
  test_quality_gate   OK
  type_discipline_gate OK
  type_check_gate     OK

* style: mutable-ok suppression on registry dicts + hook returns

* fix: SUPPORTED_EVENT_HOOKS must be GuardrailEventHooks enum, not str

LiteLLM's guardrail registry scans SUPPORTED_EVENT_HOOKS and calls
.value on each entry to build the mode allowlist. Plugin 0.2.3 shipped
bare strings, which raised AttributeError on three upstream tests
(same three as the pre-0.2.3 None-registration failure).

- Fallback stub now uses GuardrailEventHooks.pre_call.
- Docstring and pip install message updated to >=0.2.4.
- Test asserts against the enum member (which is what LiteLLM's
  registry scan actually sees).

Requires plugin conduct-litellm-guard >=0.2.4 (already tagged and
publishing).

All four budget gates verified locally green:
  ruff_strict, test_quality, type_discipline, type_check

* fix(guardrails): validate Conduct event hooks, forward tool_name, drop optional-package test skip

Pass the plugin's supported hook list into CustomGuardrail so unsupported modes
(during_call, post_call, logging_only) are rejected at config load instead of
silently doing nothing. Forward the configured tool_name to the plugin, and
replace the missing-package stub so the registry still discovers the guardrail
while construction raises an install hint.

The regression tests inject a recording guardrail class so they run without
conduct-litellm-guard installed; the previous module-level skip left the
adapter untested in CI.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): scan Responses API input through the unified Conduct bridge

The plugin's native pre_call hook only reads prompt and chat messages, so
/v1/responses requests reached Conduct with an empty prompt and were always
allowed. ConductGuardrail now implements apply_guardrail, which routes every
endpoint through LiteLLM's shared guardrail translation and feeds the
translated texts (or structured messages) to the plugin's check()

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): log Conduct apply_guardrail decisions via log_guardrail_information

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(guardrails): move the Conduct apply_guardrail bridge into an injectable function

The bridge body only ran when conduct-litellm-guard was importable, which CI
never is, so codecov/patch reported it uncovered. apply_conduct_guardrail now
takes the plugin's check coroutine and blocked-error factory as parameters, so
the package-free tests exercise every verdict branch and the plugin-bound class
is a one-line delegate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): send tool-call-only turns to Conduct and test registry wiring through config load

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): log non-blocking Conduct verdicts in standard guardrail information

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(guardrails): add Conduct config model and Admin UI garden entry

Expose ConductGuardrailConfigModel through get_config_model() so
/guardrails/ui/provider_specific_params returns the api_key, api_base,
workspace_id, tool_name, timeout and unreachable_fallback fields, and
add the Conduct Guard partner card, preset and logo to the guardrail
garden so the integration can be created from the Admin UI

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): pass unreachable_fallback directly to conduct-litellm-guard 0.2.5

The plugin renamed its constructor kwarg from fail_mode to unreachable_fallback in
0.2.5 and kept fail_mode only as a deprecated alias that warns on every init. Forward
the new kwarg and bump the documented pin to >=0.2.5. Mirrors 62325467 on #38143

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): reject conduct-litellm-guard builds that swallow unreachable_fallback

Plugin 0.2.4 accepts **kwargs, so the renamed kwarg was silently dropped and a
configured fail_open became fail_closed. Fail at import with the install hint instead

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Sudhi Seshachala <sudhi@b2bsphere.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 10:04:55 -07:00