* docs(proxy): describe /v1/decisions and /v1/systemone in Swagger UI
Add summaries, descriptions, inlined request body schemas with examples and typed 200 responses to the
decision routes and their aliases, so the OpenAPI document and the lazy snapshot show both request formats
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): cap _inlined schema walk depth for the recursion check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): move LitellmUserRoles and KeyManagementRoutes into litellm.types.proxy.auth.user_roles
Part of LIT-9423
* refactor(types): move is_jwt into litellm.types.proxy.auth.user_api_key_auth
Part of LIT-9423
* refactor(auth): alias is_jwt import so JWTHandler.is_jwt is not read as recursive
* refactor(types): type UserAPIKeyAuth, its token view and hash_token in place
Part of LIT-9423
* refactor(types): move UserAPIKeyAuth, its token view and hash_token into litellm.types.proxy.auth.user_api_key_auth
Part of LIT-9423
* fix(types): keep agent, budget snapshot and is_jwt names importable from proxy._types
Part of LIT-9423
* refactor(core_utils): move resource_ownership out of the proxy
Part of LIT-9423
* fix(types): restore main's _types.py content alongside the user_api_key_auth move
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
* style(types): drop blank lines between proxy._types re-export imports
Part of LIT-9423
* refactor(proxy): keep only user_api_key_auth re-exports in _types.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): carry main's UserAPIKeyAuth field into moved file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(types): drop blank lines between proxy._types re-export imports
Part of LIT-9423
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): move LitellmUserRoles and KeyManagementRoutes into litellm.types.proxy.auth.user_roles
Part of LIT-9423
* refactor(types): move is_jwt into litellm.types.proxy.auth.user_api_key_auth
Part of LIT-9423
* refactor(auth): alias is_jwt import so JWTHandler.is_jwt is not read as recursive
* refactor(types): type UserAPIKeyAuth, its token view and hash_token in place
Part of LIT-9423
* refactor(types): move UserAPIKeyAuth, its token view and hash_token into litellm.types.proxy.auth.user_api_key_auth
Part of LIT-9423
* fix(types): keep agent, budget snapshot and is_jwt names importable from proxy._types
Part of LIT-9423
* fix(types): restore main's _types.py content alongside the user_api_key_auth move
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
* style(types): drop blank lines between proxy._types re-export imports
Part of LIT-9423
* refactor(proxy): keep only user_api_key_auth re-exports in _types.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): carry main's UserAPIKeyAuth field into moved file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(types): drop blank lines between proxy._types re-export imports
Part of LIT-9423
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Add ScaleDown model pricing
Registers the five ScaleDown models in the pricing map. ScaleDown bills on
input tokens only, so output_cost_per_token is zero for every entry. The
classify and decisions models bill at $0.04 per million input tokens, and
extract, summarize, and compress bill at $0.05 per million.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Price all ScaleDown models at $0.04 per million and drop unverified context limits
Every ScaleDown model now bills $0.04 per million input tokens. The
32000-token max_input_tokens/max_output_tokens values were not confirmed
against the service, so they are removed rather than registered.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Price all ScaleDown models at $0.05 per million input tokens
This is the standard rate in ScaleDown's billing and matches the usage dashboard.
The Decisions usage.cost field is not used for pricing.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Add regression tests for the ScaleDown pricing entries
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Cite the source of the ScaleDown rate in the pricing tests and type them
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Add ScaleDown as a chat provider
ScaleDown serves five models over two wire protocols. extract, summarize, and
compress speak OpenAI chat on /v1/chat/completions, while classify and
decisions implement the Jev Decisions API on /v1/scaledown, taking a state
object plus a questions map and returning one typed answer per question.
The provider authenticates with x-api-key rather than a bearer token, and the
decisions models need a non-chat request body, so it routes through the HTTP
handler instead of the OpenAI passthrough. Malformed decisions requests are
rejected locally with a message naming the problem, so callers do not have to
decode an opaque 422 from upstream. Decisions bills one model call per
question, so the cost upstream reports is passed through rather than
re-estimated from token counts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Route ScaleDown domain models to the native API and stop trusting usage.cost
The /v1/chat/completions and /v1/models routes return 403 on api.scaledown.xyz,
so extract, summarize and compress now call the native /extract,
/summarization/abstractive and /compress/raw/ endpoints. The default api_base is
the bare host; a trailing /v1 is stripped and re-added only for decisions.
The Decisions usage.cost field was about 600x the dashboard amount, so it is no
longer passed through; cost is computed from input tokens at the registered rate.
Native responses are returned unchanged as JSON, including the _value/_span_anchor
wrappers on nested extraction. They carry no output token count, so
completion_tokens is reported as 0 and documented as unmeasured.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Move ScaleDown translation into llms/ and address review findings
- Dispatch moves to llms/scaledown/chat/handler.py; main.py keeps a thin call
and get_llm_provider_logic.py no longer has a ScaleDown branch, since the
config resolves key and base URL itself.
- questions passed via extra_body are merged and validated after the shared
merge (sign_request), so the documented extra_body path works.
- extra_body can no longer change the upstream model, and decisions state may
carry a document but not text, so guardrails always see the text.
- A custom api_base requires its own api_key; the environment key is only sent
to the default host or SCALEDOWN_API_BASE.
- Image content parts become the decisions document; max_completion_tokens maps
to max_tokens; stream is served as one chunk for every model.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Refuse prompt-bearing extra_body keys and empty-key fallback to the env credential
extra_body is merged after guardrails inspect the messages, so each model now
accepts only option keys there (extract threshold/top_n, compress
compression_rate, decisions questions/state); text, instructions, prompt,
context and model are rejected. An empty api_key no longer lets the
SCALEDOWN_API_KEY from the environment go to a custom api_base.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Read compress token usage from the real response shape
/compress/raw/ nests the per-prompt counts under results and reports the total
as a top-level input_tokens; the provider read a top-level original_prompt_tokens
that is not there, so compress reported zero tokens and zero cost.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Accept the router's max_retries and word the extra_body error clearly
The proxy adds max_retries to every call, which map_openai_params rejected, so
every proxied ScaleDown request failed with a 400. It is now accepted and not
forwarded.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Return extracted fields from extract, follow schema refs, and meet the type-discipline gate
- extract content is now the extracted fields (span anchors dropped, wrappers
unwrapped), so it validates against the caller's response_format; the raw
payload stays on _hidden_params['scaledown_response'].
- Local $ref/$defs in the response_format schema are resolved.
- The adapter is rewritten with Final locals, read-only annotations, one raising
helper and a typed handler; check_type_discipline reports no violations.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Fix stream_options, schema-guided extract cleaning, document input and multi-image handling
- stream_options is accepted and not sent upstream, so include_usage streams work.
- Extract output is cleaned against the requested entities, so a field named
_value or *_span_anchor is no longer dropped.
- Extract and summarize send an image or PDF as the native document fields.
- More than one image is rejected instead of silently using the first.
- The cost test derives its rate from the cost map.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Cut recursive schema refs at the cycle and cap the expanded entity count
A small response_format schema with a self-referencing $def expanded to 4^16
leaves before any request was sent. Refs that point back into themselves are now
cut off at that point, and the total expanded entities are capped at 1000 so
repeated acyclic refs are rejected with a clear error.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Bound ref chains and schema nesting so long schemas fail cleanly
Resolving a chain of thousands of distinct $refs recursed until RecursionError.
Ref resolution is now an iterative walk capped at 32 hops, and nested properties
are capped at 32 levels; both raise a clear 400 instead.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(scaledown): support native classify and preserve extraction results
* test(scaledown): include provider regressions in CI shard
* fix(scaledown): pass registry typing and bounded recursion checks
* docs(scaledown): capture provider setup before and after
* fix(scaledown): preserve nullable schemas and reject malformed requests
* fix(scaledown): isolate provider dispatch from existing providers
* fix(scaledown): preserve the shared dispatch return contract
* fix(scaledown): preserve failure logging and reject lossy schemas
* fix(scaledown): keep timeout typing annotation on its expression
* fix(scaledown): resolve escaped extraction schema references
* fix(scaledown): annotate the bounded reference walk
* chore(scaledown): remove unused loop suppression
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: moe-berri <moe@berri.ai>
* feat(otel): let a team strip captured message content from its own OTel v2 destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): split the shared callback option conflict check and sync dashboard API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): address review feedback on team capture_message_content
Classify capture_message_content as NotSecret in the credential slot registry, inject test exporters through register_exporter_factory instead of patching _exporter_from_spec, type the test helpers, and stop reassigning processor in build_tracer_provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): add integration cells for team capture_message_content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): cover weave_otel and newrelic team destinations in capture_message_content cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): stop rejecting client-sent capture_message_content at the request body boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(otel): let a team capture_message_content override the global setting in both directions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep a team's capture_message_content on its own OTel v2 destination
The setting no longer rides the flattened team callback vars, so a sibling destination and a routed provider follow the global mode. Request bodies get the base 400 back, a failure-only entry rejects the setting, and entries for one backend must agree
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): type the capture_message_content callback validation tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): type the remaining capture_message_content test signatures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): mark the new capture_message_content test locals Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): default tenant capture to callback_settings.otel and keep CaptureMessageContent formatting bare
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep CaptureMessageContent a plain str like main
The str-to-Enum switch changed the public type: the config field's type,
repr and model_dump, a constructor that now raised on unknown values, and
unknown env values coerced to no_content. CaptureMessageContent is the plain
str subclass main ships again, the field is a str, and a
captures_span_content helper replaces the enum property. A test pins that a
published preset's capture mode stays the default for an unowned
destination, which is what main forwards.
* test(otel): drop the module-level CaptureMessageContent import the rewritten test no longer uses
* fix(lint): keep the merged branch within main's type budgets
Freeze the capture-mode settings model (LIT015) and read stored logging
entries through one typed sequence helper, so the conflicting-capture checks
add no unknown types.
* fix(otel): reject a destination capture_message_content while OTel v2 is off
The legacy OTel callbacks never read the per-destination setting, so a team
could register no_content and still export prompts and replies under the
global capture mode. Registration now refuses the var unless the proxy runs
with LITELLM_OTEL_V2=true, the same rule the Arize transport and New Relic
routing vars follow.
* test(otel): type the OTel v2 fixtures and the v2-off regression test
* fix(otel): accept only no_content and span_only on a team or key destination
OTel v2 writes captured content only to span attributes, so event_only
behaved like no_content and span_and_event like span_only on a
destination. The proxy-wide setting keeps all four values
* docs(otel): say an omitted capture setting follows the proxy's own callback for that backend
* test(otel): describe the general capture fallback in the unowned destination test
---------
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* refactor(anthropic): move passthrough SSE stream assembly into litellm/llms
Part of LIT-9423
* refactor(anthropic): move raw SSE stream assembly helpers out of proxy guardrails
Part of LIT-9423
* refactor(anthropic): give the anthropic_sse re-exports reasoned noqa codes
Part of LIT-9423
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
* test(anthropic): cover the raw SSE helpers in stream_assembly
Part of LIT-9423
* refactor(anthropic): move prompt cache pricing helper out of the proxy package
Part of LIT-9423
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): move complexity router request models next to their base config
Part of LIT-9423
* test(types): cover the old auto-router import path for the moved request models
Part of LIT-9423
* refactor(anthropic): move prompt cache observation lookup out of the proxy hook
Part of LIT-9423
* test(internal_context): exempt moved prompt cache lookup helper from key-family check
* feat(proxy): require trace IDs per team on LLM, MCP and agent requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert trace ID authorization outcomes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): clear trace ID type gate errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format trace ID type check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): validate trace ID metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): use read-only trace metadata types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): skip trace ID metadata validation for teams without dict metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): check team trace ID setting before route matching
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Each spend-log delete batch and the remaining-rows probe ran inside a
prisma interactive transaction with the default 5s timeout while setting a
30s Postgres statement and lock timeout. A batch that waited on a lock for
more than 5s was closed by the query engine with a 422 and rolled back, and
after three such batches the cleanup run aborted with nothing deleted.
Reuse the partition manager's bounded_tx, which sizes the prisma timeout to
the statement bound plus commit slack, for the delete batch and the probe.
Resolves LIT-8686
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(ci): distribute unit cases and reuse security sweep connections
* revert(ci): reject first speed trial after new failures
* perf(ci): avoid repeated test discovery and parsing
* perf(ci): isolate parallel security workers
* revert(ci): restore measured security execution isolation
* perf(tests): reuse source scans and deduplicate cache cleanup
* perf(ci): restore parallel security with safe log batching
* test(ci): isolate fixture state and schema comparisons
* test(ci): make schema and scheduler fixtures deterministic
* test(ci): drain queued logging before fixture teardown
* test(ci): preserve logging delivery without idle worker restarts
* test(ci): keep guardrail sinks available during parallel bursts
* test(ci): isolate full proxy lifecycle from gateway imports
* test(ci): preserve first failure responses and sweep evidence
* test(ci): restore documentation URL settings between cases
* test(ci): isolate open MCP tools in private gateways
* test(ci): group private MCP cases and handle exited workers
* test(ci): retain proxy exits and MCP failure details
* test(ci): repair merge interactions with main
Key the schema shard's full clone on artifact-name, since main removed the
unit-flag input. Give each owned Redis its own log file again. Hand the
realtime transcription tests main added a logging object whose dispatch
handlers are awaitable, so the per-test logging drain no longer waits on a
MagicMock task forever
* perf(ci): shard owned integration cases and close lifecycle leaks
* fix(ci): qualify shard execution and harden interrupted cleanup
* fix(ci): bound worker supervision and own fault traffic
* perf(ci): reuse unparametrized fixtures across bounded case groups
* fix(ci): preserve boot fault allowances and complete exit records
* test(ci): tolerate startup retries and retain readiness details
* perf(ci): balance integration shards by recorded duration
* fix(ci): load migration database helpers only for workers
* test: move packaging and install tests to unit and integration
* test: move packaging and install checks to smoke and integration
* test: keep the boot check proxy on its scratch database in replica mode
* test: restore install imports and address review findings on packaging checks
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: move routing, caching and callback tests out of local_testing
Moves the 37 routing, caching and callback files in tests/local_testing into mirrored tests/unit paths and tests/integration sdk and security groups per the migration manifest, and deletes the 8 rows it marks for deletion
* test: cover least-busy routing and tighten migrated router, cache and pass-through tests
Add router-level least-busy tests for the three legacy rows that had no
destination, move completion fallback and model alias tests to their mirror
files, move router batch tests next to the router, and strengthen tests
whose mutants survived (specific deployment, batch pair grouping, custom
cache key, router cache ttl). Add a unit test for configured pass-through
Authorization forwarding.
* test: tighten migrated callback, proxy config and cache ttl tests
Assert the full StandardLoggingPayload key set and the wire user on callback
tests, add an async embedding failure callback test, drive the prometheus
redis failure through a real RedisCache, use the real salt key in proxy
config tests, pin the redis ttl value and rename fallback test helpers
* test: fix semantic cache assertions and order-dependent router and callback tests
Count semantic cache embeddings on the route, drain async cache writes before
the second lookup, echo the requested model from the callback provider mock so
response_cost does not depend on model_cost entries other tests register, and
move the two-key authorization assertion to the auth failure test it belongs to
* test: cover model_list fan-out and the usage-based routing tpm limit lost in the move
* test: wait for the streamed cache entry in redis and drop deleted files from the hosted mock guard
* test: restore sync callback, audio redaction and langfuse export coverage, keep redis-compat filter on the moved file
* ci: run the remaining router-keyword legacy tests on one node
* test: refuse the redis write in-process instead of resolving an invalid host
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(guardrails): split logging_only_scope direction from input-failure policy
logging_only_scope is now direction-only (input, output, unset for both
directions). The new logging_only_continue_on_input_failure flag controls
whether a flagged or raising logging_only input scan still lets the
response be scanned. "both" stays accepted everywhere as a deprecated
alias for unset scope plus the flag set to true, normalized at callback
init while stored rows keep the written value.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dashboard): narrow continue-on-input-failure update type and stay under guardrail_info max-lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dashboard): keep custom-code guardrail controls readable with logging_only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dashboard): scope continue-on-input-failure to the default direction and restore verdict assertion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): remove the both value from logging_only_scope
logging_only_scope only shipped its both value in dev prereleases, so it is
dropped instead of kept as a deprecated alias. Leaving the scope unset and
setting logging_only_continue_on_input_failure gives the same behavior. A
stored both value is now treated like any other invalid scope: it is logged
and ignored on load, and rejected with 422 on create or update.
* fix(guardrails): tolerate an invalid logging_only_scope in the v1 guardrail list
A config.yaml guardrail whose logging_only_scope is no longer valid, such as
the removed both value, made GET /guardrails/list return 500 for every
guardrail. The v1 list now parses params with the same tolerant parser as
the v2 list, info and PATCH paths, so the bad value is logged and dropped.
* test(guardrails): drive continue-flag rejection tests through the real guardrail handler
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(key mgmt): a project may only be attached to keys of its own team
A project is created under exactly one team, and its budget and models are
validated against that team's. Nothing checked that a key's team matched, so
`POST /key/generate` with `team_id: team-a` and a `project_id` owned by
`team-b` answered 200 and stored exactly that — a key belonging to team-a,
recorded under team-b's project and validated against its models and budget.
The same held with no team at all.
It needs no proxy-admin rights: an admin of team-a who is a member of no other
team can issue keys under another tenant's project with nothing but the project
id, and that tenant sees the spend without having granted anything.
`_check_project_key_limits` now takes the key's team and refuses a project
owned by a different one, before the model and budget checks so the refusal
reads as what it is. A project with no owning team is left alone — nobody owns
it, so there is no boundary to cross. `/key/update` passes the key's stored
team when the request does not carry one, and now runs the check whenever the
project itself is set or changed, which a request that moves only the project
previously skipped.
Two existing cells needed the project's team passed explicitly: they measure
the model allowlist, not tenancy, and would otherwise have been asserting model
behaviour on a request the new gate refuses. A third builds an unowned project
for the same reason.
Fixes#41089
* fix(key mgmt): check project ownership on every key mutation path
/key/update ran the project check only when project_id, models or
max_budget were supplied, so a request that changed team_id alone left a
foreign project attached. /key/regenerate never ran it at all. Both now
go through one helper that validates the key as the mutation leaves it.
On /key/generate the check moves after default_key_generate_params is
applied, because that can supply team_id; it was rejecting a valid key
whose team came from the defaults.
Tests move into the mapped test file per CLAUDE.md.
* test(key mgmt): read the project from the cache instead of patching the lookup
The test-quality gate rejected the new cells: 12 TQ008 for patching
`get_project_object`, an SDK internal, and 4 TQ001 for accept controls that
could only fail by raising.
The cells now seed `UserApiKeyCache` with the project, which is the idiom
the neighbouring project cells already use and removes the patching. The
accept controls are parametrised together with the rejecting ones, so each
test function carries a real assertion. The three regenerate seams that stay
stubbed each carry a reason.
* fix(proxy): prevent cross-team project keys
Co-authored-by: L4XB <lukas.buck@e-mail.de>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): validate bulk key team changes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): use behavioral assertions in project ownership tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): read project owner from the database for key ownership checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): read project ownership from the primary database under the lookup deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): check project moves against the primary database team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run ownership checks after existing validation without the lookup deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type ownership writer reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): make ownership tests independent of runner salt and clock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): check key project ownership after existing validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run key project ownership right before the key row write
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject key project ownership before permission and budget writes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep project object permission validation on the stored payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format key regeneration update payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): invalidate the written permission id on bulk update and regenerate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): audit project team ownership across endpoints, bulk paths and chaos
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop budget and permission rows written for a rejected cross-team key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): require project-team membership when regenerate binds a key to a team-owned project
/key/{key}/regenerate and /key/regenerate accepted a team_id + project_id
pair without checking that the caller belongs to the destination team, so a
user with no team could regenerate their own key into another team and its
project. Regenerate now requires a proxy admin or a member/admin of the
project's team whenever the change lands the key on a team-owned project.
Also adapts tests to main's premium_user_check rename, drops a duplicate
import left by the merge, and replaces the created object-permission id cast
with an isinstance check.
* fix(keys): 404 a missing project before any key write, and match the project move refusal to its siblings
A missing project_id on /key/service-account/generate and on both regenerate
routes fell through to a foreign-key 500; regenerate had already written a
deleted-token history row, and service-account generate left its budget and
object-permission rows behind. The ownership check now answers 404 with the
same message /key/generate uses, before any write, and the generate rollback
covers it. The /project/update move refusal now uses ProxyException
(bad_request, param team_id) like the delete-with-keys refusal.
Tests: integration coverage for a team admin regenerating a team key into
another team's project (403), the missing-project 404s with no history rows,
and the unit test that never reached the ownership check is removed.
* test(keys): keep the existing-permission case in the regeneration cache eviction test
Parametrize the test over a key that already has an object_permission_id and one that mints a new row, so eviction under the key's existing permission id stays covered.
---------
Co-authored-by: L4XB <lukas.buck@e-mail.de>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): let admins make per-model key/team rate limits a hard 429 instead of falling back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): sync per-model rate limit API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type per-model fallback setting access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the per-model hard cap inside the fallback loop and test it through the real limiter
* fix(proxy): list the per-model hard cap setting on the settings page and cover every per-model descriptor
* fix(proxy): read disable_fallbacks_on_per_model_rate_limits as a boolean value and cover the team-wide per-model cap
* test(proxy): cover per-model rate limit 429s skipping fallbacks on a live proxy
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN
A strands_decider deployment whose api_base is a bedrock-agentcore runtime ARN now
calls InvokeAgentRuntime on that runtime, signed with SigV4 from the deployment's aws_*
params or the default chain, reuses one session per runtime, and maps the runtime's
200 error envelopes to 400, 503, or 500
* ci: run the strands_decider unit tests in the llms job
* fix(strands_decider): take the AgentCore session id from the deployment, never the caller
A caller could pick the runtime session through extra_headers, and every new id starts another microVM. The session id now comes from the deployment's agentcore_runtime_session_id (banned in request bodies) or the ARN default, and caller-supplied host, x-amz-*, and x-amzn-bedrock-agentcore-runtime-* headers are not forwarded
* fix(strands_decider): map runtime error replies only on AgentCore ARN calls
* fix(strands_decider): sign plain decisions calls inline, off the AWS signing executor
* test(strands_decider): cover AgentCore runtime ARN calls in integration wire tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch_interception): drop stream_options when forcing stream=False and replay it on the final stream
* test(websearch_interception): type the conversion helpers and drive the replay test through mock_response
* test(websearch_interception): pin the replay test's values as Final and record follow-ups immutably
* fix(proxy): strip every agentic-loop control key from client bodies, the web search keys included
* fix(interception): move stream_options off forced non-stream calls for code interpreter and headroom, and keep adapters from replaying nested calls
* test: annotate new interception test locals with Final
* fix(responses): drop params a deployment hook removed on sync Responses calls
* test: drive the sync Responses stream_options regression through a mock transport and real Azure config
* test: type the agentic loop recorder and capture the sent request on an object
* test(integration): cover interception stream_options on chat, responses, messages and the SDK
* test(integration): assert the follow-up error text and release a held burst on failure
* test(integration): pin the follow-up failure contract and wait for the replacement worker before teardown
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(realtime): only inject response.create when a transcript guardrail suppressed VAD auto-response
The proxy sent `response.create` to the backend after every
`conversation.item.input_audio_transcription.completed`, whether or not a
`realtime_input_transcription` guardrail had disabled the backend's server-VAD
auto-response. Without such a guardrail the backend already created the turn's
response, so the injected one was a duplicate and the backend answered every
user turn with `conversation_already_has_active_response`.
Gate the re-trigger on the same predicate as the suppress side
(`_has_audio_transcription_guardrails`), via `_proxy_drives_turns()`, in both
the raw-backend and the provider_config paths. Guardrails still run on every
transcript; only the injection is gated. Transcription-only sessions keep
skipping the injection as before.
Flip the unit test that asserted the old behaviour and add a provider-path
regression test.
Fixes#31726
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(realtime): assert guardrail-gated response.create on the provider_config path
Mirror of test_realtime_guardrail_allows_clean_transcript for backends that go through
provider_config. With a realtime_input_transcription guardrail the proxy disabled the
backend's auto-response, so a clean transcript must trigger exactly one response.create.
Locks in the other half of the suppress/re-trigger pair introduced for #31726.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(realtime): type the passthrough provider_config helper's frame argument
* test(realtime): expect no proxy response.create on an unguarded voice session in the integration suite
* test(realtime): audit cells for client-owned turns without a transcript guardrail
* test(realtime): wait for the killed worker to exit, not for its parent to reap it
* test(realtime): pin each worker-kill session's outcome to the worker that accepted it
* test(realtime): drive the provider_config path through a concrete passthrough realtime config
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: support OpenAI SDK 3.x while retaining 2.x compatibility
* fix: preserve HTTPX compatibility across OpenAI SDK versions
* refactor: trim OpenAI SDK compatibility patch
* refactor: simplify SDK client defaults
* test: cover SDK compatibility across provider routes
* fix: use shared names for SDK transport helpers
* fix: address SDK compatibility review feedback
* test(openai): send requests through the SDK API factory clients
* fix(exceptions): keep the provider response on PaymentRequiredError under httpx 2
* ci(base-sdk): count httpx2 as base-only when a base dependency brings it in
OpenAI SDK 3 depends on httpx2, so the litellm-core install at highest resolution pulls it in through openai. The base-only check now resolves the declared base dependency closure of litellm and litellm-core and accepts extras-only modules owned by a distribution inside that closure
* test: cover OpenAI SDK 3 client paths end to end and read SDK HTTP headers case-insensitively
* test: read C02 spend rows once, wait out the worker healthcheck on respawn, and resolve module owners on Python 3.10
---------
Co-authored-by: Marcus Wood <marcuswood@openai.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(decisions): add the OpenAI Decisions spec types and the System One translation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): share one DecisionsModel config and require model and usage on responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): rename the shared pydantic parent to DecisionsObjectBase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): switch /v1/decisions to the OpenAI Decisions schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): build the usage the OpenAI response schema now requires
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lens): ask signal questions through the OpenAI Decisions schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): refuse safety_identifier for System One providers unless drop_params
* feat(decisions): refuse safety_identifier on providers that cannot take it unless drop_params drops it
* test(decisions): cover hosted_vllm in the safety_identifier and per-provider wire tests
* fix(decisions): check a System One request's safety_identifier against the provider too
* fix(decisions): drop a non-string safety_identifier under drop_params
A malformed safety_identifier now follows the drop_params convention in both body shapes: it answers 400 without drop_params and is dropped before the provider call with it. A string identifier on OpenAI stays on the wire either way.
* fix(decisions): let /v1/decisions drop a non-string safety_identifier under drop_params
The route checked the whole body before routing, so a non-string
safety_identifier answered 400 even when the deployment or the body set
drop_params. The route now leaves that field to the Decisions call, which
knows every drop_params source
* refactor(decisions): return the unsupported safety_identifier as a value and raise it in _prepare_call
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): map OpenAI voice names to Gemini prebuilt TTS voices
* test(gemini): type the voice mapping test helpers
* fix(gemini): leave nova out of the voice mapping since Gemini accepts it as given
* test(gemini): add integration cells for the OpenAI voice mapping on Gemini and Vertex TTS
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(databricks): keep #/$defs refs in json_schema response_format for non-Claude models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): type the json_schema response_format override
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): integration cells for json_schema $defs refs across endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): split stacked loop in streaming json_schema cell
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): mark BaseConfig dict contract on json_schema override
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): cover json_schema refs on every endpoint, edge refs, sad paths and an outage burst
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): keep GPT tool-result images beside the tool result
Bedrock gpt-6.1-sol rejects an image nested in toolResult.content. Put it on the same user turn instead.
* fix(bedrock): read tool-result image support from the price map
OpenAI GPT rows on Bedrock Converse reject an image inside toolResult.content, so those images sit beside the tool result.
* fix: sync price-map test schema and import order
CI validates model_prices_and_context_window.json against an inline schema in test_utils, and ruff checks import order in converse_transformation.
* fix(bedrock): satisfy basedpyright in tool-result image helper
Drop Final assignments inside loops and use TypedDict key checks so the lint type-check gate stays within its merge-base ceiling.
* test(bedrock): cover tool-result image placement helper
Exercise Claude no-op paths, text-only GPT tool results, status preservation, and split text/image tool content for codecov.
* refactor(bedrock): reuse the shared tool-result image placeholder text
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* test(e2e): cover tool_reference tool results on bedrock converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep tool_reference results as text on converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format tool reference parser
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover bedrock converse tool_result of only tool_reference blocks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): tag converse tool_reference case with e2e metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): avoid unchecked cast in converse tool_reference mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep tool_reference results as text on responses paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): tighten tool_reference mapping on responses paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(completion-extras): expect tool_reference names in responses input
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(bedrock): drop responses converter changes, keep converse tool_reference fix only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(proxy): keep the mapped status on assistants and threads route errors
* test(proxy): type and annotate the assistants and threads route error tests
* fix(proxy): redact internal details from assistants and threads route error messages
* test(proxy): integration cells for assistants route error mapping
* test(proxy): drop malformed JSON cases that never reach the assistants route handler
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(azure_ai): support Microsoft-Decision-1 on the decisions API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): strip project path and API suffix together in decisions base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(agents): ban semicolons and colons in human-facing text
Tighten the writing rule so ";" and ":" are only allowed where the sentence would be nonsensical without them, rewrite the existing semicolons in AGENTS.md to follow it, and note that docs (litellm-docs) keep their trailing "."
* docs(agents): rewrite semicolons in nested AGENTS.md files
* docs(agents): hand-maintain the dashboard Next.js warning and sync e2e CONTRIBUTING lines
* docs(agents): point at the PR template path that exists
* docs(agents): restore the generated Next.js block in the dashboard AGENTS.md
* feat(rust): add the DeepSeek Anthropic Messages config
* refactor(rust): drop the tool discriminator without mutation and use rstest in DeepSeek tests
