mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-26 06:03:19 +00:00
619 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kerry Lu
|
7d7e28829f
|
test(integration): delete flaky team model TPM concurrency test (#46541)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
dba46c0513
|
fix(responses): keep the provider code and message on top-level stream error events (#46438)
* fix(responses): keep the provider code and message on top-level stream error events Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): log a mid-stream failure before the router falls back CustomStreamWrapper now waits for failure logging before it raises the error the router falls back on, so the cooldown lands on the deployment that failed instead of the fallback that replaced it. Also keeps numeric error codes and the top-level message on the WebSocket and ChatGPT paths. * fix(streaming): run sync failure callbacks before a failed stream falls back to another deployment The router cools a deployment down from a sync failure callback that reads the request's model_info. A failed stream submitted that callback to the logging executor without waiting, so it often ran after the router's fallback had overwritten model_info and cooled down the healthy fallback deployment instead, leaving the model group with no deployments * test(responses): cover top-level OpenAI stream error events end to end * test(streaming): read live worker pids so the second SIGKILL chaos variant sees two workers * fix(streaming): run waited sync failure logging off the shared logging pool * revert(streaming): keep failed-stream failure logging in the background Waiting for sync failure callbacks before a failed stream raises fixed the router cooldown race, but it slowed healthy calls under many failing streams with a slow callback, since the wait shares a thread pool with every call's start. The race is pre-existing on main and is tracked separately, so this PR goes back to main's logging timing and keeps only the top-level error event changes --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Tobias Mayr
|
d03923e6a9
|
fix(responses): propagate cache_control through responses->chat completion bridge (#32511)
* fix(responses): propagate cache_control through responses->chat completion bridge User/assistant messages, tool call outputs, and function calls built from Responses API input items were dropping cache_control, breaking prompt caching for Bedrock/Anthropic when routed through the litellm_completion transform. Also propagate cache_control onto the last content element of user/assistant messages, since Anthropic/Bedrock read it from the content block rather than (or in addition to) the message level. Adds the missing cache_control field to ChatCompletionToolMessage and ChatCompletionResponseMessage TypedDicts to keep the assignments type-safe. * chore(ui): regenerate schema.d.ts for ChatCompletionToolMessage.cache_control Adding cache_control to the ChatCompletionToolMessage TypedDict changed the proxy's OpenAPI spec; regenerated via `npm run gen:api`. * fix(responses): don't overwrite existing content-block cache_control; add list-content test Per Greptile review on the PR: - The message-level cache_control was unconditionally overwriting the last content block's cache_control when that block already carried its own (preserved by _transform_responses_api_content_to_chat_completion_content). Now only propagates onto the last block when it doesn't already have one. - Added the missing test coverage for the content-block propagation path with list content, which no existing test exercised (all used plain string content, so the isinstance(content, list) branch was never hit). * style(responses): extract cache_control into a named variable, trim comment Per style review: the message-building branch read cache_control 3 different ways in 5 lines (.get() twice, then [] indexing once), unlike the sibling tool-call-output/function-call functions this PR touches, which each extract it into a named variable up front (output_cache_control, function_call_cache_control). Matched that convention here too. Also trimmed the 3-line inline comment to one line, matching the file's usual comment length for similarly subtle behavior. * chore: refresh lazy OpenAPI snapshot * fix: type cache control propagation * fix: place cache control type suppressions * style: format cache control type conversion * refactor(responses): build bridged cache_control messages in one shot * fix(responses): keep cache_control marks on bridged tool calls and within the provider cap A function_call item's mark now rides on the tool call the bridge builds, an assistant text item merged behind a function_call keeps its own mark, a developer item keeps its mark when it is mapped to a system message, and a previous_response_id replay yields its stale marks to the newest ones so the provider cap holds. The Responses layer counts the marks on function_call and function_call_output items before it places an ordinal injection point, and the chat pass replaces that count with its own instead of adding to it. * fix(responses): count each cache_control mark once and keep the bridge within the provider's limits List content carries an item-level cache_control on its last block only, so the provider sees one cache point and the breakpoint census counts one. Marked function tools and SDK-typed input items are handled by the Responses-layer reserve, assistant text merged into a function call turn keeps its mark on its own block, and the previous_response_id replay budget drops marks per turn, newest kept, instead of stripping every older turn once one overshoots. The request's own marks now fall under the same budget, newest kept, so five marked items no longer answer the provider's 400. A cache_control value that is not an ephemeral mark with an optional ttl is left off the bridged message instead of reaching the provider as sent, and a developer message maps to the system role with its content, name, and cache_control only, so unknown keys never reach a strict provider. * fix(responses): send item cache_control only to cache-point targets and follow drop_params Item marks reach the provider only when the target turns them into cache points, and every other target keeps main's behavior of stripping them. Without drop_params a mark the provider would refuse reaches it as sent, so the client gets the same 400 as on /v1/chat/completions. With drop_params a malformed mark is dropped and only the newest 4 are kept. The shared input-to-messages helper and the developer mapping in base_utils go back to main, and a marked developer item reaches a cache-point target as a system message in the bridge. * fix(responses): budget top-level marks, follow drop_params on Nova tool marks, and match chat-keeping providers * fix(responses): validate replayed marks and count input tokens without item marks * fix(responses): leave unknown Bedrock ARNs unmarked under drop_params, match OpenRouter's list, and reserve only forwarded marks * test(responses): cover item cache_control parity per chat route, drop_params trimming, and replay budgets * fix(responses): keep item marks off inspection paths and match each chat route's keep rule Only the request the bridge sends gets item marks now. Guardrails, token counting, rate limiting, and input_tokens read the input with no item marks, as on main, so an off-spec mark cannot fail a guardrail call. Tool message and tool call marks are typed inside the bridge, leaving the shared chat types and guardrail schemas as on main. Databricks keeps marks only for Claude, Snowflake uses its chat route's Claude check, and every Anthropic deployment keeps them whatever its model name. The injection census skips SDK-typed tools, and the owned-proxy test expects the marked messages on /v1/responses. * test(responses): cover guardrail reads without item marks, Snowflake name parity, and SDK-typed tools in the census * fix(responses): leave a non-list input untouched when stripping item marks * test(responses): cover string, list, object and tuple input with and without item marks kept Each input shape now answers the same whether the unmapped Bedrock ARN keeps item marks or strips them under drop_params, through the proxy and the SDK. Object input also runs through the OpenAI-compatible chat bridge, /v1/responses/input_tokens and the project input-token limiter * test(cache): derive owned-proxy cell timeouts from the harness boot budget The capped-burst cells and the uncapped response-cache cells boot their own two-worker proxy, and a fixed 240 or 360 second timeout cut them during boot on a slow runner before the harness readiness budget could decide. They now carry the same derived budget as the sibling chaos and guardrail files, with four boot and stop budgets for the restart cell that boots twice --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
f95e466bb7
|
test(integration): fix three more CircleCI integration reds from test setup (#46510)
OpenRouter basic translation expected the pre-#44788 response shape. Read refusal, logprobs and system_fingerprint from the fixture at their new positions. The team model TPM test required exactly one of three concurrent requests to be served. The pipelined limiter can reject all of them, which is an accepted trade-off, so assert no over-admission and that a lone retry is served when the burst got nothing. The Voyage host tests count every host their proxy dials. The responses cost poller could dial api.openai.com for background-response rows that other tests left in the shared worker database, so turn polling off in those proxies. |
||
|
Yassin Kortam
|
2310daef7e
|
fix(proxy-extras): run prisma migrate deploy under the pinned coordinator lock instead of Prisma's session advisory lock (#46368)
Through a transaction-mode pooler Prisma takes pg_advisory_lock(72707369) on one backend and releases it on another, so the first backend keeps the lock while idle and the next boot waits on it. Migrations now run with PRISMA_SCHEMA_DISABLE_ADVISORY_LOCK=1 while LiteLLM holds the coordinator lock and Prisma's key as transaction-level locks on one pinned connection, so nothing can outlive the migration Resolves LIT-9571 Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
b9849f3588
|
fix(router): add fallback_on_unsupported_params so param rejections can raise instead of falling back (#46350)
* fix(router): add fallback_on_unsupported_params so client-side param rejections can raise instead of falling back Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(router): wrap the fallback gate so ruff format passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): rename the flag to fallback_on_bad_request and cover provider 400s too Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): narrow the wire test responses so the type gate count does not grow Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(openai): swap the model map with setattr so the type gate count does not grow Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): import the exception classes directly so the type gate count does not grow Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): type the fallbacks list in fallback_event_handlers to pay for the new Router kwarg in the type gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): keep the LIT001 count flat on the typed fallbacks parameters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): type the fallback stream wrapper weakref so the type gate stays level Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): make fallback_on_bad_request updatable at runtime so DB-stored router_settings apply it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate schema.d.ts for UpdateRouterConfig.fallback_on_bad_request Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): rename the flag back to fallback_on_unsupported_params Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(openai): drop the reasoning_effort=none client-side check, the Router flag covers the provider 400 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rerun docs checks now that the router_settings row is on litellm-docs main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): only stop fallbacks on real 400s and cover the context window and content policy exclusions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): move the no-fallback 400 check into is_bad_request_without_fallback and pin the 402 and Vertex 403 cases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): keep falling back on router-raised 400s when fallback_on_unsupported_params is off Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): read llm_provider off the exception with getattr so the type gate stays flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
47e4ed88c2
|
fix(anthropic): keep refusal stop_reason and stop_details on the logged response (#46104)
* fix(anthropic): keep refusal stop_reason and stop_details on the logged response * fix(anthropic): keep the refusal stop-fields boundary tolerant of a non-dict upstream body * test(anthropic): use Anthropic's documented refusal example as the unit-test fixture * fix(streaming): keep provider fields on async stream cache hits and received_model_id on synthesized finish chunks * fix(caching): replay cached completion_tokens_details on stream cache hits and skip non-object cached provider fields * test(integration): cover Anthropic-family refusal stop_details on the wire, the cache and under chaos * fix(caching): skip a cached completion_tokens_details that fails validation on stream replay * fix(anthropic): keep refusal stop fields on a JSON-mode non-stream response Also adds the round 3 audit cells for JSON-mode refusals, the proxy response serialization, and wrong-typed cached usage details on both stream replays * test(anthropic): expect content_filter on streamed JSON tool refusals after main's stream change --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
f10134728d
|
fix(caching): report Redis circuit-breaker metrics through the Prometheus services logger (#46484)
* fix(caching): collect the Redis circuit-breaker state gauge as one series per state across worker processes The gauge declares multiprocess_mode=livesum, so PROMETHEUS_MULTIPROC_DIR apps get one series per state summed over the live workers instead of one series per worker pid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): register Redis circuit-breaker metrics only when LiteLLM prometheus logging is on The breaker collectors join the default registry once the prometheus or prometheus_system callback is configured or a Prometheus logger starts, with earlier breaker events tallied and seeded in, and litellm.disable_redis_circuit_breaker_metrics turns them off for apps that export their own metrics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): drop the disable_redis_circuit_breaker_metrics opt-out Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): keep the Redis breaker metrics gate to a per-state count and a publish hook Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): pin the Redis breaker metrics gate against an injected registry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): publish the breaker collectors before reading them in the existing metrics test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): report Redis circuit-breaker metrics through the Prometheus services logger Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(integrations): build the Redis breaker collectors with typed prometheus_client constructors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
83980c9d63
|
fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget (#46493)
* fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): keep only the raising call inside pytest.raises for the pre-content stream tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): yield the primed first chunk from a helper generator so the stream iterators keep their complexity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): expect the empty invoke stream to be reported from router.completion now that the first chunk is read there Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: give router stream mocks a first chunk or an ending so the eager first read neither hangs nor retries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): name the first-chunk helpers after what they do (_read_first_chunk, _FirstChunk, _chunks_with_first) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): drop the string annotation on _FirstChunk.chunk, ModelResponseStream is imported at runtime --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
542b254ca2
|
fix(anthropic): forward service_tier on /v1/messages for anthropic, azure_ai and bedrock_mantle (#46370)
* test(integration): add known-bug translation cases for request fields LiteLLM drops service_tier on /v1/messages, conversation on /v1/responses and metadata on /v1/chat/completions never reach the provider. Each gets a TranslationTestCase with the correct provider body, marked xfail(strict=True) through the new TranslationTestCase.known_bug field, plus a passing service_tier case for /v1/messages on the openai route Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): widen the known-bug cases to azure_ai, bedrock_mantle, openai/responses and azure deployments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): forward service_tier on /v1/messages for anthropic, azure_ai and bedrock_mantle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
df3b44dbee
|
feat(proxy): let proxy admins open team member defaults to team admins (#46353)
* feat(proxy): let proxy admins open team member defaults to team admins * fix(proxy): keep a team admin's never-resets duration on a team's first member budget * test(proxy): pin the existing-row and stored never-resets cases for member defaults |
||
|
yuneng-jiang
|
47c40586e1
|
fix(proxy): evict the cached user on every proxy for every /user/update and /user/bulk_update (#46500)
* fix(proxy): evict the cached user on every proxy for every /user/update and /user/bulk_update Previously the cached user row was only evicted and broadcast when the update touched max_budget, model_max_budget, tpm_limit, rpm_limit or metadata, so a models or user_role change kept being served by peer proxies until the cache TTL expired * test(proxy): cover user cache eviction past the first bulk invalidation batch |
||
|
devin-ai-integration[bot]
|
2d4472efc1
|
fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733)
* fix(vllm): write spend log rows for /vllm passthrough calls to router models The vLLM passthrough config never built a logged response for relayed calls, so stream and non-stream calls through /vllm with a router model wrote no spend row and never charged the key. vLLM now shares the OpenAI-shaped relay logging that Azure already had, moved into an openai_like helper both configs call. * ci: run the vllm passthrough unit tests in the llms-providers shard * fix(proxy): log /vllm text completions and ask router models for stream usage A chat stream relayed to a router model without stream_options now gets include_usage added on the way out, and the usage chunk vLLM appends is removed before the client sees it, so the spend row carries vLLM's own token counts instead of a local estimate. /vllm/completions calls now write a row too, stream and non-stream. * fix(vllm): keep usage, fingerprints, and disconnect rows correct on /vllm streams Inject and strip stream usage on /vllm/completions too, hold every finish frame for the fingerprint when n > 1, pass a forced-usage server's stream through untouched, stop turning continuous_usage_stats alone into per-chunk usage, close the upstream when the client hangs up so the row is written, and assemble text choices with one sort instead of a scan per choice * fix(vllm): keep n>1 stream order when holding finish chunks for the fingerprint A held finish chunk now goes out as soon as any other chunk arrives, so a choice that finishes while another still streams keeps its place. Only the finish chunks right before the usage chunk get vLLM's system_fingerprint. * fix(vllm): keep caller stream_options and log both success handlers Forward a stream_options the proxy cannot read, or a non-boolean include_usage, to vLLM as the caller sent it. Only a false always_include_stream_usage changes the /vllm relay, so clients get the same stream bytes under every setting. Record the passthrough origin when logging starts so the sync and async success handlers both parse the relayed body, hand callbacks a parsed text completion, and read relayed messages and prompt separately so a stray field no longer drops the local count. Add the audit's integration tests for the relay, the settings, and the Azure spend row. * test(vllm): expect the caller's stream under always_include_stream_usage true * fix(vllm): relay long SSE frames in linear time * test(vllm): time the relay's usage and finish_reason prefilter on one large frame --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
1d9dd7b067
|
feat(proxy): let proxy admins open soft_budget to team admins (#46485) | ||
|
Kerry Lu
|
6bd5580425
|
fix(streaming): add CustomStreamWrapper.close() and release the connection when a sync stream ends (#46354)
* fix(streaming): close sync streams so the connection goes back to the pool CustomStreamWrapper gets a sync close() (plus context manager support) that closes the wrapped provider stream, and both the sync and async wrappers close the provider stream when the stream reaches its end. The Anthropic stream iterator keeps the httpx response it reads so close()/aclose() can release the socket after message_stop instead of leaving it checked out of the pool Co-authored-by: Blair Jordan <blair.j.jordan@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): keep the basedpyright ceiling flat in CustomStreamWrapper.close() Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): close the response on the shared sync stream path too BaseModelResponseIterator gets a sync close() next to its aclose(), and the sync branch of BaseLLMHTTPHandler hands it the httpx response the way the async branch already does, so providers built on it release the socket on close() instead of waiting for garbage collection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(streaming): unit tests for CustomStreamWrapper.close() and the shared iterator close() Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): close the provider stream before re-raising a mid-stream error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Blair Jordan <blair.j.jordan@gmail.com> |
||
|
ryan-crabbe-berri
|
6e0f0f7adb
|
fix(s3_v2): bound each upload attempt, add jitter, retry 429 and transport errors in the call (#46340)
* fix(s3_v2): bound each upload attempt, add jitter, retry on 429 and 503 Each S3 PUT now carries a per-attempt httpx timeout read from S3_UPLOAD_ATTEMPT_TIMEOUT_SECONDS (default 30s, clamped to 1..3600) instead of inheriting the handler's unbounded default. Retries back off with jitter instead of a fixed 1s then 2s, 429 joins 403, 500 and 503 in the in-call retry set, and transport errors or litellm.Timeout on an attempt are retried in the same call instead of ending it. Each flush logs one warning per batch of dropped uploads that names how many were terminal and how many aged out. * fix(s3_v2): retry uploads in a bounded loop and align tests with the new retry contract The async and sync retry helpers called themselves, which the recursive-function check rejects. They are now a bounded loop over the attempt count with the same outcome. The 429 integration test now expects in-call jittered backoff, and the aged-out test matches the new drop warning text. The sync transport test drives the real HTTPHandler so a dropped per-attempt timeout or a missed litellm.Timeout fails it, and the drop-reason test uses unequal counts so swapped labels fail. --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
yuneng-jiang
|
0915691d32
|
test(integration): remove six setup causes of CircleCI integration reds (#46490)
Drain the worker's fake-upstream request log when a test's gateway fixture starts, so traffic left by earlier tests on the same xdist worker no longer shows up in a test's first observation read Generate the fine-tuning rig's provider file and job ids per fixture build, so a rebuilt module fixture in the same worker no longer collides on the managed object unique constraint Wait for every wedged team_id and team_alias invalidation publish to fail before taking the recovery marker, now that team updates publish both keys Pin the cooled-down priority scheduler request to the connection that observed the cooldown, since a sibling worker reads cooldowns from Redis on an interval Simulate upstream outages and the held budget peer on ports the test keeps owning, instead of releasing a port another worker can bind Start a peer proxy for the accounting group, which has a test that needs one |
||
|
Kerry Lu
|
f0fedda722
|
fix(caching): collect the Redis circuit-breaker state gauge as one series per state across worker processes (#46357)
The gauge declares multiprocess_mode=livesum, so PROMETHEUS_MULTIPROC_DIR apps get one series per state summed over the live workers instead of one series per worker pid Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
457b5d26c7
|
fix(anthropic): stream json_schema answers as content on Bedrock Mantle and Claude Platform, and join split answers (#46331)
* fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude
Vertex AI Claude deployments that emulate response_format through a forced
tool sometimes nest the schema object under an extra key such as
`parameters` or `input`, and that envelope reached message.content with a
200. The converter now unwraps exactly one nested object that matches the
schema, leaving a payload that is itself schema-shaped alone.
Native structured-output rows sent the deprecated top-level `output_format`,
which Vertex rejects with a 400. The Vertex request now carries it as
`output_config.format`, with an explicit caller value winning.
* fix(anthropic): unwrap tool-emulated json_schema envelopes on streams and tool-name wrappers
* test(anthropic): expect the json tool description through a shared constant
* fix(anthropic): drop the json tool description that makes sonnet-5 omit required keys
* fix(anthropic): keep schema-accepted answers whole and decide held streams at the second key
An answer the json_schema accepts is returned as is before any envelope rule runs, the tool-name
wrapper is unwrapped only when the schema rejects it, a malformed schema turns the unwrap off, a
streamed answer whose first key is a schema property named name or tool_name waits for its second
key before streaming, the held fragments go through a StringIO, and a prefix that stays undecided
past 4 KB is held to the end
* test(anthropic): keep the stream family cells to providers whose stream iterator gets the schema
* fix(vertex_ai): unwrap claude json_schema answers nested under up to three wrapper keys
A tool-emulated json_schema answer was unwrapped only when the real object sat one key down. In live QA, 1 of 58 answers (/v1/responses, claude-sonnet-5) came back as {"parameter": {"value": {...}}} and reached the client still wrapped
The unwrap now follows single-key wrappers up to three levels down and stops at the first object that has a key the schema declares. That object is returned only when it has every required key. A payload wrapped deeper than three levels, a level with more than one key, and a level whose key is one of the schema's own property names all come back exactly as they arrived
The streamed path needs no change because the buffered answer goes through the same unwrap when the tool block closes, and a test now pins that both paths agree
* fix(bedrock): unwrap streamed claude json_schema answers on claude platform and mantle
A streamed json_schema answer on bedrock/claude_platform and Bedrock Mantle Claude reached the client as tool_calls deltas holding the json_tool_call wrapper, with finish_reason tool_calls and no content. The shared HTTP handler built the async stream iterator without json_mode, and neither stream path passed the response_format schema
The handler now asks the provider config for the iterator through get_model_response_iterator_for_request, which carries json_mode and the request's optional params on both the sync and async paths. Claude Platform and Mantle override it to hand the json tool schema to the Anthropic stream iterator, so the streamed answer is the bare schema object with finish_reason stop, the same as the non-streamed one
Databricks is the only other provider on this handler whose iterator reads json_mode, so its async Claude json mode stream now returns the answer as content like its sync stream already did
The stream family integration test covers claude_platform_unmapped and bedrock_mantle again
* fix(anthropic): unwrap json tool envelopes by structure instead of schema validation
The unwrap now reads only the schema's top-level property names. A lone
undeclared key holding an object, alone or beside a name equal to the json
tool's own name, is the envelope, and the unwrap repeats while the inner
object shares no key with the schema. jsonschema and referencing are no
longer imported on the request path. The stream holds only an answer that
opens with the tool's name or with an undeclared key holding an object, and a
schema without top-level properties streams as it arrives.
* fix(llms): keep other providers' async streams as they were
The request-aware iterator hook passed json_mode on async streams to every provider on the shared HTTP handler, which changed Databricks and any other provider whose iterator reads it. The default hook now makes the same two calls the handler made before, so only Claude Platform and Bedrock Mantle change, and the Databricks test that pinned the wider change is removed
* fix(anthropic): walk json tool envelopes iteratively
* test(llms): pin which stream iterators get json_mode from the shared handler
A provider config that does not override the request-aware iterator hook still gets json_mode on a sync stream and still does not get it on an async stream, the same as before the hook existed
* fix(anthropic): keep json tool arguments too deep to decode instead of raising
* fix(anthropic): unwrap json tool envelopes beside undeclared string labels
* fix(anthropic): keep json answers whole on schemas that allow extra keys and stream held text as sent
A schema that allows undeclared keys and requires none now keeps an undeclared wrapper key on both the stream and non-stream paths. When no envelope is found, the held stream text is now sent exactly as the model wrote it. On Vertex /v1/messages, output_format now wins over output_config.format when both are set, which matches the Anthropic adapter.
* fix(anthropic): join a json_schema answer the model splits into one tool block per field
With tool emulation the model sometimes answers with one json_tool_call block per schema field, each naming the field and holding its value. The non-stream path returned only the first block and the stream sent every block's text back to back, so the client got one field or invalid JSON. Both paths now build one schema object from those blocks, and any other set of blocks keeps today's output
* refactor(anthropic): set the streamed finish reason in one expression
* test(anthropic): cover label envelopes, dynamic-key schemas, as-written held streams and output_format precedence
* fix(anthropic): unwrap a streamed values wrapper like the non-stream answer
* fix(anthropic): join streamed split answers on schemas without required keys, keep client tool calls and length on json_schema streams
* fix(anthropic): join streamed split answers that carry text labels on schemas without required keys
* test(bedrock): script the wrapped json_schema stream with the wrapper Bedrock Mantle sends
* test(bedrock): script the wrapped json_schema stream with the wrapper Bedrock Mantle sent raw
* fix(anthropic): join a split json_schema answer whose fields all hold the same string
* fix(anthropic): join a split json_schema answer sent as one partial object per field
* fix(anthropic): join field blocks after an empty streamed answer without a leading comma
|
||
|
Mateo Wang
|
389403fede
|
fix(proxy): check previous_response_id ownership on /v1/responses/compact (#46408)
* fix(proxy): check previous_response_id ownership on /v1/responses/compact * test(proxy): cover the unmanaged response id escape hatches on every response id route * test(proxy): audit compact previous_response_id ownership Add integration cells for the native /v1/responses/compact routes covering owner, admin, cross-tenant, unmanaged, malformed, duplicated and retained ids, provider errors, the response cache, id-security settings, an upstream outage, a held upstream, a worker kill and a proxy restart. Unskip the provider wire cell whose bug this branch fixes. * test(proxy): cover the no signing key path on every response id route |
||
|
Kerry Lu
|
d0f3a2dfdb
|
fix(vertex_ai): pass the per-request timeout to Gemini streaming calls (#46349)
* fix(vertex_ai): pass the per-request timeout to Gemini streaming calls Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vertex_ai): pass timeout to make_call and make_sync_call in the http 201 tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): default timeout to None on make_call and make_sync_call Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
6b7c77b7d3
|
fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355)
* fix: lower aiohttp keepalive timeout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http): restore the 15s keep-alive default and apply it to the Rust client 15s is aiohttp's own default and what every release before v1.78.0 used. The Rust HTTP client reads the same AIOHTTP_KEEPALIVE_TIMEOUT variable, so its default moves with it. * test(integration): cover upstream connections dropped after the keep-alive idle window --------- Co-authored-by: nate <nate@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Rick
|
4f3cb48dc1
|
feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos (#43706)
* feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos
DashScopeVideoConfig maps litellm's OpenAI Videos API onto Alibaba Model
Studio's async task API: create hits
/api/v1/services/aigc/video-generation/video-synthesis and status and content
poll /api/v1/tasks/{task_id}. It is registered for dashscope and for the
qwencloud and qwen_ai_platform brand aliases, following the same
get_dashscope_family_*_config pattern image generation uses. Unlike the
Sora-shaped providers the request body is nested, so prompt and media go under
input and everything else goes under parameters. X-DashScope-Async is always
sent, because DashScope rejects the call outright without it
OpenAI params translate to DashScope's contract. seconds becomes duration, and
size yields both a ratio and a named resolution tier (1280x720 gives 16:9 plus
720P), since sending only the ratio leaves the request on the default 1080P and
silently bills at that rate. input_reference becomes a single media entry,
base64 encoded into a data URI for file inputs. Which media type a model takes
(first_frame or, on the reference-to-video models, reference_image) is read
from provider_specific_entry.dashscope_video_reference_input via
get_model_info rather than hardcoded; models without it predate the media
array and get the flat img_url field. A caller-supplied media array always
wins, as that is the only way to express multi-role combinations
The create call is the only billed call, so it bills what DashScope will
deliver: the requested duration and tier, or the documented 5 second and 1080P
defaults when they are omitted. Smart duration (duration -1) lets the service
pick the length after that call, so it is rejected rather than recorded at
zero, with a 400. Task creation and task lookup both report failures as a 200 carrying
top-level code and message with no output, so that shape is raised instead of
surfacing as a queued video with an empty id. Status responses re-encode the
deployment model from the polled id, so follow-up polls and downloads route to
the same deployment. Timestamps parse as UTC+8. DashScope publishes no list,
delete or remix endpoint, so those raise rather than guess at a URL
The end-to-end tests drive the real avideo_generation() entrypoint through an
httpx MockTransport and assert on the JSON body httpx actually encoded. That is
what catches a mapper that drops input_reference before the request is built,
or a body fragment json cannot serialize, neither of which a test calling the
transform directly would see
Both model_prices maps gain wan3.0-video, wan3.0-video-prime, wan2.7-t2v,
wan2.7-i2v, wan2.7-r2v and the HappyHorse 1.0 and 1.1 t2v, i2v and r2v models,
for dashscope and the qwencloud and qwen_ai_platform aliases, priced per output
second by resolution tier from Model Studio's USD rate cards. dashscope and
qwen_ai_platform default to the Beijing host and use its rate card; qwencloud
defaults to the international host and uses Singapore's. The three
*_API_BASE_VIDEO overrides are documented in BerriAI/litellm-docs#1920
Co-authored-by: AaronHowell <237895480@qq.com>
* fix(dashscope): fail loudly on video responses that cannot be read
Addresses the review on the video generation PR. A body that is not the
documented task shape, an output block missing task_id or task_status, an
undocumented task_status and a missing output all raise DashScopeVideoError
now, instead of defaulting into a queued video with an empty id that callers
poll forever. A JSONDecodeError from a gateway response no longer escapes as
a generic connection error.
task_id and task_status are required on _TaskOutput, and a missing output
raises even when the body carries no code or message.
Download failures move into the 4xx range, since a 5xx has the proxy retry
them: 409 while the task is still running, 400 for a task that failed, 404
for an id that expired. The status a provider reported is kept when it
reported one, so an in-body failure that arrives on a 200 is billed as the
400 it is rather than as a retryable connection error.
characters and image are only dropped when drop_params is set; the params
that are rebuilt into the body or consumed by the caller (model, prompt,
user, extra_*) stay silently skipped, since the proxy stamps user from the
caller's end-user header.
Co-authored-by: AaronHowell <237895480@qq.com>
* fix(dashscope): clear the lint gates after merging main
* fix(dashscope): accept file tuples and paths as video input_reference
video_generation() types input_reference as FileTypes, which includes
(filename, content[, content_type[, headers]]) tuples and pathlib paths,
but the DashScope encoder only read bytes and file objects, so those
image-to-video calls raised before reaching DashScope. Bare strings stay
URLs and are never opened as paths.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dashscope): snap size to a supported ratio and read file tuples and paths
* test(dashscope): assert whole mapped values and bill against hand-written rates
The tier pricing test read its expected rate from the same map the cost
calculator uses, so a wrong rate passed. Billing is now checked against
fixed rates driven through the status response usage, while the map test
only asserts every supported tier has its own rate. Size mapping tests
compare the full mapped dict, and the single-test wiring class is a
plain function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(videos): download content through the caller's client and stop retrying dead tasks
Two Cursor Bugbot findings on the DashScope video PR.
The DashScope artifact lives on OSS, so transform_video_content_response makes a
second request. It built its own cached client, discarding an injected client and
its ssl_verify. Thread the client through the content response the way
transform_video_status_retrieve_response already does, and use it in the three
providers that fetch from a second URL (DashScope, RunwayML, fal.ai). The
remaining providers accept it and read the response body, matching how they
already accept an unused client on status retrieve.
A FAILED or CANCELED task reported without a code or message fell past the
in-flight check into the 502 branch, so the proxy retried a task that can never
produce a video. Raise 400 for a terminal status instead.
* fix(dashscope): answer 404, not 409, when a Wan video is downloaded before it finishes
A download that lands while the task is still PENDING or RUNNING answered 409. That
reads as a client error, which was the intent, but litellm's retry predicate counts 409
as retryable, so the proxy polled DashScope twice more before giving up and then counted
the caller's own early download against the model's error rate
404 is what the OpenAI videos API documents for content that is not ready, and it is the
one status in that family the retry predicate leaves alone. The rest of the mapping is
unchanged: 400 for a task the provider reported as failed, 404 for an expired task id,
502 when a succeeded task carries no artifact
The download tests now drive avideo_content and assert the caller sees NotFoundError
rather than an internal server error
* fix(dashscope): route a video poll back to the model group that created the task
The create response encoded the upstream model name into the returned id. Both the
dashscope and qwen_ai_platform prefixes serve the same model names from their own hosts,
so a proxy with a deployment per region had deployments that only the model group told
apart. Encoding the bare model name resolved a poll to whichever deployment the router
lists first, which sent it to the other region's host with that region's key and a task
id that region never issued. That is the reroute the earlier proof records as the
dashscope and qwen_ai_platform poll ids coming back tagged qwencloud
The id now carries the model group the create was routed to, which is what the proxy
resolves a poll back to, and falls back to the model name for a direct SDK call that has
no router
The regression test drives a real router with one deployment per region, under a
different prefix and under the same one, and asserts the poll carries the key of the
region that created the task
* fix(dashscope): read the routed model group from litellm_metadata
The router stamps the model group it picked into litellm_metadata. metadata only
carries a copy when the caller sent no metadata of its own, and every proxy request
sends some, so a proxied create still encoded the bare model name and the poll
resolved to whichever region's deployment the router lists first
The regression test now also creates the video with caller metadata, which fails
without this change
* fix(dashscope): reject unbillable video requests and keep fal and runway on their own clients
* fix(dashscope): show the caller why a video task is gone or not ready instead of the bare upstream body
* fix(dashscope): check video request values before the billed create and honor per-call drop_params
Type check documented DashScope parameters so a bad seed answers 400 instead of a billed failed task,
pass the call's and deployment's drop_params into the video mapping, answer 400 for an expired task id so
the deployment is not cooled down, read DASHSCOPE_API_BASE for video, keep polling SUSPENDED tasks, and
refuse to map size onto models litellm has no price for.
* fix(dashscope): carry the requested model name in video ids so alias and wildcard keys can poll
A key scoped to an alias or to a name a wildcard serves could create a
video and then got a 403 on the poll and the download, because the id
carried the group that served the call (wan3, qwencloud/*) and the key
was never given that name. The id now carries the name the router was
asked for on that hop, and the router resolves an alias, routing group
or wildcard name in an id back to itself when no deployment name or
provider model matches it first
* fix(dashscope): keep a routing group's video polls on the member that created the task
A routing group spans several model groups, so an id carrying the group's name
load-balanced its polls, and some left for another region's host and key with a
task id that region never issued. The id now carries the member that served the
task when the request named a routing group, and keeps the requested name for a
model group, an alias, or a wildcard match.
* fix(dashscope): carry the requested and served model in video ids so group-scoped keys poll in the creating region
* fix(proxy): pin a video's served group only after key and team aliases resolve
* test(integration): audit DashScope video create, poll, download, routing and chaos paths
* fix(proxy): keep catch-all video polls on their provider and count served polls against the requested group
* test(routing): expect a forged served group to be dropped from the polled video id
* test(integration): cover rate-limited routing group polls and catch-all polls of DashScope videos
---------
Co-authored-by: AaronHowell <237895480@qq.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
yucheng-berri
|
d17cd947b9
|
fix(sso)!: reject empty identities before session creation (#44098) | ||
|
ryan-crabbe-berri
|
12cce6c372
|
fix(anthropic): check Vertex forced-tool support on the real model, skip non-string user ids, drop speed where unsupported (#46337)
* fix(anthropic): check Vertex and Bedrock forced-tool support on the real model The Vertex and Bedrock invoke Claude transforms swapped the model for a claude-3-sonnet stand-in before calling the shared Anthropic mapping so response_format would take the tool-based path. Every model-keyed check in that mapping then ran against the stand-in, so the synthetic json tool was forced on models whose map entry says forced tool use is unsupported and the request 400ed upstream. Bedrock patched the one symptom with a backstop, Vertex had nothing. The shared mapping now asks a uses_native_structured_output hook for the structured-output decision and the two overrides answer it with their provider-scoped lookup, so the real model reaches every check and the stand-in, the restore dance and the Bedrock backstop go away. * fix(anthropic): skip non-string metadata user ids instead of raising _valid_user_id ran re.match straight on litellm_params metadata.user_id, so an integer id raised TypeError and the proxy answered 500. The user param path already skips non-strings, so the validator now does the same for any caller: a non-string id is simply not forwarded. * fix(utils): drop speed on every provider without fast mode speed rides into get_optional_params as a provider-specific kwarg, so the only gate for it lived in the Anthropic transform and Bedrock converse, OpenAI, Azure and the extra_body legs forwarded it to a 400. The gate now runs once in get_optional_params before provider-specific kwargs are copied, so every provider drops it with drop_params or rejects it with a clean 400, and a direct Anthropic fast-mode model keeps it. * fix(utils): gate speed only under drop_params and keep it for litellm_proxy Raising on speed when drop_params is off would turn requests that tolerant OpenAI-compatible backends accept into 400s, and dropping it for litellm_proxy would disable fast mode on a downstream proxy that supports it * test(anthropic): derive the forced-tool regression model from the cost map and drop new docstrings * test(utils): pin that speed is untouched without drop_params and annotate new test locals * test(integration): lock the Vertex and Bedrock invoke json tool to the real model's forced-tool support Vertex sends tool_choice only when the model accepts forced tools, reasoning_effort maps the same with and without response_format, and Bedrock invoke keeps sending the json tool unforced |
||
|
Mateo Wang
|
d81a5719ad
|
fix(cache): keep cache hits free of unset fields, and follow intentional behavior changes in CircleCI suites (#46359)
* test(integration): follow count_tokens error surfacing and decisions pricing in CircleCI suites Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): restore Responses input_file PDF pricing coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep integration AGENTS guide unchanged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cache): preserve provider field presence on cache hits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect credential display name in response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): align OpenInference refusal field expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect Akto request ingestion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): expect missing tool error wording Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect no refusal field in cursor chat wire Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a93fee9f6c
|
feat(decisions): add Laya and Bespoke providers to the Add Model form (#45260)
* feat(ui): add decisions and classifier providers to the Add Model form Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): use literal upstream decisions paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: drop duplicate TYPESAFE provider fields entry superseded by main's TypeSafe Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ui): drop stray blank line in provider_info_helpers test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): type the OSS decisions provider fields test * fix(decisions): keep Laya and Bespoke env keys on their configured server Route laya and bespoke Decisions deployments through the same oss_connection and validate_oss_model rules the pass-through and the classifier router use, so LAYA_API_KEY and BESPOKE_API_KEY only go to LAYA_API_BASE and BESPOKE_API_BASE, an api_base with credentials, a query, or a non-HTTP scheme answers 400, and an unknown checkpoint name answers 400 before any upstream call * fix(decisions): send the Laya and Bespoke env key to a form-added model on the env server A deployment added through Add Model always carries an API base, so the env key only reached deployments with no base. The env key now goes to a deployment whose base matches LAYA_API_BASE or BESPOKE_API_BASE, an empty deployment key counts as no key, and the Bespoke placeholder is the name Ollama serves. * test(decisions): cover the env-base classifier case in the complexity router test * fix(decisions): match the env server with or without /v1 and point Ollama users at bespoke/nimble * fix(decisions): call /v1/systemone once from classifiers and pass-through on a /v1 base * fix(decisions): strip /v1 from the API base path only, so a host named v1 still works --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
aa0279e0a2
|
feat(anthropic): generate auto mode safeguard_results with a classifier model on the /v1/messages chat bridge (#45600)
* feat(anthropic): generate auto mode safeguard_results with a classifier model on the /v1/messages chat bridge * fix(anthropic): check caller limits for the safeguards classifier, mark truncated tool calls, and keep pre-compaction context * fix(anthropic): log classifier spend under the caller's key hash and read list-shaped classifier replies * feat(anthropic): generate safeguard_results on the Responses bridge and native non-Claude /v1/messages providers * fix(messages): keep safeguards requests on Python under the Rust opt-in * fix(anthropic): keep non-ASCII text unescaped in the safeguards classifier prompt * test(anthropic): build the text-parts classifier reply without openai's construct path Message.model_construct goes through openai's construct_type, which caches discriminated unions in a weak-key map. On Python 3.12 a PEP 604 union cannot be weak-referenced, so the fixture raised inside the fake classifier and the test read an unavailable verdict. model_copy sets the list content without that path. * fix(anthropic): scan only new bytes for SSE frame ends on safeguarded native streams * fix(anthropic): answer safeguards on web search short circuits and widen the classifier reply cap * fix(anthropic): answer unsupported when the proxy serves no deployment for the safeguards classifier * fix(anthropic): keep a backend's stream verdicts and join split SSE data lines before stamping * fix(anthropic): map only the router's no-healthy-deployments refusal to unsupported * fix(anthropic): answer unavailable for the turn on a transient caller denial and let the classifier deployment's timeout apply * fix(anthropic): keep the user's whole opening message in the classifier transcript past its budget * fix(anthropic): run the proxy's own access checks for the safeguards classifier call The classifier and compaction sub-calls now pass through the same model access checks as the parent request (key, team, team member, user, project, end user, managed agent grants), with budgets left to the per-model budget check so nothing is reserved. They also carry the parent's logging params and headers, so a redaction header or per-request callback params apply to their own rows. Only per-model RPM limits count against a sub-call now, since a key-wide or team-wide RPM window is already spent by the parent request. The MCP auto-execute loop classifies only its final response, a chat-bridge stream without safeguards yields frames directly, and a classifier provider failure logs one warning line instead of a traceback * fix(anthropic): keep safeguards sub-calls off the parent's session routing and carry a body trace id The classifier and compaction summary calls no longer get the parent's proxy_server_request, so a Claude Code session bound to a router no longer pulls them off their configured model. Redaction still reads the headers carried in metadata. Their access checks now use the parent's resolved litellm_trace_id when the parent metadata has no trace_id, so a team that requires a trace id accepts a sub-call whose parent sent it in the body. * fix(anthropic): keep the disable-callbacks header on safeguards sub-calls * fix(proxy): charge per-model budgets from litellm_metadata when metadata holds only routing keys * test(integration): add safeguard_results cells for the bridges, native routes, caller checks, and classifier failures * fix(proxy): check the caller's budgets on the classifier call and keep failed access reads retryable --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
2f3ccbe7c7
|
feat(decisions)!: send images to Cloudflare Clef through its images field (#45734)
* feat(decisions): send images to Cloudflare Clef OpenAI-format input_image parts and the System One images field now reach Cloudflare clef and clef-flash in its images field, with remote URLs downloaded and inlined first. Providers that cannot take images drop them under drop_params and answer 400 otherwise. * fix(decisions): follow drop_params for images and use the call timeout for image downloads * fix(decisions): drop a malformed images entry on its own under drop_params * fix(decisions): refuse images next to input whatever drop_params says * fix(decisions): bound clef image fetches, honor drop_params for images, and log caller URLs Refuse more than 4 images for Cloudflare clef before any download, keep the caller's timeout as one deadline across image fetch retries, accept images next to input under drop_params, validate image strings and objects, and pass the caller's image URLs (not inlined bytes) to logging callbacks. * fix(decisions): check input_image URLs and clef image sizes * fix(decisions): accept data URL media type parameters and check clef image types * fix(decisions): count clef images after dropping unsupported types and flag clef vision --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
1aee1a21ed
|
fix(databricks): send a single plain text block as a string for non-Claude models (#45686)
* fix(databricks): send a single plain text block as a string for non-Claude models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(databricks): simplify single-text-block helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(databricks): clarify json_object validation wording Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): integration cells for single text block content shape Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): rename list-content cells to stays_list Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): expect plain string content for bridged Databricks messages Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): cover the customer's Unity Catalog model from #45617 and #45618 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(databricks): drop casts from _collapse_single_text_block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): assert the whole outgoing messages list in the collapse unit test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(databricks): cover system joins, fallback, SDK clients, and outage burst for one-block content --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b22a0bd9e5
|
test(integration): cover fine_tuning endpoint contracts (#44697)
* test(integration): cover fine_tuning endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove fine_tuning ownership hook, header spelling and drain order Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fine_tuning): assert unified job id and retrieve ownership through route-granted keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fine_tuning): assert managed file and job ownership rows in Postgres Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fine_tuning): cover validation_file ownership, default retrieve ownership and full job objects Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
b37eac29f0
|
fix(anthropic): cache tool result screenshots on the /v1/messages to Responses bridge (#46324)
* fix(anthropic): put a tool_result cache breakpoint on its hoisted images on the Responses bridge A marked tool_result whose images get moved into a trailing user message now carries the breakpoint on the last moved image instead of the function_call_output, so the prefix OpenAI caches includes the images * test(anthropic): cover a marked tool_result with two screenshots on the Responses bridge * fix(anthropic): keep every tool result screenshot inside the breakpoint that covers it on the Responses bridge A marked tool_result's screenshots now go right after its output, so earlier unmarked screenshots land inside the breakpoint and later tool outputs stay after it. A cache_control mark on a block inside a tool_result is read too. * fix(anthropic): cover only the screenshots up to a mark inside a tool result, and drop raw breakpoints under drop_params A cache mark on a text block inside a tool result now covers that result only up to the marked block, the way Anthropic caches it, so screenshots after the marked text stay outside the breakpoint. A mark on the tool result itself still covers every screenshot in it A deployment that reads no prompt cache breakpoints now drops raw prompt_cache_breakpoint marks, top-level and inside tool results, when drop_params is on, and still forwards them as sent without it * fix(anthropic): keep raw breakpoints under drop_params for models that take them * docs(anthropic): say a mark inside a tool result covers its whole text and documents --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
Oliver Jensen
|
14e820ef7c
|
fix(proxy): scope /global/activity/cache_hits analytics to the caller's own and permitted-team logs (#46210)
Non-admin callers previously received deployment-wide cache activity, error breakdowns, every key alias on the proxy, and could filter the aggregates by any other tenant's key alias. All four queries now take the caller's read scope as bind parameters, which also shrinks the key-alias and model filter options to the caller's own visibility. The three-way scope resolution (admin / internal user / fallback) now lives in resolve_spend_log_read_scope, shared with /spend/tags and /spend/logs/session/ui instead of being duplicated per endpoint. |
||
|
devin-ai-integration[bot]
|
46a389c60b
|
feat(claude-code): emit installationPreference in marketplace.json entries (#41430)
* feat(claude-code): emit installationPreference in marketplace.json entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(claude-code): satisfy marketplace CI checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(claude-code): qualify marketplace entry as read only Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(claude-code): suppress typed marketplace assembly warning Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(claude-code): restrict installation_preference to the documented values Accept only available, auto_install, or required so a typo returns 422 instead of being silently dropped, and cover the PUT full-replace clearing behavior with a test * fix(claude-code): echo installation_preference on the public skill hub * refactor(claude-code): read installation_preference with one membership check and cover hand-edited rows * test(integration): audit installation_preference on the Claude Code marketplace * test(integration): kill a serving worker in the marketplace chaos cell * test(integration): give the marketplace worker-kill cell the two-boot timeout * fix(proxy): keep installationPreference on the MarketplacePluginEntry model * fix(proxy): refuse self-installing preferences off sha256 archives, accept the camelCase key, and echo the preference on POST and PUT * fix(claude-code): serve installation_preference only on archive plugins and refuse both key spellings Claude Code 2.1.75 and older reject a marketplace.json once any plugin carries installationPreference, and every Claude Code release before archive support rejects a catalog with an archive plugin. Requiring an archive source for every value, available included, keeps the field off catalogs those clients can read. A request that sends installation_preference and installationPreference together now gets a 422. * fix(claude-code): read the preference spelling that carries a value when the other is null --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
cfa98e2f40
|
test(integration): cover pass_through endpoint contracts (#44690)
* test(integration): cover pass_through endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): run configured endpoint contracts on one proxy worker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove pass-through streams, SDK spellings, endpoint renames and bedrock stream bodies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover pass-through wire behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover Vertex and Gemini passthrough authentication Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Prove pass-through runtime method, credential-less Anthropic, stream barriers and Vertex strip layers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(pass_through): gate unquoted Bedrock stream targets and assert configured endpoint part types and methods after rename Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert pass-through query pairs so duplicate params fail Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
feb92d3255
|
test(integration): cover orgs_users_customers endpoint contracts (#44725) | ||
|
devin-ai-integration[bot]
|
5f7de8d6ac
|
test(integration): cover keys_auth endpoint contracts (#44674)
* test(integration): cover keys_auth endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover hashed, path and token spellings for keys_auth contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): strengthen JWT mapping lifecycle coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover reset_spend over plaintext and hashed path spellings with reset_to 0 and 0.02 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop Final in reset loop and use drain-then-fresh-refusal in warmed-peer delete proof Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): disable model info refresh in JWT lifecycle rig Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop unused discovery constant Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover cache invalidation and key paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> |
||
|
devin-ai-integration[bot]
|
1047a450bf
|
test(integration): pin assistants and evals endpoint contracts (#44734)
* test(integration): cover assistants_evals endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): replace exception assertions flagged by ruff Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(openai): type assistant run options Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(unit): cover assistant and eval contract paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(openai): keep assistant run options unvalidated Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin assistants and evals wire contracts as BUG skips Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover every eval route, full responses, image parts and full SDK stream payloads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): relay upstream eval run model, send reasoning_effort, cover SDK root alias, done frame and client disconnect Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): name reasoning_effort in the run BUG skip Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): use a non-null marker thread and name the eval run model restamp in its BUG skip Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): mark disconnect cases with a messages.create the proxy relays on main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(assistants_evals): cover thread message shapes, raw run options, header leaks and /v1 eval bases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect the OpenAI /cancel path for eval run cancel Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
27209b92b3
|
test(integration): cover mcp_runtime endpoint contracts (#44671)
* test(integration): cover mcp_runtime endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(integration): ruff format and import order in mcp runtime tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert out-of-path calls and untrusted forwarded headers are refused Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): move the untrusted forwarded-header discovery case into its own test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): hash the token row lookup, widen spend polling, and union open servers after listing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten mcp runtime contracts per review (sdk client, toolset scope, forwarded trust, callback html, session tools/call) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert the sdk unknown-path error without pinning the sdk's exception wrappers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover trailing-path and group csv mcp paths, rfc 9207 iss-less and error-iss callbacks, and id_token exchange Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover lite logout spelling, team-mapped and unmapped IdP subjects Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover per-server mcp-session resource and root /token spelling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): unpack the sealed client code in the step-up tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
595cf066cd
|
test(integration): cover config_utils endpoint contracts (#44737)
* test(integration): cover config_utils endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove config_utils contracts on the peer, Vault AppRole and cert logins, and admin-view checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert no Vault traffic after config override delete Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove the Vault cert login presents the client certificate over TLS Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover the blank plugin_key dashboard edit and the env-only CyberArk readback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(tests): ruff-sort imports and reformat config_utils integration tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): check plaintext Vault traffic in the cert phase with one drain Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): probe CyberArk runtime state after DELETE with a secret-resolving chat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): barrier the Langfuse absence check on the shared export channel Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): send the dashboard's blanked CyberArk field and check row, readback and runtime env Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
5261a2cc06
|
test(integration): cover responses endpoint contracts (#44733)
* test(integration): cover responses endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten responses contract tests per review Isolate team-only response-id ownership with a same-user different-team key, send every denial through the OpenAI SDK on all three aliases, make the previous_response_id affinity check deterministic, assert the whole cancel response, prove the Cursor stream text, -fast suffixes and include_obfuscation strip, add SDK compact cases, restore router pre-call checks after the affinity test, and inline the root-level responses_test_support helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for every proxy worker before routing to new deployments Restore the cross-worker convergence wait dropped when the shared helper was inlined, so the affinity and retrieve-routing tests cannot pass before the alternative deployment is loaded on a multi-worker gateway Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover Responses ownership and wire contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): address responses contract review Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect the provider model in the conversation passthrough body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expand responses endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): restore streamed response retrieval Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for proxy workers after deployment setup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): track readiness per proxy worker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
8822e086ec
|
test(integration): cover budgets_spend endpoint contracts (#44736)
* test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): harden budgets_spend contracts per review Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): make spend report window midnight safe and drop redundant comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin organization spend report scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close budgets and spend review findings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop streamed cost header absence assertion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): cover streamed spend on linked budget tiers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): wait for async spend logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
60ee966885
|
test(integration): cover anthropic_messages endpoint contracts (#44704)
* test(integration): cover anthropic_messages endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close anthropic_messages review gaps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove azure beta filtering and the safeguards allowlist Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): compare the forwarded anthropic-beta token-for-token and pin the truncated stream bug Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: strengthen endpoint wire contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: follow count_tokens upstream error relay from main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: follow device-code refresh token response from main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0ca9f8185c
|
test(integration): cover files_batches endpoint contracts (#44703)
* test(integration): cover files_batches endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): split raw cloud-storage delete test by caller Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop mutable polling state in content download test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): dispatch request spellings without rebinding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): import assert_never from typing_extensions for python 3.10 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten files_batches wire and access assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tidy files_batches access tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover repeated target models, upload expiry, batch aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove files streaming path, SDK model denial and admin GCS delete Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): give the streaming head the wire barrier's full budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: strengthen files and batches wire coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: drop duplicate parametrize from the model-routed upload helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: pass the SDK base url in the skipped repeated-plain upload test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): allow batch output metadata lookup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: avoid draining file content requests before wait Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e226016aee
|
test(integration): cover health_routing endpoint contracts (#44744)
* test(integration): cover health_routing endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten health_routing contracts per review Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close health_routing review gaps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(integration): format observed routing fallback cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close health_routing final review findings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): resolve health_routing review findings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0f0c1bc80b
|
test(integration): cover teams endpoint contracts (#44705)
* test(integration): cover teams endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): address review on teams contract tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drain shared upstream observations at fixture setup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove team block, member and bulk permission contracts at runtime Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert email admin adds and whole membership rows for team members Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drive key update grant and revoke, Z-suffixed temp budget expiry, link LIT-9257 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close review gaps in teams endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): harden team member endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
aa879f77a9
|
feat(proxy): let team admins create keys for team members behind an opt-in setting (#45870)
* feat(proxy): let team admins create keys for team members behind an opt-in setting A team admin calling POST /key/generate with a team they administer and the user_id of another member of that team got a 403, and the dashboard only offered the "Another User" owner option to proxy admins Add a create_member_keys entry to team_admin_editable_team_fields. When a proxy admin turns it on, a team admin can create a team key owned by another member of a team they administer, through the API and through the Create Key form. It stays off by default, so nothing changes until a proxy admin opts in With the setting off, the 403 a team admin gets for this call now names the setting to enable instead of saying they can only create keys for themselves * fix(ui): order the member owner picker after the team and drop the viewer from it For a team admin the owner picker depends on the team, so it now sits under the Team field, lists only the other members of that team, and the Team field says why it is required * fix(proxy): keep keys a team admin creates for a member from carrying more than LLM access A key authenticates as its owner, so with create_member_keys on a team admin could create a key owned by a proxy admin who sits in their team and act as a proxy admin with it The member now has to hold the internal_user or internal_user_viewer role, and the key is always an llm_api key, so the team admin never holds a secret that reaches management routes. Asking for a management or read_only key for a member is refused The unit tests now check the permission decision directly instead of replacing generate_key_helper_fn, and the saved key is checked against a real proxy in the team admin gate matrix * style(proxy): sort the team admin field permission imports * refactor(proxy): force the member key type on the request instead of merging a second dict * test(proxy): drive member key creation through the key generation helper The pure helpers were covered but the wiring in _common_key_generation_helper was not. Dropping the owner role lookup, the refusal call, or the user_id handed to common_key_access_checks left every unit test green * fix(proxy): skip the owner role lookup when create_member_keys is off The owner role was read before the verdict checked the setting, so with the setting off a failing user table read surfaced as a 500 instead of the 403 a team admin got before, and every such request paid an extra writer read --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
148ca66ecb
|
test(integration): cover prompts_tags_caching endpoint contracts (#44686)
* test(integration): cover prompts_tags_caching endpoint contracts * test(integration): assert prompt lifecycle requests per environment * test(integration): pin non-latest prompt versions, label/path spellings and tag header/key spellings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover production PATCH, streamed and root /responses prompts, and memory create scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): ignore proxy model discovery probes in prompt upstream doubles Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove tag update, peer prompt resync, memory team rows and Responses prompt spellings at runtime Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover memory metadata and key_prefix listing, multi-tag spellings for tag cache invalidation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> |
||
|
devin-ai-integration[bot]
|
02f56c6319
|
test(integration): cover guardrails_policies endpoint contracts (#44722)
Some checks are pending
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / llms-bedrock (push) Blocked by required conditions
unit / llms-openai-meta (push) Blocked by required conditions
unit / llms-providers (push) Blocked by required conditions
unit / llms-vertex-ai (push) Blocked by required conditions
unit / mcp-elicitation (push) Blocked by required conditions
unit / mcp-server (push) Blocked by required conditions
unit / proxy-auth (push) Blocked by required conditions
unit / proxy-endpoints (push) Blocked by required conditions
unit / proxy-extras (push) Blocked by required conditions
unit / proxy-feature-endpoints (push) Blocked by required conditions
unit / proxy-hooks-client (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Waiting to run
unit / OpenAI 3.0.0 compatibility (push) Waiting to run
unit / OpenAI 3.25.0 compatibility (push) Waiting to run
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* test(integration): cover guardrails_policies endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): let the admin submissions list carry other rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close guardrails policies review gaps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover guardrail auth and policy attachments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tidy guardrail policy test helpers * test(integration): keep plain non-admin submission refusal live * test: expand policy attachment integration coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): accept the intended 403 for plain submission keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin multi-tag header and spend row tags for policy tag scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop the duplicate pydantic import left by the main merge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Devin <noreply@devin.ai> |
||
|
devin-ai-integration[bot]
|
c9a1d9b026
|
test(integration): cover small_inference endpoint contracts (#44654)
* test(integration): cover small_inference endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert full search tool listing for scoped keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep moderation and search tool peers stateless Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover filename-agnostic OCR uploads, list queries, object rerank documents and search tool authorization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): build search tool and moderation proxy configs without mutating the loaded yaml Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover the /ocr alias, model search spelling and Cohere v2 max_tokens_per_doc, clean up non-admin search tool rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover octet-stream OCR uploads and BUG-skip the Python requests boolean spelling * test(integration): pin Exa passthrough of native fields and caller contents Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> |