846 commits

Author	SHA1	Message	Date
yuneng-jiang	6a0603e00f	feat(count_tokens)!: backport the count_tokens stack to rc/1.107.0 (#46839, #46843, #46583, #47057, #47064, #47163, #46777) (#47209) ... * fix(proxy)!: count prompt-only token_counter requests with the provider when call_endpoint is set (#46839) * fix(proxy): count prompt-only token_counter requests with the provider when call_endpoint is set * fix(proxy): keep the local count for a prompt-only token_counter request when the provider count fails * fix(proxy): log a prompt-only token count fallback as a one-line warning * fix(proxy): send a prompt-only token_counter request to the provider only on Gemini and Vertex AI * fix(proxy): count a token_counter prompt sent next to an empty messages or contents list * fix(proxy): count a prompt next to messages on the local token count instead of answering 500 * fix(proxy): answer 400 for a token count prompt next to a conversation unless drop_params is set With drop_params the conversation is counted without the prompt. On the local count, a prompt sent with system or tools is counted as a user message with them, like the messages form. * test(proxy): count tools with a prompt in the token counter fallback test * test(proxy): cover prompt counting on /utils/token_counter in the Gemini count wire tests * test(proxy): name the token counter fallback test by what it checks (cherry picked from commit 2dc6aafa6b) * fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count (#46843) * fix(count_tokens): apply context_management edits in the Gemini /v1/messages/count_tokens count /v1/messages runs clear_tool_uses context edits before it sends an Anthropic body to a Gemini or Vertex AI deployment, but /v1/messages/count_tokens counted the body unedited, so a request with older tool results cleared counted about 2.8x what inference bills. The count now runs the same history and context-edit step inference runs, honors additional_drop_params on the deployment, and skips compact edits so a count never makes a paid summary call. * fix(count_tokens): match the Gemini count to inference for compact edits and drop opt-outs Run compact edits in the count without the summary call so trigger checks and history slicing match /v1/messages, honor a request-level additional_drop_params, keep the full body when any deployment in the model group drops context_management, and answer 400 naming a block Gemini cannot count when an edit trigger fails on it * fix(count_tokens): count the unedited body when an edit fails and honor deployment ids in the drop check * fix(count_tokens): log a failed edit in the count at debug, since the send already logs it * fix(count_tokens): resolve a deployment id before wildcard routes in the drop check, as the router does * fix(count_tokens): honor the router-wide additional_drop_params opt-out in the drop check The router merges default_litellm_params between the deployment and the request, so a router-wide context_management opt-out makes /v1/messages send the full history. The count now reads it in the same order. Adds the audit's wire tests for the count's context edits. * fix(count_tokens): read the opt-out of a deployment the router serves by its litellm model name * fix(count_tokens): read the opt-out and drop_params from the deployments the router and completion_model send to * fix(messages): drop an invalid context_management spec under drop_params so the send matches the count * fix(count_tokens): give Claude Code and Codex counts the drop_params default /v1/messages gives them The proxy turns on drop_params for Claude Code and Codex user agents on /v1/messages. The count now applies the same default, so an invalid context_management spec drops on both endpoints instead of the count answering 400 while the send answers 200. The wire test also covers default_fallbacks, model_group_alias, and completion_model routes. * test(count_tokens): cover a coding agent that turns drop_params off and stop two count tests running as Claude Code (cherry picked from commit f8217d2a03) * fix(proxy)!: answer 400 for malformed tools on count_tokens instead of crashing (#46583) * fix(proxy): answer 400 for malformed tools on count_tokens instead of crashing A non-list tools value crashed the vector store lookup during auth and the built-in tool cost lookup in the failure logging path, so the proxy answered an unhandled 500 and closed the keep-alive connection. count_tokens now answers a 400 invalid_request_error for a malformed body, drops malformed tools when drop_params is on, and both lookups skip values of the wrong shape. * fix(proxy): skip non-list tools in the tool allowlist check Also validate the count_tokens body without recursion, which the recursion detector rejects. * test(integration): restore the int tools case on count_tokens The proxy now answers a non-list tools value with a 400, so the case is deterministic again. * fix(proxy): reject malformed tools at the LLM route entry and keep vector store checks on malformed ids * fix(proxy): apply drop_params like core when dropping malformed tools, and reject malformed tool vector store ids Check tools after request setup so the Claude Code default and the logged call type apply, read the request, global, and model group drop_params the way core does, drop tool_choice and friends with tools, cover Gemini generateContent, and answer 400 for a malformed tools[].vector_store_ids with the strict flag off * fix(proxy): keep the request's call type when logging a gate rejection after request setup The body-shape fallback relabeled a /v1/messages rejection as acompletion and a list-input /v1/responses rejection as aembedding, so it now only applies when the logging object carries no known call type * fix(proxy): share the malformed tools check across tool routes and honor drop_params per entry * fix(proxy): accept null vector_store_ids, read every named store id in auth, check the responses websocket * refactor(proxy): leave the responses websocket out of the request-level tool check * fix(proxy): let a request's additional_drop_params replace the deployment list and keep compact's call type on rejections * fix(proxy): read router default drop lists and deployment ids in the malformed tools check * fix(proxy): leave malformed tools to the router when it can pick another deployment Fallbacks, auto-routers, model strings, comma batches and team model names let the router pick a deployment after the proxy checks the request, so the proxy now forwards a malformed tools field there as main does, and only rejects it when the deployment it checks is the one that serves. Token counting never reroutes, so count_tokens and input_tokens keep reading the group they count with. additional_drop_params names are read the way core reads them, so a list with a non-string entry still names its strings. * test(integration): cover malformed tools on reroutable models, router defaults and deployment ids * fix(proxy): resolve malformed tools at the deployment the router picks * fix(proxy): keep provider-repaired tool schemas, honor fallback_on_unsupported_params, skip cooldown on malformed tools Empty schemas and JSON-string schemas pass the tools check the way litellm's provider paths already read them. The rejection is a BadRequestError subclass carrying the deployment's provider, so fallback_on_unsupported_params: false keeps the 400, and the router never cools a deployment down for a request litellm rejected before calling it. Adds the audit's pre_call guardrail and cross-provider bridge cells. * test(integration): expect a 400 for an empty-string tools field on /v1/messages Real Anthropic rejects tools "" with a 400 on main too, so the case joins the malformed tools test * fix(router): keep web search routing for tools passed as a tuple * fix(router): keep a deployment's own tools when drop_params drops a malformed request tools * test(router): list _merged_tools as covered through the deployment tools merge tests * fix(router): drop the request's tool settings with its malformed tools when deployment tools replace them * fix(router): skip a tool whose type is not a string in the web search filter instead of raising TypeError * fix(prometheus): keep deployment health metrics clean on a malformed tools rejection * test: type the new malformed tools tests * test: mark new test locals Final and type kwargs dicts (cherry picked from commit d76ec3a47e) * ci(cost map): classify token_count_model as an object key in the schema generator (#47057) * ci(cost map): classify token_count_model as an object key in the schema generator * ci(cost map): require non-blank token_count_model strings and test the region rules (cherry picked from commit 6e191a58f1) * ci(cost map): remove the generated model prices JSON schema (#47064) * ci(cost map): remove the generated model prices JSON schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost map): keep schema-independent cost map tests after dropping the schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(cost map): keep the schema file until main's guard stops requiring it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(cost map): let bot sync PRs keep touching the schema file until it is deleted Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit ae6f7f6613) * fix(count_tokens)!: answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens (#47163) * fix(count_tokens): answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens * fix(count_tokens): answer 400 for Vertex Mistral counts and type the input_tokens error tests Vertex has no count-tokens route for Mistral or codestral models, so they now get the same 400 as other partner models without a count API instead of a 404 that points operators at IAM. * fix(count_tokens): unwrap prefixed provider errors on /v1/responses/input_tokens * fix(count_tokens): say LiteLLM, not Vertex, cannot count a Vertex partner model * fix(count_tokens): count a Vertex partner model with its configured custom_tokenizer and unwrap prefixed flat provider errors * fix(count_tokens): keep the 400 for contents-only Vertex partner counts with a custom_tokenizer * fix(count_tokens): pass through an OpenAI provider error that omits param or code * fix(count_tokens): keep the status-based type for Anthropic count errors on the Responses route * test(count_tokens): cover the Vertex partner count gate and Responses count error shapes on a real proxy * test(count_tokens): add regression tests for the Vertex partner count gate and Responses count error shapes (cherry picked from commit ef1d3959a0) * feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) * feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response Count endpoints now say where each number came from in litellm_count_source (provider, fallback_model, local_estimate) and name the fallback model in litellm_count_fallback_model. When the provider answers 400 or 404 for a model it cannot count, the proxy counts the same request with token_count_fallback_model. token_count_framing adds a per-model constant to local estimates. Both settings ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info. * test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests * test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing * fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown Moves the cost map lookup off the event loop * ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty * fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too. * test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates * fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock calls and about 200 ms on every count, since Opus 5.5 cannot count there either. The Kimi K3 e2e also runs the large system prompt shape now. * test(count_tokens): use the shared integration port allocator in the fallback wire test * fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today * test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it * fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases. * fix(count_tokens): answer the provider's error whenever a framed estimate raises A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error. * fix(count_tokens): load the custom tokenizer inside the guarded framed estimate A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors. * fix(count_tokens): look up framing under the deployment's custom_llm_provider * fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error A framed deployment whose custom_tokenizer is not cached locally made every count wait out the Hugging Face hub's whole retry cycle when the hub could not be reached, and a burst of those loads held the default thread limiter that every count's cost map lookup also needs. Counts on a worker now share one download per minute, a failed download is reused for the rest of that minute, and after a provider error a count waits at most 5 s before answering the provider's error. The swallowed-estimate warning logs only the error type. * fix(count_tokens): keep one tokenizer download per key running and load the local route per request * test(count_tokens): type the scripted counter and wait on events instead of wall time where possible * fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500 * feat(proxy): count tokens with token_count_model in its pinned region token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable * fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys * fix(proxy): keep the upstream error envelope when adding the region hint * test(proxy): annotate the token count setting echo test * ci(cost map): require non-blank token_count_model strings and test the region rules * fix(count_tokens): answer the token_count_model's error even when framing is set * fix(count_tokens): keep the provider error for Google contents instead of a framed estimate * fix(count_tokens): estimate Google contents the route translated to messages * fix(count_tokens): keep the provider error for Google contents with non-text parts * test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock * fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags * feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer * test(bedrock): keep the Kimi K3 image check in-process * fix(count_tokens): join the region hint to an upstream message that already ends with a period Also pins that a blank token_count_model in config falls through to the cost map default * fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning * fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them A flat Responses API function or custom tool becomes the chat tool the Responses route sends to inference, and every other tool goes to the Converse body as is, so hosted tools Bedrock drops are left out of the count instead of being counted or turning it into an estimate. * test(count_tokens): annotate new token counter test locals with Final * fix(count_tokens): keep shipped count regions inside the deployment's geography A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added * fix(count_tokens): resolve count settings off the event loop (cherry picked from commit 75faaeab64) --------- Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 18:20:22 -07:00
yuneng-jiang	ab58c55d4b	fix: backport the third batch of rc/1.107.0 CI triage fixes from main (#47174) ... * test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy (#47152) * test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy These two files register deployments with their own model settings on the gateway. They now boot a module-scoped proxy on a scratch database, so their setup cannot affect other tests that share the worker's proxy * test(integration): seed the isolated gateway with the same router settings as the worker gateway (cherry picked from commit 603ba6fe00) * test(e2e): enable prometheus_system on the redis chaos proxy config (#47160) #46484 moved the litellm_redis_circuit_breaker_* metrics behind service_callback prometheus_system, so the chaos proxy, which only set callbacks prometheus, stopped exposing them and the breaker assertion read 0 (cherry picked from commit 519906e6c2) * fix(tests): wait for every thread of a killed worker before treating it as gone (#47148) WorkerKill returned once the victim's main thread was a zombie. On Linux the process keeps its sockets open until its last thread exits, so a pooled keep-alive connection to the killed worker still looked idle and the next request through it got ECONNRESET. A zombie now counts as gone only once num_threads() is at most 1. Where psutil cannot read threads of a zombie (macOS) it raises ZombieProcess, a NoSuchProcess, so the old behavior holds (cherry picked from commit 0431e75716) * test(caching): assert sync cache-hit logging without depending on thread order (#47093) Sync success callbacks run on a thread pool, so the miss and the two hits can reach the logger in any order. Compare counts instead of the list order. (cherry picked from commit 77c2ac18bd) * test(e2e): hold failing-request RSS to a loose ceiling over two measured batches (#47170) The 48 MB budget on the third batch kept failing on the x86 release gate with no leak behind it. Isolated rc/1.107.0 and v1.105.1 stacks both grew about 60 MB on the third batch with gc-tracked objects flat, and both levelled off to a couple of MB per batch after twenty batches. Measure batches 3-4 against a 200 MB per-worker ceiling instead, which a native leak of 400 KB per failed request still trips. The stored-request size check stays the deterministic catch for the v1.100.0 leak (cherry picked from commit e0514d92b8) * fix(router): read an application profile's base model from metadata on a cache hit (#47144) * fix(router): read an application profile's base model from metadata on a cache hit The 1h prompt-cache pin check read base_model only from the top level of litellm_params. On a response-cache hit the params it sees carry the base model only under metadata.model_info, so an application-profile ARN resolved to itself and the pin was written for 5 minutes while the wire carried a 1h TTL. Read it through get_base_model_from_metadata, which also checks metadata.model_info * test(router): pin top-level base_model precedence over metadata model_info (cherry picked from commit 1e3641a52a)	2026-10-17 14:21:49 -07:00
yuneng-jiang	4bf1633452	fix!: backport the second batch of rc/1.107.0 CI triage fixes from main (#47090) ... * fix(bedrock)!: block realtime replay after an input-first start and reject malformed Nova Sonic session settings (#46936) * fix(bedrock): block realtime replay after input starts the session and reject non-string instructions and voice * test(bedrock): name the malformed realtime setting test for its string cases * fix(bedrock): replay the opening realtime message until Bedrock accepts the stream and refuse any fallback after commit Also rejects a malformed modalities, input_sample_rate_hertz or tools setting with invalid_type, dropped under drop_params. * fix(bedrock): replay a refused opening message on any fallback and reject OpenAI voices and non-string audio formats * fix(realtime): read the client scope only when the websocket has one and reject any-case OpenAI and empty Bedrock voices * test(bedrock): cover realtime replay, refusal and committed failure through a real proxy Fix the scripted Bedrock peer's port import and the nested tool call cell's stale expected events, so the Nova Sonic integration file collects again (cherry picked from commit 3ef0b35255) * test(integration): write worker configs beside test-supplied configs (#47066) Under worker isolation the harness rewrote every proxy config into the owned proxy's directory, so modules a test placed next to its own config (custom_auth) no longer resolved and the proxy died at boot. The copy now lands next to a test-supplied config, while configs inside the checkout still go to the worker directory so nothing is written into the repo tree (cherry picked from commit e754857de9) * test(e2e): keep the prompt-caching affinity test on the live provider (#47072) With E2E_PROVIDER_CACHE=1 the test's Anthropic deployment went through the shared response cache. Each call's recording expires on its own, so a run could replay the cache-writing call while a follow-up went live against a cold provider cache, recording creation instead of a read. That recording then replayed as a failure for 24 hours. Mark the test provider_live, as PROVIDER_CACHE.md requires for prompt-caching tests (cherry picked from commit 6b45a196c4) * test(integration): send post-kill requests on a fresh connection in two worker-kill tests (#47080) kill() returns once the killed worker's main thread is a zombie, but its other threads can still be exiting and holding its sockets open. A keep-alive connection pooled to that worker still looks healthy, so the next request goes out on it and the closing socket answers with a reset. The old reap wait used to hide this. A new connection can only be accepted by a live worker, so the surviving-worker assertions now run on a fresh client (cherry picked from commit c32eea634a) * fix(caching): keep prompt cache usage on streamed cache hits (#47073) * fix(caching): keep prompt cache usage on streamed cache hits A stream=true request served from the response cache rebuilt its usage from the prompt, completion and total counts only, so prompt_tokens_details and the cache read and creation counts were dropped on the replayed stream. /v1/messages clients routed to Bedrock Converse saw input_tokens equal to the whole prompt and no cache tokens on a hit. Both replays now build the usage through one helper that also restores prompt_tokens_details and the integer cache read and creation counts, skipping cached values that do not validate. * fix(caching): coerce replayed token counts the way the live usage does The replay accepted only exact ints for the token counts, so an integral float or numeric string in a cached entry became 0 where the previous replay kept it. Counts now go through pydantic int coercion and fall back only when they do not validate. (cherry picked from commit d346d769f0) * fix(logging): keep the async request's logging payload when the sync success thread runs late (#47019) * fix(logging): keep the async request's logging payload when sync callbacks run late Treat the "cache" success callback as internal when deciding whether to start the sync success thread for async requests. It has no branch in the sync handler, so with `cache: true` that thread did nothing except rebuild and overwrite model_call_details["standard_logging_object"]. When the sync handler does run for an async request (a real sync-only logger is configured), it now builds its own payload and hands it only to its own callbacks. The async handler keeps sole ownership of the shared payload, so edits made by async logging hooks (for example the v3 rate limiter's mirrored x-ratelimit headers that Prometheus reads) are no longer discarded. Sync requests keep their existing behavior. * fix(logging): keep sync cache hits logged and isolate the async-request payload build The sync completion cache-hit path reused handle_sync_success_callbacks_for_async_calls, so treating "cache" as internal silently dropped sync callbacks on sync cache hits. It now submits the sync handler through its own method, as it effectively always did. For async requests the sync handler builds its payload from a copy of the call details, so message redaction cannot rewrite the shared standard_logging_object, and when it builds no payload its callbacks keep the shared one instead of getting None. * test(caching): lock sync cache-hit logging for completion, embedding and text_completion * test(logging): keep sync hook edits off a caller-supplied payload for async requests * fix(logging): give sync hooks a copy of the async payload when the thread builds none * test(logging): check every legacy sync logger gets the thread's own payload * test(logging): cover traceloop and supabase sync logging for async requests (cherry picked from commit a4e0570bea) * fix(cost): price exact cost-map names before price-free capability rules, and fix SCIM and MCP integration tests on CircleCI (#47037) * fix(test-harness): preserve sibling auth modules in isolated worker configs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect servers in MCP connect-flow descriptions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect ptu_hours in daily activity metrics after #43043 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): kill every serving worker once the SCIM push blocks on the row lock * fix(cost): price exact cost-map names before price-free capability rule matches * test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): price an explicit base_model under its own provider after candidate reordering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): retain priced generalized candidates and skip explicit-pricing lookups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): assert custom pricing makes no remote model-info requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost): keep the per-candidate pricing provider final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments * fix(cost): sort cost candidates from the static cost map and rules, without provider lookups * test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yuneng Jiang <yuneng@berri.ai> (cherry picked from commit 4a58784fa2) * test(e2e): warm the memory test up with two batches before the measured one (#47087) * test(e2e): warm the memory test up with two batches before the measured one On a cold worker the second batch of failing requests can still raise the allocator's high-water mark by about 50 MB while the live Python heap stays flat, so the 48 MB budget failed about 1 in 10 Buildkite runs without a leak. The test now runs two warmup batches, measures the third against the same budget, and records every batch's settled growth per worker as an rss_batch_growth JUnit property for later calibration * test(e2e): record each memory batch's growth as soon as it settles A failure in a later batch no longer drops the growth already measured, and the worker keys are no longer sorted, which raised on a mix of named and unnamed hosts * test(e2e): silence the xunit2 record_property warning on the memory test (cherry picked from commit 1fd7f81f97) * test: import pytest and Final for the backported #47073 test on rc On main these imports came from an earlier commit that is not on rc/1.107.0, so the picked test failed collection with NameError --------- Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 00:20:23 -07:00
yuneng-jiang	a00d8a6ee1	fix(proxy): backport the 1.107.0 rc triage fixes to rc/1.107.0 (#47043) ... * fix(proxy): give /team/delete transactions a 30s timeout (#46979) Both interactive transactions in /team/delete used Prisma's 5s default. A lock wait or a short database stall past 5s expired the transaction after the keys were already deleted, so the call returned 503 and left the team row, memberships and team models behind (cherry picked from commit 6530b329f4) * fix(sso): concurrent first sign-ins land on the winner's account, worker-kill test stops waiting on the reap (#46855) * fix(sso): concurrent first sign-ins land on the account the winner created, worker-kill test pins half its streams to each worker * test(integration): a worker kill returns once the worker is dead, the reap waits at block exit uvicorn can take a full worker healthcheck to reap a worker that died mid-ping, so kill() held the caller that long and peers held on the surviving worker hit their client timeout * fix(sso): a sign-in that loses the first-insert race joins the default teams The losing sign-in could read the winner's row before new_user added the default teams, so its CLI key was minted with no team. The race tests now run the real new_user against a fake user table (cherry picked from commit 29cd11bec6) * test(e2e): fix MCP tool search hit matching and the retired Bedrock image model (#46998) * test(e2e): compare parsed MCP tool search hit names instead of raw text * test(e2e): move Bedrock image generation off the retired Nova Canvas model (cherry picked from commit e9a8581a02) * test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) * test(integration): expect the tool search call replayed beside the function call for Anthropic targets * test(integration): wait for the team spend flush before killing a worker * test(integration): poll for the S3 attempt the batch sinks write concurrently * test(integration): queue the terminal object and its sibling into the same s3_v2 flush * test(integration): time the long Bedrock version against an equally long unversioned name * test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it * test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries (cherry picked from commit 910c30381b) * test(ui): wait for the logout navigation before checking the session is gone (#47008) (cherry picked from commit 53ef63145b) * fix(ui): preserve key settings drafts during background refreshes (#41350) * fix(ui): preserve key settings drafts during background refreshes * fix(ui): refresh untouched key settings while preserving edits * fix(ui): keep team-change MCP clears across key refreshes clearMcpGrants now marks the MCP fields dirty, so a background refresh with keepDirtyValues no longer restores the grants a team change just cleared. Also adapts the draft tests to the onSubmit(values, dirtyFields) signature on main and asserts a refresh does not mark untouched fields dirty, since the team admin member-key path rejects saves with non-budget dirty fields. KeyEditView becomes a one-line arrow to stay under the max-lines limit. * fix(ui): keep MCP tool defaults out of a key's edited fields MCPToolPermissions writes non-delete tool defaults for unrestricted servers on mount. Marking that write dirty made a team admin's budget-only save on a member key get rejected for editing mcp_tool_permissions. onChange now says whether a write is a default or a user edit, and the key editor dirties only edits. The edit session is now keyed by token or token_id, matching the form's own fallback, so switching between keys that only carry token_id starts fresh. * fix(ui): keep a new MCP server's default tool limits across refreshes Default tool writes for a server the user just added in this edit session are part of that edit, so they are marked dirty and survive a background refresh. Defaults written for an already saved server stay clean, so a team admin's budget-only save on a member key is still accepted. (cherry picked from commit 37b3a697c7) * fix(logging): stop the spend metadata merge from walking a dict the logging thread writes to (#46044) (cherry picked from commit aa32e4cc88) * test(integration): expect a pre-token invalid prompt on a bridged stream to fall back (#47020) #46493 reads the first stream chunk inside the router's retry and fallback chain, so an invalid prompt rejected before any token now falls back to the configured fallback deployment instead of answering 400. The author confirmed this is intended The fallback test now asserts the spare serves the stream. The old 400 without a fallback attempt still holds when fallback_on_unsupported_params is false, so that contract moves to a second proxy with that setting (cherry picked from commit 856a3f62f3) * test(e2e): repair fallback management test for config-owned fallbacks (#47023) * test(e2e): repair fallback round trip for config-owned fallbacks The e2e stack config sets router_settings.fallbacks, and since #45860 a /fallback write to a key the config file owns is refused with a 400. The round trip now runs on context_window fallbacks, which the stack config leaves to the database, and also deletes the rule and checks it is gone. A second test locks the new contract: a /fallback write to the config-owned fallbacks key returns 400 naming router_settings.fallbacks, and the config rule keeps being served unchanged. * test(e2e): assert the refused fallback rule is never served (cherry picked from commit 59fa785de0) * test(e2e): assert the Vertex cache-creation billing contract in the prompt caching test (#47027) The cache-creating call now reports cache_creation_input_tokens on top of Vertex's own prompt count and bills the write at the input rate, as Google does. The old assertions assumed neither, so the test went red on every run. Check the cache write, the discounted read, and that a second identical call reuses the cache instead. (cherry picked from commit 81a3802a92) --------- Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-16 19:03:48 -07:00
Kerry Lu	cb7e6a7f4c	test(e2e): mixed_size_requests_2rps perf case (coding-agent bodies 5k to 1m tokens with tools, JWT and key callers, two guardrails) (#46717) ... * test(e2e): mixed_size_requests_2rps perf case (coding-agent bodies, JWT and key callers, two guardrails) * test(e2e_harness): test_observe builds DbCounters with statements and expects the background Postgres line --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-16 03:35:35 +00:00
Kerry Lu	34d724c609	test(e2e): perf datastore limits count the Postgres statements and Redis commands a request waits on, background work reported on its own (#46798) ... * test(e2e): perf datastore limits count what a request waits on, background Postgres and Redis work reported on its own postgres_calls_per_request and redis_calls_per_request were whole-server totals (pg_stat_database commits, Redis INFO commands), so spend flushes and config reloads moved them and a new per-request lookup could hide in the noise. They now count the statements and commands a request waits on before it is forwarded (auth and routing reads, the rate-limiter script), from pg_stat_statements read before and after the run and a Redis MONITOR connection held during it. Spend writes, daily aggregates, config reloads and cache invalidations are postgres_background_calls_per_request and redis_background_calls_per_request. The run output lists the top statements and commands per bucket, and the counters are read right after the run so the test's own spend sample is not counted * test(e2e): perf datastore reads only the runtime user's pg_stat_statements rows, other users' rows have a NULL queryid * test(e2e): perf Redis classifier strips the litellm.caching: namespace the deployed gateway puts on every key * test(e2e): small_bodies_5rps limits from ten split-metric runs (postgres awaited 1.8, redis awaited 18, postgres background 12.5), spend buffers and cron locks classified as background * test(e2e): small_bodies_5rps asserts only the awaited Postgres and Redis counts, background lines stay printed * test(e2e): perf Redis classifier puts the cron lock script's GET and DEL in background, not awaited --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 20:10:17 -07:00
Nate Armstrong	f13d0430e8	test(e2e): attach perf case metrics and limits as JUnit properties (#46774) ... * test(e2e): attach perf case metrics and limits as JUnit properties The perf case only printed its metrics table, so a green run left no numbers behind. Each observed metric and its limit now land as <property> entries on the test's JUnit testcase, named perf_<metric>[_p<percentile>] and perf_<metric>[_p<percentile>]_limit, so project-releaser can collect them * test(e2e): compare perf properties exactly instead of through pytest.approx basedpyright flags pytest.approx as partially unknown in the e2e lint step, so the harness test now uses metric values that are exact in binary floating point	2026-10-15 19:12:07 -07:00
Mateo Wang	238597ed49	test(ui): send Connection: close on every integration browser request (#46805) ... * test(ui): close each federation spec API connection so the proxy cannot drop an idle keep-alive socket * test(ui): send Connection: close on every integration browser spec API call * test(ui): send Connection: close on every integration browser request from the Playwright config	2026-10-16 01:11:57 +00:00
mrinal-berri	bd60cc8608	fix(bedrock): forward Nova Sonic realtime tool calls with their arguments (#46214)	2026-10-16 01:10:53 +00:00
Kerry Lu	5fd6379a78	test(e2e): perf users start staggered with jitter, tool-call streams get a first token, small_bodies_5rps sends from the mixed case's callers (#46776) ... * test(e2e): perf streams take a reasoning or tool-call delta as the first token, tool-call replies had no time_to_first_token * test(e2e): perf users start at a random offset and jitter each interval by up to 25%, no more lockstep bursts at the planned rate * test(e2e): small_bodies_5rps sends from the mixed case's callers (two teams, JWT users, keys in and out of teams) through the pre and post guardrail --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 17:14:20 -07:00
Yassin Kortam	d113e4a0dd	fix(proxy-extras): hold only Prisma's lock key during migrate deploy so it never waits on the request log index build (#46660) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 14:47:33 -07:00
yuneng-jiang	46913be92b	test(integration): wait on the right signal in five flaky integration contracts (#46737) ... * test(integration): wait on the right signal in five flaky integration contracts The Prometheus shared-label tests waited for the latency metric and then read the request counter, which the logger writes after an await. They now wait for the counter The auto-router session poll returned on the first 200 while the caller expected two turns, so it now waits for the expected turn count The S3 queue cap test let its failing sink recover on a 15s clock that could expire mid-flush. The sink now stays down until a flush fails after every event is queued K3 normally takes 168-175s against a 180s limit, so the limit is now 300s The MCP env var browser spec reused an idle keep-alive socket that the proxy closes after 5s, so its API calls now send Connection: close * test(integration): add until_asserted so a wait polls the exact asserts it guards eventually() takes a predicate separate from the asserts that follow it, which lets a test wait on one value and then assert on another written later. until_asserted reruns a block of asserts until they all pass and re-raises the last failure on timeout The Prometheus shared-label tests and the auto-router session check now use it	2026-10-15 13:48:51 -07:00
Kerry Lu	fbd078bcbe	test(e2e): perf rig (out-of-process mocks, PerformanceTestCase runner, per-request limits) and the small_bodies_5rps case (#46713) ... * test(e2e): out-of-process mock OpenAI, Anthropic, Bedrock and guardrail servers for the perf cases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): coding-agent request corpus and real provider captures for the perf mocks Request bodies come from a deterministic coding-agent session (system prompt, Read/Edit/Write/Bash/Grep/Glob tools, tool_use and tool_result turns carrying real source files picked by the request marker) in OpenAI chat, Responses and Anthropic formats. captures/ holds one real response per provider, route, shape and stream mode (gpt-6-luna, claude-sonnet-5, us.anthropic.claude-sonnet-5), cold prompt cache, body only, raw stream chunks with time offsets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): replay module for the perf mocks, captured provider responses picked by route, mode and request size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf mocks replay the captured provider responses, knobs shrink to flake and speed The OpenAI, Anthropic and Bedrock mocks answer with the captured body or stream chunks for their route, mode and nearest request size, ids swapped per request, on the captured clock scaled by the p knob. The synthetic reply builders, the per-token stream pacing and the hand-rolled Bedrock event-stream encoder are gone Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf mock request log names its provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf guardrail mock remembers the request marker by call id for the response hook Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf mock captures retaken cold, one marker and system-prompt prefix per cell Run 2 shared one marker across cells, so every stream capture was served from the provider's prompt cache (cache_read_input_tokens up to 523946) and the json cells had small prefix hits. Every cell now carries its own marker, also prepended to the system prompt, and all 30 captures report zero cached tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf corpus reads tests/e2e files so the lane's runner image has them * test(e2e): perf long_context json captures for Claude retaken with a 16k output cap, no longer cut mid-thinking * test(e2e): perf mocks keep only what the replay path and the test read * test(e2e): perf mocks interpolate the provider's time to first byte from the captures around the request size * test(e2e): perf captures at 100k (short) and 500k (long) tokens with tools, the fixed-size set removed * test(e2e): PerformanceTestCase schema, deterministic request plan and wire records for the perf cases RequestShape names a corpus session (input_tokens, tools, output_tokens) and wire.request_body builds it with setup/corpus/session.py, so a 300k request is 85 file reads of real source in the endpoint's own format. The plan no longer predicts reply text, the mocks replay captured provider responses and the runner joins what the client saw against the mock's log by marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf Metric gains time_to_first_token_seconds Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf User loses the unused role field, every JWT user is an internal_user * test(e2e): perf RequestShape sizes requests from an input-token range, short and long stock shapes * test(e2e): perf case runner (provision, Locust plan replay, observe) and the 50 rps mixed-provider case provision.py checks the deployed config against the case and builds the callers, run.py replays the plan from a separate Locust process (locustfile.py) recording status, header, first-byte, first-token and total time per request, observe.py joins the records with what the mocks received and the workers' RSS to score every Limit, and compares each served body with the capture the mock replayed for it as a diagnostic. The case sends 15,000 requests at 50 rps over three model groups, two guardrails and the three request shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf case asserts time_to_first_token_seconds at percentile 50 and 99 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf provision creates every JWT user as internal_user * test(e2e): perf observe reads only the mock log fields that remain * test(e2e): 50 rps case sends short and long coding-agent sessions sized per request * test(e2e): perf run keeps the per-request records as perf-records.jsonl under E2E_ARTIFACT_DIR * test(e2e): the mixed small-gateway case runs at 25 rps (50 saturated one 4-worker pod on main and on v1.105.1) * test(e2e): small_bodies_high_rate perf case, the github issue 46276 shape (11 to 17 KB chat bodies, keys only, 200 rps) * test(e2e): the mixed small-gateway case runs at 10 rps for 6,000 requests (25 rps still saturated one pod on v1.105.1) * ci: python lint job gets 20 minutes, the e2e harness phase (625 tests plus the replay harness suite) pushes a run past 15 when tests/e2e changes * test(e2e): small-bodies perf case runs at 5 rps so gateway_added_seconds is per-request overhead, not queue wait * test(e2e): mixed-provider perf case runs at 5 rps x 3,000 requests from 100 users * test(e2e): mixed-provider perf case sends happy-path traffic only (no flaky deployment, no guardrail interventions or blocks) * test(e2e): perf gateway model_list is three happy-path deployments, one per provider * test(e2e): Postgres and Redis work per request as perf case limits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): perf mocks wait a case-level latency with jitter instead of replaying captured timings, cpu_seconds_per_request limit from metrics-server * test(e2e): perf suite PR carries the small_bodies_5rps case only, the mixed-provider case moves to its own PR * test(e2e): small_bodies_5rps limits from the ten v1.105.1 baselines (1.5x on datastore counts, 2x on latency and cpu, absolute on errors and memory) * test(e2e): perf mocks keep only the short captures the small case reads, captured timings and the Bedrock invoke route removed * test(e2e): perf mocks generate each reply from the request (provider schema, usage, ids, event order, chunk sizes) instead of replaying captures --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 20:14:40 +00:00
Mateo Wang	d5531234ce	test(ci): pin xai reported-cost split and coordinator-held Prisma lock timeout (#46645) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 04:08:56 -07:00
ryan-crabbe-berri	e889d9d653	fix(bedrock)!: replay redacted_thinking on Converse and keep its data as Claude's own string (#46532) ... * fix(bedrock): send redacted_thinking back to Converse as reasoningContent.redactedContent Replaying an assistant turn that carried a redacted_thinking block to Bedrock Converse turned it into an empty unsigned reasoningText, which the assistant content builder then dropped as blank text. Bedrock returned 200 so callers never noticed. Map redacted_thinking to redactedContent in both the thinking_blocks and inline content paths of the sync and async builders * fix(bedrock): drop blank redacted_thinking data and keep non-str signatures signed on Converse replay * fix(bedrock): replay redacted_thinking to Converse only when the target is Claude Nova, Llama, DeepSeek and gpt-oss reject reasoningContent.redactedContent with a 400, so a Claude history replayed to them through a Router fallback or model switch keeps dropping the redacted block the way main does. Claude is resolved from the Bedrock base model of the target, and an application inference profile ARN counts as Claude when its deployment's base_model names a Claude model. Adds integration contracts for the replayed Converse request shape, stream and non-stream, with a tool call, and for non-Claude and ARN targets. * test(e2e): cover Bedrock Converse replaying redacted_thinking live through the proxy Turn 1 triggers redacted thinking on Claude Sonnet 4.6, turn 2 replays it, and a live edge between the proxy and Bedrock checks the replayed Converse body carries the same reasoningContent.redactedContent. Covers nonstream, stream and a tool call round trip * fix(bedrock): validate litellm_params before reading the deployment base_model litellm_params arrives as an untyped dict, so passing it straight into the typed helper grew reportUnknownArgumentType. Validating it into Mapping[str, object] first keeps the read typed without growing the type-check gate * fix(bedrock): wrap non-Converse redacted_thinking data and let a named model id win over base_model * test(bedrock): cover redacted_thinking minted outside Converse, model_id and the sync Router base_model path * fix(bedrock): match bare claude- base_model names without reading the untyped cost map * fix(bedrock): keep redacted_thinking data as Claude's own string on every route Converse hands redactedContent back base64-wrapped, and that wrapper used to leak out as data, so a Converse turn replayed to Invoke or the Anthropic API was rejected. Converse replies now decode it, Converse requests wrap it exactly once, and the Anthropic-shaped builders unwrap histories saved with the old wrapping. Responses API encrypted reasoning is no longer sent to Claude on Converse, which rejected it with a 400	2026-10-14 19:08:14 -07:00
devin-ai-integration[bot]	058c84476a	fix(mcp): leave out tool texts too long to embed instead of failing tool search (#43554) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 14:53:00 -07:00
Mateo Wang	2bb8f7f219	fix(ui): label Decisions playground issues by question key and fix the Response header row (#46145) ... * fix(ui): dedupe blank-question issues and label Decisions questions by key, not position Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): make the Response card header a flex row so title and badges share one line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): fail when duplicate blank-name issues stop being deduped * fix(ui): drop the Decisions issue dedupe, which no reachable endpoint needs * fix(ui): center the Response header and test the Decisions issue labels end to end The card's border rule set 24px of bottom padding on the Response header, which put its content 8px above center. The header now keeps 8px on both sides. New tests cover the /v1/decisions position label and the score-level warning label for an all-digit key, which mutations of validatePayload left untested * test(ui): check the Decisions Response header layout in the browser * test(ui): register the Decisions header spec's model through the API so it runs in CI The CI UI stack does not read fixtures/config.yml, so the config-only fake-decisions-model never existed there and the spec could not pick it. The spec now creates a uniquely named typesafe model against the mock server, waits until /model_group/info lists it as an evaluation model and /v1/systemone answers, and deletes it afterwards --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 10:47:05 -07:00
Yassin Kortam	ae9c208b8d	fix(proxy-extras): bound migration lock waits with SET LOCAL lock_timeout in the migration SQL (#46329) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-13 20:51:03 -07:00
ryan-crabbe-berri	365c4885de	fix(openai): carry cache_control to prompt_cache_breakpoint on chat completions to gpt-5.6 (#46101) ... * fix(openai): carry cache_control to prompt_cache_breakpoint on chat completions to gpt-5.6 A cache_control mark on a system, developer, user or tool message sent to /v1/chat/completions now reaches an OpenAI deployment that reads prompt_cache_breakpoint (gpt-5.6+ on api.openai.com, or one opted in through prompt_cache_options) as a prompt_cache_breakpoint on the same content part, instead of being stripped. Marks on assistant turns and tool definitions are still dropped, and deployments that do not read breakpoints keep today's behavior. * test(openai): rewrite chat prompt_cache_breakpoint wire test as sent-vs-received literals Each test now shows the request the caller sends and the exact body OpenAI receives as plain JSON, with one helper that returns the upstream body. Drops the per-test marker plumbing and the spend-log check, which did not test the cache mark translation. * test(openai): cover a marked chat message with null content on the breakpoint path A system, developer, user or tool message that carries cache_control but whose content is neither a string nor a list of parts is handed to the existing strip untouched instead of raising inside the transform. The new case pins that the request still goes out with the mark stripped and no breakpoint added. * refactor(openai): reuse the cache control hook's breakpoint placement on chat completions The chat transform had its own copy of how a message-level cache_control lands on a part for OpenAI's prompt_cache_breakpoint. It now calls one hook helper that the injection path also uses, so which roles and part types can carry a breakpoint is decided in one place. A message-level mark on list content now lands on the last part OpenAI accepts a breakpoint on, matching injection. * test(e2e): cover cache_control reaching gpt-5.6 explicit prompt caching on chat completions Four live cases through the proxy: a marked system part, a marked string system message, a marked user part and a marked tool result. Each must write the cache on the first call and read it on the second. * fix(openai): parse content parts instead of a banned TypeGuard when renaming cache marks The strict ruff gate bans typing.TypeGuard as unverified narrowing. The part rename now validates an accepted part with pydantic before handing it to the shared rename helper.	2026-10-13 15:45:49 -07:00
devin-ai-integration[bot]	ebc8e56aec	fix(proxy): enforce the personal max_budget on proxy-admin JWTs (#43070) ... * fix(proxy): keep the personal budget on a proxy-admin user object A token carrying the proxy_admin role (JWT admin scope, SSO) replaced the fetched LiteLLM_UserTable row with a bare admin object, so max_budget never reached common_checks and admins spent past their personal budget on /chat/completions. Keep the row and force only user_role. Co-authored-by: L4XB <lukas.buck@e-mail.de> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): budget denial for a proxy-admin JWT is the 422 ExceededBudget response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): carry only the personal budget onto a forced proxy-admin user object Keeping the whole DB row also carried its models list, so a proxy admin whose row had no-default-models or a restricted list got 403 on personal requests. Build the admin object as before and copy max_budget and user_email across. The built-in admin identity (the master key and the UI username/password login) keeps ignoring its row's budget. * test(proxy): pin the contrast case in the admin model-list and admin-name budget tests Each test now also shows the same cached row refusing a non-exempt token, so the pass case cannot go green on a row the gate never read. * test(e2e): drive the personal-budget case with a proxy_admin user row and a plain JWT The Buildkite stack keeps the default admin_allowed_routes, so an admin-scope JWT gets 403 on /chat/completions before any budget check and the case could never pass there. A plain token whose sub resolves to a proxy_admin row goes through the same forced-admin rebuild. Drop the stage-mirror admin_allowed_routes edit, which the deployed stack never read. * ci: rerun e2e after an unrelated Gemini streaming spend-row timeout --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: L4XB <lukas.buck@e-mail.de> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-12 16:04:12 -07:00
devin-ai-integration[bot]	ca4169d20a	feat(providers): add CoralBricks as a native provider (#44700) ... * feat(providers): add CoralBricks as a native provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): harden cost map consistency tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): type cost helpers, read cost maps from disk, cover streaming spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): drop Final on loop-rebound names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): add the provider audit cells, the Add Model browser cell and the README row * test(coralbricks): derive UI spend from model info, type browser rows, always release held upstream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): release held upstream before the wire server joins its threads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): number forwarded chats with itertools.count instead of a mutable list Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(coralbricks): drop retired glm-5.3-flash-fp4 and test against deepseek-v4.1-flash-fast-fp4 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coralbricks): pin retries, await the respawned worker, read the UI rate from the catalog The retry cells now set num_retries on the deployment, since the CI rig posts router_settings.num_retries 0 after readiness and the router default of 2 no longer applies there. The SIGKILL chaos cell waits for the respawned worker's startup before the owned proxy is stopped and derives its timeout from the harness budget. The Add Model spec waits for the catalog the form loads, reads the expected rate from it, and finds the deployment through /v1/model/info, which does not paginate * fix(coralbricks): price the current model names and the new DeepSeek input rate CoralBricks renamed glm-5.3-fp4 to glm-5.3-fast and deepseek-v4.1-flash-fast-fp4 to deepseek-v4.1-flash-fast (the fp4 names stay as deprecated aliases), and cut DeepSeek V4.1 Flash input from $0.30 to $0.01 per 1M on 2026-10-08. Rates read from https://www.coralbricks.ai/api/public/models on 2026-10-10; a live cache write billed 6014 tokens at 9e-08 each, so the cache-write rate stays input plus cacheWritePerM. The test config builders now build new dicts instead of editing the parsed YAML, and the cost-map test locals carry Final. * fix(coralbricks): keep pricing for the deprecated fp4 slugs CoralBricks still serves * fix(coralbricks): list the catalog rows for wildcard deployments and derive the test's input limit from the row * fix(coralbricks): price the deprecated glm-5.3 and deepseek-v4.1-flash slugs --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-12 21:56:59 +00:00
devin-ai-integration[bot]	2717b5f597	test(e2e): rename load to perf and consolidate the opt-in into E2E_PERF_TESTS (#46077) ... * test(e2e): rename the load suite to perf, one E2E_PERF_TESTS opt-in, and a mixed-provider Locust perf test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): drop the mixed-provider perf test, keep the rename and the E2E_PERF_TESTS consolidation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-12 13:54:03 -07:00
devin-ai-integration[bot]	102619d47a	test: migrate proxy security, schema migration and MCP legacy tests (#45937) ... * test: migrate proxy security, schema migration and MCP legacy tests * test: run the MCP proxy e2e upstream under the SDK1 peer in integration-mcp * test: warm the baseline store before timing the locked projection * test(e2e): hold the stack exclusively while the semantic filter flag is flipped * test(mcp): scope the in-process proxy e2e fixtures to their module The session-scoped autouse fixture deleted DATABASE_URL and overrode LITELLM_MASTER_KEY for the rest of the worker, which broke every later test in the integration mcp suite * test(e2e): assert the semantic filter keeps listed tools instead of a ranking Which Datadog tool an OpenAI embedding ranks first is not something litellm owns, so the test now checks that the kept tools are a non-empty subset of the key's listing within top_k * test(mcp): keep the in-process proxy e2e tests on one worker loadgroup spread the file's tests across workers and interleaved them with other files, so the module-scoped in-process proxy was torn down and restarted between tests and requests timed out * test(mcp): name the upstream requests that carried the wrong token The bare all() assertion hid which request broke the forwarded token, which made an intermittent CircleCI failure undiagnosable --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-11 16:17:24 -07:00
devin-ai-integration[bot]	c0ca4a51f7	fix(bedrock_mantle): send max_completion_tokens for gpt-5.6 and gpt-6 chat completions (#43982) ... GPT reasoning-series models (gpt-5.x, gpt-6) on Bedrock Mantle chat completions now send max_completion_tokens, carrying a deployment or client max_tokens over with its value. Every other Mantle chat model keeps sending max_tokens.	2026-10-11 18:28:48 +00:00
devin-ai-integration[bot]	464a4dba9d	test(migrations): assert lock-timeout retry and exhaustion instead of an unbounded wait (#45955) ... * test(migrations): assert lock-timeout retry and exhaustion instead of an unbounded wait Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(migrations): accumulate follower lock wait observations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-11 17:36:47 +00:00
devin-ai-integration[bot]	21cfa0eacd	ci: run every CircleCI check from the integration workflow (#45938) ... * ci: run every CircleCI check from the integration workflow * test(ci): keep the startup test CircleCI-owned * test(ui): find reset log rows by request id so concurrent workers cannot page them out * test(ui): retry the key budget window spec until the draft refresh fix lands * ci(ui): set Playwright workers and retries on the CircleCI command only The shared playwright.config.ts also drives the Buildkite UI suite, where 3 workers ran out of memory on one of two runs. Buildkite keeps main's 1 worker and 2 retries --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-11 07:39:16 -07:00
devin-ai-integration[bot]	3090f4b23e	fix(ci): isolate integration workers and repair test fixtures (#45863) ... * test(integration): wait for customer spend rows and use owned OpenRouter cost * ci: isolate integration workers and clean up resources after crashes * docs: describe isolated extensions workers and shard census * fix(ci): restore integration service and fixture guarantees * fix(integration): preserve replay IDs and native route fixtures * fix(tests): scope integration harness imports to each unit case * fix(ci): preserve fixtures and wait for owned workers * fix(ci): isolate nested runs and telemetry cases * fix(ci): make worker readiness and fault tests deterministic * chore(ci): drop MCP fixture repair already covered by main * chore(ci): keep the local catalog fix from main --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-11 01:56:02 -07:00
devin-ai-integration[bot]	e0398455d0	test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) ... * test: migrate selected llm translation contracts * test: migrate anthropic completion translation tests * test: migrate bedrock and cohere translation tests * test: tighten bedrock translation migration * test: migrate azure and openai model translation tests * test: migrate openai and small-provider translation tests * test: migrate gemini and vertex translation tests * test: migrate optional params and rerank translation tests * test: migrate huggingface, json and openrouter translation tests * test: fix openrouter integration expectations * test: migrate databricks, xai, cometapi and fireworks translation tests * test: migrate ovhcloud, prompt and small provider translation tests * test: split reasoning effort grid into unit and e2e * test: migrate router and realtime guardrail translation tests * test: migrate google interactions translation tests * test: migrate a2a, evals, skills and hosted vllm translation tests * test: remove remaining llm translation suite * test: tighten migrated llm translation unit tests and land missing rows Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover * test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths * test: close lost-intent gaps found in review of the llm translation migration Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised * test: assert provider acceptance in the reasoning effort e2e The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid * test: cover async and stream-end paths the legacy translation suite exercised Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker * test: type the new llm translation e2e calls and mark moved fixture dirs as packages * test: name the reasoning effort e2e models as module constants * test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name * test: block the prompt the shared guardrail denies and drop the reasoning presence check * test: drop the duplicate litellm import left by the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-11 07:58:53 +00:00
devin-ai-integration[bot]	1dbde0845e	test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) ... * test: move guardrails, image, audio, batches, search legacy tests to unit * test: restore the custom guardrail and remaining chunk 1 unit rows * test: move the ocr matrix to e2e * test: drop comments from moved tests and fix vertex ocr credentials * test: move litellm utils and unified google legacy tests to their tiers * test: move logging callback and pass-through unit legacy tests to their tiers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: move store_model_in_db legacy tests and start the admin ui migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: finish the proxy job rows and drop their CircleCI jobs Moves the remaining multi-instance, openai endpoint, otel, pass-through and top-level proxy tests into tests/integration with scripted upstreams, rewrites the weak replacements to assert the outbound wire and exact responses, and removes the build_and_test job group this PR owns from CircleCI * test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names * test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist * test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in * test: type the e2e beta header, prompt cache and secret manager values * test: assert the missing model error message on the OpenAI routes * test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks * test: make the langfuse fixture directory a package for the layout check * test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane * test: keep the ocr e2e pdf fixture next to the test * test: put meta below covers on the streaming tool search e2e * test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures * test: count only OCR calls in the OCR matrix recorder * test: pin OCR parse bodies and billed cost, group new e2e cases into classes * ci: restore the python image anchor dropped in the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-11 00:17:21 -07:00
devin-ai-integration[bot]	c5ac1845b0	fix(langfuse): log Responses API token usage instead of zeros (#42212) ... * fix(langfuse): log token usage for /v1/responses calls ResponseAPIUsage exposes input_tokens/output_tokens, so the Langfuse generation logger read prompt_tokens/completion_tokens as 0 while cost was still correct. Normalize the usage with ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage before extracting usage_details. Resolves LIT-8238 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse): adapt responses-api usage regression test to langfuse v4 spans The v4 migration exports usage_details as an OTEL span attribute via an in-memory exporter instead of a mocked generation() call. Assert on the exported span's langfuse.observation.usage_details with the same token counts. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse): log token usage for streamed /v1/responses calls An assembled /v1/responses stream reaches the logger with its usage stored as a plain dict of chat fields, which the attribute reads saw as zero. Route both ResponseAPIUsage and that dict through one normalizer. * fix(langfuse): keep generations whose usage dict fails validation; cover streaming in the e2e A usage dict whose token counts are not integers falls back to the raw object, so the generation is still logged with zero usage instead of being dropped. The live e2e now runs streamed and non-streamed /v1/responses. * ci: re-run the e2e gate * test(langfuse): run the real prompt helper and label the responses usage e2e cases Drop the _add_prompt_to_generation_params pass-through patch from the Responses usage unit tests so the real helper runs with the empty metadata they already pass. Declare a Subject on both e2e parametrizations (observability, /v1/responses, openai, stream and nonstream). --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-10 16:42:59 -07:00
devin-ai-integration[bot]	a2e334febe	test: migrate local_testing SDK and provider tests out of build_and_test (#45760) ... * test: migrate local_testing SDK and provider tests out of build_and_test * test: cover sagemaker region, hf template and nova paths end to end Region rows now go through litellm.completion on the sagemaker/ handler that the legacy tests exercised, the prompt template test mocks the Hugging Face tokenizer config instead of switching to custom_prompt_dict, streaming tests assert role, finish reason and body, and Nova single turn and streaming run through litellm.completion with a recorded request body * test: tighten migrated integration tests for local OpenAI routes, timeouts and air-gapped boot Split the catch-all providers migration file by behavior. The fake endpoint route tests now hit the owned upstream directly, as the legacy harness tests did. LM Studio asserts the outbound key, path and stripped model. The embedding timeout test asserts the proxy maps a deployment timeout to a 408 and forwards the read timeout to the provider client, instead of timing out the test client The Azure streaming tool-call test also asserts the second identical stream is served from the Redis cache with the same reassembled tool call The Docker network test always skipped in CI because the security job has no litellm/litellm image and no egress to pull one, and it tested a published image rather than the checkout. It now boots the checkout proxy with every outbound HTTP connection refused and asserts readiness, the bundled cost map fallback and that the cost map host is the only egress attempted * test: tighten the migrated e2e rows for local_testing SDK and provider tests Drop the duplicated Anthropic covers marker and the Anthropic cell wrongly claimed by the Bedrock and Vertex cache tests. Keep the file-wide Anthropic model as it is on main. Send the 2023-06-01 version header to Anthropic through the deployment's extra_headers so the header actually reaches the provider, and mark the creation-token case 1h then 5m like the legacy test did Reuse the Bedrock file's own registration for the pydantic response test, swap the hand-built PNGs for the shared cat.jpg fixture, assert the SageMaker Nova image answer names the cat and that top_logprobs=3 is honored, and register the SageMaker Nova opt-in markers the same way as the other opt-in markers * test: restore lost assertions in migrated test_main tests Restore the role alternation check in test_completion_anthropic_hanging, the embedding and streaming response-header checks, the logit_bias drop check and the gemini system and safety settings checks. Add an azure 429 retry-after test for test_exception_with_headers, trigger the router fallback with mock_timeout like the legacy test, and stop the mock timeout tests sleeping 3s * test: restore lost legacy assertions in migrated vertex ai unit tests * test: tighten migrated cost, streaming and embedding tests and scope the hosted mock guard * test: strengthen migrated anthropic, azure, bedrock and cohere provider tests Assert full request bodies and usage instead of presence checks, restore the dummy-tool and custom_llm_provider assertions the legacy tests made, add the async and per-model variants the legacy parametrization covered, split streamed tool arguments across deltas, and move the Azure chat streaming tests to the azure.py mirror file * test: tighten migrated llms provider tests and cover legacy intents lost in the move Strengthen the migrated fireworks, gemini, mistral, together, openai text completion, openrouter, perplexity and openai error tests so they assert what the legacy tests proved (cost, provider tag, usage-only-on-last-chunk, function_call rebuild, reasoning param, sync/async/stream error fields). Replace the huggingface and jina tests that doubled litellm code with respx end-to-end tests. Add the tests the pre-existing mapped tests never covered: cohere 400 mapping, token-array text completion, hosted_vllm/llamafile/lm_studio env api base embeddings, watsonx embedding auth, ollama ssl_verify client and ollama vision prompt * test: cover residual legacy paths for anthropic tool streams, watsonx async stream, ollama http images and caller async embedding clients * test: restore single request assertions in the deep_infra and azure_ai embedding tests * test: drop a no-op streaming cost flag from the openai streaming cost test * test: cover residual legacy-test lines in caching, main, utils, replicate, vertex and bedrock invoke * test: cover streamed audio assembly, choice membership, untranslated responses events and the stream log guard * test: pin cost rates in unit tests, use sdk for cohere e2e, drop vertex llama e2e * test: send the vertex maps e2e request through the openai sdk * test: drop the local_testing jobs that select no tests after the migrations * test: wait for the streamed cache entry and keep custom Mistral rates test-local * test: keep untouched Mistral test lines as they were --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-10 23:14:25 +00:00
yuneng-jiang	3746056caf	test(e2e): cover persisted pricing spend (#41566) ... * test(e2e): cover persisted pricing spend * test(e2e): isolate pricing spend state * test(e2e): synchronize pricing assertions * fix(e2e): share pricing request models * test(e2e): align pricing coverage with the current harness	2026-10-10 16:06:48 -07:00
devin-ai-integration[bot]	906f927b23	fix(proxy/auth): expand all-team-models sentinel for project model access checks (#43734) ... * fix(proxy/auth): expand all-team-models sentinel for project model access checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): use CI-served models and tighten project all-team-models denial check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): declare coverage registry cells for project all-team-models tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: retrigger e2e after ephemeral stack deploy timeout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): wait for team allowlist propagation before project denial check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): annotate written_at as Final per LIT010 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: jesus <jesus@berri.ai>	2026-10-10 15:49:27 -07:00
devin-ai-integration[bot]	5003a3a11b	fix(bedrock): keep tool_reference results as text on converse (#44590) ... * test(e2e): cover tool_reference tool results on bedrock converse Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): keep tool_reference results as text on converse Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(bedrock): format tool reference parser Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover bedrock converse tool_result of only tool_reference blocks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): tag converse tool_reference case with e2e metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): avoid unchecked cast in converse tool_reference mapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): keep tool_reference results as text on responses paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): tighten tool_reference mapping on responses paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(completion-extras): expect tool_reference names in responses input Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(bedrock): drop responses converter changes, keep converse tool_reference fix only Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: kerry <kerry@berri.ai>	2026-10-09 17:52:17 -07:00
Mateo Wang	3069a467d0	docs(agents): ban semicolons and colons in human-facing text (#45684) ... * docs(agents): ban semicolons and colons in human-facing text Tighten the writing rule so ";" and ":" are only allowed where the sentence would be nonsensical without them, rewrite the existing semicolons in AGENTS.md to follow it, and note that docs (litellm-docs) keep their trailing "." * docs(agents): rewrite semicolons in nested AGENTS.md files * docs(agents): hand-maintain the dashboard Next.js warning and sync e2e CONTRIBUTING lines * docs(agents): point at the PR template path that exists * docs(agents): restore the generated Next.js block in the dashboard AGENTS.md	2026-10-09 16:36:32 -07:00
moe-berri	22ad3c5fff	refactor(lens): connect LiteLLM to the independent Lens service (#45529) ... * feat(lens): consume the independent Lens service and shared UI * fix(lens): keep embedded setup stable and vendor UI before builds * chore(lens): pin embedded UI provenance to published Lens source * fix(lens): complete embedded UI extraction across CI builds * docs(lens): record latest main and removal-boundary validation * fix(lens): wire external chart credentials and ingestion modes * test(lens): cover adapter failures and fix extraction CI * docs(lens): refresh bundled chart setup instructions * test(lens): restore gateway adapter test package marker * build(lens): pin clean-cut UI and supported storage chart * test(ui): await guardrail scope menu before checking options * perf(lens): omit unused trace-team lookup from product requests * docs(lens): record final-image adapter permission checks * test(lens): assert delegated investigation authorization * refactor(lens): remove copied trace runtime and preserve spend logging * test(lens): verify independent gateway image upgrades and outages * test(lens): pin the chart qualification database image * fix(lens): update embedded UI metadata filters * test: wait for Lens chart pod identity convergence * fix(e2e): select Lens chart pods by deployment ownership * fix(e2e): select Helm migration ownership for Lens upgrades * test(lens): retain migration Jobs through rollout assertions * docs(helm): require existing database for migration hooks * test(lens): qualify release boundaries with embedded navigation update * docs(lens): record browser qualification for embedded tabs * fix(lens): remove unrelated CI and guardrail test changes * test(lens): give qualification tenants unique key aliases * feat(lens): embed the latest canonical Lens interface * fix(lens): qualify split inference through the gateway * fix(lens): preserve scoped auth and isolate delegated credentials * chore(lens): remove unrelated documentation and lint changes * fix(lens): bound concurrent forwarding buffers through response delivery * fix(lens): preserve OAuth2 dispatch without inference policies * chore(lens): adopt current shared onboarding UI * fix(lens): refresh shared setup UI and review context * fix(lens): bound uploads before gateway authentication * docs(lens): remove extraction evidence from the gateway repo * docs(lens): refresh shared setup prompt for paired releases	2026-10-09 16:17:05 -07:00
ryan-crabbe-berri	4c78023db2	feat(teams): let proxy admins grant team admins the right to raise their team budget (#45404) ... * feat(teams): let proxy admins grant team admins the right to raise their team budget Adds a raise_max_budget entry to team_admin_editable_team_fields. With max_budget alone a team admin can still only keep or lower the team budget. Adding raise_max_budget lets them raise it, capped by the organization's max_budget when the team belongs to one, while removing the budget stays with proxy admins. The entry is rejected without max_budget, /team/info reports may_raise_max_budget to the caller, and the Admin UI nests the new checkbox under Max Budget with a tooltip and shows the team admin a hint under the budget field. * fix(ui): shorten the raise_max_budget tooltip and link it to the docs * fix(teams): pin the organization in the guarded team budget write A granted raise is checked against the team's organization at read time, so the write now also requires organization_id to be unchanged. A concurrent move into a budgeted org returns 409 instead of landing an uncapped raise. Also types the new test helpers. * test(teams): type the team row store methods and the touched race test arguments * test(teams): annotate the last fixture and parametrize arguments in the raise_max_budget tests	2026-10-09 13:39:08 -07:00
yuneng-jiang	cddc7cde97	test: fix stale and flaky tests across CircleCI, GHA and Buildkite (#45530) ... * test(integration): poll the partition lock witness off the event loop The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting * test(e2e): retry the reasoning turn until the model emits a reasoning item Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion * test(e2e): retry a Bedrock Converse model once when it reports no cache tokens Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged * test(e2e): poll until a deleted stored response stops being retrievable Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx * test(e2e): give the Together prompt cache five primed attempts Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed * test(e2e): allow 30s for the idle RSS reading at session start A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged * test(e2e): scope the MCP submission check to its own card The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card * test(router): wait for the primary success record before counting it The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs * test(proxy): give the spend-log cleanup run a 1s budget A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves * test(autorouter): release the slow token count at 2.05s instead of 2.3s The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack * test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting * test(integration): give the two worker-kill chaos tests the owned-proxy time budget Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests * test(integration): compare regional image responses without their created timestamp Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared * test(integration): count only the S3 logger's flush task The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels * test(integration): send guardrail timeout probes eight at a time All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits * test(integration): count a killed Arize chaos worker as gone once it is a zombie psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget * test(integration): send the Typesafe connection test its real model and endpoint The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly * test(integration): keep DB-stored models out of the usage-routing Redis read test The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off * test(vertex_ai): prove the batch upload streams by laziness instead of peak memory The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests * test(integration): keep the alias test_connection call as a known bug * test: type the counting mapper and the Bedrock rerun helper * test: annotate the new test locals as Final and build them as tuples * test(vertex_ai): prove pull-driven transforms with the garbage tail alone	2026-10-09 11:40:14 -07:00
yuneng-jiang	5e1c5c0bd1	test: move offline proxy auth, hook, spend and pass-through tests to tests/unit (#45568) ... * test: move offline proxy auth, guardrail hook, spend, budget reset, GCS payload and pass-through tests into tests/unit Relocate 53 legacy tests that pass with no network or keys into the tests/unit files that mirror the code they exercise, drop 2 llm_guard tests already covered by test_llm_guard_call_type_aliases, and delete the emptied legacy files. * test: freeze the reset clock and run the real reset methods in the moved budget reset tests Pin datetime in reset_budget_job and timezone_utils to a fixed instant and await the service-hook tasks the job spawns instead of sleeping. Drive the per-user failure with a malformed budget_duration row from the prisma mock so the real _reset_budget_for_key/user/team run instead of patched replacements. * test: use fixed timestamps in the moved pass-through and spend log tests * test: mock the moderation HTTP call and type the moved moderation and spend log tests Serve the OpenAI moderation response through an httpx MockTransport so the real async_make_request runs, annotate the moved tests' locals with Final, and pass get_logging_payload typed kwargs instead of an untyped input_args dict. * test: drive the moved proxy tests through boundaries and annotate their locals with Final Serve GCS downloads and the Router moderation call through httpx mocks with fake Google credentials, let the real pass-through success handler fail on the mocked upstream response, and fold the Vertex live route endpoint check into the existing route test. Annotate moved locals with Final, build the budget reset rows without rebinding, and type the remaining fakes without **kwargs	2026-10-09 09:49:13 -07:00
devin-ai-integration[bot]	bbe5595a99	fix(responses): keep a flagged hosted deployment's prompt cache breakpoint on the chat bridge (#45500) ... * fix(responses): keep a flagged hosted deployment's prompt cache breakpoint on the chat bridge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): read the serving provider's row for prompt cache breakpoints on the chat bridge and base_model * fix(anthropic_cache_control_hook): rename the module-level provider resolver so the recursion check passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): follow the credential dialog's new name field and auth method id in the federation e2e spec Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-09 04:00:39 -07:00
devin-ai-integration[bot]	5c6ea040b0	test: update stale MCP call_tool and usage card assertions, add timeout headroom to request-log index boot test (#45523) ... Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-08 23:44:46 -07:00
devin-ai-integration[bot]	8bae65e0b6	test(ui-e2e): update usage page selectors after the #45221 redesign (#45347) ... * test(ui-e2e): update usage page selectors after the #45221 redesign * test(ui-e2e): drop the top keys locator comment --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-08 15:55:22 -07:00
devin-ai-integration[bot]	577d74c1ce	fix(bedrock): count tokens on bedrock-mantle when bedrock-runtime cannot count a Claude model (#45317)	2026-10-08 15:26:49 -07:00
Mateo Wang	ce6889bbf7	fix(responses): keep tool_result next to tool_use on Anthropic previous_response_id continuations (#45322) ... * fix(responses): keep tool_result next to tool_use on Anthropic previous_response_id continuations Replayed history no longer re-adds each stored turn's instructions, and the current request's instructions go before the replayed history instead of between the last tool_use and its tool_result * test(responses): fake the Anthropic transport instead of acompletion in the continuation regression test * fix(responses): keep the previous turn's instructions when a continuation sends none * fix(responses): carry only the latest stored instructions when a continuation sends none Replaying each spend log's own instructions put a system message between a replayed tool_use and its tool_result, which Anthropic rejects with a 400	2026-10-08 09:44:11 -07:00
devin-ai-integration[bot]	0e26edfdb9	test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) ... * test: make legacy live tests in spend, batches, openai endpoints and audio dirs offline (partial) * test: migrate wave-1b live tests offline (guardrails, images, ocr, search, openai endpoints) * test: fix wave-1b review items, add responses/ocr integration tests and firecrawl unit test * test: anthropic messages router/bedrock/openai-bridge unit tests for wave-1b nodes * test: anthropic messages logging, prompt-caching and tool-search unit tests; drop migrated base nodes * test: finish pass_through_unit_tests nodes, logging drain fix and mutations * test: migrate anthropic passthrough tests to integration wire tests * test: fix passthrough migration wire spend row lookup and wildcard config * test: migrate hosted vllm and openai file passthrough tests offline * test: move assemblyai and vertex passthrough nodes to in-process unit tests * test: restore unlisted router node and fix logging worker drain in passthrough unit tests * test: drop spend-row BUG skip and sharpen non-streaming skip reason for anthropic messages * test: use public presidio alias after merge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: restore the batch and file unit tests the migration rewrote away * test: restore full legacy intent in anthropic messages router unit tests Drop the false BUG skip on non-streaming aanthropic_messages logging (success callbacks do fire; the skipped body filtered on the wrong model), assert the logged model_group, messages, cost and usage, cover streaming logging for both Anthropic and Bedrock invoke, assert dict content blocks for Anthropic, Bedrock invoke and the OpenAI bridge, add the Bedrock invoke leg of the router test, fall back from a real 401, test system-prompt caching and streaming message_start cache fields on converse and invoke, send the legacy tool-search tools and beta header, remove the type: ignore and bare dict helper, and add in-process native /anthropic passthrough spend logging tests * test: fix passthrough migration wire tests and drop false native spend BUG skip The native spend row test read rows by the shared master-key digest, so it matched other tests' rows; read each row by its own message id instead and assert tokens, total, spend, tags, provider, api_base and end_user on both the non-streaming and streaming native routes. Merge the streaming test that never checked spend, assert exact tags from litellm_metadata, stop rebinding a Final in a loop, require cost > 0, and add the chat-completions bridge cost case the legacy test covered * test: harden the spend, OCR, image, search and guardrail migration replacements The OCR spend tests now build fresh kwargs per case instead of mutating a shared fixture, and every payload case asserts the exact logged spend. The OCR wire test reads its spend row by request id and checks exact page pricing. Image edit, Nova Canvas, DuckDuckGo, Firecrawl, Bedrock guardrail and Presidio replacements now fake only the provider HTTP boundary (respx or an in-process aiohttp server) and assert the outbound request, so the DuckDuckGo limit, Azure base_model pricing and guardrail masking are proven rather than assumed * test: cover Exa and Perplexity search structure and max_results offline and retire the two base search methods * test: drive batch and file replacements through the provider HTTP boundary and real logging callback Replace monkeypatched litellm.afile_content and AsyncHTTPHandler doubles with respx routes, read batch logging metadata from a registered success callback instead of get_logging_payload, use the real managed-files hook for the GEN-2166 regression, assert outbound request bodies, pin poller ownership explicitly in the migrated DB-sync tests, and require the scripted upstream to be hit in the responses error-status wire tests * test: assert file content download headers pass through the proxy * test: point spend coverage references at the tests that replaced the retired spend job * test: assert passthrough identity, spend and route dispatch from the code under test The AssemblyAI non-admin test asserted metadata it wrote itself and leaked a background poll to the real AssemblyAI host. It now drives assemblyai_proxy_route with a real Request and waits for the success callback for its own transcript id. The Vertex spend test matches its log by call id instead of taking the first event. The OpenAI files wire test hit the native /{provider}/v1/files route; it now calls /openai/files so the passthrough is what forwards the upload and delete. * test: mock only the HTTP boundary in the migrated audio tests Vertex TTS tests no longer replace _ensure_access_token or AsyncHTTPHandler.post; the token comes from a mocked Google OAuth endpoint and the synthesize call from respx. Speech tests assert the outbound body, the transcription cache test polls for the cache write instead of relying on test ordering, and the model pass-through test checks the multipart model field per model. * test: wait on a logger event instead of polling the clock in anthropic messages unit tests Recorders keep payloads in a rebound tuple and set an asyncio.Event; tests await it with asyncio.wait_for instead of a sleep-and-deadline poll loop * test: freeze module-level batch and file response fixtures as Final MappingProxyType * test: fake the presidio analyzer with an in-memory aiohttp connector The blocked-entity tests started an aiohttp TestServer, which binds a local socket. They now hand the guardrail a ClientSession whose connector answers /analyze and /anonymize in process, so no socket is opened and the outbound analyze text and entities are still asserted * test: type anthropic messages router test helpers with LiteLLM's Anthropic TypedDicts Messages, cached system blocks and tool-search tools now use AnthropicMessagesUserMessageParam, AnthropicMessagesTextParam, AnthropicToolSearchToolRegex and AnthropicMessagesTool instead of bare dict shapes; tools are converted to plain dicts only at the acreate call, whose tools parameter is list[dict] * test: type batch limiter helpers with TypedDicts and wait on the logging callback event instead of polling * test: give the migrated OCR, image and presidio helpers precise types OCR spend helpers take ReadOnly TypedDicts for kwargs and responses and use LiteLLM's OCRResponse/OCRUsageInfo instead of local pydantic stand-ins; spend metadata is validated with a TypeAdapter. The presidio fake uses LiteLLM's PresidioAnalyzeRequest/ResponseItem types, and the image-edit logger validates the logged payload instead of storing an untyped dict * test: signal callback and cache events instead of polling Recorders keep tuples and set an asyncio.Event, thread-safely, when the payload for this test's transcript id or upstream URL arrives. The transcription cache test waits on a Cache subclass that signals after async_add_cache. No clock polling or sleeps remain in these tests. * test: assert the batch limiter hook updates the caller's request in place * test: tolerate model-list probes and read native passthrough rows by owned key The router's OpenAI-compatible model-info refresh (litellm/router.py:10710) sends GET /v1/models to configured openai api_bases, so the wire answers it with an empty list and excludes it from the provider-call assertions. Native /anthropic spend rows are now read by a per-request virtual key digest and call_type, then the row's request_id is checked against the message id * test: expect the OCR alias in the proxy response model The proxy restamps every OpenAI-compatible response model to the name the client requested (_override_openai_response_model), so /v1/ocr returns the scenario alias. The upstream model is now checked on the drained request body instead of inside the peer, where a failed assert never reached the test --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-08 02:38:12 -07:00
devin-ai-integration[bot]	da2bb5a6b1	test(e2e): cover responses API gaps on gemini, azure, compact and context management (#45248) ... * test(e2e): cover responses API gaps on gemini, azure, compact and context management * test(e2e): assert streamed usage cost and split anthropic strict schema test * test(e2e): assert compaction items and final stream event in responses e2e --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-07 22:48:29 -07:00
moe-berri	e9cfba2c17	feat(lens): isolate ingestion and investigations in a Rust service (#45148) ... * feat(lens): isolate trace storage and investigation in a Rust service * fix(lens): include Rust sources in the image build context * feat(lens): wire service setup, scoped delivery receipts and lease attempts * fix(lens): complete service routing and reject stale investigation results * fix(lens): retry key propagation and validate isolated Compose setup * fix(lens): seed through isolated ingestion and preserve upstream queue fixes * chore: sync schema.prisma copies from root * fix(lens): bind nullable due timestamps as text for Prisma * chore(ui): remove stale lint suppressions * fix(lens): address CI failures and review findings * refactor(lens): remove retired Python worker and run evaluations in Rust * fix(lens): reuse control connections and satisfy review checks * test(lens): install and upgrade both Helm charts on Kubernetes * test(lens): run connection reuse coverage as an integration test * fix(ui): upgrade Next.js to 16.3.8 security release * fix(lens): fence stale attempts and preserve reviewed evidence * Revert "fix(ui): upgrade Next.js to 16.3.8 security release" This reverts commit 2f79a51b25. * fix(lens): stop failed investigations and stream history excerpts * test(lens): cover model tool and result contracts * test(lens): fix retired routes and reuse installation build artifacts * test(lens): use portable grep in Helm installation smoke * test(lens): wait for migrations before forwarding Helm services * fix(lens): keep failed evidence reads retryable * fix(lens): preserve sandbox output during process exit --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-10-07 17:18:58 -07:00
ryan-crabbe-berri	679f7e636e	test(e2e): record steps for raw transport calls and poll helpers (#45150) ... Add @step labels to the HttpTransport methods, the poll and wait helpers and the boot helpers that did real IO without recording a step, so a test that reaches the proxy through them no longer reports an empty or gappy step timeline in the JUnit report.	2026-10-07 15:21:26 -07:00
ryan-crabbe-berri	048a1500df	test(e2e): move the harness self-tests out of tests/e2e (#45172) ... * test(e2e): move the harness self-tests out of tests/e2e The nightly Buildkite run copies tests/e2e into the runner image and runs bare pytest, so the 672 tests of the harness itself (fixture parsing, JUnit properties, the stack lock, the load aggregators, the Claude Code driver) counted as e2e tests on the status page even though none of them reaches a proxy. They now live in tests/e2e_harness, mirroring the tests/e2e layout, and run in the GitHub Actions lint job and the CircleCI provider_replay_harness job instead * fix(ci): point the providers replay controls at tests/e2e_harness The providers integration job still selected the four replay-control tests under tests/e2e/test_provider_edge.py, so pytest exited before they ran. The raw-HTTP check's file walk also drops to one loop per comprehension * style(tests): mark the raw-HTTP check's bindings Final	2026-10-07 22:20:02 +00:00
devin-ai-integration[bot]	eb9355d11a	feat(ui): configure Anthropic workload identity federation from the dashboard (#44889) ... * feat(ui): configure Anthropic workload identity federation from the dashboard Add Credential and Edit Credential offer workload identity federation for Anthropic, Add Model creates a federated credential and attaches it, and the credentials table marks federated credentials. Editing a credential now sends only the values the admin changed, and switching provider no longer leaves the previous provider's default base URL on screen * fix(ui): keep a credential edit to what the admin set in the federation form * fix(ui): lock the provider in the federation dialog opened from Add Model * fix(ui): require one federation id when the identity source is the proxy environment * test(credentials): cover the Anthropic federation dashboard and credential routes Integration cells for the credential routes every dashboard shape writes (round trips, PATCH set and delete, malformed bodies, non-admin refusals, the token-file allowlist and exchange-host checks, every identity source through chat and messages against a scripted exchange, concurrent writes across two workers and a worker kill mid burst), plus Playwright specs for the Add Credential, Edit Credential and Add Model federation flows and the team-admin view. The owned proxies boot with a 2 s config reload so both workers serve a stored credential inside the fixture budget. * test(e2e): type the federation spec's captured bodies and clean up the Add Model deployment by its created id captureRequestBody and postAsMaster take a type parameter instead of returning Record<string, any>, the spec names the credential and model write shapes it captures, and the Add Model cell reads the deployment id from the /model/new response right after the click so a later failing check no longer leaves the deployment behind. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-07 13:38:58 -07:00