100 commits

Author	SHA1	Message	Date
devin-ai-integration[bot]	c80f1c2ec8	fix(router)!: count native Responses WebSocket turns toward deployment limits and budgets (#45727) ... * fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns. * fix(responses): bill the usage a failed native WebSocket turn reports * fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect The v3 limiter stashed whichever litellm_call_id the request body carried as the slot owner, so a client-supplied id on a pass-through body left the proxy's own success and failure events unable to release the slot. The owner is now the proxy's logging call id. A WebSocket connection whose turns log under their own ids never settles the TPM, ITPM, or OTPM estimate the connection reserved, so the disconnect release now refunds any reservation no callback settled, once. * fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect A failed turn's reported usage now counts toward the key's TPM and the project ITPM/OTPM scopes the same way a success does, and closing the socket hands back only the estimate still unused in the window that holds it, so a window that already rolled is left alone. An id-less error event is booked against the newest pending turn, the one the backend rejected at parse time, and a turn that nests its input under response logs its user message. A connection's one budget reservation goes to the first turn that settles it, every later turn charging its cost directly. * test(proxy): expect the settling call id on a pass-through budget reservation * fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM A native Responses WebSocket turn reaches the rate limiter's success callback with its raw event list as the response and no combined_usage_object, so the turn's tokens never reached the key's TPM window or the project's input and output token limits. Read the usage from the turn's events when the call type is a Responses WebSocket turn. * fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas The limiter's failure hook looked up the parallel slot by the request body's litellm_call_id, while the slot belongs to the proxy's logging call id, so a pass-through request a pre-call hook rejected held its slot for an hour when its body carried an id. Both now resolve the id the same way. Each response.create frame already charges its estimate to project ITPM/OTPM, so the turn's own usage no longer adds a second count. The cost callback's failure hook read the body's litellm_call_id through a strict model and raised on a non-string value, which dropped the failure row. It now reads a string id or none. * fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right Free a rejected request's parallel slot through the request it runs in, and skip native Responses WebSocket connections, whose disconnect frees the slot. Count WebSocket turns toward the connection's tag TPM. In collector mode, claim the shared budget reservation before the spend event is built, so only the first turn settles it. Store each turn's own frame as its request body, and send an id-less error to the oldest waiting turn unless an earlier turn is streaming. * fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag * fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls * fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes * fix(router): count native Responses WebSocket turns toward deployment limits and budgets * fix(router): re-check deployment budget and RPM on each Responses WebSocket turn * test(router): list the WebSocket turn admission method as covered through its partial * fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn * fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure * fix(proxy): log the routed model group on a WebSocket turn's failure row * fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn * test(integration): cover per-turn admission on Responses WebSockets Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP controls for TPM counting and the failure row model_group * fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success callback such as the response cache no longer runs each CustomLogger's sync success hook on top of the async one. That second run counted every turn's tokens twice toward deployment TPM. A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket closes with 1011, and the error becomes the socket's failure, on both the native and the managed relay, instead of the socket dropping with no frame and no failure row. A refused turn's failure row now carries the request's tags from the router's metadata, the same way it already carries the model group. * test(integration): cover a crashed WebSocket turn check and failure-row tags A turn whose admission check raises gets a 500 server_error frame, the socket closes with 1011 and its one failure row carries the error, on both relays. The refused turn's failure row keeps the caller's tags, and HTTP failure rows on chat, responses and messages keep them too. * fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which a client that sends its next response.create early would lose. A refused turn's failure row now takes the router's tags only when every tag is a string, so the fallback never copies a caller's unvalidated tags onto the row. * test(integration): cover a turn still streaming when the next WebSocket turn's check crashes A native turn still streaming when the next turn's admission check raises now has its own cell. It must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for one more turn, so a relay that stopped checking turns after a crash goes red. * fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets A turn after the first now runs the project quota check, then the proxy's pre_call guardrails, then the deployment's turn admission, on both the native and the managed relay, the order HTTP calls use. A turn a guardrail blocks gets the guardrail's error even past the budget and no longer counts toward the deployment's rpm or runs its pre-call checks. * fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it A turn the per-turn guardrail passes and the deployment then refuses now writes its own failed spend row with that turn's guardrail information, on both relays, the way a guardrail-blocked frame and an HTTP refusal already do * fix(responses): bill a refused WebSocket turn on one row only A refused turn whose guardrails recorded a cost is already billed on its own row, so it no longer also becomes the connection's close-time failure row * fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them * fix(responses): bill a WebSocket's refused turns when the session limit closes it * fix(responses): bill refused WebSocket turns when the session limit cancels the relay * fix(responses): free the WebSocket slot at the session limit without waiting on the failure row * fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot * fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown * fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after * fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it * fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close * fix(proxy): always commit spend again after the shutdown wait for pending logging tasks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 15:35:20 +00:00
alexcastgonz	6762d65eb8	fix(vertex_ai): price streamed pass-through calls with the regional uplift (#44795) ... * fix(vertex_ai): price streamed pass-through calls with the regional uplift The streamed Vertex pass-through handler builds the response with stream_chunk_builder, which stamps a cost into _hidden_params without the vertex_location. The handler then prices the response again with the location from the URL, but only into kwargs, and the logger prefers the stamped cost. Streamed calls on a regional endpoint were billed at the global rate. Stamp the location-aware cost on the response in the shared generateContent payload builder so the logger records it. Fixes #44755 * style(vertex_ai): keep the response_cost assignment within the 120-char limit * refactor(vertex_ai): set the passthrough response cost with set_hidden_param Writing _hidden_params directly mutated a parameter and needed a pyright ignore, which the type discipline gate counts as LIT004 and LIT011. set_hidden_param is the shared helper for this. * test(integration): check Vertex pass-through spend rows keep the regional uplift when streamed * fix(proxy): price streamed Vertex pass-through spend like the non-streamed call Drop the location-blind cost stream_chunk_builder stores instead of stamping a cost, so the logger prices streamed generateContent and rawPredict calls with the URL's location and the deployment's custom pricing, the same way it prices non-streamed calls * fix(proxy): price custom-priced Vertex pass-through deployments at their own rates The stamped response_cost from the generateContent logging helper is returned before the logger checks custom pricing, so a deployment with custom input/output rates was billed at the cost-map price. The helper now passes custom_pricing and the deployment id into completion_cost, so the stamp holds the deployment rate for streamed and non-streamed calls, and deployments without custom pricing keep the regional uplift. * fix(proxy): bill Vertex /v1beta and interrupted pass-through streams by the deployment location and price * fix(proxy): give the new pyright ignore a reason for the LIT004 gate * test(integration): bill a custom-priced Vertex pass-through stream at the deployment price, cut-off streams included * fix(proxy): let the logger price non-streamed /gemini pass-through calls so both modes bill the same * fix(proxy): keep a client base_model out of pass-through pricing and price gateway Gemini streams as Gemini A top-level base_model in a pass-through body now reaches the logging params only for a key or team with allow_client_pricing_override, so a plain key cannot pick a cheaper model's price once the logger prices these calls. The test that pinned every litellm-owned body name into the logging params now leaves base_model to its own key-gated test. The Vertex handler now classifies a non-Google upstream host as Vertex only for Vertex resource paths, so a /gemini stream through a GEMINI_API_BASE gateway is priced as Gemini without the regional uplift. Adds the audit's integration spend test for Claude on Vertex, /v1beta, the named custom-priced /gemini deployment, and a cut OpenAI stream. * test(integration): price the gateway /gemini stream case as Gemini and cover client base_model and gateway host pricing The cost harness sends /gemini through GEMINI_API_BASE, which the pass-through now logs and bills as gemini, so the streamed case moves from the Vertex cases to the Gemini cases under the Gemini price. New spend tests cover a body base_model repricing only for keys allowed client pricing, a GEMINI_API_BASE gateway billing as Gemini in both modes, and a configured non-Google gateway picking the provider from the path. * fix(proxy): price pass-through Gemini by the route and ignore a non-string base_model The Vertex pass-through handler picked Gemini pricing from the URL path, so a configured pass-through to a Vertex gateway with a short path (no /projects/ segment) was billed as Gemini, and its Claude streams at $0. The provider now comes from the route: a call through /gemini or to generativelanguage.googleapis.com prices as Gemini, and everything else as Vertex, the same rule main used. Matching the GEMINI_API_BASE host was rejected because one gateway can serve both APIs, and the path rule was rejected because short gateway paths look like Gemini paths. A non-string base_model (a number, list or object) made the cost calculation fail and bill $0 once the stamped stream costs were gone. get_base_model_from_metadata now ignores it and falls back to the deployment's base_model, so the call bills the list price. A 400 was the other option, but the logger runs after the upstream answered. * test(integration): bill a non-string body base_model at the list price on every pass-through route * fix(proxy): bill a non-string model_info base_model at the list price and drop a non-object client model_info A key allowed client pricing can send metadata.model_info in a pass-through body. A non-string model_info.base_model made the cost lookup fail, and once the stamped stream and Gemini costs were gone those calls billed $0 on /gemini and on streamed /vertex_ai and /v1beta. The lookup now ignores a base_model that is not a string, the same rule the root base_model got, so the call bills the list price. A model_info that is not an object (a string or a list) broke several logging steps that read it as a mapping, so a non-streamed /gemini call fired no success callback and no pass-through call wrote a spend row. The pass-through route now leaves a non-object model_info out of the logging metadata, so the call bills the list price. A 400 was the other option, but it would turn calls that answer 200 today into errors, and guarding every reader of model_info would spread the same check over many files. * fix: reject a non-object model_info before the provider call again The previous commit made the base_model lookup return None for a model_info that is not an object. That lookup also runs during call setup on every router route and SDK entrypoint, where such a value used to fail before any provider request. With the early return the call went upstream, the provider billed it, and the request then failed in a later reader that still expects an object. On /v1/messages and /v1/responses the key was also charged for a response the client never got. The lookup again reads base_model from any truthy model_info, so a non-object value fails during setup the way it did before this PR. A non-string base_model inside an object still bills the list price, and the pass-through route still drops a non-object client model_info at intake, so pass-through calls keep billing the list price. * fix: match the Gemini API host exactly when picking the pass-through cost provider The provider check accepted any host ending in generativelanguage.googleapis.com, so a Vertex host such as vertex-generativelanguage.googleapis.com was priced as Gemini and lost the regional uplift. It now accepts that host or a subdomain of it. * test: pin that a subdomain of the Gemini API host keeps the Gemini price --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 08:10:52 +00:00
joshua-berri	9c958aff34	fix(logging): deliver completed batch success callbacks consistently (#47192) ... * fix(batches): dispatch configured success callbacks during reconciliation * fix(batches): retain billed job claim when sync dispatch shuts down * fix(batches): finish reconciliation when logging executor is stopped * fix(logging): avoid duplicate custom hooks during async batch dispatch * fix: preserve sync-only custom callback delivery * fix: scope callback deduplication to per-call async recipients * refactor: type per-call async success callbacks * refactor: retain logging constructor callback compatibility * test: support current batch reconciliation metadata * fix(logging): honor per-logger redaction during sync success delivery * fix(logging): centralize completed success callback dispatch * fix(guardrails): avoid duplicate success callback dispatch --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-18 09:19:04 -07:00
yuneng-jiang	44d99475a5	test(integration): stop GET /__observations from draining the upstream log (#47256) ... * test(integration): stop GET /__observations from draining the upstream log GET /__observations used to empty the queue, so a second read in a test saw nothing and a "never reached upstream" check could pass whether or not the request was forwarded. GET now returns every request since the last reset and DELETE /__observations is the reset. An autouse fixture resets the worker upstream before each test, which also covers tests that read without the gateway fixture. Discarded reads used as resets become DELETEs, helpers that accumulated drained reads now read one snapshot, and tests with several phases reset between them. The three vacuous negative checks (rag allowlist, guardrail opt-out chaos, xAI batch output) now assert the absence on the same read that shows a request which did reach the upstream. * style(integration): space the observations unit test like its neighbours	2026-10-18 09:58:05 +00:00
yuneng-jiang	fc27dfdc7d	test(integration): give each owned proxy a private database and Redis by default (#47213) ... * test(integration): give each owned proxy a private database and Redis by default owned_proxy and owned_proxy_process now boot on a scratch database and a per-proxy redis-server unless the caller opts into the worker's state with state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips DATABASE_URL. Under the replica profile the private database gets the writer and reader roles so the reader path stays covered. Gateway carries the database_url its proxy writes to, and the Scenario cleanup checks and delete_key_if_present read from it, so they keep checking something on a private database. The 189 test files with owned proxy sites that depend on the worker's database or Redis opt in with a shares_worker_state module marker, which a module-scoped conftest fixture turns into the shared state. The 7 helper modules with such sites opt in per call, since conftest fixtures and other test files reach them. * test(integration): reach session fixtures with the shared marker and keep the worker retry default The module-scoped fixture that applied shares_worker_state ran after session-scoped fixtures, so proxies those booted went private in marked files. The marker is now applied around each marked item's setup, call and teardown, which covers fixtures of every scope. Owned proxies on the worker database inherited the worker gateway's router_settings.num_retries of 0 from it. A private proxy now gets the same default in its config copy unless the config sets num_retries itself. Five providers files read the owned proxy's spend rows or response cache through the worker's database or Redis and opt into sharing.	2026-10-18 03:37:41 +00:00
Mateo Wang	75faaeab64	feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) ... * feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response Count endpoints now say where each number came from in litellm_count_source (provider, fallback_model, local_estimate) and name the fallback model in litellm_count_fallback_model. When the provider answers 400 or 404 for a model it cannot count, the proxy counts the same request with token_count_fallback_model. token_count_framing adds a per-model constant to local estimates. Both settings ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info. * test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests * test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing * fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown Moves the cost map lookup off the event loop * ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty * fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too. * test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates * fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock calls and about 200 ms on every count, since Opus 5.5 cannot count there either. The Kimi K3 e2e also runs the large system prompt shape now. * test(count_tokens): use the shared integration port allocator in the fallback wire test * fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today * test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it * fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases. * fix(count_tokens): answer the provider's error whenever a framed estimate raises A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error. * fix(count_tokens): load the custom tokenizer inside the guarded framed estimate A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors. * fix(count_tokens): look up framing under the deployment's custom_llm_provider * fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error A framed deployment whose custom_tokenizer is not cached locally made every count wait out the Hugging Face hub's whole retry cycle when the hub could not be reached, and a burst of those loads held the default thread limiter that every count's cost map lookup also needs. Counts on a worker now share one download per minute, a failed download is reused for the rest of that minute, and after a provider error a count waits at most 5 s before answering the provider's error. The swallowed-estimate warning logs only the error type. * fix(count_tokens): keep one tokenizer download per key running and load the local route per request * test(count_tokens): type the scripted counter and wait on events instead of wall time where possible * fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500 * feat(proxy): count tokens with token_count_model in its pinned region token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable * fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys * fix(proxy): keep the upstream error envelope when adding the region hint * test(proxy): annotate the token count setting echo test * ci(cost map): require non-blank token_count_model strings and test the region rules * fix(count_tokens): answer the token_count_model's error even when framing is set * fix(count_tokens): keep the provider error for Google contents instead of a framed estimate * fix(count_tokens): estimate Google contents the route translated to messages * fix(count_tokens): keep the provider error for Google contents with non-text parts * test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock * fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags * feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer * test(bedrock): keep the Kimi K3 image check in-process * fix(count_tokens): join the region hint to an upstream message that already ends with a period Also pins that a blank token_count_model in config falls through to the cost map default * fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning * fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them A flat Responses API function or custom tool becomes the chat tool the Responses route sends to inference, and every other tool goes to the Converse body as is, so hosted tools Bedrock drops are left out of the count instead of being counted or turning it into an estimate. * test(count_tokens): annotate new token counter test locals with Final * fix(count_tokens): keep shipped count regions inside the deployment's geography A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added * fix(count_tokens): resolve count settings off the event loop	2026-10-17 17:11:27 -07:00
alberto	bbd446b476	fix(proxy): report requested model in streamed Responses response.model (#46662) ... * chore: Add failing test case Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Restamp nested response.model in streamed Responses events Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Add tests Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Run format and lint, and fix Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Type nested Responses restamp to satisfy basedpyright gate Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(proxy): restamp the nested model on streamed response.queued events Background streams send response.queued as a GenericEvent, so its nested response.model still carried the backend name. Also drop the new comments * fix(proxy): restamp a nested response kept as a dict on streamed lifecycle events * test(proxy): drive typed events through the nested restamp no-op test * fix(proxy): restamp the nested model on guardrail block frames in streamed responses * test(proxy): assert a cache-polled background response reports the requested model * fix(proxy): restamp nested model on dict events and on every SSE data line --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 19:52:10 +00:00
Mateo Wang	4a58784fa2	fix(cost): price exact cost-map names before price-free capability rules, and fix SCIM and MCP integration tests on CircleCI (#47037) ... * fix(test-harness): preserve sibling auth modules in isolated worker configs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect servers in MCP connect-flow descriptions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect ptu_hours in daily activity metrics after #43043 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): kill every serving worker once the SCIM push blocks on the row lock * fix(cost): price exact cost-map names before price-free capability rule matches * test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): price an explicit base_model under its own provider after candidate reordering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): retain priced generalized candidates and skip explicit-pricing lookups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): assert custom pricing makes no remote model-info requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost): keep the per-candidate pricing provider final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments * fix(cost): sort cost candidates from the static cost map and rules, without provider lookups * test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yuneng Jiang <yuneng@berri.ai>	2026-10-17 06:46:52 +00:00
yuneng-jiang	910c30381b	test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001) ... * test(integration): expect the tool search call replayed beside the function call for Anthropic targets * test(integration): wait for the team spend flush before killing a worker * test(integration): poll for the S3 attempt the batch sinks write concurrently * test(integration): queue the terminal object and its sibling into the same s3_v2 flush * test(integration): time the long Bedrock version against an equally long unversioned name * test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it * test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries	2026-10-16 22:45:29 +00:00
Deepanshu Pal	d5ce7dbb8c	fix(proxy)!: honor caller litellm_session_id in spend log session_id (#44199) ... * fix(responses): scope session history lookup to the referenced key * Implement SQLite-backed DB for session tests Added a SQLite-backed database class for testing and a new test to ensure session isolation between different API keys. * fix(proxy): honor caller litellm_session_id in spend log session_id Updated docstring to clarify session ID resolution logic for spend log rows. * test(proxy): caller litellm_session_id wins over the trace id * style(proxy): fix blank lines around new tests * fix(proxy): read the caller session from the logging payload for spend logs * fix(responses): match user and team too when replaying session history * fix(proxy): keep bridged Responses chains whole and scope session history by owner * fix(proxy): keep a metadata session next to a trace header and let session headers win * fix(proxy): send the trace-header session only to spend logs and let session headers win beside a trace header * fix(proxy): drop a client-sent caller session and keep it out of requester metadata * fix(proxy): keep the trace id as the spend log session when the caller session contains a NUL byte * fix(responses): keep a continued bridged chain's session under request_correlation_in_logs With the flag on, the caller session outranked the chain session the bridge sets for a continued Responses chain, so a chain whose session changed or first appeared on a later turn split. The continuation now logs under the chain's session and drops the turn's caller session key, as on main. Adds the /audit integration cells for the spend log caller session. * fix(responses): log a continued bridged chain under its session through the spend-log key only The previous commit set the chain session as the turn's litellm_session_id, which also moved the OTel conversation id of a continued turn away from its trace header. The bridge now writes the chain session into the internal spend-log session key of the metadata the spend log reads, so only the spend log session changes, as on main. Adds an OTel cell for a continued chain. * test(spend): cover bridged chains that share one caller session * test(spend): cover the caller session on configured pass-through routes * test(spend): type the SQLite fake and split stacked comprehensions in the caller session tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 13:15:05 +00:00
devin-ai-integration[bot]	4e2dd65549	fix(interactions): keep background interaction bills through crashes, late restarts, and deployment keys (#45743) ... * fix(interactions): keep background interaction bills through crashes, late restarts, and deployment keys A settlement claim is now a lease, so a row whose settler died between claim and outcome is claimable again after the lease and boot resumes it. A resumed poll gets the full poll window and fetches with the serving deployment's credentials from the live router. The stored create context keeps the request tags and the request metadata that the creating worker's own spend row used * fix(interactions): bill a lapsed claim's takeover only when no spend row exists A settler that billed and then died, or failed to record the outcome, left the row unsettled, so the replica that took over its lapsed claim billed the interaction a second time. A takeover now looks up the spend row keyed by the interaction id and records the row as billed when it finds one. * fix(interactions): settle Vertex AI and edited deployments, never double bill with spend logs off A takeover with spend logs off records the interaction unsettled instead of billing it again, and a release records its outcome before freeing the budget reservation. A resumed or live fetch carries the deployment's Vertex AI project, location, and credentials, and a stale deployment id falls back to the model group only when one same-provider deployment serves it. A poll stops at the first 400, 401, 403, or 404, string request tags no longer fail the create, and settled_at gets an index built concurrently. * fix(interactions): leave a live replica's interaction to it when a resumed poll cannot read it A replica that boots with credentials unable to read an interaction got a 404 on its resumed poll and settled the row as unsettled at once, so the replica that created it, still polling, found it settled and never billed it. A resumed poll that gets a 400, 401, 403, or 404 now stops fetching and keeps checking the row until its poll window ends, and settles it as unsettled only when no other replica did. * fix(interactions): keep the creator's metadata on a resumed bill and retry outcome writes inside the claim lease A bill another replica writes now carries the metadata the creating worker's own spend row is built from, so a client's metadata.model_group no longer relabels it. A failed outcome write is retried for half the claim lease before the budget reservation is left to a takeover, and the creating replica retries every fetch error for its whole window again, as main does. The resume integration case now checks that a resumed fetch uses the deployment's own key. * fix(interactions): resume background interaction polls only after database models load Boot resume ran before the proxy loaded the models stored in the database, so a resumed poll for a model added through /model/new or the Admin UI fetched with env credentials. A 4xx there stops a resumed poll, so the bill was lost. The settlement now installs after the database models load. The integration test resumes a database model with an eager first fetch and fails with the old startup order. * fix(interactions): renew a settling claim so a slow settle is never taken over and billed twice A settle runs every success callback in turn, and one slow callback could hold it past the 5 minute claim lease. A booting replica or a DELETE then took the live claim over, found no spend row yet, and billed the interaction a second time. The settler now renews its claim every third of the lease until it records the outcome, so only a settler that stopped renewing is taken over. A takeover also read the spend log after writing its claim, so a failed read left the row under the taker's own fresh claim, which its poll then took for another settler's and left unbilled until a reboot. The spend row is now read before the claim is written, so a failed read takes nothing and the claim is retried on the next poll. * test(spend): cover lapsed claim takeovers, rotated deployment keys, late restarts, and resumed bill tags --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 12:00:08 +00:00
Kerry Lu	5ed3629242	fix(responses): write one spend-log row per native WebSocket turn (#46768) ... * fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection logs and bills as its own call when its terminal event arrives, keyed by the resp_ id the client saw, instead of one row at socket close that sums every turn * test(responses): mock the websocket slot release and mark the per-turn start time naive * style(responses): keep the per-turn start time line within ruff format * refactor(responses): type websocket turn events as read-only mappings * fix(responses): settle the websocket budget reservation on the first turn only Every per-turn logging copy carried the connection's admission reservation, so two turns whose cost callbacks overlapped could both reconcile it. The first turn keeps the reservation and later turns charge their cost directly * fix(proxy): keep the legacy limiter's websocket slot until disconnect A websocket turn callback no longer decrements current_requests in the legacy limiter. The socket's slot is released once from arelease_max_parallel_requests_on_disconnect, which now dispatches to the legacy handler as well --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 19:05:22 -07:00
Mateo Wang	d25b58f744	fix(count_tokens): count Anthropic-format requests for OpenAI the way /v1/messages inference converts them (#46759) ... * fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * test(count_tokens): expect the converted Responses input in the budgeted-key count integration test --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>	2026-10-16 01:58:46 +00:00
devin-ai-integration[bot]	67bfb7a148	fix(ollama): prevent reused response ids from dropping spend logs (#39227) ... * fix(spend_tracking): key completion spend logs off the per-call id so duplicate provider response ids cannot drop rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_tracking): narrow the provider response id with isinstance instead of cast Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gcs_bucket): store request payloads under the spend log request id so the logs viewer can find them Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_tracking): validate callback fields for current type gates * test(gcs_pubsub): assert provider response id in spend metadata * fix(ollama): normalize reused response ids automatically --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 17:17:51 -07:00
Mateo Wang	7090588bdf	feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240) ... * feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions * fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel * feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing early-access row, carrying the same limits and capabilities. The AI Studio row mirrors the early-access entry; the Vertex row differs only in provider and pricing source. Prices follow once Google publishes them. Extend the mapped metadata test to cover all three ids, that each routes to its provider, that its backup copy matches, and that the public rows carry the same limits as the early-access row. * feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops * chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap * fix(gemini): follow continuation hops in a loop instead of recursion The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls * fix(gemini): bill Google Search grounding from every continuation request Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain * fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch * chore(gemini): name the type errors the dual-mode stream iterator trips The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up * fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment * fix(gemini): keep one response id across a continued answer A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key * test(gemini): add integration cells for long decode continuation Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once * fix(vertex_ai): fetch the access token again before each continuation request A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged * fix(vertex_ai): start each continuation stream with an empty parser buffer When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async * fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way * fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes * fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not * fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0 A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer Adds the audit's wire tests for both, which fail 24 cells before this change * test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed * fix(vertex_ai): log a continued answer's raw response as one object A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch * test(integration): check a continued answer logs one merged raw response A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch * test(vertex_ai): cover a follow-up tool call and its thought signature in long decode * test(gemini): pin that a one-request answer logs Google's body unchanged	2026-10-15 22:54:23 +00:00
Dushyant Acharya	06efa5de85	fix(proxy): honor disable_end_user_cost_tracking in spend logs and end-user tables (#40945) ... * fix(proxy): strictly honor disable_end_user_cost_tracking in SpendLogs * fix(proxy): drop end-user id on failed requests and ClickHouse rows when end-user cost tracking is off * fix(clickhouse): drop the end-user id from spend log metadata when end-user cost tracking is off * test(spend): cover disable_end_user_cost_tracking across endpoints, failures, cache hits and chaos --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 21:48:20 +00:00
devin-ai-integration[bot]	e344dbd8db	feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043) ... * feat(proxy): split Azure PTU deployments across teams by share A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on, routing hides the deployment from teams holding no share, the v3 limiter enforces each share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team Usage page report PTU-hours next to tokens. * fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name The PTU flag is read from the process environment on every check instead of through the secret-manager path, so a proxy with a hosted secret manager does not pay a round trip per request. A team's ceiling now carries the model's cached-input ratio and settlement charges cache reads at that ratio, matching how Azure sizes a PTU. A deployment registered for one team is found by its team_public_model_name for PTU-hours and the ceiling, since its model_name is rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and parsed_ptu_shares refuses booleans, strings, and non-string team ids, a fractional ptu_count reserves nothing, and models_processed counts distinct deployments rather than holdings. * fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations * fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id * fix(ptu): refuse ptu_shares naming a team that does not exist * refactor(ptu): keep Azure detection under llms/azure and check share teams with one query * fix(proxy): drop the reverted TeamIdSearchFilter import after merging main Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally * fix(proxy): chunk the ptu_shares team lookup under the IN-list bound * fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling * fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling * fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour * fix(ui): seed total_ptu_hours in the empty daily activity metadata * fix(ptu): size usage rows by the name the request used * fix(ptu): size a team's usage rows by the deployment it is served from * fix(ptu): never size a team's usage row by another team's reservation * fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length * fix(ptu): read a share map only through the terms registration requires A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and ptu_effective_from, or one that does not add up to ptu_count, is refused at registration but was enforced once loaded: every other team got the 400 and the share holder a ceiling while no flat cost accrued. _deployment_shares now reads the shares through ptu_terms, so such a row reserves nothing, the same as before the map was read. * fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off 6fb24d1d read ptu_shares through the pricing terms so a termless map reserved nothing, but with the flag off the router skips ptu_config_error for config.yaml entries, so such an entry loaded and served every team, against the rule that a declared split restricts access whether or not cost attribution is on. _deployment_shares reads the declared map again, and the router now raises ptu_config_error for a config-sourced entry declaring ptu_shares regardless of the flag, so a malformed split is refused at boot instead of being honoured or ignored silently. * fix(ptu): charge wildcard-served names, pass-through usage, and team-less settlements against the team's PTU ceiling A name served by a wildcard route now resolves to the shared deployment behind it, so the limiter's ceiling and the team daily activity sizing both find it. Pass-through responses settle the PTU counter from combined_usage_object in Azure-normalized tokens, and a reservation taken at admission is settled even when the success metadata lost the team. * fix(ptu): resolve a wildcard-served name to the one pattern the router picks, and a deployment id before any wildcard A team's wildcard reservation behind an open proxy-wide * route was charged and sized for traffic the * route served, and naming a shared deployment by id next to a catch-all wildcard escaped its per-team ceiling. Router.wildcard_route_deployments now owns the most-specific-proxy-wide-else-team precedence, and both the limiter and the consumption page resolve through it after listed rows and deployment ids. * test(router): pin the wildcard route precedence that PTU sizing and the router share * fix(proxy): accept whole-number PTU shares written as floats, as ptu_count already does A share of 2.0 now reads as 2 on POST /model/new and in config.yaml, where it was refused with a 422 and dropped at load. Fractional, infinite, boolean and string shares stay refused. Adds unit tests for the malformed-rate, provider-over-prefix, zero-token Responses usage, fractional-count and sizing-warning paths. * fix(proxy): settle team PTU reservations in their own minute and keep billed PTU days when shares move * fix(proxy): write each PTU deployment day's team charges in one transaction The backfill counts a deployment day as priced once any team's row exists for it, so a rollup that wrote team A's share and lost team C's left team C unbilled for good. Each deployment day's team rows now go through one Prisma batch with the retry around the whole batch, so a day is fully written or not written at all and the next backfill prices it for every team. * fix(proxy): read the PTU flag from a secret manager, serve share holders from their share, and refuse unfittable calls and share-holding team deletes * test(router): list the cached PTU ceiling helpers in the router coverage check * fix(proxy): read the PTU flag only for share holders, guard org and SCIM team deletes, keep share holders on their shared deployment through retries and the scheduler, and date PTU-hours by the terms in force * fix(proxy): bill a rerun PTU day to the current share holders alone * fix(proxy): replace a PTU day's rows by day and deployment, not a negated team list * fix(proxy): read the PTU flag from the secret manager first, like get_secret * fix(router): scope retry health lookups to the caller only for PTU share holders * fix(router): move the share-holder retry check into ptu_shares Adds ptu_hours and total_ptu_hours to the daily activity integration goldens and fixtures, since every route now reports them * test(integration): cover PTU shares across routing, delete guards, cost attribution and config regex timing --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 18:44:30 +00:00
devin-ai-integration[bot]	fbc2e358bc	fix(responses): bill a background response once, on the first read or poll that sees it finished (#45703) ... * fix(responses): bill a background response once, from the poll job Reading a completed background response with GET /v1/responses/{id} wrote a full-cost spend row on every read, on top of the row the background cost poll writes. Reads of a Responses object (GET, DELETE, cancel) now never bill, and the poll job is the single biller. The poll job now bills with the creating request's attribution (key hash, team, org, user, aliases, tags) read off the managed object row, through a helper it shares with the batch cost poll. The background create persists the creating key and request tags so that attribution survives until the poll runs. * test(proxy): a background response read reports zero cost * fix(responses): bill a background response once, on the first read or poll that sees it finished A conditional update moves the stored row out of queued or in_progress so only one worker or pod bills it, through one re-read stamped with the creator's attribution. A failed billing read puts the row back for the next poll * fix(responses): keep the caller's response id out of the billing failure warning * fix(responses): keep the creating client's User-Agent tags on the one background charge * fix(responses): keep managed background reads free and bill each response once The managed files hook marks reads and cancels of proxy-issued response ids, so only those stay free. SDK and database-less reads of a background response still bill, extra_body background creates are tracked, and the poll's status read no longer cools down a deployment on a 404. * fix(responses): bill finished background responses from the fetched body and store rows Postgres accepts * fix(responses): give the background billing event its own spend log row and bill responses the create left untracked * fix(responses): narrow the managed files hook with a protocol instead of an enterprise import * fix(responses): leave a read the caller retargets with client-side credentials to the cost poll * fix(responses): bill background responses on the creator's end user and report the charge on the settling read The read that settles the bill returns the charge in x-litellm-response-cost, the billed event carries the creator's end user and the key, team, and user budget fields, and its spend row is named after the id the client got. Override reads, cancels, and deletes settle from the configured deployment, a streamed create marks its row billed, stored rows keep no model output, and model insights counts the billed row's spend without a request. * test(responses): cover background response billing across proxies, reads and chaos * fix(responses): settle background billing only for callers allowed to address the response A delete now settles the bill only after the response id access check passes, a streamed create is marked billed when it sets background inside extra_body or its completed event arrives after the last chunk, the billing event keeps the deployment's litellm_params base_model, a key with no user and no team bills no user instead of its owner scope, and the enterprise cost check import is optional * fix(responses): record the bound on the pending status IN filters * fix(responses): send the billed cost split on the settling read and keep the batch cost event as it was The read that settles a background response now sends the whole cost header family from the billed event, so the breakdown headers add up to the charge. Batch cost events go back to main's attribution (creator id as the user, no budget fields); only background responses get the end user and budgets. Integration tests expect the charge on the settling read (or none when the poll billed first), one charge for a streamed create, and SDK flows that poll with the id each read returns. * fix(responses): bill each background response once across upgrades, streams, and failed settles Rows stored before this billing model are found by the issued id and rekeyed by the provider id, and the cost poll retires one a read already billed. A streamed background create is free on its own and bills the finished response through the same managed row before the client sees response.completed, so a racing read cannot bill it again. A settle that fails retries in the background, a retargeted read with no row tracks and bills from the configured deployment, and a delete keeps its billed cost header. Reverts the unrelated json.dumps narrowing in the completion transformation. * fix(responses): bill cache-polled background streams and match poll rows by the issued id * test(responses): cover deletes, failed settles, budgets, base models, and cache-polled background billing Adds integration cells for the background response billing paths the audit inventory lists: a DELETE that settles the bill and refuses a stranger, a settle the database refuses that retries once it recovers, the Prometheus remaining budget, base_model pricing from model_info and litellm_params, the daily model usage rollup, the end user, the settling read's cost split, stored rows without model output, a forged managed marker on the create, streamed creates with background in extra_body, and the cache-polled create --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 11:00:14 -07:00
yucheng-berri	1f4ef4b7d1	feat(proxy): general_settings.disable_requester_ip_address_logging keeps caller IP out of spend logs (#45804) ... * feat(proxy): general_settings.disable_requester_ip_address_logging keeps caller IP out of spend logs When the setting is on, spend log rows store requester_ip_address as null in the column, the metadata and the stored request (which also drops client-IP forwarding headers), and IP addresses in stored error messages are masked, so the Logs page shows no caller IP. The setting is read the way the settings API validates it. Capture is unchanged, so guardrails, logging callbacks and metrics still receive the IP. Toggleable from the Admin UI General Settings. Resolves LIT-7442 * fix(proxy): mask IPv6 addresses that end or start with :: across eight groups The masking pattern allowed seven colons, so valid forms like 1:2:3:4:5:6:7:: and ::1:2:3:4:5:6:7 stayed in stored error text with disable_requester_ip_address_logging on. Also lists the request metadata walker in the recursion check, since it only walks the tree the spend log sanitizer already returned. * fix(proxy): mask IPs in every stored header value and after labels or underscores With disable_requester_ip_address_logging on, client IPs carried in headers outside the forwarding list (do-connecting-ip, incap-client-ip, custom names) and addresses written as host:2001:db8::2, peer_2001:db8::2, socket_192.0.2.1 or an unbracketed IPv6 with a port were still stored in spend logs. * fix(proxy): mask request header IPs before long values are truncated A header value longer than the spend log string limit kept the part of an IP that fell before the truncation cut. Request headers in metadata and requester_metadata are now masked before the body is truncated, like error text already was. IPv4 followed by a dotted port (192.0.2.1.443) is masked again. * fix(proxy): mask the IP piece a header truncation cut leaves instead of pre-masking headers Masking header values before truncation scanned the whole client value, including text the cut drops, and walked hook-made header cycles. Mask after truncation instead, then mask the hex and digit run touching each side of the truncation marker when it is short enough to be part of an IP address. Every nested headers mapping gets the same treatment. * docs(proxy): say the requester IP setting masks IPs in other stored header values * fix(proxy): mask stored error text after truncation and harden the cut-piece pass Error messages and tracebacks were regex-scanned in full before the DB cap, which costs seconds on multi-megabyte provider errors. Mask them after the cap like header values, with the same pass over the run beside each truncation marker. That pass now handles every marker, skips long hex tokens so an address beside one is still masked, keeps at most one address length, and also masks colon-only IPv6 groups. * fix(proxy): drop the requester IP from the originating request an auto router classifier row stores * fix(proxy): drop the requester IP from the classifier input an auto router classifier row stores * refactor(proxy): mask stored IPs with the content filter's ipv4 and ipv6 patterns The requester IP setting now reuses the content filter guardrail's prebuilt ipv4 and ipv6 patterns instead of its own address regexes, so spend logs mask what that guardrail masks. Addresses glued to a label or underscore stay as the content filter leaves them. * fix(proxy): mask stored IPs with the spend-log matcher again The content filter's ipv4 and ipv6 patterns left labelled IPv6 (host:2001:db8::2), unbracketed IPv6 with a port and IPv4 after an underscore in stored error text and header values, and masked parts of text like std::io::Error. Restore the address-validating matcher and its regression cases. This reverts commit 79537a8aa5. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-14 18:16:40 -07:00
Mateo Wang	d81a5719ad	fix(cache): keep cache hits free of unset fields, and follow intentional behavior changes in CircleCI suites (#46359) ... * test(integration): follow count_tokens error surfacing and decisions pricing in CircleCI suites Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): restore Responses input_file PDF pricing coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep integration AGENTS guide unchanged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cache): preserve provider field presence on cache hits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect credential display name in response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): align OpenInference refusal field expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect Akto request ingestion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): expect missing tool error wording Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect no refusal field in cursor chat wire Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 08:56:05 -07:00
Oliver Jensen	14e820ef7c	fix(proxy): scope /global/activity/cache_hits analytics to the caller's own and permitted-team logs (#46210) ... Non-admin callers previously received deployment-wide cache activity, error breakdowns, every key alias on the proxy, and could filter the aggregates by any other tenant's key alias. All four queries now take the caller's read scope as bind parameters, which also shrinks the key-alias and model filter options to the caller's own visibility. The three-way scope resolution (admin / internal user / fallback) now lives in resolve_spend_log_read_scope, shared with /spend/tags and /spend/logs/session/ui instead of being duplicated per endpoint.	2026-10-14 10:01:21 +02:00
devin-ai-integration[bot]	cfa98e2f40	test(integration): cover pass_through endpoint contracts (#44690) ... * test(integration): cover pass_through endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): run configured endpoint contracts on one proxy worker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove pass-through streams, SDK spellings, endpoint renames and bedrock stream bodies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover pass-through wire behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover Vertex and Gemini passthrough authentication Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Prove pass-through runtime method, credential-less Anthropic, stream barriers and Vertex strip layers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(pass_through): gate unquoted Bedrock stream targets and assert configured endpoint part types and methods after rename Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert pass-through query pairs so duplicate params fail Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 07:38:17 +00:00
devin-ai-integration[bot]	feb92d3255	test(integration): cover orgs_users_customers endpoint contracts (#44725)	2026-10-14 07:38:06 +00:00
devin-ai-integration[bot]	5f7de8d6ac	test(integration): cover keys_auth endpoint contracts (#44674) ... * test(integration): cover keys_auth endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover hashed, path and token spellings for keys_auth contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): strengthen JWT mapping lifecycle coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover reset_spend over plaintext and hashed path spellings with reset_to 0 and 0.02 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop Final in reset loop and use drain-then-fresh-refusal in warmed-peer delete proof Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): disable model info refresh in JWT lifecycle rig Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop unused discovery constant Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover cache invalidation and key paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai>	2026-10-14 07:12:28 +00:00
devin-ai-integration[bot]	8822e086ec	test(integration): cover budgets_spend endpoint contracts (#44736) ... * test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): harden budgets_spend contracts per review Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): make spend report window midnight safe and drop redundant comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin organization spend report scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close budgets and spend review findings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop streamed cost header absence assertion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): cover streamed spend on linked budget tiers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): wait for async spend logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 06:58:22 +00:00
devin-ai-integration[bot]	0f0c1bc80b	test(integration): cover teams endpoint contracts (#44705) ... * test(integration): cover teams endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): address review on teams contract tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drain shared upstream observations at fixture setup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove team block, member and bulk permission contracts at runtime Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert email admin adds and whole membership rows for team members Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drive key update grant and revoke, Z-suffixed temp budget expiry, link LIT-9257 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close review gaps in teams endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): harden team member endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 06:29:20 +00:00
devin-ai-integration[bot]	148ca66ecb	test(integration): cover prompts_tags_caching endpoint contracts (#44686) ... * test(integration): cover prompts_tags_caching endpoint contracts * test(integration): assert prompt lifecycle requests per environment * test(integration): pin non-latest prompt versions, label/path spellings and tag header/key spellings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover production PATCH, streamed and root /responses prompts, and memory create scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): ignore proxy model discovery probes in prompt upstream doubles Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove tag update, peer prompt resync, memory team rows and Responses prompt spellings at runtime Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover memory metadata and key_prefix listing, multi-tag spellings for tag cache invalidation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai>	2026-10-14 04:10:31 +00:00
devin-ai-integration[bot]	32f0d680d2	test(integration): cover google_genai endpoint contracts (#44680) ... * test(integration): cover google_genai endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten google_genai contract assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover non-admin agent api_base guard and exactly-once spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expand Google GenAI route contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover adapter responseJsonSchema, tool-loop history, /models alias and anthropic stream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin raw SSE id/retry framing and agent list/versions key guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): strengthen Gemini endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 04:01:35 +00:00
devin-ai-integration[bot]	90a503d98c	test(integration): cover videos_containers endpoint contracts (#44691) ... * test(integration): cover videos_containers endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): end the first container page on an ownerless id Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert container upload parts, SDK video extend, bare video ids, cost from map Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): catch a stale container list cache, cover video edit JSON, bracketed video refs and object input_reference Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin container optional fields, video client extras and the /videos alias on the wire Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin multipart character id decoding on video create as a BUG skip Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 03:49:01 +00:00
devin-ai-integration[bot]	117684cce5	test(integration): cover agents_a2a endpoint contracts (#44650) ... * test(integration): cover agents_a2a endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin agent daily spend to the logged request day and assert cleanups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect agent daily activity newest day first Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover A2A 1.0 task aliases, agent id headers, declared access groups and per-token spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert full A2A stream, header, alias, task and spend contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert forwarded a2a-version header on A2A sends Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): release A2A stream gate only after a full SSE frame Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin A2A trace id, stream headers, denied alias and task methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin A2A card protocol version Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert the whole accepted push config response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 03:40:07 +00:00
jesus-berri	fead47fa7b	fix(proxy): skip failure rows for expired Admin UI session keys on dashboard info routes (#45881) ... * fix(proxy): skip failure rows for expired Admin UI session keys on dashboard info routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): mark DB-boundary write assertion as test-quality-ok Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-12 13:56:05 -07:00
devin-ai-integration[bot]	02feb67c20	test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037) ... * test(integration): build worker stacks concurrently and record proxy boot phases The controller built every xdist worker's database, Redis, upstream and gateway proxy one after another before any test ran, about 95 s per extensions shard, and since every multi-worker suite is now isolated the same serial wait hits mcp, security and cost too. The stacks now start on a thread pool bounded by the CPU count. Each stack reads its endpoints and results directory from an explicit environment instead of a patched os.environ, every stack that started is registered for cleanup, and a failure in one stops the others before the error is raised Owned proxy startup.json now records the wall-clock spawn time, the worker count and when the Prisma CLI child ran, when the port started accepting and when the first worker answered readiness, and exit.json records when the stop started and ended, so the boot time can be split and overlapping gateway boots can be seen in the artifacts * test(integration): boot one uvicorn worker where a test does not assert cross-worker behaviour A 2-worker owned proxy takes about twice as long to become ready as a 1-worker one, because the uvicorn parent imports the app and each spawned worker imports it again. The passthrough upstream error visibility cases, the single-request S3 dedupe cases and the single-request spend log payload cases send one or two requests and assert on the response, the proxy log and the spend row, none of which depends on a second worker, so they now boot the default single worker Cases that do rely on two workers keep them, the concurrent S3 burst through a sink outage, previous_response_id replay, response cache hits, the concurrent spend bursts and upstream stop and recovery * test(observability): settle OTel excluded-services traces by a quiet period instead of a full window _trace_spans slept through its whole 15 or 30 s window on every call even when the trace had long stopped growing. It now polls and returns once the trace has spans and has not gained one for five tenant export intervals, and the module pins the tenant batch exporter's schedule delay to 1 s so that bound is set by the test rather than by the SDK default. Absence checks still run only after the trace stopped growing, and against a build that leaks postgres spans to the tenant they still fail These cases also boot one uvicorn worker now. None of them asserts anything about a second worker --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai>	2026-10-12 09:48:42 -07:00
devin-ai-integration[bot]	7efce7f8e0	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) ... * test(integration): give chaos and timing tests a harness API and gate raw signals Chaos tests killed and suspended proxy workers with raw os.kill and psutil calls and bounded elapsed time with fixed literals, so they raced the uvicorn worker healthcheck and failed on slow CI boxes OwnedProxy now offers kill_worker and replacing_worker, which return only once a replacement answers readiness with its own pid, and suspended, which is refused unless the proxy was launched with a lenient worker healthcheck. timing.py bounds durations against a measured control or the injected delay. run.py measures the proxy import time once and derives readiness deadlines and the per-test timeout from it, writing boot-calibration.json check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time asserted under a numeric literal) for tests/integration outside _support, and every call site is migrated so both report zero with no suppressions * test(integration): steer the assistants wire burst onto both workers The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split * test(integration): restore product-enforced timing bounds lost in the migration Some TQ011 migrations dropped or loosened a bound whose time is the behaviour under test. The websocket session cap only kept its floor, so a cap closing at 300 s would have passed, and several timeouts were bounded by the injected stall instead of the configured product value assert_enforced_within bounds an elapsed time by the configured value plus explicit slack, with an optional floor. The websocket cap, the MCP validation timeout, the guardrail litellm_params timeout, the prisma command timeouts, the baseline projection lock timeout and the key metadata statement timeout use it. The device code refusal keeps its 15 s margin, and the langfuse retried prompt fetch is now bounded by a cold fetch of a steady prompt as well as by the ignored retry-after * fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat in a ping on the killed worker for up to that long before respawning it. It now uses the graceful stop budget, and a worker kill returns only after the victim is reaped, so pooled connections no longer race the dead worker's socket. Several tests on main picked the multiprocessing resource tracker as their victim and never killed a worker. Now that they kill a real one, presidio counts streams cut by the death, delete_model waits for the config reload that removes a deleted model from a replacement worker, and the wedged coordination Redis test kills before the wedge and waits for every parked publish to fail before resuming Redis. The redis cluster test imports the harness through the tests package so it collects in the unit job, and the per-model fallback rig exposes the owned proxy it now needs. * test(observability): only require S3 deliveries the killed worker could not have buffered A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai>	2026-10-12 05:27:33 -07:00
devin-ai-integration[bot]	780c504e00	test: migrate tests/proxy_behavior into integration and unit tiers (#45939) ... * test: migrate tests/proxy_behavior into integration and unit tiers (partial) * test(integration): run settings-dependent management tests on an owned proxy * test(integration): assert stored values in the rewritten management rows * test(integration): pin sibling team allocations against the org guaranteed throughput cap * test(integration): keep Prisma clients out of the shared integration conftest import path --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-11 08:28:42 -07:00
Yassin Kortam	3f16213906	feat(usage): add top users by spend and user activity views (#45310) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-10 12:21:43 -07:00
yuneng-jiang	9b53ee878f	perf(ci): isolate and shard integration suites (#44146) ... * perf(ci): distribute unit cases and reuse security sweep connections * revert(ci): reject first speed trial after new failures * perf(ci): avoid repeated test discovery and parsing * perf(ci): isolate parallel security workers * revert(ci): restore measured security execution isolation * perf(tests): reuse source scans and deduplicate cache cleanup * perf(ci): restore parallel security with safe log batching * test(ci): isolate fixture state and schema comparisons * test(ci): make schema and scheduler fixtures deterministic * test(ci): drain queued logging before fixture teardown * test(ci): preserve logging delivery without idle worker restarts * test(ci): keep guardrail sinks available during parallel bursts * test(ci): isolate full proxy lifecycle from gateway imports * test(ci): preserve first failure responses and sweep evidence * test(ci): restore documentation URL settings between cases * test(ci): isolate open MCP tools in private gateways * test(ci): group private MCP cases and handle exited workers * test(ci): retain proxy exits and MCP failure details * test(ci): repair merge interactions with main Key the schema shard's full clone on artifact-name, since main removed the unit-flag input. Give each owned Redis its own log file again. Hand the realtime transcription tests main added a logging object whose dispatch handlers are awaitable, so the per-test logging drain no longer waits on a MagicMock task forever * perf(ci): shard owned integration cases and close lifecycle leaks * fix(ci): qualify shard execution and harden interrupted cleanup * fix(ci): bound worker supervision and own fault traffic * perf(ci): reuse unparametrized fixtures across bounded case groups * fix(ci): preserve boot fault allowances and complete exit records * test(ci): tolerate startup retries and retain readiness details * perf(ci): balance integration shards by recorded duration * fix(ci): load migration database helpers only for workers	2026-10-10 11:15:19 -07:00
moe-berri	22ad3c5fff	refactor(lens): connect LiteLLM to the independent Lens service (#45529) ... * feat(lens): consume the independent Lens service and shared UI * fix(lens): keep embedded setup stable and vendor UI before builds * chore(lens): pin embedded UI provenance to published Lens source * fix(lens): complete embedded UI extraction across CI builds * docs(lens): record latest main and removal-boundary validation * fix(lens): wire external chart credentials and ingestion modes * test(lens): cover adapter failures and fix extraction CI * docs(lens): refresh bundled chart setup instructions * test(lens): restore gateway adapter test package marker * build(lens): pin clean-cut UI and supported storage chart * test(ui): await guardrail scope menu before checking options * perf(lens): omit unused trace-team lookup from product requests * docs(lens): record final-image adapter permission checks * test(lens): assert delegated investigation authorization * refactor(lens): remove copied trace runtime and preserve spend logging * test(lens): verify independent gateway image upgrades and outages * test(lens): pin the chart qualification database image * fix(lens): update embedded UI metadata filters * test: wait for Lens chart pod identity convergence * fix(e2e): select Lens chart pods by deployment ownership * fix(e2e): select Helm migration ownership for Lens upgrades * test(lens): retain migration Jobs through rollout assertions * docs(helm): require existing database for migration hooks * test(lens): qualify release boundaries with embedded navigation update * docs(lens): record browser qualification for embedded tabs * fix(lens): remove unrelated CI and guardrail test changes * test(lens): give qualification tenants unique key aliases * feat(lens): embed the latest canonical Lens interface * fix(lens): qualify split inference through the gateway * fix(lens): preserve scoped auth and isolate delegated credentials * chore(lens): remove unrelated documentation and lint changes * fix(lens): bound concurrent forwarding buffers through response delivery * fix(lens): preserve OAuth2 dispatch without inference policies * chore(lens): adopt current shared onboarding UI * fix(lens): refresh shared setup UI and review context * fix(lens): bound uploads before gateway authentication * docs(lens): remove extraction evidence from the gateway repo * docs(lens): refresh shared setup prompt for paired releases	2026-10-09 16:17:05 -07:00
yuneng-jiang	cddc7cde97	test: fix stale and flaky tests across CircleCI, GHA and Buildkite (#45530) ... * test(integration): poll the partition lock witness off the event loop The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting * test(e2e): retry the reasoning turn until the model emits a reasoning item Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion * test(e2e): retry a Bedrock Converse model once when it reports no cache tokens Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged * test(e2e): poll until a deleted stored response stops being retrievable Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx * test(e2e): give the Together prompt cache five primed attempts Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed * test(e2e): allow 30s for the idle RSS reading at session start A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged * test(e2e): scope the MCP submission check to its own card The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card * test(router): wait for the primary success record before counting it The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs * test(proxy): give the spend-log cleanup run a 1s budget A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves * test(autorouter): release the slow token count at 2.05s instead of 2.3s The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack * test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting * test(integration): give the two worker-kill chaos tests the owned-proxy time budget Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests * test(integration): compare regional image responses without their created timestamp Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared * test(integration): count only the S3 logger's flush task The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels * test(integration): send guardrail timeout probes eight at a time All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits * test(integration): count a killed Arize chaos worker as gone once it is a zombie psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget * test(integration): send the Typesafe connection test its real model and endpoint The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly * test(integration): keep DB-stored models out of the usage-routing Redis read test The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off * test(vertex_ai): prove the batch upload streams by laziness instead of peak memory The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests * test(integration): keep the alias test_connection call as a known bug * test: type the counting mapper and the Bedrock rerun helper * test: annotate the new test locals as Final and build them as tuples * test(vertex_ai): prove pull-driven transforms with the garbage tail alone	2026-10-09 11:40:14 -07:00
devin-ai-integration[bot]	f09408dd37	test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) ... * test: replace 61 live logging and otel tests with offline unit and integration coverage * test: restore datadog formatting, deliver datadog logs and redis failures over the wire, assert full router hook payloads, tighten stream usage and otel checks Restores the pre-existing test_datadog.py lines the branch had reflowed. Datadog success, failure and redis-failure replacements now assert the gzip body posted to the intake, with redis failing through a real cache on a closed local port. Router hook sequence recorder checks the legacy field types and asserts concrete payloads, exact streaming and fallback sequences. Stream usage asserts the default include_usage request body and the redacted messages value. Otel asserts response id and token counts. * test: assert budget envelope figures, guardrail inspection and exact prometheus samples per request * test: isolate the prometheus latency test on its own deployment so counts do not depend on order * test: cover timed slack delivery, keep the redis failure test off the network, and make new payload types read-only * test: drain the redis test's logging and scope otel span checks to the test's own trace * test: drive the periodic slack flush without a wall-clock interval * test: scope router hook events to the test, script the db clock, split a nested comprehension --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-08 15:55:57 -07:00
devin-ai-integration[bot]	406514fcaf	feat(spend_logs): configure which metadata fields are stored in LiteLLM_SpendLogs (#44659) ... * feat(spend_logs): configure which metadata fields are stored in LiteLLM_SpendLogs Adds general_settings.spend_logs_metadata_fields with mutually exclusive include and exclude lists. The filter runs on a copy of the row right before it is queued for Postgres, so daily spend rollups, budgets and callbacks still see every key. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): keep excluded auto-router savings keys out of published spend log metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend_logs): cover metadata retention across endpoints, failures, cache hits, batches, runtime updates and auto-router publication Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate schema.d.ts for spend_logs_metadata_fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): filter metadata at DB write so guardrail usage sees full rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend_logs): poll guardrail daily metrics instead of reading once Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend_logs): drop timeout comment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): read spend_logs_metadata_fields through typed general settings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-08 14:51:44 -07:00
devin-ai-integration[bot]	b7ecb05340	fix(proxy): mark background responses stale_expired when provider returns 404 (#41724) ... * fix(proxy): mark background responses stale_expired when provider returns 404 Responses deleted upstream (store=false / ZDR rows dropped after provider retention) now move to stale_expired on the poll instead of being retried every cycle until the 7 day stale cleanup. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): drop inline comment in check_responses_cost Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive _get_response seam in 404 stale_expired tests Assigning AsyncMock on the instance keeps the new tests inside the TQ002/TQ008 test-quality ceilings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only expire 404 responses polled through a resolved deployment A NotFoundError from the bare SDK fallback can mean a missing or misconfigured deployment, which a config fix inside the staleness window can still recover. Rows whose poll went through their router deployment are the ones the provider actually dropped, so only those move to stale_expired. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve the deployment before fetching so via_router binds once Drops the reinitialised loop flag flagged by review and modernizes the moved annotation. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive the real response fetch in the poll 404 regression tests * fix(enterprise): expire background response rows on any provider 404 status The GET path maps the provider's 404 body to litellm.BadRequestError that still carries status_code 404, so an except on NotFoundError never fired and the poll job kept retrying the row every cycle. Key the terminal decision on the status code instead of the class * fix(enterprise): expire a background response only on a 404 naming it and skip a bad deployment entry per job * chore(enterprise): record the poll-cycle bound on the managed-object IN lists * test(integration): cover background response poll retirement on the real proxy --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-08 10:35:52 -07:00
devin-ai-integration[bot]	d5173e3d70	test: move 76 live top-level tests to offline unit and integration coverage (#45352) ... * test: move 76 live top-level tests to offline unit and integration coverage * test: keep the mock-param opt-in guard valid once the top-level senders are gone * test: address review on offline replacements * docs(test): drop the deleted harness test file from the harness check command * test: make the key rebind, team member delete and routes integration tests exercise the legacy paths * test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing * test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks * test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds * test: scope the admin health check to the test's own deployment --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-08 10:33:35 -07:00
Mateo Wang	58258409c9	refactor(proxy): expose public names for private proxy helpers (#45170) ... * refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 18:59:27 -07:00
devin-ai-integration[bot]	c3a23fe499	refactor: expose core private helpers under public names (#44871) ... * refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 00:05:27 +00:00
Hektor Jacynycz Garcia	157d96dcd0	fix(logging): log spend once for large non-streaming requests on the chat to Responses bridge (#44508) ... * fix(logging): claim the async success dedup flag before the first await Nested @client wrappers on the chat -> Responses bridge schedule two async success tasks on one logging object. The dedup flag was checked at the top of _async_success_handler_body but only set after several awaits; the worker-thread base64 offload for messages >= 256 KiB yields there, so the second task passed the same check and the request was logged twice (two SpendLogs rows, key/team/daily spend counted twice). Fixes #44500 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(logging): serialise async success handlers per logging object instead of claiming early Claiming the dedup flag before the first await (previous commit) let a first handler that is cancelled before its callbacks run (e.g. by the logging worker's per-coroutine timeout) suppress the second one, losing the request's success accounting. Serialise non-streaming handlers on the same Logging object with an asyncio.Lock kept in a module-level WeakKeyDictionary: the second handler waits, then skips if the first logged, or logs itself if the first raised or was cancelled first. Streaming chunk logging is not locked. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(integration): pin one spend row and one key charge for large bridged chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for the large request's spend row before asserting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(logging): use fixed timestamps in async success dedup tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(logging): trim the async success dedup lock comment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): reset background interaction dedupe under the success dedup lock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(logging): use a fixed start time in the interactions logging helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: claim the async success log at schedule time instead of locking the handler Nested @client wrappers (chat over the Responses bridge, Anthropic Messages over the chat adapter) exit with one shared Logging object and each queued their own async success handler. The handler checked has_logged_async_success before its awaited work and set it after, so both could pass the check and the request was logged and billed twice. _schedule_async_success_logging now claims the log on the Logging object synchronously through claim_async_success_log, and a later wrapper returns without enqueuing. There is no await between the check and the decision, so no lock is needed. The per-object asyncio.Lock, the WeakKeyDictionary that held it and the locked block in async_log_background_interaction_completion are gone. Tests: the regression test moves to tests/unit/test_utils.py and drives two _dispatch_success_logging exits with one Logging object through the logging worker, asserting one log with the inner result. It fails without the claim. The cancelled-handler test is dropped since a second wrapper no longer retries. The background interaction completion test stays. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(logging): pin the background completion event order and name the innermost wrapper in the claim docstrings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): send one bridged request and assert the inner Responses result is the logged one Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-06 21:23:19 +00:00
moe-berri	5147aefa8b	fix(lens): reuse trace reviews and consolidate findings across runs (#44778) ... * fix(lens): reuse trace reviews and consolidate findings across runs * chore: sync schema.prisma copies from root * fix(lens): preserve partial reviews and bound budget admission * fix(lens): resolve CI regressions and clarify reused runs * test(lens): exercise review reuse in worker container smoke * fix(lens): show actual reviews when a run finishes * fix(lens): distinguish reuse plans and stop blocked scans * fix(lens): retain partial findings when model requests stop * docs(lens): clarify budget edits during active runs * fix(lens): publish findings only after reconciliation completes * fix(lens): guide insufficient-budget runs to budget settings * fix(lens): renew budget holds and bound admission waits * fix(lens): preserve provider errors during budget cleanup * fix(ci): update PgBouncer and sharp for current builds * fix(lens): serialize settlement and fence review checkpoints * fix(lens): defer generated Prisma client type import * test(lens): verify checkpoint and progress rollback together --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-10-06 12:09:56 -07:00
nate-berri	b8239a9873	fix(vertex_ai): add regional endpoint uplift to gemini-3.1-flash-image (#44673) ... * fix(vertex_ai): add regional endpoint uplift to gemini-3.1-flash-image Google prices Gemini 3.1 Flash Image at 1.1x on non-global endpoints for input, text output and image output, but the cost map row had no regional_endpoint_uplift_multiplier, so regional calls billed at the global rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(integration): cover regional uplift spend for gemini-3.1-flash-image * test(integration): read the proxy salt from the environment in the uplift spend cells --------- Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-05 21:00:22 -07:00
nate-berri	922ea71fe5	fix(vertex_ai): apply regional endpoint uplift on image generation cost path (#44679) ... * fix(vertex_ai): apply regional endpoint uplift on image generation cost path completion_cost had vertex_location but never passed it to the image generation cost router, so a regional_endpoint_uplift_multiplier on a Vertex image row would be ignored. No image row carries the multiplier yet, so nothing is misbilled today. Pass the location through to the Vertex image calculator for both the token-based price and the per-image fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(vertex_ai): cover regional image cost through the proxy logging path Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(images): hand image_edit's vertex_location to its cost resolver * test(integration): audit Vertex image regional uplift billing * test(integration): require every spend row after a proxy restart * test(integration): reject extra spend rows after a proxy restart --------- Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-05 20:02:22 -07:00
moe-berri	d232b008bc	test(lens): use current worker protocol in billing integration (#44760)	2026-10-05 18:39:38 -07:00
devin-ai-integration[bot]	a1a42768c1	fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses (#43560) ... * fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(spend-tracking): share the short-lived miss cache write Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): cover key alias recovery after spend log lookup failures across usage routes * test(spend): bound outage alias lookups per miss window instead of a fixed count * fix(spend-tracking): treat any spend-log lookup failure as a short-lived miss The Prisma client raises a plain AttributeError when the database drops the connection mid-query, so the PrismaError catch let it through and the whole usage call answered 500. Any failure now keeps the 30 second backoff only, and the integration proxy patches its test entitlement at import so uvicorn's spawned workers inherit it * test(integration): audit spend-log metadata recovery under timeouts and dropped connections Cover the daily activity routes, the usage AI chat, the Vantage and CloudZero dry runs and exports under a locked spend-log table and under a database connection dropped mid-lookup, on a two-worker proxy, with the recovery after the outage asserted through the proxy's own miss TTL. Add a dropped_connection_relay that closes only the connection whose bytes carry a trigger, so a cell can drop the one connection the recovery query runs on while the rest of the pool keeps serving. Rewrite the sweep and JWT cells for the merged main: the export route reads metadata by SQL join and never calls the recovery, the search routes answer key rows and find deleted keys by alias, and the daily-spend owner recovery names the user while the alias stays blank. The sweep cell now times out a second lookup under the same lock, which pins the keys blank on the merge base and recovers on this branch. * test(integration): match a dropped-connection trigger split across two reads The dropped-connection relay checked each TCP read on its own, so a SQL marker that straddled two reads never tripped it and the outage cells would run without the outage they meant to exercise. Carry the tail of the previous read into the next check, as the held-statement relay already does, and pin that with a unit test that splits the trigger across two writes. * test(integration): scan relay triggers through an in-process helper The dropped-connection relay now matches its SQL trigger through a TriggerScanner that carries the previous read's tail, and the unit test exercises that scanner directly instead of opening loopback sockets, which tests/unit forbids. The relay's end to end behavior stays covered by the integration cells --------- Co-authored-by: gabriele <gabriele@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-05 22:36:25 +00:00