* fix(responses): run pre_call guardrails on every Responses WebSocket turn and bill each turn its own guardrail cost
* test(responses): keep the websocket unit test file on main's formatting
* fix(responses): type the websocket frame guard for the lint gates
* fix(responses): pass the websocket frame guard as a named parameter and mark the guardrail dict edits
* fix(responses): bill a guardrail-rejected Responses WebSocket frame as a failed request with its guardrail cost
* fix(responses): type the recorded guardrail records as Mapping for the lint gate
* fix(responses): bill a rejected websocket frame through the spend logger only so the connection keeps its rate-limit slot
* fix(responses): guard nested response.create frames, strip forged user_api_key_* frame metadata, copy guarded tools back and give each rejected frame its own spend row id
* fix(responses): mark the spend logger's bare-dict failure hook for the type gate
* fix(responses): reject response.create frames that carry input at both levels and keep billing of a rejected frame non-blocking
* fix(responses): name the non-blocking billing catch for the strict lint gate
* fix(responses): keep the admitted guardrail list on every websocket frame, a frame cannot replace it with its own guardrails list
* test(responses): wait for the key spend total before asserting it in the websocket guardrail cost test
* fix(responses): copy the admitted metadata shallowly per websocket frame, the OTEL parent span cannot be deep-copied
* fix(responses): reject a WebSocket turn a pre_call guardrail rerouted to another model
* fix(responses): bill a rerouted WS turn and carry Presidio pii_tokens from the frame guard to the socket unmasker
* fix(responses): default pii_tokens through default_factory, Python 3.13 rejects a mappingproxy dataclass default
* fix(responses): run the per-frame guardrail walk on managed Responses WebSockets too
* fix(responses): unmask managed Responses WebSocket replies with the frame guard's Presidio pii_tokens
* fix(responses): strip client guardrail cost records from WS frames so a blocked frame bills the real scan
* fix(responses): forward a WS frame without the fields a guardrail deleted
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(test-harness): preserve sibling auth modules in isolated worker configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect servers in MCP connect-flow descriptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect ptu_hours in daily activity metrics after #43043
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): kill every serving worker once the SCIM push blocks on the row lock
* fix(cost): price exact cost-map names before price-free capability rule matches
* test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price an explicit base_model under its own provider after candidate reordering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): retain priced generalized candidates and skip explicit-pricing lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): assert custom pricing makes no remote model-info requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): keep the per-candidate pricing provider final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments
* fix(cost): sort cost candidates from the static cost map and rules, without provider lookups
* test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* fix(azure): add responses_api_version so the Responses route stops using the deployment chat api_version
* fix(azure_ai): fold Grok reasoning tokens into completion_tokens like the xai provider
* fix(utils): refresh provider model sets after register_model
* fix(cost-map): align azure gpt-6.1-sol reasoning effort flags with the openai row
* test(e2e): cover Azure Responses on a deployment pinned to a dated chat api_version
* fix(utils): keep router deployment ids out of provider model sets
register_model now refreshes the provider sets, so the per-request and
router registrations keyed by a deployment id put those opaque ids into
wildcard model listings. Those callers now pass
add_to_provider_model_sets=False.
* fix(azure): carry responses_api_version through the chat to Responses bridge
completion() dropped responses_api_version from the litellm_params it hands
the Responses bridge, so azure/responses/<model> on a dated chat api_version
still built the 404ing /openai/responses URL. Also cover the auto-router
marker never forwarding the new param.
* fix(azure): stop listing a wrong set of reasoning efforts when none is rejected
The message said only low, medium and high are supported, which is wrong for
models that also take xhigh such as gpt-6.1-sol.
* test(azure): use the latest grok and gpt-5.4 rows in the new azure tests
* fix(proxy): list each wildcard-expanded model once
A provider set can hold a model both bare and provider-prefixed, which now
happens after any priced request registers its openai/<model> key, so the
openai/* listing showed that model twice.
* fix(azure): route Responses past dated GA api_versions by default
A deployment pinned to a dated GA api_version such as 2024-10-21 for chat now sends Responses calls (create, get, delete, cancel, input_items and the azure/responses/ bridge) to the default Responses version, because those dated versions have no Responses route on Azure and 404. responses_api_version still overrides it, -preview, v1, latest and preview versions are kept, an api-version in the api_base query still wins, and chat keeps api_version as before
* chore(azure_ai): mark the grok chunk_parser dict as an override contract
* fix(azure): rewrite only the route's /openai segment when moving a call to v1
The v1 rewrite in the Azure URL builder replaced every "/openai" substring in the path, so a gateway
api_base like https://apim.example.com/openai-gw/team turned into /openai/v1-gw/team/openai/v1/responses.
Dated GA Responses calls now land on the v1 route by default, which made this reachable for them too,
so the rewrite now only touches the last /openai path segment.
The Responses version resolver no longer duplicates the builder's default for an unset api_version.
Tests cover the gateway api_base case, the env AZURE_API_BASE query-string precedence, and the e2e suite
now has one test for the dated GA default and one where only responses_api_version can make the call work
* test(integration): cover Azure Responses api_version, Grok reasoning spend, wildcard listing and reasoning_effort none
Wire tests pin the Azure route and query for chat, Responses and the azure/responses bridge on a dated GA api_version, and check responses_api_version never reaches the request body. Cost cases check azure_ai Grok folds reasoning tokens into completion_tokens and spend, streaming and not. A config-driven proxy checks /v1/models next to an openai/* wildcard lists each id once and never a deployment id. Azure gpt-6.1-sol rejects reasoning_effort none before reaching upstream and drops it with drop_params
* fix(utils): list ai21 chat and vertex code models under their provider wildcard
register_model now files ai21 chat models into ai21_chat_models, which models_by_provider never read, so a registered ai21 chat model dropped out of ai21/* listings. models_by_provider now reads ai21_chat_models and the vertex code chat and code text sets too.
* fix(main): keep caller-chosen model names out of provider wildcard listings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* test(cost): remove price fields the real gemini and azure rows do not carry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): key vertex gemini cases on the vertex_ai prefixed rows and add gemini-4-argon
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add current gemini and vertex usage keys to stored responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add bedrock mantle claude-opus-5-5 cost tracking cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add bedrock mantle gpt cost tracking cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add tier 1 cost tracking cases for gpt-5.6-terra, gemini-4-argon and gemini-3.1-pro-preview
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add audit 3 shape cases for vertex_ai and azure_ai claude
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): mark azure gpt-5.6-terra priority case as known bug
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(realtime): keep turn_detection.type when a transcript guardrail disables auto-response
* test(realtime): type the GA session.update helper and mark its locals Final
* test(realtime): expect typed turn_detection on the voice guardrail session updates
* test(realtime): accept either gate path for the client's first voice session.update
* fix(gemini): drop VAD tuning when the guardrail turns auto-response off
* fix(gemini): check create_response without a redundant isinstance
* fix(vertex_ai): keep the client's VAD tuning on Vertex when auto-response is off
* fix(realtime): keep the transcript-guardrail gate typeless on xAI so it does not auto-answer
* fix(realtime): replace any client turn_detection with the untyped gate on xAI
xAI answers on its own under a typed server_vad even with create_response off,
so a client's typed turn_detection must not reach xAI while the transcript
guardrail gates the turn.
* fix(realtime): keep a client's early turn_detection and hide the guardrail cancel's error
A session.update sent before session.created reached OpenAI ahead of the proxy's own gate, so the gate replaced the client's turn_detection. The proxy now re-sends the client's gated turn_detection after the gate when the two differ.
The guardrail's response.cancel now carries a litellm_guardrail_ event_id, and the error OpenAI returns for it (response_cancel_not_active when nothing is answering) no longer reaches the client.
* fix(realtime): send the guardrail gate before a client's early session.update and let Muse and Chirp fall back to server_vad under drop_params
* fix(realtime): hide only the cancel errors that echo the guardrail's own response.cancel ids
Track the event ids of the guardrail cancels this session sent and drop a backend error only when its
error.event_id is one of them, so a client's own event id with the same prefix still gets its error.
Cover the scripted response.cancel reply in the integration upstream, update the stale list-mode test
to the typed gate, and mark the xAI cancel error cell as an expected failure, since xAI replies with
its own event id.
* test(integration support): deliver the disconnect a real server sends after the client goes away
The scripted upstream now keeps reading frames after a reply hits a closed socket, so the unit test's
fake ASGI receive has to end with the websocket.disconnect message a real server delivers.
* fix(realtime): delete the blocked transcript item before the guardrail's refusal turn
OpenAI GA answered the blocked question in some refusal turns because the
blocked user item stayed in the conversation. The raw OpenAI-protocol path
now sends conversation.item.delete for it right after the cancel. Provider
config backends have no such event, so that path no longer passes an item id.
Also bound the guardrail integration file by the shared graceful-stop
timeout and make the xAI cancel-error xfail require an AssertionError.
* fix(realtime): keep the blocked item on xAI, which cannot delete an audio input item
* test(realtime): assert the xAI voice block sends no conversation.item.delete upstream
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): bill Responses API image_generation_call tool usage
Price completed image_generation_call output items from tool_usage.image_gen token counts when present, otherwise per image from the quality and size on the item, with the image model taken from tools[].model (default gpt-image-1). Add the result into tool_usage_cost alongside web search and file search, and read tool_usage from the top level or usage.tool_usage. Squashed and rebased onto main
* style(cost): annotate read-only views and mark casts for the type discipline gate
* fix(cost): price Responses image tool from tools[].model only, no default model or provider
* refactor(cost): drop redundant status check, unread usage fields and unused output_type literal
* test(cost): integration case for Responses image_generation tool token billing
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the streamed usage.cost matches the spend row on every cost case
* test(integration): type the streamed usage frames as JsonValue
* test(integration): only require streamed usage.cost when the client received a usage frame
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price bridged /v1/messages reasoning tokens from the chat completion, not the served Anthropic body
* fix(cost): keep the type gate flat for the bridged /v1/messages billing path
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add tier 1 claude messages stream twins and recapture drifted usage shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): freeze the login throttle clock so the retry-after assertion is deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add Azure AI Claude Haiku 5.5 cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): declare service tier cache price fields on CostMapEntry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): drop CostMapEntry reorder now declared on main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add live-captured cache-read fixtures and align replay usage blocks with current provider responses
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): keep the input/output split on streamed xai spend rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): split xAI's reported total across input and output by token math
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): price an explicit base_model from its own row when the deployment provider row has no rates
A chatgpt/ deployment with model_info.base_model recorded zero spend because the
base model was looked up under the chatgpt provider, which resolves to the
deliberately unpriced chatgpt/ row. When the deployment provider's row for the
base model declares no rates, price it under the provider of the base model's
own row instead. Deployments whose provider row declares rates are unchanged.
(cherry picked from commit 42d9a8693e)
* test(integration): bill a chatgpt base_model deployment from the priced row on a live proxy
Co-authored-by: sla55er <stiliyanmarkov@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: sla55er <stiliyanmarkov@gmail.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add xai grok-4.7, Bedrock runtime chat completions and Bedrock Mantle cost cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): bring back known_bug and carry the xai grok-4.7 stream breakdown bug as a strict xfail case
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): add BridgeParityTestCase with a gpt-6-sol /v1/messages to Responses pair
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): rename BridgeParityTestCase to BridgeCostParityTestCase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): collect BRIDGE_COST_PARITY only from modules that define one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): bill the regional row for inference-profile ARNs that embed a cross-region prefix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep the ARN regional-row regression tests to the touched lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): apply the anthropic fast multiplier to cache cost breakdown
Anthropic fast mode billed totals at provider_specific_entry.fast while the
spend log cost_breakdown kept cache_read_cost and cache_creation_cost at 1x.
Move the speed lookup into a shared helper, multiply it into the billed token
rates used by get_token_type_cost_breakdown, and have the Anthropic calculator
read the same helper so totals and breakdown cannot drift apart.
Production change carried from #44074.
Co-authored-by: kerry <kerry@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): narrow the provider entry instead of defaulting to a dict literal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* feat(decisions): bill decisions and systemone from <provider>/decisions/<model> cost map rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): mark decisions pricing rows with mode decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): sync generated API types for decisions mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): update pricing expectations in proxy tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): price classifier calls from decision rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): drop redundant evaluation handler after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): inject pricing row for deployed classifier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): ship the decisions pricing lookup without cost map rows
Restore both cost maps to main. Tests inject their own
<provider>/decisions/<model> rows. The Add Model dropdown hides those
pricing keys and offers mode decisions instead of evaluation.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): expect the Decisions mode option in the Add Model form
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): bill decisions and systemone from _decisions cost map fields
Replace the provider/decisions/model pricing rows with <field>_decisions
fields on the model's own row. cost_per_token overlays them for the
decisions, adecisions, systemone and asystemone call types and prices
through generic_cost_per_token; chat, responses and batch keep the plain
fields. gpt-6-luna gains its Decisions rates (input 1e-07, output 0,
cached and cache-write input 1e-07, 2x above 272k tokens).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): satisfy type gate, schema sync and test_utils schema for _decisions fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): import cost_calculator module to keep basedpyright unknown-variable count flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): drop stray fireworks import from __init__ and annotate test cost rows for the type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover gpt-6-luna decisions and systemone above the 272k tier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): keep the decisions pricing block clear of main's stale azure comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost_calculator): drop stale azure models doc comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(model-prices): add missing OpenAI dated snapshots for deep research, search preview, audio mini and computer use
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): add OpenAI shutdown date to dated snapshots, absorb Together Tev1 cache price removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger e2e-tests after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): correct Perplexity pplx-decider price and add pplx-decider-v1.1-27b
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(fireworks_ai): add Decisions support for accounts/fireworks/routers/rune on /v1/systemone and /v1/decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(fireworks_ai): point the rune cost row at the public Fireworks docs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): patch litellm.callbacks with monkeypatch in the Fireworks cached-cost test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): move the openai cost tracking cases from cost_tracking_cases.json to CostTrackingTestCase literals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): move the remaining cost tracking cases, the cost map and the batch/realtime cases from cost_tracking_cases.json to Python literals (#45885)
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): streamed request bills the same SpendLogs row as the non-streamed request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): write the stream parity cells as StreamParityTestCase literals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): express stream parity cells as CostTrackingTestCase pairs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stream parity asserts exactly one SpendLogs row per leg after a settle window
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stream parity forces a spend-log flush with a /v1/responses row instead of a settle sleep
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stream parity covers messages -> chat and messages -> converse bridges
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stream parity bridges use xai/grok-4.7 and us.anthropic.claude-opus-5-5
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): use latest frontier models in stream parity cells (gpt-5.6, claude-opus-5-5)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): switch OpenAI parity cells to gpt-6.1-sol and the chat -> responses bridge to gpt-5.5-pro
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions
The System One request format moves to /v1/systemone and /systemone. /v1/decisions and
/decisions now accept the OpenAI Decisions API format, translate it into a System One
request, route it through the same pipeline, and translate the answers back.
* fix(decisions): import assert_never from typing_extensions for Python 3.10
* fix(decisions): cap OpenAI-format questions at the System One limit
/v1/decisions accepted up to 200 questions, but the System One request it translates to takes at most 128, so 129 to 200 questions failed with an internal validation error. Both now share MAX_DECISION_QUESTIONS.
* fix(decisions): return 400 for bodies that are not JSON
* test(decisions): send System One bodies to /v1/systemone in integration tests
* fix(decisions): return guardrail_information on OpenAI-format decisions when requested
* refactor(decisions): validate proxy request data before reading guardrail settings
* fix(realtime): skip guardrail VAD session.update injection for transcription sessions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(realtime): flag transcription sessions from the route intent and backend events only
A client session.update declaring session.type transcription on a voice
session no longer sets the transcription flag, so it cannot switch off the
guardrail's create_response gate or skip the transcript guardrail
* fix(realtime): flag transcription sessions from provider-transformed session events
* test(integration): cover transcription sessions skipping the VAD auto-response injection
Adds the realtime transcript guardrail audit cells: transcription sessions on all three
realtime routes, the OpenAI SDK, the beta protocol, the whisper default deployment, the Azure
GA path over a TLS scripted upstream, and Meta Muse push-to-talk sessions keep the client's
session.update verbatim and get their transcript, while voice sessions keep the injected
create_response gate and a client-declared transcription type no longer bypasses it. Sad,
edge, and chaos cells cover malformed session fields, duplicate and older backend session
events, unauthenticated upgrades, repeated sessions, an upstream outage under open sessions,
and a worker kill with a proxy restart.
The scripted upstream now answers session.update the way the vendor does (session.updated,
or the missing turn_detection.type and session-type errors), records every websocket frame,
serves the Azure and Muse realtime paths, and can run over TLS from an owned upstream.
---------
Co-authored-by: gabriele <gabriele@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): price batch image completion tokens at image batch rate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover default vertex batch output transformation for image cost
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): sync model prices schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): simplify batch completion cost with rate helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): keep global batch pricing fallback for image-only deployment rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(test): rename connect tunnel helper to https redirect
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): merge deployment batch rates over global pricing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): put flash-image batch image rate on the GA row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost): drop redundant comment in batch fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): use _batch_or_half for the batch image rate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): price nano banana 2.1 fixtures off the real cost map rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): use the vertex 4k image token count in the batch cost test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: align integration fixtures with current behavior
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): wait for the spend flush before asserting its trace placement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): key response-cache entries by model group from litellm_metadata
/v1/responses routes through the router with model_group in litellm_metadata, which the cache key ignored, so identical requests to different model groups sharing one underlying model hit each other's cached responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): register typesafe as a provider so Jev deployments load
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): move provider endpoints under llms and validate proxy bodies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): add Cloudflare Clef and Strands Decider backends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): register decisions routes for managed agents and gateway
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): use raw regex for cloudflare missing account match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): avoid cast in Cloudflare response unwrapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): default model, evaluation health probe, short Cloudflare names
The proxy validates only state and questions, so a request without a
model falls through to the configured default model like every other
route. Health checks probe evaluation-mode deployments through the
Decisions API instead of failing with an unsupported mode, and
cloudflare/clef and cloudflare/clef-flash get cost-map rows so the short
names resolve a mode and a price. The registry no longer claims typed
decisions for a provider with no backend.
* fix(decisions): let health_check_params override the evaluation probe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit the decisions endpoint across providers, limits, health and chaos
Adds the /v1/decisions audit cells: one wire contract per provider (path, key, body and cost-map billing), the gateway-only fields and tags, the sad paths (invalid bodies, unknown model, key checks, api_base in the body, upstream 401/429/500, a 200 without answers, an unreachable upstream), the two evaluation-mode health probes, and three chaos cells (a mixed-failure burst over both routes, a worker SIGKILL mid-burst, an upstream outage and restart on the same port).
The PR's cost case read the upstream observations through the gateway, which answers 404 for that path; it now reads them from the upstream URL. The owned proxy harness takes extra CLI arguments, and its graceful stop waits as long as a worker boot may take, since a worker still starting honors SIGTERM only once it is up and the 30 second wait forced a cleanup under load.
* fix(decisions): send env API keys to a configured api_base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): add zero-cost evaluation cost-map entry for Strands Decider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): register the routes through the lazy feature registry
The Decisions router was included at import, ahead of the config and DB
pass-through endpoints, so a pass-through configured at /v1/decisions
was skipped and answered 400 as an unknown Decisions provider. The
routes now register through LAZY_FEATURES, which splices them in after
every eager route, so a pass-through at /v1/decisions keeps its route
while /decisions still serves natively. The lazy OpenAPI snapshot carries
the two paths so the schema shows them before the first call.
The audit cells add the env-key egress to a configured api_base, the
client api_base opt-in shared with chat, the pass-through precedence on
an owned proxy, and the Strands evaluation health check resolved from
the cost map. The integration config exports the Perplexity env key the
first cell needs.
* fix(decisions): keep the Cloudflare api_base message in its transformation and read the audit upstream once per cell
* fix(proxy): let a config pass-through beat a lazily registered route in eager mode
With LITELLM_DISABLE_LAZY_ROUTES set the decisions routes are registered at
startup, so SafeRouteAdder treated a config pass-through at exactly
/v1/decisions as already registered and dropped it. In lazy mode a pass-through
created through the API after the first native call was skipped the same way.
Routes a lazy feature owns no longer count as registered, and a route added at
one of their paths is placed ahead of them, the precedence lazy mode gives a
config pass-through when the feature has not loaded yet.
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The gpt-5.6-responses_previous_response_id case sent a literal id the proxy never issued, which the Responses id security hook refuses with a 403 at production defaults. The case now primes a response through the proxy and chains the id it hands back, so the harness drops allow_unmanaged_response_ids and the security hook stays exercised
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
