* test(cost): add xai grok-4.7, Bedrock runtime chat completions and Bedrock Mantle cost cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): bring back known_bug and carry the xai grok-4.7 stream breakdown bug as a strict xfail case
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): add per-agent a2a_protocol_version override for langgraph-api 0.15 cards
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): cover streamed 0.3 replies through the real client
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): accept new kwargs in create_a2a_client fake; add protocol-version integration tests
The create_a2a_client fake in tests/agent_tests/test_a2a_agent.py listed a
fixed signature, so asend_message's new protocol_version= kwarg raised
TypeError in test_a2a_non_streaming and test_a2a_streaming. Accept and
ignore extra kwargs with **_, matching the new _capture fake in test_main.py.
Add tests/integration/compatibility cells for litellm_params.a2a_protocol_version
against a real 2-worker proxy with Postgres and Redis and scripted A2A peers
modeled on langgraph-api 0.15.4: pinned send/stream on all three routes, aliases,
the 1.0 opt-out, unchanged genuine v1 agents, PATCH taking effect on both
workers, persistence via /v1/agents, the localhost retry, invalid values,
upstream errors, the dialect hint, and the completion bridge stripping the key.
Merge base 5f969982fc: 33 red, 7 green. Head ba709d3de5: 40/40 green twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(focus): add ternary destination for FOCUS export
Add a `ternary` FOCUS export destination (plus a `ternary` callback) so a LiteLLM proxy can push FOCUS-formatted spend to Ternary with `callbacks: ["ternary"]` -- no separate export job
- FocusTernaryDestination: a pure sink modeled on the sibling vantage/mavvrik/gcs destinations; POSTs the FOCUS CSV to the Ternary cost-ingestion endpoint, chunking an oversized backfill (csv-module repartition, quoted-newline safe; oversized single row raises rather than silently dropping) with a stable X-Ternary-Upload-Id + 0-based index/total so the receiver stages-then-swaps. connection_id is validated + URL-encoded; base_url must be HTTPS (loopback exempt for local); the upload has a bounded timeout; it aborts on the first failed chunk so nothing partial lands
- TernaryLogger: presets the FOCUS pipeline (CSV) and enriches the FOCUS Tags JSON with the per-request token counts the shared transformer drops (Ternary weights allocation by tokens); the enrichment is confined to this path and only adds keys. Cadence is daily (default) or interval (validation loops, an explicit interval-seconds required); the export window is day-aligned and drops days before the window start so a whole-day-replace receiver never truncates
- Registration mirrors vantage/mavvrik: ternary callback literal, logging dispatch, scheduler job, and CustomLoggerRegistry entry -- no shared-code changes
- Tests: mocked unit tests only (no live calls), 100% line coverage of both new modules
* fix(focus): warn instead of silently dropping token tags on mismatch
_merge_token_tags returned the frame unchanged when the transformed data
had no `Tags` column or its row count disagreed with the source, so every
token count was dropped with no signal -- the receiver then lands NULLs
that read as "this gateway reports no tokens". Emit a warning on each
degenerate path (mirroring _drop_days_before), and cover both with caplog
tests asserting the warning and the unchanged return.
* feat(focus): upload one UTC day per Ternary signed-URL PUT
Replace the multipart POST with chunking by a per-day signed-URL upload,
matching the Mavvrik destination's pattern. The logger partitions the raw
usage rows on `date` (dropping unparseable dates with a warning), then
transforms, enriches and delivers each day separately; one day's failure
is logged and the remaining days still upload, with a single error raised
at the end. The destination requests a signed URL for the day, validates it
is an HTTPS GCS URL, and PUTs the CSV with exactly the signed headers.
Removes the 30 MiB / 100k-row chunking, the X-Ternary-* upload headers and
the .partN filenames; a re-sent day overwrites its object, so retries stay
idempotent without them.
* fix(focus): re-send the last two whole days so late-flushed spend lands
LiteLLM writes DailyUserSpend in batches, so spend dated yesterday can be
flushed after the 00:05 UTC daily export. That export only covered the
previous UTC midnight onward, and the next day's export moved past it, so
the tail of each day was never sent. Start the window at UTC midnight two
days back: the following run re-sends yesterday whole, overwriting its
per-day object with the complete day.
* fix(focus): lazy-load the Ternary engine, fetch whole days, keep ternary out of the generic factory
Every proxy imported polars at boot through the logger registry, so a proxy without polars failed to start even with no FOCUS callback. The polars-backed engine now lives in its own module and loads on the first export
A non-UTC Postgres session timezone shifted the naive updated_at filter, so the oldest day in each run lost rows and Ternary's whole-day replace made the loss permanent. The Ternary read now starts one day early with no upper bound and keeps rows by their own UTC date
callbacks: ["focus"] with FOCUS_PROVIDER=ternary uploaded a parquet blob of one interval over a whole-day CSV file. The generic factory no longer builds a Ternary destination and the Ternary engine injects its own
* fix(focus): keep only the days a Ternary export window covers
Dropping the updated_at upper bound let a bounded export_usage_data window upload every day through today. Rows are now kept by their UTC date between the window's first day and the day before its end instant
* refactor(focus): leave the shared FOCUS export engine as on main
The Ternary engine sets the base engine's fields itself and passes its own
destination, so focus/export_engine.py and its test match main again.
---------
Co-authored-by: Aaron Sproul <aaron@ternary.app>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(guardrails): add Wingback guardrail integration
Mirrored from #38685 at d73b66544d (net diff against its merge base, applied on main)
* fix(guardrails): dispatch Wingback through the unified guardrail
The proxy only runs a unified guardrail when apply_guardrail is defined on
the guardrail's own class, so WingbackGuardrail now binds the parent's
decorated method in its class body. Chat, messages and responses pre, post
and during call checks reach Wingback again, and each check logs once.
Also types the initializer and config model for the lint gates and uses
monkeypatch.setenv in the unit test
* chore(guardrails): drop the Wingback dispatch comment
The unit and integration hook tests already fail when the apply_guardrail binding is removed, so the comment is redundant under the repo comment rule
---------
Co-authored-by: bsk-wb <bharath@wingback.ai>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(cost): add BridgeParityTestCase with a gpt-6-sol /v1/messages to Responses pair
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): rename BridgeParityTestCase to BridgeCostParityTestCase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): collect BRIDGE_COST_PARITY only from modules that define one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): bill the regional row for inference-profile ARNs that embed a cross-region prefix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep the ARN regional-row regression tests to the touched lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): accept service_tier on requests and carry the served tier on responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): mark the in-place service_tier store as rebind-ok
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(databricks): wrap the service_tier store to the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): keep the rebind-ok suppression on the service_tier line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): only set service_tier on responses when the upstream returned one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): name the B010 suppression on the service_tier setattr
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): shorten the service_tier suppression to the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): read service_tier from the Responses request only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(edenai): settle native Messages reported cost
* test(rust): add live EdenAI Messages contract cases
* refactor(rust): settle call cost once as an ExecutionFacts variant
The route now decides Cost (Reported, CacheHit or Deferred) from the
result source and the provider's reported figure. The legacy callback
adapter only projects a Reported amount onto provider_reported_cost and
no longer writes response_cost, which the Python calculator derives
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(rust): keep EdenAI wire shaping independent of Invoke
* style(rust): format cost settlement imports
* fix(rust): trust provider costs without bypassing configured pricing
* fix(rust): clear untrusted response costs before spend settlement
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(cost_calculator): apply the anthropic fast multiplier to cache cost breakdown
Anthropic fast mode billed totals at provider_specific_entry.fast while the
spend log cost_breakdown kept cache_read_cost and cache_creation_cost at 1x.
Move the speed lookup into a shared helper, multiply it into the billed token
rates used by get_token_type_cost_breakdown, and have the Anthropic calculator
read the same helper so totals and breakdown cannot drift apart.
Production change carried from #44074.
Co-authored-by: kerry <kerry@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): narrow the provider entry instead of defaulting to a dict literal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* test(responses): pin bridged commentary plus final answer as one chat choice
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): pin reasoning, two-message and two-tool-call bridge shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): number bridged chat tool calls 0, 1 instead of all 0
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(gemini): tool-use prompt tokens must be priced as input
Signed-off-by: Delweng <delweng@gmail.com>
* fix(gemini): price tool-use prompt tokens as input
toolUsePromptTokenCount was folded into usage.prompt_tokens but left out
of prompt_tokens_details.text_tokens, and generic_cost_per_token bills
input from text_tokens, so URL context / code execution / File Search
tool tokens were counted but never charged. Search-grounded responses
keep adding zero via the grounding-aware billable count, and
promptTokensDetails absent still leaves text_tokens None so billing
falls back to the full prompt_tokens
Signed-off-by: Delweng <delweng@gmail.com>
---------
Signed-off-by: Delweng <delweng@gmail.com>
* fix(responses): fold bridged Responses output into one chat choice
A Chat Completions request bridged to /responses got one choice per
Responses message item, so a reply shaped [message, message,
function_call] put the tool call on choices[1] and clients reading
only choices[0] never saw it. Fold every message text, annotation,
reasoning item and tool call into choices[0]
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop mutable annotations and the extra cast in the choice fold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): remove a cast-ok marker that suppressed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): join folded message texts with a newline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(responses): join folded message texts with no separator so stream and non-stream match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(otel): let a team or key excluded_services override the global setting for its OTel v2 destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format otel excluded_services changes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: use raw regex for excluded_services match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): reject non-string destination callback vars
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): harden destination callback var validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): compare parsed values in callback var conflict check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Add user field to AI Guard execute-policy payload for dashboard attribution
Send the top-level OpenAI request `user` field in the execute-policy
JSON payload so AI Guard can surface usernames on its dashboard. This is
independent of the existing user_api_key_* headers which remain unchanged.
The upstream guardrail framework already passes the original request_data
(containing the `user` field) through to response-path guardrail calls,
so this works for both IN and OUT directions including streaming.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Add type annotations and tests for user field in execute-policy payload
- Add Final annotation to user_for_payload
- Add str | None type to user parameter in make_zscaler_ai_guard_api_call
- Add 3 tests: user included when present, omitted when absent,
and user in/out of JSON body sent to AI Guard
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(zscaler_ai_guard): inject the http client in user field tests
Pass an AsyncHTTPHandler into ZscalerAIGuard so the user field tests read the
real execute-policy body off a MockTransport instead of patching class methods,
and drop the comments that restated the lookup
* fix(zscaler_ai_guard): forward only non-empty string users
Read the request user through a typed helper and type the execute-policy call
params so basedpyright stays under the merge-base count, and skip non-string
values the vendor rejects with a 500
* test(zscaler_ai_guard): drop the test-only http_client
Run the user field tests against an in-process aiohttp server through api_base
so the guardrail keeps its original constructor
---------
Co-authored-by: Syamala Keerthana Chegu <schegu@zscaler.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): keep the personal budget on a proxy-admin user object
A token carrying the proxy_admin role (JWT admin scope, SSO) replaced the fetched LiteLLM_UserTable row with a bare admin object, so max_budget never reached common_checks and admins spent past their personal budget on /chat/completions. Keep the row and force only user_role.
Co-authored-by: L4XB <lukas.buck@e-mail.de>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): budget denial for a proxy-admin JWT is the 422 ExceededBudget response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): carry only the personal budget onto a forced proxy-admin user object
Keeping the whole DB row also carried its models list, so a proxy admin
whose row had no-default-models or a restricted list got 403 on personal
requests. Build the admin object as before and copy max_budget and
user_email across. The built-in admin identity (the master key and the UI
username/password login) keeps ignoring its row's budget.
* test(proxy): pin the contrast case in the admin model-list and admin-name budget tests
Each test now also shows the same cached row refusing a non-exempt token, so the
pass case cannot go green on a row the gate never read.
* test(e2e): drive the personal-budget case with a proxy_admin user row and a plain JWT
The Buildkite stack keeps the default admin_allowed_routes, so an admin-scope JWT
gets 403 on /chat/completions before any budget check and the case could never
pass there. A plain token whose sub resolves to a proxy_admin row goes through the
same forced-admin rebuild. Drop the stage-mirror admin_allowed_routes edit, which
the deployed stack never read.
* ci: rerun e2e after an unrelated Gemini streaming spend-row timeout
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: L4XB <lukas.buck@e-mail.de>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(sagemaker): inference components and OpenAI-compatible embeddings
Send model_id on a SageMaker embedding deployment as the invoke call's
InferenceComponentName, sync and async, and keep it out of the request body.
Add the sagemaker/openai/<endpoint> route, which sends the OpenAI embeddings
request shape ({"input": [...]} plus extra_body fields) and reads
data[].embedding and usage from the response for containers that serve the
OpenAI embeddings API. Plain sagemaker/<endpoint> deployments keep the
Hugging Face shape
* refactor(sagemaker): type validate_environment messages and cover base64 embeddings
* fix(sagemaker): name model_id in inference component errors and reject a non-string model_id on embeddings
* fix(sagemaker): fall back to the error text when SageMaker sends a null message
A ClientError whose Error.Message is null (SageMaker's InternalFailure shape) reached the
inference component hint as None and crashed with a TypeError; the message now falls back
to the error's own text on the chat and embedding paths
* test(sagemaker): audit matrix for inference-component embeddings
* fix(sagemaker): read the error status and text through one shape-checked helper
botocore's ConnectionClosedError carries response=None, so the except blocks that
read e.response.get(...) crashed with an AttributeError and the proxy answered 400
"Invalid request format" for a dropped connection. client_error_details validates
the botocore error shape with pydantic and falls back to 500 plus the error's own
text for any other shape (a missing response, an httpx Response, a plain exception).
* fix(sagemaker): drop the inference component hint from chat and derive the SDK wire expectation from the request
The chat except blocks never see SageMaker's text: HTTPHandler.post raises from
raise_for_status before the handler's own SagemakerError line, so the hint check ran
on httpx's generic "Client error '400 Bad Request' for url" line and could not match on
any chat path. The hint now names litellm.embedding only and the chat blocks keep the
status and text they had, without the dead check. The two OpenAI SDK wire tests pinned
the SDK's default encoding_format; they now expect whatever the SDK sent.
* refactor(sagemaker): meet main's per-rule lint and type ceilings
Main started gating every lint rule on its merge-base count and added LIT015
(frozen pydantic models) after this branch was cut, so the merge with main
pushed LIT001, LIT011, LIT015, BLE001 and reportArgumentType past their
ceilings. The response models are frozen with tuple fields, the OpenAI
embedding response is built with model_copy instead of writing into the
caller's object, the invoke_endpoint arguments are a TypedDict, and
optional_params and extra_body are validated into Mapping[str, object] before
they are read
* refactor(sagemaker): pick the embedding config in one place and stop pinning upstream error text
The openai/, Voyage and Cohere checks lived in both get_model_config and
get_supported_openai_params, so they now share one lookup. Tests compare error
messages against the error's own text, or against botocore's and json's own
wording, instead of copying that wording, and assert whole messages where the
text is ours
* fix(caching): key direct calls on model_id so inference components don't share cached embeddings
* test(sagemaker): name the fake-boto3 SDK test class for what it covers
* test(sagemaker): kill and stop chaos workers through the owned-proxy harness
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
cost_discount_config keys may now be <provider>/<model-pattern> with fnmatch globs, so a discount can target a subset of a provider's models. An exact pattern beats a glob, the glob with the longest literal wins, and the bare provider key is the fallback. PATCH /config/cost_discount_config validates the provider half only, and the admin UI cost tracking page gains a Model pattern field and a Models column
Resolves LIT-8465
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A caller hitting /v1/messages with a cache_control mark and an OpenAI
gpt-5.6 or newer deployment behind it lost the mark when the request was
rebuilt as a Responses request, so OpenAI never cached at the boundary the
caller chose. The bridge now rewrites each marked system, user and
tool_result block to prompt_cache_breakpoint on the same block when the
deployment reads breakpoints, keeps the system prompt as a developer
message so its mark survives, and still strips the mark for deployments
that do not read it
The hook's gate becomes public so the bridge and the hook share one
answer for which deployments read breakpoints, and the one-block swap
lives next to the existing breakpoint helper in prompt_templates
* feat(providers): add CoralBricks as a native provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): harden cost map consistency tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): type cost helpers, read cost maps from disk, cover streaming spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): drop Final on loop-rebound names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): add the provider audit cells, the Add Model browser cell and the README row
* test(coralbricks): derive UI spend from model info, type browser rows, always release held upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): release held upstream before the wire server joins its threads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): number forwarded chats with itertools.count instead of a mutable list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(coralbricks): drop retired glm-5.3-flash-fp4 and test against deepseek-v4.1-flash-fast-fp4
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): pin retries, await the respawned worker, read the UI rate from the catalog
The retry cells now set num_retries on the deployment, since the CI rig posts
router_settings.num_retries 0 after readiness and the router default of 2 no
longer applies there. The SIGKILL chaos cell waits for the respawned worker's
startup before the owned proxy is stopped and derives its timeout from the
harness budget. The Add Model spec waits for the catalog the form loads, reads
the expected rate from it, and finds the deployment through /v1/model/info,
which does not paginate
* fix(coralbricks): price the current model names and the new DeepSeek input rate
CoralBricks renamed glm-5.3-fp4 to glm-5.3-fast and deepseek-v4.1-flash-fast-fp4
to deepseek-v4.1-flash-fast (the fp4 names stay as deprecated aliases), and cut
DeepSeek V4.1 Flash input from $0.30 to $0.01 per 1M on 2026-10-08. Rates read
from https://www.coralbricks.ai/api/public/models on 2026-10-10; a live cache
write billed 6014 tokens at 9e-08 each, so the cache-write rate stays input
plus cacheWritePerM.
The test config builders now build new dicts instead of editing the parsed
YAML, and the cost-map test locals carry Final.
* fix(coralbricks): keep pricing for the deprecated fp4 slugs CoralBricks still serves
* fix(coralbricks): list the catalog rows for wildcard deployments and derive the test's input limit from the row
* fix(coralbricks): price the deprecated glm-5.3 and deepseek-v4.1-flash slugs
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ci): wait for SIGSTOP to take effect before handing back a suspended proxy process
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ci): resume a process whose SIGSTOP never lands before failing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(model_hub): surface model_info.description in model group info and Model Hub UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): aggregate model group description without in-place mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): skip background health check DB writes when the latest-row read fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock_mantle): map the OpenAI-shaped context overflow envelope too
Mantle reports context overflow in two envelopes. #37862 covered the
structured `validation_error` one carrying token counts
("prompt tokens (N) exceed model maximum (M)"). Mantle also returns the
OpenAI-shaped body:
{"error":{"code":"context_length_exceeded","message":"Your input
exceeds the context window of this model. ...","param":"input",
"type":"invalid_request_error"}}
That passes the existing `invalid_request_error` gate but matches neither
the token-count pattern nor `is_error_str_context_window_exceeded`'s
substring list, so it fell through to a generic BadRequestError. Clients
such as Claude Code only reactive-compact on a recognised overflow
phrase, so an overflowing request surfaced as an ordinary error with no
recovery.
Return the same normalised "prompt is too long: ..." message for this
envelope when it carries no counts, keeping the numeric wording when the
provider supplies them.
* fix(bedrock_mantle): map the streamed context overflow and keep its 400 on /v1/messages streams
* fix(anthropic_adapter): keep only the context overflow's 400 on streamed /v1/messages so other provider 4xx still fall back
* test(anthropic_adapter): wrap a parametrize line over the 120 char limit
* test(integration): audit Bedrock Mantle context overflow mapping across routes, clients, and fallbacks
* test(integration): move the streamed /v1/messages overflow spend row into a BUG-skipped test
* test(integration): keep every chaos cell in the burst and skip only the missing spend row
* fix(bedrock_mantle): answer 400 on an in-stream context overflow
The Mantle overflow mapper built its ContextWindowExceededError on the
original exception's response, so an overflow that Mantle reports inside
a 200 stream (the chat-mode token-count envelope, which the OpenAI chunk
parser wraps as a 500) kept that 500 and the streamed /v1/messages route
answered HTTP 500 api_error instead of 400 invalid_request_error. The
mapper now builds the error on a 400 response that keeps the original
headers and request, and the mapping test covers an in-stream 500 and 503.
The integration suites pin the streamed /v1/messages overflow to the
pre-content 400 route under a retry budget and to the message_start plus
invalid_request_error event without one, drop the chaos carve-out that
expected no spend row, and give the fallbacks file a timeout mark sized to
the proxy's graceful stop.
* fix(bedrock_mantle): keep the provider's 400 body and match the overflow code only as error.code
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(decisions): bill decisions and systemone from <provider>/decisions/<model> cost map rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): mark decisions pricing rows with mode decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): sync generated API types for decisions mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): update pricing expectations in proxy tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): price classifier calls from decision rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): drop redundant evaluation handler after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): inject pricing row for deployed classifier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): ship the decisions pricing lookup without cost map rows
Restore both cost maps to main. Tests inject their own
<provider>/decisions/<model> rows. The Add Model dropdown hides those
pricing keys and offers mode decisions instead of evaluation.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): expect the Decisions mode option in the Add Model form
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): bill decisions and systemone from _decisions cost map fields
Replace the provider/decisions/model pricing rows with <field>_decisions
fields on the model's own row. cost_per_token overlays them for the
decisions, adecisions, systemone and asystemone call types and prices
through generic_cost_per_token; chat, responses and batch keep the plain
fields. gpt-6-luna gains its Decisions rates (input 1e-07, output 0,
cached and cache-write input 1e-07, 2x above 272k tokens).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): satisfy type gate, schema sync and test_utils schema for _decisions fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): import cost_calculator module to keep basedpyright unknown-variable count flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): drop stray fireworks import from __init__ and annotate test cost rows for the type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover gpt-6-luna decisions and systemone above the 272k tier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): keep the decisions pricing block clear of main's stale azure comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost_calculator): drop stale azure models doc comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): skip failure rows for expired Admin UI session keys on dashboard info routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): mark DB-boundary write assertion as test-quality-ok
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): rename the load suite to perf, one E2E_PERF_TESTS opt-in, and a mixed-provider Locust perf test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop the mixed-provider perf test, keep the rename and the E2E_PERF_TESTS consolidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): support Clef-omni audio and video
* fix(decisions): accept Clef-omni media types with parameters and cover the proxy routes
The clip patterns now mirror Cloudflare's input schema, so a data URL or
content_type carrying parameters like audio/webm;codecs=opus is accepted.
The four proxy routes get media tests of their own, the backup cost map
is synced with the new Clef-omni rows, and the docs page moves to the
docs repo as a follow-up.
* fix(decisions): refuse parameterized media-type data URLs and refresh the OpenAPI snapshot
Cloudflare answers 422 to a data URL whose media type carries parameters even though its schema only requires the data:audio/ prefix, so LiteLLM now refuses those with a 400 before the call. The object form keeps passing parameters through, which Cloudflare accepts. The lazy OpenAPI snapshot and the dashboard API types are regenerated for the new audio and videos fields
* fix(decisions): gate media by cost-map flags, treat null clips as none, drop unsupported clips before validation
* fix(decisions): keep the SDK's positional order and gate media flags on evaluation-mode models
* fix(decisions): gate media on both decisions model modes and move the audit cells onto the harness helpers
* test(decisions): cover the Fireworks System One router in the media refusal cells
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): do not inject temperature for Gemini 3+ models when omitted
map_openai_params added temperature=1.0 to every Gemini 3+ request that did not set one, on both the gemini and vertex_ai routes. Google is deprecating temperature, top_p and top_k for Gemini 3+ and says upcoming models will reject them, so this default turns a request that never set temperature into one that will fail
An omitted temperature now falls back to Google's own default, which their docs give as 1.0 for all Gemini 3 models. Explicit temperature, top_p and top_k are still forwarded
Fixes#38663
* test(gemini): annotate sampling regression tests
* test(gemini): drop the injected temperature from Gemini translation cases
The nine Gemini base cases for chat completions, responses and messages send no temperature, so the provider request no longer carries the removed 1.0 default. Each new body was sent to the real Gemini API for gemini-3.5-flash, gemini-3.8-flash and gemini-3.1-pro-preview and returned 200, and the stored provider responses are unchanged
* test(gemini): drop the default-temperature wording from the thinking level test docstring
* test(gemini): cover omitted and explicit sampling on both Gemini routes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(model_prices): add meta/muse-wonder-1.0
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): mark muse-wonder-1.0 as not supporting reasoning_effort none
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(model-prices): add missing OpenAI dated snapshots for deep research, search preview, audio mini and computer use
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): add OpenAI shutdown date to dated snapshots, absorb Together Tev1 cache price removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger e2e-tests after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): correct Perplexity pplx-decider price and add pplx-decider-v1.1-27b
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop Greptile from the PR template, AGENTS.md and CI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: require bot reviews to finish before ticking the review checkbox
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(fireworks_ai): add Decisions support for accounts/fireworks/routers/rune on /v1/systemone and /v1/decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(fireworks_ai): point the rune cost row at the public Fireworks docs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): patch litellm.callbacks with monkeypatch in the Fireworks cached-cost test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
