Add ptu_count, cost_per_ptu_per_hour, ptu_effective_from and ptu_effective_to to
ModelInfo so a model deployment can carry the inputs for provisioned-throughput
flat-cost attribution. ModelInfo validates per-field bounds (positive count,
non-negative rate, effective_to after effective_from); model/new and
model/{id}/update enforce the cross-field invariant (count and rate set together,
team_id required) on the effective model_info so partial updates validate the
merged result, and v1/model/info returns the fields.
LiteLLM_DailyTeamSpend gains ptu_flat_cost and ptu_source_model_id columns plus a
sentinel api_key constant; the daily rollup that writes them lands in a follow-up
PR. Adding the optional model_info fields is backward compatible; models without
them are unaffected.
ptu_effective_from is required alongside the count and rate rather than optional. Flat
cost accrues from that instant, so an absent start has to be inferred, and inferring it
let a deployment configured today be billed for days it did not exist. Both PTU validators
also run over the merged view before any write on the update path, beside the premium check the create path
already runs there: the team ACL update below autocommits, so a validator raising further
down left the team mutated and the deployment row never written.
The update path validates the model_info a patch would store rather than the patch
alone. An invariant holds over the deployment as it will exist, not over whichever
subset of fields a caller sent, and validating the patch rejected raising the rate on
an already configured model because that patch carries no start of its own.
The summed job deadline alone did not protect the test budget. Setup that
overran its allowance still ate into pytest's window, which is the same
failure this change set out to remove, just with more headroom.
Every step before pytest now carries its own ceiling, and their sum is the
`setup-timeout-minutes` default. Setup can no longer overrun into the test
budget without failing its own step first, and a slow setup step now reports
as a red step naming itself rather than a cancelled shard whose tests passed.
Model the workflow YAML the guard reads with Pydantic instead of bare dicts,
so the shapes it depends on are validated once at the boundary. A workflow
that does not parse is now reported as a finding rather than a traceback.
`prisma generate` runs `npm install prisma@<version>` whenever the
prisma-client-py binary cache directory has no CLI entrypoint, pulling ~85 MB
of query and schema engines over the network. Every workflow pointed
PRISMA_BINARY_CACHE_DIR at `${{ runner.temp }}/prisma-cache`, which GitHub
wipes and recreates per job, so that cache was empty on every job of every
run and the download was never avoidable.
The download is normally a few seconds and occasionally minutes. On one
proxy-db run it took 5m18s on a single shard against 3.8s on its eleven
siblings, which pushed the job past its 15 minute timeout and cancelled a
shard whose tests were at 99% and all passing.
Leave PRISMA_BINARY_CACHE_DIR unset so the binaries land in the
prisma-client-py default, which is already keyed by prisma and engine
version, and restore both that path and the @prisma/engines staging cache
through a shared composite action.
Job timeouts also counted setup against the test budget. `timeout-minutes`
now bounds the pytest step, with a separate allowance for checkout,
dependency install, and client generation, so slow setup shows up as a slow
job instead of a cancelled test run.
check_prisma_binary_cache.py guards all three invariants: no workflow
reintroduces the override, every job that generates the client restores the
cache, and the version the action greps out of uv.lock still resolves.
Adds 'agentcore' as a search provider backed by an AgentCore Gateway MCP web-search target, usable from litellm.search()/`/search` and as a websearch_interception backend. Supports SigV4 (AWS_IAM gateways) and bearer tokens (CUSTOM_JWT gateways) via a new BaseSearchConfig.sign_request hook.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
xAI already converts Responses usage to chat Usage so web_search_calls survive
cost tracking. The chat completions bridge then re-ran the Responses usage
transform and crashed on missing input_tokens. Pass through already-chat Usage
and chat-shaped dumps instead
Gate web search like OpenAI (output/annotations/web_search_requests).
xAI uses server_side_tool_usage_details only for per-call cost math, with
web_search_requests mirrored in llms/xai for existing gate compatibility.
Treat positive web_search_calls as a web-search signal in built-in tool
cost gating, and mirror counts onto prompt_tokens_details.web_search_requests
when attaching xAI tool usage details so charges are not skipped.
Use usage.server_side_tool_usage_details.web_search_calls at $5/1k calls
instead of legacy num_sources_used/web_search_requests. Preserve tool usage
details through Responses usage transform for accurate response cost.
A poll of a Vertex passthrough batch wrote nothing to the managed-object row,
so status and file_object stayed frozen at the create-time snapshot and
GET /v1/batches served a stale status and an empty output file id for the life
of the batch. Only the create may claim a batch, but every observation of one
may refresh its state.
store_unified_object_id takes create_if_missing, which the poll clears: it
refreshes status and file_object through update_many, and leaves a row that is
absent absent rather than creating one owned by the observer, since created_by
and team_id are written by whoever reaches the create branch. The update payload
is now shared with the upsert so it cannot drift into writing api_key,
request_tags, created_by or team_id.
The passthrough identity re-assertion that was previously part of this PR ships
separately in #36121, so this PR keeps only the batch attribution work.
The creating key owns user_api_key_alias only when it actually has one. Guarding
the overwrite on the presence of a key rather than on a resolved alias nulled the
field out for every key generated without key_alias, and for any key rotated or
deleted before its batch finished, losing the creating user's alias that the spend
row previously carried. The guard now matches the team-alias line below it.
* fix(otel): mark v2 server spans as failed for pre-call errors (LIT-4780)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): authenticate malformed-body requests before rejecting them (LIT-4780)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): cover malformed-body rejection when auth error is recovered
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): skip authorization for a request whose body never parsed
Deferring the parse failure ran the full auth phase, including budget reservation, whose reserved amount is only released by the endpoint's post call path; the endpoint never runs, so malformed requests leaked reservations and locked a budgeted key out. Authorization now runs only when the body parsed, and a parse failure with a rejected key keeps returning the 400 it returned before.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Since #35491, every Router joins the module-global _live_routers weak set at
construction, and every model cost map swap replays the deployments of every
member on top of the freshly adopted map. #36039 isolated the register_model
ledger half of that replay but not this half: under pytest-xdist, a Router
created by an earlier test in the same worker that was still referenced (or
simply not yet garbage collected) re-registered its deployments during
TestPriceDataReloadIntegration::test_distributed_reload_check_function, and
register_model hydrated the sparse mocked gpt-3.5-turbo entry into a full
ModelInfo dict, failing the exact-equality assert (reruns cannot help since
the polluting router survives in the worker process)
The autouse isolate_litellm_state fixture now snapshots _live_routers before
each test and restores its membership on teardown, so a test's routers stop
contributing to cost map rebuilds once the test ends. A canary pair in
test_conftest_isolation.py asserts the rollback
