* feat(batches): emit per-line JSONL batch records to callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): only emit line items for final batches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): bump soupsieve to 2.9.2 for GHSA-gjv8-xp57-g29c and GHSA-j934-xhv5-fg8f
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(batches): satisfy ruff format for line item logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for store_batch_line_items_in_callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): cover line-item edge shapes and general settings propagation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep deployment credentials out of callback-visible params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(batches): declare credential transport attribute on Logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): integration contract for per-line batch callback events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve store_batch_line_items_in_callbacks through the settings store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): assert per-line events beside the aggregate and create events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop resurrected use_legacy_interactions_schema shim from merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): assert line item events carry no credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reset batch line-item flag when the setting is absent
Removing store_batch_line_items_in_callbacks from the DB general settings left the module flag at its previous value because the effect early-returned when the key was neither in the DB nor owned by YAML. The proxy owns the global, so an absent resolved value now writes False.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): map anthropic batch lines through the anthropic message transformer
Anthropic result lines carry a message-shaped body, so ModelResponse(**body) produced empty choices and a usage key that blocked the batch-rate usage attach. Extract the existing AnthropicResponse -> ModelResponse conversion used for /v1/messages logging into anthropic_message_to_model_response and reuse it for anthropic line items.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: tag the anthropic result cast with a cast-ok reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: shorten cast-ok reason so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip line-item callbacks for providers without a line reconstruction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): reconstruct bedrock and mistral batch lines through their provider transformations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: wrap widened provider literals so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): pair bedrock lines by recordId and type them from the reconstructed response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep line item events out of proxy limiters and strip credentials from child params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep cast-ok tags on the flagged cast lines for the discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip failed line item events in parallel request limiter failure hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): tolerate lifecycle spend rows when asserting no line item rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): place mutable-ok tags on the lines the discipline gate flags
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip router budget accounting for batch line item events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep anthropic line errors and skip native vertex batch rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): restore test-quality tags on line item tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move batch line-item unit test to tests/unit after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): emit batch line-item callbacks once per batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): release line-item claim when fan-out emitted nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): make the line-item claim release ownership-safe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep bodyless failed batch lines reaching callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): seed batch line hidden params in get_hidden_params defaults
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(batches): reuse fetched batch result files for line-item logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): release line-item claim when nothing was emitted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): type batch line-item callback sites to satisfy basedpyright budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: wrap batch hidden-params cast for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): reason suppressions for batch line-item lint deltas
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop stale mutable-ok suppressions after LIT002 removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters (#44486)
* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters
Batch line-item callbacks (call_type=acompletion + litellm_params.batch_parent_id)
bypassed the call-type-only batch guards:
- ClickHouseSpendLogger wrote one spend row per line on top of the aggregate
aretrieve_batch row, billing every batch twice with the flag on.
- Router deployment callbacks and the routing-strategy usage handlers charged the
historical batch's tokens/rpm to the live TPM/RPM counters, so after
Router.aretrieve_batch the next chat raised RouterRateLimitError.
Skip these events wherever the aggregate batch retrieve is already skipped.
* fix(logging): keep internal litellm_params authoritative in model_call_details
A request body key named litellm_params lands in optional_params, and
update_environment_variables merged **optional_params after the internal
entry, so a caller could overwrite the logging kwargs' litellm_params and
forge markers like batch_parent_id. Live-reproduced: a Gemini chat whose
body carried {"litellm_params": {"batch_parent_id": ...}} returned 200
with no ClickHouse spend row. Re-assert the internal litellm_params after
the merge so every sink (ClickHouse, Postgres, router quota counters)
reads the same trusted value, and assert the aggregate aretrieve_batch
row still writes in the ClickHouse line-item test (review finding).
* fix(clickhouse): skip failed batch line items in the spend sink too
A failed JSONL line is part of the aggregate aretrieve_batch row's
request counts; writing it its own spend_logs row inflated request
counts with the line-item flag on. Extend the batch line-item guard to
the failure handler (review finding) and pin both directions: a failed
line item writes no row, a genuine live failure still does.
* fix(batches): declare batch_line_items service family on claim calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(integrations): keep batch line items out of the built-in metering sinks
With store_batch_line_items_in_callbacks on, per-line batch events carry a
real response_cost and reached six metering sinks that lacked the
is_batch_line_item_event guard, so a completed batch metered aggregate +
per-line (roughly 2x true spend): prometheus (litellm_spend_metric and
request counters), openmeter, lago, datadog cost management (FOCUS
BilledCost), newrelic metrics, and the OTel v1/v2 gen_ai.usage.cost
metrics. Skip line-item events in each sink's success/failure metering
entry points, matching the guards already on the Postgres and ClickHouse
spend sinks. OTel spans for line items are still emitted; only their
cost/token metrics are skipped. Tests drive real sink instances on the
real dispatch lists through a completed 2-line batch and assert each
sink meters exactly the aggregate.
* style: format metering-sinks test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): skip synthetic line-item durations in v2 metrics; isolate prometheus test
Follow-ups from review on the sink guards: record_failure() now skips
batch line items too (a failed line's interval measures retrieval and
emission, not its model call), and the prometheus guard test snapshots
and restores the process-wide registry collectors instead of leaving
them unregistered. The OTel v2 test is now a flag on/off A/B over the
whole metric snapshot: cost and usage values must match exactly and no
metric may gain datapoints with the flag on.
* chore: finish main merge conflict resolution for batch line items
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: sort imports in litellm_logging after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): use the public hook class names main now exports
* fix(batches): only the batch writer can mark a callback event as a line item
amoderation, transcription and speech build litellm_params from every request kwarg,
so a body field named batch_parent_id reached Logging.litellm_params and every
spend, budget and rate-limit hook skipped the request. update_environment_variables
now drops an incoming batch_parent_id; the line-item writer sets it through
Logging.mark_batch_line_item. Batch line hidden params are emitted only when set.
* chore(batches): keep the lint gates within main's ceilings after the merge
Typed the helpers main now types (Mapping litellm_params, Any Span), replaced the
banned cast with a literal match, dropped isinstance checks main made redundant,
and moved test-quality reasons onto the lines the gate reads.
* fix(anthropic): read the message id off the validated message
The anthropic_messages logging path can hand over an already parsed message object,
which has no .get; main read its id through getattr, so take it from the validated
AnthropicResponse instead.
* fix(batches): an unreadable error file no longer drops the output line events
Costing counts an error file it could not fetch as no failed lines; line logging
refetched it and the raise aborted the whole fan-out, so the successful lines were
never emitted. The error file now reads as empty with a warning.
* test(batches): pin billing for batch_parent_id on every endpoint and value shape
Chat completions, embeddings, responses and messages join moderations, speech and
transcriptions, and speech is driven with an empty, int, list, object and 5 KB value.
* test(batches): bill every request of a concurrent burst mixing forged and plain bodies
* revert: keep the line-item fan-out all-or-nothing on an unreadable error file
85fc4498cc emitted the output lines and kept the batch claim when the error file
could not be read, so its failed lines were never retried. Raising releases the
claim and the next retrieve retries the whole fan-out.
* test(batches): pin the all-or-nothing retry and warm the burst model first
An unreadable error file emits no lines and the next retrieve emits all of them.
The burst cell drives one speech model after a warm-up request so a cold worker
cannot fail it, and waits for the batched key spend.
* fix(batches): run the line fan-out outside the aggregate event's deadline
The logging worker cancels a callback coroutine after its per-callback
deadline, and the line fan-out ran inside the aggregate aretrieve_batch event
before its callbacks, so a large batch or a slow sink could cancel the
aggregate billing event. The fan-out now runs in its own task, built from a
copy of the parent's litellm_params taken before the aggregate callbacks
write their request-local markers.
Anthropic keeps no input file and reports input_file_id "None", which the
fan-out fetched and failed on, so no Anthropic line ever arrived. That id is
now treated as no input file.
* fix(batches): let running line fan-outs finish before proxy shutdown closes their clients
A started fan-out keeps its batch claim, so a shutdown that cut it short
skipped the unsent lines until the claim expired. proxy_shutdown_event now
waits up to MAX_TIME_TO_CLEAR_QUEUE for them first. Also replace a test's
type: ignore with an isinstance check.
* fix(batches): keep batch line fields off callback kwargs and skip fan-outs at shutdown
Line bodies were spread into the callback kwargs, so a line's metadata could settle router rate limit reservations; they now stay in optional_params, and the router rate limit and prompt caching hooks skip line events.
A retrieve logged after the shutdown drain, or by the logging worker's exit flush, no longer starts a fan-out its loop would destroy while it holds the batch claim.
The output file is fetched before the error file again.
* fix(batches): log batch lines under the retrieve's callback settings and keep them out of alerting and shadow eval
Line events inherit the retrieve request's key and team callback settings, so turn_off_message_logging and per-team logging credentials apply to them, and they keep the system prompt in the logged messages.
Shadow eval, Datadog metrics, Slack daily reports and PagerDuty skip line events, since the batch API served them and the aggregate event already counts the batch.
* test(batches): line events follow the key's message logging opt-out
* test(batches): line events follow team logging settings, keep the line system prompt and survive a restart once
* fix(batches): keep in-memory line claims for 10,000 batches and pin managed batch emission across poller and retrieves
* Keep batch line unit tests in-process and drop repeated per-sink comments
* Share one batch provider alias across batch_utils and line item logging
* Report Bedrock line status codes, skip unknown Bedrock lines without raising, fetch the input file after the native Vertex check
* Read a swallowed Redis claim failure as unavailable, price unpriceable lines at zero, keep the OTel failure docstring
* Call handle_completed_batch when line items are off, keep a cache test from leaking Redis into the line claim cache, pin an unpriceable line's zero cost on the callback kwargs
* Hand the cost poller's credential snapshot to line item file reads, so a managed Bedrock batch's lines read its input from the deployment bucket
* Log batch lines through the sync callbacks a direct async call runs, keep failed lines out of router cooldown counts, and emit an error line whose body is text
* fix(batches): log failed Bedrock embedding lines as embeddings with their input
A Bedrock batch input record carries no url, so a failed Titan v2 line fell back to acompletion and its
inputText never reached the event messages. The failed line now takes aembedding from the deployment
model and the record's inputText is the line input
* fix(batches): keep a Bedrock line's system prompt in its logged messages
Bedrock batch input records hold the system prompt as content blocks, which logging only reads off the kwargs as text, so Claude and Nova lines dropped it
* feat(batches): store per-line batch requests and responses in S3/GCS via batch_line_item_callbacks
Replace the per-line logging path with a dedicated hook. When the hourly batch cost
poller finishes a managed batch, it pairs every output and error line with its input
line, prices each line, and hands one payload to every logger listed in
litellm_settings.batch_line_item_callbacks. s3_v2 and gcs_bucket write it as one
NDJSON object per batch under batch_line_items/<day>/<batch id>.jsonl.
Bodies are dropped and the header kept when the creator key or team, the global
setting, or the logger itself turns off message logging, and when the creator can
no longer be found. A failed upload is logged and never blocks spend tracking.
* fix(batches): store batch line items after the batch is billed and retired
Send a billed batch's lines only after its row is marked complete, so a slow or
failing sink never holds a claimed batch open. Skip reading the input file when
the bodies are dropped anyway, still send the error lines of a batch whose output
file is gone, and end every NDJSON record with a newline.
* test(batches): cover team-level and deleted-key redaction of batch line items
* fix(batches): store batch line items in the background so a hung bucket never holds up billing
The poller now retires a batch first and hands its line items to a per-worker background store that runs one batch at a time. A slow, hung or failing bucket no longer delays billing or retiring the other batches in the poll cycle. While a poll cycle's worth of batches is still waiting to store, the next batch's lines are dropped with an error.
* fix(batches): store an unbilled batch's line items only once it is retired
When retiring an expired, cancelled or all-failed batch fails, the batch stays on the poll page, so its lines are no longer read and stored again on every poll cycle until it is retired. The poller also releases a billed batch's output bytes before reading the next batch's output, as it did before line items existed.
* fix(batches): price and write batch line items off the event loop
Pricing every line of a large batch and serializing its object ran on the
proxy's event loop, so a 50k-line batch stalled the worker for seconds.
Both now run in a worker thread. An object the bucket refuses for good is
no longer kept in s3_v2's drop marks, which only a log-queue flush clears.
Vertex AI input rows carry the custom_id in their key or request labels, so
their lines now get the request too.
* fix(batches): pair no request for a shared custom_id and wait for an unclaimed batch's retry
A Vertex AI embeddings entry whose input is a list is split into one row per
item, all with the same custom_id, so its line now gets no request instead of
the last row. On a database without batch_processed nothing claims a billed
batch, so when marking it complete fails it stays in the poll page and its
lines now wait for the next poll instead of being written on every cycle.
* fix(batches): store batch line items only once the poll has billed every batch
Pricing a large batch's lines on a worker thread still competes with the poll for the GIL, so the next batches
in the same poll were billed and retired seconds later. The background stores now wait until the poll cycle that
queued them has finished.
* fix(batches): hash line-item objects off the event loop and keep lines with a lone surrogate
A batch's whole line-items object was hashed for its S3 PUT on the event loop, and a line
holding a lone surrogate from a provider's JSON escapes made the object impossible to
encode as UTF-8, so the batch's object was never written. The object is now prepared in a
worker thread, and only the line that cannot be encoded is written with escaped characters.
* fix(batches): keep request and response for batches created without a virtual key
A batch created with a JWT has no API key on its row, and its line items were stored without
request and response as if its creator had been deleted. Such a batch now follows its team's
and the proxy's message logging settings, the same as a master key batch. A deleted key or team
or a failed lookup still drops the bodies.
* fix(batches): pair line items with the right request on every deployment and skip empty objects
A batch retired with nothing billable now reads its input file by the id the provider reported, so a batch that
ran on the second deployment of a model group keeps each line's request. A Vertex AI line read as custom_id
"unknown" pairs with no input row, and a batch whose files hold no readable line writes no object
* fix(batches): let OAuth2 batches follow team and proxy message logging settings
The line-item redaction check looked up every saved creator credential other than the master key or a CLI session as a virtual key. An OAuth2 batch saves the raw token, so it read like a deleted key and lost its request and response
It now looks up only a saved value shaped like a virtual key's hash, reusing is_valid_sha256_hash, so OAuth2 batches follow their team's logging settings and the proxy-wide setting while deleted keys still fail closed
* fix(batches): hash the creator credential and write batch line items in parts
The creator credential reaches line metadata only as a sha256 hash, so an
OAuth2 token or the master key alias is never written to the bucket. A queued
store keeps only whether the batch was billed and downloads the output file
again at its turn, logging an error naming the batch if that fails. Lines are
built and uploaded one part at a time, each closing once its raw request and
result bytes reach BATCH_LINE_ITEMS_MAX_PART_BYTES (16 MiB by default), with
the input file indexed by byte span instead of parsed up front.
* docs(batches): say async_log_batch_line_items is called once per part
* fix(batches): log an error when a batch's deployment is gone before its line items are stored
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
A chat completions request to gpt-4.1 (or any OpenAI model older than 5.6) with drop_params on still sent the marker inside messages and tools. It is now removed there, while drop_params off keeps sending it through and models that read it keep it either way
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Responses route now resolves openrouter alongside openai. The
OpenRouter config reuses the OpenAI request and response transforms and
owns only its bearer credential (OPENROUTER_API_KEY, then OR_API_KEY) and
base URL (OPENROUTER_API_BASE). Wire model ids follow Python's
get_llm_provider, including the openrouter/auto native id with an
explicit provider. The Python route host no longer hands litellm.openai_key
to OpenRouter.
The OpenRouter live test lives in tests/live/openrouter.rs and returns
early when the provider key is not exported, so cargo test stays offline
by default instead of hiding it behind --ignored.
Co-authored-by: nate-berri <nate@berri.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(responses): bill a background response once, from the poll job
Reading a completed background response with GET /v1/responses/{id} wrote a
full-cost spend row on every read, on top of the row the background cost poll
writes. Reads of a Responses object (GET, DELETE, cancel) now never bill, and
the poll job is the single biller.
The poll job now bills with the creating request's attribution (key hash, team,
org, user, aliases, tags) read off the managed object row, through a helper it
shares with the batch cost poll. The background create persists the creating
key and request tags so that attribution survives until the poll runs.
* test(proxy): a background response read reports zero cost
* fix(responses): bill a background response once, on the first read or poll that sees it finished
A conditional update moves the stored row out of queued or in_progress so only one worker or pod bills it, through one re-read stamped with the creator's attribution. A failed billing read puts the row back for the next poll
* fix(responses): keep the caller's response id out of the billing failure warning
* fix(responses): keep the creating client's User-Agent tags on the one background charge
* fix(responses): keep managed background reads free and bill each response once
The managed files hook marks reads and cancels of proxy-issued response ids, so only those stay free. SDK and database-less reads of a background response still bill, extra_body background creates are tracked, and the poll's status read no longer cools down a deployment on a 404.
* fix(responses): bill finished background responses from the fetched body and store rows Postgres accepts
* fix(responses): give the background billing event its own spend log row and bill responses the create left untracked
* fix(responses): narrow the managed files hook with a protocol instead of an enterprise import
* fix(responses): leave a read the caller retargets with client-side credentials to the cost poll
* fix(responses): bill background responses on the creator's end user and report the charge on the settling read
The read that settles the bill returns the charge in x-litellm-response-cost, the billed event
carries the creator's end user and the key, team, and user budget fields, and its spend row is
named after the id the client got. Override reads, cancels, and deletes settle from the
configured deployment, a streamed create marks its row billed, stored rows keep no model
output, and model insights counts the billed row's spend without a request.
* test(responses): cover background response billing across proxies, reads and chaos
* fix(responses): settle background billing only for callers allowed to address the response
A delete now settles the bill only after the response id access check passes, a streamed
create is marked billed when it sets background inside extra_body or its completed event
arrives after the last chunk, the billing event keeps the deployment's litellm_params
base_model, a key with no user and no team bills no user instead of its owner scope, and the
enterprise cost check import is optional
* fix(responses): record the bound on the pending status IN filters
* fix(responses): send the billed cost split on the settling read and keep the batch cost event as it was
The read that settles a background response now sends the whole cost header
family from the billed event, so the breakdown headers add up to the charge.
Batch cost events go back to main's attribution (creator id as the user, no
budget fields); only background responses get the end user and budgets.
Integration tests expect the charge on the settling read (or none when the
poll billed first), one charge for a streamed create, and SDK flows that poll
with the id each read returns.
* fix(responses): bill each background response once across upgrades, streams, and failed settles
Rows stored before this billing model are found by the issued id and rekeyed by the
provider id, and the cost poll retires one a read already billed. A streamed background
create is free on its own and bills the finished response through the same managed row
before the client sees response.completed, so a racing read cannot bill it again. A
settle that fails retries in the background, a retargeted read with no row tracks and
bills from the configured deployment, and a delete keeps its billed cost header.
Reverts the unrelated json.dumps narrowing in the completion transformation.
* fix(responses): bill cache-polled background streams and match poll rows by the issued id
* test(responses): cover deletes, failed settles, budgets, base models, and cache-polled background billing
Adds integration cells for the background response billing paths the audit
inventory lists: a DELETE that settles the bill and refuses a stranger, a
settle the database refuses that retries once it recovers, the Prometheus
remaining budget, base_model pricing from model_info and litellm_params, the
daily model usage rollup, the end user, the settling read's cost split, stored
rows without model output, a forged managed marker on the create, streamed
creates with background in extra_body, and the cache-polled create
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): allow model instances in LiteLLM_EndUserTable validation
* fix(proxy): preserve property-backed attributes when coercing spend
* refactor(proxy): default null end-user spend with a field validator
* fix(proxy): drop model-wide from_attributes from the end-user table
* test(proxy): unskip the /customer/block integration contracts
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(scim): write a group's roster in one locked transaction instead of one member at a time
POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the
membership rows and the users' team tags in one transaction under the team's advisory lock, so a
500-member push runs a fixed number of statements instead of thousands. PUT now answers with the
roster it just wrote
* test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers
* fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write
A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock
Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize
The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock
* fix(scim): evict the cached team on a group rename and read a no-op roster from the writer
A rename-only PUT or PATCH wrote the new alias and left the team cached under
its id and old alias on every worker, so keys kept logging spend against the
old team alias until the TTL expired. The group write now evicts the cached
team (and broadcasts the eviction) right after the alias or metadata update.
sync_team_roster reads the team from the writer when the plan names nobody,
so a replica cannot hand back the pre-rename row; a plan already matching the
roster returns inside the lock without writing, evicting, or emitting the
members metric; the response is built from the written row, so lastModified
reflects the roster write; and detaching a removed member bumps the user's
updated_at like every other user write.
* fix(scim): bump detached members' updated_at in UTC
The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts
into Prisma's naive timestamp column in the session timezone, so on a database
whose session is not UTC the removed member's row landed hours away from the
UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and
have the bulk roster cell assert the bumped row sits between two reads of the
database's own UTC clock taken around the replace.
* fix(scim): chunk the group write's member lookups under the bind cap
Every Prisma "in" filter the group write issues bound one parameter per
member with nothing bounding the list, which check_unbounded_in_lists
reports as 11 new findings. The single-field lookups now go through
litellm.repositories.chunked_in (caller_transaction inside the roster
transaction, per_chunk_ok for the role recompute), and the one
cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records
the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two
slices, and a member named only in the last slice still resolves.
* fix(proxy): send bulk cache evictions in slices so the broadcaster drops none
Veria pointed out that the invalidation publisher caps its pending tasks at
1,024 and drops the excess with a warning, and that a group push now detaches
every removed member's keys in one call instead of one member at a time, so a
removal of more than roughly 340 members (three cache keys each) silently left
stale key objects on the other replicas. The merge base never hit the cap
because each member was a separate request.
Bulk evictions (team, user, organization, and session key deletes, plus the
roster settle) now publish in slices of 256 and wait, bounded at two seconds,
for the backlog already handed to the event loop before each slice. A Redis
that never answers costs a bulk caller at most that wait per slice instead of
wedging the request, and single-key callers are untouched. Regression tests
push 1,500 keys and a 500-member removal through a slow publisher and assert
every eviction reaches Redis; both fail against the previous tip.
* fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts
Greptile pointed out that pacing the broadcasts in slices also paced the
local eviction: a bulk delete waited for the publish backlog before dropping
the next slice from this worker's own memory, so on a Redis that accepts
connections but never answers a key deleted from the database kept
authenticating on the handling worker for up to two seconds per slice. The
merge base cleared every key locally at once.
Bulk evictions now drop every affected entry from this worker's memory first,
through the cache's own partition lookup since key objects live apart from
the other auth objects, and only then publish in paced slices. The roster
settle does the same for the deleted tokens, the JWT mapping keys and every
changed member's user and membership entries before any step that can wait.
Regression tests run both paths against a publisher that never answers and
check that nothing is still cached on this worker before the first bound
elapses; both fail with the local eviction removed.
* fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts
* docs(proxy): drop the module docstring that repeated the PR description
* fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps
Bulk key evictions now drop entries from memory, delete them from Redis in one
chunked DEL, and broadcast every key in one pipelined publish, so a large roster
removal no longer overflows the publisher's pending cap or costs a Redis round
trip per member. A push waits for in-flight membership loads before evicting.
User rows are locked in byte order to match the spend writer, a bulk user delete
takes every team lock before its first user write, and a POST whose roster write
fails deletes the empty team so the IdP retry creates it again.
* test(team): observe delete broadcasts at the Redis layer
* fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently
* fix(team): chunk the email lookup behind the member row lock
* fix(scim): create a member deleted mid-push again instead of failing the group write
* fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch
A member deleted mid-push is created again only when scim_upsert_user allows it,
under the verb's own created_via. New group members are checked against the
license seat limit as one batch before any is created. Auth cache publish drops
log the key count instead of the key.
* fix(scim): lock member rows while recomputing admin roles after a group push
* fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member
* fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member
* fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first
A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time.
The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team.
A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup.
* fix(user-delete): keep the team alias label on the team members gauge after removals
* test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them
* fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404
* fix(scim): answer 404 and roll back when the team is deleted during a roster write
* test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast
* feat(proxy): report when each per-model budget window resets
model_max_budget_usage on /key/info, /user/info and /team/info now carries budget_reset_at per model, the open window's start plus its period, or null when no window is open. The window start is read in the same batched lookup as the spend counter
* fix(proxy): count a monthly budget window from its own start when reporting its reset
duration_in_seconds measures Nmo from now, so a window opened on January 15 and read on February 10 reported February 12, then null while the counter still refused requests until February 15. duration_in_seconds_from measures the month from the window start, and duration_in_seconds now shares that month arithmetic
* fix(proxy): measure months from timezone-aware datetimes and assert whole usage values
datetime.fromtimestamp without a tz tripped DTZ006. The month arithmetic now runs on an aware local datetime, which keeps the same wall-clock result. The duration tests move to parametrized functions, since subTest payloads carrying datetimes cannot cross the xdist worker boundary, and the new usage tests compare the whole entry
* fix(budgets): end a monthly budget window a calendar month after it opens
Each charge recounted the month from the charge date, so a monthly window
opened in a month of another length expired days before or after the reset
/key/info reports. The window is now measured from its start, by the same
helper the usage report uses. A period ending after year 9999 reports no
reset instead of failing the read.
* fix(budgets): count monthly budget windows in UTC for every router budget
Months were counted on the host's local calendar, so replicas with different TZ
settings disagreed on when a monthly window ends. Provider, deployment, tag and
per-model budgets now all measure a window from its start in UTC, and a test
covers the router budgets. A reset past year 9999 is documented as null.
* test(budgets): cover monthly tag budgets in the router window test
* test(budgets): add integration cells for budget_reset_at and monthly budget windows
* test(budgets): find a monthly window in another span on every calendar day
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): list models from team access groups
* 修复 PR #32599 的类型检查并补齐模型列表回退测试
* fix(proxy): keep a key's own model list narrow when adding team access group models
A key with its own model list was shown every model from its team's access
groups on /v1/models, /v1/model/info and /model_group/info, though auth
denies those calls. Team access group models now widen a key's list only when
the key inherits the team's models. /v1/model/info reuses the shared listing
helper without a catch-all, and route-level tests replace the mock-only ones
* fix(proxy): skip the list build in the unrestricted model info check and test every listing route
The unrestricted check for /v1/model/info only needs to know whether the key
and team lists are empty, so it no longer builds and expands the model list a
second time. The route test now covers /v1/models and /model_group/info next
to both model info routes
* fix(proxy): scope model info for personal and agent keys with no model list
* fix(proxy): keep config keys' team access groups and own-team team_id scope in model listing
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(cost): add tier 1 claude messages stream twins and recapture drifted usage shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): freeze the login throttle clock so the retry-after assertion is deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(docker): manage a quickstart install with status, stop, start, and uninstall
Rerunning either quickstart now opens a menu for the existing install
instead of installing again, and both scripts take an action: status,
start (up), stop (down), restart, logs, open, update, uninstall
Each install gets a litellm-gateway command (litellm-gateway.cmd on
Windows) in its folder, linked into ~/.local/bin when that is on PATH,
so stopping and starting no longer means finding the containers by hand
- stop keeps the database; uninstall asks before deleting it, keeps the
images unless asked, and checks for leftovers
- a busy port names the process holding it
- -y / -Yes answers every question for scripts and CI
* chore(docker): run the quickstart on the latest image instead of main-stable
The docs mark main-stable as deprecated, and on the registry both tags
point at the same image digest today. The quickstart is for local
evaluation, and the file already says to pin a release tag for anything
beyond that
* fix(docker): trust a container's compose project label only when it is a valid project name
When another container holds the port, the quickstart offers to stop it
with docker compose -p <project> stop, taking the project from the
container's label. Whoever started the container sets that label, and
quickstart.sh split it into words, so a crafted label could add
arguments to the stop command
- a label Compose would not accept as a project name now counts as no
project, and the container is stopped by name instead
- quickstart.sh runs the stop with quoted arguments rather than
splitting a command string
* feat(docker): show how many requests the quickstart gateway has handled
The status box, shown by litellm-gateway status and by the rerun menu,
gains a Requests row, so a quick test shows the gateway is working
- the count comes from /global/activity with the admin key from .env,
failed requests included; curl reads the key from stdin, off the
command line
- the row is left out when the gateway cannot say, such as an older
image
- help is now listed in the usage text; help, -h, and --help already
printed it
* fix(docker): move to a free port instead of stopping an unrelated container
When an install's saved port is taken, the quickstart now only offers to stop
the holder if it is another quickstart install. For anything else it offers the
next free port and saves it in .env, so it never stops someone else's app. A
port set through LITELLM_PORT is never moved.
* fix(jwt): honor end_user_id_jwt_field on the JWT-to-virtual-key mapped path
A JWT mapped to a virtual key dropped the end-user claim, so spend rows had
no end user and customer budgets, limits and region never applied. The mapped
token now carries the claim, and it takes precedence over the body user and
the x-litellm-end-user-id header, which remain the fallback when the claim is
missing or empty.
* test(jwt): budget-check the body or header end user when a mapped JWT has no end-user claim
* fix(jwt): replace the annotation offsets with a targeted pyright suppression to keep LIT001 flat
* test(cost): add Azure AI Claude Haiku 5.5 cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): declare service tier cache price fields on CostMapEntry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): drop CostMapEntry reorder now declared on main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): open faked stream items empty so tool arguments and text arrive once
* fix(responses): open faked stream items in progress with a fresh empty list, and open mid-stream guardrail blocks empty
* fix(responses): open faked refusals, reasoning summaries, custom tool calls and MCP calls empty so each value arrives once
* test(integration): cover the empty-open Responses stream items on every fake, replay and block path
Integration cells for the faked /v1/responses stream, the chat and messages bridges, cache replay, guardrail block events and the MCP gateway call. Each opened item starts empty and each value arrives once, checked with raw httpx, the OpenAI SDK and the Anthropic SDK, plus upstream errors, 5 KB values, a client drop, a dropped upstream connection and a worker SIGKILL
* feat(rust-bridge): scope catalog rules with a contextvar
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): select native integration routes with scoped rules
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): run the Rust extension suite without LITELLM_RUST
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): use scoped catalog rules in dispatch tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* refactor(rust-bridge): keep catalog rules a static global and override them once per test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(providers): add Inference APIs as OpenAI-compatible provider
Registers https://api.inferenceapis.com/v1 as a JSON-configured provider (chat completions and Responses API).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(providers): list Inference APIs in Add Model and the provider docs index, add request-level tests
- provider_create_fields.json entry so the Add Model dropdown test passes
- provider_endpoints_support.json (+ backup) entry so the provider docs check passes
- tests now capture the outgoing HTTP request (URL, auth header, body) instead of only inspecting registrations
- drop a redundant comment
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(providers): inject a recording HTTP session instead of patching httpx.Client
Request-level tests now set litellm.client_session to an httpx.Client with a MockTransport, so nothing is
monkeypatched on a class.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4
* chore(providers): drop redundant comment on the inferenceapis entry
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4
* feat(inferenceapis): add cost map rows for the 12 chat models and test the responses route
Prices and context windows come from https://inferenceapis.com/pricing and GET /v1/models.
Moves the provider test to tests/unit and adds the interactions flag the endpoint matrix now carries.
* feat(inferenceapis): price the vendor's chat model aliases and mark /v1/messages supported
* fix(inferenceapis): serve /v1/messages natively and keep audio routes off the provider
* fix(json providers): send provider-specific params in extra_body and report context overflow as ContextWindowExceededError
* fix(exception mapping): match the Inference APIs overflow phrase instead of any context_length_exceeded substring
* fix(exception mapping): map context overflow to ContextWindowExceededError for Inference APIs only
Providers without their own mapper keep reporting an overflow as BadRequestError, since the router raises a ContextWindowExceededError instead of trying generic fallbacks when context_window_fallbacks has no entry for the group. Adds the integration cells for the provider, its fallbacks, and the extra_body and overflow families.
* test(inferenceapis): hold upstream replies across the worker kill in the chaos cell
A reply the killed worker sent before SIGKILL can lose its queued spend row, which the
product never promised to keep. Holding the upstream until after the kill means only the
surviving worker answers, so every 200 must log exactly once.
* test(inferenceapis): make the chaos cell kill the quieter worker and always release held replies
The survivor now must hold at least one in-flight request, so the burst check cannot pass as 0 == 0, and the held vendor is released on exit so a failed assertion cannot stall the wire server teardown
---------
Co-authored-by: TaylorHawkes <857858+TaylorHawkes@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): keep provider headers on MidStreamFallbackError
A 429 raised while opening a stream is wrapped in MidStreamFallbackError,
which dropped the litellm_response_headers exception_type() attaches to
the wrapped error. retry-after and x-request-id reached callers on the
non-streaming path but not on the streaming one, so retry backoff,
provider request id logging and proxy error headers lost them.
* refactor(streaming): drop explanatory comment from MidStreamFallbackError header copy
* test(streaming): cover sync and async header carry on MidStreamFallbackError
Parametrizes the provider-header test over the sync and async stream
wrappers, and drops the hasattr test, which pinned how the attribute is
stored rather than what any caller observes
* test(streaming): add integration cells for stream-open provider headers
Cover the SDK stream openers (gemini, vertex_ai, codestral, predibase and the
eager providers), the Router stream retry and fallback paths, and the proxy on
both the router and the config-less serving paths, including a mixed burst, a
worker SIGKILL and a SIGTERM restart. Add a helper that boots a config-less
proxy owned by the test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): route Model Garden models through the generateContent adapter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vertex_ai): annotate generateContent route test parameters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): type litellm_params in get_vertex_ai_model_route
* fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions
Streamed Model Garden calls through the generateContent adapter now request
stream usage where the deployment supports it, keep cached-token details, and
look up the provider-relative cost key, so they bill like the non-stream call.
The adapter forwards router metadata, passes upstream errors through with their
status, countTokens falls back to local counting for non-Gemini routes, and the
native Gemini config is limited to the Gemini routes.
* fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing
Streamed generateContent through the chat adapter now ends with the upstream usage, honors
always_include_stream_usage false and the caller's stream_options, counts one failure per
failed call, and routes Vertex deployments by their base_model like chat does
* fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage
* test(google_genai): expect the trailing usage chunk on bridged OpenAI streams
The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone.
* fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk
* fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent
* test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire
* test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: align token counting with inference context
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: preserve Vertex Gemini count message shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: forward Google countTokens tools in shapes every provider counter accepts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: add package marker for Vertex count-token unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): ignore deployment system_instruction when counting tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): forward system and tools on Google AI Studio, keep strict on Anthropic tools
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(count_tokens): ruff format gemini count handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(count_tokens): return Mapping from count request body builders
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(count_tokens): truncate long Anthropic tool names on Vertex count, update stale strict test expectations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(count_tokens): simplify Gemini count context and tests
Put the Vertex Gemini count branch back in the same shape as main and the partner model branch, keeping only the system and tools context, the provider error mapping and the gemini tokenizer default
Share one error response helper between the Vertex and AI Studio callers, drop the unread kind fields and the extra validation of map_function output, and say Gemini instead of Vertex AI in the error that AI Studio users also see
Tighten the tests by removing a stray parametrize on a helper, sharing the SDK count args, asserting the exact forwarded system and tools, and folding the Anthropic tool shape tests into one table
* fix(count_tokens): reuse inference tool translation for Vertex Gemini count
Vertex and AI Studio count now convert tools with the same functions inference uses, translate_anthropic_tools_to_openai and the Responses tool converter, before VertexGeminiConfig.map_function. Unnamed and Anthropic hosted tools now count the same declarations inference sends instead of being dropped
Tools that cannot be converted, such as a non list tools value, a non object tool or a schema that is not a JSON object, now return a 400 before any request instead of being counted without them
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: keep model capability flags out of provider requests
A supports_* flag or another model map field set on a deployment's litellm_params was forwarded
to the provider as a request field, so Bedrock answered 400 Extra inputs are not permitted on
chat traffic and on the background health check alike. The shared owned-kwarg predicate now
treats every ProviderSpecificModelInfo field as litellm-owned.
Builds on the health-check-only approach in #40952 by leilei3167.
* fix: treat every supports_ model map flag as a capability field and warn when one sits in litellm_params
* fix(router): skip declared litellm params in the capability flag warning and warn for runtime-added deployments
* fix(router): strip model capability flags from the router's deployments instead of every request
* fix(router): keep capability flags out of health probes and read them from the loaded catalog
* perf(router): read the catalog's supports_ field set once per cost map load
Cache the loaded catalog's field names when the cost map snapshot is taken so the capability-flag check stops walking every catalog entry per deployment, and add integration cells that drive the flagged and control deployments through chat, streaming, messages, responses, embeddings, health and model management against a vendor that rejects the flags
* fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove an OpenAI-format box answer reaches the caller as a box
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): give the box wire test a deployment that advertises supports_box_questions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results
* fix(vertex_ai): use object instead of Any in schema-ref traversal
* fix(vertex_ai): use immutable tuple traversal to satisfy type-discipline gate
* test(vertex_ai): add direct contract test for _contains_json_schema_ref
* fix(vertex_ai): type the $ref walk over JsonValue and drop the private-helper test
* fix(vertex_ai): wrap a tool result only when a $ref holds a non-empty string
Gemini rejects {"$ref": "<name>"} in function_response.response but accepts null, empty,
numeric, boolean, object, and list values, so those tool results stay structured
* perf(vertex_ai): skip the $ref walk when the tool result text cannot spell the key
* test(vertex_ai): cover a $ref on the tool result's root object
* docs(vertex_ai): cite where the Gemini $ref rule was observed
* fix(vertex_ai): keep a $ref under a list of lists structured
Gemini accepts a $ref reached through a list nested directly in a list,
so those tool results keep their structured form. Adds the audit's
integration cells for the tool result wire, chaos and edge shapes
* test(vertex_ai): teach the fake Gemini servers the list of lists rule
The audit's scripted upstreams now skip lists inside a list like Gemini
does, add list of lists cells on every endpoint, and give the 8 MiB
timing calls their own client timeout
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(types): replace Any with proven types in 84 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep vector store search locals untyped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep runtime payload validation out of the sweep
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep only annotation-only Any fixes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): baseline breaker metrics from live state after gc
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): collect garbage before the services logger gauge test reads live breakers
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(minimax): add MiniMax-H3 and H3-Max video generation via /v1/videos
MinimaxVideoConfig maps litellm's OpenAI Videos API onto MiniMax's V2 task
API. Create, status, list, content and delete hit /v2/video_generation and
/v2/query/video_generation. Remix is not exposed: MiniMax's
/v2/video_regeneration only upscales a finished 768P task to 2K and ignores any
prompt, so it raises instead of returning a video that silently drops the edit
OpenAI params translate to MiniMax's contract: seconds becomes duration, size
reduces via gcd to a valid ratio, and text-to-video requests default to ratio
16:9, resolution 768P and duration 5 when unset. input_reference becomes a
first_frame content item, base64 encoded into a data URI for file inputs, and
multimodal reference media passes through a content array in extra_body. Caller
content is validated strictly as image, video or audio items, and the scanned
prompt is always the one text item sent, so text can't reach MiniMax past the
guardrails; anything else in content returns a 400. api_base
shares MINIMAX_API_BASE with the chat config (a trailing /v1 or /v2 is stripped
because the video API carries its own version). The list endpoint pages by
number with no cursor or sort, so after and an ascending order are rejected
with a 400 rather than ignored, and page_num passes through extra_query. Responses
validate into frozen pydantic models at the boundary, video ids are
provider-encoded so status and content route back to MiniMax, and create
reports duration_seconds and video_resolution in usage for cost tracking
End-to-end tests drive the real avideo_generation() entrypoint through an httpx
MockTransport and assert on the encoded request body, which is what catches a
mapper that drops input_reference before the request is built
Both model_prices maps gain minimax/MiniMax-H3 and minimax/MiniMax-H3-Max,
priced per output second by resolution tier from MiniMax's pay-as-you-go list
(H3 $0.08 at 768P and $0.13 at 2K, H3-Max $0.05 at 480P and $0.08 at 768P).
MiniMax also bills input images beyond a per-model free allowance (5 for H3, 2
for H3-Max), so each entry carries input_cost_per_image and
provider_specific_entry.minimax_free_input_images, and the create response
reports the full charge as provider_reported_cost_usd. Reference video input is
rejected with a 400, because MiniMax bills its duration only after the create
call that litellm bills
Co-authored-by: AaronHowell <237895480@qq.com>
* fix(minimax): fail closed on invalid video params and malformed task responses
Addresses review on #43705:
- A create, status or delete response without a task id (or a task) is a
502 instead of a queued object with an empty id
- An unmapped task status is a 502 instead of defaulting to queued
- seconds/duration must be a positive whole number and size a positive
WIDTHxHEIGHT or W:H, otherwise 400; a 0x0 size no longer divides by zero
and an unparseable duration no longer falls back to the billed default
- characters and image are rejected with UnsupportedParamsError unless
drop_params is set, instead of being dropped silently
- Downloading a task that is not ready or failed is a 400 that keeps its
message through the handler, instead of a ValueError surfaced as a
retried 500
- The video download goes through safe_get / async_safe_get
Co-authored-by: AaronHowell <237895480@qq.com>
* test(minimax): cover the video download path and invalid-param branches
The download now goes through safe_get, so the happy path and the
finished-task-without-a-url branch are asserted, alongside the non-object
parameters rejection and the remix response stub.
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
* fix(minimax): return 400 for video remix and clear new lint gate counts
* fix(minimax): send file-tuple images and keep input_reference next to content media
* fix(minimax): answer 404 when an H3 video is downloaded before it finishes
A download that lands while the task is still queued or running answered 400. That is a
client error, which was the intent, but 404 is what the OpenAI videos API documents for
content that is not ready, and it is the status the sibling DashScope and Volcengine
video paths now answer, so a client polling all three reads one shape
A failed task keeps its 400, since that is not a timing mistake the caller can retry
away. The download test drives avideo_content and asserts the caller sees NotFoundError
rather than an internal server error
* fix(minimax): answer 400 for unsupported video params, early downloads and unknown ids
A create param MiniMax cannot honor (seed, n, quality and the like) was dropped
without a word. It now answers 400 unless drop_params is on, like other providers
A download before the video is ready and a status call for an unknown task id
now answer 400. The router cools a deployment down on a 404 and retries a 5xx,
and neither is a deployment fault. DashScope on main answers 400 for the same case
Input images past the free allowance are now billed through the shared video cost
path, so a deployment with custom pricing bills them too. Status usage no longer
carries MiniMax's token counts, list sets has_more to false since MiniMax pages by
number, and the price map stops listing video as an input modality
* fix(minimax): keep the polled video id, honor deployment drop settings, and match chat's key order
Status re-encodes the id with the polled id's model when MiniMax omits it, drop_params and
additional_drop_params never reach MiniMax and listed keys are dropped, MINIMAX_API_KEY wins over
litellm.api_key, create sets created_at, and a content variant other than the video answers 400
* fix(minimax): list a task without a model under the routed group
MiniMax leaves `model` off some listed tasks. The caller polls the ids a list hands back, and the
proxy resolves a poll by the model in the id, so an id encoded with an empty model no longer routes
back to the deployment that served the task. Fall back to the group the router was asked for.
* fix(minimax): honor deployment drop keys inside parameters and bill images at the map rate when a deployment prices only seconds
A field the deployment lists in additional_drop_params no longer reaches MiniMax when it arrives inside parameters, and a deployment that sets only its per-second price now bills reference images past the free allowance at the cost map's per-image rate instead of $0
* fix(minimax): keep deployment-dropped fields out of the body when they arrive through extra_body
* fix(minimax): answer 400 for a video resolution with no price unless drop_params is on
* fix(minimax): send the default resolution when a deployment drops resolution
---------
Co-authored-by: AaronHowell <237895480@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Happy <yesreply@happy.engineering>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(images): preserve explicit provider for async generation
* test(images): prove a bare model name reaches its explicit provider on the async path
* fix(images): validate async provider hints instead of reading untyped kwargs
Read custom_llm_provider and api_base in aimage_generation through a TypeAdapter over a small TypedDict, so the provider lookup gets typed values and the basedpyright gate stays within its merge-base ceiling
* fix: answer bad provider hints with a 400 and keep the explicit provider for async speech and transcription
* fix: keep the explicit provider for realtime and cover the audio bad-hint 400
* fix: keep the explicit provider on the realtime HTTP routes and read base_url in async audio
* fix: stop async image generation from rejecting a base_url it never reads
* fix(audio): skip base_url validation when api_base is set on async speech and transcription
* test(integration): cover a declared provider with a bare model on image, audio, and realtime routes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): return 400 with file line number for malformed batch JSONL row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(vertex_ai): keep yield outside the JSON decode guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): surface BaseLLMException raised while streaming a request body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(http_handler): only follow __cause__ when unwrapping streamed body errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): avoid recursive streamed error unwrapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): return a 400 naming the line for wrong-shape and non-UTF-8 batch rows
Keep main's error path for a directly raised provider error and only unwrap
one found behind a transport error. Add the parser column to invalid JSON
errors, and cover the real aiohttp chain in the _handle_error test
* fix(vertex_ai): report every row conversion error as a 400 naming its line
A message with no role, a non-string role, a bad temperature, or a litellm
BadRequestError raised while converting a row still returned a 500, and the
async upload lost the 400 for litellm's own exceptions. The stream now turns
any error from one row's conversion into a 400 naming that line.
_handle_error only unwraps an httpx.TransportError, which is what the aiohttp
transport raises around a failing request body, so other callers keep their
own status. A file that starts with a UTF-8 byte order mark now uploads, and
provider messages start with 'Batch input file line N' like the proxy check.
* fix(vertex_ai): reject null or missing batch bodies and unparseable rows with a 400 naming the line
A chat row whose body is missing or not an object now raises a 400 naming
its line instead of uploading a blank prompt, and a non-object body on
line 1 no longer crashes the GCS object name lookup. Rows too deeply nested
or too long to parse return a 400 from both the SDK and the proxy's own
batch file check. JSON error columns now count leading whitespace.
* fix(vertex_ai): keep batch rows on their file line through router and guardrail rewrites
The Router model rewrite and batch guardrails now keep blank lines and blank out dropped records, so a provider error naming a line still points into the caller's file. The rewrite strips a leading BOM per line, and a non-string body.model on line 1 is reported as a 400 naming the line
* test(integration): cover Vertex batch row line errors across routes, rewrites, SDK and worker chaos
* fix(router): keep blank lines only for Vertex batch uploads and leave guardrail rewrites compact
* test(integration): score the whole OpenAI upload after a guardrail drop and cover a mixed OpenAI and Vertex batch upload
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(llms): sanitize trailing slashes in provider get_complete_url
* test(llms): move trailing-slash url tests into each provider's mapped unit file
* fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI
* fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders
* fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests
* fix(gigachat): strip a trailing slash from the pass-through api_base
* test(integration): cover trailing-slash api_base across provider URL builders
Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex): report an unreadable credentials file as a read failure
Tell a path apart from inline JSON by shape, not os.path.exists().
Fixes#40166
* test(vertex): narrow pytest.raises to the expected message
* fix(vertex): log the credentials path instead of returning it to the caller
* refactor(vertex): log the credential source decision where it is made
* fix(vertex): handle paths open() rejects with a bare ValueError
* fix(vertex): redact the credentials path before it reaches the log
* test(vertex): cover the brace fallback and non-text credentials file
* test(vertex): exclude the unreachable exhaustiveness guard from coverage
* refactor(vertex): read the failure's fields off the matched object
CodeQL's py/uninitialized-local-variable does not follow keyword capture
patterns in a match statement and reported `path` and `reason` as possibly
unbound. Narrowing on the class and reading the attributes says the same thing
without a suppression.
* fix(vertex): accept a read-only mapping in extract_aws_params
* fix(vertex): keep encoded credentials out of the log and cover the legacy route
* fix(vertex): give one message for every credentials file failure and pick file or JSON by shape in Rust
* fix(vertex): read a `{`-named key file in Rust, name base64 there, and give one message when Google auth rejects a credentials file
* fix(vertex): read an unreadable `{`-named key file as a file, cap key file reads, and drop storage advice from the file message
* test(vertex): integration cells for credentials file and inline value failures
* test(vertex): drive the Rust messages cell through Vertex Claude
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic/adapter): stop translate_anthropic_tools_to_openai mutating the caller's input_schema
`translate_anthropic_tools_to_openai` assigned the caller's `input_schema`
straight into `function_chunk["parameters"]`, then merged the tool's remaining
top-level keys into that same dict. Because the assignment aliased rather than
copied, keys such as `display_width_px` on a computer tool were written back
into the caller's schema.
That matters because callers reuse the tool list across passes: the pass-through
guardrail translates the tools, and the request is then translated again for the
real call. The first pass pollutes the source and the second forwards a schema
carrying non-schema keys.
Shallow-copying fixes it. `translate_anthropic_output_format_to_openai` in the
same file already copies its schema for this reason, so this brings the two into
line.
Rebased onto current litellm_internal_staging; the original branch had gone to
conflict on surrounding drift while the line it fixes is unchanged.
* fix(anthropic/adapter): drop the inline comment and test the strict tool shape that Anthropic rejects
* fix(anthropic/adapter): deep copy input_schema so downstream schema edits never reach the caller's tool
A shallow copy still shared nested properties with the caller, and the Gemini, vLLM and watsonx bridges strip nested additionalProperties in place, so a fallback to Claude sent a strict tool Anthropic rejects. A deep copy also keeps a null or malformed input_schema as sent, as before this PR
* fix(anthropic/adapter): deep copy the incoming tools so downstream edits never reach the caller's tools
Copying only input_schema left the extra keys merged into parameters, such as
input_examples, and OpenAI-format function tools shared with the caller, so
Gemini's strict strip still edited the caller's tool before a fallback to Claude
* fix(anthropic/adapter): keep translating tools that deepcopy cannot copy
A tool passed as a frozen mapping, or with a schema nested past Python's recursion limit, made copy.deepcopy raise, so the request failed before it reached the provider. Copy each tool on its own and keep the original object when the copy fails, which is how these tools were handled before
* fix(anthropic/adapter): copy tools through JSON so the Responses bridge and frozen tools keep the caller's schema
* test(anthropic/adapter): parse stub bodies with json and cover the tool copy fallback
* test(anthropic/adapter): close the connection after the malformed tools request
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): scope /spend/tags aggregates to the caller's own and permitted-team logs
Non-admin callers previously received the proxy-wide per-tag spend
aggregation. The endpoint now resolves the same read scope as
/spend/logs/ui and /spend/logs/session/ui, and the query honors the
start_date/end_date parameters it previously accepted but ignored.
* fix(proxy): use Annotated auth dependency on /spend/tags and sync dashboard API types
* fix(proxy): bind /spend/tags dates as UTC datetimes, not session-timezone strings
Raw YYYY-MM-DD strings cast to timestamptz are read in the database
session timezone, shifting the day window by that offset. Parse them
to UTC-aware datetimes first, like every sibling spend query in this
file, and reject malformed dates with a 400 before they reach the
database.
* fix(auth): return a generic 401 for an unknown key
The wrong-key 401 shipped the internal verification table name and the
submitted key's full SHA-256 hash to any unauthenticated caller. Both now
go to the proxy's own log and the response carries only "Invalid API key
provided."
The Admin UI detected a dead session by string-matching that message, so it
now reads the structured error type instead.
* Update ui/litellm-dashboard/src/lib/http/client.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(auth): log the missing key hash where every resolve path passes
Review catch: JWT key-mapping misses resolve through IdentityStore without
touching the main key path, so logging the hash there left those callers with
nothing but the generic 401. Log it at the store's raise site instead, which
every resolve path shares, and drop the now-duplicate log.
* test(auth): inject a key-less client instead of patching the DB fetch
The store test tripped TQ008 by patching _fetch_key_object_from_db_with_reconnect.
Hand IdentityStore a client whose get_data returns no row instead, so the miss is
driven through the real lookup rather than around it.
* fix(auth): log the unknown key's hash once on the auth failure line
The per-lookup warnings in get_key_object and IdentityStore logged whatever value
they were handed, which on the failure-logging path is the raw key a caller sent
(a provider key included), let a newline in ?key= forge log lines, and fired on
lookups that expect a miss. The hash now rides KeyNotFoundError onto the single
existing auth failure log line next to the requester IP, with control characters
escaped. Also drops the new comments and updates skipped tests that quoted the
old message.
* test(auth): add the blank lines ruff format expects before _InFlightCountingPrisma
* test(integration): expect the generic wrong-key 401 in the management lifecycle tests
* fix(auth): keep the operator wording on the unknown-key log line and drop the redundant 401 rewrite
* test(auth): cover the generic unknown-key 401 across routes, proxies, and a worker kill
* test(auth): cover the realtime upgrade refusing an unknown key without echoing it
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: inject clocks and wait on conditions in three wall-clock-flaky unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: keep generic API logger tests on 3.10 and cover automatic sending
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: pin router usage minute in shared rate-limit header test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: stabilize three wall-clock-sensitive unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): cap cold schema validation workers at the CPUs the proxy can use
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: inject the cache clock and wait on lock contention instead of 1s deadlines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): wait on listing events instead of 1s deadlines in the discovery drain test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
* fix(proxy): resolve model aliases to their target deployment for listing metadata
A model alias (a key's `aliases`, a team's `model_aliases`, or the router's
`model_group_alias`) is rewritten at request time and owns no deployment row,
so `get_model_listing_info` returns None for it and the alias name is not a
cost-map key either. Both metadata candidate sources come up empty and the
listing entry carries no `mode`, `max_input_tokens`, or `max_output_tokens`,
while the deployment it routes to reports them on the same response.
Resolve a listed alias to the deployment it routes to for the metadata lookup,
keeping the alias as the response id. Maps are applied in order, each against
the result of the last, mirroring how `litellm_pre_call_utils` applies the team
map and then the key map, so a chained alias reports the deployment the caller
actually reaches rather than an intermediate one. An alias shadowing a
same-named deployment follows the alias, because the request rewrite does too.
Fixes the model_aliases half of #40553.
* fix(proxy): look listed aliases up through the shared alias chain
Drop the separate alias resolver and retarget a listed alias in
alias_listing_entries and alias_target instead, so /v1/models and
/v1/models/{id} report the model /chat/completions rewrites it to, and
only when the caller can list that model
* fix(proxy): keep deployed listed models' own rows and resolve team alias targets at list and retrieval
* fix(proxy): resolve router_settings aliases, keep alias labels, and skip paused targets in model listing
* fix(proxy): leave an alias to an undiscoverable model bare in model listing
A team alias whose target has discoverable: false showed the hidden model's mode and limits on GET /v1/models and retrieved it by alias. Undiscoverable groups now count as hidden for alias resolution, as on main, while admins still see them.
* fix(proxy): leave an alias to a model a listing callback hides bare in model listing
* fix(proxy): keep aliases bare for models a listing callback never judged or another team owns
---------
Co-authored-by: haydster7 <haydster7@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
/public/providers/fields re-read and re-parsed the bundled 85K
provider_create_fields.json on every request, and /public/agents/fields did the
same for that file plus agent_create_fields.json. Both are unauthenticated
public endpoints, and both did the work inside an `async def`, so the read and
the parse ran on the event loop and serialized against every other in-flight
request. Measured against the bundled files, that is roughly 0.40 ms of blocking
work per request for the provider endpoint and 0.45 ms for the agent endpoint
Both now memoize with `lru_cache`, which is what `/public/endpoints` in the same
module already does through its `_cached_endpoints` global, so each file is read
once per process rather than once per request
The agent endpoint mutated the dicts it had just loaded, appending inherited
provider credential fields and popping `inherit_credentials_from_provider`. That
is only safe while the data is re-read every time, so caching it meant making the
merge pure: `_agent_with_inherited_credentials` builds a new mapping per agent,
and the cached value is the merged result rather than the raw file contents.
Responses are byte identical before and after, checked on both endpoints against
a live proxy
The bundled records are typed as Mapping[str, object] and narrowed with
isinstance at the point of use, rather than typing.Any, which ruff-strict bans
Side effects: 3 of the repo's ASYNC230 violations go away, and basedpyright
errors in the file drop from 63 to 52
Rebased onto main after /public/providers/fields started validating the bundled
JSON through a pydantic adapter on every request. That validation now runs once
and the validated records are cached; the agent endpoint keeps merging the raw
provider records, so both responses stay byte identical to main.
Co-authored-by: Alok Sharma <alok44170@gmail.com>
