Commit graph

1040 commits

Author SHA1 Message Date
devin-ai-integration[bot]
689494ad5f test(passthrough): type the team deployment helper and mark test locals Final 2026-11-05 16:09:04 +00:00
devin-ai-integration[bot]
78f53a1a5e fix(passthrough): skip team deployments when picking proxy-wide pass-through credentials on lts/1.108.x 2026-11-05 16:02:13 +00:00
kerry
d07d295ad5 bump: version 1.108.1
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 01:39:40 +00:00
Kerry Lu
9f0484b987 fix(openai): send chat completions through client.post so the SDK's per-message param transform stops holding the event loop (#46967)
* fix(openai): send chat completions through client.post so the SDK's per-message param transform stops holding the event loop

create() walks every message, content part and tool of the request through
openai._utils._transform before posting, 91 ms at 300k tokens and ~300 ms at
1M tokens on the event loop, and returns the same dict for the plain dicts
litellm builds. client.post is what create() calls after that walk, same
request, headers, retries, raw response and stream objects, so the chat
helpers now call it directly through the helper embeddings already used

* fix(openai): type the chat client.post result as LegacyAPIResponse so the type gate sees .headers and .parse

* fix(openai): assert the chat client.post result is a LegacyAPIResponse instead of casting it

* test(openai): stub client.post with a LegacyAPIResponse spec in the empty-response and no-leak tests

* test(core-utils): stub client.post with a LegacyAPIResponse spec in the agentic loop leak test

* test: every client.post stub on the OpenAI chat path returns a LegacyAPIResponse-spec'd mock

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit e8550c1454)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 01:39:40 +00:00
paveln83
d089d524a3 fix(caching): key cached async clients by a per-loop token instead of id(loop) (#48877)
* fix(caching): key cached async clients by a per-loop token instead of id(loop)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): drop deferred closes queued on a collected event loop

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): pin that a close queued on a live loop waits for that loop

* test(caching): run loop-reuse tests until a loop address repeats

The collected-loop tests ran a fixed 50 loops and passed with the fix reverted whenever no loop reused an earlier address. They now stop at the first reuse and skip if none happens, through a shared helper.

* test(caching): cover the cache key fallback for loops without weak references

Also corrects the evicted client closer docstring: a dropped bucket's clients are left to the garbage collector.

* fix(http): free short-lived event loops whose handler finalizer close finished as the loop stopped

* fix(caching): share sync clients across event loops and catch a failed finalizer close

* fix(agentops): stop closing the shared sync http client after the auth call

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 4ce724e3df)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 01:39:40 +00:00
Kerry Lu
79a3567b66 feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos (#49785)
Backport of #49785 to lts/1.108.x.
Cherry-picked from fcef9cd0a3 (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 01:39:40 +00:00
Kerry Lu
f559536757 fix(logging): bill each SDK call its own native Bedrock guardrail trace when calls share one metadata dict (LIT-10239) (#48923)
Backport of #48923 to lts/1.108.x.
Cherry-picked from e21de0e418 (main).
2026-11-05 01:39:40 +00:00
Kerry Lu
a2e7cdcf7c feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost (#48278)
Backport of #48278 to lts/1.108.x.
Cherry-picked from 0b66a27528 (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 01:39:40 +00:00
devin-ai-integration[bot]
8fa525c90d fix(vertex-ai): sync tier prices and retirement floors (#49964)
Backport of #49964 to lts/1.108.x.
Cherry-picked from 62f85afacf (main). The cost-map schema keys and the schema test were re-applied by hand onto the line's test_utils.py, and the priority-markup test the line does not have was left out.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:26:44 +00:00
Kerry Lu
4f6bc3c507 fix(proxy): cancel the Admin UI's raw spend-log reads in Postgres after 30s (#49965)
Backport of #49965 to lts/1.108.x.
Cherry-picked from 2d0c720425 (main). The line keeps its inline session page query, so that read and the grouped count go through _query_spend_logs directly and _SpendLogsReader is not ported.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:26:44 +00:00
Kerry Lu
30c7420c4e fix(cost): bill gemini audio input and cached audio at flex, priority and batch rates (#48086)
Backport of #48086 to lts/1.108.x.
Cherry-picked from dc0237ccd7 (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:26:44 +00:00
Kerry Lu
de425fb956 fix(watsonx): read pass-through credentials from use_in_pass_through deployments (#49462)
Backport of #49462 to lts/1.108.x.
Cherry-picked from 948445c828 (main). get_proxy_wide_deployment_credentials hand-ported onto the line's _resolve_matching_deployment_api_key, since the line has no _get_deployment_credentials.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:26:44 +00:00
Kerry Lu
7dffccccae fix(batches): cost an unmanaged gs:// batch through the deployment that created it (#49301)
Backport of #49301 to lts/1.108.x.
Cherry-picked from 298911d61b (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:26:44 +00:00
Kerry Lu
277ee2e943 fix(proxy): abort the request with a 499 when the client disconnects during the body upload (#48640)
Backport of #48640 to lts/1.108.x.
Cherry-picked from 59b8c73930 (main). Dropped the pick's edit to the ValidationError branch the line does not have.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:26:44 +00:00
Kerry Lu
bc4c89bfc3 fix(proxy): cancel /team/spend/by_user in Postgres after 30s (#49922)
Backport of #49922 to lts/1.108.x.
Cherry-picked from 391c8013c9 (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:26:43 +00:00
Kerry Lu
6becdf304e fix(proxy): buffer tool discoveries in Redis and write LiteLLM_ToolTable in one statement per drain, on its own lock (#48089)
Backport of #48089 to lts/1.108.x.
Cherry-picked from e6a32bbeeb (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:15:40 +00:00
Kerry Lu
8304f16942 fix(router): reject string fallback targets at Router init (LIT-10238) (#48943)
Backport of #48943 to lts/1.108.x.
Cherry-picked from 96fe78f96e (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:15:40 +00:00
Kerry Lu
825a555b5b fix(router): cool down failing deployments when fallbacks use string targets (LIT-10238) (#48913)
Backport of #48913 to lts/1.108.x.
Cherry-picked from 84c4c5aa5c (main).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:15:40 +00:00
Kerry Lu
3a5abcfe1c fix(router): keep a lone deployment warm under router allowed_fails unless the policy names the exception (#48644)
Backport of #48644 to lts/1.108.x.
Cherry-picked from 8f31c7171d (main). Dropped the pick's edit to a stream-stall test the line does not have.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-05 00:15:40 +00:00
Rick
3df724803f fix(logging): drop a first-token time left over from a previous attempt (#47541)
Backport of #47541 to lts/1.108.x.
Cherry-picked from ced995df6e (main).
2026-11-05 00:15:40 +00:00
Kerry Lu
90bdf7fd01 fix(aiohttp_transport): keep the owned session open while a response stream is still being read (LIT-10237) (#48920)
Backport of #48920 to lts/1.108.x.
Cherry-picked from 4a70d80aef (main).
2026-11-05 00:15:40 +00:00
Kerry Lu
987819afd1 fix(anthropic): count search results and reject unsupported token-count blocks (#48699)
Backport of #48699 to lts/1.108.x.
Cherry-picked from 7acdf9f00d (main).
2026-11-05 00:15:40 +00:00
Ambuj Upadhyay
e7cb9a6483 fix(streaming): dict-usage arm drops prompt_tokens_details/completion_tokens_details its sibling arms preserve (#36678)
Backport of #36678 to lts/1.108.x.
Cherry-picked from cb70bc560e (main).
2026-11-05 00:15:40 +00:00
Kerry Lu
5b73742e4e fix(aiohttp_transport): close an owned aiohttp session when its transport is collected (#48650)
Backport of #48650 to lts/1.108.x.
Cherry-picked from d6edd37810 (main).
2026-11-05 00:15:40 +00:00
Kerry Lu
4695ad1deb fix(bedrock): price claude_platform calls from their own cost map rows (#48150)
Backport of #48150 to lts/1.108.x.
Cherry-picked from 560372456e (main).
2026-11-05 00:15:39 +00:00
Yuneng Jiang
bbc8484790
test(utils): stop the register_model replay test leaking gpt-4o pricing on 1.108
Brings over the test_utils.py isolation fix from #47135. The replay test registered openai/gpt-4o with supports_function_calling false into the shared cost map, so a later function calling test on the same worker failed depending on xdist ordering
2026-11-04 12:23:06 -08:00
Yuneng Jiang
0b5b41ca1b
test(bedrock): let GovCloud lag on native structured output on 1.108
Brings over the test-only GovCloud exemption from #46870. The 1.108 cost map already mirrors main, where us-gov Sonnet 5 profiles mark supports_bedrock_native_structured_output false while the commercial base marks it true
2026-11-04 12:04:41 -08:00
Yuneng Jiang
003192d5df
test(proxy): drop stale team cache update test on 1.108
main removed test_team_cache_update_called in #35643. The 1.108 line already carries the auth cache freshness backport, so update_cache no longer reads the team cache and the test fails
2026-11-04 12:01:20 -08:00
Yuneng Jiang
5352ed94fb
fix(proxy): backport auth cache freshness and spend updates
(cherry picked from commit b55024658c)
2026-11-02 13:58:11 -08:00
yuneng-jiang
76e42d73bd
fix(prometheus): retain deployment attribution on proxy failures (#49717)
(cherry picked from commit 1563776cd8)
2026-11-02 13:09:39 -08:00
devin-ai-integration[bot]
f86f570aca
fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure (#45136)
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure

A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first.

* fix(proxy): log a provider error frame through an attempt-scoped logging copy

The error-frame failure stashed the superseded attempt's usage on the logging
object the retry shares, so the retry's success log carried the failed
attempt's token counts. The failure now logs through a copy of the logging
object with its own model_call_details, and the helper moves into the
Anthropic passthrough logging handler so the proxy streaming handler only
routes by endpoint type.

* fix(proxy): scope the error-frame failure log to streams closed at the frame

A stream that ran to its end at the provider's error frame kept logging
through the success route before this PR, and the attempt-scoped failure
log is now taken only for a stream the router closed at that frame, with
the copy made at the close so the retry's deployment never reaches it

* chore: drop duplicate litellm import and mark the stream_ended rebind

* fix(proxy): log only router-superseded attempts as error-frame failures

A client that hangs up at a forwarded error frame closed the stream the same way the router closes a superseded attempt, so a non-retried stream lost its spend. The router now closes a superseded attempt through aclose_superseded_attempt, and only that close routes the attempt to the failure log

* test: type the _settle helper's recorded events

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 8bb8ac7f18)
2026-10-31 18:02:04 -07:00
Yuneng Jiang
8415e2d44b
Merge remote-tracking branch 'origin/rc/1.108.0' into litellm_chore_575e7d 2026-10-31 16:43:45 -07:00
yucheng-berri
57b539bf99
Merge pull request #49482 from BerriAI/litellm_backport_49262_49264_rc_1_108_0
fix(guardrails): backport #49262 and #49264 to rc/1.108.0, name only the pipeline steps that ran in x-litellm-applied-guardrails
2026-10-31 15:57:50 -07:00
Mateo Wang
43dcc2d80a
feat(bedrock): route token counts from model metadata (#47441)
* feat(bedrock): route token counts from model metadata

* fix(bedrock): keep counting metadata compatible with older proxies

(cherry picked from commit 7b82b2a3c3)
2026-10-31 15:46:49 -07:00
yuneng-jiang
03f7015f5f
ci: run the integration groups on GitHub Actions (#48985)
* test: add a manually dispatched integration trial on larger runners

Runs the providers, database and management integration groups on 16-core GitHub-hosted runners with Postgres and Redis as service containers, to measure wall time and how far each group scales with more workers

run_integration.sh now keeps a preset INTEGRATION_WORKERS, takes the service container names from the environment, and seeds test order from the GitHub run when there is no CircleCI one

* test: run the integration trial on push to this branch

* test: give integration trial workers a local redis-server

* test: run the providers trial on a 64-core runner with 32 workers

The 16-core run reached 53% of providers in 45 minutes with 16 workers, which points at CPU. Postgres gets max_connections=1000 and a 1 GB shm because one server now backs 32 worker proxies, where each CircleCI machine backs 4

* test: run every integration group in the trial on sized runners

* test: add a scan that refuses to upload integration results holding the license

* test: drop the member-name pass the archive byte scan already covers

* test: use a secret whose base64 and base64url encodings differ

* ci: run integration groups on run-ci pull requests and scheduled main runs

* ci: scope the integration license to a protected environment

Backport note: tests/integration/AGENTS.md keeps rc/1.108.0's text, since main's copy of that paragraph also carries later changes rc does not have

(cherry picked from commit 62b8ee78de)
2026-10-31 14:56:29 -07:00
yuneng-jiang
839bc9fda2
Merge pull request #49167 from BerriAI/litellm_backport_rc_1_108_0_eddb61
fix(mcp)!: backport #48939 to rc/1.108.0, restore upstream-token admission for legacy delegate servers
2026-10-31 14:46:52 -07:00
ryan-crabbe-berri
cd84c7e08c
test(budgets): age the monthly window past any calendar month so the reset test passes on the 31st (#49372)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 5029d027a1)
2026-10-31 13:00:28 -07:00
yucheng-berri
8be835c28a fix(guardrails): drop the response pipeline steps that never ran from x-litellm-applied-guardrails (#49264)
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran

* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails

* fix(guardrails): drop the response pipeline steps that never ran from x-litellm-applied-guardrails

* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails

* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line

* fix(guardrails): keep request steps when a response pipeline blocks before a later request pipeline

(cherry picked from commit 969ded2b54)
2026-10-31 12:12:55 -07:00
yucheng-berri
62e9c0ed71 fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails (#49262)
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran

* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails (#49263)

* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails

* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails

* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line

(cherry picked from commit fbc157437c)
2026-10-31 12:12:55 -07:00
shrey
18f0d8e0ce
fix(mcp)!: restore upstream-token admission for legacy delegate servers (#48939)
(cherry picked from commit fd8b89fbec)
2026-10-29 18:50:23 -07:00
Yuneng Jiang
82925fc3c0
test: accept the mirrored cost map's extraction mode, new row fields and openinfer rows
The mirrored map carries extraction mode rows, explicit_cache_read_input_token_cost and supports_bedrock_native_structured_output, and openinfer rows for a provider this line has no model set for. stable/1.106.x widened the same checks in ebf92c9a45 and f44d4a70c0
2026-10-27 19:57:39 -07:00
yucheng-berri
375df19d63
fix(security)!: build proxy-only failure logs from an allowlist and drop caller guardrail info (#48599)
* fix(proxy): build proxy-only failure logs from an allowlist and ignore request-time GCS credentials

* fix(proxy): drop caller metadata from proxy-only failure logs

metadata and litellm_metadata leave the proxy-only failure-log allowlist, so a rejected request can no longer point its failure log at its own Langsmith or Langfuse host through nested metadata.

* fix(proxy): keep allowlisted metadata in proxy-only failure logs

* test(proxy): isolate proxy-only failure log metadata tests

* fix(logging): ignore non-list request tags when building the logging payload

A guardrail-rejected request with metadata tags set to a non-list value crashed
the standard logging payload, so its spend row lost model attribution.

* fix(proxy): keep router attribution and drop unauthenticated tags in proxy-only failure logs

A request rejected before routing now logs the configured deployment and model group even when its metadata or litellm_metadata carries its own values. Requests rejected at auth no longer write caller tags into failure logs and tag spend.

* fix(proxy): always strip client guardrail info and mirror failure attribution into litellm_metadata

allow_client_pricing_override no longer keeps a caller-supplied standard_logging_guardrail_information on the chat or pass-through paths, since its guardrail_cost is billed. Proxy-only failure logs now also carry key and team attribution in litellm_metadata.

* refactor(proxy): keep the proxy-only failure-log fixes inside the type gates

Types the picked helpers so basedpyright and the type-discipline gate stay at main's counts. No runtime change.

* fix(proxy): keep the WS frame guardrail-info strip and string interaction tags as sent

The guardrail-info key left the pricing strip list, so the Responses WS frame guard now drops it by name. get_request_tags no longer raises on string tags, so background interaction polling checks for them before keeping them as sent.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 788868a52f)
2026-10-27 19:57:39 -07:00
Ishaan
319030c73b
fix(ci): unblock dependency security checks (#48055)
* fix(deps): upgrade vulnerable Python dependencies

* fix(deps): lock patched Python dependency versions

* fix(otel): preserve JSON export with the updated SDK

* test(otel): cover JSON export with the updated SDK

* test(otel): update SDK log records and worker discovery

* fix(ui): align Lens peer dependency with patched Next.js

* fix(otel): hand the AgentOps exporter a session it owns for the JWT header

OpenTelemetry 1.45 moved the OTLP HTTP exporter's requests.Session behind a
transport, so writing the minted JWT to the exporter's private _session raised,
was swallowed, and every AgentOps batch went out unauthenticated. The exporter
now creates the session, passes it through the public session= argument and
sets the Authorization header on it

The tests now drive a real export, serving the token endpoint with respx and
capturing the OTLP POST with requests_mock, instead of reading SDK internals

* fix(otel): skip FastAPI instrumentation once the app has started

opentelemetry-instrumentation-fastapi 0.66b1 wraps build_middleware_stack instead
of calling add_middleware, so instrumenting an app whose middleware stack is
already built no longer raises. The call silently did nothing while we still
registered the global response propagator. Check the built stack up front and
skip both, which is what the existing test expects

* test(otel): assert exported requests and span flags instead of SDK internals

OpenTelemetry 1.45 dropped the exporter's private _headers and sets the W3C
random-trace-id flag on new traces, so traceresponse ends in -03. The Langtrace
and Langfuse tests now export through requests_mock and check the URL and headers
the collector receives, and the traceresponse tests expect the server span's own
trace flags

* fix(deps): bump dashboard zod to 4.6.6 and make the vendored Lens use it

zod 4.6.5 is flagged by the dependency scan and blocks osv-scan on every PR. A
root bump alone leaves the vendored Lens tarball, which pins zod 4.6.5 exactly,
on its own nested copy, so a top-level zod override points every consumer at the
root version and the lock carries a single zod 4.6.6

* test(ui): stub the tag summary fetch the client panel retry refreshes

Retrying the model breakdown refetches /tag/summary, which this integration test
left on the real network. When that request failed before the assertion, the
panel switched to its summary error and unmounted the model row, so the test
passed or failed by timing

* fix(deps): drop the duplicated Lens override from the dashboard package.json

* style(ui): format the client panel fetch stub

---------

Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
(cherry picked from commit 8f9894fbd6)
2026-10-27 19:57:39 -07:00
Kerry Lu
6c971b7412
fix(proxy): attribute retry attempt spend rows to their deployment and keep them out of daily request counts (#47658)
* fix(proxy): attribute retry attempt spend rows to their own deployment and keep them out of daily request counts

Each <request_id>-attempt-<n> row now carries the model, provider, api_base and model group of the deployment that billed it (looked up by model_info.id through the proxy router, injected as deployment_lookup) instead of copying the final row's, and is stamped internal_call_origin=billed_retry_attempt so the daily spend tables keep its spend and tokens while counting zero api_requests and failed_requests for it

* fix(proxy): declare DeploymentLookup as a TypeAlias and leave the test file's untouched lines as they were

* fix(proxy): mark attempt rows with billed_retry_attempt instead of internal_call_origin so model usage keeps their spend

Bugbot: an internal_call_origin stamp made build_model_usage_transaction drop the attempt rows from LiteLLM_DailyModelUsage. The rows now carry their own SpendLogsMetadata flag, which only the daily request counters read

* fix(proxy): read the billed_retry_attempt flag through a typed helper so the basedpyright gate holds

* fix(proxy): keep the typed litellm_params local in _deployment_provider so the basedpyright gate holds

* fix(proxy): annotate the validated litellm_params as LiteLLM_Params in _deployment_provider

* fix(proxy): keep billed retry attempts out of daily request errors and strip the flag from caller metadata

Bugbot: _record_request_error counted every attempt row as a caller failure in LiteLLM_DailyRequestErrors. Veria: a caller could send metadata.billed_retry_attempt to zero its own request counts, so the key joins the untrusted metadata fields the proxy strips on the way in

* fix(proxy): stamp billed_retry_attempt explicitly in the spend log metadata so the basedpyright gate holds

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 5dce13d2c2)
2026-10-27 19:57:39 -07:00
Kerry Lu
e1dc2eb201
fix(logging): spilled over PTU requests without base_model log the served model's rates (#47655)
* fix(logging): report pay-as-you-go rates for spilled over PTU requests without base_model

* fix(logging): keep the provider response model so spilled over PTU payloads price the served model

* fix(logging): keep a provider_response_model the response already carries

* fix(logging): read the carried provider_response_model through a typed adapter

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 6b1f4ed46f)
2026-10-27 19:57:39 -07:00
Kerry Lu
de53be5bc6
fix(proxy): price earlier timed-out retry attempts at their own deployment's model rate (#47648)
* fix(proxy): bill every earlier timed-out retry attempt at its own deployment's model rate

* fix(router): read the retry record's base_model through the typed metadata helper

* fix(router): keep the four-record retry breadcrumb cap

* fix(router): carry a deployment's litellm_params base_model on its retry record

* chore(router): place the cast-ok reason on the cast line and list the base_model helper as covered through log_retry

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 27efc387f7)
2026-10-27 19:57:38 -07:00
Kerry Lu
99fe893640
fix(router): price cost-based routing through model_info.base_model (#47617)
* fix(router): price cost-based routing through model_info.base_model

* fix(router): keep cost-based routing pricing helpers fully typed

* fix(router): drop typing.cast from cost-based routing pricing helpers

* fix(router): read the cost map through a typed accessor in cost-based routing

* fix(router): avoid a Final rebind inside the cost-based routing loop

* refactor(router): share deployment_pricing_models across spend, listing and cost-based routing

* refactor(router): keep the /v1/models listing key inline, share pricing models for spend and routing

* refactor(router): inline the deployment pricing-model wrapper in cost-based routing

* refactor(router): read deployment pricing models through one shared helper

deployment_pricing_models(deployment) now owns the model_info.base_model / litellm_params.base_model /
litellm_params.model unpacking, used by cost-based routing and the /v1/models listing. pricing_models(base_model, model)
stays for the failure-row spend estimate, which already holds the two names

* refactor(router): share the str-keyed mapping validator with cost-based routing

* ci: re-run after flaky test_aiohttp_transport read timeout in llms-providers

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 0c6731eb5c)
2026-10-27 19:57:38 -07:00
Kerry Lu
5ebf406610
fix(proxy): price a 499 spend row only for the proxy's own disconnect, and never after the call already completed (#47652)
* fix(proxy): price a 499 spend row only for the proxy's own disconnect, and never after the call already completed

* fix(proxy): gate the zero-cost 499 row on the success log being enqueued, not merely claimed

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit ffb7133d32)
2026-10-27 19:57:38 -07:00
Kerry Lu
642be9bceb
fix(proxy): scope the search free-tier counter to the request's provider (#47646)
* fix(proxy): scope the search free-tier counter to the request's provider

* fix(proxy): read the provider with get so partial spend payloads still get the free tier

* test(proxy): give the grounded spend payload fixture its provider

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 640f2d865b)
2026-10-27 19:57:38 -07:00
Kerry Lu
432d39c949
fix(proxy): price a billed failure row at the deployment's long-context tier like the success path (#47647)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 70ece3501f)
2026-10-27 19:57:38 -07:00