* fix(openai): send chat completions through client.post so the SDK's per-message param transform stops holding the event loop
create() walks every message, content part and tool of the request through
openai._utils._transform before posting, 91 ms at 300k tokens and ~300 ms at
1M tokens on the event loop, and returns the same dict for the plain dicts
litellm builds. client.post is what create() calls after that walk, same
request, headers, retries, raw response and stream objects, so the chat
helpers now call it directly through the helper embeddings already used
* fix(openai): type the chat client.post result as LegacyAPIResponse so the type gate sees .headers and .parse
* fix(openai): assert the chat client.post result is a LegacyAPIResponse instead of casting it
* test(openai): stub client.post with a LegacyAPIResponse spec in the empty-response and no-leak tests
* test(core-utils): stub client.post with a LegacyAPIResponse spec in the agentic loop leak test
* test: every client.post stub on the OpenAI chat path returns a LegacyAPIResponse-spec'd mock
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit e8550c1454)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): key cached async clients by a per-loop token instead of id(loop)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): drop deferred closes queued on a collected event loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): pin that a close queued on a live loop waits for that loop
* test(caching): run loop-reuse tests until a loop address repeats
The collected-loop tests ran a fixed 50 loops and passed with the fix reverted whenever no loop reused an earlier address. They now stop at the first reuse and skip if none happens, through a shared helper.
* test(caching): cover the cache key fallback for loops without weak references
Also corrects the evicted client closer docstring: a dropped bucket's clients are left to the garbage collector.
* fix(http): free short-lived event loops whose handler finalizer close finished as the loop stopped
* fix(caching): share sync clients across event loops and catch a failed finalizer close
* fix(agentops): stop closing the shared sync http client after the auth call
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 4ce724e3df)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #49785 to lts/1.108.x.
Cherry-picked from fcef9cd0a3 (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48278 to lts/1.108.x.
Cherry-picked from 0b66a27528 (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #49964 to lts/1.108.x.
Cherry-picked from 62f85afacf (main). The cost-map schema keys and the schema test were re-applied by hand onto the line's test_utils.py, and the priority-markup test the line does not have was left out.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #49965 to lts/1.108.x.
Cherry-picked from 2d0c720425 (main). The line keeps its inline session page query, so that read and the grouped count go through _query_spend_logs directly and _SpendLogsReader is not ported.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48086 to lts/1.108.x.
Cherry-picked from dc0237ccd7 (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #49462 to lts/1.108.x.
Cherry-picked from 948445c828 (main). get_proxy_wide_deployment_credentials hand-ported onto the line's _resolve_matching_deployment_api_key, since the line has no _get_deployment_credentials.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #49301 to lts/1.108.x.
Cherry-picked from 298911d61b (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48640 to lts/1.108.x.
Cherry-picked from 59b8c73930 (main). Dropped the pick's edit to the ValidationError branch the line does not have.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #49922 to lts/1.108.x.
Cherry-picked from 391c8013c9 (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48089 to lts/1.108.x.
Cherry-picked from e6a32bbeeb (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48943 to lts/1.108.x.
Cherry-picked from 96fe78f96e (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48913 to lts/1.108.x.
Cherry-picked from 84c4c5aa5c (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48644 to lts/1.108.x.
Cherry-picked from 8f31c7171d (main). Dropped the pick's edit to a stream-stall test the line does not have.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Brings over the test_utils.py isolation fix from #47135. The replay test registered openai/gpt-4o with supports_function_calling false into the shared cost map, so a later function calling test on the same worker failed depending on xdist ordering
Brings over the test-only GovCloud exemption from #46870. The 1.108 cost map already mirrors main, where us-gov Sonnet 5 profiles mark supports_bedrock_native_structured_output false while the commercial base marks it true
main removed test_team_cache_update_called in #35643. The 1.108 line already carries the auth cache freshness backport, so update_cache no longer reads the team cache and the test fails
Scope the Prisma migration lock test's advisory lock check to its own database, so a lock another xdist worker holds in its database no longer trips it
Keep in-process MCP peers' idle connections open for 30s instead of 1s, so the gateway's pooled HTTP client doesn't reuse a connection the peer is closing and hang the tool call
Delete database guardrails in the logging-only scope tests through DELETE /guardrails/{id} on the shared gateway instead of raw SQL, so a default-on guardrail stops applying to the next test on the same worker right away
Bound the wedged coordination Redis /customer/update check by the shortest Redis wait the proxy uses, like the burst check, so a slow runner can't fail it while a handler that waits on Redis still does
(cherry picked from commit 4f2c75ddf5)
Takes only #47108's (e85cb4052e) changes to coordination_redis_proxy_config.yaml and test_user_updates_wedged_coordination_redis.py. They set the coordination Redis socket timeout and define the shortest Redis wait bound that #49554's /customer/update fix uses
test_client_disconnect_before_any_content_sends_no_response_scan released the gated upstream right after
the client closed, so the upstream's next chunk raced the disconnect into the proxy. When the chunk won,
the proxy had " tail" on record for the client and the cut-short scan recorded a post_call success on
the spend row, which the test asserts is absent (CircleCI integration-extensions job 2263560 on
68db022d57: `AssertionError: (('success',),)`).
The test now reads the spend row first, which is written once the proxy has settled the disconnect, and
releases the upstream after it. The proxy's behavior is unchanged; only the test's ordering is.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
(cherry picked from commit 89a194bbd0)
Takes only the max_workers change in test_team_delete_large_membership.py from 7e8be61fa0. The 16-thread key warmup slowed the /team/delete that follows it to 5.4-11.6s locally against the client's 15s timeout, and 2 threads brings it to 4.1-5.4s
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure
A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first.
* fix(proxy): log a provider error frame through an attempt-scoped logging copy
The error-frame failure stashed the superseded attempt's usage on the logging
object the retry shares, so the retry's success log carried the failed
attempt's token counts. The failure now logs through a copy of the logging
object with its own model_call_details, and the helper moves into the
Anthropic passthrough logging handler so the proxy streaming handler only
routes by endpoint type.
* fix(proxy): scope the error-frame failure log to streams closed at the frame
A stream that ran to its end at the provider's error frame kept logging
through the success route before this PR, and the attempt-scoped failure
log is now taken only for a stream the router closed at that frame, with
the copy made at the close so the retry's deployment never reaches it
* chore: drop duplicate litellm import and mark the stream_ended rebind
* fix(proxy): log only router-superseded attempts as error-frame failures
A client that hangs up at a forwarded error frame closed the stream the same way the router closes a superseded attempt, so a non-retried stream lost its spend. The router now closes a superseded attempt through aclose_superseded_attempt, and only that close routes the attempt to the failure log
* test: type the _settle helper's recorded events
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 8bb8ac7f18)
rc/1.108.0 carries OpenTelemetry 1.45.1 (#48055), #47611 and the #47639 base_model strip, so the OTel traceresponse header, langtrace outage, guardrail concurrent-disconnect and Bedrock lookaround tests expected the old behavior. This takes #48665's hunks for those four files only, without the unhandled to non_object_body rename, which depends on #48438 that rc does not have
(cherry picked from commit 21ab2cb87b, four test files only)
rc/1.108.0 carries the #43935 backport, which enforces gpt-5's supports_none_reasoning_effort and supports_xhigh_reasoning_effort flags, so the test now reads the levels it parametrizes from the cost map like main does. Only this file of #47600 is taken. Its other files follow changes rc does not have
(cherry picked from commit e78f7b4602, tests/integration/providers/test_cursor_responses_wire.py only)
