mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-03 07:23:33 +00:00
33457d3ac9
* test(proxy): cover per-frame model checks and session pricing on the OpenAI websocket passthrough * fix(proxy): check the model in every OpenAI websocket passthrough frame and price the session * fix(proxy): bill each OpenAI websocket turn on its own row and run full auth per frame Each finished turn (completed, incomplete, or failed) now logs its own spend row under the model and access groups that allowed it, so group budgets see spend as turns end instead of at close. Every model a frame names goes through the key's full auth, and a stored prompt reference with no model is refused for a key held to some models. A frame still in flight when the client leaves is recorded before the relay gives up on the send * fix(proxy): keep the stored prompt refusal's first sentence inside the websocket close reason The close reason carries the refusal's first sentence, and the stored prompt sentence ran past the 123 bytes a close frame allows, so it was cut mid-word. The sentence is shorter now and the rest of the explanation stays in the error event. * fix(proxy): bill pipelined websocket turns to their own frame and close the review gaps * fix(proxy): log no extra websocket spend row for frames after the last turn * fix(proxy): price websocket Responses turns by served tier and hosted tools, and bill turns a client walked out of A websocket Responses turn is now priced through the same Responses cost path the HTTP passthrough uses, so a priority-tier answer and a hosted web search cost what they cost over HTTP. When the client leaves mid-answer, the gateway keeps reading the provider's socket until that turn ends, up to 10 minutes, so the turn the provider still bills reaches spend * fix(proxy): bill websocket turns by lane, skip turns OpenAI refuses, and stop frame checks counting the session's own reservation An error event refuses a queued response.create, so the next turn is no longer charged to the refused frame's model. Turns on separate stream_id lanes are charged to the frame that started each one. The frame model gate settles the connect reservation before it re-runs auth, so a key with less budget left than one turn's max cost is no longer refused on its first frame. * fix(proxy): bill each websocket lane cut off at session end on its own row When a client leaves while turns on several stream_id lanes are still running, each unfinished turn now gets its own spend row under the groups and model its own frame asked for, instead of one merged row. Also pins the PR's new single-assigned test variables as Final and swaps datetime.now() in the logging-handler and success-handler fixtures for fixed timestamps. * test(proxy): mark the remaining new websocket test variables Final Builds the logging mocks with model_call_details in the constructor instead of setting it afterwards. * fix(proxy): check every model the realtime URL names before the OpenAI websocket relay connects * fix(proxy): keep reading running answers after a refused websocket frame so they are billed * fix(proxy): bill realtime transcription sessions on the OpenAI websocket passthrough A transcription session names its model only in session.updated under audio.input.transcription, and reports usage on each completed transcription instead of response.done, so its row logged unknown at $0 and a budgeted key never spent. * test(proxy): mark the turn-taking client's delivered list Final * fix(proxy): hold a websocket session's parallel slot until the socket closes Each finished turn logged its spend row under the session's own call id, so the first turn's success callback freed the key's max_parallel_requests slot while the socket was still open, and a key limited to one session could open several at once. The session now claims its slot under its own call id, every turn row logs under a call id of its own, and the slot is released when the session ends, as before Priced turn rows also take their row's call id as their id, so two turns that start in the same instant no longer share a spend log id and drop one row * fix(proxy): bill realtime transcription clips that finish outside an answer A conversation.item.input_audio_transcription.completed frame that arrived after a session's last response.done was never billed, because the session end only owes a row for a turn still running. A clip that completes while no answer is running now logs its own row, and a clip inside a running answer stays in that answer's row Each row also carries the session's latest session.updated frame, since later rows lost the transcription model it names and priced clips against the realtime model. A newer session.updated replaces the older one, so the stale model is never read first It also drops the docstrings this PR had added, since the names already say what each helper does * fix(cost): price a realtime transcription clip at the transcription model the session set A realtime conversation session opens with session.created naming the realtime model and no transcription model, and the client sets the transcription model later in session.update. The cost code took the first session frame and fell back to its session model, so a clip was priced at the realtime model. That model has no per-second rate, so a gpt-transcribe clip cost $0, and a token-billed clip used the realtime model's audio rate instead of the transcription model's. Now the newest session frame that names a transcription model sets the price, and the first session model is only the fallback when no frame names one, as before * fix(cost): price each realtime transcription clip at the model set when it ran A session that changes its transcription model mid-way kept every logged clip on the newest model. Each clip now takes the transcription model from the last session event before it, and falls back to the session's own model * fix(passthrough): keep the session update a buffered clip ran under until its answer is billed A transcription that completes while an answer streams waits in the buffer for that answer's response.done. A session.updated in between used to drop the earlier update, so the clip was priced at the opening configuration. An earlier update now stays while an unbilled clip follows it, and drops at the next update once that clip is billed. * fix(passthrough): keep reading a websocket turn the client left before OpenAI started it A client that sends response.create and hangs up before OpenAI answers with response.created made the walkout checks see no turn in flight, so the gateway closed the provider socket and the turn OpenAI still ran and billed never reached spend. The relay now counts every forwarded create until the provider starts or refuses it, and keeps draining while one is owed * fix(passthrough): settle a websocket turn that ends without its start frame A turn the provider ended without sending response.created first left its create counted as not started, so a client leaving later held the provider socket for the whole walkout drain with nothing left to bill. A turn's end now settles one waiting create when its lane saw no start frame * fix(proxy): refuse unparseable and unroutable websocket frames and hold legacy slots per socket A client frame too deep or too long for the JSON parser closed the provider socket mid-turn, so the turn in flight was never billed. The gate now refuses that frame with an error event and keeps draining the open turn. A stream_id OpenAI rejects comes back as an error that names no lane, which dropped another turn's authorization. The gate now refuses those frames before they reach OpenAI. With the legacy parallel request limiter, each per-turn spend row freed the key's slot while the socket stayed open. Websocket passthrough rows now free nothing, and the socket frees its slot once when it closes. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_admin_viewer_handler_access.py | ||
| test_auth_checks.py | ||
| test_auth_checks_object_access_and_lookup.py | ||
| test_auth_exception_handler.py | ||
| test_auth_hot_path_network_requests.py | ||
| test_auth_object_prefetch.py | ||
| test_auth_utils.py | ||
| test_authorization.py | ||
| test_banned_params_extra_body.py | ||
| test_cli_auth.py | ||
| test_custom_auth_end_user_budget.py | ||
| test_default_end_user_budget_simple.py | ||
| test_fallback_budget.py | ||
| test_fallback_model_access.py | ||
| test_handle_jwt.py | ||
| test_info_routes.py | ||
| test_jwt.py | ||
| test_litellm_license.py | ||
| test_login_utils.py | ||
| test_master_key_boot_check.py | ||
| test_mcp_ip_filtering.py | ||
| test_model_access_group_budgets.py | ||
| test_model_checks.py | ||
| test_model_checks_fallbacks.py | ||
| test_models_fallback_endpoint.py | ||
| test_multi_budget_windows.py | ||
| test_multipart_bypass_repro.py | ||
| test_network.py | ||
| test_oauth2_proxy_hook.py | ||
| test_object_permission_loading.py | ||
| test_onboarding.py | ||
| test_organization_budget_enforcement.py | ||
| test_password_hashing.py | ||
| test_password_policy.py | ||
| test_personal_key_team_model_access.py | ||
| test_proxy_routes.py | ||
| test_rds_iam_token.py | ||
| test_resolvers_exceptions.py | ||
| test_resolvers_grants.py | ||
| test_resolvers_models.py | ||
| test_resolvers_seam.py | ||
| test_resolvers_store.py | ||
| test_route_checks.py | ||
| test_router_override_fallback_auth.py | ||
| test_search_tool_deny_by_default.py | ||
| test_team_grants.py | ||
| test_team_member_budget.py | ||
| test_unmapped_model_budget_enforcement.py | ||
| test_user_api_key_auth.py | ||
| test_user_api_key_auth_request_flow.py | ||