mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-26 06:03:19 +00:00
6762d65eb8
* fix(vertex_ai): price streamed pass-through calls with the regional uplift The streamed Vertex pass-through handler builds the response with stream_chunk_builder, which stamps a cost into _hidden_params without the vertex_location. The handler then prices the response again with the location from the URL, but only into kwargs, and the logger prefers the stamped cost. Streamed calls on a regional endpoint were billed at the global rate. Stamp the location-aware cost on the response in the shared generateContent payload builder so the logger records it. Fixes #44755 * style(vertex_ai): keep the response_cost assignment within the 120-char limit * refactor(vertex_ai): set the passthrough response cost with set_hidden_param Writing _hidden_params directly mutated a parameter and needed a pyright ignore, which the type discipline gate counts as LIT004 and LIT011. set_hidden_param is the shared helper for this. * test(integration): check Vertex pass-through spend rows keep the regional uplift when streamed * fix(proxy): price streamed Vertex pass-through spend like the non-streamed call Drop the location-blind cost stream_chunk_builder stores instead of stamping a cost, so the logger prices streamed generateContent and rawPredict calls with the URL's location and the deployment's custom pricing, the same way it prices non-streamed calls * fix(proxy): price custom-priced Vertex pass-through deployments at their own rates The stamped response_cost from the generateContent logging helper is returned before the logger checks custom pricing, so a deployment with custom input/output rates was billed at the cost-map price. The helper now passes custom_pricing and the deployment id into completion_cost, so the stamp holds the deployment rate for streamed and non-streamed calls, and deployments without custom pricing keep the regional uplift. * fix(proxy): bill Vertex /v1beta and interrupted pass-through streams by the deployment location and price * fix(proxy): give the new pyright ignore a reason for the LIT004 gate * test(integration): bill a custom-priced Vertex pass-through stream at the deployment price, cut-off streams included * fix(proxy): let the logger price non-streamed /gemini pass-through calls so both modes bill the same * fix(proxy): keep a client base_model out of pass-through pricing and price gateway Gemini streams as Gemini A top-level base_model in a pass-through body now reaches the logging params only for a key or team with allow_client_pricing_override, so a plain key cannot pick a cheaper model's price once the logger prices these calls. The test that pinned every litellm-owned body name into the logging params now leaves base_model to its own key-gated test. The Vertex handler now classifies a non-Google upstream host as Vertex only for Vertex resource paths, so a /gemini stream through a GEMINI_API_BASE gateway is priced as Gemini without the regional uplift. Adds the audit's integration spend test for Claude on Vertex, /v1beta, the named custom-priced /gemini deployment, and a cut OpenAI stream. * test(integration): price the gateway /gemini stream case as Gemini and cover client base_model and gateway host pricing The cost harness sends /gemini through GEMINI_API_BASE, which the pass-through now logs and bills as gemini, so the streamed case moves from the Vertex cases to the Gemini cases under the Gemini price. New spend tests cover a body base_model repricing only for keys allowed client pricing, a GEMINI_API_BASE gateway billing as Gemini in both modes, and a configured non-Google gateway picking the provider from the path. * fix(proxy): price pass-through Gemini by the route and ignore a non-string base_model The Vertex pass-through handler picked Gemini pricing from the URL path, so a configured pass-through to a Vertex gateway with a short path (no /projects/ segment) was billed as Gemini, and its Claude streams at $0. The provider now comes from the route: a call through /gemini or to generativelanguage.googleapis.com prices as Gemini, and everything else as Vertex, the same rule main used. Matching the GEMINI_API_BASE host was rejected because one gateway can serve both APIs, and the path rule was rejected because short gateway paths look like Gemini paths. A non-string base_model (a number, list or object) made the cost calculation fail and bill $0 once the stamped stream costs were gone. get_base_model_from_metadata now ignores it and falls back to the deployment's base_model, so the call bills the list price. A 400 was the other option, but the logger runs after the upstream answered. * test(integration): bill a non-string body base_model at the list price on every pass-through route * fix(proxy): bill a non-string model_info base_model at the list price and drop a non-object client model_info A key allowed client pricing can send metadata.model_info in a pass-through body. A non-string model_info.base_model made the cost lookup fail, and once the stamped stream and Gemini costs were gone those calls billed $0 on /gemini and on streamed /vertex_ai and /v1beta. The lookup now ignores a base_model that is not a string, the same rule the root base_model got, so the call bills the list price. A model_info that is not an object (a string or a list) broke several logging steps that read it as a mapping, so a non-streamed /gemini call fired no success callback and no pass-through call wrote a spend row. The pass-through route now leaves a non-object model_info out of the logging metadata, so the call bills the list price. A 400 was the other option, but it would turn calls that answer 200 today into errors, and guarding every reader of model_info would spread the same check over many files. * fix: reject a non-object model_info before the provider call again The previous commit made the base_model lookup return None for a model_info that is not an object. That lookup also runs during call setup on every router route and SDK entrypoint, where such a value used to fail before any provider request. With the early return the call went upstream, the provider billed it, and the request then failed in a later reader that still expects an object. On /v1/messages and /v1/responses the key was also charged for a response the client never got. The lookup again reads base_model from any truthy model_info, so a non-object value fails during setup the way it did before this PR. A non-string base_model inside an object still bills the list price, and the pass-through route still drops a non-object client model_info at intake, so pass-through calls keep billing the list price. * fix: match the Gemini API host exactly when picking the pass-through cost provider The provider check accepted any host ending in generativelanguage.googleapis.com, so a Vertex host such as vertex-generativelanguage.googleapis.com was priced as Gemini and lost the regional uplift. It now accepts that host or a subdomain of it. * test: pin that a subdomain of the Gemini API host keeps the Gemini price --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| fixtures | ||
| golden | ||
| __init__.py | ||
| _background_responses.py | ||
| _daily_activity_fixtures.py | ||
| test_agent_spend.py | ||
| test_autorouter_session_rollup.py | ||
| test_background_interaction_settlement.py | ||
| test_background_response_billed_once.py | ||
| test_background_response_billing_proxies.py | ||
| test_background_response_poll_retirement.py | ||
| test_baseline_accounting.py | ||
| test_batch_completion_accounting.py | ||
| test_batch_enqueued_tokens_redis_lua.py | ||
| test_batch_observability.py | ||
| test_batch_poll_starvation.py | ||
| test_budget_limit_envelopes.py | ||
| test_cache_activity.py | ||
| test_cache_and_quota.py | ||
| test_chaos_burst_spend_once.py | ||
| test_container_spend.py | ||
| test_daily_activity_aggregated_breakdowns.py | ||
| test_daily_activity_key_alias_probes.py | ||
| test_daily_activity_key_metadata_query_timeout.py | ||
| test_daily_activity_key_owner.py | ||
| test_daily_activity_key_owner_faults.py | ||
| test_daily_activity_key_owner_traffic.py | ||
| test_daily_activity_repository.py | ||
| test_daily_activity_routes.py | ||
| test_daily_rollup_retry.py | ||
| test_daily_tag_spend_retention.py | ||
| test_disable_end_user_cost_tracking.py | ||
| test_disconnected_bedrock_messages_stream_billing.py | ||
| test_end_user_budget_enforcement.py | ||
| test_end_user_spend_without_proxy_user.py | ||
| test_expired_ui_session_failure_rows.py | ||
| test_failed_dispatch_tokens.py | ||
| test_filtered_ledger.py | ||
| test_global_spend_report.py | ||
| test_global_spend_reset.py | ||
| test_google_genai_native_spend.py | ||
| test_image_generation_key_spend.py | ||
| test_key_budget_lockout.py | ||
| test_key_metadata_recovery_dropped_connection.py | ||
| test_key_metadata_recovery_probe_bounds.py | ||
| test_legacy_spend_logs_row_cap.py | ||
| test_messages_stream_usage_cost.py | ||
| test_model_router_selected_model.py | ||
| test_normalized_error_long_message.py | ||
| test_org_budget_cli_session_token.py | ||
| test_passthrough_budget_reservation.py | ||
| test_passthrough_request_tags.py | ||
| test_passthrough_stream_pricing_paths_spend.py | ||
| test_prompt_caching_requests_pagination.py | ||
| test_ptu_share_attribution.py | ||
| test_redis_ttl_preserving_token_increment.py | ||
| test_requester_ip_address_logging.py | ||
| test_reset_budget_leader_election.py | ||
| test_responses_bridge_spend_once.py | ||
| test_responses_cache_write_itemization.py | ||
| test_responses_websocket_turn_rows.py | ||
| test_roi_branch_spend.py | ||
| test_scoped_spend_reports.py | ||
| test_service_tier_stream_billing.py | ||
| test_session_total_spend.py | ||
| test_shutdown_flush.py | ||
| test_spend_attribution_contracts.py | ||
| test_spend_calculate.py | ||
| test_spend_capture_rate_captured_spend.py | ||
| test_spend_log_caller_session.py | ||
| test_spend_log_read_scope.py | ||
| test_spend_log_tool_payload_content.py | ||
| test_spend_log_write_batching.py | ||
| test_spend_logs_metadata_fields.py | ||
| test_spend_rollup_accuracy.py | ||
| test_stream_alias_billing.py | ||
| test_tag_budget_enforcement.py | ||
| test_team_budget_enforcement.py | ||
| test_team_daily_activity_aggregated.py | ||
| test_team_member_budget_alerts.py | ||
| test_team_member_spend.py | ||
| test_team_member_spend_flush.py | ||
| test_user_budget_on_team_keys.py | ||
| test_vertex_image_regional_uplift.py | ||
| test_vertex_passthrough_regional_uplift_spend.py | ||
| test_vertex_regional_uplift_spend.py | ||