mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-29 06:33:28 +00:00
6762d65eb8
* fix(vertex_ai): price streamed pass-through calls with the regional uplift The streamed Vertex pass-through handler builds the response with stream_chunk_builder, which stamps a cost into _hidden_params without the vertex_location. The handler then prices the response again with the location from the URL, but only into kwargs, and the logger prefers the stamped cost. Streamed calls on a regional endpoint were billed at the global rate. Stamp the location-aware cost on the response in the shared generateContent payload builder so the logger records it. Fixes #44755 * style(vertex_ai): keep the response_cost assignment within the 120-char limit * refactor(vertex_ai): set the passthrough response cost with set_hidden_param Writing _hidden_params directly mutated a parameter and needed a pyright ignore, which the type discipline gate counts as LIT004 and LIT011. set_hidden_param is the shared helper for this. * test(integration): check Vertex pass-through spend rows keep the regional uplift when streamed * fix(proxy): price streamed Vertex pass-through spend like the non-streamed call Drop the location-blind cost stream_chunk_builder stores instead of stamping a cost, so the logger prices streamed generateContent and rawPredict calls with the URL's location and the deployment's custom pricing, the same way it prices non-streamed calls * fix(proxy): price custom-priced Vertex pass-through deployments at their own rates The stamped response_cost from the generateContent logging helper is returned before the logger checks custom pricing, so a deployment with custom input/output rates was billed at the cost-map price. The helper now passes custom_pricing and the deployment id into completion_cost, so the stamp holds the deployment rate for streamed and non-streamed calls, and deployments without custom pricing keep the regional uplift. * fix(proxy): bill Vertex /v1beta and interrupted pass-through streams by the deployment location and price * fix(proxy): give the new pyright ignore a reason for the LIT004 gate * test(integration): bill a custom-priced Vertex pass-through stream at the deployment price, cut-off streams included * fix(proxy): let the logger price non-streamed /gemini pass-through calls so both modes bill the same * fix(proxy): keep a client base_model out of pass-through pricing and price gateway Gemini streams as Gemini A top-level base_model in a pass-through body now reaches the logging params only for a key or team with allow_client_pricing_override, so a plain key cannot pick a cheaper model's price once the logger prices these calls. The test that pinned every litellm-owned body name into the logging params now leaves base_model to its own key-gated test. The Vertex handler now classifies a non-Google upstream host as Vertex only for Vertex resource paths, so a /gemini stream through a GEMINI_API_BASE gateway is priced as Gemini without the regional uplift. Adds the audit's integration spend test for Claude on Vertex, /v1beta, the named custom-priced /gemini deployment, and a cut OpenAI stream. * test(integration): price the gateway /gemini stream case as Gemini and cover client base_model and gateway host pricing The cost harness sends /gemini through GEMINI_API_BASE, which the pass-through now logs and bills as gemini, so the streamed case moves from the Vertex cases to the Gemini cases under the Gemini price. New spend tests cover a body base_model repricing only for keys allowed client pricing, a GEMINI_API_BASE gateway billing as Gemini in both modes, and a configured non-Google gateway picking the provider from the path. * fix(proxy): price pass-through Gemini by the route and ignore a non-string base_model The Vertex pass-through handler picked Gemini pricing from the URL path, so a configured pass-through to a Vertex gateway with a short path (no /projects/ segment) was billed as Gemini, and its Claude streams at $0. The provider now comes from the route: a call through /gemini or to generativelanguage.googleapis.com prices as Gemini, and everything else as Vertex, the same rule main used. Matching the GEMINI_API_BASE host was rejected because one gateway can serve both APIs, and the path rule was rejected because short gateway paths look like Gemini paths. A non-string base_model (a number, list or object) made the cost calculation fail and bill $0 once the stamped stream costs were gone. get_base_model_from_metadata now ignores it and falls back to the deployment's base_model, so the call bills the list price. A 400 was the other option, but the logger runs after the upstream answered. * test(integration): bill a non-string body base_model at the list price on every pass-through route * fix(proxy): bill a non-string model_info base_model at the list price and drop a non-object client model_info A key allowed client pricing can send metadata.model_info in a pass-through body. A non-string model_info.base_model made the cost lookup fail, and once the stamped stream and Gemini costs were gone those calls billed $0 on /gemini and on streamed /vertex_ai and /v1beta. The lookup now ignores a base_model that is not a string, the same rule the root base_model got, so the call bills the list price. A model_info that is not an object (a string or a list) broke several logging steps that read it as a mapping, so a non-streamed /gemini call fired no success callback and no pass-through call wrote a spend row. The pass-through route now leaves a non-object model_info out of the logging metadata, so the call bills the list price. A 400 was the other option, but it would turn calls that answer 200 today into errors, and guarding every reader of model_info would spread the same check over many files. * fix: reject a non-object model_info before the provider call again The previous commit made the base_model lookup return None for a model_info that is not an object. That lookup also runs during call setup on every router route and SDK entrypoint, where such a value used to fail before any provider request. With the early return the call went upstream, the provider billed it, and the request then failed in a later reader that still expects an object. On /v1/messages and /v1/responses the key was also charged for a response the client never got. The lookup again reads base_model from any truthy model_info, so a non-object value fails during setup the way it did before this PR. A non-string base_model inside an object still bills the list price, and the pass-through route still drops a non-object client model_info at intake, so pass-through calls keep billing the list price. * fix: match the Gemini API host exactly when picking the pass-through cost provider The provider check accepted any host ending in generativelanguage.googleapis.com, so a Vertex host such as vertex-generativelanguage.googleapis.com was priced as Gemini and lost the regional uplift. It now accepts that host or a subdomain of it. * test: pin that a subdomain of the Gemini API host keeps the Gemini price --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| audio_utils | ||
| llm_cost_calc | ||
| llm_response_utils | ||
| prompt_templates | ||
| specialty_caches | ||
| __init__.py | ||
| conftest.py | ||
| event_loop_lag.py | ||
| fake_secret_vault.py | ||
| messages_with_counts.py | ||
| test_agentic_followup_kwargs.py | ||
| test_anthropic_dedup_factory.py | ||
| test_api_route_to_call_types.py | ||
| test_app_crypto.py | ||
| test_audio_utils.py | ||
| test_aws_partition.py | ||
| test_bedrock_converse_dedup_factory.py | ||
| test_bug_report.py | ||
| test_chat_completion_agentic_loop.py | ||
| test_classifier_logging.py | ||
| test_cli_token_utils.py | ||
| test_cloud_storage_security.py | ||
| test_codestral_provider_routing.py | ||
| test_core_helpers.py | ||
| test_coroutine_checker.py | ||
| test_cost_discount.py | ||
| test_custom_logger_registry.py | ||
| test_dd_tracing.py | ||
| test_decisions_pricing.py | ||
| test_decode_special_tokens.py | ||
| test_default_encoding.py | ||
| test_dot_notation_indexing.py | ||
| test_duration_parser.py | ||
| test_env_utils.py | ||
| test_error_normalization.py | ||
| test_exception_mapping_utils.py | ||
| test_extract_base64_image.py | ||
| test_fallback_generalizations.py | ||
| test_fallback_utils.py | ||
| test_get_litellm_params.py | ||
| test_get_llm_provider_endpoint_match.py | ||
| test_get_llm_provider_logic.py | ||
| test_get_model_cost_map.py | ||
| test_get_supported_openai_params.py | ||
| test_health_check_helpers.py | ||
| test_hidden_params.py | ||
| test_image_handling.py | ||
| test_initialize_dynamic_callback_params.py | ||
| test_internal_call_metadata.py | ||
| test_json_fragment_accumulator.py | ||
| test_json_schema_validation.py | ||
| test_litellm_logging.py | ||
| test_llm_judge.py | ||
| test_llm_request_utils.py | ||
| test_logging_callback_manager.py | ||
| test_logging_utils.py | ||
| test_logging_worker.py | ||
| test_max_streaming_duration.py | ||
| test_model_param_helper.py | ||
| test_model_response_utils.py | ||
| test_moderation_standard_logging.py | ||
| test_oauth_token_exchange.py | ||
| test_optional_imports.py | ||
| test_path_utils.py | ||
| test_private_json.py | ||
| test_provider_affinity.py | ||
| test_provider_specific_headers.py | ||
| test_ptu_pricing.py | ||
| test_realtime_errors.py | ||
| test_realtime_streaming.py | ||
| test_redact_messages.py | ||
| test_request_timeout_resolver.py | ||
| test_retry_after_headers.py | ||
| test_safe_divide_seconds.py | ||
| test_safe_json_dumps.py | ||
| test_sensitive_data_masker.py | ||
| test_sentry_scrubbing.py | ||
| test_served_output_texts.py | ||
| test_stream_usage_logging.py | ||
| test_streaming_chunk_builder_cursor.py | ||
| test_streaming_chunk_builder_server_tool_use.py | ||
| test_streaming_chunk_builder_utils.py | ||
| test_streaming_handler.py | ||
| test_streaming_overhead.py | ||
| test_text_completion_conversion.py | ||
| test_thread_pool_executor.py | ||
| test_token_counter.py | ||
| test_token_counter_tool.py | ||
| test_token_counter_tool_data.py | ||
| test_tokenizer.py | ||
| test_tool_fields.py | ||
| test_tool_search_spend_logging.py | ||
| test_url_utils.py | ||
| test_xai_oauth_routing.py | ||