* fix(vertex_ai): price streamed pass-through calls with the regional uplift
The streamed Vertex pass-through handler builds the response with
stream_chunk_builder, which stamps a cost into _hidden_params without the
vertex_location. The handler then prices the response again with the
location from the URL, but only into kwargs, and the logger prefers the
stamped cost. Streamed calls on a regional endpoint were billed at the
global rate.
Stamp the location-aware cost on the response in the shared
generateContent payload builder so the logger records it.
Fixes#44755
* style(vertex_ai): keep the response_cost assignment within the 120-char limit
* refactor(vertex_ai): set the passthrough response cost with set_hidden_param
Writing _hidden_params directly mutated a parameter and needed a pyright ignore, which the type discipline gate counts as LIT004 and LIT011. set_hidden_param is the shared helper for this.
* test(integration): check Vertex pass-through spend rows keep the regional uplift when streamed
* fix(proxy): price streamed Vertex pass-through spend like the non-streamed call
Drop the location-blind cost stream_chunk_builder stores instead of stamping a cost, so the logger prices streamed generateContent and rawPredict calls with the URL's location and the deployment's custom pricing, the same way it prices non-streamed calls
* fix(proxy): price custom-priced Vertex pass-through deployments at their own rates
The stamped response_cost from the generateContent logging helper is returned before the
logger checks custom pricing, so a deployment with custom input/output rates was billed at
the cost-map price. The helper now passes custom_pricing and the deployment id into
completion_cost, so the stamp holds the deployment rate for streamed and non-streamed calls,
and deployments without custom pricing keep the regional uplift.
* fix(proxy): bill Vertex /v1beta and interrupted pass-through streams by the deployment location and price
* fix(proxy): give the new pyright ignore a reason for the LIT004 gate
* test(integration): bill a custom-priced Vertex pass-through stream at the deployment price, cut-off streams included
* fix(proxy): let the logger price non-streamed /gemini pass-through calls so both modes bill the same
* fix(proxy): keep a client base_model out of pass-through pricing and price gateway Gemini streams as Gemini
A top-level base_model in a pass-through body now reaches the logging
params only for a key or team with allow_client_pricing_override, so a
plain key cannot pick a cheaper model's price once the logger prices
these calls. The test that pinned every litellm-owned body name into the
logging params now leaves base_model to its own key-gated test.
The Vertex handler now classifies a non-Google upstream host as Vertex
only for Vertex resource paths, so a /gemini stream through a
GEMINI_API_BASE gateway is priced as Gemini without the regional uplift.
Adds the audit's integration spend test for Claude on Vertex, /v1beta,
the named custom-priced /gemini deployment, and a cut OpenAI stream.
* test(integration): price the gateway /gemini stream case as Gemini and cover client base_model and gateway host pricing
The cost harness sends /gemini through GEMINI_API_BASE, which the pass-through now logs and bills as gemini, so the streamed case moves from the Vertex cases to the Gemini cases under the Gemini price. New spend tests cover a body base_model repricing only for keys allowed client pricing, a GEMINI_API_BASE gateway billing as Gemini in both modes, and a configured non-Google gateway picking the provider from the path.
* fix(proxy): price pass-through Gemini by the route and ignore a non-string base_model
The Vertex pass-through handler picked Gemini pricing from the URL path,
so a configured pass-through to a Vertex gateway with a short path (no
/projects/ segment) was billed as Gemini, and its Claude streams at $0.
The provider now comes from the route: a call through /gemini or to
generativelanguage.googleapis.com prices as Gemini, and everything else
as Vertex, the same rule main used. Matching the GEMINI_API_BASE host
was rejected because one gateway can serve both APIs, and the path rule
was rejected because short gateway paths look like Gemini paths.
A non-string base_model (a number, list or object) made the cost
calculation fail and bill $0 once the stamped stream costs were gone.
get_base_model_from_metadata now ignores it and falls back to the
deployment's base_model, so the call bills the list price. A 400 was the
other option, but the logger runs after the upstream answered.
* test(integration): bill a non-string body base_model at the list price on every pass-through route
* fix(proxy): bill a non-string model_info base_model at the list price and drop a non-object client model_info
A key allowed client pricing can send metadata.model_info in a pass-through
body. A non-string model_info.base_model made the cost lookup fail, and
once the stamped stream and Gemini costs were gone those calls billed $0
on /gemini and on streamed /vertex_ai and /v1beta. The lookup now ignores
a base_model that is not a string, the same rule the root base_model got,
so the call bills the list price.
A model_info that is not an object (a string or a list) broke several
logging steps that read it as a mapping, so a non-streamed /gemini call
fired no success callback and no pass-through call wrote a spend row. The
pass-through route now leaves a non-object model_info out of the logging
metadata, so the call bills the list price. A 400 was the other option,
but it would turn calls that answer 200 today into errors, and guarding
every reader of model_info would spread the same check over many files.
* fix: reject a non-object model_info before the provider call again
The previous commit made the base_model lookup return None for a
model_info that is not an object. That lookup also runs during call
setup on every router route and SDK entrypoint, where such a value used
to fail before any provider request. With the early return the call went
upstream, the provider billed it, and the request then failed in a later
reader that still expects an object. On /v1/messages and /v1/responses
the key was also charged for a response the client never got.
The lookup again reads base_model from any truthy model_info, so a
non-object value fails during setup the way it did before this PR. A
non-string base_model inside an object still bills the list price, and
the pass-through route still drops a non-object client model_info at
intake, so pass-through calls keep billing the list price.
* fix: match the Gemini API host exactly when picking the pass-through cost provider
The provider check accepted any host ending in
generativelanguage.googleapis.com, so a Vertex host such as
vertex-generativelanguage.googleapis.com was priced as Gemini and lost
the regional uplift. It now accepts that host or a subdomain of it.
* test: pin that a subdomain of the Gemini API host keeps the Gemini price
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
6762d65eb8