Cache rates were derived as ratios of the dollar input rate (1.25x write,
0.1x read) while input and output derive from the published DBU table
times $0.070. Databricks publishes cache write and cache read DBU per
model, and those are not exact multiples of the input DBU, so the two
rules disagreed by up to 0.1 percent.
Rewrites 43 cache literals across 31 entries to published_cache_DBU x
$0.070. Skips databricks-gemini-2-5-pro and databricks-gemini-2-5-flash,
whose input and output rates predate the current table by a 1.25x
increase; their cache rates stay tied to their own input rate so each
entry remains internally consistent.
Replaces the ratio assertions with a test pinning the absolute published
DBU figures for the five new models, and adds a test pinning the
older-vintage exception. Corrects the metadata note on the five new
entries, which claimed the reference-only *_dbu_cost_per_token fields
drive cost calculation.
The shared cost calculator treats a missing cache rate as free, so routing
Databricks through it billed cached tokens at zero on the 14 entries that
publish no cache pricing. On a 10,000 token prompt with 8,000 cache reads
that is $0.0010000 against the correct $0.0050001, a fivefold undercharge.
Those entries now declare cache rates equal to their input rate, which is
what a model with no caching discount should charge, and a test pins every
priced Databricks entry to declaring cache rates so no future entry can
regress into it.
Also repoints the provider-neutral generalization test off an id the new
Opus 5 entry now shadows, adds backup-to-main parity tests for the five new
entries, pins that Databricks Claude is never auto-injected with cache
control despite reporting caching support, and trims the Sonnet 5 pricing
note, which is served on an unauthenticated route.
* test: add regression coverage for twelve closed issues
Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:
- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges
Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.
* fix: stop the new regression tests polluting and tripping over shared global state
Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.
test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.
The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
The introductory DBU rates run through 2026-08-31 and pricing carries no
expiry date, so a static introductory entry would undercharge by a third
from September 1 and let spend outrun enforced budgets. Ship the standard
rates, which match Sonnet 4.5 and 4.6, and keep the introductory numbers
in the entry notes.
Also give the new cost calculator tests full type annotations.
Databricks cost calculation multiplied every prompt token by the input rate, so
a cache read cost the same as an uncached token. Route it through
generic_cost_per_token, which already understands cache reads and cache writes,
and add the cache rates the registry was missing.
Adds Claude Opus 4.7, Opus 4.8, Opus 5, Sonnet 5 and Fable 5 on Databricks.
A reasoning item id is not an Anthropic signature. Passing it off as one got the
block replayed to Anthropic and Bedrock as if it were real, and every backend that
verifies signatures rejected the turn. Thinking blocks now come back unsigned, and
the streaming path no longer emits a signature_delta for them.
Azure AI Foundry, Fireworks, and vLLM reject unknown message fields, so they now
strip reasoning_content alongside thinking_blocks the way Mistral already did.
The thinking-block helpers take ChatCompletionThinkingBlock and
ChatCompletionRedactedThinkingBlock instead of loose mappings.
The experimental /v1/messages adapters lost prior-turn reasoning three
different ways once the request left for an OpenAI-shaped backend.
On the Responses path, thinking blocks were flattened into output_text
inside the assistant message, so the model read its own private reasoning
back as visible prose and no reasoning item was ever sent. They now become
Responses reasoning input items, grouped by signature so summary parts that
arrived as one item go back as one item. The response direction stops
hardcoding signature=None and carries the reasoning item id, which is what
lets the next turn regroup them; the streaming wrapper emits the matching
signature_delta.
On the chat completions path the adapter attached thinking_blocks but never
set reasoning_content, so Moonshot and DeepSeek substituted a single-space
placeholder and other providers sent nothing. It is now derived from the
thinking blocks.
With use_chat_completions_url_for_anthropic_messages and a model that itself
bridges to /v1/responses, the assistant message was dropped whole: reasoning,
text, and all. That branch now emits the reasoning items and the message
content alongside the tool calls.
Fixes#24985
* fix(otel): emit LLM Call spans for speech, image, moderation, ocr and transcription
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): log the image request before caller headers are merged in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): map non-chat routes to standard genai operations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): stop caller image headers aliasing the logged request body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep resolved api_base in async moderation pre_call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): log resolved client endpoint for speech pre_call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(otel): justify mutable request payloads in speech and image pre_call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep caller headers out of the logged speech request body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
Twenty-three tests across eleven files opened with litellm.set_verbose = True
and never put it back, so the flag stayed on for everything that ran after them
in the same process. None of those files read the output it produces: no
caplog, no capsys, no assertion on a log line, so the flag was left over from
debugging. Deleting it beats restoring it, since restoring keeps the noise.
Ten of the eleven stop leaving the flag on. test_volcengine_embedding.py still
ends with it set, from something it exercises rather than from the test itself,
which is worth its own look.
Fifteen tests opened with litellm.set_verbose = True and never put it back, so
the flag stayed on for everything that ran after them in the same process.
Nothing in the file reads the output it produces: there is no caplog, no capsys
and no assertion on a log line, so the flag was left over from debugging.
Deleting it beats restoring it, since restoring keeps the noise.
Nine tests in test_http_handler.py captured litellm.disable_aiohttp_transport,
force_ipv4, ssl_ecdh_curve or the request_timeout pair, wrapped their whole body
in a try, and put the value back in a finally. monkeypatch.setattr does all of
that, so the captures, the try and the finally go away and the bodies lose a
level of indentation. The class-scoped restore_request_timeout fixture existed
only for that same bookkeeping and goes with them.
litellm.in_memory_llm_clients_cache is left alone on purpose: the eviction tests
assert a handler is garbage collected, and monkeypatch holds the replaced value
alive until teardown, which keeps the weakref they check from clearing.
test_zai_provider.py set LITELLM_LOCAL_MODEL_COST_MAP and litellm.model_cost
directly and never put them back, so every test that ran after it in the same
process saw a local cost map instead of the real one. The two respx tests did
the same to litellm.disable_aiohttp_transport with no restore at all.
Both now go through monkeypatch, which restores on teardown including when the
test fails. The cost-map setup moves into a fixture requested by exactly the
five tests that read the cost map.
* test: use monkeypatch.setenv for env writes in tests/test_litellm
`os.environ["X"] = v` inside a test leaks the value into every test that runs
after it in the same worker, so ordering decides the result. 262 of those
writes across 40 files now go through pytest's `monkeypatch` fixture, which
restores the previous value at teardown.
The rewrite skips any test that a mock.patch-family decorator wraps, any test
with defaulted positional parameters, any test whose own name is called
directly elsewhere, and rebinds nothing inside nested defs, because in each of
those cases appending a fixture parameter changes what pytest or mock binds.
Ratchets the TQ004 ceiling from 768 to 506.
* fix(test): delete the key through monkeypatch instead of popping it first
Five tests popped a key straight out of `os.environ`, ran, then restored it with
`monkeypatch.setenv`. By the time monkeypatch saw the name it was already gone,
so it recorded "absent" as the value to go back to and deleted the key at
teardown. On a worker that inherited a real `RESEND_API_KEY`, `SENDGRID_API_KEY`,
`UI_PASSWORD`, `LITELLM_SALT_KEY` or `OPENAI_API_KEY`, every test after the first
one ran without it.
`monkeypatch.delenv(..., raising=False)` removes the key and restores whatever
was there, so the try/finally the manual restore needed goes with it.
* chore(test): leave the two cost-calc files to the PR that rewrites them fully
Both files are also in #37815, which converts the module-global writes as well
as the env writes and folds them into one fixture. Two PRs rewriting the same
lines differently is a conflict nobody benefits from resolving, so this one
drops back to staging on those two and keeps the other 39.
TQ004 clears 200 here instead of 275; the rest moves with #37815.
Twenty tests in test_request_metadata.py assigned the global directly and
leaned on an autouse fixture to put it back afterwards. monkeypatch.setattr
does both jobs at the point of use, so each test now says what it sets and the
fixture that existed only to undo them goes away.
The Responses WebSocket path, the pre-call deployment hook and the
per-frame project quota hook are all selected by small predicates that
nothing asserted directly. Mutating those four decisions left 4 of 6
mutants alive against the mapped test file.
Cover them at the boundary: the rust WebSocket path needs both the
openai provider and the rust flag, a plain CustomLogger must not
advertise a pre-call deployment hook while an overriding or inheriting
one must, and only callbacks that actually expose a callable
enforce_project_io_token_quota_for_frame reach the WebSocket loop.
Kill rate on those four decisions goes 2/6 -> 6/6; the file goes 66 -> 75
passing.
get_llm_provider ran in the messages handler and again inside completion,
so a provider/vendor/model id lost its vendor segment and reached upstream
bare. Pass the caller's unresolved model down the bridge instead, and move
the responses marker into the canonical provider/responses/model slot.
Reporting stays provider-local on both bridges: message_start names the id
the provider itself knows, through a shared local_model_name helper.
Fixes#37716
Four more ruff rules for code the test suite runs but never checks. F601 is the
one that paid: the duplicate key it flagged in a get_form_data fixture was the
mock reproducing the production bug fixed in the previous commit.
B025 removed two unreachable handlers, one of them a pytest.skip shadowed by an
earlier `pass`, so an upstream Vertex flake reported green having asserted
nothing. F632 turned an `is ""` identity check, which passes only on CPython
interning, into the `== ""` it meant. B023 fixed three closures over loop
variables, all latent today but one iteration-order change away from checking the
last case N times.
* fix(anthropic_messages): gate sampling params on /v1/messages like /chat/completions
/v1/messages forwarded temperature/top_p/top_k raw to models that removed
sampling params (supports_sampling_params: false — Claude 4.7+/Fable 5),
producing provider 400s that router fallbacks mask as silent model
downgrades. The chat path already gates these via
AnthropicModelInfo._apply_sampling_param; reuse it in
get_requested_anthropic_messages_optional_param so both endpoints agree:
drop under drop_params, else raise the clean client-side 400.
Fixes#35053
* test(anthropic): drive new sampling-param tests off the kwarg, not the global
The five tests added here set `litellm.drop_params = True` under a manual
try/finally. That trips TQ005 (module-global mutation, 10 new violations
over the ceiling) and it leaks process-wide if the finally is ever
skipped, which is what the save/restore conftest exists to paper over.
`get_requested_anthropic_messages_optional_param` already takes
`drop_params` as a kwarg, and that is the path /v1/messages actually
uses, so pass it directly. `monkeypatch.setattr` pins the global to
False so each test proves the per-request flag alone is sufficient and
cannot pass on a leaked global.
Verified: TQ gate clean, all 10 tests pass, and the 3 that assert the
new gating still fail with the fix in utils.py reverted.
---------
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
add_provider_specific_headers_to_request tagged the client's Authorization header
with the same provider list as anthropic-beta and anthropic-version, so an
sk-ant-oat subscription token was sent to AWS Bedrock and Google Vertex AI as
well. On Bedrock it replaced the SigV4 signature, or the deployment's own API
key, and AWS answered 403 "Invalid API Key format". On Vertex it went out as a
second Authorization header next to the Google one and Google answered 401
ACCESS_TOKEN_TYPE_UNSUPPORTED.
The credential and those API headers need different scopes, so a request can now
carry more than one ProviderSpecificHeader entry. The API headers keep the
provider list they already had and the credential gets its own entry scoped to
anthropic alone. get_provider_specific_headers takes either a single entry or a
sequence and merges only the entries whose provider list matches, so callers that
pass one entry keep working unchanged.
Bedrock SigV4 signing and the deliberate extra_headers Authorization pass-through
in _sign_request are left alone.
Routing rerank through get_request_headers also picked up its
AWS_BEARER_TOKEN_BEDROCK branch. Bedrock API keys are only valid for
Bedrock and Bedrock Runtime actions, not for Agents for Amazon Bedrock
Runtime ones, and rerank is served by bedrock-agent-runtime, so AWS
rejects a bearer-signed rerank call. Opt the rerank handler out of the
bearer path so it keeps signing with SigV4.
A test that asserts on the error inside its own except block passes when the
call stops raising, because nothing runs the handler. That is the exact case
the test exists to catch, so the regression lands green.
Rewrites all 111 such blocks into pytest.raises, which fails when the call
succeeds, and selects PT017 in ruff-tests.toml so no new one lands.
* fix(vertex_ai): convert messages to contents in gemini count_tokens
acount_tokens passed contents=None to the Vertex Gemini countTokens
endpoint when called with messages=, causing a silent zero token count.
The Gemini branch of VertexAITokenCounter.count_tokens never read the
messages parameter, so the request body was {"contents": null}, which
Vertex accepts with HTTP 200 and no totalTokens field.
Convert messages to Gemini contents format using the existing
_gemini_convert_messages_with_history helper when contents is None.
Treat a response without totalTokens as a failure so the caller falls
back to local token counting instead of returning a silent zero.
Fixes#36921
* style: apply ruff format to common_utils.py
Resolves lint CI failure on PR #36981.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: suppress LIT002 on messages fallback for Gemini token counter
Adds `# mutable-ok:` suppression on the `messages or []` fallback passed
to `_gemini_convert_messages_with_history`. The [] is a None-fallback;
the helper signature requires list[AllMessageValues], so a tuple would
violate the type contract. Resolves type-discipline-budget CI failure
on PR #36981.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: suppress reportPrivateUsage on _gemini_convert_messages_with_history import
Adds `# pyright: ignore[reportPrivateUsage]` on the import of the
shared `_gemini_convert_messages_with_history` helper. The function is
already used by gemini/chat, context_caching, and
vertex_and_google_ai_studio_gemini; reusing it here avoids duplicating
the OpenAI-to-Gemini message conversion. Resolves basedpyright budget
CI failure on PR #36981.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
Per-model budgets were three separate things pretending to be one. The
enforcement check, the post-call increment and the info endpoints each derived
their own cache key, so a budget could refuse traffic at 429 while /key/info
reported zero usage, and a Bedrock model id never matched a budget keyed on the
bare family name. /user/new echoed a model_max_budget back and stored an empty
dict, and nothing enforced a user-scoped per-model budget at all.
One owner now builds the counter key from the configured budget model, and
enforcement, the increment and the info endpoints all read it. Bedrock ids
resolve through the model-cost map. Auth carries the user's budget onto the
token on every branch that reaches the spend hook, including JWT and
auto-registration. Native passthrough attaches the three budget metadata keys
its StandardLoggingUserAPIKeyMetadata does not carry, so /anthropic/... and
/bedrock/... traffic is counted and capped like /v1/chat/completions.
The dashboard gains the per-model budget editor it never had, on the key create,
key edit and internal-user edit forms. It is read-only without an enterprise
license, matching the write gate the proxy already enforces, and an untouched
budget is left out of an update so an unrelated edit cannot trip that gate.
The editor hydrates from either BudgetConfig spelling, since model_max_budget is
a plain dict that the proxy stores exactly as the client sent it, and it carries
through the fields it does not model. Without both, editing one model would drop
another model row entirely and silently discard its tpm_limit and rpm_limit.
/user/info refreshes its local copy of the user field by field after a save, so
model_max_budget joins that list. Left out, a saved cap read back as the old one
when the form was reopened, and clearing the row to recover would then wipe the
value that had actually persisted.
A zero-dollar cap is the strictest limit expressible, not the absence of one,
so it is enforced rather than skipped on falsiness, spend exactly at the cap is
refused the way every sibling budget check already refuses it, and a counter
that was never written reads as zero spend rather than as unknown. The usage
endpoints read every counter in one batched lookup, so a large model_max_budget
cannot fan out into one concurrent cache call per configured model.
Every auth path honours the same zero-cost skip flag, so none of them can refuse
a free request that another serves. The custom-auth helper gains the flag it
never had, which also changes its pre-existing key and end-user checks.
The compaction summary gate checks the user scope alongside the key and end-user
ones. This file propagates all three budgets into the summary subrequest, so
enforcing only two let compaction increment a counter it could not be refused by.
Custom auth attaches the user's budget to the token unconditionally, since the
post-call spend hook reads it there: gating the attach on the same condition as
enforcement left the counter uncharged whenever the request was not itself
enforceable. An entry that will not validate is skipped rather than raised on,
so one malformed scope cannot abort every other scope's increment or turn a
config typo into a 500.
The edit forms re-seed the budget editor when a different key or user is loaded.
Its rows are seeded once and cannot re-read their own value prop, so without
this a save wrote the previously loaded record's budgets onto the current one.
Only the built-in provider pass-through routes carry the budget metadata.
get_model_from_request deliberately resolves no model for a user-defined
pass-through, since its body is forwarded verbatim and names an upstream model,
so attaching there would charge a counter nothing on that route can refuse.
GPT-5.6 Sol, Terra and Luna reached the bedrock-runtime data plane on
2026-08-17, separately from the existing bedrock-mantle path. On runtime
they are served only through cross-region inference profiles, so
bedrock/us.openai.gpt-5.6-* had no cost map entry and fell through to the
Invoke route, which rewrites the token cap to max_tokens and is rejected
as unsupported_parameter on both /v1/chat/completions and /v1/responses.
Register the Geo and Global profiles as bedrock_converse so routing
reaches Converse, which AWS documents and serves for these models, and
price each profile from its own published rate table. No bare key: the
control plane reports inferenceTypesSupported INFERENCE_PROFILE with no
on-demand throughput, so a bare id is not invocable.
Declare the published cache-read and cache-write rates. Bedrock rejects
an explicit cachePoint block for these models, so supports_prompt_caching
stays off, but it caches long prefixes implicitly and reports the cache
tokens in usage either way. Without the cost fields a cache-read turn
bills only its uncached tokens: measured against live Bedrock, a
15609-token cached prefix came to $0.000176 instead of $0.00876095.
Clients that resend a long prefix every turn are the worst affected.
Reasoning stays unadvertised. Converse rejects the Anthropic-shaped
thinking block LiteLLM sends for reasoning_effort; the shape these models
accept is additionalModelRequestFields {"reasoning": {"effort": ...}},
which needs a transform change tracked by #34105. Advertising it without
that change is what made the earlier attempt in #37307 fail.
Resolves the three conflicts against the JSON provider registry refactor. The
hardcoded api.scx.ai base-url branch in get_llm_provider_logic.py is dropped in
favour of the generic JSONProviderRegistry.get_by_base_url lookup, which reads
the same base_url and api_key_env from providers.json and additionally honours
an explicitly passed api_key. constants.py and types/utils.py keep both the
cognition and scx-ai entries added on either side.
Retypes the 54 highest-density reportAny/reportExplicitAny sources with real
types instead of shuffling the ceilings around: typed prisma table Protocols so
the untyped client surface stops at the query, local TypedDicts for JSON and
dict payloads, concrete chunk and logging types on the streaming and callback
surfaces, and 3-argument getattr with a Callable annotation where an SDK object
is genuinely duck-typed
No cast(), no type: ignore, no noqa, no suppression comments, and no new Any
annotations. Whole-tree basedpyright drops 1,941 errors with no rule rising
anywhere, and all three budget files are ratcheted so the cleared headroom
cannot silently grow back
Adds a GDC regression test pinning the named AttributeError that the typed
credential accessor now raises when with_gdch_audience is missing
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
sagemaker_chat never put X-Amzn-SageMaker-Inference-Component on the request, so any endpoint
backed by inference components answered 400 INFERENCE_COMPONENT_NAME_MISSING and the call never
reached the container. The legacy sagemaker provider has built that header from model_id since
#8889, and this brings the chat provider in line. It goes on in validate_environment, which runs
before the request is SigV4-signed, so the signature covers it
The request body also always named the endpoint rather than the served model, which containers
that validate the body's model answer with a 404. hf_model_name now becomes the body's model
when it is set, and endpoints that do not set it keep sending exactly what they send today
