Operators can now set aws_session_tags on a Bedrock or SageMaker
deployment and every AssumeRole call carries them as STS session tags,
so trust policies gated on sts:TagSession admit the session and
CloudTrail and cost reports see the tags. Tags are validated up front,
sorted into the credential cache key so tag order does not fork
sessions, stripped from invoke and embedding request bodies, and
blocked from client request bodies like the other AWS identity params.
Based on #34073.
Co-authored-by: Satya Yedida <222552552+satya-rubrik@users.noreply.github.com>
Bedrock Guardrails resolved credentials and signed inline on the loop in
its three async paths, Titan embeddings signed each batch item inline in
the async loop, and Bedrock Mantle requests slipped past the off-loop
gate because BedrockMantleAuthMixin composes a BaseAWSLLM instead of
inheriting from it. Introduce the SignsRequestsWithAWS marker that both
BaseAWSLLM and the Mantle mixin carry so sign_request_off_loop_if_aws
covers Mantle, and move the guardrail and embedding signing into
asyncio.to_thread. Every new test fails at the previous tip.
Marengo 3.0 requests now get a 400 naming any textTruncate, lengthSec,
useFixedLengthSec, or minClipSec parameter, and any video or audio option
sent with a text, image, text_image, or multi_input request, instead of
silently dropping them. drop_params (global, per deployment, or per
request) drops them instead. Pydantic validation errors name the field
and the reason, and the 3.0 marker is the exact "marengo-embed-3-" model
id segment.
AWS prices Marengo 2.7 and 3.0 text and image embeddings per request, never per
token, and their responses carry no token count. The old transform estimated
prompt tokens from the vector length, which billed a text request at 128 tokens
times the per-token rate (0.00896 instead of 0.00007). Marengo responses now
report zero tokens with query_count and image_count derived from the request
batch, and all six Marengo cost-map entries price per request (with the video
and audio per-second and per-image rates on the base entries). query_count is a
new prompt_tokens_details field wired to input_cost_per_query in the cost
calculator.
A deployment authenticating with api_key or AWS_BEARER_TOKEN_BEDROCK still ran
boto3's credential chain before every call, so an unloadable default profile
(a login_session profile without botocore[crt]) made Converse, embeddings,
image generation, image edit, and the Bedrock guardrail hook fail with
MissingDependencyException even though the bearer token alone signs the
request. The chain now runs only when no bearer token is configured
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
Twenty-three tests across eleven files opened with litellm.set_verbose = True
and never put it back, so the flag stayed on for everything that ran after them
in the same process. None of those files read the output it produces: no
caplog, no capsys, no assertion on a log line, so the flag was left over from
debugging. Deleting it beats restoring it, since restoring keeps the noise.
Ten of the eleven stop leaving the flag on. test_volcengine_embedding.py still
ends with it set, from something it exercises rather than from the test itself,
which is worth its own look.
Fifteen tests opened with litellm.set_verbose = True and never put it back, so
the flag stayed on for everything that ran after them in the same process.
Nothing in the file reads the output it produces: there is no caplog, no capsys
and no assertion on a log line, so the flag was left over from debugging.
Deleting it beats restoring it, since restoring keeps the noise.
* test: run the 30 test files stranded in the second mirror
tests/litellm sat beside tests/test_litellm, which is the mirror the repo
convention names, and no job collected it. The allowlist called the directory
unresolved and assumed it was a duplicate. It is not: 30 of its 34 files have no
counterpart in the real mirror, so they are tests nobody has run since they were
written, not copies of tests that run elsewhere.
Moving them in is byte-identical, and it is what makes them run. Every one is
now claimed by a shard's test-path rather than by an allowlist entry, and the
216 tests they hold pass. Directories that needed to become packages did, since
several files are named test_transformation.py and pytest cannot import two of
those from non-package directories in one session.
Never running is why three assertions had drifted away from the code:
* nvidia.nemotron-super-3-120b max_output_tokens, 32000 -> 32768
* sambanova/MiniMax-M2.7 max_input_tokens, 204800 -> 196608
* the Vertex text-to-speech handler moved from data= to json=, so the test
reads the decoded body off the json kwarg instead of parsing the data one
The first two follow model_prices_and_context_window.json, which the catalog
sync keeps current; the third follows the handler. In all three the test was the
stale side.
The lint workflow ran test_no_hardcoded_secrets.py by path and now points at the
new one.
Four files stay behind. Each shares a filename with a live test whose contents
are disjoint from it, so landing those means merging test bodies, which is a
content review rather than a move. The allowlist entry now names those four and
records how many tests each would bring, in place of calling the whole
directory unresolved.
* fix(ci): keep the secret scan out of the mirror's conftest
The secret-scan job runs pytest under uv run --no-project, so its environment
holds pytest and nothing else. That worked while the file sat in tests/litellm,
which has no conftest, and broke the moment it moved into tests/test_litellm,
whose conftest imports litellm on collection: ModuleNotFoundError: No module
named 'dotenv', before a single test ran.
The file is a repo-wide static scan that imports only base64, os, re and pytest,
so it belongs with the other repo-wide checks in tests/code_coverage_tests,
which has no conftest, rather than in the package mirror. Installing the full
dependency set into a 15-second job to satisfy a conftest it does not use would
be the wrong trade.
Verified with the job's exact command:
uv run --no-project --with 'pytest==9.0.2' pytest \
tests/code_coverage_tests/test_no_hardcoded_secrets.py -q
1 passed in 0.47s
* fix(bedrock/cohere): wrap embedding_types as list in map_openai_params
Bedrock Cohere expects embedding_types as a JSON array but
encoding_format was passed through as a raw string, causing:
Malformed input request: #/embedding_types: expected type: JSONArray, found: String
* test(bedrock/cohere): assert embedding_types is sent as JSON array
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
Bedrock multimodal embedding models (Titan and Nova) were being costed
using the per-token text rate instead of the correct flat per-image rate
($0.00006/image). The pricing data was correct but never applied because
image_count was never populated in prompt_tokens_details.
Pass batch_data to Titan/Nova response transformers so they can count
image inputs and set PromptTokensDetailsWrapper(image_count=N) on Usage,
mirroring the existing Vertex AI pattern from PR #9623. Also fix the
text_tokens fallback in generic_cost_per_token to not override
text_tokens=0 when image_count > 0 (image-only requests).
- Test encoding_format='float' parameter mapping and response handling
- Test encoding_format='base64' parameter mapping to binary format
- Verify parameter transformation and response processing
- Mock AWS API responses for both float and binary formats
- Ensure OpenAI compatibility with new encoding_format support
* add support of bearer token for bedrock integration
* fix linting issue
* fix type checking issue
* reoder arguments to address type checking issue
* switch to use get_secret_str to fetch env variable
Co-authored-by: 0x-fang <fanggong@amazon.com>
