Every Bedrock and SageMaker call site hand-copied the same nine aws_* kwargs
into BaseAWSLLM.get_credentials, so each new auth param has to be threaded
into a dozen places and any site that misses one silently assumes the role
with the wrong parameters.
Introduce AwsAuthParams, a frozen pydantic model whose fields are exactly the
credential-shaped params get_credentials accepts, plus resolve_credentials on
BaseAWSLLM and pop_aws_auth_params for the call sites that must strip the keys
out of optional_params. Deriving AWS_AUTH_PARAM_KEYS from the model's fields
means the mirror list in common_utils can no longer drift from the struct.
Behavior is unchanged: the same values reach STS from the same call sites.
Dropping any one field from the resolver fails one of the new tests.
Claude-Session: https://claude.ai/code/session_01E6zsK1DBcXfbetkgX86fw2
A deployment with base_model set was priced from the region's own row once
completion_cost forwarded the response region into cost_per_token, which now
strips the provider prefix and finds bedrock/<region>/<base_model>. Explicit
pricing (base_model or custom pricing) suppresses the region for cost_per_token
the same way _select_model_name_for_cost_calc already does
Strengthens the serverless test to assert ResourceType is signed rather
than merely present, ties _uses_tls to the redis-py kwarg coercion so the
two cannot drift, locks the stripped-kwarg name tuple to the test's
expectations, and adds "off" and "True" sentinel flag values.
Renames the provider builder's parameter to redis_settings.
The ElastiCache IAM gate read `aws_iam_auth` and `ssl` with a helper that
only accepted the literal string "true", while the kwarg coercion that runs
later accepts "true", "1" and "yes". Type coercion happens after the gate, so
`REDIS_AWS_IAM_AUTH=1` silently skipped IAM auth and `REDIS_SSL=1` made the
"requires TLS" check fail closed on a connection that was in fact TLS. Both
helpers now share `_str_to_bool`.
AWS signs serverless cache tokens with an extra `ResourceType=ServerlessCache`
query parameter, so tokens minted for a serverless cache were rejected. Adds an
`aws_iam_serverless` setting (`REDIS_AWS_IAM_SERVERLESS`) that puts the
parameter into the signed URL, and lowercases the cache name because
ElastiCache lowercases it at creation time.
The comment documents that the raw Azure client id, tenant id and secret
are deliberately kept off the connect function, so this branch should
never have dropped it
Deferred annotation evaluation keeps the type-checking-only botocore
import off the runtime path, so the alias only reintroduced typing.Any,
which the strict ruff budget now bans
get and post already take follow_redirects. put built the request and sent it
with the client default, so a caller uploading to a URL it did not choose had
no way to refuse a redirect. Same plumbing as the other two methods.
OpenInferenceMapper spelled every captured prompt and response message out as
two indexed attributes with no bound. A few dozen turns overran the OTel SDK's
128-attribute span limit, which evicts oldest first, so the gen_ai.* model,
provider, usage, cost and finish reason written before it were what got
dropped. Both directions now share one MAX_MESSAGE_ATTRS_PER_SPAN ceiling, the
response keeps at least half of it, and input.value / output.value still carry
the complete conversation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(prometheus): bucket latency by input sequence length
* style: format startup resolver call
* fix(prometheus): handle unknown input lengths
* fix(prometheus): preserve disabled custom input length labels
* test(prometheus): seed startup snapshot in mocked logger test
* fix(proxy): preserve database setting types during startup
* fix(prometheus): distinguish missing usage and preserve config persistence
Keep quoted database-storage config values intact for legacy persistence readers while using a local boolean for early callback discovery. Distinguish absent provider usage from an explicitly reported zero when labeling latency metrics.
* fix(proxy): normalize input length flag from secret managers
* fix(prometheus): isolate input buckets and preserve missing usage
Keep built-in buckets on latency histograms, preserve unrelated custom labels, and classify raw incomplete usage and upstream total-only headers as unknown. Cover count conservation, failure callbacks, explicit zero, startup snapshots, and direct caller compatibility. Drop earlier branch budget changes.
* fix(proxy): defer Prometheus alerting until stored settings load
Reuse successful startup storage resolution and preserve callback deduplication across alerting reloads.
* test(prometheus): restore input length flag between tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "test(prometheus): restore input length flag between tests"
This reverts commit f302e0b7bd.
* refactor(prometheus): make input length flag config/env only
Drop the Admin UI General Settings row, the safe DB override entry, the startup reorder that loaded DB litellm_settings before Prometheus callbacks, and the alerting-only Prometheus path. The flag now behaves like prometheus_emit_stream_label: litellm_settings in config.yaml or an os.environ reference, applied on restart.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A 200 body that is JSON but lacks a boolean allowed was recorded as a Defender
Block and rejected with 400. It is malformed, so it now routes through the
same unreachable_fallback handling as non-JSON and non-object bodies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
RefreshableCredentials (IMDS roles) can refresh between the access key, secret and token reads SigV4 performs, producing a mixed-generation signature that S3 rejects with 403 and the log is dropped. Snapshot the credentials with get_frozen_credentials before signing, treat 403 like 500/503 in the upload retry loop, and fetch credentials plus sign again on every attempt in both the async and sync upload paths. Tests load a real botocore credential_process profile and fake only the HTTP boundary with httpx.MockTransport
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound concurrent key and spend-counter DB lookups to stop prisma pool thrash
A cache-miss burst fanned every get_key_object DB fallback and SpendCounterReseed
point lookup into the prisma query-engine httpx pool at once. httpcore's request
assignment is O(queued x connections) per event, so the event loop spent most of its
time in pool bookkeeping and the logging worker's 20s wait_for tripped. Callers now
wait on a small shared semaphore (PROXY_DB_LOOKUP_MAX_CONCURRENCY, default 25)
instead of queueing inside httpcore
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the in-flight counting prisma fake
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop module docstring from db_lookup_gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the in-flight counting table fake
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(openai): drop tool schema regex patterns OpenAI's validator cannot compile
OpenAI validates function tool parameters with jsonschema's format checker,
which compiles every pattern with Python re. Claude Code's Artifact tool ships
an ECMA-262 pattern with \p{..} Unicode property escapes, so any OpenAI target
behind /v1/messages, /v1/responses or /v1/chat/completions 400s with
"Invalid schema for function 'Artifact': '...' is not a 'regex'" for every
model family. Drop only the patterns Python re rejects, keep the rest, at the
same seams that already flatten top-level combinators.
* fix(openai): walk only schema positions, iteratively, and drop regexes for every openai deployment
Review round: the regex sanitizer now walks JSON Schema applicator positions
only (properties, items, prefixItems, combinators, $defs, additionalProperties
and the rest), so a pattern key inside default, examples, const or a vendor
extension is data and stays. It also drops patternProperties keys Python re
cannot compile, which OpenAI checks the same way. The walk is level-order and
rebuilt deepest level first instead of recursive, so the code-quality recursion
gate passes and there is no depth cap below what a JSON parser admits. On the
chat wire an openai deployment with a custom api_base now drops such regexes
too, since that base is usually a proxy in front of the same validator, while
the lossier combinator flattening stays limited to api.openai.com hosts.
