litellm_request_total_latency_metric's start_time is set inside
common_processing_pre_call_logic, which only runs after user_api_key_auth
has already succeeded, so the metric silently excluded authentication and
pre-call setup time despite being documented as total request latency. The
sibling litellm_request_queue_time_seconds metric had the same problem:
its arrival_time was captured after auth too, despite its own comment
claiming to track when the request arrived at the proxy.
request.state.litellm_received_at is now stamped unconditionally at the
very first line of user_api_key_auth (previously only when OTEL was
configured), giving a timestamp that precedes all auth work. Both metrics
now derive from it: queue_time_seconds genuinely spans arrival through the
start of pre-call processing, and the total-latency metric adds that
queue time on top of its existing start/end window so it becomes true
end-to-end latency.
queue_time_seconds ends exactly at start_time rather than a separately
captured timestamp, so its window and the total-latency window share a
boundary instead of overlapping and double-counting a few lines of setup
work on every request.
The ceiling used to go through `int(... or 3)`, so anything `int()` accepted
worked. Tightening the new shared validator to `isinstance(int)` turned a
config that boots today into a proxy that refuses to start, because
`max_agentic_loops: os.environ/MAX_AGENTIC_LOOPS` is resolved to a string
before it reaches either check, and a YAML-quoted "5" is a string too.
Accept ints, integral floats, and strings that parse to a whole number. Keep
refusing bools, fractional floats, words, and anything below 1.
* fix(otel): emit LLM Call spans for speech, image, moderation, ocr and transcription
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): log the image request before caller headers are merged in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): map non-chat routes to standard genai operations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): stop caller image headers aliasing the logged request body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep resolved api_base in async moderation pre_call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): log resolved client endpoint for speech pre_call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(otel): justify mutable request payloads in speech and image pre_call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep caller headers out of the logged speech request body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The ceiling was only checked at the feature level, on
litellm_settings.websearch_interception_params. The per-deployment
litellm_params.max_agentic_loops, which wins over it, went straight into
int(kwargs.get("max_agentic_loops", 3) or 3), so a 0 was swallowed by the
falsy fallback and read as the default 3. Asking for the tightest ceiling
handed you the loosest one. A non-integer booted the proxy and then failed
every request to that model with "invalid literal for int() with base 10".
Both settings now share one validator, which names the field it rejected,
and the per-deployment value is checked while the model list is read at
startup so a bad value stops the proxy rather than surfacing per request.
The check sits in load_config rather than on LiteLLM_Params because the
proxy builds its router with ignore_invalid_deployments=True, where a
validation error drops the deployment silently instead of refusing to
start. This is the same placement the complexity_router_config plugin
check already uses.
Chat completions read the same key through a separate path that turned 0
into 1 and true into a ceiling of 1, so it now shares the validator too
and the key means one thing on both surfaces.
A capped turn on a streaming request is rebuilt into SSE by
FakeAnthropicMessagesStreamIterator. It emitted content_block_stop for
every block but content_block_start only for text, thinking,
redacted_thinking and tool_use, so a web search turn's server_tool_use
and web_search_tool_result blocks produced stops with no matching start.
Anthropic's SDK accumulator appends on content_block_start and then
indexes content[event.index] on content_block_delta, so the orphan stops
shifted every later index and client.messages.stream() raised IndexError
on the text block. Unknown block types now pass through with a start of
their own, which keeps position equal to index.
Also corrects two claims that said no current caller reaches the loop
with stream=True. AgenticStreamingIterator does, and it keeps raising,
because its events are already on the wire.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
Fifteen tests assigned litellm.callbacks directly and left the conftest global
snapshot to clean up after them. monkeypatch.setattr restores it as part of the
test, so the file no longer depends on that safety net to stay isolated.
Ten tests set litellm.s3_callback_params by hand. Four of them reset it to None
on the last line of the test body, which only runs when the test passes; the
other six wrap the body in try/finally to put the old value back. Raising inside
test_s3_verify_false_handling on the current file leaves the whole callback
config, bucket, endpoint and keys, set in the process for whatever runs next.
monkeypatch.setattr covers both shapes and restores on failure, so the 28 TQ005
violations and the try/finally scaffolding come out together.
51 tests pass, and the wider tests/test_litellm/integrations tree is unchanged.
The five TQ002 mock-echo tests in this file are left alone; those need a
judgement about what S3 logging should assert, not a mechanical sweep.
Both datadog test files hand-roll what monkeypatch.setenv already does: read the
old value, write the test value, put the old one back on the way out. The cost
management fixture checks the old value for truthiness rather than for None, so
an operator running the suite with DD_API_KEY set to the empty string gets it
deleted rather than restored. Starting from DD_API_KEY="" and running test_init
leaves it None on the current file, and "" after this.
13 raw os.environ writes become monkeypatch.setenv, the two fixtures stop being
yield fixtures because there is nothing left to do on the way out, and the now
unused os import goes with them.
27 tests pass across the two files, 88 across tests/test_litellm/integrations/datadog.
* test: use monkeypatch.setenv for env writes in tests/test_litellm
`os.environ["X"] = v` inside a test leaks the value into every test that runs
after it in the same worker, so ordering decides the result. 262 of those
writes across 40 files now go through pytest's `monkeypatch` fixture, which
restores the previous value at teardown.
The rewrite skips any test that a mock.patch-family decorator wraps, any test
with defaulted positional parameters, any test whose own name is called
directly elsewhere, and rebinds nothing inside nested defs, because in each of
those cases appending a fixture parameter changes what pytest or mock binds.
Ratchets the TQ004 ceiling from 768 to 506.
* fix(test): delete the key through monkeypatch instead of popping it first
Five tests popped a key straight out of `os.environ`, ran, then restored it with
`monkeypatch.setenv`. By the time monkeypatch saw the name it was already gone,
so it recorded "absent" as the value to go back to and deleted the key at
teardown. On a worker that inherited a real `RESEND_API_KEY`, `SENDGRID_API_KEY`,
`UI_PASSWORD`, `LITELLM_SALT_KEY` or `OPENAI_API_KEY`, every test after the first
one ran without it.
`monkeypatch.delenv(..., raising=False)` removes the key and restores whatever
was there, so the try/finally the manual restore needed goes with it.
* chore(test): leave the two cost-calc files to the PR that rewrites them fully
Both files are also in #37815, which converts the module-global writes as well
as the env writes and folds them into one fixture. Two PRs rewriting the same
lines differently is a conflict nobody benefits from resolving, so this one
drops back to staging on those two and keeps the other 39.
TQ004 clears 200 here instead of 275; the rest moves with #37815.
When the bounded loop cap or the repeated tool-call fingerprint guard refused a
rerun, the raise escaped the parent agentic frame and the client got the raw model
turn back: HTTP 200 carrying an unresolved tool_use block for the internal
litellm_web_search tool and stop_reason "tool_use". The client never declared that
tool, so it had no way to answer it and the conversation could not continue
The safety check now raises AgenticLoopSafetyError, a ValueError subclass, and
_call_agentic_completion_hooks catches it and returns a finalized response: the
blocks belonging to the refused tool calls are dropped, and stop_reason is closed
out to end_turn when nothing the client declared is still waiting. Refused blocks
are matched by the ids and names of the tool calls the rail refused rather than by
hardcoding the web search tool name
Only the non-streaming anthropic messages path ends the turn this way. A streaming
caller has already sent the original message by the time the hooks run, so a
finalized turn would arrive as a second message rather than replace the first, and
the responses surface carries a pydantic model this finalizer does not rewrite.
Both keep raising, exactly as they did before
Also adds max_agentic_loops to websearch_interception_params so the ceiling can be
set once for the whole feature. A per deployment litellm_params.max_agentic_loops
still wins over it, and the field stays on the proxy's untrusted root list so a
client cannot raise its own ceiling
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
S3SigV4Auth now signs these requests, which fixes an object key containing a space.
The path itself is still produced by requests.Request(...).prepare(), so it is not
percent-encoded the way S3 canonicalizes it and three shapes still misbehave.
"+" and "&" are left literal where S3 encodes them as %2B and %26, so the canonical
request differs from ours and the upload fails with 403 SignatureDoesNotMatch. "#"
is worse: requests treats it as a fragment delimiter, truncates the key there, and
the log is written to the wrong S3 object with a 200 and no error anywhere.
Build the object URL once, percent-encoding the key a single time and normalizing
the result with httpx so the string that is signed is byte-identical to the one
httpx transmits, and hand that URL straight to AWSRequest instead of routing it
through requests.
The key is encoded without being decoded first, so an alias holding a literal
percent escape stays a single path segment; decoding it would turn %2F into real
separators and move the object out of its prefix, and out of its bucket on the
path-style URL.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* test: enforce PT012 so a pytest.raises block cannot hide dead assertions
`with pytest.raises(...)` stops at the first statement that raises. Anything
sequenced after it inside the block never runs, so an assertion written there is
never checked and the test still reports green.
Two sites were doing exactly that, and both assertions turned out to be wrong
once they started running. tests/llm_translation/test_prompt_factory.py asserted
the bedrock rejection names "requires at least one non-system message", which
holds. tests/proxy_unit_tests/test_proxy_server.py asserted the prisma startup
failure mentions "httpx.ConnectError", which never appears: the failure is an
httpx.ConnectError whose message is "All connection attempts failed", so that
test now asserts the type. Its DATABASE_URL override moves to monkeypatch, since
the old restore sat below the assertion and leaked the invalid URL into every
later DB test the moment the assertion started being able to fail.
The remaining 72 sites are rewritten without changing what they exercise: setup
that cannot raise moves above the block, a nested `patch` moves outside it, and
bodies with real control flow (a stream drain, an if/else on sync_mode, a
retry loop) move into a local closure the block calls.
Fixing PT012 unmasked two B017s, since ruff only reports a blind
pytest.raises(Exception) once the block holds a single statement.
tests/proxy_unit_tests/test_auth_checks.py narrows to the ProxyException
can_key_call_model actually raises. tests/local_testing/test_completion_cost.py
was asserting vertex_ai/medlm-medium has no cost entry, which stopped being true
at some point; that dead first half is gone and the rest of the test, which
checks medlm pricing resolves above zero, now runs instead of being skipped.
* chore(ci): ratchet TQ004 to 768 after the prisma test moved to monkeypatch
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
* feat(otel): route Phoenix traces to per-key/team projects under otel v2
The v2 arize_phoenix preset read PHOENIX_PROJECT_NAME once at startup into a
static resource attribute, silently dropping the per-key/team project routing
v1 supported. Route it via Phoenix's x-project-name OTLP/HTTP header instead:
the env var stays the global default, and a phoenix_project_name (or
phoenix_project_name_override) in key/team metadata sends that key's traces
to the named project.
The project comes only from user_api_key_auth_metadata (server-set at auth),
never from client request metadata or StandardCallbackDynamicParams, since
choosing the telemetry destination is a data-exfiltration primitive. The
header is appended to the exporter's static headers rather than replacing
them, so the preset's Authorization survives, and it is gated to OTLP/HTTP
exporters because Phoenix only reads it on /v1/traces.
Also unban the bare phoenix_project_name fields from the request-body gate:
the proxy integrations ignore them (only user_api_key_auth_metadata routes,
and that stays banned), so rejecting them just broke SDK-style callers.
* fix(otel): root project-routed Phoenix spans in their own trace
Phoenix assigns a whole trace to one project by whichever span arrives
first. The request's auth/db/root spans always export through the default
provider without the project header, so a project-routed LLM span parented
into that trace got dragged back into the default project and the header
did nothing (verified against a live Phoenix instance). Detach the routed
span into its own trace with a link back to the request trace, mirroring
how the v1 Phoenix logger exported each request under its own local parent.
* fix(otel): drain in-flight spans before shutting down evicted providers
LRU eviction shut a routed provider down immediately, but an LLM span
opened at pre_call stays open until the later success or failure callback;
with more than 256 overlapping credential/project routes that in-flight
span was silently dropped instead of exported. Refcount open spans per
provider (hold at span open, release when the carrier is removed on close,
carrier-map eviction, or MCP stray-carrier cleanup) and defer a retired
provider's shutdown until its last open span closes.
* fix(otel): take the provider hold inside route_for to close the eviction race
pre_call can run on thread-pool workers, so between route_for returning a
provider and the caller recording its open span, a concurrent request could
overflow the LRU and shut that provider down with a zero span count, dropping
the routed trace. route_for now increments the open-span count in the same
locked critical section as the cache update and hands back an already-held
provider; every caller releases it once its span has landed. The lock also
makes the cache mutations safe under that same thread-pool concurrency.
* fix(otel): skip tenant routing on deferred pre_call
route_for ran before the recordable-parent check, so a thread-pool
pre_call still built or LRU-touched a tenant provider and could evict
an idle one even though the hold was released immediately and close
re-routed. Only route when the span actually opens
* add somethign
* Revert "add somethign"
This reverts commit 2f2cf84c5a.
* fix(otel): cap retired tenant providers draining open spans
* docs(otel): justify the retired-provider cap
* test: run the 30 test files stranded in the second mirror
tests/litellm sat beside tests/test_litellm, which is the mirror the repo
convention names, and no job collected it. The allowlist called the directory
unresolved and assumed it was a duplicate. It is not: 30 of its 34 files have no
counterpart in the real mirror, so they are tests nobody has run since they were
written, not copies of tests that run elsewhere.
Moving them in is byte-identical, and it is what makes them run. Every one is
now claimed by a shard's test-path rather than by an allowlist entry, and the
216 tests they hold pass. Directories that needed to become packages did, since
several files are named test_transformation.py and pytest cannot import two of
those from non-package directories in one session.
Never running is why three assertions had drifted away from the code:
* nvidia.nemotron-super-3-120b max_output_tokens, 32000 -> 32768
* sambanova/MiniMax-M2.7 max_input_tokens, 204800 -> 196608
* the Vertex text-to-speech handler moved from data= to json=, so the test
reads the decoded body off the json kwarg instead of parsing the data one
The first two follow model_prices_and_context_window.json, which the catalog
sync keeps current; the third follows the handler. In all three the test was the
stale side.
The lint workflow ran test_no_hardcoded_secrets.py by path and now points at the
new one.
Four files stay behind. Each shares a filename with a live test whose contents
are disjoint from it, so landing those means merging test bodies, which is a
content review rather than a move. The allowlist entry now names those four and
records how many tests each would bring, in place of calling the whole
directory unresolved.
* fix(ci): keep the secret scan out of the mirror's conftest
The secret-scan job runs pytest under uv run --no-project, so its environment
holds pytest and nothing else. That worked while the file sat in tests/litellm,
which has no conftest, and broke the moment it moved into tests/test_litellm,
whose conftest imports litellm on collection: ModuleNotFoundError: No module
named 'dotenv', before a single test ran.
The file is a repo-wide static scan that imports only base64, os, re and pytest,
so it belongs with the other repo-wide checks in tests/code_coverage_tests,
which has no conftest, rather than in the package mirror. Installing the full
dependency set into a 15-second job to satisfy a conftest it does not use would
be the wrong trade.
Verified with the job's exact command:
uv run --no-project --with 'pytest==9.0.2' pytest \
tests/code_coverage_tests/test_no_hardcoded_secrets.py -q
1 passed in 0.47s
A proxy on the default remote cost map never produced a prompt cache
breakpoint: the published map has the gpt-5.6 entries without
supports_prompt_cache_breakpoint, so the model-map gate returned False
for every listed model and only LITELLM_LOCAL_MODEL_COST_MAP=True (the
repo .env, hence the passing unit tests) made the feature work. The hook
now honors the flag when the entry carries one, True or False, and
otherwise applies the GPT-5.6+ version rule to the model name, so a map
that lags the flag still gets the OpenAI dialect. The model-map tests
pin litellm.model_cost to the bundled backup map and a new test drives
the hook against an unflagged gpt-5.6 entry.
completion() and acompletion() take base_url as an alias for api_base
that only lands on api_base after the cache control hook ran, so a
GPT-5.6 call at a non-OpenAI gateway given through base_url still got
the dialect. Both seed calls and the unstamped request-params read now
look at base_url too.
ResponsesAPIRequestUtils.merge_prompt_management_input reshaped hook
output in place, retyping text parts to input_text on the caller's own
message objects. The merge now shapes a copy of each message as it
emits it, so the identity-based merge keeps working on the hook's
objects and nothing the hook or the client owns is mutated.
The cache control hook also runs on litellm.responses() input. On a
GPT-5.6 deployment it wrapped a string-content item into a chat-shaped
{"type": "text"} part, which the Responses API rejects, and it never
marked input_text, input_image or input_file parts, so no breakpoint and
no prompt_cache_options reached the provider. Add the Responses part
types to the eligible block set and translate chat-shaped text parts on
non-assistant items to input_text in
ResponsesAPIRequestUtils.merge_prompt_management_input, which both the
async and the sync prompt management sites go through.
The dialect also fired for any GPT-5.6 name that resolved to provider
openai, including deployments pointed at a custom api_base that does not
understand prompt_cache_breakpoint. Decide it once per request from the
provider, the model map and the resolved api_base (request, then
litellm.api_base, then OPENAI_BASE_URL / OPENAI_API_BASE): only
api.openai.com and *.api.openai.com hosts speak the dialect, a top-level
prompt_cache_options opts a custom target in, and litellm_proxy/ targets
never get it. maybe_seed_default_injection_points takes api_base and
stamps the finished decision on the points as _litellm_openai_dialect so
the sync completion() path, whose hook params do not carry api_base,
honors it; maybe_inject_cache_control takes api_base from the
/v1/messages handler.
Eligibility now comes from a supports_prompt_cache_breakpoint model map
flag on the OpenAI gpt-5.6 entries, exposed through
litellm.utils.supports_prompt_cache_breakpoint, with the GPT version rule
kept only for models the map does not know. The OpenAI dialect no longer
reserves a slot for tool_config points, which OpenAI has no cache block
for, and with_prompt_cache_breakpoint plus the chat bridge helper return
a new block instead of mutating their input.
When the resolved deployment is provider openai and the model is GPT-5.6 or
newer, the cache control hook now writes prompt_cache_breakpoint on the
targeted content block and sets prompt_cache_options to explicit mode unless
the caller already passed one. The /v1/messages bridges carry the marker
through (the Responses bridge moves a marked system prompt into a developer
message, since top-level instructions cannot hold one). Breakpoint counting
and the stand-down check recognise both marker kinds, and client breakpoints
already present in messages are no longer subtracted from the cap twice.
Fixes#37509
Every dynamic tracer-provider build called Resource.create, which scans the entry
points of every installed distribution, roughly 3ms and 200 file opens. The dynamic
providers reach it from the async logging path, which runs on the event loop serving
requests, so past the provider cache bound every request paid it and delayed the
requests in flight alongside it
The value derives only from the logger's config and process environment, so it is
built once per logger and reused. This logger's own init-time providers share it,
which also removes redundant startup builds. ArizeLogger overrides _init_tracing and
still builds its own, so it keeps one extra build
Refs LIT-5437
* feat(otel): attribute Prisma database spans to PostgreSQL instead of localhost
Prisma reaches PostgreSQL through a query engine on loopback, so transport
instrumentation attributes database waits to localhost and operators cannot
tell the work is PostgreSQL or correlate it with the database's own metrics.
Datastore service spans now carry db.system.name, db.system, db.operation.name
and, for PostgreSQL, server.address, server.port and db.namespace derived from
DATABASE_URL, and are emitted as CLIENT spans. Only host, port, database and
schema are read, so no credential reaches an exporter. Endpoint attributes are
omitted when a read replica is configured, because routing is decided per Prisma
call underneath the span.
* fix(otel): reject a mis-split DSN authority and name socket-only databases
An unencoded '/' in the password truncates the URL authority, so urlparse
reports the username as the host and the password tail as the database, which
put credential material in db.namespace. Postgres drivers reject that DSN
outright, so the only safe reading is no endpoint at all.
A hostless 'postgresql:///litellm' is a valid local-socket DSN that Prisma
accepts, and it now yields db.namespace with no server address rather than
nothing. The default schema is matched case-insensitively, since an unquoted
PostgreSQL identifier folds and one deployment must yield one namespace.
* fix(otel): keep a non-default schema in db.namespace
Prisma quotes the schema name, so a DSN with ?schema=PUBLIC provisions a
second schema alongside public rather than reusing it. Observed on a live
proxy: the PUBLIC schema came up with its own 70 tables next to public's 78,
and a key created under one was not visible under the other.
Case-folding the two into a single namespace therefore reported two different
schemas as the same database, which is the misattribution this feature exists
to remove. Match the default literally.
* fix(otel): reject any DSN whose userinfo fell outside the authority
An unencoded '#' or '?' in the password sends the tail to the fragment or
query, leaving the path empty, so the marker check on the database segment
never fired and urlparse's hostname (the database username) was exported as
server.address.
The stranded userinfo '@' is the general tell for every mis-split, so guard on
that instead of enumerating the characters that cause it.
* fix(otel): allow an at-sign inside a well-formed DSN query
The previous guard rejected any DSN whose userinfo at-sign fell outside the
netloc, which also caught libpq parameters that legitimately carry one, so
?application_name=svc@prod and ?user=admin@company.com lost their endpoint
attributes.
Discriminate instead: a PostgreSQL DSN never has a fragment, its database name
cannot hold an unencoded at-sign or slash, and an at-sign in the query is only
suspicious when the query did not parse as parameters.
* fix(otel): resolve the database endpoint per span instead of once per process
The endpoint was cached for the process lifetime on the premise that
DATABASE_URL is deployment-static. It is not. The RDS IAM refresh rebuilds the
URL from DATABASE_HOST/PORT/NAME/SCHEMA on every rotation, the reconnect path
re-reads DATABASE_URL, and the DB-backed environment_variables config overlay
sets arbitrary keys post-startup with no blocklist covering DATABASE_*. A
process that had genuinely failed over kept exporting the old server.
Read the environment per span, which is also what Prisma connects with, so the
span can no longer name a different server than the one serving the query;
get_secret_str consulted a secret manager first and could diverge from it. Only
the parse is memoized, keyed on the URL.
* fix(otel): reject a question-mark mis-split whose tail parses as parameters
A '?' in a password strands the rest of the authority in the query, and that
tail can still parse as key=value, so testing only for an unparseable query let
the login through as server.address. One spelling hijacked the host= parameter
and put the password suffix there directly.
A legitimate at-sign in a query always follows a database path, and a
'?'-mis-split never leaves one, so require both.
* refactor(otel): drop the DSN parse cache that retained rotated credentials
The cache was keyed on the full DATABASE_URL, so up to eight complete DSNs,
each carrying a password or a retired IAM token, stayed referenced for the
process lifetime and outlived the rotation that replaced them. Nothing reached a
span, but a heap dump or crash report would have surfaced them.
Parsing costs about four microseconds against a span emission that costs orders
of magnitude more, so the cache bought nothing worth that.
* fix(otel): avoid a set construction the tightened LIT002 budget rejects
* fix(otel): refuse an ambiguous DSN authority instead of guessing at it
A password holding both an unencoded slash and a query-like tail defeated all
three shape checks: the slash left a clean path carrying the password
remainder, the query still parsed as parameters, and no fragment survived. The
login went out as server.address, the password's leading digits as server.port
and the rest as db.namespace.
A DSN whose at-sign sits in a query parameter is indistinguishable from that
mis-split by any property of the parse; both leave no userinfo, a host, a port
and a path. Guessing wrong publishes a credential fragment, so the ambiguity
now resolves to refusing the endpoint. Such a DSN loses server.address and
db.namespace and keeps the rest of the span; percent-encoding the at-sign
restores them.
Also honour port= alongside host=, which libpq pairs and this read ignored.
* docs(otel): fix a spliced sentence and a stale cache claim in db_endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): bound and shut down credential-scoped tracer providers
Each credential-scoped TracerProvider owns a BatchSpanProcessor worker thread that
only stops on shutdown, and the v1 cache holding them was an unbounded, unsynchronized
dict that never shut anything down. Every distinct team/key credential set therefore
added a thread for the life of the process, and concurrent first-requests for the same
credential set orphaned duplicate providers outright.
Make the cache a lock-guarded bounded LRU that shuts down whatever it drops, matching
the v2 TenantTracerCache. Providers wrapping a caller-supplied SpanExporter instance
share that exporter with the logger's own provider, so they are dropped without
shutdown; those use SimpleSpanProcessor and own no thread.
* fix(otel): reclaim dropped providers on a dedicated executor
Sustained credential churn queues one blocking shutdown per eviction, so using the
shared logging executor let an unreachable tenant endpoint stall unrelated logging
work behind the OTLP retry budget. Give provider shutdown its own bounded pool; its
threads spawn lazily, so a proxy that never evicts still pays nothing.
* fix(otel): decide provider shutdown from the victim, not the evicting request
Both dynamic entry points share one provider cache, so it can hold providers of
mixed exporter ownership. Reading the ownership flag from the evicting request
therefore stopped a shared caller-supplied exporter in one direction, silencing
telemetry process-wide, and leaked a BatchSpanProcessor thread in the other.
Cache ownership alongside the provider so the drop decision reads the victim's
own flag.
* fix(otel): honor the widened header mapping type instead of dict only
Widening the header parameter to Mapping left the isinstance check on dict, so a
non-dict Mapping silently returned no headers at all, which for the OTLP path means
an unauthenticated exporter and no traces with nothing raised. The dict branch also
returned the caller's own object, and dropping the defensive copy at the call site
let that alias reach a long-lived exporter. Match on Mapping and copy.
* fix(otel): do not give a provider we may never stop an interpreter-exit hook
Every TracerProvider registers an atexit hook by default, and that hook holds a strong
reference. Providers wrapping a caller-supplied exporter are dropped without shutdown,
so they stayed pinned for the life of the process and then stopped the shared exporter
at exit. Tie shutdown_on_exit to ownership: those providers use SimpleSpanProcessor and
buffer nothing, so they lose no flush, while providers that own their exporter keep the
hook and their exit flush.
Also stop the victim the eviction test leaves behind, and trim the added comments.
A pass that raises (a missing Slack webhook, say) now waits the daily interval instead of logging the
same exception every 30 seconds, and the Admin UI alerting settings list the new alert type so it can be
toggled like the others
An empty pass no longer holds the daily lock, a False lock claim (held or redis
error) is retried on the next 30 second poll instead of sleeping a day, and a
sent alert is stamped in the shared cache for a day so sibling pods and restarts
stay quiet
* fix(shadow_eval): copy messages before router call and raise judge output cap
* fix(shadow_eval): lead failure detail with location and pin post-failure continuation
Shadow eval only answered "should this key adopt this auto-router". Once a key
is on the router it is invisible to the feature, because the sampling gate skips
any request the shadowed router already served, so post-adoption quality
regressions go unmeasured.
Reverse mode inverts the arms: sample the traffic the router did serve and
duplicate it against a fixed baseline_model, judged by the same blind pairwise
judge. Same job table, same attempt rows, same aggregates.
real_* stays the arm the caller was served and shadow_* the duplicated one, so
in reverse real_model is the router's pick and shadow_model is the baseline. The
active-job slot becomes one per (key, direction) so both directions can run at
once, and tier attribution in reverse reads the control request's routing
decision rather than the shadow call's write-back.
update_trace_keys lets a caller name which request metadata entries get copied
onto an existing trace, and the name is unrestricted. Sending
update_trace_keys: ["user_api_key_auth"] with existing_trace_id serializes the
resolved auth object, including the team callback credentials it carries, onto
the trace through Langfuse.trace(**trace_params). TraceBody is Extra.allow, so
an unexpected key ships rather than being dropped.
Any holder of a team key can do this and read the result in the destination the
team already logs to, so the feature is now inert unless an operator turns it on
with langfuse_enable_update_trace_keys.
Request metadata carries the whole UserAPIKeyAuth object, whose team_metadata
holds the customer's own langfuse callback_vars. The only filter on the emitted
blob was a four key deny list written as a circular reference crash guard, so
those credentials reached the customer's own langfuse traces.
The emitted blob is now the StandardLoggingPayload allowlist plus the litellm
computed enrichments, and nothing is copied across from raw request metadata.
That makes the credential exclusion structural rather than a filter someone has
to keep correct. Steering keys keep reading raw metadata, matching literal_ai.
Proxy callers are unaffected: their request metadata already rides under the
allowlisted requester_metadata key, nesting intact.
debug_langfuse dumped raw request metadata into the trace as a second copy of
the same leak. It now emits caller scalars only.
When StandardLoggingPayload is absent the trace is still emitted with the
existing trace_id fallback, so failure traces survive.
langfuse_* request headers land in metadata as strings, but the trace path reads
mask_input/mask_output with a bare truthiness check and iterates update_trace_keys
directly. A header saying mask_input: false redacted the payload it was asked to
keep, and update_trace_keys was walked one character at a time so every requested
key silently failed to match
* fix(langfuse): emit otel trace version and release on the keys langfuse v4 reads
The langfuse_otel exporter wrote version to langfuse.generation.version and
langfuse.trace.version, and release to langfuse.trace.release. Langfuse v4
recognizes neither, so both landed in the generic span attribute bag and every
trace reported version and release as null. v4 has a single langfuse.version
key, lifted to the trace when it sits on the root span, plus langfuse.release.
Also routes the otel v2 preset's per-request headers through the shared builder
so key-scoped and team-scoped exports carry x-langfuse-ingestion-version like
the other three exporter paths already do.
* fix(langfuse): give trace_version precedence over version on the shared v4 key
Matches the documented contract in docs/observability/langfuse_integration.md
and the legacy langfuse SDK callback, which both treat trace_version as the
authoritative trace version with version as its fallback.
