A body that serializes litellm_trace_id as null or an empty string carries no identity, so it must not
block the server span fallback. Also mark the nested metadata write as an out-param store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
UserAPIKeyAuth.parent_otel_span is Any at runtime (opentelemetry is an optional extra), so the OTel
trace-id fallback must only format an int trace id, otherwise an object that merely quacks like a span
turns the whole request into a 500
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Pass user_api_key_dict.parent_otel_span into the outgoing W3C injection so the
legacy otel callback propagates its litellm_request span, falling back to the
otel_v2 request root span and then the ambient span. Extend the mapped unit
tests to assert the propagated trace and span ids over real captured headers
for HTTP and WebSocket passthrough with forwarding on and off.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
When the otel callback is enabled and the client sends no trace or session identity, the request now inherits the W3C trace id of the proxy's server span as litellm_trace_id and metadata.trace_id. The missing_session_id policy and SpendLogs then persist that value as session_id, so a trace in the OTel backend and its row in the Logs UI carry the same id. Explicit x-litellm-trace-id, traceparent, body metadata.trace_id and litellm_trace_id keep priority.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The fallback retry in _pre_call_with_fallbacks re-entered common_processing_pre_call_logic with data already enriched by the first pass, so add_litellm_data_to_request deep-copied a metadata dict holding the live OTel span and the request failed with a 500 (cannot pickle '_thread.RLock') instead of the intended 429 or fallback. Capture the configured fallbacks and a snapshot of the client request before the first pass, look up the fallback chain by the normalized model group after the limiter raises, and run each fallback attempt on a fresh copy of that snapshot. Replaces the mock-heavy tests with a rig that runs the real v3 limiter and a live OTel span through the proxy_logging_obj seam
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
tests/integration/conftest.py imported the bare `integration` package. Because
tests/__init__.py and tests/integration/__init__.py both exist, pytest's default
prepend import mode puts only the repo root on sys.path, so that name resolved
only under the PYTHONPATH that tests/integration/run.py injects. Every other
invocation died at conftest import with
ModuleNotFoundError: No module named 'integration' and exit 4, including the
command test_oci_integration.py documents in its own docstring.
The imports now use the tests.integration._support path that pytest actually
resolves, matching the 120 other `from tests.` imports in the suite. run.py's
PYTHONPATH still works because it already puts the repo root on the path.
tests/code_coverage_tests/test_integration_suite_imports.py collects every file
under tests/integration with PYTHONPATH scrubbed and asserts a non-zero
collection count, so an unresolvable import fails the code-quality job instead
of only the developers who run these files by hand. CI runs the three
pre-existing files through the allowlist rather than executing them, which is
why nothing caught this.
The first cache-enabled litellm-e2e build (211) showed three gaps in the shared provider cache:
Every OpenAI response carries Cloudflare bot-management Set-Cookie headers, and the capture rejected any response with Set-Cookie, so no OpenAI response was ever recorded (179 of 372 misses rejected). The edge already withholds Set-Cookie from the proxy, so drop it before validating and storing instead of rejecting.
The provider prompt-caching tests need fresh provider state: a replayed priming response reports cache creation rather than a cache read, and the TPM test then trips the key limit. Mark both modules provider_live.
TestApiBaseSeam::test_live_mode_returns_none ran inside the cache-enabled runner and saw the shared edge; isolate it from E2E_PROVIDER_CACHE.
Together moved openai/gpt-oss-20b off serverless and three tests in
test_completion.py died on a live 400. None of them needed Together to be up:
streaming is already covered live by tests/e2e/llm_translation/test_together_ai_e2e.py,
which picks its model from the cost map instead of pinning one, and the other
two are request-shape questions. Delete all three and assert the two shapes in
the mapped transformation file: the provider prefix is stripped without eating
the rest of a slashed model name, and custom role wrappers never reach the
request.
Already shaped ProxyException and HTTPException errors passing through the moderations, audio speech, Anthropic Messages, and handle_exception_on_proxy paths now answer with the x-litellm-call-id header the route logged under, without overwriting a header the exception was raised with. The GET /v1/batches failure hook receives the resolved request data so the spend log request_id matches the response header and the error log
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Together moved openai/gpt-oss-20b off serverless, so three tests started
failing with a 400 model_not_available from the live API. None of them was
really testing Together: they cover provider-prefix parsing, prompt shaping
and streaming, all litellm side. Mock the transport and assert those, so the
tests answer to our code instead of a vendor catalog.
The reset job moved from zeroing spend to an atomic decrement of the amount
it cleared, so every batched write now carries {"decrement": <cleared>}
instead of 0. Four tests still pinned 0 and had been failing since, which
also meant they no longer checked the amount at all. Assert the decrement
equals each row's own pre-reset spend, so a wrong amount fails the test.
simple_shuffle logs the selected deployment at INFO whenever a weight set
applies, so the fallback group's selection line lands between the fallback
notice and the success notice and pushed the notice out of the tail-3 window.
Filter it the same way the neighbouring get_available_deployment noise is
already filtered.
