scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-27 01:22:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 80
litellm/tests/test_litellm/integrations
History
devin-ai-integration[bot] 1edc4ba580
fix(logging): pass provider response headers to callbacks on every endpoint (#42824) 
* fix(logging): pass provider response headers to callbacks on every endpoint

Custom callbacks only received kwargs["response_headers"] for chat
completions. Responses, image generation and edit, speech, and
transcription calls either never recorded the provider's headers or
recorded them in one place and not the other.

Every handler now records the provider's httpx headers on the response's
hidden params as "headers" (raw) and "additional_headers" (processed,
with LiteLLM's own entries winning on a clash), and the logging object
derives model_call_details["response_headers"] from those hidden params
before cost calculation on the non-stream and both streaming success
paths, keeping a handler-set value authoritative. Binary speech responses
expose their hidden params to the standard logging payload, and the sync
OpenAI transcription request always fetches the raw response.

* test(images): point the legacy image and speech fakes at the raw response surface

Image generation now goes through the SDK's raw response so the provider headers can be read, and the speech binary response now carries hidden params. The unit fakes in the image generation, xinference, proxy provider, image edit, Vertex speech, and otel suites still pinned the old call surface and the old "no hidden params" assertion, so they read an uncalled mock or a fake response without headers.

* test(images): drop the rewritten mock comments and the generated edit PNGs

* test(images): move the llm-span test's image fake to the raw response surface

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-24 13:01:12 -07:00
..
arize feat(arize): per-team success and error sampling rates for the Arize AX callback (#42383) 2026-09-22 11:21:40 -05:00
azure_storage fix(azure): restrict the storage credential chain to deployment identities (#39637) 2026-09-03 18:29:32 -07:00
bitbucket test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
cloudzero fix(cloudzero): preserve late resource tags (#39873) 2026-09-05 12:10:05 -07:00
code_interpreter_interception feat(sandbox): reuse e2b container across requests when metadata.session_id is set (#31688) 2026-06-30 18:58:09 -07:00
datadog fix(types): blank non-string datadog tool text fields, look the spend table up by name 2026-09-21 14:10:09 -07:00
dotprompt Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_dotprompt_model_swap 2026-08-26 15:13:57 -07:00
focus fix(spend-tracking): hand plain dict rows to polars in the CloudZero and Focus exports 2026-09-03 18:46:01 -07:00
gitlab test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
langfuse fix(langfuse): warn and drop invalid LANGFUSE_TRACING_ENVIRONMENT instead of failing requests (#38582) 2026-08-27 18:03:42 -07:00
levo test: migrate wave 1 phase 2 legacy unit tests to tests/unit 2026-09-20 07:50:43 +00:00
newrelic feat(newrelic): export team max and remaining budget gauges to the Metric API (#40542) 2026-09-10 17:16:15 -07:00
open_telemetry test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
otel fix(logging): pass provider response headers to callbacks on every endpoint (#42824) 2026-09-24 13:01:12 -07:00
SlackAlerting test(alerting): inject the webhook client and extend the mapped test files 2026-09-21 14:16:37 -07:00
websearch_interception fix(websearch_interception): keep intercepted searches under the parent request's session and trace (#41711) 2026-09-22 12:35:27 -07:00
conftest.py fix(otel): honor SSL_CERT_FILE and ssl_verify in OTLP HTTP exporters (#42106) 2026-09-22 20:50:53 +00:00
rubrik_test_helpers.py Litellm oss staging 04 21 2026 2 (#26569) 2026-05-20 21:25:19 -07:00
test_agentops.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_anthropic_cache_control_hook.py chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
test_athina.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_azure_sentinel.py test(azure_sentinel): pin batch_size as a per-request bound under concurrent events (#40320) 2026-09-08 16:55:40 -07:00
test_braintrust_logging.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_braintrust_span_name.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_custom_guardrail.py Revert "Merge pull request #41220 from BerriAI/litellm_post_call_guardrail_context" 2026-09-19 17:45:15 +00:00
test_custom_guardrail_recursion.py fix: recursive pydantic issue (#19531) 2026-01-22 19:56:41 -08:00
test_custom_prompt_management.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_deepeval.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_galileo.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_guardrail_logging_sync.py fix(guardrails): keep guardrail information in spend logs when the caller sends its own metadata 2026-07-24 16:20:44 -07:00
test_helicone.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_langfuse.py refactor(langfuse): remove unreachable langfuse v1 logging path 2026-09-17 20:04:50 +00:00
test_langfuse_otel.py feat(langfuse): support langfuse_environment as a per-key dynamic callback param (#38264) 2026-08-26 16:56:55 -07:00
test_langsmith_init.py fix(langsmith): json.dumps with default=str so non-serializable metadata does not crash batch flush (#42424) 2026-09-22 12:21:11 -07:00
test_lunary.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_mlflow.py fix(mlflow): prevent _stream_id_to_span leak and mlflow 2.x end_trace TypeError (#39049) 2026-09-08 16:18:06 -07:00
test_openmeter.py test: unwind environment writes in tests/test_litellm with monkeypatch (#37806) 2026-08-21 20:28:37 -07:00
test_opentelemetry.py fix(otel): honor SSL_CERT_FILE and ssl_verify in OTLP HTTP exporters (#42106) 2026-09-22 20:50:53 +00:00
test_opentelemetry_dynamic_imports.py Make grpc dependency optional (#19447) 2026-01-20 19:03:52 -08:00
test_opik_utils.py chore(ci): merge oss branch (#33784) 2026-07-17 23:22:13 +00:00
test_otel_guardrail_violation_spans.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_otel_team_attributes_matrix.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_prometheus_api_promql_escape.py fix(prometheus): quote api_key for PromQL string literal in spend lookup 2026-05-01 21:38:19 +00:00
test_prometheus_budget_metric_guard.py fix(prometheus): skip budget metric DB lookups when gauges are NoOpMetric (#32834) 2026-07-10 20:25:47 -07:00
test_prometheus_budget_metrics_db_lookups.py perf(auth): negative-cache missing user/key lookups on the request hot path (#32368) 2026-07-08 09:59:57 +03:00
test_prometheus_budget_metrics_timeout.py feat: litellm oss staging (#31935) 2026-07-03 09:27:31 +05:30
test_prometheus_cache_metrics.py fix(prometheus): populate cache write token metrics for OpenAI-style usage (#34803) 2026-07-27 12:28:19 -07:00
test_prometheus_caller_identity.py test(prometheus): cover caller-identity config failure cases (#38380) 2026-08-26 11:53:49 -07:00
test_prometheus_carried_budget_state.py perf(proxy): one MGET and one pipeline for post-call spend counters, no team/user/org refetch on the response path (#40841) 2026-09-12 16:05:50 +00:00
test_prometheus_client_ip_user_agent.py feat(prometheus): bucket latency by input sequence length (#40059) 2026-09-10 00:29:28 -07:00
test_prometheus_custom_metadata_label_counts.py Fix Prometheus remaining metric zero values (#27348) 2026-05-06 17:22:20 -07:00
test_prometheus_deployment_state_proxy_rejects.py fix(proxy): attribute provider and model_info on pre_call_hook rejections (#41077) 2026-09-22 14:14:13 -07:00
test_prometheus_end_user_cardinality.py test(prometheus): cover customer budget series expiry by end_user ttl 2026-09-16 19:22:30 +00:00
test_prometheus_input_sequence_length_label.py feat(prometheus): bucket latency by input sequence length (#40059) 2026-09-10 00:29:28 -07:00
test_prometheus_invalid_key_filtering.py fix(prometheus): count 401 auth failures in litellm_proxy_failed_requests_metric 2026-09-15 02:09:00 +00:00
test_prometheus_labels.py fix(prometheus): add model_group label to deployment request and rate limit metrics (#42966) 2026-09-24 12:15:01 -07:00
test_prometheus_mcp_tool_metrics.py feat(prometheus): expose MCP tool metadata in Prometheus metrics (#31899) 2026-07-02 10:56:35 +03:00
test_prometheus_media_generation_metrics.py feat(prometheus): expose video duration and image count consumption metrics (#33138) 2026-07-13 18:51:13 -07:00
test_prometheus_metric_name_consistency.py test(prometheus): restore the full registry and build admission metrics fresh in the dashboard fixture 2026-09-17 09:21:19 +00:00
test_prometheus_metrics_endpoint.py perf(prometheus): render /metrics off the event loop and coalesce concurrent scrapes (#37702) 2026-08-20 16:08:22 -07:00
test_prometheus_missing_metrics.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_prometheus_none_metadata.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_prometheus_overhead_with_guardrails.py feat(prometheus): add litellm_total_overhead_latency_metric (SDK overhead + guardrails) (#31593) 2026-06-30 17:34:17 +08:00
test_prometheus_queue_guardrail_metrics.py fix(prometheus): fold auth/pre-call time into litellm_request_total_latency_metric (#37958) 2026-08-22 14:25:55 -07:00
test_prometheus_rate_limit_labels.py feat(prometheus): expose per-key and per-team rate limit allowed and used gauges (#39236) 2026-09-01 18:03:18 -07:00
test_prometheus_remaining_tokens_router_fallback.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_prometheus_requested_model_cardinality.py fix(proxy): attribute provider and model_info on pre_call_hook rejections (#41077) 2026-09-22 14:14:13 -07:00
test_prometheus_service_tier_label.py feat(otel): stamp service tier attributes on inference spans (#35679) 2026-08-03 23:10:01 -07:00
test_prometheus_services.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_prometheus_spend_logs_metadata.py fix(prometheus): expose project_alias in custom metadata labels (LIT-3741) (#31784) 2026-07-01 10:44:02 +08:00
test_prometheus_stream_label.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_prometheus_token_detail_metrics.py fix(prometheus): populate cache write token metrics for OpenAI-style usage (#34803) 2026-07-27 12:28:19 -07:00
test_prometheus_user_team_metrics.py fix(prometheus): skip customer budget cache read when end_user tracking is off 2026-09-16 20:23:59 +00:00
test_prometheus_zero_cost_metric.py feat(cost): warn and count $0 cost on billable requests (#42345) 2026-09-21 19:49:26 -07:00
test_prompt_manager_ssti.py fix(security): sandbox jinja2 in gitlab/arize/bitbucket prompt managers 2026-05-02 09:14:02 +00:00
test_responses_background_cost.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_rubrik.py fix(guardrails/rubrik): attribute blocked requests to the caller that made them (#35734) 2026-08-03 19:56:24 -07:00
test_s3.py test(s3): type the prompts-only test helpers 2026-09-15 23:40:35 +00:00
test_s3_v2.py fix(s3_v2): bound concurrent S3 uploads per flush and add opt-in JSONL batch files (#41258) 2026-09-24 07:32:00 +00:00
test_shadow_eval_logger.py fix(shadow-eval): replay approved pre-call guardrail snapshots (#42774) 2026-09-23 17:43:54 -07:00
test_weave_otel.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00