mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-03 07:23:33 +00:00
250af8f027
* fix(responses): log the encrypted response id on streamed /v1/responses spend rows
With response id security on, the stream hook encrypts the resp_ id on every
frame, but logging had already received the SDK's own copy of the terminal
response, so the spend row, the standard logging payload and callbacks got the
managed id (or the chatcmpl id on a bridged stream) the client never saw. The
hook now records the exact id it advertises on the request's logging object,
and the logging layer stamps that id on the logged streamed response
* fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id
Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the
id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new
session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes
to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response,
and a raw provider id still gets a managed id
Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or
<id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's
request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The
background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id
* fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner
A GET or cancel of a stored response now logs a new encrypted id of its own,
so callbacks and spend rows never reuse the create's id, while the owner of
the id still gets back the exact id they sent. An admin reading someone
else's response gets an id bound to the admin, as before. The spend row id
suffix for reads and cancels is gone, so with response id security off the
rows and session history match the default branch. A streamed GET follows
the same rules.
* fix(responses): echo the sent id to every reader that passes the response access check
A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed
the access check, not only when the caller is the user or team the id was issued to. An admin, the master key
or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not
reach the first turn from it, so a bridged follow-up through that id lost the conversation history
Every read and cancel still logs a fresh id bound to the reader, never the create's id
* fix(responses): keep a read or cancel row in session history only when it brings new output
A GET, streamed GET or cancel sent with a session header joins the
create's session, and its stored response repeats the create's output.
The replay then sent the same assistant turn once per read.
Session history now keeps an aget_responses or acancel_responses row
only when its stored Responses output has an item id that no earlier
kept row carried. A polled background response keeps the first poll
that returns the finished output, since its queued create row has none,
and later polls of the same output are dropped. Every other row keeps
today's rule
* fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart
A create row now owns its output items in session history, and a read or cancel row adds only items no
create and no earlier read carried, in one linear pass, so a background response polled while it grows,
or a read that ends before its streamed create, replays the answer once and after its prompt
DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id
like a read does
Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row
id, so a streamed background create and its cost event, which now share the client's id, keep separate
files
The integration tests that decoded the managed id now match the exact id the client received
* fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events
A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh
encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET
Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage
file and unbatched GCS object, so batch retrieves and events with no id keep main's names
An item a session read replays takes the newest read's copy at the first read's place, so an item a
poll caught half written replays finished
Tests pin the client's id over the router's continued response id, and the storage test no longer reads
the clock after the upload
* fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items
A streamed background create and its later cost event share the client's
id, so OpenMeter dropped the second event and DynamoDB overwrote the
create's item. Both now key the cost event as <id>_background_cost, like
its spend row and storage file.
Session history again replays a read's output items that carry no id,
as main did.
* fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras
* fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id
A streamed background create and its later cost event now share the id the
client received, so ClickHouse and DeepEval saw both events under one id.
Route both through logged_event_key, the way the Azure storage, GCS,
OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse
request_id matches its spend row's `<id>_background_cost` and its DeepEval
span gets its own uuid. ClickHouse response_id stays the raw provider id.
* test(deepeval): annotate the background cost span test's locals with Final
* test(responses): run the DELETE wire test that pins the echoed client id
DELETE /v1/responses/{id} now answers with the id the caller sent, so the
skip that marked it as a known bug no longer applies. The test passes on
the /v1, /openai/v1 and bare routes.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|---|---|---|
| .. | ||
| _azure_storage_support.py | ||
| _langfuse_otel_metadata_support.py | ||
| _logging_only_scope_support.py | ||
| _openinference_support.py | ||
| _s3_v2_support.py | ||
| _ternary_support.py | ||
| azure_dispatch_support.py | ||
| conftest.py | ||
| logging_worker_drain_canary.py | ||
| test_akto_logging_only.py | ||
| test_apply_guardrail_endpoint.py | ||
| test_arize_otel_v2_openinference_chaos.py | ||
| test_arize_otel_v2_openinference_family.py | ||
| test_arize_otel_v2_openinference_modes.py | ||
| test_arize_otel_v2_openinference_sad_edge.py | ||
| test_arize_otel_v2_openinference_spans.py | ||
| test_azure_content_safety_audit.py | ||
| test_azure_content_safety_dispatch.py | ||
| test_azure_content_safety_dispatch_resilience.py | ||
| test_azure_content_safety_endpoints.py | ||
| test_azure_storage_chaos.py | ||
| test_azure_storage_client_ttl.py | ||
| test_azure_storage_file_names.py | ||
| test_batch_line_items_cold_storage.py | ||
| test_bedrock_converse_guardrail_rewrites.py | ||
| test_bedrock_converse_stream_post_call_guardrail.py | ||
| test_bedrock_error_request_id.py | ||
| test_bedrock_guardrail_attachments.py | ||
| test_bedrock_invoke_stream_post_call_guardrail.py | ||
| test_cache_hit_guardrail_metrics.py | ||
| test_cache_hit_guardrail_metrics_chaos.py | ||
| test_callback_delivery.py | ||
| test_content_filter_logging_only.py | ||
| test_custom_logger_hook_counts.py | ||
| test_custom_logger_hook_counts_chaos.py | ||
| test_custom_logger_hook_counts_modes.py | ||
| test_custom_logger_hook_counts_post_call_block.py | ||
| test_custom_logger_hook_counts_vendors.py | ||
| test_custom_logger_redaction.py | ||
| test_datadog_logging_delivery.py | ||
| test_file_usage_counter_span.py | ||
| test_generic_guardrail_api_wire.py | ||
| test_generic_guardrail_num_retries.py | ||
| test_grayswan_wire.py | ||
| test_grayswan_wire_chaos.py | ||
| test_guardrail_attachment.py | ||
| test_guardrail_effects.py | ||
| test_guardrail_load_balancing.py | ||
| test_guardrail_pipeline_applied_header.py | ||
| test_guardrail_stream_scope.py | ||
| test_guardrail_stream_scope_chaos.py | ||
| test_guardrail_stream_scope_matrix.py | ||
| test_guardrail_timeout_all_providers.py | ||
| test_langfuse_delivery.py | ||
| test_langfuse_otel_console_output.py | ||
| test_langfuse_otel_legacy_destinations.py | ||
| test_langfuse_otel_metadata_chaos.py | ||
| test_langfuse_otel_metadata_clients.py | ||
| test_langfuse_otel_metadata_destinations.py | ||
| test_langfuse_otel_metadata_identity.py | ||
| test_langfuse_otel_tenant_only.py | ||
| test_langfuse_otel_trace_fields.py | ||
| test_langsmith_logging_delivery.py | ||
| test_langtrace_delivery.py | ||
| test_logging_only_scope_chaos.py | ||
| test_logging_only_scope_config.py | ||
| test_logging_only_scope_runtime.py | ||
| test_logging_worker_drain.py | ||
| test_otel_cache_hit_cost.py | ||
| test_otel_conversation_id.py | ||
| test_otel_excluded_services.py | ||
| test_otel_excluded_services_matrix.py | ||
| test_otel_team_capture_message_content.py | ||
| test_otel_team_sampling_across_pods.py | ||
| test_otel_text_completion_choices.py | ||
| test_otel_trace_response_header.py | ||
| test_otel_v1_request_trace.py | ||
| test_ovalix_guardrail.py | ||
| test_panw_airs_block_masked_response.py | ||
| test_passthrough_guardrail_scan_text.py | ||
| test_passthrough_operation_name.py | ||
| test_passthrough_upstream_error_chaos.py | ||
| test_passthrough_upstream_error_visibility.py | ||
| test_presidio_entity_masking.py | ||
| test_presidio_masking_endpoints.py | ||
| test_presidio_overlapping_spans.py | ||
| test_presidio_streaming_output.py | ||
| test_prometheus_deployment_failure_api_provider.py | ||
| test_prometheus_project_alias_label.py | ||
| test_prometheus_provider_auth_failure.py | ||
| test_prometheus_provider_auth_failure_chaos.py | ||
| test_prometheus_request_metrics.py | ||
| test_prometheus_series_cap.py | ||
| test_prometheus_series_cap_chaos.py | ||
| test_prometheus_shared_custom_labels.py | ||
| test_realtime_guardrail_translation.py | ||
| test_s3_v2_flush_surfaces.py | ||
| test_s3_v2_partition_granularity.py | ||
| test_s3_v2_streaming_failure_dedupe.py | ||
| test_s3_v2_upload_fanout.py | ||
| test_signoz_delivery.py | ||
| test_straiker_v3_platform.py | ||
| test_stream_error_alert_api_base.py | ||
| test_ternary_focus_export.py | ||
| test_weave_key_scoped_delivery.py | ||
| test_wingback_guardrail.py | ||
| test_xecguard_wire.py | ||