* fix(openai): carry cache_control to prompt_cache_breakpoint on /v1/responses to gpt-5.6
Native /v1/responses used to strip every Anthropic cache_control mark, so a
gpt-5.6 deployment in explicit mode never wrote a cache. Marks on system,
developer and user parts and on function_call_output now become
prompt_cache_breakpoint on the same block when the deployment reads
breakpoints, and are stripped exactly as before everywhere else
* fix(openai): read str providers and base_url in the prompt cache breakpoint gate
Responses configs generated from a provider JSON file, and openai_like
itself, name their provider as a plain str, so .value crashed every native
Responses request through them. The gate also only read api_base, while
responses() keeps a caller's base_url as an extra field on the params, so an
injected mark on a custom host was rewritten to a breakpoint
* test(openai): cover the pass-through branches of the prompt cache breakpoint swap
A part type OpenAI does not cache keeps no breakpoint and an item mark skips
it, and an item or content of a shape OpenAI will reject reaches the strip
untouched instead of failing inside the proxy
* fix(prompt_caching): keep a client-written prompt_cache_breakpoint when a cache_control mark lands on the same block
* fix(proxy): broadcast team cache invalidation on team writes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover cross-instance team cache invalidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): source Home What's new from day 0 support blog posts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): Home cards read card_title and card_summary from blog tags with provider logo
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): redact nested callback input and provider response fields
* fix(logging): keep shared request body for per-callback redaction and drop hidden provider fields
The deprecated per-callback message_logging=False path ran perform_redaction on
the shared model_call_details, which now also redacted the nested
proxy_server_request body. The spend log Request column and
previous_response_id replay read that body, so one opted-out callback blanked
them for everyone. That path now keeps the shared body and hidden params, and
redact_streaming_responses_for_custom_logger hands the opted-out callback its
own redacted litellm_params.
Redacted litellm_params also clear provider_specific_fields copied into
metadata.hidden_params (Anthropic thinking text), in litellm_params and in the
nested request body, and the nested body redacts system and instructions.
* fix(logging): keep replayed instructions and router usage marker under redaction
The shared request body keeps system and instructions again, so spend logs
store them and previous_response_id follow-ups replay the real instructions.
Per-callback copies still redact them.
Provider-specific fields in metadata.hidden_params are cleared in place, so
the metadata dict keeps its identity and the router's counted-usage marker
stays visible to deployment_callback_on_success (no double tpm/rpm count).
* fix(logging): never copy shared metadata dicts during redaction
The shared redaction pass now leaves litellm_params metadata dicts as they
are, so a hidden_params write from the other success handler between the
in-place clear and the copy can no longer move the router usage marker to a
stale dict. The key-presence predicate change is reverted as unneeded.
* refactor(logging): redact nested request body and provider fields with the existing message rules
Run perform_redaction on a copy of proxy_server_request.body, the same call the
spend log already makes, and redact reasoning inside provider_specific_fields the
way the message's own reasoning_content and thinking_blocks are redacted, instead
of clearing every provider field. Opaque fields such as Gemini thought_signatures
stay, and the metadata dicts and streaming objects are handled as on main.
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): return provider-matching 4xx for the LIT-8062 status code follow-ups
- OCR multipart errors are 400 invalid_request_error on the file param
- a missing model is a 400 on chat, responses and messages even with a wildcard deployment
- responses max_output_tokens <= 0 reaches the provider instead of being raised to 16
- daily activity rejects end_date before start_date with a 400
- vector store file attributes are sent in full so the provider enforces its key limit
- router-wide defaults no longer satisfy a param the Router takes positionally, or a model the router does not serve
- a JSON null user_config is treated as absent
* chore: keep the type gates at their merge-base counts
* fix(proxy): keep fixed-target wildcards serving a missing model and close router-default gaps
- a missing model is rejected only when the wildcard would forward it into the target
- moderation only counts router defaults for models get_model_list lists
- user_config defaults and deployment params never satisfy a positionally bound param
* test(integration): pin the fixed-target wildcard path and no provider call on positional-param 400s
* chore: validate pattern-matched deployments instead of reading untyped dicts
* fix(proxy): keep router defaults counting for a2a agent models
* test: drop the signature-walk test and type the new test helpers
* test(integration): audit matrix for the status-code follow-ups
Drives every mode the fix branches on through a real proxy, Postgres, Redis and
the scripted upstream: positional params against router, deployment and
user_config defaults; moderation listed vs unlisted; unknown and a2a models;
every user_config shape on chat, responses and messages, streaming and SDK
clients; forwarding vs fixed-target wildcards for missing, null and empty
models; Responses max_output_tokens edges; OCR parse errors; vector store file
attributes; every daily activity route; and a provider outage mid burst plus a
killed worker on a two-worker proxy. Product gaps found are skipped as BUG.
* style: ruff format the audit test files
* test: name the auth pre-check as the source of the JSON array 500
* test: build the audit helpers without mutation and share one sentinel
* test: build every audit request body as one literal
* fix(proxy): keep router defaults and model-less dispatch for requests a fallback chain serves
* fix(proxy): count request, key and team fallbacks when an unlisted model inherits router defaults
* fix(proxy): skip key and team fallbacks on the client-credential dispatch path
* fix(proxy): keep router defaults when the router's own * fallback serves an unlisted model
* fix(proxy): count client-style fallbacks, forward an empty model, report invalid usage chat dates
- The router-default pre-check now treats client-style request, key and team fallbacks
([{"model": ...}]) as serving an unlisted model, as the router does
- The forwarding-wildcard gate only rejects a missing or null model; an empty model
reaches the provider as before (vLLM serves its loaded model for it)
- The /usage/ai/chat tools hand the model the date-range error instead of an empty
dataset or a generic fetch error
* test(usage): type the invalid-date tool test
* fix(proxy): keep rejecting an empty model behind a forwarding wildcard
Key and team model checks skip an empty model, so forwarding it let a key
restricted to another model reach a server's default model (vLLM serves
its loaded model for ""). Pin that with a restricted-key integration cell.
* fix(usage): report the query layer's date 400 to the usage chat model
Checking dates before the fetch refused formats the global usage tool's query
accepts (2026/10/01 returned spend on main). Catch the daily activity query's
own 400 instead, so each tool rejects exactly what it rejected before and only
the message changes.
* fix(proxy): restore three model-less and unlisted-model paths main served
An empty model is looked up as sent, so next to a forwarding */* wildcard it
still reaches the fixed * deployment it matches. A null model with a prompt_id
is left to the prompt manager, which may pick the model. A Claude Code
subagent turn on a session-bound auto/complexity router counts as served, so
the router default max_tokens still applies.
* fix(proxy): only skip the model check for a prompt_id when a prompt manager is registered
Without a callback prompt manager nothing can pick the model, so a null model
with a prompt_id behind a forwarding wildcard gets the same 400 as any other
null model.
* fix(proxy): leave a model-less Claude Code subagent turn to its session router
A subagent turn on a session bound to an auto, complexity, adaptive or
quality router is answered by that router whatever model it names, so a
null or missing model behind a forwarding wildcard is no longer rejected.
An empty model stays rejected.
* fix(proxy): skip the wildcard model check only for a bound Claude Code session
Read the session's router binding, as the router does, instead of only
checking that the request could resume one, so a subagent turn without a
model on an unbound session is still rejected before a forwarding
wildcard sends it upstream.
* refactor(proxy): stop predicting routing in the proxy pre-checks
- drop the wildcard missing-model gate: a model-less request behind a forwarding wildcard goes to the router and gets the provider's own status, as on main
- drop the "model the router serves" check: router defaults count for any named model again, except moderation, where the Router merges them only for listed models
- keep the positional-param, moderation and null user_config fixes
* test(ocr): cover the form, size, empty body, invalid JSON and file-document 400s
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
Each test below passed or failed on the same commit depending on test order
or port timing in the CircleCI integration workflow
- readiness probe only polls a port the launched proxy owns, so a stolen
port no longer sends GET /health/integration-readiness to another test's
wire server
- team update tests seed fresh member users, since a proxy admin relocation
adds team members to the destination org and leaked org_b membership into
the shared INTERNAL_USER
- realtime whisper tests wait until every proxy worker serves only the new
gpt-realtime-whisper deployment, via a per-worker deployment id route
- configured fallbacks test runs on its own proxy and database, because
/config/update persists router settings and cannot unset fallbacks, which
left [] and changed the no-fallback error text other tests assert
- otel excluded services tests each get a private Redis so a sibling proxy
cannot warm the budget registry cache the tenant span depends on
- Langfuse audit upstream uses unique response ids instead of len(targets),
which could repeat under concurrency and drop SpendLogs rows
* test(cost): add xai grok-4.7, Bedrock runtime chat completions and Bedrock Mantle cost cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): bring back known_bug and carry the xai grok-4.7 stream breakdown bug as a strict xfail case
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): add per-agent a2a_protocol_version override for langgraph-api 0.15 cards
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): cover streamed 0.3 replies through the real client
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): accept new kwargs in create_a2a_client fake; add protocol-version integration tests
The create_a2a_client fake in tests/agent_tests/test_a2a_agent.py listed a
fixed signature, so asend_message's new protocol_version= kwarg raised
TypeError in test_a2a_non_streaming and test_a2a_streaming. Accept and
ignore extra kwargs with **_, matching the new _capture fake in test_main.py.
Add tests/integration/compatibility cells for litellm_params.a2a_protocol_version
against a real 2-worker proxy with Postgres and Redis and scripted A2A peers
modeled on langgraph-api 0.15.4: pinned send/stream on all three routes, aliases,
the 1.0 opt-out, unchanged genuine v1 agents, PATCH taking effect on both
workers, persistence via /v1/agents, the localhost retry, invalid values,
upstream errors, the dialect hint, and the completion bridge stripping the key.
Merge base 5f969982fc: 33 red, 7 green. Head ba709d3de5: 40/40 green twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(focus): add ternary destination for FOCUS export
Add a `ternary` FOCUS export destination (plus a `ternary` callback) so a LiteLLM proxy can push FOCUS-formatted spend to Ternary with `callbacks: ["ternary"]` -- no separate export job
- FocusTernaryDestination: a pure sink modeled on the sibling vantage/mavvrik/gcs destinations; POSTs the FOCUS CSV to the Ternary cost-ingestion endpoint, chunking an oversized backfill (csv-module repartition, quoted-newline safe; oversized single row raises rather than silently dropping) with a stable X-Ternary-Upload-Id + 0-based index/total so the receiver stages-then-swaps. connection_id is validated + URL-encoded; base_url must be HTTPS (loopback exempt for local); the upload has a bounded timeout; it aborts on the first failed chunk so nothing partial lands
- TernaryLogger: presets the FOCUS pipeline (CSV) and enriches the FOCUS Tags JSON with the per-request token counts the shared transformer drops (Ternary weights allocation by tokens); the enrichment is confined to this path and only adds keys. Cadence is daily (default) or interval (validation loops, an explicit interval-seconds required); the export window is day-aligned and drops days before the window start so a whole-day-replace receiver never truncates
- Registration mirrors vantage/mavvrik: ternary callback literal, logging dispatch, scheduler job, and CustomLoggerRegistry entry -- no shared-code changes
- Tests: mocked unit tests only (no live calls), 100% line coverage of both new modules
* fix(focus): warn instead of silently dropping token tags on mismatch
_merge_token_tags returned the frame unchanged when the transformed data
had no `Tags` column or its row count disagreed with the source, so every
token count was dropped with no signal -- the receiver then lands NULLs
that read as "this gateway reports no tokens". Emit a warning on each
degenerate path (mirroring _drop_days_before), and cover both with caplog
tests asserting the warning and the unchanged return.
* feat(focus): upload one UTC day per Ternary signed-URL PUT
Replace the multipart POST with chunking by a per-day signed-URL upload,
matching the Mavvrik destination's pattern. The logger partitions the raw
usage rows on `date` (dropping unparseable dates with a warning), then
transforms, enriches and delivers each day separately; one day's failure
is logged and the remaining days still upload, with a single error raised
at the end. The destination requests a signed URL for the day, validates it
is an HTTPS GCS URL, and PUTs the CSV with exactly the signed headers.
Removes the 30 MiB / 100k-row chunking, the X-Ternary-* upload headers and
the .partN filenames; a re-sent day overwrites its object, so retries stay
idempotent without them.
* fix(focus): re-send the last two whole days so late-flushed spend lands
LiteLLM writes DailyUserSpend in batches, so spend dated yesterday can be
flushed after the 00:05 UTC daily export. That export only covered the
previous UTC midnight onward, and the next day's export moved past it, so
the tail of each day was never sent. Start the window at UTC midnight two
days back: the following run re-sends yesterday whole, overwriting its
per-day object with the complete day.
* fix(focus): lazy-load the Ternary engine, fetch whole days, keep ternary out of the generic factory
Every proxy imported polars at boot through the logger registry, so a proxy without polars failed to start even with no FOCUS callback. The polars-backed engine now lives in its own module and loads on the first export
A non-UTC Postgres session timezone shifted the naive updated_at filter, so the oldest day in each run lost rows and Ternary's whole-day replace made the loss permanent. The Ternary read now starts one day early with no upper bound and keeps rows by their own UTC date
callbacks: ["focus"] with FOCUS_PROVIDER=ternary uploaded a parquet blob of one interval over a whole-day CSV file. The generic factory no longer builds a Ternary destination and the Ternary engine injects its own
* fix(focus): keep only the days a Ternary export window covers
Dropping the updated_at upper bound let a bounded export_usage_data window upload every day through today. Rows are now kept by their UTC date between the window's first day and the day before its end instant
* refactor(focus): leave the shared FOCUS export engine as on main
The Ternary engine sets the base engine's fields itself and passes its own
destination, so focus/export_engine.py and its test match main again.
---------
Co-authored-by: Aaron Sproul <aaron@ternary.app>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(guardrails): add Wingback guardrail integration
Mirrored from #38685 at d73b66544d (net diff against its merge base, applied on main)
* fix(guardrails): dispatch Wingback through the unified guardrail
The proxy only runs a unified guardrail when apply_guardrail is defined on
the guardrail's own class, so WingbackGuardrail now binds the parent's
decorated method in its class body. Chat, messages and responses pre, post
and during call checks reach Wingback again, and each check logs once.
Also types the initializer and config model for the lint gates and uses
monkeypatch.setenv in the unit test
* chore(guardrails): drop the Wingback dispatch comment
The unit and integration hook tests already fail when the apply_guardrail binding is removed, so the comment is redundant under the repo comment rule
---------
Co-authored-by: bsk-wb <bharath@wingback.ai>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(cost): add BridgeParityTestCase with a gpt-6-sol /v1/messages to Responses pair
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): rename BridgeParityTestCase to BridgeCostParityTestCase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): collect BRIDGE_COST_PARITY only from modules that define one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): bill the regional row for inference-profile ARNs that embed a cross-region prefix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep the ARN regional-row regression tests to the touched lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): apply the anthropic fast multiplier to cache cost breakdown
Anthropic fast mode billed totals at provider_specific_entry.fast while the
spend log cost_breakdown kept cache_read_cost and cache_creation_cost at 1x.
Move the speed lookup into a shared helper, multiply it into the billed token
rates used by get_token_type_cost_breakdown, and have the Anthropic calculator
read the same helper so totals and breakdown cannot drift apart.
Production change carried from #44074.
Co-authored-by: kerry <kerry@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): narrow the provider entry instead of defaulting to a dict literal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* test(responses): pin bridged commentary plus final answer as one chat choice
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): pin reasoning, two-message and two-tool-call bridge shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): number bridged chat tool calls 0, 1 instead of all 0
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): fold bridged Responses output into one chat choice
A Chat Completions request bridged to /responses got one choice per
Responses message item, so a reply shaped [message, message,
function_call] put the tool call on choices[1] and clients reading
only choices[0] never saw it. Fold every message text, annotation,
reasoning item and tool call into choices[0]
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop mutable annotations and the extra cast in the choice fold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): remove a cast-ok marker that suppressed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): join folded message texts with a newline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(responses): join folded message texts with no separator so stream and non-stream match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(otel): let a team or key excluded_services override the global setting for its OTel v2 destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format otel excluded_services changes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: use raw regex for excluded_services match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): reject non-string destination callback vars
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): harden destination callback var validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): compare parsed values in callback var conflict check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sagemaker): inference components and OpenAI-compatible embeddings
Send model_id on a SageMaker embedding deployment as the invoke call's
InferenceComponentName, sync and async, and keep it out of the request body.
Add the sagemaker/openai/<endpoint> route, which sends the OpenAI embeddings
request shape ({"input": [...]} plus extra_body fields) and reads
data[].embedding and usage from the response for containers that serve the
OpenAI embeddings API. Plain sagemaker/<endpoint> deployments keep the
Hugging Face shape
* refactor(sagemaker): type validate_environment messages and cover base64 embeddings
* fix(sagemaker): name model_id in inference component errors and reject a non-string model_id on embeddings
* fix(sagemaker): fall back to the error text when SageMaker sends a null message
A ClientError whose Error.Message is null (SageMaker's InternalFailure shape) reached the
inference component hint as None and crashed with a TypeError; the message now falls back
to the error's own text on the chat and embedding paths
* test(sagemaker): audit matrix for inference-component embeddings
* fix(sagemaker): read the error status and text through one shape-checked helper
botocore's ConnectionClosedError carries response=None, so the except blocks that
read e.response.get(...) crashed with an AttributeError and the proxy answered 400
"Invalid request format" for a dropped connection. client_error_details validates
the botocore error shape with pydantic and falls back to 500 plus the error's own
text for any other shape (a missing response, an httpx Response, a plain exception).
* fix(sagemaker): drop the inference component hint from chat and derive the SDK wire expectation from the request
The chat except blocks never see SageMaker's text: HTTPHandler.post raises from
raise_for_status before the handler's own SagemakerError line, so the hint check ran
on httpx's generic "Client error '400 Bad Request' for url" line and could not match on
any chat path. The hint now names litellm.embedding only and the chat blocks keep the
status and text they had, without the dead check. The two OpenAI SDK wire tests pinned
the SDK's default encoding_format; they now expect whatever the SDK sent.
* refactor(sagemaker): meet main's per-rule lint and type ceilings
Main started gating every lint rule on its merge-base count and added LIT015
(frozen pydantic models) after this branch was cut, so the merge with main
pushed LIT001, LIT011, LIT015, BLE001 and reportArgumentType past their
ceilings. The response models are frozen with tuple fields, the OpenAI
embedding response is built with model_copy instead of writing into the
caller's object, the invoke_endpoint arguments are a TypedDict, and
optional_params and extra_body are validated into Mapping[str, object] before
they are read
* refactor(sagemaker): pick the embedding config in one place and stop pinning upstream error text
The openai/, Voyage and Cohere checks lived in both get_model_config and
get_supported_openai_params, so they now share one lookup. Tests compare error
messages against the error's own text, or against botocore's and json's own
wording, instead of copying that wording, and assert whole messages where the
text is ours
* fix(caching): key direct calls on model_id so inference components don't share cached embeddings
* test(sagemaker): name the fake-boto3 SDK test class for what it covers
* test(sagemaker): kill and stop chaos workers through the owned-proxy harness
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A caller hitting /v1/messages with a cache_control mark and an OpenAI
gpt-5.6 or newer deployment behind it lost the mark when the request was
rebuilt as a Responses request, so OpenAI never cached at the boundary the
caller chose. The bridge now rewrites each marked system, user and
tool_result block to prompt_cache_breakpoint on the same block when the
deployment reads breakpoints, keeps the system prompt as a developer
message so its mark survives, and still strips the mark for deployments
that do not read it
The hook's gate becomes public so the bridge and the hook share one
answer for which deployments read breakpoints, and the one-block swap
lives next to the existing breakpoint helper in prompt_templates
* feat(providers): add CoralBricks as a native provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): harden cost map consistency tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): type cost helpers, read cost maps from disk, cover streaming spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): drop Final on loop-rebound names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): add the provider audit cells, the Add Model browser cell and the README row
* test(coralbricks): derive UI spend from model info, type browser rows, always release held upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): release held upstream before the wire server joins its threads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): number forwarded chats with itertools.count instead of a mutable list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(coralbricks): drop retired glm-5.3-flash-fp4 and test against deepseek-v4.1-flash-fast-fp4
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): pin retries, await the respawned worker, read the UI rate from the catalog
The retry cells now set num_retries on the deployment, since the CI rig posts
router_settings.num_retries 0 after readiness and the router default of 2 no
longer applies there. The SIGKILL chaos cell waits for the respawned worker's
startup before the owned proxy is stopped and derives its timeout from the
harness budget. The Add Model spec waits for the catalog the form loads, reads
the expected rate from it, and finds the deployment through /v1/model/info,
which does not paginate
* fix(coralbricks): price the current model names and the new DeepSeek input rate
CoralBricks renamed glm-5.3-fp4 to glm-5.3-fast and deepseek-v4.1-flash-fast-fp4
to deepseek-v4.1-flash-fast (the fp4 names stay as deprecated aliases), and cut
DeepSeek V4.1 Flash input from $0.30 to $0.01 per 1M on 2026-10-08. Rates read
from https://www.coralbricks.ai/api/public/models on 2026-10-10; a live cache
write billed 6014 tokens at 9e-08 each, so the cache-write rate stays input
plus cacheWritePerM.
The test config builders now build new dicts instead of editing the parsed
YAML, and the cost-map test locals carry Final.
* fix(coralbricks): keep pricing for the deprecated fp4 slugs CoralBricks still serves
* fix(coralbricks): list the catalog rows for wildcard deployments and derive the test's input limit from the row
* fix(coralbricks): price the deprecated glm-5.3 and deepseek-v4.1-flash slugs
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ci): wait for SIGSTOP to take effect before handing back a suspended proxy process
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ci): resume a process whose SIGSTOP never lands before failing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(model_hub): surface model_info.description in model group info and Model Hub UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): aggregate model group description without in-place mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): skip background health check DB writes when the latest-row read fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock_mantle): map the OpenAI-shaped context overflow envelope too
Mantle reports context overflow in two envelopes. #37862 covered the
structured `validation_error` one carrying token counts
("prompt tokens (N) exceed model maximum (M)"). Mantle also returns the
OpenAI-shaped body:
{"error":{"code":"context_length_exceeded","message":"Your input
exceeds the context window of this model. ...","param":"input",
"type":"invalid_request_error"}}
That passes the existing `invalid_request_error` gate but matches neither
the token-count pattern nor `is_error_str_context_window_exceeded`'s
substring list, so it fell through to a generic BadRequestError. Clients
such as Claude Code only reactive-compact on a recognised overflow
phrase, so an overflowing request surfaced as an ordinary error with no
recovery.
Return the same normalised "prompt is too long: ..." message for this
envelope when it carries no counts, keeping the numeric wording when the
provider supplies them.
* fix(bedrock_mantle): map the streamed context overflow and keep its 400 on /v1/messages streams
* fix(anthropic_adapter): keep only the context overflow's 400 on streamed /v1/messages so other provider 4xx still fall back
* test(anthropic_adapter): wrap a parametrize line over the 120 char limit
* test(integration): audit Bedrock Mantle context overflow mapping across routes, clients, and fallbacks
* test(integration): move the streamed /v1/messages overflow spend row into a BUG-skipped test
* test(integration): keep every chaos cell in the burst and skip only the missing spend row
* fix(bedrock_mantle): answer 400 on an in-stream context overflow
The Mantle overflow mapper built its ContextWindowExceededError on the
original exception's response, so an overflow that Mantle reports inside
a 200 stream (the chat-mode token-count envelope, which the OpenAI chunk
parser wraps as a 500) kept that 500 and the streamed /v1/messages route
answered HTTP 500 api_error instead of 400 invalid_request_error. The
mapper now builds the error on a 400 response that keeps the original
headers and request, and the mapping test covers an in-stream 500 and 503.
The integration suites pin the streamed /v1/messages overflow to the
pre-content 400 route under a retry budget and to the message_start plus
invalid_request_error event without one, drop the chaos carve-out that
expected no spend row, and give the fallbacks file a timeout mark sized to
the proxy's graceful stop.
* fix(bedrock_mantle): keep the provider's 400 body and match the overflow code only as error.code
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(decisions): bill decisions and systemone from <provider>/decisions/<model> cost map rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): mark decisions pricing rows with mode decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): sync generated API types for decisions mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): update pricing expectations in proxy tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): price classifier calls from decision rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): drop redundant evaluation handler after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): inject pricing row for deployed classifier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): ship the decisions pricing lookup without cost map rows
Restore both cost maps to main. Tests inject their own
<provider>/decisions/<model> rows. The Add Model dropdown hides those
pricing keys and offers mode decisions instead of evaluation.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): expect the Decisions mode option in the Add Model form
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): bill decisions and systemone from _decisions cost map fields
Replace the provider/decisions/model pricing rows with <field>_decisions
fields on the model's own row. cost_per_token overlays them for the
decisions, adecisions, systemone and asystemone call types and prices
through generic_cost_per_token; chat, responses and batch keep the plain
fields. gpt-6-luna gains its Decisions rates (input 1e-07, output 0,
cached and cache-write input 1e-07, 2x above 272k tokens).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): satisfy type gate, schema sync and test_utils schema for _decisions fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): import cost_calculator module to keep basedpyright unknown-variable count flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): drop stray fireworks import from __init__ and annotate test cost rows for the type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover gpt-6-luna decisions and systemone above the 272k tier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): keep the decisions pricing block clear of main's stale azure comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost_calculator): drop stale azure models doc comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): skip failure rows for expired Admin UI session keys on dashboard info routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): mark DB-boundary write assertion as test-quality-ok
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): support Clef-omni audio and video
* fix(decisions): accept Clef-omni media types with parameters and cover the proxy routes
The clip patterns now mirror Cloudflare's input schema, so a data URL or
content_type carrying parameters like audio/webm;codecs=opus is accepted.
The four proxy routes get media tests of their own, the backup cost map
is synced with the new Clef-omni rows, and the docs page moves to the
docs repo as a follow-up.
* fix(decisions): refuse parameterized media-type data URLs and refresh the OpenAPI snapshot
Cloudflare answers 422 to a data URL whose media type carries parameters even though its schema only requires the data:audio/ prefix, so LiteLLM now refuses those with a 400 before the call. The object form keeps passing parameters through, which Cloudflare accepts. The lazy OpenAPI snapshot and the dashboard API types are regenerated for the new audio and videos fields
* fix(decisions): gate media by cost-map flags, treat null clips as none, drop unsupported clips before validation
* fix(decisions): keep the SDK's positional order and gate media flags on evaluation-mode models
* fix(decisions): gate media on both decisions model modes and move the audit cells onto the harness helpers
* test(decisions): cover the Fireworks System One router in the media refusal cells
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): do not inject temperature for Gemini 3+ models when omitted
map_openai_params added temperature=1.0 to every Gemini 3+ request that did not set one, on both the gemini and vertex_ai routes. Google is deprecating temperature, top_p and top_k for Gemini 3+ and says upcoming models will reject them, so this default turns a request that never set temperature into one that will fail
An omitted temperature now falls back to Google's own default, which their docs give as 1.0 for all Gemini 3 models. Explicit temperature, top_p and top_k are still forwarded
Fixes#38663
* test(gemini): annotate sampling regression tests
* test(gemini): drop the injected temperature from Gemini translation cases
The nine Gemini base cases for chat completions, responses and messages send no temperature, so the provider request no longer carries the removed 1.0 default. Each new body was sent to the real Gemini API for gemini-3.5-flash, gemini-3.8-flash and gemini-3.1-pro-preview and returned 200, and the stored provider responses are unchanged
* test(gemini): drop the default-temperature wording from the thinking level test docstring
* test(gemini): cover omitted and explicit sampling on both Gemini routes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(model-prices): add missing OpenAI dated snapshots for deep research, search preview, audio mini and computer use
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): add OpenAI shutdown date to dated snapshots, absorb Together Tev1 cache price removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger e2e-tests after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): correct Perplexity pplx-decider price and add pplx-decider-v1.1-27b
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(fireworks_ai): add Decisions support for accounts/fireworks/routers/rune on /v1/systemone and /v1/decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(fireworks_ai): point the rune cost row at the public Fireworks docs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): patch litellm.callbacks with monkeypatch in the Fireworks cached-cost test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(managed_files): pin ownership checks on batch and fine-tuning create
Tenant B can create a batch or fine-tuning job from tenant A's managed
file id today. These fail on main for acreate_batch input_file_id and
acreate_fine_tuning_job training_file and validation_file
* fix(managed_files): check input file ownership on batch and fine-tuning create
Batch create and fine-tuning create only mapped the managed input file to
provider ids and never checked who owned it, so any key with access to the
model could build a batch or job on another tenant's file and then own the
outputs. Both now run check_file_ids_access like chat and responses do,
which 403s a non-owner and 404s a managed id with no ownership row.
Fine-tuning also checks validation_file, which was never looked at
* test(batches): pin that a pre-call ownership denial stops every batch create branch
LIT-4797 noted an endpoint-level check missed the load-balanced branch
because it dispatches first. The ownership check lives in the pre-call
hook, which runs before branch selection, so a 403 there must keep both
the load-balanced and unified-file branches from reaching the router
* test(integration): pin managed-file ownership on batch and fine-tuning create
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(managed_files): check ownership before the batch rate limiter reads the input file and keep key-owned resources across key regenerate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): build worker stacks concurrently and record proxy boot phases
The controller built every xdist worker's database, Redis, upstream and gateway
proxy one after another before any test ran, about 95 s per extensions shard,
and since every multi-worker suite is now isolated the same serial wait hits
mcp, security and cost too. The stacks now start on a thread pool bounded by
the CPU count. Each stack reads its endpoints and results directory from an
explicit environment instead of a patched os.environ, every stack that started
is registered for cleanup, and a failure in one stops the others before the
error is raised
Owned proxy startup.json now records the wall-clock spawn time, the worker
count and when the Prisma CLI child ran, when the port started accepting and
when the first worker answered readiness, and exit.json records when the stop
started and ended, so the boot time can be split and overlapping gateway boots
can be seen in the artifacts
* test(integration): boot one uvicorn worker where a test does not assert cross-worker behaviour
A 2-worker owned proxy takes about twice as long to become ready as a 1-worker
one, because the uvicorn parent imports the app and each spawned worker imports
it again. The passthrough upstream error visibility cases, the single-request
S3 dedupe cases and the single-request spend log payload cases send one or two
requests and assert on the response, the proxy log and the spend row, none of
which depends on a second worker, so they now boot the default single worker
Cases that do rely on two workers keep them, the concurrent S3 burst through a
sink outage, previous_response_id replay, response cache hits, the concurrent
spend bursts and upstream stop and recovery
* test(observability): settle OTel excluded-services traces by a quiet period instead of a full window
_trace_spans slept through its whole 15 or 30 s window on every call even when
the trace had long stopped growing. It now polls and returns once the trace
has spans and has not gained one for five tenant export intervals, and the
module pins the tenant batch exporter's schedule delay to 1 s so that bound is
set by the test rather than by the SDK default. Absence checks still run only
after the trace stopped growing, and against a build that leaks postgres spans
to the tenant they still fail
These cases also boot one uvicorn worker now. None of them asserts anything
about a second worker
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* test(integration): give chaos and timing tests a harness API and gate raw signals
Chaos tests killed and suspended proxy workers with raw os.kill and psutil
calls and bounded elapsed time with fixed literals, so they raced the uvicorn
worker healthcheck and failed on slow CI boxes
OwnedProxy now offers kill_worker and replacing_worker, which return only once
a replacement answers readiness with its own pid, and suspended, which is
refused unless the proxy was launched with a lenient worker healthcheck.
timing.py bounds durations against a measured control or the injected delay.
run.py measures the proxy import time once and derives readiness deadlines and
the per-test timeout from it, writing boot-calibration.json
check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time
asserted under a numeric literal) for tests/integration outside _support, and
every call site is migrated so both report zero with no suppressions
* test(integration): steer the assistants wire burst onto both workers
The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split
* test(integration): restore product-enforced timing bounds lost in the migration
Some TQ011 migrations dropped or loosened a bound whose time is the behaviour
under test. The websocket session cap only kept its floor, so a cap closing at
300 s would have passed, and several timeouts were bounded by the injected stall
instead of the configured product value
assert_enforced_within bounds an elapsed time by the configured value plus
explicit slack, with an optional floor. The websocket cap, the MCP validation
timeout, the guardrail litellm_params timeout, the prisma command timeouts, the
baseline projection lock timeout and the key metadata statement timeout use it.
The device code refusal keeps its 15 s margin, and the langfuse retried prompt
fetch is now bounded by a cold fetch of a steady prompt as well as by the
ignored retry-after
* fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations
The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat
in a ping on the killed worker for up to that long before respawning it. It now uses
the graceful stop budget, and a worker kill returns only after the victim is reaped,
so pooled connections no longer race the dead worker's socket.
Several tests on main picked the multiprocessing resource tracker as their victim and
never killed a worker. Now that they kill a real one, presidio counts streams cut by
the death, delete_model waits for the config reload that removes a deleted model from
a replacement worker, and the wedged coordination Redis test kills before the wedge
and waits for every parked publish to fail before resuming Redis.
The redis cluster test imports the harness through the tests package so it collects
in the unit job, and the per-model fallback rig exposes the owned proxy it now needs.
* test(observability): only require S3 deliveries the killed worker could not have buffered
A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* test(integration): isolate every multi-worker integration suite by default
Per-worker database, Redis, upstream and proxy were opt-in for management,
accounting, providers and extensions. mcp, security and cost ran 4 to 8
xdist workers against one shared proxy, database and Redis, and those suites
hold most of the integration flakes CircleCI records. Every run with more than
one worker now builds a private stack per worker, run_integration.sh no longer
starts or exports the shared proxy, upstream, database or Redis in that mode,
and each worker's scripted upstream runs in its own process instead of a
thread in the pytest controller.
Worker gateways get what the shared proxy used to provide: a peer for
management and mcp, the scripted cost map for cost, stdio MCP servers, and
coverage for mcp. mcp coverage now measures lines through sys.monitoring,
because branch measurement on CPython 3.12 uses the trace function and that
made a covered proxy take about 4x as long to become ready, which is why the
shared mcp proxy kept missing its 90 s readiness wait. Routing parity
snapshots now also read the worker databases, since its cost and mcp runs use
several workers. The security per-worker fixture is redundant and removed
* test(integration): give the nested probe root the files a worker gateway needs
Two worker probes now run isolated, so the nested controller boots a real per-worker upstream and proxy from the probe root. That upstream imports the fake OpenAI server and the cost map modules, and the proxy reads proxy_config.yaml, none of which the probe copied. The nested run also drops INTEGRATION_COVERAGE_FILE so it never writes into an outer coverage file
* test(integration): budget nested multi-worker runs for their worker stack boots
A nested run.py database run with more than one worker now boots a private database, Redis, upstream and proxy per worker, one after another, before any case runs. The fixed 60 s subprocess timeout left no room for that, so the 2 worker probes sat at 57-60 s on CircleCI. The subprocess budget is now the old 60 s plus 30 s per worker stack, and the outer pytest timeout keeps its previous 30 s of slack on top. 30 s is the slowest worker proxy readiness recorded in startup.json on the integration-cost job of workflow e09b108b (21.6-30.3 s on 4 CPUs, 2026-10-12)
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
The redis-compat matrix and mcp-integration ran unit tests on five large
CircleCI machines per integration run. redis-compat now runs as a redis-py
matrix in the unit workflow with the same pinned redis-server 7.2.16 binary,
no reruns, and the redis-compat Codecov flag on the 5.3.1 leg. The MCP unit
directories not already claimed by mcp-elicitation and mcp-oauth run in a new
mcp-server shard whose report is also uploaded under the mcp-integration flag.
The Windows release wheel job installs Python from a checksum-verified
python.org installer with a bounded wait instead of Chocolatey, helm chart
testing waits on the rollout instead of sleeping, and roi-database no longer
checks out the repository twice.
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* test(integration): migrate before timed owned proxy launch and gate credential probe on every worker
* test(integration): retry closed-connection samples and require both workers in the credential gate
---------
Co-authored-by: yuneng <yuneng@berri.ai>
