* test(integration): drop the contracts.json manifest and the covers requirement
Groups live as a GROUPS literal in run.py, the browser expectations move next to the
browser tests, and the runner fails only on pytest failure, collection errors or a
selected file that collects zero tests. The covers marker stays registered for the
existing tests but is no longer checked. The mcp directory gets its own group
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): run mcp as its own shard with xdist and a peer proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): INTEGRATION_COVERAGE=1 runs the proxy under coverage for the MCP modules
The mcp shard sets it. The proxy and its peer start under coverage run in parallel mode,
get SIGTERM after the tests so coverage flushes, and the combined text and HTML reports
land in the suite results that CircleCI already stores as artifacts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(integration): let the test proxy flush coverage when uvicorn re-raises SIGTERM
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add SSE, stdio, scripted, OpenAPI and OAuth 2.1 MCP peer doubles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add MCP transport and access-control matrices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add MCP credential and OAuth flow coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add MCP LLM endpoint, accounting, guardrail, resilience and lifecycle coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stop the same-URL grant test from counting a late initialize as a leaked call and satisfy the test-tree lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the REST denied-server listing is refused or empty
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin the REST denied-server listing to 403 access_denied
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(e2e): move the compat-matrix populator from a GCE VM to a Render cron job
The daily Claude Code compatibility-matrix job ran as a systemd timer on
the litellm-compatibility-matrix-populator VM in the vertex-check GCP
project. Replace that with a Render Docker cron job built from a new
Dockerfile in tests/e2e/claude_code/cron_vm: pinned and checksummed
debian base, gh, uv, and Claude Code CLI, a non-root populator user, and
run_daily.sh as the entrypoint. run_daily.sh now clones a fresh blobless
checkout per run (Render cron disks are ephemeral), reads the publish PAT
from the github-token secret file under CREDENTIALS_DIRECTORY, and its
comments no longer describe systemd. The .service and .timer units are
gone; README.md and the env example describe the Render service, its
secret files, and the local docker build instead.
* docs(e2e): name the plan and trigger route Render's cron-job API accepts
Render answers a bare 404 for the legacy pro_max plan name on a cron job
(4c-16g is the same 4 CPU / 16 GB size) and the manual trigger route is
/v1/cron-jobs, not /v1/cronjobs.
* fix(e2e): install the published litellm wheel instead of building the tag from source
The tag builds a Rust extension through maturin, which needs a C and Rust
toolchain the cron image does not carry, so the first Render run failed at
uv sync with "linker cc not found". Sync the locked dependencies with
--no-install-project, install the PyPI wheel (what users run) with
--no-build, and pass --no-sync to every uv run so uv never puts the source
build back.
* fix(e2e): keep the SKIP_PUBLISH matrix where a Render run can read it
The validation run wrote the matrix into the image checkout, which nobody
can read once the container exits. Save it under HOME and print it at the
end of the log instead.
* fix(e2e): let the stale compat-matrix PR sweep see past the newest 100 docs PRs
The docs repo has a few hundred open PRs, so a 100-item list never
reached the week-old compat-matrix PR and the sweep left it open on
every run.
* docs(e2e): say the Render cron needs a manual deploy after each merge
Pushes never started a deploy during setup because Render only hears
about them through its GitHub app, which the org does not have, so the
README now carries the deploy command and the wait-for-live rule
* ci: build the compat-matrix cron image on pull requests
The CI coverage gate requires every Dockerfile to be built by a job, and
building this one on each PR that touches it also catches a broken pin
or checksum before Render does
* fix(e2e): shim the whole tests/e2e tree into the compat-matrix worktree
The five-file helper allowlist missed fixture_mode, which e2e_config now
imports, so the first Render run died at conftest load with
ModuleNotFoundError. Copy the image's whole tests/e2e tree instead and
keep pytest from loading the EKS-harness conftest with --confcutdir
* fix(e2e): scope the compat-matrix sweep to the publishing account's own PRs
The stale-PR sweep selected every open docs PR whose head branch starts
with compat-matrix/, so a contributor's fork PR under that name would
have been closed once a newer matrix PR existed. The sweep now resolves
the publishing login from the token and only closes same-repo PRs that
account opened
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(logger): add shared Rust diagnostics and Python logging bridge
* feat(logger): dispatch diagnostic processing through Rust
* chore: regenerate Cargo.lock after rebase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: allowlist bounded logging tree walkers in recursive detector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(logger): skip decoding plain access arguments
* test(logger): skip embedded-python logger test when litellm deps are absent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: cargo fmt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: expect NativeDiagnosticProcessor in the native public surface
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(stub): export NativeDiagnosticProcessor via __new__ in _native.pyi
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tracing): rename logger crate and document host sink contract
* test(logger): cover exc, stack, and nested extras in the diagnostic filter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logger): keep rendered redacted line when template scan flags a key pattern
The blanket REDACTED for a changed msg/color template discarded lines
whose rendered form was already redacted by the same pipeline, e.g.
'password=%s' became 'REDACTED' instead of 'password=REDACTED'. Only
fall back to REDACTED when the rendered form did not change either,
which is where interpolation can mangle the key pattern the scrub
would otherwise see.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(rust): install python deps so the logger bridge test runs
The end-to-end bridge test skipped silently when litellm's Python deps
were absent. uv sync --no-install-project installs them without a
maturin build, and PYTHONPATH makes them visible to the embedded
interpreter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ci): close open pull requests superseded by a merged fix on their linked issue
* fix(ci): replace the mixed-anchor release-line regex with a plain predicate
* fix(ci): recover a half-done close, page linked pull requests, trust only the workflow's marker
* fix(ci): close superseded pull requests whatever closed the issue and fail the job when the script throws
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): add a HashiCorp Vault secret manager lane
key_management_system had no end-to-end coverage: the Rust crates and the
Python unit tests all run against mocked managers. This adds a secret_manager
suite that drives a proxy configured with hashicorp_vault against a real Vault.
The tests seed a fresh secret name per test with the runner's OPENAI_API_KEY and
register a deployment pointing at os.environ/<name>. The proxy's env never holds
that name, so get_secret's os.environ fallback cannot mask a broken manager, and
a bogus value in Vault must come back as the provider's 401. Virtual keys are
checked written to and removed from Vault under prefix_for_stored_virtual_keys.
The setting is global to the proxy, so the lane has its own config and the
secret_manager_vault opt-in marker, and stays out of the per-PR selector.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): make the secret manager suite backend-agnostic
One marker and opt-in (secret_manager / E2E_SECRET_MANAGER=<system>) pick the
backend from secret_backends.BACKENDS. The tests reach the manager through a
SecretStore protocol, and each backend contributes a secret_store_<system>.py
module, a registry entry, and gateway/secret_manager_<system>_ci_config.yml.
requires_capability deselects tests a backend cannot support (CyberArk does
not delete), and test_secret_backends.py checks every lane config against its
backend without a live stack.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): add a CyberArk Conjur secret manager lane
Adds cyberark as the second secret_manager backend: a Conjur store over its
REST API (policy-declared variables, raw-text values, policy-patch teardown),
its lane config, and a registry entry without deletes_stored_keys, since the
proxy's CyberArk delete answers not_supported and Conjur keeps the key.
secret_manager/backend.sh up|down <system> boots any backend in Docker and
writes proxy.env and tests.env, so every lane runs the same way; the registry
test checks the script boots exactly the registered backends. e2e_http gains
send_text_external for APIs that speak raw text rather than JSON.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(e2e): give the secret manager suite a client with .proxy and address review
The shared resources fixture reads client.proxy, so a bare ProxyClient errored every
live test at setup. backend.sh now writes its env under a per-user directory with
umask 077, the markerless unit tests are gone per tests/e2e/AGENTS.md, and routine
comments are trimmed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Eight directories the misc shard named moved to tests/unit on 2026-09-20, and one
missing path makes pytest-xdist collect [0 items] for the whole shard, which the
exit-5 tolerance turned into a green required check running nothing. The shared
Run tests step now drops a path that does not exist with a :⚠️: and runs
pytest over the rest, keeping option tokens verbatim.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(otel): honor SSL_CERT_FILE and ssl_verify in OTLP HTTP exporters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): assert OTLP HTTP TLS behavior against a real TLS sink
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): assert rejected exports by outcome, not by exception type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): honor SSL_CERT_FILE and ssl_verify in the v2 OTLP HTTP exporters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): hoist otlp_tls imports and type the TLS sink fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): gate the OTLP TLS export test behind an otel_tls opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(e2e): drop CONTRIBUTING.md edit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(langsmith): json.dumps with default=str so non-serializable metadata does not crash batch flush
Serialize the runs/batch payload with json.dumps(default=str, allow_nan=False) and send it as content= with an explicit Content-Type, so datetime, Decimal and similar metadata values no longer raise TypeError and drop the batch. Forward content= on the AsyncHTTPHandler retry path so a retried batch re-sends the identical body
Replaces #39133, which was cut from the retired staging branch and conflicts with main
Co-authored-by: Damien Smrt <dsmrt@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): drop test docstrings and replace monkeypatch with a client-injecting handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): add live e2e for non-native metadata reaching LangSmith
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): scope the e2e docstring to the values the test injects
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(http_handler): close injected retry clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): deselect the LangSmith live e2e on the stage-mirror stack
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Damien Smrt <dsmrt@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(e2e-stack): print add-mask lines only under GitHub Actions
* refactor(e2e-stack): inline the add-mask lines into main
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(otel v2): map completions, images, speech, transcription and moderation output onto the generation output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(redaction): redact text completion choices in the standard logging payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): compare decoded generation output text and follow the live moderation verdict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel v2): compare logged byte counts with the received media and move e2e schemas into models.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): keep the otel_v2 Langfuse output e2e file out of the stage-mirror gate it cannot run in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: benchmark and gate an installed release wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: simplify installed-wheel benchmark check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add native tokenizer codec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): route Python tokenization through the Rust extension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(lint): format tokenizer call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(packaging): restore runtime dependencies and native images
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve Python SDK behavior with Rust tokenizers
* fix(tokenizer): restore compatibility paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): count custom tokenizers directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve caller-supplied Python tokenizer counts
* fix(tokenizer): reuse packaged vocabularies in the native wheel
* refactor(rust_bridge): route token counting through the catalog as RUST_OPT_IN
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): compare tokenizer groups by value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): re-resolve filelock under the <4.0 pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): align transformation override signatures with base configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): use fat LTO to keep the native wheel under the 35 MB limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): preserve Python defaults with opt-in Rust dispatch
* test(proxy): tolerate missing litellm.utils.Tokenizer when patching it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): patch the tokenizer dispatch function instead of the removed alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): give the Rust wrappers the tiktoken and tokenizers surface
Callers of litellm.encoding and litellm.create_tokenizer must see the same
read-only API whichever backend the catalog selects.
- OpenAIEncoding mirrors tiktoken.Encoding: n_vocab, max_token_value,
token_byte_values, encode_single_token, encode_with_unstable,
encode_to_numpy, decode_with_offsets, is_special_token, repr; the Rust
tiktoken crate keeps a Vocabulary beside each CoreBPE and reports the
requested encoding name (gpt2 stays gpt2).
- HuggingFaceTokenizer mirrors the read-only tokenizers.Tokenizer surface
(token_to_id, id_to_token, get_vocab, get_vocab_size,
get_added_tokens_decoder, num_special_tokens_to_add, padding, truncation,
encode_special_tokens, from_buffer); HuggingFaceEncoding gains the
char/word/token lookups, pad, truncate, set_sequence_id and merge.
Mutators stay on the Python tokenizer.
- from_json/from_pretrained claim the fork gate only when the huggingface
feature is compiled in; the surrogate fallback matches on the Codec.
- Tokenizer caching is keyed on the same catalog Context the dispatch runs
on; rust_tokenizer reads the encoding name without loading an encoding;
LITELLM_RUST parsing is cached.
- Drop the unused tiktoken_encoding_for_model export and Error::Download.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tokenizer): close the exhaustive matches with assert_never
CodeQL reads a `match` over a Literal with no default arm as an implicit
`None` return. `assert_never` makes the exhaustiveness explicit for both the
HuggingFace tokenizer loader and the Rust token-counter factory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tokenizer): derive the fast counter from the shared tokenizer
The count-only counter (`fast` feature) and the codec each parsed the same
artifact: TokenCounter took the Anthropic JSON and the tiktoken rank files
from Python while Tokenizer loaded them again. One parse now serves both.
- FastTokenizer builds from a model another loader holds: `from_shared`
takes the Arc<tokenizers::Tokenizer> the HF codec keeps, and
`from_*_pairs` take the ranks the tiktoken vocabulary already parsed.
- `FastCounter::fast_counter` in the core crate derives it from either codec;
encodings the fast scanner does not reproduce are refused.
- Native `Tokenizer.count(text, fast=False)` opts into that counter, built
once per tokenizer on first use; `TokenCounter.from_tokenizer(tokenizer,
fast=False)` replaces the JSON and rank-file constructors.
- The Python route counts over the native tokenizers the codec path shares
(`native_encoding`, `native_anthropic`) and no longer reads rank files;
the packaged Anthropic tokenizer has one loader, `tokenizer_dispatch.anthropic`.
- Public wrappers gain `count(text, fast=False)`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The Buildkite ephemeral stack runs the gateway in another pod, so it cannot reach the pytest host's provider edge. The GitHub changed-e2e lane runs gateways on the runner and sets E2E_PROVIDER_EDGE_HOST_REACHABLE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The changed-test gate booted its stage-mirror stack without files_settings
or finetune_settings, so every raw upload with a custom_llm_provider hit a
500, and it exported the whole provider env into the gateways, so the
AWS_ROLE_NAME the assume-role test needs made the GovCloud deployment run
an AssumeRole with its static keys. The gate also deleted its pytest output,
so a red run left nothing to read. The mirror config now carries the
openai, azure, and vertex_ai file settings, gateways start without
AWS_ROLE_NAME, and the workflow uploads the pass logs and junit files with
every secret value, every field of a JSON-valued secret, and their
XML-escaped forms replaced before the raw files are removed.
