mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-08 03:08:45 +00:00
fix(lens): retry key propagation and validate isolated Compose setup
This commit is contained in:
moe-berri
parent
fc7707f168
commit
15599ad919
7 changed files with 191 additions and 114 deletions
|
|
@ -1,112 +1,114 @@
|
|||
# Lens worker
|
||||
# Lens service
|
||||
|
||||
Lens reviews recorded activity and saves evidence-linked findings in the LiteLLM dashboard under Observability, Lens (`/ui/lens/`)
|
||||
Lens records agent activity and investigates it in a separate Rust service. LiteLLM serves model requests, the dashboard, and investigation settings. Lens owns trace ingestion and ClickHouse access; PostgreSQL stays with LiteLLM
|
||||
|
||||
## Install
|
||||
Agent exporters send traces directly to Lens. LiteLLM sends its optional request logs through a bounded background queue. If Lens or ClickHouse is unavailable, model requests continue; traces can be delayed or dropped according to the exporter's retry policy. The gateway never waits for ClickHouse during startup or inference
|
||||
|
||||
Build LiteLLM and its worker from the same source commit with the same release identity. The worker runs separately and connects to your gateway using a limited worker token
|
||||
## New local installation
|
||||
|
||||
### New local installation
|
||||
|
||||
Install Docker with Compose and Git. This builds LiteLLM and its worker from the same checkout and starts the existing local tracing stack:
|
||||
Install Docker with Compose and Git, then build the gateway and Lens from one checkout:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/BerriAI/litellm.git
|
||||
cd litellm
|
||||
export LITELLM_RELEASE_TAG="sha-$(git rev-parse HEAD)"
|
||||
export LENS_WORKER_IMAGE="litellm-lens-worker:${LITELLM_RELEASE_TAG}"
|
||||
export OPENAI_API_KEY='sk-...'
|
||||
docker build --build-arg LITELLM_RELEASE_TAG="$LITELLM_RELEASE_TAG" \
|
||||
-f deploy/lens/Dockerfile -t "$LENS_WORKER_IMAGE" .
|
||||
export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 24)"
|
||||
export LITELLM_LENS_SERVICE_TOKEN="$(openssl rand -hex 32)"
|
||||
export OPENAI_API_KEY='<your-provider-key>'
|
||||
docker compose -f docker/docker-compose.tracing.yml up -d --build
|
||||
```
|
||||
|
||||
Open `http://localhost:4002/ui/` and sign in as `admin` with the key saved in `.lens-dev/master_key`. Go to **Lens > Investigations > Connect worker**, choose a model and monthly budget, then **Get install command**. Expand **Using Docker Compose or Helm?** and copy the worker token. In the same terminal, run:
|
||||
Save the generated keys privately and reuse them when restarting or upgrading. This stack binds to localhost and uses development database passwords; use your normal secrets, TLS, backups, and ingress for a hosted deployment
|
||||
|
||||
```bash
|
||||
export LITELLM_URL=http://litellm:4000
|
||||
export LENS_WORKER_TOKEN='<paste-your-worker-token>'
|
||||
docker compose -f docker/docker-compose.tracing.yml -f deploy/lens/compose.yaml up -d
|
||||
```
|
||||
Open `http://localhost:4002/ui/` and sign in as `admin` with `LITELLM_MASTER_KEY`. Under **Lens > Traces > Set up tracing**, generate a tracing key and copy the ingestion URL. Local exporters use `http://localhost:4318`. Model calls keep their existing LiteLLM URL and model key
|
||||
|
||||
The worker joins the gateway's Docker network, and the dashboard shows **Worker connected**. Save the token privately for restarts and upgrades
|
||||
Under **Lens > Investigations > Connect worker**, choose an analysis model and monthly budget. The deployed service connects automatically after you save these settings. There is no worker command or second token to copy
|
||||
|
||||
This stack is for local evaluation: it binds to localhost and uses development database credentials. For a hosted deployment, keep your normal database, keys, networking, and deployment process. Build both images from one source revision with the same `LITELLM_RELEASE_TAG`, publish the worker to your registry, and set `LENS_WORKER_IMAGE` on LiteLLM to that image
|
||||
## Existing LiteLLM installation
|
||||
|
||||
### Existing LiteLLM installation
|
||||
Keep your gateway, PostgreSQL database, deployment tool, and existing encryption keys. Deploy the matching Lens image, give it access to ClickHouse, and configure the service connection on LiteLLM
|
||||
|
||||
Keep your deployment and PostgreSQL database. A working gateway/worker pair can stay as it is until you upgrade both. For a gateway built from source, use its exact commit and `LITELLM_RELEASE_TAG`; a release version or the latest commit on `main` is not a substitute for that source identity
|
||||
| Variable | LiteLLM | Lens service |
|
||||
| --- | --- | --- |
|
||||
| `LITELLM_LENS_SERVICE_TOKEN` | Same private random secret, at least 32 characters | Same secret |
|
||||
| `LITELLM_LENS_URL` | Internal Lens URL, such as `http://lens-worker:4318` | Not needed |
|
||||
| `LITELLM_LENS_PUBLIC_URL` | Ingestion base URL reachable by your agents | Not needed |
|
||||
| `LITELLM_URL` | Not needed | LiteLLM URL reachable from Lens |
|
||||
| `CLICKHOUSE_URL` | Remove it from Lens tracing configuration | ClickHouse HTTP URL with credentials |
|
||||
| `CLICKHOUSE_DATABASE` | Not needed for Lens | Existing database name, defaults to `litellm` |
|
||||
| `AGENT_TRACING_RETENTION_DAYS` | Not needed for Lens | Retention for traces and Lens request logs, defaults to `14` |
|
||||
|
||||
The public development package is `ghcr.io/berriai/litellm-lens-worker-dev:sha-<full-commit>`. It publishes amd64 images on Lens-related changes, so an arbitrary source commit may have no image. Check the exact image exists before using it. If it is unavailable, your gateway uses a different release identity, or you need native arm64, build the worker from the gateway's checkout:
|
||||
Remove the old `general_settings.tracing.store` configuration used for Lens from LiteLLM. Keep unrelated logging integrations and their configuration. Only Lens should reach its ClickHouse database. The shared service secret is an infrastructure credential: keep it out of browser code, agent exporters, screenshots, and public ingress headers
|
||||
|
||||
Expose the Lens HTTP listener on port 4318 through TLS. Route `/lens-ingest` on your existing hostname directly to Lens at the load balancer, then set `LITELLM_LENS_PUBLIC_URL=https://<your-host>/lens-ingest`. The gateway must not proxy these uploads. Alternatively use a separate hostname and forward `/v1/` to Lens. Keep `/internal/` private; it requires the service secret
|
||||
|
||||
### Standalone Docker or a container host
|
||||
|
||||
Build from the same source commit and `LITELLM_RELEASE_TAG` as your running gateway:
|
||||
|
||||
```bash
|
||||
export LITELLM_RELEASE_TAG='<gateway-release-identity>'
|
||||
export LENS_WORKER_IMAGE='<your-registry>/litellm-lens-worker:<your-image-tag>'
|
||||
export LENS_WORKER_IMAGE='<your-registry>/litellm-lens-worker:<image-tag>'
|
||||
docker build --build-arg LITELLM_RELEASE_TAG="$LITELLM_RELEASE_TAG" \
|
||||
-f deploy/lens/Dockerfile -t "$LENS_WORKER_IMAGE" .
|
||||
```
|
||||
|
||||
For a remote worker host, publish that image to a registry the host can pull from. Set the gateway's `LENS_WORKER_IMAGE` to the resulting image reference, restart the gateway using its normal deployment process, then copy its install command. Prefer the published image digest for hosted installations. Do not change the gateway's release identity just to accept another worker
|
||||
Publish that image to a registry your host can pull from. Prefer a digest reference for hosted deployments. Public development images use `ghcr.io/berriai/litellm-lens-worker-dev:sha-<full-commit>`; check that the exact image exists before selecting it. An arbitrary commit may not have a published image
|
||||
|
||||
For Kubernetes or Render, run the standalone worker using `LITELLM_URL` and `LENS_WORKER_TOKEN` from setup. Keep existing databases and secrets. The worker needs no inbound port.
|
||||
The image supports native amd64 and arm64. For worker-only Compose, use `deploy/lens/compose.yaml` with a private environment file containing `LENS_WORKER_IMAGE`, `LITELLM_URL`, `LITELLM_LENS_SERVICE_TOKEN`, and `CLICKHOUSE_URL`:
|
||||
|
||||
## Helm
|
||||
```bash
|
||||
docker compose --env-file /path/to/private/lens.env \
|
||||
-f deploy/lens/compose.yaml up -d
|
||||
```
|
||||
|
||||
The componentized source chart at `helm/litellm` includes an optional Lens worker. Use the chart from the same checkout as your gateway and keep your component image overrides in your values. Configure PostgreSQL and ClickHouse as usual, install the chart, then obtain a limited worker token from Lens setup. Store it in a Kubernetes Secret and enable the worker in your values:
|
||||
The Compose listener binds to localhost. Your reverse proxy must reach it. On Render, run Lens as a web service with the same environment and listener port 4318, not an outbound-only background worker. Use `/health/live` for process health and `/health/ready` to check storage and tracing credentials
|
||||
|
||||
Lens does not need provider credentials, PostgreSQL credentials, a GPU, or the LiteLLM Python package. The image includes a small CPython runtime only for the investigator's confined calculation tool. Keep the shipped security settings, temporary filesystem, and resource limits
|
||||
|
||||
### Kubernetes with Helm
|
||||
|
||||
Both `helm/litellm` and `helm/litellm-helm` support the Lens service. Keep your existing release, namespace, values, and database configuration. Create two Secrets through your normal secret manager: `litellm-lens-service` with key `service-token`, and `litellm-lens-clickhouse` with key `url`
|
||||
|
||||
```yaml
|
||||
lensWorker:
|
||||
enabled: true
|
||||
image:
|
||||
repository: <your-worker-image-repository>
|
||||
repository: <matching-worker-image-repository>
|
||||
digest: sha256:<matching-worker-image-digest>
|
||||
tokenSecret:
|
||||
name: litellm-lens-worker
|
||||
key: token
|
||||
serviceTokenSecret:
|
||||
name: litellm-lens-service
|
||||
key: service-token
|
||||
clickhouseSecret:
|
||||
name: litellm-lens-clickhouse
|
||||
key: url
|
||||
publicUrl: https://<your-litellm-host>/lens-ingest
|
||||
```
|
||||
|
||||
Set the worker repository and digest explicitly to an image built from the gateway's source commit and release identity. The chart connects the worker to the backend service. Keep these values and the Secret when upgrading the chart and update the gateway and worker image overrides together. `lensWorker.replicaCount` controls simultaneous investigations. To use a private registry or external proxy, set `lensWorker.image.repository`, `lensWorker.image.digest` (or `tag` for a source build), and `lensWorker.url`. A digest takes precedence over the tag. The dashboard uses the chart's worker image for standalone install commands too
|
||||
When the chart's main ingress is enabled, it routes `/lens-ingest` directly to Lens. With a custom ingress, add that route yourself. For a dedicated hostname, use `lensWorker.ingress.enabled`, `host`, `className`, and `tls`, and set `publicUrl` to that hostname. The chart connects LiteLLM to Lens internally and gives both services the shared secret
|
||||
|
||||
## Standalone worker
|
||||
|
||||
Start with a source deployment that includes Lens, PostgreSQL, and agent tracing, and prepare its matching worker as described above. Configure one ClickHouse URL for trace writes, bounded reads, and Lens queries:
|
||||
|
||||
```yaml
|
||||
general_settings:
|
||||
tracing:
|
||||
store:
|
||||
type: clickhouse
|
||||
url: os.environ/CLICKHOUSE_URL
|
||||
retention_days: 14
|
||||
```
|
||||
|
||||
The URL, database, and retention settings can also come from `CLICKHOUSE_URL`, `CLICKHOUSE_DATABASE`, and `AGENT_TRACING_RETENTION_DAYS` when omitted from YAML. A YAML value wins when both are set. The database defaults to `litellm`. `retention_days` defaults to 14 and applies to both traces and spend logs
|
||||
|
||||
Retention changes require a proxy restart. ClickHouse removes expired rows during background merges, not immediately at startup. Enable request/response logging to analyze LLM requests. Lens can only inspect content you actually retain
|
||||
|
||||
In **Lens > Investigations**, click **Connect worker**, choose an analysis model and monthly limit, then **Get install command**. Use **Advanced options** to select an existing virtual key or change the proxy URL if the server running Docker needs a different network address. Copy the command and run it on your server. The dashboard shows **Worker connected** when the container checks in
|
||||
|
||||
The command already contains the compatible worker image and one worker token. The selected virtual key stays on the proxy; its secret is never sent to the worker. Once the matching image is available on the worker host, no second LiteLLM deployment is needed. Keep the command private because it includes the token. The LiteLLM release provides the dashboard and APIs; the container only runs background analysis
|
||||
|
||||
The dashboard uses the gateway's `LENS_WORKER_IMAGE` override when set. Public `:sha-<commit>` development images must match both the gateway commit and release identity. Build from source for the worker host's native architecture
|
||||
|
||||
After upgrading the gateway, update the worker image and redeploy it while keeping its proxy URL and token. Existing containers do not update automatically. If an investigation reports a worker compatibility error, update the image before retrying
|
||||
|
||||
For deployments managed with Compose, download `compose.yaml` and provide `LITELLM_URL`, `LENS_WORKER_TOKEN`, and an explicit `LENS_WORKER_IMAGE` in a private environment file:
|
||||
Update your existing component image overrides to matching builds, then use the chart from that checkout:
|
||||
|
||||
```bash
|
||||
docker compose --env-file /path/to/lens.env -f compose.yaml up -d
|
||||
helm upgrade --install litellm ./helm/litellm \
|
||||
--namespace litellm -f values.yaml --wait
|
||||
```
|
||||
|
||||
To work on Lens itself, `make lens-dev` runs the proxy, a worker from source and the hot-reload dashboard together; set `LENS_DEV_PROXY_PORT` / `LENS_DEV_UI_PORT` to move them off 4000/3000. For a local container build, set `LENS_WORKER_IMAGE=litellm-lens-worker:local` and `LITELLM_RELEASE_TAG` to the gateway's release tag, then use `docker compose -f deploy/lens/compose.yaml -f deploy/lens/compose.build.yaml up -d --build`
|
||||
Use `./helm/litellm-helm` if that is your existing chart. `lensWorker.replicaCount` scales ingestion and investigations. Each replica needs access to the same ClickHouse and gateway. Credentials refresh every 30 seconds; a newly created key may briefly receive a retryable 429. Revocations propagate on refresh, and a replica stops accepting traces when its credential snapshot reaches 90 seconds
|
||||
|
||||
The generated command gives the worker 1 GiB of temporary memory-backed storage, shared across parallel reviews. Change `size=1g` in the Docker command or set `LENS_WORKER_TMP_SIZE` with Compose to fit your server and workload. Python reports storage failures to the reviewer and cleans up temporary files, so the reviewer can retry a smaller computation or report insufficient evidence. The worker remains available for other scans. Existing workers must be recreated with the new image and mount options
|
||||
## Upgrade
|
||||
|
||||
The worker needs outbound HTTPS access to LiteLLM. It needs no inbound ports, provider keys, direct database access, or GPU. The proxy calls your selected model through its normal virtual-key authorization and inference pipeline; trace content reaches that model provider. Use a model with JSON output support and known token prices. One worker handles up to three investigations concurrently and can serve multiple lenses. For more throughput, start another worker with a separate credential
|
||||
Upgrade LiteLLM and Lens from the same source commit and release identity. For a coordinated published release, use its matching worker version; `deploy/lens/stack.yaml` starts LiteLLM, Lens, PostgreSQL, and ClickHouse for new installations. Standalone images remain available. Publishing an image does not update running containers
|
||||
|
||||
If your deployment restricts `allowed_ips`, allow the worker's address. For workers behind a reverse proxy with `use_x_forwarded_for: true`, also configure `mcp_trusted_proxy_ranges` with that proxy's CIDRs and, when needed, `mcp_xff_num_trusted_hops`. Lens reuses these existing trusted-proxy settings. Forwarded addresses without an established trust boundary are rejected by the allowlist; accepting them would let a worker impersonate an allowed address
|
||||
Keep the same databases, encryption keys, shared service secret, and public ingestion URL. Pause scheduled investigations and finish or cancel active runs, update both images through your usual deployment process, then check ingestion and run an investigation before resuming schedules. Do not run `docker compose down -v`
|
||||
|
||||
Setup, manual runs, feedback, and worker credentials are restricted to proxy administrators. Proxy-admin viewers can inspect results. Regular user and team keys cannot access the Lens API. Worker credentials can serve the administrator’s lenses. Revoke it in the connection dialog when retiring a worker. Redeploy the worker alongside proxy upgrades so their API versions match
|
||||
When upgrading from the Python worker, replace it with the Rust Lens service, move the existing ClickHouse connection to Lens, and configure the service URLs and secret on LiteLLM. Existing trace data remains in the same ClickHouse database; findings and settings remain in PostgreSQL. Stop the old worker. Generate dedicated tracing keys and change agent exporters to the ingestion URL. A virtual model key no longer authorizes uploads; the old gateway upload endpoints return 410 with setup guidance
|
||||
|
||||
If you retain an explicit `LENS_WORKER_TOKEN`, it remains an optional investigation credential. Normal setup uses the shared service connection and registers one managed worker identity. Configure the analysis model and billing key in the dashboard; provider keys stay on LiteLLM
|
||||
|
||||
## Development
|
||||
|
||||
`make lens-dev` starts LiteLLM, the Rust Lens service, and the hot-reload dashboard. Set `LENS_DEV_PROXY_PORT` and `LENS_DEV_UI_PORT` to change the local ports. For containers, pass the same release identity to both builds. Unversioned or incompatible workers are refused before claiming work
|
||||
|
||||
## Configure a lens
|
||||
|
||||
|
|
@ -116,7 +118,7 @@ Describe how the agent should behave and optionally add specific checks. Select
|
|||
|
||||
Choose your analysis model, parallelism and monthly budget. Parallelism controls simultaneous model calls, not the number of runs selected. New lenses run once by default. Turn on monitoring to repeat the same setup at a custom interval. **Run now** uses the same saved settings immediately, including the same lookback window and sampling. Each scan recalculates the window and reuses completed reviews when the selected trace content, expected behavior, enabled checks and analysis model are unchanged. Budget, name and schedule edits preserve reuse. Duplicate a lens when you want a separate investigation without changing an existing monitor
|
||||
|
||||
Pausing stops future scheduled scans; cancel the active scan separately if needed. The worker polls every two seconds; creating a lens or clicking Run now queues a scan, and due schedules are queued when the worker polls. Scans for the same lens never overlap, and its next interval starts after completion. Closing the browser does not stop the worker. A running scan retains its analysis settings and selected execution IDs across retries. Budget edits apply to subsequent model calls, including those in an active scan
|
||||
Pausing stops future scheduled scans; cancel the active scan separately if needed. The worker polls every 2 to 15 seconds, backing off while idle; creating a lens or clicking Run now queues a scan, and due schedules are queued when the worker polls. Scans for the same lens never overlap, and its next interval starts after completion. Closing the browser does not stop the worker. A running scan retains its analysis settings and selected execution IDs across retries. Budget edits apply to subsequent model calls, including those in an active scan
|
||||
|
||||
## Read the results
|
||||
|
||||
|
|
|
|||
|
|
@ -32,6 +32,7 @@ pub struct Snapshot {
|
|||
|
||||
struct ActiveSnapshot {
|
||||
received: Instant,
|
||||
issued_at: u64,
|
||||
expires_at: u64,
|
||||
keys: HashMap<String, Credential>,
|
||||
}
|
||||
|
|
@ -91,8 +92,16 @@ impl Credentials {
|
|||
if keys.len() != count {
|
||||
return Err(Error::Unavailable);
|
||||
}
|
||||
*self.0.write().map_err(|_| Error::Unavailable)? = Some(ActiveSnapshot {
|
||||
let mut current = self.0.write().map_err(|_| Error::Unavailable)?;
|
||||
if current
|
||||
.as_ref()
|
||||
.is_some_and(|active| active.issued_at > snapshot.issued_at)
|
||||
{
|
||||
return Err(Error::Unavailable);
|
||||
}
|
||||
*current = Some(ActiveSnapshot {
|
||||
received: Instant::now(),
|
||||
issued_at: snapshot.issued_at,
|
||||
expires_at: snapshot.issued_at + SNAPSHOT_TTL.as_secs(),
|
||||
keys,
|
||||
});
|
||||
|
|
@ -114,14 +123,24 @@ impl Credentials {
|
|||
}
|
||||
|
||||
pub fn tenant(&self, headers: &HeaderMap) -> Result<Tenant, Error> {
|
||||
let hash = format!("{:x}", Sha256::digest(bearer(headers)?.as_bytes()));
|
||||
let token = bearer(headers)?;
|
||||
let hash = format!("{:x}", Sha256::digest(token.as_bytes()));
|
||||
let guard = self.0.read().map_err(|_| Error::Unavailable)?;
|
||||
let snapshot = guard.as_ref().ok_or(Error::Unavailable)?;
|
||||
let now = unix_seconds();
|
||||
if snapshot.received.elapsed() >= SNAPSHOT_TTL || snapshot.expires_at <= now {
|
||||
return Err(Error::Unavailable);
|
||||
}
|
||||
let key = snapshot.keys.get(&hash).ok_or(Error::Unauthorized)?;
|
||||
let pending = token
|
||||
.strip_prefix("lens-trace-")
|
||||
.and_then(|value| value.split_once('-'))
|
||||
.and_then(|(issued, _)| issued.parse::<u64>().ok())
|
||||
.is_some_and(|issued| issued >= snapshot.issued_at && issued <= now.saturating_add(5));
|
||||
let key = snapshot.keys.get(&hash).ok_or(if pending {
|
||||
Error::CredentialsPending
|
||||
} else {
|
||||
Error::Unauthorized
|
||||
})?;
|
||||
if key.expires_at.is_some_and(|expiry| expiry <= now) {
|
||||
return Err(Error::Unauthorized);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -33,6 +33,8 @@ pub enum Error {
|
|||
Configuration(&'static str),
|
||||
#[error("credential is invalid or expired")]
|
||||
Unauthorized,
|
||||
#[error("tracing credentials have not propagated yet")]
|
||||
CredentialsPending,
|
||||
#[error("Lens is temporarily unavailable")]
|
||||
Unavailable,
|
||||
#[error("request exceeds the size limit")]
|
||||
|
|
@ -69,6 +71,7 @@ impl Error {
|
|||
pub fn status(&self) -> StatusCode {
|
||||
match self {
|
||||
Self::Unauthorized => StatusCode::UNAUTHORIZED,
|
||||
Self::CredentialsPending => StatusCode::TOO_MANY_REQUESTS,
|
||||
Self::TooLarge => StatusCode::PAYLOAD_TOO_LARGE,
|
||||
Self::InvalidRequest => StatusCode::BAD_REQUEST,
|
||||
Self::TraceChanged => StatusCode::CONFLICT,
|
||||
|
|
@ -117,10 +120,14 @@ impl IntoResponse for Error {
|
|||
StatusCode::CONFLICT => "trace_changed",
|
||||
StatusCode::PAYLOAD_TOO_LARGE => "too_large",
|
||||
StatusCode::UNAUTHORIZED => "unauthorized",
|
||||
StatusCode::TOO_MANY_REQUESTS => "pending_credentials",
|
||||
_ => "unavailable",
|
||||
};
|
||||
let mut response = (status, Json(serde_json::json!({"code": code}))).into_response();
|
||||
if status == StatusCode::SERVICE_UNAVAILABLE {
|
||||
if matches!(
|
||||
status,
|
||||
StatusCode::SERVICE_UNAVAILABLE | StatusCode::TOO_MANY_REQUESTS
|
||||
) {
|
||||
response
|
||||
.headers_mut()
|
||||
.insert("retry-after", http::HeaderValue::from_static("5"));
|
||||
|
|
|
|||
|
|
@ -316,3 +316,45 @@ async fn replacing_credentials_revokes_previous_keys() {
|
|||
.unwrap();
|
||||
assert_eq!(response.status(), 401);
|
||||
}
|
||||
|
||||
#[rstest]
|
||||
#[tokio::test]
|
||||
async fn newly_created_key_is_retryable_until_this_replica_has_refreshed() {
|
||||
let server = serve("http://127.0.0.1:1", true).await;
|
||||
let now = unix_seconds();
|
||||
let token = format!("lens-trace-{now}-new-key");
|
||||
let client = http_client().unwrap();
|
||||
let pending = client
|
||||
.post(format!("{}/v1/traces", server.url))
|
||||
.bearer_auth(&token)
|
||||
.json(&export())
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(pending.status(), 429);
|
||||
assert_eq!(pending.headers()["retry-after"], "5");
|
||||
let older = format!("lens-trace-{}-invalid-key", now - 100);
|
||||
let denied = client
|
||||
.post(format!("{}/v1/traces", server.url))
|
||||
.bearer_auth(&older)
|
||||
.json(&export())
|
||||
.send()
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(denied.status(), 401);
|
||||
assert!(
|
||||
server
|
||||
.state
|
||||
.credentials
|
||||
.replace(Snapshot {
|
||||
issued_at: now - 1,
|
||||
keys: vec![],
|
||||
})
|
||||
.is_err()
|
||||
);
|
||||
let headers = http::HeaderMap::from_iter([(
|
||||
http::header::AUTHORIZATION,
|
||||
http::HeaderValue::from_str(&format!("Bearer {KEY}")).unwrap(),
|
||||
)]);
|
||||
assert!(server.state.credentials.tenant(&headers).is_ok());
|
||||
}
|
||||
|
|
|
|||
|
|
@ -165,8 +165,8 @@ async def service_connection(auth: Auth) -> ServiceConnection:
|
|||
|
||||
|
||||
async def credential_snapshot() -> IngestionSnapshot:
|
||||
keys: Final = await repository().ingestion_keys()
|
||||
now: Final = int(datetime.now(timezone.utc).timestamp())
|
||||
keys: Final = await repository().ingestion_keys()
|
||||
return IngestionSnapshot(
|
||||
issued_at=now,
|
||||
keys=tuple(
|
||||
|
|
|
|||
|
|
@ -64,7 +64,7 @@ def new_key(request: IngestionKeyRequest, user_id: str) -> IngestionKeyCreated:
|
|||
now: Final = datetime.now(timezone.utc)
|
||||
if request.expires_at is not None and request.expires_at <= now:
|
||||
raise ValueError("Choose an expiry in the future")
|
||||
token: Final = "lens-trace-" + secrets.token_urlsafe(40)
|
||||
token: Final = f"lens-trace-{int(now.timestamp())}-" + secrets.token_urlsafe(40)
|
||||
digest: Final = hashlib.sha256(token.encode()).hexdigest()
|
||||
return IngestionKeyCreated(
|
||||
key=token,
|
||||
|
|
|
|||
|
|
@ -3,7 +3,8 @@ set -euo pipefail
|
|||
|
||||
worker_image() {
|
||||
env -u LENS_WORKER_IMAGE -u LITELLM_VERSION \
|
||||
LITELLM_URL=http://litellm:4000 LENS_WORKER_TOKEN=config-test "$@" \
|
||||
LITELLM_URL=http://litellm:4000 LITELLM_LENS_SERVICE_TOKEN=config-test-service-secret-32-characters \
|
||||
CLICKHOUSE_URL=http://clickhouse:8123 "$@" \
|
||||
docker compose --env-file /dev/null -f deploy/lens/compose.yaml config --images
|
||||
}
|
||||
[[ "$(worker_image LENS_WORKER_IMAGE=registry.example/lens:source)" == registry.example/lens:source ]]
|
||||
|
|
@ -18,13 +19,14 @@ qa_dir=$(mktemp -d)
|
|||
master_key="sk-$(openssl rand -hex 16)"
|
||||
compose=(docker compose -p lens-compose-ci --env-file "$qa_dir/env" -f deploy/lens/stack.yaml)
|
||||
cleanup() {
|
||||
"${compose[@]}" --profile lens down -v --remove-orphans >/dev/null 2>&1 || true
|
||||
"${compose[@]}" down -v --remove-orphans >/dev/null 2>&1 || true
|
||||
rm -rf "$qa_dir"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
umask 077
|
||||
printf 'LITELLM_VERSION=0.0.0-lens-ci\nLITELLM_PORT=4418\nLITELLM_MASTER_KEY=%s\nLITELLM_SALT_KEY=sk-%s\n' \
|
||||
"$master_key" "$(openssl rand -hex 32)" > "$qa_dir/env"
|
||||
printf 'LITELLM_LENS_SERVICE_TOKEN=%s\nLENS_PORT=4419\n' "$(openssl rand -hex 32)" >> "$qa_dir/env"
|
||||
printf 'POSTGRES_PASSWORD=%s:/?#@%%\nCLICKHOUSE_PASSWORD=%s:/?#@%%\n' \
|
||||
"$(openssl rand -hex 32)" "$(openssl rand -hex 32)" >> "$qa_dir/env"
|
||||
docker tag "${LITELLM_IMAGE:?Set LITELLM_IMAGE to the built gateway image}" ghcr.io/berriai/litellm:0.0.0-lens-ci
|
||||
|
|
@ -52,7 +54,22 @@ jq -n --arg trace "$trace_id" --arg span "$span_id" --arg at "$start_ns" \
|
|||
kind:1,startTimeUnixNano:$at,endTimeUnixNano:$at,
|
||||
attributes:[{key:"openinference.span.kind",value:{stringValue:"AGENT"}}],status:{code:1}}]}]}]}' \
|
||||
> "$qa_dir/trace.json"
|
||||
api /v1/traces -d "@$qa_dir/trace.json" > /dev/null
|
||||
api /lens/tracing/keys -d '{"name":"Compose smoke"}' > "$qa_dir/tracing-key.json"
|
||||
tracing_key=$(jq -r '.key' "$qa_dir/tracing-key.json")
|
||||
trace_sent=false
|
||||
for attempt in $(seq 1 60); do
|
||||
if curl --fail --silent --show-error --max-time 10 \
|
||||
-H "Authorization: Bearer $tracing_key" -H 'Content-Type: application/json' \
|
||||
-d "@$qa_dir/trace.json" http://127.0.0.1:4419/v1/traces > /dev/null 2>&1; then
|
||||
trace_sent=true; break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
[[ "$trace_sent" == true ]]
|
||||
status=$(curl --silent -o /dev/null -w '%{http_code}' \
|
||||
-H "Authorization: Bearer $master_key" -H 'Content-Type: application/json' \
|
||||
-d "@$qa_dir/trace.json" http://127.0.0.1:4418/v1/traces)
|
||||
[[ "$status" == 410 ]]
|
||||
trace_saved() {
|
||||
for attempt in $(seq 1 60); do
|
||||
if api "/v1/traces/$trace_id" > "$qa_dir/saved-trace.json" 2>/dev/null && \
|
||||
|
|
@ -70,13 +87,13 @@ key_id=$(jq -r '.token_id // empty' "$qa_dir/key.json")
|
|||
if [[ -z "$key_id" ]]; then
|
||||
key_id=$(jq -rj '.key' "$qa_dir/key.json" | openssl dgst -sha256 | awk '{print $NF}')
|
||||
fi
|
||||
jq -n --arg key "$key_id" '{name:"Lens Compose CI",analysis_key_id:$key}' > "$qa_dir/registration.json"
|
||||
jq -n --arg key "$key_id" '{name:"Lens Compose CI",analysis_key_id:$key,managed:true}' > "$qa_dir/registration.json"
|
||||
api /lens/workers/register -d "@$qa_dir/registration.json" > "$qa_dir/worker.json"
|
||||
jq -e '.image == "ghcr.io/berriai/litellm-lens-worker:v0.0.0-lens-ci"' "$qa_dir/worker.json" > /dev/null
|
||||
printf 'LENS_WORKER_TOKEN=%s\n' "$(jq -r '.token' "$qa_dir/worker.json")" >> "$qa_dir/env"
|
||||
jq -e '.managed == true and .token == ""' "$qa_dir/worker.json" > /dev/null
|
||||
worker_id=$(jq -r '.worker.id' "$qa_dir/worker.json")
|
||||
heartbeat_after=$(date -u +'%Y-%m-%dT%H:%M:%S')
|
||||
"${compose[@]}" --profile lens up -d
|
||||
"${compose[@]}" up -d
|
||||
|
||||
connected() {
|
||||
for attempt in $(seq 1 60); do
|
||||
|
|
@ -87,36 +104,45 @@ connected() {
|
|||
fi
|
||||
sleep 2
|
||||
done
|
||||
"${compose[@]}" --profile lens logs lens-worker
|
||||
"${compose[@]}" logs lens-worker
|
||||
return 1
|
||||
}
|
||||
connected
|
||||
printf 'Fresh Compose stack: matching worker image and authenticated heartbeat passed\n'
|
||||
|
||||
for target in db:5432 clickhouse:8123; do
|
||||
service=${target%:*}
|
||||
port=${target#*:}
|
||||
address=$(docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$("${compose[@]}" ps -q "$service")")
|
||||
"${compose[@]}" exec -T lens-worker python -c '
|
||||
database_address=$(docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$("${compose[@]}" ps -q db)")
|
||||
"${compose[@]}" exec -T lens-worker python3.13 -I -S -c '
|
||||
import socket, sys
|
||||
for host in (sys.argv[1], sys.argv[2]):
|
||||
for host in ("db", sys.argv[1]):
|
||||
try:
|
||||
connection = socket.create_connection((host, int(sys.argv[3])), timeout=2)
|
||||
connection = socket.create_connection((host, 5432), timeout=2)
|
||||
except OSError:
|
||||
continue
|
||||
connection.close()
|
||||
raise SystemExit("Worker can reach a datastore directly")
|
||||
' "$service" "$address" "$port"
|
||||
done
|
||||
printf 'Worker can reach the proxy but cannot connect directly to PostgreSQL or ClickHouse\n'
|
||||
raise SystemExit("Lens can reach PostgreSQL directly")
|
||||
with socket.create_connection(("clickhouse", 8123), timeout=2):
|
||||
pass
|
||||
' "$database_address"
|
||||
"${compose[@]}" exec -T litellm python3 -c '
|
||||
import socket
|
||||
try:
|
||||
connection = socket.create_connection(("clickhouse", 8123), timeout=2)
|
||||
except OSError:
|
||||
pass
|
||||
else:
|
||||
connection.close()
|
||||
raise SystemExit("Gateway can reach ClickHouse directly")
|
||||
'
|
||||
printf 'Datastore isolation: Lens reaches ClickHouse, gateway reaches Postgres, neither reaches the other datastore\n'
|
||||
service_token=$(sed -n 's/^LITELLM_LENS_SERVICE_TOKEN=//p' "$qa_dir/env")
|
||||
|
||||
status=$(curl --silent --show-error -o "$qa_dir/mismatch.json" -w '%{http_code}' -X POST \
|
||||
-H "Authorization: Bearer $(jq -r '.token' "$qa_dir/worker.json")" \
|
||||
-H "Authorization: Bearer $service_token" \
|
||||
'http://127.0.0.1:4418/lens/worker/claim?protocol_version=4&worker_release=v0.0.0-old')
|
||||
[[ "$status" == 409 ]]
|
||||
jq -e '.detail | contains("Upgrade the Lens worker")' "$qa_dir/mismatch.json" > /dev/null
|
||||
|
||||
"${compose[@]}" --profile lens restart litellm lens-worker
|
||||
"${compose[@]}" restart litellm lens-worker
|
||||
heartbeat_after=$(date -u +'%Y-%m-%dT%H:%M:%S')
|
||||
connected
|
||||
trace_saved
|
||||
|
|
@ -125,32 +151,13 @@ jq -e --arg id "$worker_id" --arg key "$key_id" \
|
|||
'.workers[] | select(.id == $id and .analysis_key_id == $key)' "$qa_dir/restarted.json" > /dev/null
|
||||
printf 'Compose restart: trace, worker identity, token and billing assignment preserved; wrong release rejected\n'
|
||||
|
||||
cat > "$qa_dir/unversioned.yaml" <<'EOF'
|
||||
services:
|
||||
litellm:
|
||||
environment:
|
||||
LITELLM_RELEASE_TAG: ""
|
||||
EOF
|
||||
"${compose[@]}" -f "$qa_dir/unversioned.yaml" up -d litellm
|
||||
"${compose[@]}" stop clickhouse
|
||||
"${compose[@]}" restart litellm
|
||||
for attempt in $(seq 1 90); do
|
||||
if api /health/liveliness > /dev/null 2>&1; then break; fi
|
||||
sleep 2
|
||||
done
|
||||
api /health/liveliness > /dev/null
|
||||
status=$(curl --silent --show-error --max-time 30 -o "$qa_dir/unversioned-registration.json" -w '%{http_code}' \
|
||||
-H "Authorization: Bearer $master_key" -H 'Content-Type: application/json' \
|
||||
-d "@$qa_dir/registration.json" 'http://127.0.0.1:4418/lens/workers/register')
|
||||
[[ "$status" == 503 ]]
|
||||
jq -e '.detail | contains("no release identity")' "$qa_dir/unversioned-registration.json" > /dev/null
|
||||
status=$(curl --silent --show-error --max-time 30 -o "$qa_dir/unversioned-claim.json" -w '%{http_code}' -X POST \
|
||||
-H "Authorization: Bearer $(jq -r '.token' "$qa_dir/worker.json")" \
|
||||
'http://127.0.0.1:4418/lens/worker/claim?protocol_version=4&worker_release=')
|
||||
[[ "$status" == 503 ]]
|
||||
jq -e '.detail | contains("no release identity")' "$qa_dir/unversioned-claim.json" > /dev/null
|
||||
api /lens > "$qa_dir/unversioned-workers.json"
|
||||
jq -e --arg id "$worker_id" '.workers | length == 1 and .[0].id == $id' "$qa_dir/unversioned-workers.json" > /dev/null
|
||||
"${compose[@]}" up -d litellm
|
||||
heartbeat_after=$(date -u +'%Y-%m-%dT%H:%M:%S')
|
||||
connected
|
||||
"${compose[@]}" start clickhouse
|
||||
trace_saved
|
||||
printf 'Unversioned gateway: setup and claims refused without guessing; original worker and trace recovered\n'
|
||||
printf 'Gateway cold startup succeeds with ClickHouse stopped; trace reads recover after storage restarts\n'
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue