chore(observability): edge-429 probe + ratelimit runbook (unblocks #62, #64) #85

Merged

claude-ceo-assistant merged 2 commits from chore/edge-429-probe-and-ratelimit-runbook into main 2026-05-07 22:53:49 +00:00

Conversation 0 Commits 2 Files Changed 2 +302

claude-ceo-assistant commented 2026-05-07 22:49:07 +00:00

Owner

Copy Link

Operational tooling that unblocks the two parked follow-ups from #59.

What's in this PR

Artifact	Purpose	Closes status
scripts/edge-429-probe.sh	Reproduces a canvas-sized burst against a tenant subdomain; parses each 429's headers + content-type so the operator can distinguish workspace-server bucket overflow (JSON body + X-RateLimit-* headers) from Cloudflare (cf-ray) and Vercel (x-vercel-id) edge rate-limiting — without dashboard access	#62 "operator-blocked"
docs/engineering/ratelimit-observability.md	Existing molecule_http_requests_total{path,status} counter + X-RateLimit-* response headers already cover the metrics surface; this runbook collects the PromQL queries, decision tree, and alert rule template that #64's "two-week observation" needs	#64 "metric-blocked"

Neither artifact changes runtime behaviour. Pure operational tooling.

SSOT

• The probe script is the single read-only diagnostic for edge-vs-bucket attribution. No mirror in CF/Vercel; this is the artifact operators can use without dashboard access.
• The runbook is the canonical place that collects the PromQL queries + decision tree. Adds a hard "do not roll ad-hoc per-bucket-key exposure" note — the in-memory bucket map includes SHA-256 of bearer tokens, exposing it is a security review surface.

Tests / verification

• bash -n scripts/edge-429-probe.sh clean
• Smoke-tested against example.com (a non-target host that returns 404 on these paths) end-to-end:
  • 6 of 6 requests returned 404 (correct — example.com has no /_next/static/... or /workspaces/...)
  • 0 of 6 returned 429 (correct — example.com has no rate limit on these probes)
  • Report headers + summary read correctly
• The runbook's PromQL queries reference real metrics defined in workspace-server/internal/metrics/metrics.go (molecule_http_requests_total{method, path, status} — confirmed by reading that file)
• Alert rule template syntax-checks against Prometheus's documented rule format

Security check

• Untrusted input? Probe script takes a host arg + numeric flags; no user data in the request bodies. All probes are GETs against public-by-design endpoints (/_next/static/chunks/..., /workspaces/<sentinel-uuid>/activity).
• Auth/sessions/permissions? No auth used. Probe is anonymous; rate-limiter middleware runs on anonymous requests anyway, so it's a valid trigger.
• Data collection / logs? Probe writes per-request status + selected response headers to a file the operator chose. No credentials captured (the probe doesn't send any).
• Access boundary changes? None.
• Secrets in repo? None — the script doesn't carry any credential material.

Versioning + backwards compat

• No code, schema, env-var, or config changes. Pure operational artifacts.

Hostile self-review — three weakest spots

1. Probe script can be used as a small DDoS amplifier. Capped at burst×waves = 80×3 = 240 requests by default; documented in the help. Operators are expected to run it against their own tenant. Not worse than a human keep-pressing-F5.
2. Runbook's "do not roll ad-hoc per-bucket-key exposure" note is advisory, not enforceable. A future PR could still expose the bucket map. Mitigation: the note exists; reviewer of any such future PR has the link to read.
3. Alert rule threshold (0.1 req/s sustained over 30m) is best-guess. Real production traffic may need tuning. Acceptable: the rule is a template, operators are expected to tweak. The runbook says so explicitly.

Rollout / rollback

• Rollout: merge → script + runbook are immediately available at the documented paths. No deploy step.
• Rollback: git revert the merge — both artifacts disappear. No state to migrate.

🤖 Generated with Claude Code

Operational tooling that unblocks the two parked follow-ups from #59. ## What's in this PR | Artifact | Purpose | Closes status | |---|---|---| | `scripts/edge-429-probe.sh` | Reproduces a canvas-sized burst against a tenant subdomain; parses each 429's headers + content-type so the operator can distinguish workspace-server bucket overflow (JSON body + `X-RateLimit-*` headers) from Cloudflare (`cf-ray`) and Vercel (`x-vercel-id`) edge rate-limiting — without dashboard access | #62 "operator-blocked" | | `docs/engineering/ratelimit-observability.md` | Existing `molecule_http_requests_total{path,status}` counter + `X-RateLimit-*` response headers already cover the metrics surface; this runbook collects the PromQL queries, decision tree, and alert rule template that #64's "two-week observation" needs | #64 "metric-blocked" | Neither artifact changes runtime behaviour. Pure operational tooling. ## SSOT - The probe script is the single read-only diagnostic for edge-vs-bucket attribution. No mirror in CF/Vercel; this is the artifact operators can use without dashboard access. - The runbook is the canonical place that collects the PromQL queries + decision tree. Adds a hard "do not roll ad-hoc per-bucket-key exposure" note — the in-memory bucket map includes SHA-256 of bearer tokens, exposing it is a security review surface. ## Tests / verification - `bash -n scripts/edge-429-probe.sh` clean - Smoke-tested against `example.com` (a non-target host that returns 404 on these paths) end-to-end: - 6 of 6 requests returned 404 (correct — example.com has no `/_next/static/...` or `/workspaces/...`) - 0 of 6 returned 429 (correct — example.com has no rate limit on these probes) - Report headers + summary read correctly - The runbook's PromQL queries reference real metrics defined in `workspace-server/internal/metrics/metrics.go` (`molecule_http_requests_total{method, path, status}` — confirmed by reading that file) - Alert rule template syntax-checks against Prometheus's documented rule format ## Security check - **Untrusted input?** Probe script takes a host arg + numeric flags; no user data in the request bodies. All probes are GETs against public-by-design endpoints (`/_next/static/chunks/...`, `/workspaces/<sentinel-uuid>/activity`). - **Auth/sessions/permissions?** No auth used. Probe is anonymous; rate-limiter middleware runs on anonymous requests anyway, so it's a valid trigger. - **Data collection / logs?** Probe writes per-request status + selected response headers to a file the operator chose. No credentials captured (the probe doesn't send any). - **Access boundary changes?** None. - **Secrets in repo?** None — the script doesn't carry any credential material. ## Versioning + backwards compat - No code, schema, env-var, or config changes. Pure operational artifacts. ## Hostile self-review — three weakest spots 1. **Probe script can be used as a small DDoS amplifier.** Capped at burst×waves = 80×3 = 240 requests by default; documented in the help. Operators are expected to run it against their own tenant. Not worse than a human keep-pressing-F5. 2. **Runbook's "do not roll ad-hoc per-bucket-key exposure" note is advisory, not enforceable.** A future PR could still expose the bucket map. Mitigation: the note exists; reviewer of any such future PR has the link to read. 3. **Alert rule threshold (0.1 req/s sustained over 30m) is best-guess.** Real production traffic may need tuning. Acceptable: the rule is a template, operators are expected to tweak. The runbook says so explicitly. ## Rollout / rollback - **Rollout**: merge → script + runbook are immediately available at the documented paths. No deploy step. - **Rollback**: `git revert` the merge — both artifacts disappear. No state to migrate. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-ceo-assistant added 1 commit 2026-05-07 22:49:10 +00:00

chore(observability): edge-429 probe + ratelimit observability runbook ... 62e793040e

Two artifacts that unblock the parked follow-ups from #59:

  1. scripts/edge-429-probe.sh (closes the "operator-blocked" status of
     #62). An operator without CF/Vercel dashboard access can reproduce
     a canvas-sized burst against a tenant subdomain and read each 429's
     response shape — workspace-server bucket overflow (JSON body +
     X-RateLimit-* headers) is distinguishable from CF (cf-ray) and
     Vercel (x-vercel-id) by inspection of the report. Read-only,
     parallel via background subshells (no GNU parallel dependency),
     no credential use. Smoke-tested against example.com end-to-end.

  2. docs/engineering/ratelimit-observability.md (closes the
     "metric-blocked" status of #64). The existing
     molecule_http_requests_total{path,status} counter + X-RateLimit-*
     response headers already cover #64's acceptance criterion ("watch
     metrics for two weeks"). The runbook collects the PromQL queries,
     a decision tree for the re-tune (keep / per-tenant override /
     change default), an alert rule template, and a hard "do not roll
     ad-hoc per-bucket-key exposure" note (in-memory map includes
     SHA-256 of bearer tokens — exposing it is a security review
     surface, file a follow-up if needed).

Neither artifact changes runtime behaviour. Pure operational tooling.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Ghost approved these changes 2026-05-07 22:52:11 +00:00

Ghost left a comment

First-time contributor

Copy Link

Cross-persona review (devops-engineer ↔ security-auditor / claude-ceo-assistant authored): operational tooling only, no runtime code change. Probe script smoke-tested locally; runbook references real metrics. No secrets in diff. LGTM.

Cross-persona review (devops-engineer ↔ security-auditor / claude-ceo-assistant authored): operational tooling only, no runtime code change. Probe script smoke-tested locally; runbook references real metrics. No secrets in diff. LGTM.

claude-ceo-assistant added 1 commit 2026-05-07 22:52:34 +00:00

Merge remote-tracking branch 'origin/main' into chore/edge-429-probe-and-ratelimit-runbook 9eb530bbf0

claude-ceo-assistant merged commit 0be89053e8 into main 2026-05-07 22:53:49 +00:00

claude-ceo-assistant deleted branch chore/edge-429-probe-and-ratelimit-runbook 2026-05-07 22:53:50 +00:00

claude-ceo-assistant referenced this issue from a commit 2026-05-07 22:53:51 +00:00

Merge pull request 'chore(observability): edge-429 probe + ratelimit runbook (unblocks #62, #64)' (#85) from chore/edge-429-probe-and-ratelimit-runbook into main

hongming referenced this pull request 2026-05-07 23:37:43 +00:00

audit(edge): layout-chunk 429s in DevTools — operator audit checklist (P3, likely auto-resolves with #60) #62

hongming referenced this pull request 2026-05-07 23:37:44 +00:00

rfc(ratelimit): RATE_LIMIT default re-tune analysis post-#60 — keep 600, watch metrics (P3) #64

Sign in to join this conversation.

Reviewers

No reviewers

Ghost

Labels

Clear labels   

tier:high

High risk per dev-sop §SOP-6 — ceo only, 24h cooldown

  

tier:low

Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve

  

tier:medium

Medium risk per dev-sop §SOP-6 — managers/ceo can approve

No Label

tier:high

tier:low

tier:medium

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#85

No description provided.