molecule-ai/molecule-core
53
0
Fork 2
Code Issues 188 Pull Requests 42 Actions 3 Packages Projects Releases Wiki Activity

obs gap: Gitea CI action logs are not shipped to Loki — debugging requires ssh+zstdcat on the operator #2272

New Issue
Open
opened 2026-06-05 01:46:11 +00:00 by hongming · 0 comments
hongming commented 2026-06-05 01:46:11 +00:00
Owner
Copy Link
Copy Source

Problem

Diagnosing CI / staging-e2e failures currently requires ssh root@operator + zstdcat /opt/molecule/gitea/actions_log/.../<task>.log.zst — because Gitea Actions per-task logs are not shipped to the obs system (Loki). Verified 2026-06-05: the operator Vector config (/etc/vector/vector.yaml) has NO gitea/actions source; service/container/source label values in Loki contain operator docker containers + JSONL ops streams + tenant platform logs, but zero CI job output. This violates the obs-first rule for the entire CI/e2e surface.

Why it's not a trivial Vector file-source

Gitea stores action logs zstd-compressed (<task_id>.log.zst) on task completion; in-progress tasks are plaintext (<task_id>.log), in a nested layout /opt/molecule/gitea/actions_log/<owner>/<repo>/<NN>/<task_id>.log[.zst]. A bare type: file tail can't read the .zst.

Proposed fix (mirrors the existing operator JSONL→Vector pattern)

A small operator shipper (cron or inotify on /opt/molecule/gitea/actions_log) that, on task completion, decompresses the .zst and appends NDJSON lines to /var/log/gitea-actions.jsonl with fields {repo, run_id, job, task_id, ts, line} (run/job/task resolvable from the path + the action_task/action_run tables). Then add a Vector type: file source for /var/log/gitea-actions.jsonl + a remap transform stamping service=gitea-ci, repo, run_id (mirroring parse_disk_gc/parse_runner_health in /etc/vector/vector.yaml), appended to sinks.loki.inputs. Then CI logs are queryable as {service="gitea-ci", repo="molecule-core"} |= "<run_id>" instead of SSH+zstdcat.

  • Idempotent (track last-shipped task id); cap line size; only ship completed tasks (or tail live .log for in-progress, dedupe on completion).
  • Assets live in operator-config (the obs/Vector home); note /etc/vector/vector.yaml is currently untracked (internal#242) and obs Vector snippets are manual-scp (not auto-synced).

Related obs gaps (separate, for visibility)

  • Staging/prod CP logs are on Railway, not Loki — a Railway→Loki bridge (or Railway log drain) is a separate item.
  • Workspace/agent container logs excluded by design (internal#107); RFC#640 is the in-flight reversal (owes ToS check).

Surfaced while debugging the core#2261 reconciler live-e2e (had to SSH+zstdcat to read the HTTP-400 cause). Ref: obs recipe in reference_obs_system_access.

## Problem Diagnosing CI / staging-e2e failures currently requires `ssh root@operator` + `zstdcat /opt/molecule/gitea/actions_log/.../<task>.log.zst` — because **Gitea Actions per-task logs are not shipped to the obs system (Loki).** Verified 2026-06-05: the operator Vector config (`/etc/vector/vector.yaml`) has NO gitea/actions source; `service`/`container`/`source` label values in Loki contain operator docker containers + JSONL ops streams + tenant platform logs, but **zero CI job output.** This violates the obs-first rule for the entire CI/e2e surface. ## Why it's not a trivial Vector file-source Gitea stores action logs **zstd-compressed** (`<task_id>.log.zst`) on task completion; in-progress tasks are plaintext (`<task_id>.log`), in a nested layout `/opt/molecule/gitea/actions_log/<owner>/<repo>/<NN>/<task_id>.log[.zst]`. A bare `type: file` tail can't read the `.zst`. ## Proposed fix (mirrors the existing operator JSONL→Vector pattern) A small operator shipper (cron or inotify on `/opt/molecule/gitea/actions_log`) that, on task completion, decompresses the `.zst` and appends NDJSON lines to `/var/log/gitea-actions.jsonl` with fields `{repo, run_id, job, task_id, ts, line}` (run/job/task resolvable from the path + the `action_task`/`action_run` tables). Then add a Vector `type: file` source for `/var/log/gitea-actions.jsonl` + a `remap` transform stamping `service=gitea-ci`, `repo`, `run_id` (mirroring `parse_disk_gc`/`parse_runner_health` in `/etc/vector/vector.yaml`), appended to `sinks.loki.inputs`. Then CI logs are queryable as `{service="gitea-ci", repo="molecule-core"} |= "<run_id>"` instead of SSH+zstdcat. - Idempotent (track last-shipped task id); cap line size; only ship completed tasks (or tail live `.log` for in-progress, dedupe on completion). - Assets live in operator-config (the obs/Vector home); note `/etc/vector/vector.yaml` is currently untracked (internal#242) and obs Vector snippets are manual-scp (not auto-synced). ## Related obs gaps (separate, for visibility) - **Staging/prod CP logs** are on **Railway**, not Loki — a Railway→Loki bridge (or Railway log drain) is a separate item. - **Workspace/agent container logs** excluded by design (internal#107); RFC#640 is the in-flight reversal (owes ToS check). Surfaced while debugging the core#2261 reconciler live-e2e (had to SSH+zstdcat to read the HTTP-400 cause). Ref: obs recipe in `reference_obs_system_access`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) godwin hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 hongming-personal infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#2272
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?