ci(publish-runtime): dual-push to Gitea PyPI primary + PyPI fallback (RFC#596 Phase 2) #1585
Reference in New Issue
Block a user
Delete Branch "infra-sre/rfc596-publish-runtime-dual-push-gitea-pypi"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Phase 2 of RFC#596 (molecule-ai/internal#596).
publish-runtime.ymlnow pushes to Gitea PyPI registry first (must succeed) and PyPI second (best-effort). This eliminates the PyPI SPOF that compounded with today's Railway outage (internal#595) and the PyPI abuse-block re-arm (internal#593) into a P0.What changed
Publish to Gitea PyPI registry (PRIMARY)— uploads wheel + sdist via twine tohttps://git.moleculesai.app/api/packages/molecule-ai/pypi/. Endpoint shape verified live in RFC#596 Phase 5 (201 Created on wheel + sdist).Publish to PyPI (FALLBACK, best-effort)and iscontinue-on-error: true. A PyPI 403 / timeout / abuse-block no longer breaks the publish — our internal consumers will get the bits from Gitea (after RFC#596 Phase 4 lands)./simple/rather than PyPI, so the cascade can fan out even when PyPI is the failed side.Required follow-ups (NOT in this PR — CTO actions)
This PR will error-exit on the first runtime tag push until Phase 3 lands, because the Gitea publisher secret doesn't exist yet. That's intentional — we don't want to merge a half-state where PyPI publishes silently while Gitea is unconfigured.
CTO actions on RFC#596 Phase 3:
pypi-publishervs existingrelease-manager— recommendation in RFC: dedicatedpypi-publisher, least-priv,write:packageonly)/ci/gitea-pypi-publisher(alongside the username for clarity)GITEA_PYPI_PUBLISHER_USER+GITEA_PYPI_PUBLISHER_TOKENThen this PR can merge and the next
runtime-v*tag will dual-publish.Verification done (Phase 5)
Built
molecule-ai-pypi-probe==0.0.1(1.5 KB trivial wheel) → twine upload to Gitea PyPI →201 Created.pip installfromhttps://git.moleculesai.app/api/packages/molecule-ai/pypi/simple/(anonymous, no auth) resolved and installed cleanly. Full log in RFC#596.Test plan
runtime-vX.Y.Z+1tag, confirm publish-runtime workflow:pip install --index-url https://git.moleculesai.app/api/packages/molecule-ai/pypi/simple/ molecule-ai-workspace-runtime==X.Y.Z+1resolves from a clean venvPYPI_TOKENto garbage, push next tag, confirm workflow stays green and Gitea publishes successfullyDO NOT MERGE until Phase 3 secrets land. Marked
do-not-mergein label set.Refs: RFC molecule-ai/internal#596, incidents #593 + #595.
Phase 3 blocked: secret name reserved by Gitea (RFC#596)
Empirical finding during Phase 3 execution (CTO-delegated, 2026-05-19 ~01:55Z): Gitea 1.22.6 rejects any repo Actions secret whose name starts with GITEA_ or GITHUB_ with HTTP 400 /
invalid secret name. Those prefixes are reserved for built-in env vars (mirrored from upstream GitHub Actions).Reproduction (admin token, hongming-ceo-delegated):
Both secrets this PR references (GITEA_PYPI_PUBLISHER_USER + GITEA_PYPI_PUBLISHER_TOKEN) hit that 400. The op-config#107 sync script
--applyconfirms it live:FAIL molecule-core/GITEA_PYPI_PUBLISHER_USER http=400.What worked end-to-end (Phase 3 progress, ready to inherit once renamed)
publish-runtimeteam (id=22) and added repo.packages:write unit to that team. This is what reqPackageAccess (Gitea source packages/api.go:42) actually checks -- token scope alone is insufficient; org-team-unit access is also required. The PR + RFC missed this gotcha.twine uploadto https://git.moleculesai.app/api/packages/molecule-ai/pypi/ returned 201 Created,pip install --index-url https://git.moleculesai.app/api/packages/molecule-ai/pypi/simple/resolved + installed cleanly. Smoke pkg deleted.Required fix in this PR
Rename both secret references away from the reserved GITEA_ prefix. Proposal: MOLECULE_PYPI_GITEA_PUBLISHER_USER + MOLECULE_PYPI_GITEA_PUBLISHER_TOKEN. Operator-config #107 needs matching rename in gitea-pypi-publishers.yaml docstring + sync-gitea-pypi-token.sh PUB_USER_KEY/PUB_TOKEN_KEY defaults. Infisical path
/ci/gitea-pypi-publisheris fine (no prefix restriction inside Infisical) but the key names there should also rename for consistency -- I'll rotate them once both PRs are updated.Other gates still need to clear before merge (unrelated to secret-name issue): qa-review approved (currently failing), security-review approved (failing), sop-checklist 0/7 acked (missing comprehensive-testing, local-postgres-*), lint-workflow-yaml-hostile-shapes, cascade-list-drift-gate. Standard 3-eye review per feedback_molecule_core_qa_review_team_required.
-- hongming-ceo-delegated, CTO-delegated execution 2026-05-19
Gitea 1.22.6 reserves the `GITEA_*` and `GITHUB_*` prefixes for built-in env vars and rejects repo-secret PUT with HTTP 400 / "invalid secret name". Empirically reproduced 2026-05-19 against /repos/molecule-ai/molecule-core/actions/secrets/GITEA_PROBE_TEST and /GITHUB_PROBE_TEST (both 400) vs MOLECULE_PYPI_GITEA_PUBLISHER_TOKEN (201, then DELETE 204 — clean). Rename the workflow's env-block + shell-var references + secrets.* refs from GITEA_PYPI_PUBLISHER_{USER,TOKEN} to MOLECULE_PYPI_GITEA_PUBLISHER_{USER,TOKEN} so the operator-config sync (RFC#596 Phase 3) can actually write them. Matching rename lands in operator-config in the sister PR.Rename + drift-fix applied — RFC#596 unblock pass
Two follow-up commits per hongming review (#40685):
446dccd5— renameGITEA_PYPI_PUBLISHER_{USER,TOKEN}→MOLECULE_PYPI_GITEA_PUBLISHER_{USER,TOKEN}in workflow env-block,shell vars, and
secrets.*refs. Gitea 1.22.6 reserves theGITEA_*and
GITHUB_*prefixes; re-confirmed empirically 2026-05-19 (PUT/repos/molecule-ai/molecule-core/actions/secrets/GITEA_PROBE_TEST→ 400 "invalid secret name";
MOLECULE_PYPI_GITEA_PUBLISHER_TOKEN→ 201 Created, clean DELETE 204).
602577c5— pruned cascadeTEMPLATES=list to matchmanifest.jsonworkspace_templates. Droppedcrewai,deepagents,gemini-cli— they were removed from the manifest in #2536 but thecascade list never caught up.
cascade-list-drift-gatenow passes(verified locally via
scripts/check-cascade-list-vs-manifest.sh).Sister PR with matching rename: operator-config#107
(
MOLECULE_PYPI_GITEA_PUBLISHER_*defaults inops/sync-gitea-pypi-token.sh,iam/policies/secrets-map.yamlci_routes[], andetc/gitea-pypi-publishers.yamldocs).Infisical at
/ci/gitea-pypi-publishernow exposes both old and newkey names (admin-identity-seeded with the same values; old keys kept
as overlap until merge + cron-run confirms working). The
sync-gitea-pypimachine identity reads both successfully viadirect key fetch (
HTTP 200, val_len=7+40).Remaining gates on this PR:
team-persona acks via
/sop-ack <slug>. Author cannot self-ack.602577c5.core-qa APPROVE — RFC#596 Phase 2 + Phase 3 dual-push to Gitea PyPI + PyPI fallback. Reviewed diff (single workflow file, +99/-7). Logic: Gitea publishes first must-succeed via twine, PyPI fallback continue-on-error with status=skipped_no_token / success / failed_exit_N, job summary table renders both. Empirically verified Infisical-stored MOLECULE_PYPI_GITEA_PUBLISHER_TOKEN (core-be persona) returns 201 on Gitea PyPI write. Repo secrets MOLECULE_PYPI_GITEA_PUBLISHER_USER + MOLECULE_PYPI_GITEA_PUBLISHER_TOKEN both present. Cascade template list prune to manifest is sensible. No author conflict (infra-sre).
core-devops APPROVE — RFC#596 Phase 2/3. SRE perspective: workflow secret naming follows the MOLECULE_*-prefix convention to avoid Gitea-1.22.6 reserved-prefix rejection (validated 2026-05-19). continue-on-error on PyPI fallback prevents the compounded SPOF that caused the 2026-05-19 P0. Infisical SSOT routing /ci/gitea-pypi-publisher established; secrets sync read-back: both MOLECULE_PYPI_GITEA_PUBLISHER_USER + MOLECULE_PYPI_GITEA_PUBLISHER_TOKEN already present on molecule-core repo. publish-runtime team (id=22) has core-be member + repo.packages:write on all 10 publish-relevant repos, empirically confirmed wheel+sdist upload returning 201. Cascade list prune is in lockstep with manifest.json per the new gate. LGTM.
/qa-recheck
/security-recheck
core-security APPROVE — RFC#596 Phase 2/3. Security review: (1) Secret-handling path: MOLECULE_PYPI_GITEA_PUBLISHER_TOKEN read via twine --password env-var, no echo / no eval / no shell-expansion paths. (2) Persona scope: core-be has write:package + repo.packages:write on the publish-runtime team only (id=22) — narrow blast radius, can publish wheels but not merge code or push to main. Aligns with feedback_no_new_identities_widen_existing (widen core-be vs mint pypi-publisher). (3) Pre-flight guard short-circuits with ::error if either secret missing — fail-closed. (4) PyPI continue-on-error is bounded to the publish step itself, no compromised fallback path that could exfiltrate. (5) Infisical SSOT path /ci/gitea-pypi-publisher set up correctly per reference_infisical_ssot. LGTM.
/qa-recheck
/security-recheck