molecule-ai/molecule-core
51
0
Fork 2
Code Issues 164 Pull Requests 154 Actions 14 Packages Projects Releases Wiki Activity

feat(provisioner): inject per-persona Gitea credential helper at provision time #1523

Closed
hongming wants to merge 1 commits from feat/provisioner-inject-gitea-credential-helper into staging
pull from: feat/provisioner-inject-gitea-credential-helper
merge into: molecule-ai:staging
Conversation 1 Commits 1 Files Changed 6
+636
hongming commented 2026-05-18 20:01:33 +00:00
Owner
Copy Link
Copy Source

Summary

Closes the workspace git-auth gap: prod-team workspaces (agent-dev-a/agent-dev-b and the rest of the persona-backed fleet) now ship a working git push https://git.moleculesai.app/... out of the box, with the credential helper wired by the platform — NOT by the open-source workspace templates.

Architecture (per CTO 2026-05-18 standing rule)

Follows feedback_open_source_templates_no_hardcoded_org_internals — template-codex / template-hermes / template-openclaw are public repos and MUST NOT carry git.moleculesai.app literals. Integration boundary is the platform (this repo).

  • Docker mode: prepareProvisionContext writes cfg.ConfigFiles[".gitconfig.molecule"] + cfg.ConfigFiles[".git-credentials.molecule"] — same path as .auth_token (via WriteFilesToContainer).
  • SaaS mode: provisionWorkspaceCP calls writeGiteaCredentialHelperViaEIC after cpProv.Start returns instance_id — same EIC channel the canvas Files API uses (writeFileViaEIC).
  • Internal claude-code template (workspace/entrypoint.sh, NOT open-source): 26 lines added to copy the files into agent's home + git config --global include.path so the helper fires under gosu. Idempotent.
  • Open-source templates (codex/hermes/openclaw): untouched in this PR. Follow-up PRs in each repo will add a generic if [ -f /configs/.gitconfig.molecule ]; then … source line — no host literals.

Empirical probe (the load-bearing finding)

2026-05-18 probe on prod Dev-A 4ca4c06c-4e5d-4d80-a2ac-14daad8abc5d:

=== GITEA_TOKEN bytes ===
0
=== GITEA_USER ===

=== MOLECULE_AGENT_ROLE ===

GITEA_TOKEN was NOT arriving in container env. Root cause: agent-dev-a/b persona dirs ship only token + universal-auth.env — NO env file (the post-suspension scoped-identity shape where model creds resolve via Infisical UA at runtime). loadPersonaEnvFile parsed the missing env, silently no-op'd, and the workspace booted with zero git-auth env vars.

What this PR does

Part 1 — get GITEA_TOKEN into env (org_helpers.go)

Extended loadPersonaEnvFile to fall through to a new loadPersonaTokenFile helper that populates GITEA_TOKEN + GITEA_USER + GITEA_USER_EMAIL when the persona ships a bare token file. Precedence: explicit env file always wins. Regular personas (core-be, core-devops, …) unaffected.

Part 2 — write the credential helper (workspace_provision_gitea_creds.go)

New file with pure render functions (renderGitConfigFragment, renderGitCredentialsLine) + validateGitHost (defence-in-depth — refuses https://…, whitespace, embedded @/://) + resolvePersonaGitIdentity + the two injectors. Best-effort failure model matches mintWorkspaceSecrets.

Part 3 — wire into provision flow

  • workspace_provision_shared.go — Docker injection just after buildProvisionerConfig.
  • workspace_provision.go — SaaS injection just after cpProv.Start + instance_id persist.
  • workspace/entrypoint.sh — copy files into /home/agent + git config --global include.path.

Tests (19 new test functions, all passing)

  • renderGitConfigFragment / renderGitCredentialsLine literal-output pins (regression breaks every persona-backed workspace).
  • validateGitHost positive + negative cases (whitespace, scheme prefix, embedded //@/:).
  • resolvePersonaGitIdentity precedence + override + bad-host paths.
  • injectGiteaCredentialHelperConfigFiles happy path + no-op no-clobber.
  • loadPersonaTokenFile fallback (the empirically-verified Dev-A/B shape), env-file-wins precedence, missing-token-file no-op, empty-token-file safety.
ok  github.com/Molecule-AI/molecule-monorepo/platform/internal/handlers  15.747s
go vet ./... — clean

Test plan

  • Unit tests added + green (19/19)
  • Full handlers package suite green (15.7s)
  • go vet ./... clean
  • go build ./... clean
  • Manual live verification on Dev-A/B after merge + staging deploy: provision a fresh workspace under agent-dev-a role, exec git ls-remote https://git.moleculesai.app/molecule-ai/internal HEAD — expect HTTP 200 lines, not 401
  • Hot-patch existing Dev-A/B containers via the SAME mechanism (writeFileViaEIC) — separate burst, NOT this PR

Question for review (load-bearing)

Is the Part-1 fix (loadPersonaTokenFile fallback) the right level? I added it because the empirical probe showed GITEA_TOKEN=0 bytes in prod Dev-A. The alternative is to backfill an explicit env file on the operator host for every scoped-identity persona — but that duplicates SSOT (token lives in token, also lives in env). The fallback keeps token as the single source. Confirm or push back.

Standing-by

DO NOT MERGE. Standing by for review wave. Will drive the live verification + hot-patch on Dev-A/B once the merged change rolls to staging.

Refs

  • Closes the prod-team git-auth gap (Dev-A/B unblock thread)
  • Standing rule: feedback_open_source_templates_no_hardcoded_org_internals
  • Sibling rules: feedback_per_agent_gitea_identity_default, feedback_verify_actual_endstate_not_ack_follow_sop, feedback_dispatch_empirical_probe_first_not_guess, feedback_surface_actionable_failure_reason_to_user
  • Memory: reference_prod_team_infisical_identities (Dev-A/B persona shape)
## Summary Closes the workspace **git-auth gap**: prod-team workspaces (`agent-dev-a`/`agent-dev-b` and the rest of the persona-backed fleet) now ship a working `git push https://git.moleculesai.app/...` out of the box, with the credential helper wired by the **platform** — NOT by the open-source workspace templates. ## Architecture (per CTO 2026-05-18 standing rule) Follows `feedback_open_source_templates_no_hardcoded_org_internals` — `template-codex` / `template-hermes` / `template-openclaw` are public repos and MUST NOT carry `git.moleculesai.app` literals. Integration boundary is the platform (this repo). - **Docker mode:** `prepareProvisionContext` writes `cfg.ConfigFiles[".gitconfig.molecule"]` + `cfg.ConfigFiles[".git-credentials.molecule"]` — same path as `.auth_token` (via `WriteFilesToContainer`). - **SaaS mode:** `provisionWorkspaceCP` calls `writeGiteaCredentialHelperViaEIC` after `cpProv.Start` returns `instance_id` — same EIC channel the canvas Files API uses (`writeFileViaEIC`). - **Internal claude-code template** (`workspace/entrypoint.sh`, NOT open-source): 26 lines added to copy the files into agent's home + `git config --global include.path` so the helper fires under gosu. Idempotent. - **Open-source templates (codex/hermes/openclaw):** untouched in this PR. Follow-up PRs in each repo will add a generic `if [ -f /configs/.gitconfig.molecule ]; then …` source line — no host literals. ## Empirical probe (the load-bearing finding) **2026-05-18 probe on prod Dev-A `4ca4c06c-4e5d-4d80-a2ac-14daad8abc5d`:** ``` === GITEA_TOKEN bytes === 0 === GITEA_USER === === MOLECULE_AGENT_ROLE === ``` GITEA_TOKEN was **NOT** arriving in container env. Root cause: `agent-dev-a/b` persona dirs ship only `token` + `universal-auth.env` — NO `env` file (the post-suspension scoped-identity shape where model creds resolve via Infisical UA at runtime). `loadPersonaEnvFile` parsed the missing `env`, silently no-op'd, and the workspace booted with zero git-auth env vars. ## What this PR does ### Part 1 — get GITEA_TOKEN into env (`org_helpers.go`) Extended `loadPersonaEnvFile` to fall through to a new `loadPersonaTokenFile` helper that populates `GITEA_TOKEN` + `GITEA_USER` + `GITEA_USER_EMAIL` when the persona ships a bare `token` file. Precedence: explicit `env` file always wins. Regular personas (`core-be`, `core-devops`, …) unaffected. ### Part 2 — write the credential helper (`workspace_provision_gitea_creds.go`) New file with pure render functions (`renderGitConfigFragment`, `renderGitCredentialsLine`) + `validateGitHost` (defence-in-depth — refuses `https://…`, whitespace, embedded `@`/`:`/`/`) + `resolvePersonaGitIdentity` + the two injectors. Best-effort failure model matches `mintWorkspaceSecrets`. ### Part 3 — wire into provision flow - `workspace_provision_shared.go` — Docker injection just after `buildProvisionerConfig`. - `workspace_provision.go` — SaaS injection just after `cpProv.Start` + `instance_id` persist. - `workspace/entrypoint.sh` — copy files into `/home/agent` + `git config --global include.path`. ## Tests (19 new test functions, all passing) - `renderGitConfigFragment` / `renderGitCredentialsLine` literal-output pins (regression breaks every persona-backed workspace). - `validateGitHost` positive + negative cases (whitespace, scheme prefix, embedded `/`/`@`/`:`). - `resolvePersonaGitIdentity` precedence + override + bad-host paths. - `injectGiteaCredentialHelperConfigFiles` happy path + no-op no-clobber. - `loadPersonaTokenFile` fallback (the empirically-verified Dev-A/B shape), env-file-wins precedence, missing-token-file no-op, empty-token-file safety. ``` ok github.com/Molecule-AI/molecule-monorepo/platform/internal/handlers 15.747s go vet ./... — clean ``` ## Test plan - [x] Unit tests added + green (19/19) - [x] Full handlers package suite green (15.7s) - [x] `go vet ./...` clean - [x] `go build ./...` clean - [ ] Manual live verification on Dev-A/B after merge + staging deploy: provision a fresh workspace under `agent-dev-a` role, exec `git ls-remote https://git.moleculesai.app/molecule-ai/internal HEAD` — expect HTTP 200 lines, not 401 - [ ] Hot-patch existing Dev-A/B containers via the SAME mechanism (writeFileViaEIC) — separate burst, NOT this PR ## Question for review (load-bearing) **Is the Part-1 fix (`loadPersonaTokenFile` fallback) the right level?** I added it because the empirical probe showed GITEA_TOKEN=0 bytes in prod Dev-A. The alternative is to backfill an explicit `env` file on the operator host for every scoped-identity persona — but that duplicates SSOT (token lives in `token`, also lives in `env`). The fallback keeps `token` as the single source. Confirm or push back. ## Standing-by DO NOT MERGE. Standing by for review wave. Will drive the live verification + hot-patch on Dev-A/B once the merged change rolls to staging. ## Refs - Closes the prod-team git-auth gap (Dev-A/B unblock thread) - Standing rule: `feedback_open_source_templates_no_hardcoded_org_internals` - Sibling rules: `feedback_per_agent_gitea_identity_default`, `feedback_verify_actual_endstate_not_ack_follow_sop`, `feedback_dispatch_empirical_probe_first_not_guess`, `feedback_surface_actionable_failure_reason_to_user` - Memory: `reference_prod_team_infisical_identities` (Dev-A/B persona shape)
hongming added 1 commit 2026-05-18 20:01:34 +00:00
feat(provisioner): inject per-persona Gitea credential helper at provision time
CI / Canvas Deploy Reminder (pull_request) Blocked by required conditions
Details
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
Details
CI / Detect changes (pull_request) Successful in 5s
Details
CI / Shellcheck (E2E scripts) (pull_request) Successful in 9s
Details
E2E API Smoke Test / detect-changes (pull_request) Successful in 10s
Details
Harness Replays / detect-changes (pull_request) Successful in 5s
Details
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
Details
publish-runtime-autobump / bump-and-tag (pull_request) Has been skipped
Details
E2E Chat / detect-changes (pull_request) Successful in 16s
Details
gate-check-v3 / gate-check (pull_request) Successful in 7s
Details
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 15s
Details
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 16s
Details
sop-checklist / na-declarations (pull_request) N/A: (none)
Details
qa-review / approved (pull_request) Successful in 8s
Details
sop-checklist / all-items-acked (pull_request) Successful in 9s
Details
security-review / approved (pull_request) Successful in 11s
Details
sop-tier-check / tier-check (pull_request) Successful in 15s
Details
Harness Replays / Harness Replays (pull_request) Successful in 2s
Details
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 44s
Details
publish-runtime-autobump / pr-validate (pull_request) Successful in 55s
Details
E2E Chat / E2E Chat (pull_request) Failing after 56s
Details
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 56s
Details
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 1m23s
Details
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2m12s
Details
audit-force-merge / audit (pull_request) Waiting to run
Details
CI / Canvas (Next.js) (pull_request) Successful in 4m29s
Details
CI / Platform (Go) (pull_request) Successful in 4m58s
Details
CI / Python Lint & Test (pull_request) Successful in 6m23s
Details
CI / all-required (pull_request) Successful in 6m29s
Details
1de7e26015 
Closes the workspace git-auth gap: prod-team workspaces (agent-dev-a /
agent-dev-b and the rest of the persona-backed fleet) now ship a working
`git push https://git.moleculesai.app/...` out of the box, with the
credential helper wired by the platform — NOT by the open-source
workspace templates.

Architecture (per CTO 2026-05-18 standing rule
feedback_open_source_templates_no_hardcoded_org_internals):

  - template-codex / template-hermes / template-openclaw are public
    repos. They MUST NOT carry `git.moleculesai.app` or any other
    molecule-AI host literal — a third-party fork against THEIR git
    host has to work without editing the template.
  - The integration boundary is the platform (this repo). The
    provisioner drops two files into /configs at provision time —
    `.gitconfig.molecule` (the include-able credential helper fragment)
    and `.git-credentials.molecule` (one URL line: scheme + user + token
    + host) — using the same channel that already delivers .auth_token
    and .platform_inbound_secret. Docker mode writes via
    cfg.ConfigFiles; SaaS mode writes via the existing EIC channel
    (writeFileViaEIC, the same path the canvas Files API uses) after
    cpProv.Start returns the EC2 instance_id.
  - Internal claude-code template (workspace/entrypoint.sh, which lives
    in THIS repo and is not open-source) gets a 5-line copy step + a
    `git config --global include.path` so the helper fires under gosu.
  - Follow-up PRs against template-codex / template-hermes /
    template-openclaw will add a generic no-op-if-absent source line
    — no host literals — to wire the platform-written files into
    those open-source images.

Empirical probe (2026-05-18) on workspace 4ca4c06c-…-aa2c-14daad8abc5d
(prod Dev-A) confirmed the gap: `printenv GITEA_TOKEN | wc -c` → 0,
GITEA_USER empty, MOLECULE_AGENT_ROLE empty. Two-part root cause —

  Part 1: GITEA_TOKEN was never reaching the container. The
  agent-dev-a/b persona dirs ship ONLY a `token` file (post-suspension
  scoped-identity shape — model creds resolve via Infisical universal
  auth, no plaintext `env` file). loadPersonaEnvFile parsed the missing
  `env` and silently no-op'd, leaving the workspace with zero git auth
  env. Fix: extended loadPersonaEnvFile to fall through to a new
  loadPersonaTokenFile helper that populates GITEA_TOKEN +
  GITEA_USER + GITEA_USER_EMAIL when the persona ships a bare `token`
  file. Precedence: explicit `env` file values always win
  (env-file-first → token-file-fallback). Regular personas (core-be,
  core-devops, …) ship `env` and are unaffected.

  Part 2: even with GITEA_TOKEN in env, the container had no
  credential helper. Fix: new workspace_provision_gitea_creds.go renders
  two files, plugs the Docker injection into
  prepareProvisionContext (just after buildProvisionerConfig) and the
  SaaS injection into provisionWorkspaceCP (just after cpProv.Start +
  instance_id persist). Best-effort failure model matches
  mintWorkspaceSecrets — missing creds skip silently; bad GITEA_HOST
  values log and skip; EIC write failures log and continue. `git push`
  surfaces any residual gap loudly (CTO directive
  feedback_surface_actionable_failure_reason_to_user).

Tests added (19 new test functions, all passing):
  - renderGitConfigFragment / renderGitCredentialsLine literal-output
    pins (regression here breaks every persona-backed workspace).
  - validateGitHost positive + negative cases (whitespace, scheme
    prefix, embedded slash/@/colon).
  - resolvePersonaGitIdentity precedence + override + bad-host paths.
  - injectGiteaCredentialHelperConfigFiles happy path + no-op
    no-clobber.
  - loadPersonaTokenFile fallback (the empirically-verified Dev-A/B
    shape), env-file-wins precedence, missing-token-file no-op,
    empty-token-file safety (won't expose GITEA_USER without a token).

Full handlers package: 15.7s, all green. go vet clean. Build clean.

Refs: feedback_open_source_templates_no_hardcoded_org_internals,
feedback_per_agent_gitea_identity_default,
feedback_verify_actual_endstate_not_ack_follow_sop,
feedback_empirical_probe_first_not_guess,
reference_prod_team_infisical_identities,
feedback_surface_actionable_failure_reason_to_user.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hongming commented 2026-05-18 20:04:50 +00:00
Author
Owner
Copy Link
Copy Source

Superseded by #1525 — closing.

This PR uses platform-side ~/.gitconfig + ~/.git-credentials file-write via EIC. CTO 2026-05-18 directed env-only injection instead: GIT_ASKPASS + a generic molecule-askpass shell helper that reads GITEA_USER/GITEA_TOKEN from env. Cleaner architecture (no file-write hooks, no path-mapping concerns, less coupling), and the generic helper is open-source-template-friendly (no git.moleculesai.app literals).

Go to #1525 for the env-only platform PR + 3 sibling template PRs (codex#12, hermes#28, openclaw#24).

The loadPersonaTokenFile fallback finding (agent-dev-a/b have no env file, only token + universal-auth.env) is preserved in #1525's scope — same fix needed regardless of file vs env injection.

**Superseded by #1525 — closing.** This PR uses platform-side `~/.gitconfig` + `~/.git-credentials` file-write via EIC. CTO 2026-05-18 directed env-only injection instead: `GIT_ASKPASS` + a generic `molecule-askpass` shell helper that reads `GITEA_USER`/`GITEA_TOKEN` from env. Cleaner architecture (no file-write hooks, no path-mapping concerns, less coupling), and the generic helper is open-source-template-friendly (no `git.moleculesai.app` literals). Go to **#1525** for the env-only platform PR + 3 sibling template PRs (codex#12, hermes#28, openclaw#24). The `loadPersonaTokenFile` fallback finding (agent-dev-a/b have no `env` file, only `token` + `universal-auth.env`) is preserved in #1525's scope — same fix needed regardless of file vs env injection.
hongming closed this pull request 2026-05-18 20:04:51 +00:00
Some optional checks failed
CI / Canvas Deploy Reminder (pull_request) Blocked by required conditions
Details
Block internal-flavored paths / Block forbidden paths (pull_request) Successful in 4s
Details
CI / Detect changes (pull_request) Successful in 5s
Details
CI / Shellcheck (E2E scripts) (pull_request) Successful in 9s
Details
E2E API Smoke Test / detect-changes (pull_request) Successful in 10s
Details
Harness Replays / detect-changes (pull_request) Successful in 5s
Details
Handlers Postgres Integration / detect-changes (pull_request) Successful in 5s
Details
publish-runtime-autobump / bump-and-tag (pull_request) Has been skipped
Details
E2E Chat / detect-changes (pull_request) Successful in 16s
Details
gate-check-v3 / gate-check (pull_request) Successful in 7s
Details
Secret scan / Scan diff for credential-shaped strings (pull_request) Successful in 15s
Details
Runtime PR-Built Compatibility / detect-changes (pull_request) Successful in 16s
Details
sop-checklist / na-declarations (pull_request) N/A: (none)
Details
qa-review / approved (pull_request) Successful in 8s
Details
sop-checklist / all-items-acked (pull_request) Successful in 9s
Required
Details
security-review / approved (pull_request) Successful in 11s
Details
sop-tier-check / tier-check (pull_request) Successful in 15s
Details
Harness Replays / Harness Replays (pull_request) Successful in 2s
Details
lint-required-no-paths / lint-required-no-paths (pull_request) Successful in 44s
Details
publish-runtime-autobump / pr-validate (pull_request) Successful in 55s
Details
E2E Chat / E2E Chat (pull_request) Failing after 56s
Details
Runtime PR-Built Compatibility / PR-built wheel + import smoke (pull_request) Successful in 56s
Details
E2E API Smoke Test / E2E API Smoke Test (pull_request) Failing after 1m23s
Details
Handlers Postgres Integration / Handlers Postgres Integration (pull_request) Successful in 2m12s
Details
audit-force-merge / audit (pull_request) Waiting to run
Details
CI / Canvas (Next.js) (pull_request) Successful in 4m29s
Details
CI / Platform (Go) (pull_request) Successful in 4m58s
Details
CI / Python Lint & Test (pull_request) Successful in 6m23s
Details
CI / all-required (pull_request) Successful in 6m29s
Required
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
area/ci
CI/CD pipeline issues
 do-not-auto-merge
Opt out of autonomous merge-queue merging
 kind/infrastructure
Infrastructure-related issues
 merge-queue
Ready for serialized Gitea merge queue
 merge-queue-hold
Temporarily hold PR in merge queue
 platform/go
Go platform test issues
 release-blocker
Blocks the staging→main promotion / a release
 release-test
security
test-label-sre
tier:high
High risk per dev-sop §SOP-6 — ceo only, 24h cooldown
 tier:low
Low risk per dev-sop §SOP-6 — engineers/managers/ceo can approve
 tier:medium
Medium risk per dev-sop §SOP-6 — managers/ceo can approve
 triage-test
test
 wip
Work in progress — do not auto-merge
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-dev-a agent-dev-b agent-pm agent-researcher agent-reviewer agent-reviewer-1 agent-reviewer-cr2 app-fe (Molecule AI · app-fe) app-lead (Molecule AI · app-lead) app-qa (Molecule AI · app-qa) claude-ceo-assistant claude-ci-reader claude-status-reaper core-be (Molecule AI · core-be) core-devops (Molecule AI · core-devops) core-fe (Molecule AI · core-fe) core-lead (Molecule AI · core-lead) core-offsec (Molecule AI · core-offsec) core-qa (Molecule AI · core-qa) core-security (Molecule AI · core-security) core-uiux (Molecule AI · core-uiux) cp-be (Molecule AI · cp-be) cp-lead (Molecule AI · cp-lead) cp-qa (Molecule AI · cp-qa) cp-security (Molecule AI · cp-security) cui (Zhanlin Cui) dev-lead (Molecule AI · dev-lead) devops-engineer documentation-specialist (Molecule AI · documentation-specialist) fullstack-engineer (Molecule AI · fullstack-engineer) hongming hongming-ceo-delegated hongming-codex-laptop hongming-kimi-laptop hongming-pc2 infra-lead (Molecule AI · infra-lead) infra-runtime-be (Molecule AI · infra-runtime-be) infra-sre (Molecule AI · infra-sre) integration-tester (Molecule AI · integration-tester) mc-drift-bot molecule-code-reviewer plugin-dev (Molecule AI · plugin-dev) pm publish-runtime-bot pypi-publisher (Molecule AI PyPI Publisher (RFC#596)) release-manager (Molecule AI · release-manager) sdk-dev (Molecule AI · sdk-dev) sdk-lead (Molecule AI · sdk-lead) sop-drift-bot sop-tier-bot (SOP Tier-Check Bot) technical-writer (Molecule AI · technical-writer) triage-operator (Molecule AI · triage-operator)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: molecule-ai/molecule-core#1523
Delete Branch "feat/provisioner-inject-gitea-credential-helper"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?