Skip to content

Troubleshooting

Google status reports the wrong quota project

Section titled “Google status reports the wrong quota project”

Symptom. gcloud’s selected project differs from the project charged or checked for quota by a client library or Cloud MCP call.

Cause. gcloud config get-value project is the CLI resource default, while ADC contains its own quota_project_id. Earlier auth status output incorrectly labeled the CLI default as the ADC quota project. Environment overrides can select another quota project or another credential file entirely.

Confirm. Run bash ~/dotfiles/install/auth.sh google-status. Compare gcloud-project, adc-quota-project, and the override labels without dumping the full credential JSON.

Fix. Update the checkout and use the corrected status command. Change ADC only when the intended project is known, with gcloud auth application-default set-quota-project YOUR_PROJECT_ID. Do not change the CLI project merely to make the two labels agree. See Gemini and Google APIs.

Gemini discovery works but a model request fails

Section titled “Gemini discovery works but a model request fails”

Symptom. df-gemini models lists a model, but generation or Live fails.

Cause. Discovery access does not establish generation quota, model availability, billing, or region access. A 429 can mean an exhausted or zero project quota; a 503 indicates temporary service unavailability. Google-account Pro status does not establish paid Developer API access.

Confirm. Check the same project’s AI Studio rate-limit page and bash ~/dotfiles/install/auth.sh google-status. The probe returns failure and a safe error class/status without exposing its key or WebSocket URL.

Fix. For a 503, retry with backoff. For quota/billing failures, select a model with available quota or deliberately configure paid billing. Do not create extra API keys expecting more quota: project keys share the same pool. See the Google API guide.

Symptom: Fish starts with source: No such file or directory naming an env.fish under an old ~/.local/plat_*/bin directory.

Root cause: uv’s standalone installer wrote ~/.config/fish/conf.d/uv.env.fish with the installation’s fixed path. Changing the runtime layout did not update that snippet; Fish runs conf.d files even for noninteractive commands.

Confirm: inspect ~/.config/fish/conf.d/uv.env.fish, then run fish -lc 'command -s uv; printf "%s\n" $_LOCAL_PLAT'.

Fix: update the managed Fish files with chezmoi apply --exclude=scripts ~/.config/fish/conf.d and open Fish again. The managed environment resolves the runtime root at startup and replaces the legacy uv snippet with an empty ownership marker. The Python installer now uses UV_NO_MODIFY_PATH=1 to prevent future standalone installations from rewriting shell startup. See Fish setup.

Push validation cannot mount its isolated checkout on NFS

Section titled “Push validation cannot mount its isolated checkout on NFS”

Symptom: native tests, documentation, and lint pass, then Docker reports permission denied for .ci-validation/push.*/repo before container tests start.

Root cause: mktemp -d creates a private directory. On an NFS export with root squashing, the Docker daemon cannot traverse that directory even though the invoking user can read the checkout. A restrictive caller umask can also make cloned files unreadable to the container.

Confirm: inspect the permissions of .ci-validation/ and the push.* ancestor while validation runs. A bind mount of the same test source succeeds once the gate-owned ancestors allow traversal and the source is readable.

Fix: use the updated tests/pre-push.sh and retry git push. The gate creates a readable snapshot with traversable ancestors and no write access for other users; it leaves the original checkout permissions unchanged. The checkout’s existing parent directories must also be accessible to Docker. See validation before publishing.

Docker bootstrap build reports invalid Ubuntu package signatures

Section titled “Docker bootstrap build reports invalid Ubuntu package signatures”

Symptom: ./tests/run.sh fails during apt-get update with At least one invalid signature was encountered, before running tests.

Possible root cause: A full Docker data disk prevents complete package index writes. On the affected Colima VM, /var/lib/docker was at 100% with zero available space. This observation does not rule out other signature errors.

Confirm: Use colima ssh -- df -h / /var/lib/docker and docker system df to distinguish VM storage exhaustion from host free space.

Fix: After choosing an explicitly authorized cleanup or disk expansion, restore free space and rerun ./tests/run.sh. Do not disable signature checking or treat this pre-test environment failure as a passing suite.

For an isolated test profile, verify the daemon before running the gate:

Terminal window
env -u DOCKER_HOST DOCKER_CONTEXT=colima-PROFILE docker info --format '{{.Name}}'

Replace PROFILE with an existing profile. An inherited DOCKER_HOST can keep Docker pointed at the default socket despite selecting another context.

Symptom: Validation prints Fast Bats tests, then stops before the test count. A bats-preprocess process remains idle.

Observed cause: On macOS with Homebrew Bash 5.3.15, a process sample showed bats-preprocess blocked in heredoc_write → write while preparing a here-string. This is a shell redirection stall before test execution; it does not indicate a failing test.

Confirm: Inspect the command and sample the affected PID:

Terminal window
ps -axo pid,ppid,etime,command | rg bats-preprocess
macprof sample PID 2 -o - --keep-idle

Workaround: Use Bash’s temporary-file redirection behavior for this invocation:

Terminal window
BASH_COMPAT=50 bats --count tests/agents.bats
BASH_COMPAT=50 ./tests/ci.sh full
# Or apply the same setting to the required outgoing-commit gate:
BASH_COMPAT=50 git push origin main

The count returned 22 on the affected checkout. The full gate still must pass. Keep this setting local to the command; compatibility mode changes other Bash semantics too. See the Bash maintainer’s explanation of here-document compatibility.

Agent shell loops report installed commands as missing

Section titled “Agent shell loops report installed commands as missing”

Symptom: rg, curl, or stat works initially, then a generated zsh loop prints repeated command not found errors. Assigning status can also fail.

Root cause: zsh’s path array is tied to PATH; assigning a file name to path replaces command search directories. status is a read-only parameter.

Fix: Use variables such as file_path and exit_status. Select Bash explicitly for scripts written for Bash. This is a shell-variable problem, not a reason to reinstall the missing-looking commands.

Bundled skill validation fails with missing yaml

Section titled “Bundled skill validation fails with missing yaml”

Symptom: quick_validate.py exits with ModuleNotFoundError: No module named 'yaml' before validating a skill.

Root cause: The bundled validator imports PyYAML using the plain python3 environment. PyYAML was absent from the managed interactive library manifest.

Confirm: python3 -c 'import yaml; print(yaml.__version__)'.

Fix: PyYAML is now declared in packages/python.txt; install/python.sh installs it alongside SymPy and the public runtime verifier checks both imports. For an isolated invocation using a different Python, run uv run --with pyyaml /path/to/quick_validate.py /path/to/skill.

Agent test commands retry unsupported quiet or filter arguments

Section titled “Agent test commands retry unsupported quiet or filter arguments”

Symptom: cargo nextest run -q rejects -q, or cargo test first second rejects the second positional filter.

Fix: Follow the project’s test recipe. For nextest, use cargo nextest run --cargo-quiet --status-level fail --final-status-level fail. For a selected nextest subset, use its -E 'test(first) | test(second)' expression. Cargo test accepts one positional substring filter; run separate commands for independent filters, and run cargo test --doc for doctests. Keep test exit codes visible when filtering logs.

Large tool responses defeat output compression

Section titled “Large tool responses defeat output compression”

Symptom: Returning whole downloads or JSON documents fills agent context even with RTK enabled.

Fix: Save complete downloads/logs to files, return selected fields using jq or the CLI’s --json/--jq options, and inspect bounded excerpts. RTK’s estimated savings are not a guarantee that every command is compressed. Batch independent reads and prefer a completion-aware wait (for example gh run watch RUN_ID --exit-status) over rapid repeated status polling.

ShellCheck rejects the runtime-selected platform environment

Section titled “ShellCheck rejects the runtime-selected platform environment”

Symptom: Quality and macOS smoke fail with SC1090 in install/_lib.sh, while the Docker bootstrap suites pass.

Root cause: The platform environment path contains a runtime-selected $PLAT, so ShellCheck cannot resolve it. The Docker suite does not run the separate CI lint gate.

Confirm: Run shellcheck -S warning bootstrap.sh install/*.sh install/plat/*/.plat_env.sh home/dot_local/bin/executable_git-wt from the repo.

Fix: Mark only the dynamic import with source=/dev/null and lint every actual platform environment explicitly in both CI jobs. Keep the global lint severity unchanged; a passing bootstrap suite does not replace this check. The jobs now share tests/ci.sh; enable the push gate so the same checks validate the outgoing commit before Git publishes it.

Local secret scanning reports files outside the outgoing branch

Section titled “Local secret scanning reports files outside the outgoing branch”

Symptom: The local push gate reports archived transcript paths that do not exist in the outgoing branch, while hosted CI sees no corresponding files.

Root cause: Gitleaks defaults to scanning all local refs. An independent validation clone can also contain unrelated checkpoint branches; tool names in those transcripts can resemble credential formats.

Confirm: Inspect the redacted finding’s commit and run git for-each-ref --contains COMMIT --format='%(refname)'. Do not print raw matches or assume every finding is a false positive.

Fix: The shared quality command uses gitleaks git --log-opts=HEAD to scan the outgoing commit and its full history. It does not bypass secret checks or ignore credentials removed from earlier commits in that history.

Hosted Codex runtime and backup tests fail while local tests pass

Section titled “Hosted Codex runtime and backup tests fail while local tests pass”

Symptom: Several Codex runtime tests fail together, and the history/backup tests report uv exiting 127 on the hosted Quality runner.

Root cause: These tests use uv to run Python validation. It was present on the developer machine and installed by Docker bootstrap, but absent from the separately provisioned hosted Quality environment.

Confirm: Check for Command not found on uv run in the Bats warnings. The fast test entrypoint now requires uv before it starts any fixtures.

Fix: Provision uv through the existing SHA-pinned validation-tool action. Its manifest pins uv’s version with the action revision. Keep the dependency check and real runtime/backup tests enabled.

Local fixture tests overwrite a managed tool

Section titled “Local fixture tests overwrite a managed tool”

Symptom: Remote-bootstrap tests fail to find their fake chezmoi, while the real chezmoi --version unexpectedly prints chezmoi version test. Fish profile fixtures can also resolve live runtime directories instead of their fake home.

Root cause: Setting a fixture’s HOME does not override exported DF_TOOLS_ROOT, DF_STATE_ROOT, CODEX_HOME, or other managed runtime variables. The fixture can therefore write into the real installation. A host overlay can also restore these roots if the test only unsets the variables.

Confirm: Inspect the executable reported by command -v chezmoi with file; the fixture stub is a short shell script containing the test version. Inspect path variables without printing credential values.

Fix: Stop the affected test run, preserve the confirmed stub outside the managed executable path, and restore chezmoi with bash install/chezmoi.sh. Use the updated tests/ci.sh, whose Bats subprocess clears inherited runtime and build settings and exports empty root overrides. Writing fixtures also set their temporary roots explicitly. Rerun ./tests/ci.sh fast; the regression checks require inherited tool and state directories to remain untouched.

Quality CI fails with chezmoi: command not found

Section titled “Quality CI fails with chezmoi: command not found”

Symptom: The fast Bats profile-rendering test exits 127, while the Docker bootstrap jobs pass.

Root cause: The Quality job runs directly on the hosted runner. Unlike the bootstrap container, it did not install chezmoi before rendering login profiles.

Confirm: Check the failed step for tests/profiles.bats and the missing chezmoi command; this is a test-environment dependency failure.

Fix: The Quality job installs a pinned, checksum-verified chezmoi release binary into its tool directory before running Bats. It also installs zsh because the test sources both bash and zsh profiles. Keep these dependencies in .github/workflows/ci.yml; do not skip the rendering test. Use the release binary because this chezmoi version’s Go module has exclude directives and cannot be installed with go install github.com/twpayne/chezmoi/v2@version.

Native OpenCode test cannot find an installed binary

Section titled “Native OpenCode test cannot find an installed binary”

Symptom: A fresh Linux bootstrap installs OpenCode successfully, but tests/opencode-native-argv.bats fails with FileNotFoundError: 'opencode'.

Root cause: The test entrypoint resolved its Node path before bootstrap installed nvm. Bootstrap runs in a child shell, so its updated PATH cannot reach the parent Bats runner. Tests that source installer helpers again can pass while tests using the inherited environment fail.

Confirm: Read the native argv test’s captured exception and check whether the runner’s PATH contains the installed nvm default’s bin directory.

Fix: tests/entrypoint.sh resolves the nvm default after bootstrap and adds it to the shared test environment. Keep this setup in the entrypoint so native commands work without per-test environment repairs.

Local agents default to MLX but localhost:8080 is unavailable

Section titled “Local agents default to MLX but localhost:8080 is unavailable”

Symptom: Pi or OpenCode starts with the configured local model but cannot connect to port 8080.

Root cause: Local models are the macOS client defaults, while heavyweight login services are intentionally disabled. A working CLI does not mean its inference backend is loaded.

Confirm: Inspect local-agent status and the client model selection. A managed default uses mlx/qwen3.6-27b; a different configured backend should be started through its own workflow.

Fix: Run local-agent pi or local-agent opencode. The helper verifies the configured model, starts the managed MLX server on demand if the port is free, and leaves login policy unchanged. It uses cached weights only; run bash ~/dotfiles/install/local-llm.sh pull-models explicitly when weights are missing. Logs are in $LOCAL_PLAT/share/mlxserve/on-demand/server.log.

For fresh DF_PROFILE=core bootstraps, local inference is now skipped by default because its packages belong to the full Python manifest. OpenCode’s config still reconciles independently.

Upgrade retains an old OpenCode or Ollama package owner

Section titled “Upgrade retains an old OpenCode or Ollama package owner”

Symptom: An existing machine still has the OpenCode Homebrew formula or an Ollama.app server after its package registry selects npm OpenCode and formula Ollama.

Root cause: Removing a package declaration does not uninstall an existing package or transfer a running service.

Confirm: Use brew list --versions opencode ollama, command -v opencode, and curl --noproxy '*' -fsS http://127.0.0.1:11434/api/version.

Fix: Rerun bash ~/dotfiles/install/node.sh for OpenCode and bash ~/dotfiles/install/macos-services.sh for Ollama. The migrations verify the replacement, preserve rollback receipts, and retain user configuration and models. Ollama migration stops when models are loaded, connections are active, or port ownership cannot be verified; finish those requests before retrying.

Quick reference for when things go wrong. Check here before digging into scripts.


Colima intermittently rewrites or breaks ~/.ssh/config

Section titled “Colima intermittently rewrites or breaks ~/.ssh/config”

Symptom: tools that parse ~/.ssh/config fail intermittently, or chezmoi diff alternates between a clean file and a leading Colima Include.

Root cause: with sshConfig: true, every Colima start may prepend its exact ~/.colima/ssh_config include. A globbed include in the managed template does not satisfy Colima’s exact-path check, so Colima and chezmoi repeatedly rewrite the same file. Colima writes the file in place, which can also expose a partial config to concurrent readers.

Confirm:

Terminal window
grep '^sshConfig:' ~/.colima/*/colima.yaml
grep -n '\.colima/.*ssh_config' ~/.ssh/config
chezmoi diff ~/.ssh/config

Fix: update the dotfiles checkout and rerun bootstrap.sh. The macOS services installer sets sshConfig: false in Colima’s default template and all existing profiles, while chezmoi removes the Colima include from the managed SSH config. Docker is unaffected. Use colima ssh to enter the VM, or use the separately generated config without loading it globally:

Terminal window
ssh -F ~/.colima/ssh_config colima

Terminal window
echo "$_PLAT" "$_LOCAL_PLAT" # capability + install root
ls "$_LOCAL_PLAT/bin/" # chezmoi, uv, claude should be here
ls "$_LOCAL_PLAT/cargo/bin/" # fd, sd, zoxide, etc.
which fd # should point under $_LOCAL_PLAT

$_LOCAL_PLAT is $HOME/.local by default (flat layout) or $HOME/.local/$_PLAT when PLAT isolation is enabled. If $_PLAT or $_LOCAL_PLAT is empty, .zprofile wasn’t sourced. Open a new login shell (zsh -l) or source it:

Terminal window
source ~/.zprofile

Codex install fails with marketplace unavailable: openai-bundled

Section titled “Codex install fails with marketplace unavailable: openai-bundled”

Symptom: install/codex.sh reports that openai-bundled is unavailable even though codex login status says the user is authenticated.

Root cause: openai-bundled is a local marketplace owned and registered by Codex Desktop. Authentication does not expose it to the standalone CLI, whose built-in marketplace is openai-curated. The CLI-managed packages/codex-plugins.txt must therefore contain only plugins from marketplaces reported by codex plugin marketplace list.

Confirm:

Terminal window
codex login status
codex plugin marketplace list --json

Fix: update the dotfiles checkout and rerun bootstrap.sh. Bundled plugins remain owned by Codex Desktop; do not register the app’s internal plugin path manually because that can conflict with the app’s marketplace reconciliation.


Codex plugin healthcheck reports a plugin was dropped upstream

Section titled “Codex plugin healthcheck reports a plugin was dropped upstream”

Symptom: install/codex.sh dies with Declared Codex plugin was dropped upstream: <plugin>@openai-curated, even though the plugin still appears in the curated marketplace.

Root cause: codex-cli 0.153.0 changed plugin list --json: uninstalled plugins are omitted unless --available is passed, and an unscoped listing can prefer a same-named remote plugin over an installed local-marketplace plugin. Older install/codex.sh code read that incomplete global inventory and mistook the missing row for an upstream removal.

Confirm using the marketplace-scoped complete inventory:

Terminal window
codex plugin list --marketplace openai-curated --available --json \
| jq -r '.installed[].pluginId, .available[].pluginId'

Fix: update the dotfiles checkout and rerun bootstrap.sh. The installer now queries each declared marketplace with --available, so collisions with remote plugins do not hide installed curated plugins. If the scoped inventory really does omit a declared selector, prune or replace that selector in packages/codex-plugins.txt; declarations remain a hard contract.

The WARNING: failed to clean up stale arg0 temp dirs: Directory not empty line from codex-cli is unrelated NFS noise (.nfs* files in its temp dir) — harmless.


Claude cannot load required remote managed settings

Section titled “Claude cannot load required remote managed settings”

Symptom: install/claude.sh stops at the marketplace inventory with Your organization requires remote managed settings to load, but they could not be loaded.

Root cause: an organization with forceRemoteSettingsRefresh requires a fresh policy fetch before Claude starts. A cached policy does not satisfy that requirement. Authentication commands remain available so the user can sign in again. See server-managed settings.

Confirm with claude plugin marketplace list. For the fetch cause, use a private debug log:

Terminal window
umask 077
debug_dir=$(mktemp -d "${TMPDIR:-/tmp}/claude-policy.XXXXXX")
claude --debug-file "$debug_dir/debug.log" plugin marketplace list

Inspect the Remote settings lines locally. http_401 means the server rejected the credential; claude auth status reporting loggedIn: true alone does not prove the server accepts it. Console API keys are static credentials, while OAuth profiles support automatic token refresh. See authentication.

Fix for an authentication rejection: refresh the sign-in, verify that the inventory loads, then rerun bootstrap:

Terminal window
claude auth login
claude plugin marketplace list
~/dotfiles/bootstrap.sh upgrade

claude auth login opens the supported sign-in flow. For network or policy-delivery failures instead, restore access to api.anthropic.com or contact the organization administrator. Preserve the managed policy and cache; changing them does not repair rejected credentials. The installer checks this prerequisite once and stops before dependent plugin operations if it fails.


Claude plugin fails: Plugin "X" not found in any configured marketplace

Section titled “Claude plugin fails: Plugin "X" not found in any configured marketplace”

Symptom: install/claude.sh logs [warn] fail <plugin>: … ✘ Failed to install plugin "<plugin>": Plugin "<plugin>" not found in any configured marketplace, but the plugin visibly exists in the marketplace’s GitHub repo.

Root cause: plugin installs resolve against the local marketplace clones under ~/.claude/plugins/marketplaces/, and with DISABLE_AUTOUPDATER=1 those never refresh themselves. A plugin added upstream after the clone date is invisible (the claude-plugins-official clone once sat 4 months stale while math-olympiad existed upstream). claude.sh used to refresh catalogs only in upgrade mode — and even that call was broken, passing a nonexistent --all flag whose error was silenced by >/dev/null || true, so no mode ever refreshed. It now refreshes (with the correct no-name form) in every mode.

Confirm — compare the clone date against upstream:

Terminal window
git -C ~/.claude/plugins/marketplaces/<marketplace> log -1 --format=%cd
jq -r '.plugins[].name' \
~/.claude/plugins/marketplaces/<marketplace>/.claude-plugin/marketplace.json | grep <plugin>

Fix: update the checkout and rerun bootstrap.sh (or install/claude.sh). Manual one-off:

Terminal window
claude plugin marketplace update <marketplace>
claude plugin install <plugin>@<marketplace>

The same VS Code / Cursor extensions report fail on every upgrade run

Section titled “The same VS Code / Cursor extensions report fail on every upgrade run”

Symptom: bootstrap.sh upgrade logs fail <ext-id> for a fixed set of extensions, run after run — yet the extensions are installed and working in the editor. Install mode never reports them.

Root cause: upgrade mode used to reinstall each declared extension with --install-extension <id> --force, which re-resolves the ID against the editor’s marketplace. Two categories can never satisfy that:

  • Extensions the editor now bundles. VS Code ships github.copilot-chat built in (0.59.0); the marketplace copy is older (0.48.1) and the CLI refuses the downgrade outright: is a built-in extension … and cannot be downgraded.
  • IDs the marketplace doesn’t carry. Cursor resolves against Open VSX, so Microsoft-proprietary IDs fail with Extension '<id>' not found even when the extension is installed — Cursor imported it from VS Code on first run, a path the CLI can’t reproduce. nvidia.nsight-vscode-edition is refused explicitly: not available in Cursor for the Mac Silicon.

Confirm — run the install by hand to reproduce the marketplace error reported by the now-failing bootstrap step:

Terminal window
code --install-extension <id> --force # or: cursor --install-extension …

Fix: update the checkout and rerun. vscode.sh / cursor.sh now upgrade with a single --update-extensions bulk pass instead of per-extension --force, which only touches what the editor can actually resolve. Drop bundled extensions from packages/vscode-extensions.txt, and keep IDs Open VSX can’t serve out of packages/cursor-extensions.txt (that file’s header lists the known-unavailable set and the Anysphere forks to use instead).


Cursor reports ENOENT for User/settings.json and ignores user settings

Section titled “Cursor reports ENOENT for User/settings.json and ignores user settings”

Symptom: Cursor logs or displays an error such as:

ENOENT: no such file or directory, open '.../Cursor/User/settings.json'

The native file is still a symlink, but its managed target is missing or empty:

Terminal window
ls -l "$HOME/Library/Application Support/Cursor/User/settings.json"
jq -e 'type == "object"' ~/.config/cursor/settings.json
git diff -- home/dot_config/cursor/settings.json

Root cause: Cursor writes the symlinked settings file non-atomically. The Cursor agent hook could run after the file was truncated but before the replacement contents arrived, and chezmoi add then copied the empty file into the repo. The hook’s jq cleanup also accepted empty input as success, preserving the damage. A later failed rewrite can leave the native symlink dangling.

Fix: update the checkout. The hook now accepts only a complete settings object or keybindings array, then validates the chezmoi source after import and restores its previous contents if the file changed during the copy. To recover an already-empty source after confirming the diff contains no wanted edits:

Terminal window
git restore --source=HEAD -- home/dot_config/cursor/settings.json
chezmoi apply ~/.config/cursor/settings.json ~/.cursor/hooks/sync-dotfiles-cursor.sh

If an open window still shows defaults after the native symlink resolves, run Developer: Reload Window from Cursor’s command palette.


Cursor computer use reports Ready: no, or screenshots never appear

Section titled “Cursor computer use reports Ready: no, or screenshots never appear”

Symptom: agent worker debug shows a worker, but Computer use Ready: no and screenshot or click tasks fail. The My Machines worker may be loaded.

Confirm:

Terminal window
launchctl print "gui/$(id -u)/dev.cade.cursor-worker"
ls -d "$HOME/.cursor/cursor-computer-use/Cursor Computer Use.app"
agent worker debug

Root cause: computer use is implemented by Cursor Computer Use (co.anysphere.cursor-computer-use), a helper installed on the first --computer-use start. Accessibility and Screen Recording must be granted to that app, not Terminal or Cursor.app. Sequoia and later may re-prompt Screen Recording. agent worker debug does not prove those grants. This repo does not write TCC.db.

Only one worker daemon can own ~/.local/share/cursor-agent. Cursor’s in-app worker (often named like church) holds that lock without computer use. df-cursor-worker passes --wait so the LaunchAgent waits for that process instead of crash-looping. To take over with computer use, stop the in-app worker, then bash ~/dotfiles/install/cursor.sh sync-worker.

Fix: open System Settings → Privacy & Security, grant Accessibility and Screen Recording to Cursor Computer Use, then rerun a screenshot task. If the helper app is missing, start the worker once (bash ~/dotfiles/install/cursor.sh sync-worker) so it can install, then grant TCC. To stop the worker: DF_CURSOR_WORKER=0 bash ~/dotfiles/install/cursor.sh.

Cloud /in-cloud sessions use native VM computer use and dashboard MCP, not this Mac’s helper app or ~/.cursor/mcp.json.


Symptom: bash install/cursor.sh check fails on exploreSubagentModel, or jq -r .exploreSubagentModel ~/.cursor/cli-config.json is default after a Cursor or agent start. permissions.allow may shrink to Shell(ls).

Confirm:

Terminal window
jq '{exploreSubagentModel,permissions}' ~/.cursor/cli-config.json
bash ~/dotfiles/install/cursor.sh check

Root cause: ~/.cursor/cli-config.json is app-owned at runtime. The installer merge-pins Explore to Composer 2.5 and unique-appends Shell(*); the running app rewrites those keys. Parent selectedModel is intentionally left alone.

Fix: bash ~/dotfiles/install/cursor.sh sync-cli. That restores Explore and Shell(*) without changing the parent chat picker.


Brew bundle fails: No available formula … This command requires the tap

Section titled “Brew bundle fails: No available formula … This command requires the tap”

Symptom: brew bundle errors with No available formula with the name "owner/tap/formula". This command requires the tap owner/tap. If you trust this tap, tap it explicitly and then try again: brew tap owner/tap — even though the Brewfile has the tap "owner/tap" line and the tap is already trusted.

Root cause: two separate Homebrew gates protect third-party taps — trust (HOMEBREW_REQUIRE_TAP_TRUST) and the tap actually being cloned. Homebrew no longer auto-taps from a fully-qualified formula name, and brew bundle can hit formula resolution before executing the Brewfile’s own tap directive — in particular the upgrade check for a formula already installed under the same name from homebrew/core (seen with rtk: core keg installed, rtk-ai/tap/rtk in the Brewfile, tap trusted but never tapped → resolution error every run).

Confirm:

Terminal window
brew tap # tap missing from the list
jq . ~/.homebrew/trust.json # …while already trusted here

Fix: update the checkout and rerun — ensure_brewfile_taps() (_lib.sh) now trusts and taps every tap referenced by the Brewfile before the bundle. Manual one-off: brew tap owner/tap, then rerun install/homebrew.sh.


Brew bundle appears stuck after Using lines

Section titled “Brew bundle appears stuck after Using lines”

Symptom: bootstrap prints its package inventory, ending in a line such as Using surrealdb/tap/surreal, then produces no visible build output.

Root cause: current Homebrew Bundle lists satisfied and actionable entries before running one batched install. Without --verbose, it captures that child’s output and can discard it on success. The last inventory line does not identify the running package or prove the process is hung.

Confirm: on Linux, inspect the original bootstrap’s descendants:

Terminal window
pgrep -af 'bootstrap.sh'
pstree -ap BOOTSTRAP_PID

A compiler beneath brew establishes an active build. A different installer means Bundle has already returned. If the tree names google-mcp.py, check Python’s cache lock wait.

Fix: the Linux and macOS package installers now pass --verbose to Bundle, including Linux’s single verified retry. After the original process exits, rerun ~/dotfiles/bootstrap.sh upgrade with the updated sources. This exposes build output; it does not shorten a build or bypass dependency checks. Sources: Homebrew Bundle, linux-packages.sh, and homebrew.sh.

Symptom: Python setup stops at Preparing the refreshable Google MCP transport; uv run --locked --script ... --check-runtime stays idle.

Root cause: the script environment cache lives under a shared NFS tool root. uv locks its environment before installing dependencies, and an NFS lock request can wait in the kernel even though no local process holds the lock. On Blackwell, observed on 2026-10-12, the cache was on NFSv3 and a uv worker’s wait channel was nlmclnt_wait. The same locked check completed with its cache on the host’s ext4 state volume.

Confirm on Linux: select the affected uv PID and inspect its actual cache and worker threads:

Terminal window
pgrep -af 'uv.*google-mcp.py'
findmnt -T "$UV_CACHE_DIR" -o TARGET,FSTYPE
ps -L -p UV_PID -o pid,tid,stat,wchan:30,args

If the calling shell does not have UV_CACHE_DIR, read only that variable from /proc/UV_PID/environ; do not print the complete environment.

Fix: configure a persistent host-local DF_STATE_ROOT with bash ~/dotfiles/install/host.sh configure. Current installers and managed Google/Gemini launchers then use $DF_STATE_ROOT/uv/cache. An explicit UV_CACHE_DIR still wins. Stop the stuck bootstrap in its terminal before starting another; with Blackwell’s existing local state configuration, the one-invocation recovery is:

Terminal window
UV_CACHE_DIR="$DF_STATE_ROOT/uv/cache" ~/dotfiles/bootstrap.sh upgrade

The first run populates a fresh cache. Existing processes retain the old path; do not delete a lock file or disable locking. This change does not relocate installed Python/tool environments or fix project environments stored on NFS. Sources: uv cache safety, _runtime-paths.sh, python.sh, and package cache defaults.

Brew bundle reports Upgrading X has failed! after installing X

Section titled “Brew bundle reports Upgrading X has failed! after installing X”

Symptom: bootstrap.sh upgrade pours and links a formula successfully, then reports Upgrading <formula> has failed!. Nearby errors name a vanished file in ~/.cache/Homebrew/downloads/, such as No such file or directory @ dir_s_rmdir - ...bottle_manifest.json. Several unrelated formulae can fail this way in one run.

Root cause: Homebrew Bundle defaults to as many as four package workers. Those workers launch separate brew install or brew upgrade processes that share one download cache and run install cleanup against it. One worker can remove a cache entry after another has inspected it, turning successful installs into nonzero exits. Homebrew tracks the broader parallel-worker race as Homebrew/brew#23328; the Homebrew manpage documents the auto job default and the sequential override.

Confirm after the original bootstrap process has exited:

Terminal window
formula=tree
brew list --versions "$formula"
brew outdated --formula "$formula"
brew linkage --test "$formula"

If the new version is listed, brew outdated prints nothing, and linkage passes, the install succeeded and only its cleanup path failed.

Graphviz has a second version of this symptom. Netpbm fetches its source and manual from Subversion. Bundle can queue those SVN fetches before it finishes installing Graphviz’s Subversion dependency, record You must: brew install svn, then install Subversion, retry both checkouts, and successfully build Netpbm and Graphviz. The early fetch result still makes Bundle print Upgrading graphviz has failed! after the successful install.

Fix: update the checkout and rerun bootstrap.sh upgrade. The shared installer environment now disables Bundle package jobs; downloads and each source build can still run concurrently, but package installs and cleanup are serialized. On Linux it also installs a working Subversion before a Brewfile containing Graphviz, then runs brew bundle check after any nonzero Bundle exit. If that check passes, it retries Bundle once and reports recovery only when the clean retry exits zero. Manual one-off:

Terminal window
brew bundle install --jobs=1 --file="$HOME/dotfiles/packages/Brewfile"

Do not start the retry while the first bootstrap is still running.


Brew cleanup fails with Device or resource busy .../.nfs...

Section titled “Brew cleanup fails with Device or resource busy .../.nfs...”

Symptom: an upgrade prints a formula’s beer-mug success line, then fails while cleanup removes an unrelated old keg:

Error: Device or resource busy @ apply2files - .../Cellar/expat/<version>/lib/.nfs...

Root cause: NFS renames an unlinked-but-open file to .nfs* and keeps it until the last process closes it. Any Homebrew executable or shared library can be a holder: this first appeared with the Bash running bootstrap, then with dozens of long-lived dbus-daemon processes mapping an old libexpat.so. Homebrew runs formula cleanup after installs and, every 30 days, a full cleanup. That cleanup exception changes the command’s exit status after the package succeeds, so Bundle misleadingly reports Upgrading <formula> has failed! for each later package too.

Confirm which processes still hold the file:

Terminal window
lsof /path/from/the/error/.nfs...

Fix: wait for the original bootstrap to exit, update the checkout, and retry. On an NFS Homebrew prefix, linux-packages.sh sets the documented HOMEBREW_NO_INSTALL_CLEANUP switch for the run. Installs and upgrades still happen, but cleanup cannot turn their success into failure. The switch does not disable an explicit cleanup; after every process shown by lsof has exited, reclaim the retired keg with:

Terminal window
brew cleanup expat

Old kegs consume some disk until that maintenance succeeds. Do not delete the .nfs* file manually or terminate unrelated holders just to make cleanup pass.


Ruby upgrade writes outside its new keg during make install

Section titled “Ruby upgrade writes outside its new keg during make install”

Symptom: upgrading vim, ccache, or another Ruby dependent builds Ruby and then fails under the global Homebrew Ruby directory:

Dir.mkdir: Permission denied @ dir_s_mkdir - .../brew/lib/ruby

The same contamination can later surface as Errno::ENOENT under .../brew/lib/ruby/gems/... during RubyGems setup.

Root cause: the formula adds the versioned [email protected] path as a compatibility fallback. runruby supplies the build directory through LD_LIBRARY_PATH, but Homebrew’s GCC emits DT_RPATH, which the dynamic loader searches first. During make install, the new Cellar lib directory is not populated yet, so the build executable falls through to the previous keg’s libruby. The source RUBYLIB also lacks RubyGems’ optional defaults/operating_system.rb; its require can therefore find the previous keg’s file. Those old Homebrew defaults redirect Gem.default_dir and Gem.ruby outside the new keg. The Linux filesystem sandbox correctly rejects that write; the prefix permissions are not broken.

Fix: install/patch-homebrew-ruby.sh patches the local formula before Brew Bundle runs. It enables new ELF dtags so DT_RUNPATH yields to the build-tree library path, retains the new keg before the versioned fallback after install, and adds a build-local empty RubyGems packager-default file so the previous keg’s override cannot leak in. The formula replaces that empty file with the current Homebrew configuration after installation.

This formula-local DT_RUNPATH is a deliberate exception to the prefix’s usual DT_RPATH policy: the build runner must let its temporary LD_LIBRARY_PATH select the new build-tree libruby.

Do not chmod the prefix or disable Homebrew’s Linux sandbox. Wait for any active Homebrew process to exit, then rerun ~/dotfiles/bootstrap.sh upgrade so the formula refresh and patch happen in the intended order.


Symptom: a source build of apache-serf invokes a brewed GCC directly and fails through Homebrew’s glibc headers:

glibc/include/bits/socket.h: fatal error: asm/socket.h: No such file or directory

Root cause: Homebrew’s standard build environment already puts the installed [email protected] include directory in CPATH. Serf’s SConstruct creates a new SCons child environment that does not inherit that variable, so the direct GCC command loses the kernel-header path. Adding the path to superenv or CPATH again does not cross this second environment boundary.

Fix: install/patch-homebrew-apache-serf.sh adds a direct Linux dependency on [email protected] and passes its stable opt_include path through Serf’s supported CPPFLAGS SCons variable. The patch fails closed if the formula structure changes instead of silently starting another known-broken source build.

A trailing Clang warning about which GCC installation it may prefer is separate from this GCC compile failure. Wait for any active Homebrew process to exit, then rerun ~/dotfiles/bootstrap.sh upgrade so the refreshed formula is patched before Bundle starts.


Gecode patch reports that its configure target moved

Section titled “Gecode patch reports that its configure target moved”

Symptom: bootstrap records this degradation even though MiniZinc may still finish installing:

gecode configure patch target not found — formula may have changed

Root cause: Homebrew changed the Gecode formula from Autotools flags such as --enable-qt to CMake settings such as GECODE_ENABLE_GIST. The dependency guard could still apply while the old configure anchor no longer existed, leaving a partial formula edit and a misleading successful patch status.

Fix: install/patch-homebrew-gecode.sh recognizes both formula shapes, forbids the upstream bottle on Linux, and sets Gist and Qt off while retaining the macOS bottle and GUI. The bottle gate matters because build flags cannot change a bottle that already contains libgecodegist and Qt dependencies. The installer rebuilds an existing Gist-bearing keg from source and requires both brew linkage --test gecode and the absence of libgecodegist.so; a moved anchor stops source builds instead of leaving a partial formula edit.


Clang cannot load libz3 during an in-progress upgrade

Section titled “Clang cannot load libz3 during an in-progress upgrade”

Symptom: a formula failure ends with a separate loader error such as:

clang: error while loading shared libraries: libz3.so.4.15: cannot open shared object file

Root cause: Bundle upgraded Z3 before unversioned LLVM. The installed LLVM still needs Z3’s previous major SONAME, while opt/z3 already selects the new keg. This is independent of a formula that was compiled with GCC and usually repairs itself when the same Bundle run reaches LLVM.

The Linux installer reconciles this pair before Bundle: it upgrades an outdated LLVM in upgrade mode, reinstalls a current keg whose linkage is broken, and checks both brew linkage and the unversioned Clang executable again after Bundle.

Do not point the old SONAME at the new major library or repoint opt/z3 during the active transaction. After all Homebrew processes exit, rerun ~/dotfiles/bootstrap.sh upgrade. For a manual check, run:

Terminal window
"$(brew --prefix)/bin/clang" --version

If it still reports the old Z3 SONAME, upgrade an outdated LLVM or reinstall a current but broken keg, then verify its linkage:

Terminal window
if [[ -n "$(brew outdated --formula llvm)" ]]; then
brew upgrade llvm
else
brew reinstall llvm
fi
brew linkage --test llvm
"$(brew --prefix)/bin/clang" --version

OpenSSH upgrade fails with inreplace failed ... sshd_config

Section titled “OpenSSH upgrade fails with inreplace failed ... sshd_config”

Symptom: OpenSSH finishes make install, then Homebrew aborts while replacing its Cellar prefix in the persistent configuration:

Error: inreplace failed
.../brew/etc/ssh/sshd_config:
expected replacement of ".../Cellar/openssh/<version>" with ".../opt/openssh"

Root cause: Homebrew preserves files under etc across upgrades. After the first install, sshd_config already contains opt/openssh; a later install has no Cellar path left to replace, but the formula treats that valid no-op as an error.

Fix: install/patch-homebrew-openssh.sh guards the replacement with a content check. It leaves an already-normalized configuration untouched, while the formula’s test still rejects any Cellar path that remains.


Brew has the current keg but still uses an older version

Section titled “Brew has the current keg but still uses an older version”

Symptom: brew list --versions glib or another formula lists the current version, and brew outdated prints nothing, but opt/<formula>, bin/<tool>, or pkg-config still resolves an older keg. This can follow an interrupted or failed upgrade.

Inspect both the selected keg and the current keg’s receipt:

Terminal window
formula=glib
prefix=$(brew --prefix)
brew info --json=v2 "$formula" | jq '.formulae[0] | {linked_keg, installed}'
readlink -f "$prefix/opt/$formula"
ls "$prefix/Cellar/$formula"/*/INSTALL_RECEIPT.json

If the current keg has an install receipt, its direct executable works, and brew linkage --test "$formula" passes, preview and repair only the links:

Terminal window
brew link --overwrite --dry-run "$formula"
brew link --overwrite "$formula"

If its receipt is absent or brew info --json=v2 reports a null install time, the keg is incomplete. Do not force-link it; rebuild it:

Terminal window
HOMEBREW_NO_INSTALL_CLEANUP=1 brew reinstall --build-from-source "$formula"

This run found both forms: Fish 4.8.1 was complete but its bin/fish symlink still named 4.5.0, while GLib 2.88.3 lacked a receipt and had to be rebuilt.

OpenSSL 4 retries collide with an existing manpage

Section titled “OpenSSL 4 retries collide with an existing manpage”

On Linux, an interrupted OpenSSL 3-to-4 transition can leave a complete but unlinked openssl@4 keg while openssl@3 still owns the global links. A retry can install over leftover output and fail with Errno::EEXIST for share/man/man1/asn1parse.1ssl.gz: the formula tries to create a compressed manpage symlink that already exists. A long-prefix bottle warning explains the source build; it does not explain this file collision.

After all installs using the shared prefix have stopped, apply the receipt, direct-executable, and linkage checks above to openssl@4. If they pass, repair the transition without rebuilding the existing keg:

Terminal window
brew unlink openssl@3
brew link openssl@4
brew postinstall openssl@4
HOMEBREW_NO_INSTALL_CLEANUP=1 brew upgrade wget
brew linkage --test openssl@4 wget
wget --version
wget --spider --timeout=20 --tries=1 https://docs.brew.sh/

Unlinking preserves OpenSSL 3’s installed keg and opt/openssl@3 path, but source-built tools with an undeclared OpenSSL dependency may still need repair. Check the checksum execution described under checksum loader failures before proceeding to other installers. OpenSSL 4 should become globally linked, its post-install step should create the CA-certificate link, and wget should upgrade without rebuilding OpenSSL. If the keg fails the checks, use the source-reinstall path above instead; do not delete individual manpages or lock files.

On blackwell (Linux x86-64, shared NFS prefix, 2026-09-28), this repaired OpenSSL 4.0.2_1 and installed wget 1.25.0_2; Bundle check and the HTTPS request passed. The wget linkage audit still exited nonzero for indirect libunistring linkage: the Linux formula declares it transitively through dependencies, while this source build links it directly. It reported no missing libraries; this formula-declaration issue is separate from the OpenSSL installation failure and remains unresolved.


Brew Bundle reports a circular libtiff, webp dependency

Section titled “Brew Bundle reports a circular libtiff, webp dependency”

Symptom: brew bundle check refuses to sort its graph even though both current formulae are installed:

Formulae dependency graph sorting found a circular dependency:
libtiff, webp

Root cause: the installed WebP receipt can retain an old libtiff dependency, while the current libtiff formula depends on WebP. Generated keg receipts are installation records; do not hand-edit them. Reinstalling WebP regenerates its receipt from the current one-way dependency graph:

Terminal window
HOMEBREW_NO_INSTALL_CLEANUP=1 brew reinstall --build-from-source webp
brew bundle check --file="$HOME/dotfiles/packages/Brewfile"

Rust fails after Homebrew says its packages are satisfied

Section titled “Rust fails after Homebrew says its packages are satisfied”

Symptom: install/rust.sh reports Homebrew rustup not found even though brew list rustup and brew --prefix rustup succeed.

Root cause: Homebrew’s keg-only rustup formula removed rustup-init. Older bootstrap runs left ~/.local/cargo/bin/rustup pointing at the removed /opt/homebrew/bin/rustup-init, and rust.sh incorrectly required that removed binary before accepting the installed formula.

Confirm:

Terminal window
brew info rustup | grep -E 'keg-only|no longer provides'
readlink ~/.local/cargo/bin/rustup
ls "$(brew --prefix rustup)/bin/rustup"

Fix: update the checkout and run bash ~/dotfiles/install/rust.sh. The installer now links Homebrew’s individual keg wrappers into the managed Cargo bin directory and initializes stable with rustup toolchain install; it does not depend on rustup-init.


macOS syntax checks report an unmatched quote in a formula patch

Section titled “macOS syntax checks report an unmatched quote in a formula patch”

Symptom: ./tests/ci.sh macos fails in patch-homebrew-mesa.sh or patch-homebrew-systemd.sh with unexpected EOF while looking for matching a single quote, although the same scripts pass Linux checks.

Cause: Bash 3.2 misparses the embedded Python heredoc inside command substitution. Quoting the outer assignment does not resolve this parser failure. A newer Bash on Linux does not reproduce it.

Confirm: run /bin/bash -n install/patch-homebrew-mesa.sh and /bin/bash -n install/patch-homebrew-systemd.sh on macOS. These commands parse the files without installing packages or changing formulas.

Fix: update the checkout. Each patch now puts its Python heredoc in a shell function and captures that function’s output. Formula transformations and Linux-only execution remain the same. The macOS CI mode explicitly validates with /bin/bash and runs both formula fixture suites under its constrained system PATH, even when Homebrew Bash is installed.

See the Mesa source, systemd source, and Bash’s heredoc reference.

Symptom: Linux bootstrap stops during Homebrew preparation with mesa pyyaml patch target not found — formula changed.

Cause: Mesa added r.test? to its Python resource filter. The old exact replacement only recognized the earlier filter, so it rejected the new formula.

Fix: update the checkout and rerun the maintained patch:

Terminal window
bash ~/dotfiles/install/patch-homebrew-mesa.sh

Expect a successful patch report, then an already-applied report on a repeat run. The patch preserves the formula’s test/resource exclusions and existing bindgen compiler workaround, selects the formula’s Python expression, and uses the declared PyYAML version. Linux requires that version’s binary wheel; unsupported formula or bindgen shapes fail before changing the file. This prepares the formula without rebuilding Mesa. install/linux-packages.sh calls it before Bundle; DF_PATCH_BREW_MESA=0 explicitly skips it.

See the patch source and the Homebrew formula.


Symptom: Linux bootstrap reports systemd patch target not found after a Homebrew formula update.

Cause: the old patch matched a literal "python3.14" interpreter and pinned lxml to 6.0.2. The formula now passes a python3 variable to virtualenv_create and declares its own newer lxml resource. Matching only the new interpreter spelling would still install the wrong lxml version.

Confirm: inspect the interpreter and resource in the local formula:

Terminal window
brew cat systemd | grep -A 5 -E 'resource "lxml"|virtualenv_create'

Fix: update the checkout and apply the maintained patch:

Terminal window
bash ~/dotfiles/install/patch-homebrew-systemd.sh

Expect a successful patch report, or an already-applied report on the next run. The patch retains the formula’s interpreter expression and obtains lxml’s version from resource("lxml").version. Linux builds require a wheel for that exact version and fail if none is available; macOS retains its original resource install. Unsupported formula shapes produce a visible failure without being rewritten. DF_PATCH_BREW_SYSTEMD=0 explicitly skips the patch.

This repairs the formula preparation step; it does not rebuild systemd. install/linux-packages.sh applies the patch before its package pass. See the patch source and Homebrew’s formula cookbook.


Symptom: rust-docs-mcp or another Cargo source fallback fails in openssl-sys with version: 4_0_3 and a message that only OpenSSL 1.x or 3 is supported. OPENSSL_DIR unset is diagnostic output, not the failure itself.

Cause: Homebrew’s global pkg-config search finds OpenSSL 4, while the crate’s locked openssl-sys version rejects that major. An installed OpenSSL 3 keg is insufficient unless the build selects its versioned path. A later successful doctor check can belong to the previously installed binary; the failed replacement still makes install/rust.sh fail.

Confirm: compare default discovery with the available OpenSSL 3 headers:

Terminal window
pkg-config --modversion openssl
brew --prefix openssl@3
test -f "$(brew --prefix openssl@3)/include/openssl/ssl.h"

Fix: update the checkout and retry the Rust installer. If OpenSSL 3 is absent, install its declared Brewfile dependency first:

Terminal window
brew install openssl@3
bash ~/dotfiles/install/rust.sh

The installer selects opt/openssl@3 for source builds through OPENSSL_DIR, adds its library directory to Linux runtime search paths, preserves explicit OpenSSL path overrides, and leaves global Homebrew links unchanged. Expect the source-build log to name OpenSSL 3 and the final count to report zero failures. If you explicitly selected OpenSSL 4, remove or correct that override before retrying. See Cargo package behavior and the upstream build variables.


Bootstrap upgrade leaves GUI apps outdated

Section titled “Bootstrap upgrade leaves GUI apps outdated”

Symptom: formula upgrades complete, but VS Code, ChatGPT, or other casks remain outdated. The output says Cask upgrades deferred.

Root cause: the cask pass requires cached sudo authentication. Previously, bootstrap only checked sudo -n true and skipped every cask if it failed, even when individual apps did not need privileges. A manual sudo -v before starting was unreliable because lengthy formula upgrades could outlast the timestamp.

Confirm: inspect the bootstrap output for the deferral message and compare Homebrew’s records with the app’s own version:

Terminal window
brew outdated --cask --greedy

Auto-updating apps can have stale Homebrew receipts, so this list alone does not establish the version of the app bundle.

Fix: run the updated ~/dotfiles/bootstrap.sh upgrade from an interactive terminal. Bootstrap authenticates once and maintains the timestamp until exit. Keep DF_SUDO=auto and the desired cask/MAS upgrade gates enabled. Unattended runs without authentication report the skipped upgrades in the degradation summary. See sudo session behavior for configuration and limitations.

If the log first reports sudo credential renewal failed, the credential was lost during the run. Updated installers recheck before cask, App Store, and TeX updates and can visibly reauthenticate in an interactive terminal. The keeper preserves sudo’s error and retries without prompting. Inspect the effective policy with sudo -l if renewal repeatedly fails; do not change sudo timeout or security policy merely to suppress the warning. An unattended run still needs an interactive follow-up to complete skipped updates.

Native Codex installed but the shell runs an older npm version

Section titled “Native Codex installed but the shell runs an older npm version”

Symptom: ~/.local/bin/codex --version is newer than codex --version, or bootstrap reports Codex PATH conflict after installing the complete runtime.

Root cause: nvm precedes the native bin directory on PATH, and a previous global npm installation left its codex launcher in the node version’s bin directory. Restarting the shell recreates the same ordering. Older cleanup code also skipped the managed nvm directory when ~/.local resolved to scratch storage outside $HOME, leaving that launcher in place after installation.

Confirm: compare command -v codex and the versions reported by that path and the managed $LOCAL_PLAT/bin/codex (~/.local/bin/codex without isolation).

Fix: run bash ~/dotfiles/install/codex.sh install, then rehash in zsh or hash -r in Bash. The installer validates the native runtime, retires known npm launcher symlinks in the resolved managed root (including scratch storage and PLAT layouts) and ~/.nvm, and verifies PATH resolution. It preserves package files for active sessions and refuses to replace unknown launchers. See native CLI migration.


Cask upgrade fails: It seems there is already an App at '/Applications/X.app'

Section titled “Cask upgrade fails: It seems there is already an App at '/Applications/X.app'”

Symptom: bootstrap.sh upgrade reports Some greedy cask upgrades failed, and brew upgrade --cask --greedy ends with Error: Problems with multiple casks: naming an app (or a binary, e.g. already a Binary at '/opt/homebrew/bin/dnx'). Homebrew reverts the upgrade, so the same failure repeats every run.

Root cause: Homebrew refuses to overwrite an artifact it doesn’t have a receipt for. Two ways an auto-updating cask gets there:

  • Upstream renames the app bundle. The cask’s app stanza changes name, the self-updater has already written the new bundle, and brew’s receipt still points at the old one — so brew tries to create a file that exists. This is what codex-app did: OpenAI folded the Codex desktop app into ChatGPT and renamed Codex.app → ChatGPT.app (same com.openai.codex bundle ID). The codex-app cask is deprecated with chatgpt as its replacement, and the auto-migration leaves an orphan Caskroom/codex-app/ directory behind, so brew list --cask still shows it while brew info says “Not installed”.
  • A leftover symlink from a previous install. dnx pointing into /usr/local/share/dotnet/ blocked every dotnet-sdk upgrade.

Confirm:

Terminal window
brew outdated --cask --greedy # which casks are stuck
ls /opt/homebrew/Caskroom/<cask> # receipt version vs the running app
/usr/libexec/PlistBuddy -c "Print :CFBundleIdentifier" /Applications/X.app

Two apps printing the same bundle ID means one is a stale copy under the old name, not a second product.

Fix: brew install --cask <name> --force — it overwrites the unmanaged artifact, removes the old-named bundle, and re-establishes the receipt at the current version. Delete any orphan Caskroom/<old-cask>/ directory (it holds only a symlink and metadata; rm -rf on it does not follow the symlink) and remove stray binaries before retrying. For a renamed cask, also update packages/Brewfile to the replacement name.


A Homebrew package is months behind upstream (Linux)

Section titled “A Homebrew package is months behind upstream (Linux)”

Symptom: a formula installs at a version far older than upstream stable, and re-running bootstrap never moves it. Seen with glab, which stuck at 1.89.0 while upstream was 1.109.0 — old enough that glab skills install --global failed with Unknown command "skills", so install/skills-sync.sh reported fail glab / missing declared skill: glab on every run.

Root cause: install/linux-packages.sh sets HOMEBREW_NO_AUTO_UPDATE=1 (so an implicit refresh can’t revert the in-place formula patches) and, until now, never ran an explicit brew update. The homebrew-core clone therefore froze at whatever date it was first cloned, and HOMEBREW_NO_INSTALL_FROM_API=1 forces every lookup through that frozen clone. One machine sat 4.5 months stale with 193 of 336 installed formulae behind upstream.

Confirm:

Terminal window
git -C "$(brew --repo homebrew/core)" log -1 --format=%ci # tap's age
brew info <formula> | head -1 # frozen version

Fix: rerun install/linux-packages.sh — it now discards the formula patches, runs brew update, and re-applies each patch against the fresh formula. This refreshes definitions only; DF_BREW_UPGRADE still governs whether installed kegs move, so a plain run leaves working binaries alone.

Upgrade a single package without touching the tap:

Terminal window
env -u HOMEBREW_NO_INSTALL_FROM_API -u HOMEBREW_NO_AUTO_UPDATE brew upgrade <formula>

Expect patch anchors to rot across a long refresh — see the patch-anchor gotcha in .claude/rules/homebrew.md.


Upgrade finishes installing, then the version audit calls newer tools outdated

Section titled “Upgrade finishes installing, then the version audit calls newer tools outdated”

Symptom: bootstrap.sh upgrade installs current Homebrew packages, passes runtime verification, then fails its strict version audit because an installed patch version is newer than the reported exact baseline.

Root cause: an unpinned package-manager tool was audited as an exact pin. A Homebrew upgrade could therefore make the machine newer than the repo baseline, and the audit categorized every exact mismatch as outdated.

Confirm:

Terminal window
bash install/audit-versions.sh | jq '.[] | select(.status != "current")'
brew info --json=v2 rustup juliaup | jq '.formulae[] | {name, stable: .versions.stable, installed: .installed[0].version}'

Fix: update the repo, then rerun bootstrap.sh upgrade. Unpinned rustup, juliaup, and unversioned Homebrew zig releases use minimum-version floors; genuinely pinned toolchains remain exact.

Quarto follows the same ownership rule: the macOS Homebrew cask uses a minimum floor of 1.10.18, so an installed 1.10.19 passes. Linux uses an exact release archive pin, including an explicit DF_QUARTO_VERSION override. An older audit applied the Linux pin on macOS and incorrectly rejected newer cask releases. After updating the audit, rerun bash install/audit-versions.sh --strict; a successful report requires every row to have status: current and exits zero.


Skill upgrade reports both “up to date” and committed-digest drift

Section titled “Skill upgrade reports both “up to date” and committed-digest drift”

Symptom: the skills updater says global skills are current, then install/skills-sync.sh reports content differs from committed digest.

Root cause: the mutable installer receipt and committed review digest serve different purposes. A legitimate upstream update advances the receipt, while the committed digest stays unchanged until the new content is reviewed. Local trees that differ from their installer receipt are preserved instead.

Confirm with the read-only audit:

Terminal window
bash install/skills-sync.sh check

After reviewing the changed skill trees, refresh the review baseline with bash install/skills-sync.sh lock. Do not use npx skills check for this; it updates installed content. Self-installed skills must declare their own force flag. The glab row uses --force, because glab skills install otherwise refuses an existing file but exits successfully.


Section titled “Cursor skills remain a directory instead of the shared symlink”

Symptom: readlink ~/.cursor/skills prints nothing, Cursor has a separate skill tree, or chezmoi apply cannot replace ~/.cursor/skills with its managed link.

Root cause: an older Cursor installation created ~/.cursor/skills as a real directory before this repository made it a symlink to ~/.claude/skills. The pre-chezmoi ownership migration preserves that directory as an adjacent backup, copies only top-level entries that do not already exist in the shared tree, and does not overwrite a shared collision.

Confirm the current shape without changing it:

Terminal window
test -d ~/.cursor/skills && ! test -L ~/.cursor/skills && echo 'real directory'
readlink ~/.cursor/skills

Fix: run the migration, then rerun bootstrap so it records its chezmoi source and creates the managed link:

Terminal window
bash ~/dotfiles/install/skills-sync.sh adopt
~/dotfiles/bootstrap.sh update
readlink ~/.cursor/skills

The last command should print ../.claude/skills. The original complete tree remains at ~/.cursor/skills.pre-chezmoi-*; inspect it before deleting anything. For a name collision, the shared ~/.claude/skills/<name> stays authoritative and the Cursor version remains only in that backup.


GLIBC_x.y not found from a Homebrew binary (Linux)

Section titled “GLIBC_x.y not found from a Homebrew binary (Linux)”

Symptom: a brew-installed binary refuses to start, blaming Homebrew’s own libc:

.../opt/binutils/bin/as: .../opt/glibc/lib/libc.so.6: version `GLIBC_2.38' not found

Only some binaries are affected — the recently installed ones — and the error names whichever binary you happened to run, so it reads like a problem with that package. ldd disagrees and shows the system libc, because it resolves through the system loader while the binary itself runs under brew/lib/ld.so.

Root cause: the glibc keg is older than the bottles. Homebrew’s Linux bottles carry the glibc floor of the CI image that built them, and when homebrew-core moves that image it bumps the glibc formula in the same breath (Ubuntu 22.04 → 24.04, glibc 2.35 → 2.39, July 2026). Nothing upgrades an installed glibc keg on its own — it isn’t in the Brewfile — so every formula poured after the move lands with a floor the keg can’t meet. Aug 2026: seven kegs (binutils, gcc@15, texlab, tinymist, cadical, harper, juliaup) broke at once, all poured by one brew bundle run days after the builder moved.

Confirm:

Terminal window
brew outdated --formula --verbose glibc # glibc (2.35_2) < 2.39_1
ldd --version | head -1 # host glibc
jq -r '.built_on.os_version' "$(brew --cellar)"/<formula>/*/INSTALL_RECEIPT.json

Fix: rerun install/linux-packages.sh. It reconciles the keg against the formula before the bundle, and checks every keg installed since the last run against what the keg provides. The broken kegs need no reinstall — they were fine all along; only the loader under them was too old.

Upgrading by hand needs one guard:

Terminal window
HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1 brew upgrade glibc

A bare brew upgrade glibc moves the keg in ~3 minutes and then spends hours source-rebuilding every dependent whose linkage it considers stale, holding a formula lock the whole time so every other brew command fails with “has already locked”. The rebuild buys nothing: glibc is backward compatible, and the kegs built against the old one keep running.

Homebrew refuses to build a glibc newer than the host’s, so a host older than the formula can’t be fixed this way. That combination has no in-place remedy: either the host glibc moves, or the keg goes and every formula is reinstalled against the host loader. The script warns rather than pretending.

Related: brew reinstall glibc fails with Errno::ENOENT ... gcc-13. Reinstall unlinks the keg before building, which leaves brew/lib/ld.so dangling, and no brew binary — including the compiler — can start. Upgrades build the new keg first and are safe; a keg can only be rebuilt at the same version by rebuilding the prefix.


NODE_MODULE_VERSION mismatch / ERR_DLOPEN_FAILED from an npm tool

Section titled “NODE_MODULE_VERSION mismatch / ERR_DLOPEN_FAILED from an npm tool”

Symptom: an npm-installed CLI dies loading a native addon, usually mid-script:

Error: The module '.../node_modules/better-sqlite3/build/Release/better_sqlite3.node'
was compiled against a different Node.js version using NODE_MODULE_VERSION 141.
This version of Node.js requires NODE_MODULE_VERSION 147.

Root cause: the package was installed under one Node major and is being run by another. Native addons are ABI-locked per major, and the bin shebang is #!/usr/bin/env node — so whichever Node is first on PATH wins. Two ways in:

  • Two Node layers. Globals installed under Homebrew’s node keg and under nvm. List them: npm ls -g --depth=0 --prefix "$(brew --prefix)" should show only npm. Remove strays with npm uninstall -g --prefix "$(brew --prefix)" <pkg> — nvm is the layer that owns npm globals.
  • PATH order. brew shellenv puts brew’s bin ahead of nvm’s; install/_lib.sh restores nvm-first for install scripts, but an ad-hoc shell can still invert it.

It hides well: only subcommands that actually load the addon fail. qmd collection show works while qmd update aborts, which reads like a broken index rather than a broken interpreter.

Fix: run it under the Node that installed it (command -v node should be under $NVM_DIR), or reinstall the package under the current Node (npm install -g <pkg>).


nvm rejects prefix or globalconfig in ~/.npmrc

Section titled “nvm rejects prefix or globalconfig in ~/.npmrc”

Symptom: install/node.sh stops during nvm use or after installing a Node version:

Your user’s .npmrc file (${HOME}/.npmrc)
has a `globalconfig` and/or a `prefix` setting, which are incompatible with nvm.

Root cause: a legacy prefix=~/.npm sends every global package to one shared tree. nvm instead gives each Node version its own global prefix under $NVM_DIR; mixing the two makes package binaries and native addons run under the wrong Node ABI. globalconfig can redirect npm to another file containing the same conflict.

Fix: rerun install/node.sh. Before loading nvm it atomically removes an active prefix from the user .npmrc, preserves the remaining npm policy, and keeps the file private. If .npmrc sets globalconfig, the installer leaves it unchanged and stops: copy any needed registry/auth/policy from the referenced file into ~/.npmrc, remove the globalconfig line, then rerun. nvm owns the Node/npm prefix; packages/npm.txt owns the global CLI set. Do not export NPM_CONFIG_PREFIX or add another npm prefix for this setup.

Verify the selected runtime and global tree agree:

Terminal window
source "$NVM_DIR/nvm.sh"
nvm use default --silent
command -v node
npm prefix -g
dirname "$(dirname "$(nvm which default)")"

The last two paths must match.


uv update reports a checksum mismatch because sha256sum cannot start

Section titled “uv update reports a checksum mismatch because sha256sum cannot start”

Symptom: both uv self-update and its standalone installer print sha256sum: error while loading shared libraries: libcrypto.so.3, followed by ERROR: checksum mismatch with an empty got: value. The checksum program did not run; this output does not establish a corrupt download.

Cause: a source-built Coreutils can auto-detect globally linked OpenSSL without the Homebrew formula declaring that dependency. Its checksum binaries then search the global Brew library directory. Transitioning global links from OpenSSL 3 to 4 removes libcrypto.so.3 from that directory even though the OpenSSL 3 keg remains installed. Keeping that keg alone does not repair the binary’s search path.

Confirm the selected binary’s execution and ELF dependencies. If inspecting runtime resolution, use its own ELF interpreter with --list; system ldd can mislead by resolving libraries against the system glibc instead:

Terminal window
command -v sha256sum
printf abc | sha256sum
readelf -d "$(readlink -f "$(command -v sha256sum)")"

Fix: after all other installs using the prefix have stopped, apply the managed formula patch and rebuild Coreutils:

Terminal window
bash ~/dotfiles/install/patch-homebrew-coreutils.sh
HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_FROM_API=1 \
HOMEBREW_NO_INSTALL_CLEANUP=1 brew reinstall --build-from-source coreutils
printf abc | sha256sum
DF_MODE=upgrade bash ~/dotfiles/install/python.sh

The digest must be ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad. The patch adds an explicit openssl@4 dependency and --with-openssl=yes, preserving OpenSSL acceleration while giving the build a versioned library search path. linux-packages.sh checks the known digest before and after Bundle and rebuilds a broken installed Coreutils; python.sh preserves the original self-update failure instead of claiming every failure means an unsupported uv build.

DF_PATCH_BREW_COREUTILS=0 or DF_PATCH_BREW_ALL=0 disables this Linux patch and automatic repair. The check covers checksum execution, not every installed program. Do not skip checksum verification, globally set LD_LIBRARY_PATH, or relink OpenSSL 3 over OpenSSL 4. The authoritative patch is patch-homebrew-coreutils.sh; GNU documents the build option in its Coreutils manual.


A Homebrew binary can’t find libX.so.N after an upgrade

Section titled “A Homebrew binary can’t find libX.so.N after an upgrade”

Symptom: one program stops starting, naming a library version that used to exist:

node: error while loading shared libraries: libllhttp.so.9.3: cannot open shared object file

Root cause: brew upgrade <formula> doesn’t stop at the formula — it then rebuilds every dependent whose linkage the upgrade invalidated. Interrupt it in between (Ctrl-C, a killed terminal, a timeout) and you’re left with the new dependency and the old dependent: the dependent’s RPATH points at opt/<dep>/lib, which now holds only the new soname. Both kegs are usually still in the Cellar, so brew list looks healthy.

Confirm:

Terminal window
brew list --versions <dep> # e.g. llhttp 9.3.1 9.4.3
readelf -d "$(brew --prefix)/opt/<formula>/bin/<prog>" | grep RPATH

Fix: rebuild the dependent — brew upgrade <formula> (or reinstall). Don’t relink the old dependency keg; that just moves the breakage to whatever wanted the new one.

Prevention: pass HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1 when you upgrade something with many dependents, so nothing half-finishes in the first place.


nvm.sh is lazy-loaded in interactive shells only. Login profiles put the bin directory selected by nvm’s default alias on PATH. A standalone non-interactive script should activate that same alias explicitly:

Terminal window
source "$NVM_DIR/nvm.sh"
nvm use default --silent

The cached values live in ~/.config/chezmoi/chezmoi.toml. To reset:

Terminal window
chezmoi init --data=false

To pre-seed without prompting:

Terminal window
DF_NAME="Your Name" DF_EMAIL="[email protected]" chezmoi init

Another program modified a managed file. Common culprits:

  • uv auto-adds source lines to .zshrc/.bashrc for its bin/env files
  • Claude Code updates ~/.claude/settings.json when plugins are installed
  • Other tools may modify shell configs without asking

Options:

Terminal window
chezmoi diff # see what changed
chezmoi apply --force # overwrite with repo version (safe for shell configs)
chezmoi add ~/.claude/settings.json # pull the live version into the repo (for config files)

For shell configs (.zshrc, .zprofile, .bash_profile), always use chezmoi apply --force to restore the clean template. These files should never be manually edited.


PATH order is wrong — wrong binary is resolving

Section titled “PATH order is wrong — wrong binary is resolving”

Expected priority (highest to lowest). $_LOCAL_PLAT collapses to $HOME/.local in flat-mode (default).

$_LOCAL_PLAT/cargo/bin Rust tools (fd, sd, zoxide, bat, rg, etc.)
$_LOCAL_PLAT/nvm/.../bin Node.js (highest installed version)
$_LOCAL_PLAT/bin chezmoi, uv, claude, codex, uv-tool entrypoints
~/.local/bin arch-neutral scripts (collapses to $_LOCAL_PLAT/bin in flat mode — deduped via typeset -U)
/opt/homebrew/bin Homebrew (macOS) — also where rustup lives
/opt/homebrew/sbin Homebrew sbin
/usr/bin system

Diagnose with:

Terminal window
which <tool> # where it's resolving from
type -a <tool> # all locations on PATH
echo $PATH | tr ':' '\n' # full PATH in order

If a Homebrew tool is shadowing a cargo tool, check packages/cargo.txt and packages/Brewfile for duplicates — remove the one you don’t want.

The other classic shadowing footgun: legacy binaries at ~/.local/bin/<tool> from before a layout migration. The [[ -x "$ARCH_BIN/<tool>" ]] install checks in current scripts catch most of these, but if <tool> --version shows an unexpectedly old version, check ls ~/.local/bin/<tool>* for backups (*.preplat-bak.* or stale binaries) and delete them.


nsys / ncu resolve to the CUDA toolkit copy, not the standalone install

Section titled “nsys / ncu resolve to the CUDA toolkit copy, not the standalone install”

Symptom. You added a newer Nsight Systems / Nsight Compute to dotfiles-nvidia/packages/{nsys,ncu}-versions.txt, the installer reports ok, nsys_list / ncu_list show it — but nsys --version still prints the older version, and which nsys points at $_LOCAL_PLAT/.cuda/bin/nsys.

Root cause chain. The CUDA toolkit bundles its own nsys and ncu. In the shell profiles the ### CUDA ### block runs before the Nsight blocks, and cuda_use prepends $CUDA_HOME/bin to PATH. The Nsight auto-activation used to be guarded on ! command -v nsys (“activate only if not already in PATH”) — by that point the toolkit had always put one there, so the guard never fired and the standalone install was permanently shadowed.

A second, independent trap: Nsight Compute ships ncu at the root of its tree, not under bin/ (Nsight Systems does use bin/). So even when ncu_use did run, prepending $NCU_HOME/bin added a nonexistent directory.

Confirm.

Terminal window
which nsys ncu # .cuda/bin/... means it's shadowed
ls "$_LOCAL_PLAT/.ncu" # ncu at top level, no bin/
readlink "$_LOCAL_PLAT/.nsys" # which standalone version is active

Fix. Both are fixed in the profile templates: activation is now unconditional whenever $_LOCAL_PLAT/.nsys / .ncu exists (the standalone prepends after cuda_use, so it wins), and ncu_use falls back to $NCU_HOME when there is no bin/. Run chezmoi apply and start a new login shell. If it still resolves wrong, the version symlink is the likely culprit — cuda.sh/nsys.sh/ncu.sh deliberately never overwrite an existing .nsys / .ncu / .cuda, so a new install does not become active on its own:

Terminal window
nsys_switch tarball_nsys_2026.1.3.425
ncu_switch tarball_ncu_2026.2.1.5

Check the build log via the API:

Terminal window
ACCOUNT="YOUR_CLOUDFLARE_ACCOUNT_ID"
TOKEN="..."
# List recent deployments
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/pages/projects/dotfiles/deployments" \
-H "Authorization: Bearer $TOKEN" | python3 -m json.tool | grep -E '"id"|"status"'
# Get logs for a specific deployment
DEPLOY_ID="..."
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/pages/projects/dotfiles/deployments/$DEPLOY_ID/history/logs" \
-H "Authorization: Bearer $TOKEN" | python3 -c "
import sys, json
for e in json.load(sys.stdin)['result']['data']: print(e['line'])
"

Common causes:

  • Unsupported Node version — the handbook requires Node 22.12 or newer; CI selects Node 24. Check the build environment and site/package.json.
  • Chromium executable missing — Mermaid renders during the build. Install the matching browser with npx --prefix site --no-install playwright install chromium; keep PLAYWRIGHT_BROWSERS_PATH identical for installation and the build.
  • Invalid link or missing feature coverage — run ./tests/ci.sh docs, then fix the named Markdown link, source path, or explicit entry in docs/_data/features.json. Do not disable the checker.
  • Build output not found — confirm destination_dir = "site/dist" in infra/cloudflare/main.tf. The Astro build is verified before upload.

Two machines fighting over dotfiles on a shared home

Section titled “Two machines fighting over dotfiles on a shared home”

This happens when a template renders differently on each machine (e.g. using {{ .chezmoi.arch }}). The rule: templates must be arch-neutral. Arch-specific logic belongs in shell runtime code, not templates.

Check which template is causing the conflict:

Terminal window
chezmoi diff # shows what chezmoi wants to change vs what's on disk

The fix is almost always to replace a template variable with a shell runtime expression. See Managing dotfiles → Shared home safety.


A symlinked ~/.codex or ~/.claude turned back into a real directory

Section titled “A symlinked ~/.codex or ~/.claude turned back into a real directory”

Symptom: you hand-symlinked ~/.codex (or ~/.claude) to scratch, and some time later it is a plain directory again holding only the managed config — while the scratch copy sits frozen at the date the link died. Nothing logged an error.

Root cause: chezmoi manages files inside both directories (config.toml, AGENTS.md, hooks.json, profiles, rules/, themes/, agents/ for Codex; settings.json, skills/, hook scripts for Claude). A source directory means chezmoi’s target state for that path is “directory” — so on the next chezmoi apply it removes whatever is there, symlink included, and recreates a real directory with just the managed files. Everything else is orphaned wherever the link pointed.

Confirm — a stale target next to a fresh $HOME copy is the tell:

Terminal window
chezmoi managed | grep -x '.codex' # non-empty ⇒ chezmoi owns this path
stat -c '%n %y' ~/.codex ~/scratch/.codex # scratch frozen, home current

Fix: don’t symlink the managed directory. For Codex on a shared home, configure a host-local runtime and run bash install/codex.sh sync-runtime. The scratch installer no longer moves Codex children or SQLite files. For Claude, it still supports selected unmanaged children. See Scratch space.

Then reconcile the orphaned copy by hand — the script won’t touch it, because it cannot tell your stale data from a deliberate second install:

Terminal window
du -sh ~/scratch/.codex # what was orphaned

The supported CODEX_HOME boundary now includes managed-configuration sync; see Codex host-local state.

Codex background server has incompatible feature settings

Section titled “Codex background server has incompatible feature settings”

Symptom: interactive Codex requires api_key_model_discovery and offers Run without daemon this time; restarting is disabled because the server is not managed by Codex.

Root cause: the CLI and the shared app server have different effective startup features. A desktop/SSH-launched app server can outlive CLI upgrades, and the CLI cannot automatically restart a server it does not manage. A version difference is useful evidence, but does not by itself establish which feature differs.

Confirm: compare the CLI and running server through the native command:

Terminal window
codex --version
codex app-server daemon version
codex features list

Fix for the current invocation: select Run without daemon this time, or:

Terminal window
codex --no-daemon

This preserves the shared server and its existing clients. For a persistent repair, restart it through the desktop/SSH client that owns it when active and queued work may be interrupted, using the shared features named by the compatibility dialog. Changing a file does not update an already running server’s startup features. Sources: official CLI reference and the installed CLI’s codex --help.

SSH works but the remote Codex desktop cannot initialize its database

Section titled “SSH works but the remote Codex desktop cannot initialize its database”

Symptom: ordinary SSH succeeds, while app-server startup fails with a damaged-database recovery error and Cross-device link (os error 18).

Root cause: individually symlinked SQLite files and their recovery directory can cross filesystem boundaries. Separately, placing WAL databases on NFS allows incompatible multi-host access. Moving them to another NFS export does not solve that problem. A clean SQLite integrity check does not establish safe WAL locking or successful recovery renames.

Confirm on Linux: inspect the actual filesystem and link targets, not the directory names. Avoid printing authentication files.

Terminal window
findmnt -T ~/.codex -o TARGET,FSTYPE
readlink -f ~/.codex/state_5.sqlite
readlink -f ~/.codex/db-backups
tail -n 30 ~/.codex/app-server-control/app-server.log

Fix: stop writers and make a full verified backup of both the legacy root and any existing destination first, using codex.sh backup SOURCE DESTINATION as described in host-local state. Select persistent host-local state with install/host.sh configure, run bash install/codex.sh sync-runtime, and reconnect the desktop’s SSH connection. Verify its daemon received the new CODEX_HOME and that its Unix socket and databases are local. Keep the old shared tree for explicit history recovery. Do not delete a database merely because startup called it damaged, and do not symlink app-server-control: the native Unix listener requires a real directory. See host-local state.

If the error still names ~/.codex after the desktop is working locally, inspect the failing launcher’s environment. An old terminal/tmux shell may retain the previous codex() function, and non-login shells never read login profiles. Current Bash/Zsh wrappers resolve host policy lazily; reload the shell once after deployment (exec zsh -l or exec bash -l). A new desktop process using the local root does not prove an old terminal inherited that same environment.


Duplicate PLAT paths in PATH (both v3 and v4 showing up)

Section titled “Duplicate PLAT paths in PATH (both v3 and v4 showing up)”

Only relevant with DF_USE_PLAT=1. Fixed in current versions — .zprofile/.bash_profile resolve ~/.local symlinks before setting _LOCAL_PLAT so PATH entries use the same physical path.

If you upgraded from before that fix:

Terminal window
chezmoi apply ~/.zprofile ~/.bash_profile
exec zsh -l # or: exec bash -l
echo "$PATH" | tr ':' '\n' | grep plat # all entries should share the same PLAT prefix

In flat mode (DF_USE_PLAT=0, the default), this failure mode doesn’t apply — there’s no $PLAT segment in $_LOCAL_PLAT.


Zsh history lives at ~/.zsh_history (the conventional default; survives any ~/.local cleanup). Bash history at ~/.bash_history. The bash sidecar command log (richer: timestamps, exit codes, cwd) at ~/.bash_log — search via bash_log_search <pattern>.

If you have history under the old location (~/.local/state/{zsh,bash}/), one-time migrate:

Terminal window
[ -f ~/.local/state/zsh/history ] && mv ~/.local/state/zsh/history ~/.zsh_history
[ -f ~/.local/state/bash/history ] && mv ~/.local/state/bash/history ~/.bash_history
[ -f ~/.local/state/bash/log ] && mv ~/.local/state/bash/log ~/.bash_log

If you set up with DF_USE_PLAT=1 and want to switch to flat (or vice-versa), the layout in ~/.local/ is stable as long as one mode is active — but switching strands GBs in the unused tree. Decommission tool:

Terminal window
# After setting DF_USE_PLAT=0 (or removing use_plat=true from chezmoi data):
bash ~/dotfiles/install/plat-decommission.sh

Refuses to run if DF_USE_PLAT=1 is currently set (won’t nuke the active install). See PLAT isolation for the full migration story.


Brew zsh tab completion leaves remnant characters (Linux)

Section titled “Brew zsh tab completion leaves remnant characters (Linux)”

Symptom: after pressing Tab, stale characters remain on the line instead of being erased.

Root cause chain:

  1. Brew zsh’s RUNPATH loads Homebrew’s own glibc (brew/opt/glibc/lib/libc.so.6)
  2. Homebrew’s glibc ships no lib/locale/ data → setlocale() silently falls back to C/ASCII
  3. In the C locale, wcwidth() returns byte counts instead of display columns
  4. Every cursor-position calculation in ZLE/completion is off → artifacts

Confirm by checking the codeset inside brew zsh:

Terminal window
zsh --no-rcs -c 'zmodload zsh/langinfo; echo $langinfo[CODESET]'
# broken: ANSI_X3.4-1968
# working: UTF-8

Fix: linux-packages.sh generates en_US.UTF-8 and C.UTF-8 locale data for brew’s glibc into $LOCAL_PLAT/locale/ using brew’s own localedef. The shell profiles export LOCPATH pointing there so brew zsh picks it up at startup.

Both locales are needed: non-login processes can inherit LC_CTYPE=C.UTF-8 while the profile sets LANG=en_US.UTF-8. Without the category locale, brewed Perl reports Setting locale failed on each tlmgr invocation. Generating the matching data preserves the requested UTF-8 locale without suppressing warnings.

If you installed before this fix:

Terminal window
# Regenerate locale data
bash ~/dotfiles/install/linux-packages.sh
# Apply updated shell profiles (adds LOCPATH export)
chezmoi apply ~/.zprofile ~/.bash_profile
# Open a new login shell and verify
exec zsh -l
zsh --no-rcs -c 'zmodload zsh/langinfo; echo $langinfo[CODESET]' # UTF-8

Test suite: bash ~/dotfiles/tests/test-locale.sh


Copy/paste from a remote SSH session pastes as mojibake (’, Â, é)

Section titled “Copy/paste from a remote SSH session pastes as mojibake (’, Â, é)”

Symptom: text copied out of a remote Linux session pastes with latin-1 garbage where punctuation, accents, or spaces should be: ’ becomes ’, é becomes é, non-breaking spaces surface as  . Two common shapes:

  • plain ssh + tmux — the display itself is garbled, and copies carry it
  • VS Code/Cursor Remote-SSH embedded terminal — display may look fine, but copying agent TUI output (Claude Code renders with padding/NBSP characters) pastes with stray  accent characters; no tmux involved

The terminal emulator (iTerm2, xterm.js) is innocent: the bytes are already mangled before they reach it.

Root cause chain:

  1. On macOS, .zprofile exports LC_ALL=en_US.UTF-8
  2. macOS ships SendEnv LANG LC_* in /etc/ssh/ssh_config, and Linux sshd accepts LC_* by default — the Mac’s LC_ALL lands in the remote environment. This covers Remote-SSH too: the VS Code/Cursor server is started over that same ssh connection, and every embedded terminal inherits its environment
  3. LC_ALL overrides LANG, defeating the deliberate LANG-only locale setup in the Linux shell profiles (see the entry above). Embedded terminals are hit hardest: they spawn non-login shells, so a profile-only guard never even runs there
  4. On hosts whose system glibc has no en_US.UTF-8 compiled (minimal server images — the brew-glibc LOCPATH data doesn’t help system binaries), setlocale() falls back to C/ASCII
  5. Anything in that C locale that re-encodes the byte stream (system tmux is the classic offender) treats each UTF-8 byte as a separate latin-1 character — the display, and therefore anything selected and copied from it, is mojibake

Confirm on the remote, inside the garbling session:

Terminal window
locale; echo "LC_ALL=$LC_ALL"; locale -a 2>/dev/null | grep -iE 'en_US|utf'
printf 'caf\xc3\xa9 \xe2\x80\x94 \xe2\x80\x9cok\xe2\x80\x9d\n' # should render: café — “ok”

Broken looks like: a “cannot change locale” warning or LC_CTYPE="C" in the locale output, and the printf line rendering as café — “okâ€.

Fix: the locale guard (unset LC_ALL before exporting LANG, from the locale-env.sh shared partial) runs in the Linux shell profiles AND the interactive rc files — the rc copy is what protects non-login embedded terminals. Then:

Terminal window
chezmoi apply ~/.zprofile ~/.bash_profile ~/.zshrc ~/.bashrc
tmux kill-server # the tmux server caches the locale it started with
exec zsh -l # or reconnect / open a fresh embedded terminal

Note the fix cleans the encoding; agent TUIs like Claude Code still put invisible layout characters (padding spaces, hard wraps) into the scrollback, so terminal-selection copies of long output stay imperfect. For clean text use /export or copy from the paired web/mobile session instead.

If it’s still garbled, the host has no UTF-8 locale usable by system binaries at all — check locale -a; export LANG=C.UTF-8 (built into every modern glibc) is the fallback.

Note the tempting client-side fix does NOT work: SendEnv -LC_* in ~/.ssh/config is a no-op here, because ssh reads the user config before /etc/ssh/ssh_config and -pattern removals apply at parse time — the system default adds the patterns after your removal runs.


Symptom: python --version reports 2.7, or python -c 'import sympy' fails after bootstrap.

Cause: The interactive Python environment or its entrypoint symlink is missing for the active PLAT. This commonly appears after moving between hosts with different CPU capability levels: another PLAT may have Python packages, but its binaries are intentionally not on this host’s PATH.

Fix: Re-run the Python installer for the active platform:

Terminal window
bash ~/dotfiles/install/python.sh
hash -r
python -c 'import sys, sympy; print(sys.version, sympy.__version__)'

The installer owns $LOCAL_PLAT/python and exposes its interpreter as $LOCAL_PLAT/bin/python; project-specific dependencies still belong in each project’s uv environment.

WhisperX upgrade fails with SQLAlchemy duplicate normalized extra name

Section titled “WhisperX upgrade fails with SQLAlchemy duplicate normalized extra name”

Symptom: uv tool upgrade --all fails while building SQLAlchemy 2.1.0, reporting duplicate normalized extra name mssql-pymssql.

Root-cause chain: WhisperX depends indirectly on SQLAlchemy through pyannote and Optuna. SQLAlchemy’s source metadata declares both mssql-pymssql and the legacy alias mssql_pymssql; uv rejects this collision. On 2026-09-24, the PyPI release initially lacked a Python 3.12 macOS ARM64 wheel, exposing the source metadata failure. Later that session, the matching wheel became available and the same installer upgrade succeeded with SQLAlchemy 2.1.0. This bypasses the source-package defect; it does not repair its metadata.

Confirm: inspect the failing package/version and the platform wheels on the SQLAlchemy release page. The extra-name normalization specification explains why underscores and hyphens collide.

Fix: once a compatible wheel is published, retry the managed upgrade:

Terminal window
DF_MODE=upgrade bash ~/dotfiles/install/python.sh

Expected result: all tool upgrades and the installer’s entrypoint checks pass. Do not edit uv’s cached source or change the default Python solely to bypass this error. Other platforms without a compatible wheel may still encounter it.

Python 3.14 from Homebrew has build issues on some Linux systems:

  1. UUID module detection failure - configure detects libuuid but the build fails
  2. test_datetime hangs during PGO - Profile-guided optimization runs the test suite, but test_datetime hangs on some CPUs (timezone-related)

Fix: Patches are applied automatically by install/patch-homebrew-python.sh during bootstrap. If you need to re-apply manually:

Terminal window
bash ~/dotfiles/install/patch-homebrew-python.sh
brew reinstall --build-from-source [email protected]

The patches:

  • Set py_cv_module__uuid=n/a to disable the uuid module
  • Patch Makefile’s PROFILE_TASK to skip test_datetime during PGO

Environment variables in .zprofile/.bash_profile prevent Homebrew from auto-updating and overwriting these patches:

  • HOMEBREW_NO_AUTO_UPDATE=1 - prevents tap updates
  • HOMEBREW_NO_INSTALL_FROM_API=1 - forces local formula usage

cass source build fails with rustc 1.94.0 is not supported or E0554

Section titled “cass source build fails with rustc 1.94.0 is not supported or E0554”

On a host with glibc < 2.38 (e.g. Ubuntu 22.04) cass has no usable prebuilt, so memory.sh builds it from source — and you see one of:

rustc 1.94.0 is not supported by the following packages: [email protected] requires rustc 1.95 …
# or, on a newer stable:
error[E0554]: `#![feature]` may not be used on the stable release channel

Two root causes stacked:

  1. cass requires nightly. A dependency gates #![feature(try_trait_v2)] and the repo pins channel = "nightly". Stable can’t build it — an old stable fails the MSRV check, a new stable fails E0554.
  2. A stray Homebrew rust shadows rustup. A rust formula (a lingering build dependency — not in the Brewfile, nothing depends on it) puts cargo/rustc in brew/bin at an old version. In bootstrap’s PATH that shadows rustup, so cargo resolved to brew’s 1.94.0 even after rust.sh updated rustup’s stable to 1.97.1.

Confirm:

Terminal window
which -a cargo # a brew/bin/cargo at an old version is the smoking gun
rustup toolchain list # is `nightly` installed?

Fix (already baked into current memory.sh — this is for older checkouts or manual recovery):

Terminal window
brew uninstall rust # remove the orphan shadow (safe: nothing depends on it)
rustup toolchain install nightly --profile minimal
$CARGO_HOME/bin/cargo +nightly install --git \
https://github.com/Dicklesworthstone/coding_agent_session_search \
coding-agent-search --bin cass --locked --root "$LOCAL_PLAT"

_cass_build_from_source now installs nightly on demand and calls $CARGO_HOME/bin/cargo +nightly explicitly, so it no longer depends on PATH resolution or the default toolchain.


cass search misses sessions you know happened

Section titled “cass search misses sessions you know happened”

Symptom: history you remember from Codex or Cursor never surfaces, even on exact phrases, while Claude Code sessions from the same week come back fine.

Root cause: cass ingest is append-only per conversation. Once a conversation is in the canonical DB it is never re-read, so every parser improvement since it was first indexed only reaches new sessions. Old ones keep whatever subset the parser of the day extracted. cass index --full does not fix this — it forces a full scan and a lexical rebuild, then logs skipping historical salvage because canonical database is already populated.

Confirm — compare the DB against a fresh parse of the same files:

Terminal window
sqlite3 -readonly ~/.cass/agent_search.db \
"select a.name, count(m.id) from messages m
join conversations c on c.id=m.conversation_id
join agents a on a.id=c.agent_id group by 1 order by 2 desc;"
wc -l < ~/.codex/sessions/2026/*/*/rollout-*.jsonl # rough upper bound per file

Measured 2026-08-01: codex held 15,037 messages where the same 88 rollouts parse to 85,488 today, and cursor 3,384 vs 6,949 — 82% and 51% of that history unsearchable.

Fix, one connector at a time. Every step matters:

Terminal window
# 0. verify each source file still exists — forget is only safe if it can come back
sqlite3 -readonly ~/.cass/agent_search.db \
"select c.source_path from conversations c join agents a on a.id=c.agent_id
where a.name='codex';" | while read -r p; do [ -f "$p" ] || echo "MISSING: $p"; done
# 1. back up via sqlite, NOT cp — a plain copy can tear a live WAL
sqlite3 ~/.cass/agent_search.db ".backup '$HOME/.cass/agent_search.db.bak'"
# 2. drop the stale rows (dry-run first: omit --apply)
cass forget --source-glob "$HOME/.codex/sessions/**" --apply
# 3. reset the connector watermark — --full still honours it, and rollouts dated
# months ago never beat a watermark stamped today, so the scan finds nothing
sqlite3 ~/.cass/agent_search.db \
"update meta set value='0' where key='last_scan_ts:connector:codex';"
# 4. re-ingest, then clean up after a cass bug: forget leaks tail-state rows keyed
# by the deleted conversation_id, and that column is a plain rowid — SQLite
# reuses freed ids, so a future conversation would inherit a stale
# "ingested through idx N" marker and be silently truncated
cass index --full
sqlite3 ~/.cass/agent_search.db \
"delete from conversation_tail_state
where conversation_id not in (select id from conversations);"
# 5. rebuild vectors in bounded batches; repeat until the backlog is empty
bash ~/dotfiles/install/memory.sh semantic

Skipping step 3 is the usual failure — the run exits 0, having ingested nothing.


cass index fails with graph topology attestation failed

Section titled “cass index fails with graph topology attestation failed”
build HNSW index failed: hnsw error: graph topology attestation failed:
parallel construction failed (search entry origin 6219 reaches only 92908/97513
points at the base layer); serial rebuild also failed

Transcripts contain thousands of byte-identical tool stubs — [Tool: apply_patch] alone repeats 4,383 times — and identical text embeds to identical vectors. Under DistDot those form zero-distance cliques larger than the layer-0 fanout (max_nb_connection 16 → ~32 links), so a clique fills every member’s neighbour list with its own duplicates and nothing outside ever links in. HNSW reachability is directional, so the whole group is unreachable from the entry point and cass’s attestation rejects the graph.

It is deterministic: retrying fails identically, which is why the serial rebuild also failed and why retryable=true in the error is misleading.

Fix: drop --build-hnsw (the manual memory.sh semantic mode does not use it). HNSW only backs --approximate; exact search over ~100k vectors is fast enough, and the flag has never once succeeded on this archive — every semantic_manifest.json here records "hnsw": null. Restore it if cass starts deduping identical vectors before insert.


Every cass command fails: unable to open database file (but sqlite3 opens it fine)

Section titled “Every cass command fails: unable to open database file (but sqlite3 opens it fine)”
opening frankensqlite db readonly at /Users/cade/.cass/agent_search.db:
unable to open database file: '/Users/cade/.cass/agent_search.db'

cass doctor reports archive-db-unreadable, cass index --full refuses to run (“index refused to modify an unhealthy canonical archive”), and the older cass-watch/cass-semantic LaunchAgents may crash-loop with exit 5 — yet sqlite3 -readonly ~/.cass/agent_search.db "pragma quick_check;" says ok.

frankensqlite pins the database’s file identity — device id + inode — in the agent_search.db-fsqlite-ns-use sidecar (record: 8-byte FSQLNS01 magic, 1-byte version, then tag/dev/ino big-endian). On macOS, APFS volume device ids are assigned at mount time and can change across reboots. After the id shifts, every read-only open compares recorded vs live identity and fails closed with SQLITE_CANTOPEN. A read-write open would rewrite the record and self-heal, but cass health-gates every mutating command behind a read-only open first, so nothing ever reaches the heal path.

Confirm.

Terminal window
stat -f "dev=%d ino=%i" ~/.cass/agent_search.db # live identity
hexyl -n 40 ~/.cass/agent_search.db-fsqlite-ns-use
# bytes 10..18 = recorded dev (BE), bytes 18..26 = recorded ino (BE)

Inode matches, device id doesn’t → this bug. If the inode differs, the db file was actually replaced — stop and investigate before touching anything.

Fix. Stop all cass processes, then patch the recorded dev to the live value (here only the last byte differed, 0x10 → 0x0d at offset 17):

Terminal window
printf '\x0d' | dd of="$HOME/.cass/agent_search.db-fsqlite-ns-use" \
bs=1 seek=17 count=1 conv=notrunc
cass doctor # database failure should be gone
bash ~/dotfiles/install/memory.sh index
bash ~/dotfiles/install/memory.sh semantic # one bounded vector batch

Recurs whenever the Data volume mounts with a different device id. Do not run cass doctor --fix for this: on 0.6.23 it enters unbounded recursion in the reconstruct path (observed: 1.6 h at 100% CPU, ~50 GB RSS, no output) — kill it if started; it only touches lock files before hanging.


macOS still asks: “cass would like to access data from other apps”

Section titled “macOS still asks: “cass would like to access data from other apps””

After an upgrade, an old launchd job may still be scanning the session archive. The current setup never schedules cass indexing: qmd stays warm for Markdown search, while cass runs only when you invoke memory.sh index or semantic.

Older dotfiles deployed a dev.cade.cass-watch LaunchAgent that ran cass index every 300 s, and the aider connector crawls $HOME. Aider histories are project-local (.aider.chat.history.md in each repo), so discovery walks its root — and that root defaults to $HOME. The walk enters ~/Pictures, ~/Music, ~/Documents, ~/Desktop, ~/Downloads, and ~/Library, so it asks for Photos, MediaLibrary, AddressBook, Calendar, AppData — and AllFiles.

It is tempting to blame the connectors that read ~/Library/Application Support (cursor, chatgpt, copilot) — don’t. Measured 2026-08-05: a scan that opens 13 Cursor state.vscdb files raises zero TCC requests, because Cursor is a non-sandboxed Electron app with no registered container, so its app-support dir isn’t protected. Excluding those connectors costs you cross-harness session coverage and fixes nothing.

“Allow” doesn’t make it stop because cass is ad-hoc signed (codesign -dv → Signature=adhoc, no Team ID), so a grant is pinned to the binary’s cdhash and is voided at the next cass upgrade. There is also no System Settings pane for App Data grants, so a stale one can’t be repaired from the UI.

See exactly what cass is asking for — this is the diagnostic that matters, since the cass logs only record paths it opens deliberately, not what a directory walk touches:

Terminal window
/usr/bin/log show --last 30m --predicate 'process == "tccd"' --info \
| rg 'Sub:\{.*/\.local/bin/cass\}' | rg -o 'kTCCService[A-Za-z]+' | sort | uniq -c

(/usr/bin/log explicitly — log is a shell function in this repo’s profiles.)

Fix — remove the old scheduler and bound any manual crawl. First rerun the current memory setup, which unloads the legacy cass jobs. Then keep the aider discovery root bounded for manual indexing:

Terminal window
export CASS_AIDER_DATA_ROOT="$HOME/dev" # aider discovery root, not $HOME

Applied on macOS by the manual install/memory.sh index modes. Aider still indexes normally — just only under the given root, so aider projects elsewhere go unindexed. CASS_AIDER_DATA_ROOT takes a single path, so $HOME is the only “covers everything” value and it is what causes the problem.

Verify with the log show command above: a scan should now produce no cass entries at all. Check a scan really ran, or the empty result proves nothing — rg 'skipping disabled connectors' ~/.local/share/cass/stderr.log | tail -1.

The alternative to all of this is granting ~/.local/bin/cass Full Disk Access (kTCCServiceSystemPolicyAllFiles is one of the things it asks for), which covers every service at once — but it must be re-added after every cass upgrade, and it hands a self-updating ad-hoc-signed binary read access to Mail, Messages, and Safari history.


Section titled “npm install -g fails with EBUSY … unlink '.nfsXXXX' (qmd upgrade)”

bootstrap.sh upgrade (or install/node.sh) dies upgrading a global npm package — almost always @tobilu/qmd:

npm error code EBUSY
npm error EBUSY: resource busy or locked, unlink
'.../@tobilu/qmd/node_modules/sqlite-vec-linux-x64/.nfs000000001f79d0f000015a88'
[fail] node.sh failed

Root cause: NFS “silly-rename”. The qmd MCP daemon (qmd mcp --http --port 8181) keeps native addons (sqlite-vec, node-llama-cpp, better-sqlite3) mmap’d. When npm deletes the old package tree to swap in the new one, NFS can’t remove a file the daemon still has open, so it renames it to .nfsXXXX and keeps it until that fd closes. npm then can’t unlink the .nfs* file and aborts with EBUSY. Only happens on NFS homes (the Linux clusters) — macOS local disks unlink open files fine, so this is gated to Linux.

node.sh now stops the daemon around the qmd upgrade and restarts it (via the qmd_daemon_* helpers in _lib.sh), so a normal upgrade no longer trips on it. To recover a checkout that predates the fix, or if you hit it by hand:

Terminal window
pkill -f "qmd[^ ]* mcp --http" # 1. stop the daemon → NFS reaps .nfs* files
npm install -g @tobilu/qmd@latest # 2. re-run the upgrade (or: bash install/node.sh)
qmd mcp --http --daemon & # 3. restart (a new shell also lazy-starts it)

A failed swap can also leave a broken husk — a qmd/ dir with only an empty node_modules/ plus a dangling bin/qmd symlink — in a different npm prefix than the one which qmd resolves to (nvm’s). Delete the husk; the live copy is the one on PATH.

qmd is running but its health check cannot connect to 127.0.0.1:8181

Section titled “qmd is running but its health check cannot connect to 127.0.0.1:8181”

Recent qmd releases may bind localhost:8181 only on IPv6 (::1). A check hard-coded to IPv4 then fails even though the daemon and the configured MCP URL are healthy. Confirm with curl http://localhost:8181/health; it should return {"status":"ok",...}. The shared health check now uses localhost, matching the MCP server URL in packages/mcp-servers.txt.

import sage.all / cysignals dies with TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object

Section titled “import sage.all / cysignals dies with TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object”

Symptom. Importing cysignals.pysignals (directly, or transitively via passagemath’s sage.all) raises the TypeError above. Plain import cysignals.signals (and cypari2) works fine.

Root cause. This macOS release (Darwin 25.x) pre-installs C-level fault handlers (SIGILL, SIGABRT, SIGFPE, SIGBUS, SIGSEGV) in every process. signal.getsignal() reports a handler Python didn’t install as None, and cysignals’ pysignals init saves + re-installs existing handlers — re-setting None is rejected by CPython. Not sandbox-, uv-, or Python-version-specific: reproduced on uv’s python-build-standalone 3.12/3.13 and Homebrew 3.14.

Confirm.

Terminal window
python3 -c "import signal; print(signal.getsignal(signal.SIGSEGV))" # → None

Fix. Reset the fault handlers from Python before anything imports cysignals.pysignals (~/dev/math-lab/sagefix.py does exactly this):

import signal
for s in (signal.SIGILL, signal.SIGABRT, signal.SIGFPE, signal.SIGBUS, signal.SIGSEGV):
signal.signal(s, signal.SIG_DFL)

Related trap: pinning cysignals older than what passagemath wheels were built against fails later with cysignals.signals does not export expected C function _do_raise_exception — keep the resolver’s cysignals (1.12.x), fix the handlers instead.

TinyTeX refuses package installation until tlmgr is updated

Section titled “TinyTeX refuses package installation until tlmgr is updated”

Symptom: Linux bootstrap stops at the TeX baseline with tlmgr itself needs to be updated, followed by latex.sh failed.

Cause: TeX Live requires critical package-manager updates before it permits package installation. The installer previously attempted tlmgr install before its upgrade-mode self-update, so it could never reach the repair command.

Fix: update the checkout and run the installer again:

Terminal window
DF_MODE=upgrade bash ~/dotfiles/install/latex.sh

The Linux installer now runs tlmgr update --self --all before installing the baseline in upgrade mode. Install/update modes run tlmgr update --self first, leaving broad package upgrades opt-in. It then installs the baseline and checks pdflatex, latexmk, chktex, texcount, and latexdiff in the managed tool directory. Expect TeX ready; a failed manager update, missing command, or failed package operation still stops the installer. This does not bypass TeX Live’s repository/release compatibility checks.

The implementation is install/latex.sh; see the upstream tlmgr manual for self-update and package-update behavior.


pdflatex: command not found on macOS with MacTeX installed

Section titled “pdflatex: command not found on macOS with MacTeX installed”

Symptom. brew list --cask shows mactex and /Library/TeX/texbin/pdflatex exists and is executable, but pdflatex, latexmk, chktex, and texcount all report “command not found”.

Root cause. MacTeX installs into /Library/TeX/texbin, which is on no default PATH. Its installer drops a /etc/paths.d/TeX entry, but that only reaches path_helper-processed shells, and these profiles rebuild PATH themselves. install/latex.sh verified the binary by absolute path, so the step reported [okay] while nothing was actually reachable.

Confirm.

Terminal window
ls /Library/TeX/texbin/pdflatex # exists
command -v pdflatex # nothing

Fix. Handled by both shell profiles:

Terminal window
[ -d /Library/TeX/texbin ] && path=($path /Library/TeX/texbin) # zprofile

Appended, not prepended, so Linux’s TinyTeX binaries (symlinked into $ARCH_BIN by latex.sh) keep priority on a machine with both. Run chezmoi apply ~/.zprofile ~/.bash_profile and start a new shell.

brew bundle installs nothing new and exits 0

Section titled “brew bundle installs nothing new and exits 0”

Symptom. brew bundle install prints only Using <formula> lines and succeeds. Packages just added to the Brewfile never appear, and no error names them.

Root cause. A cask-only package declared as brew "..." instead of cask "...". Homebrew resolves the whole dependency graph before installing anything, so one unsatisfiable entry aborts the entire run — every other new package is collateral, which is what makes this read as a no-op rather than a failure. Hit Aug 2026 with brew "quarto": homebrew-core has no quarto formula at all, only a cask.

Confirm.

Terminal window
brew bundle check --file=packages/Brewfile --verbose
# → Formula quarto needs to be installed or updated.
brew info --formula quarto
# → Error: No available formula ... Found a cask named "quarto" instead.

Fix. Move it into the if OS.mac? block as cask "quarto". Casks are macOS-only, so a cask-only tool has no Homebrew route on Linux — install it another way there rather than leaving a brew line that breaks every bundle run. Check a new entry with brew info --formula <name> before committing.

A GUI app’s config is permanently dirty in chezmoi status

Section titled “A GUI app’s config is permanently dirty in chezmoi status”

Symptom. chezmoi status shows MM on an app’s config file every time you look, even when you changed nothing. Running chezmoi apply “fixes” it, then it comes back after the app runs. Worse, bootstrap (which runs chezmoi apply --force) silently reverts real in-app settings changes along the way.

Root cause. Two writers on one file. The app owns and rewrites its config, and a statically chezmoi-managed copy fights it. LinearMouse is the sharpest case: it stamps "$schema": "https://schema.linearmouse.app/<app version>" into the file, so the file goes dirty on a timer — every app update produces a diff with no setting change behind it. That trains you to ignore the dirty status, which is exactly when a real reverted setting slips past.

Confirm.

Terminal window
chezmoi diff ~/.config/linearmouse/linearmouse.json
# - "$schema" : ".../0.11.3" <- what the app wrote
# + "$schema" : ".../0.11.2" <- what apply would force back

Fix. Don’t let chezmoi manage app-owned configs. Use the apply/sync split (install/linearmouse.sh, install/claude-desktop.sh, install/codex-desktop.sh): the tracked source lives under install/<app>/, apply merges it into the live file live-first so app-owned keys survive, and sync captures in-app changes back. For LinearMouse specifically the tracked source omits $schema entirely, so only genuine setting changes ever diff.

Terminal window
bash install/linearmouse.sh sync # capture in-app changes → repo
bash install/linearmouse.sh # push repo settings → app (default: apply)

Adding a new app to this pattern means deleting its home/ chezmoi source (chezmoi then leaves the live file alone), adding the script, and wiring a DF_DO_* flag in bootstrap.sh.


codex -p <domain> cannot see its expected MCP tools

Section titled “codex -p <domain> cannot see its expected MCP tools”

Symptom. A new Codex session using -p browser, creative, or another domain profile starts, but the expected tools are absent or Codex rejects the profile.

Root cause. Domain overlays are generated files in the active CODEX_HOME. Older configurations selected core MCPs only; current Codex sync enables every managed server unless DF_MCP_PROFILES explicitly narrows that selection. An older checkout, an incomplete bootstrap, or a registry change can leave the configuration or overlays stale. Codex accepts one profile per invocation.

Confirm and fix. Regenerate the configuration, then run the same parser check used by bootstrap:

Terminal window
env -u DF_MCP_PROFILES bash ~/dotfiles/install/codex.sh sync-config
bash ~/dotfiles/install/codex.sh check
codex -p browser debug prompt-input "healthcheck"

Set DF_MCP_PROFILES=core or a named selection only when a narrower baseline is wanted. Disabled custom entries outside the registry remain user-controlled. Start a new Codex task after the check, because an existing task keeps its initial tool inventory and may retain an old MCP subprocess environment.


Persistent Playwright profile loses login cookies after close

Section titled “Persistent Playwright profile loses login cookies after close”

Symptom. A Playwright CLI 0.1.19 profile reopens with local storage intact, but its persistent login cookie has disappeared.

Root cause. The CLI daemon’s context-close callback reenters graceful shutdown. On this Mac the second shutdown force-kills Chrome before cookies are flushed to disk. A persistent profile directory alone does not verify working login persistence.

Fix. df-browser close includes a guarded compatibility adapter for owned sessions. It removes only the known duplicate-shutdown listener before normal close, retaining Playwright’s completion listeners. Unknown implementations leave the profile open with a clear error. The browser workspace guide documents the adapter and the real-browser regression test to run after updates.

Google Cloud MCP stops authenticating during a long session

Section titled “Google Cloud MCP stops authenticating during a long session”

Symptom. An official Google Cloud MCP connects successfully, then starts returning authentication errors after its initial access token expires.

Root cause. Older Codex/OpenCode launchers minted GOOGLE_MCP_TOKEN once at startup, and Claude’s old headersHelper minted it at connection time. Neither path refreshed the token while the same connection remained active.

Fix. The current registry emits df-google-mcp as a local stdio transport for all four harnesses. It uses Google’s ADC refresh interface before HTTP requests. Refresh the installed helpers and affected configuration:

Terminal window
bash ~/dotfiles/install/agent-tools.sh
df-google-mcp --check-runtime
bash ~/dotfiles/install/codex.sh sync-config
bash ~/dotfiles/install/claude.sh sync-mcp
bash ~/dotfiles/install/opencode.sh sync-config
bash ~/dotfiles/install/cursor.sh sync-mcp

Restart the affected harness connection so it starts the relay. Existing ADC continues to work; revoked credentials, disabled services, and missing IAM permissions still need their own fixes. See Google Cloud MCP credentials.

Codex MCP login times out in an SSH shell after browser sign-in

Section titled “Codex MCP login times out in an SSH shell after browser sign-in”

Symptom. The provider reports successful sign-in on the laptop, but the remote Codex CLI waits and eventually reports timed out waiting for OAuth callback. The browser’s localhost callback may show connection refused.

Root cause. Codex’s callback listener is on the workstation. The laptop browser reaches its own loopback interface unless the same port is forwarded over SSH. ForwardAgent yes forwards SSH signing access; it does not deliver an HTTP callback. By default the MCP callback port changes between logins.

Confirm. Read the callback host and port from the login URL’s redirect_uri parameter. While login is waiting, compare ss -ltnp on the workstation with lsof -nP -iTCP:<port> -sTCP:LISTEN on the laptop. Keep callback query strings out of shared logs: they contain authorization codes and state.

Fix. Configure a host-specific DF_CODEX_MCP_CALLBACK_PORT and matching loopback-only LocalForward, then retry a fresh login. If sign-in already completed while the CLI is still waiting, establish the matching tunnel and use the provider’s Finish OAuth button. See MCP login from SSH for commands, concurrent-host behavior, and existing SSH connection handling.


Codex MCP OAuth fails: “Authorization server response missing required issuer”

Section titled “Codex MCP OAuth fails: “Authorization server response missing required issuer””

Symptom. codex mcp login <server> (or first use of an OAuth MCP server in Codex) opens the browser, auth succeeds there, then the CLI dies with failed to handle OAuth callback … Authorization server response missing required issuer: expected <server url>. The same server connects fine from Claude Code.

Root cause. A Codex regression, not a server or config problem. Codex 0.143.0+ looks for an iss field in the token endpoint’s JSON response body — where RFC 6749 doesn’t put one — instead of using the RFC 9207 iss callback parameter it already validated. Spec-compliant authorization servers (Cloudflare’s among them) fail the check. Tracked in openai/codex#31573; introduced via a modelcontextprotocol/rust-sdk change.

Confirm. codex --version ≥ 0.143.0, the issue above still open, and the server works from another harness. For Cloudflare specifically, prove the server itself is healthy with a direct handshake:

Terminal window
curl -s -X POST https://mcp.cloudflare.com/mcp \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"probe","version":"0.0.0"}}}'
# → {"result":{…"serverInfo":{"name":"cloudflare-api"…

Fix. Sidestep OAuth in Codex with a static bearer: the --codex-bearer <ENV_VAR> annotation in packages/mcp-servers.txt makes install/codex.sh emit bearer_token_env_var = "<ENV_VAR>" for that server while every other harness keeps OAuth. Cloudflare rides --codex-bearer CLOUDFLARE_API_TOKEN (from ~/.cloudflare.env, bash install/auth.sh cloudflare); mcp.cloudflare.com accepts API-token bearers directly, verified with the handshake above. Caveat: the token’s scopes bound what the tools can do — OAuth carried your full user grant, so mint a broader token if a tool call 403s. Remove the annotation once the upstream fix ships.


Mac App Store reports No downloads initiated

Section titled “Mac App Store reports No downloads initiated”

Symptom: bootstrap.sh upgrade reaches Upgrading Mac App Store apps (mas), then prints one or more messages such as:

Error: No downloads initiated for ADAM ID 682658836

Root cause: mas found a newer catalog version, but macOS’s private CommerceKit API returned no download object and no underlying error. The message alone does not distinguish App Store sign-in or account ownership problems from an App Store service or cache failure. Homebrew has already finished by this point, so this refusal must not turn the whole Homebrew reconciliation into a failure.

Confirm the pending apps and the local App Store configuration:

Terminal window
mas outdated
mas config

mas 7 also requires root privileges for updates. The bootstrap only runs it when sudo is already cached because its password prompt is hidden inside the logged command. Run sudo -v before bootstrap to enable the automatic attempt.

Fix: open App Store, sign in or accept any pending account terms, and update the named apps there. Then rerun mas outdated; once it is empty, rerun ~/dotfiles/bootstrap.sh upgrade. The bootstrap preserves mas warnings but continues after Homebrew itself has reconciled successfully.


Homebrew LLDB installs but fails its runtime checks

Section titled “Homebrew LLDB installs but fails its runtime checks”

Symptom: Homebrew successfully pours the lldb bottle, then lldb.sh reports Homebrew LLDB failed its runtime checks after installation.

Root cause: the runtime probe asked LLDB to load /bin/true. Linux provides that path, but macOS provides true at /usr/bin/true, so a healthy macOS LLDB failed the probe before lldb-dap was checked.

Fix: update the dotfiles checkout and rerun:

Terminal window
~/dotfiles/bootstrap.sh upgrade

The probe now loads /bin/sh, which exists on both supported platforms.


Python tools repeatedly report missing entrypoints

Section titled “Python tools repeatedly report missing entrypoints”

Symptom: python.sh successfully installs a package and lists its executables, then immediately reports Installed tool has missing entrypoints; several large tools may be reinstalled on every bootstrap run.

Root cause: the validator treated every console script advertised by a Python package as part of the dotfiles contract and gave each script a 10-second cold start. Those scripts are not equivalent to selected tools. For example, mlx-lm advertises mlx_lm.evaluate, but that optional command imports lm-eval, which the base mlx-lm install intentionally does not include. Heavy ML commands could also exceed the cold-start bound and pass on the next run after their caches were warm.

Fix: update the dotfiles checkout and rerun the upgrade. Each pip*.txt declaration now names the command the dotfiles actually require with entry=. The installer validates only those commands, uses a 30-second Python CLI bound, and reports the exact package and command when validation fails.


Python bytecode caching stays disabled after updating dotfiles

Section titled “Python bytecode caching stays disabled after updating dotfiles”

Symptom: uv run --offline python -c 'import sys; print(sys.dont_write_bytecode)' prints True, or a terminal still exports CMAKE_CXX_COMPILER_LAUNCHER=ccache.

Root cause: a long-lived terminal, IDE, or agent inherited the former PYTHONDONTWRITEBYTECODE=1 and compiler-cache policy. Editing the profile cannot change an already-running parent process, and the old _PROFILE_SOURCED guard could prevent a nested shell from refreshing it.

Fix: apply the managed dotfiles, then run . ~/.profile in Bash/Zsh or start a new Fish shell. Cache policy now refreshes before the guard, and Fish imports explicit unset operations. Python should report False, with bytecode under ~/.cache/python/pycache. A process launched directly by an old IDE can still inherit old values until that IDE restarts. An explicit python -B remains a command-specific bypass. See compiler-cache migration.


Login-profile fixture intermittently misses QMD startup

Section titled “Login-profile fixture intermittently misses QMD startup”

Symptom: the push gate fails the QMD call-count assertion in tests/profiles.bats, but rerunning the focused test passes.

Root cause: login profiles detach QMD inside a subshell. A later wait in the sourcing shell cannot join that grandchild, so the fixture can inspect its log before the fake QMD process writes it.

Confirm: run bats --filter 'rendered Linux login profiles' tests/profiles.bats. The fixture uses a fake home and fake QMD; it does not start the real service.

Fix: update to the fixture that polls for both recorded startup calls with a bounded deadline, then checks their exact arguments and the absence of CASS calls. Keep the push gate enabled; changing the real QMD startup behavior is unnecessary.


Home directory is near quota despite install/scratch.sh running

Section titled “Home directory is near quota despite install/scratch.sh running”

Symptom: df -H ~ reports the home filesystem close to full even though .local, .cache, and the other DF_LINKS entries are already symlinked to scratch space, and disk-dependent tools (e.g. an MCP server’s cache doctor check) start failing on free-space thresholds.

Root cause: install/scratch.sh only migrates a fixed, named list of cache-shaped directories (DF_LINKS, plus DF_CLAUDE_LINKS/DF_CODEX_LINKS one level under the chezmoi-managed ~/.claude/~/.codex). Two other classes of growth are not covered automatically:

  1. Ad hoc project/output directories created directly under ~ (build trees, bug-repro dumps, scratch clones) — install/dirs.sh already provisions ~/dev, ~/bones, and ~/misc as scratch-backed symlinks for exactly this, but nothing enforces using them; a directory created as ~/<name> instead of ~/misc/<name> sits on the small quota indefinitely.
  2. New subdirectories an app adds after the DF_CODEX_LINKS/DF_CONFIG_LINKS default list was written — e.g. Codex’s db-backups/ (pre-migration SQLite schema-upgrade snapshots) grew unnoticed under ~/.codex for months because the migration list only knew about backups/, a similarly-named but different directory.

Confirm: du -sh --exclude=.local -- ~/.[!.]* ~/* 2>/dev/null | sort -rh | head -20 to find what’s actually large; for a chezmoi-managed dir like ~/.claude or ~/.codex, run du -sh ~/.codex/* ~/.codex/.[!.]* one level down, since the directory itself can’t be a symlink (see the “never symlink a chezmoi-managed directory” gotcha in .claude/rules/install-scripts.md).

Fix: for case 1, move the stray directory under ~/misc/<name> (or ~/dev, ~/bones) and update anything that referenced the old path; going forward, create new project/output directories there directly rather than in ~. For case 2, add the missing subdir name to the relevant _DEFAULT_*_LINKS list in install/scratch.sh and rerun it — Codex/Claude Code must not be running at the time, since scratch.sh refuses to migrate ~/.codex while a process holds its files open (a live app-server can otherwise keep writing to an unlinked inode across the migration).

Remote Codex repeatedly warns about MCP authentication

Section titled “Remote Codex repeatedly warns about MCP authentication”

Symptom. Startup warnings reappear during ongoing work or after a client reconnect. Remote logs report failed to read OAuth tokens from keyring or refusing file fallback, even though .credentials.json exists.

Root cause. Automatic OAuth storage selection can resolve a desktop keyring that the headless app-server cannot read reliably. Runtime MCP refresh and new agent initialization surface the connection failures again. Separately, GitHub’s GH_TOKEN shell-function export is absent when an app starts the Codex binary directly. Neither condition is fixed by a longer startup timeout.

Confirm and fix. Inspect the active CODEX_HOME, server startup errors, and credential presence without printing secrets. Update dotfiles and run bash ~/dotfiles/install/codex.sh sync-config on each remote host. Linux sync selects file OAuth storage and GitHub uses its connection-time credential helper. Restart affected clients when idle. Authenticate a missing server with codex mcp login <server> using the SSH callback tunnel, then verify a read-only operation. Keep existing credential files private; do not copy rotating OAuth credentials between hosts or delete them as a reset.

An untrusted-project warning naming the legacy ~/.codex is a separate config layer warning after moving CODEX_HOME. Inspect the effective config and actual project root. Do not trust an entire home directory just to silence it. The warning does not mean that the active host-local global config was disabled.

Codex MCP login fails with EOF parsing .credentials.json

Section titled “Codex MCP login fails with EOF parsing .credentials.json”

Symptom. HTTP MCP servers report Unknown authentication and login fails with failed to parse credentials file and EOF while parsing a value at line 1 column 0. The active $CODEX_HOME/.credentials.json is empty. A second login can separately report Address in use when the first still owns the configured OAuth callback port.

Root cause. An empty credential file is invalid JSON, so Codex cannot read the store to update it. Disk exhaustion can leave a credential write incomplete. On Blackwell, inspected September 23, 2026, the ext4 credential file’s modification time was September 16 at 04:58:24.871558 UTC; the Codex logs_2.sqlite database contained 24 No space left on device errors in the 04:58:20–04:58:30 UTC window. This strongly implicates a failed write during disk exhaustion; the logs do not identify the credential writer itself. Freeing space afterward does not repair the file. The Codex 0.154.0 credential writer calls set_len(0) before write_all, so a failed replacement write can leave an empty store. A healthcheck detects this damage; it cannot make upstream writes atomic.

Confirm. Check the active directory, capacity, and callback listener without printing credentials. On Linux:

Terminal window
runtime_dir="${CODEX_HOME:-$HOME/.codex}"
stat -c '%n bytes=%s modified=%y' "$runtime_dir/.credentials.json"
df -h "$runtime_dir"
df -i "$runtime_dir"
ss -ltnp 'sport = :45231' # Blackwell; use the configured port on other hosts
bash ~/dotfiles/install/codex.sh check

Fix. Resolve capacity exhaustion first. End an outstanding login in its original terminal before starting another. Preserve the damaged file privately; if it is confirmed empty, replace it atomically with a mode-0600 file containing {}. Do not overwrite a nonempty store or silently reset it during bootstrap. Reauthenticate each affected server and verify a read-only tool operation. A successful codex mcp list --json with not_logged_in confirms that parsing is repaired, not that authentication is restored. Legacy NFS credentials may contain stale rotating refresh tokens; their presence alone does not establish a safe recovery copy.

Keep active SQLite state host-local; SQLite WAL does not support network filesystems. Relocating the runtime does not fix a failed credential writer; protect runtime storage from build/cache exhaustion and retain private, consistent backups. See Codex runtime storage.

Codex lifecycle hook times out after three seconds

Section titled “Codex lifecycle hook times out after three seconds”

Symptom. Hook failed: hook timed out after 3s appears at a lifecycle event, including after the model finishes a response. The managed command is ~/.local/bin/df-task hook.

Root cause. The deadline includes the shell runner, helper launcher, and checkpoint write. Older launchers started Bash, which can read .bashrc under SSH, then resolved host policy and CPU platform again despite an inherited runtime. The host resolver also started a process for each policy variable. Older checkpoint code ignored DF_STATE_ROOT, placing records on a shared home filesystem despite a configured local state directory.

Fix. Update dotfiles and run bash ~/dotfiles/install/codex.sh sync-runtime. The resolver inspects exported names without per-variable processes. The POSIX hook launcher reuses an executable resolved Python runtime, and checkpoint writes honor the host state root. Existing metadata remains readable during migration. The three-second timeout and 200 ms lock-wait bound remain unchanged. No daemon restart or prompt replay is needed for these helper changes.

Verify. Run the native hook regression test (bats tests/codex-hooks.bats). It completes a real app-server turn using a local model fixture and checks its lifecycle checkpoints. Codex normally provides a non-login environment shell; $SHELL -lc is its fallback. A manual full login-shell timing can overstate the normal hook cost. Inspect the active shell path and native hook completion results before changing a timeout. Interrupt and SessionEnd are capped at three seconds by Codex, and requesting a larger value also changes the effective trust hash. See durable work.

Codex upgrade rejects a valid 0.163.0 package

Section titled “Codex upgrade rejects a valid 0.163.0 package”

Symptom. install/codex.sh upgrade downloads codex-package-aarch64-apple-darwin.tar.gz, then fails with is incomplete or reports the wrong version; keeping active runtime.

Root cause. Codex 0.163.0 removed its bundled patched zsh, but the installer still required codex-resources/zsh/bin/zsh. This rejected an otherwise valid package before activation. The old runtime stays active. This failure is independent of sudo, App Store, and optional MCP authentication warnings.

Confirm. On macOS arm64, the official 0.163.0 archive was verified on October 19, 2026 against its published SHA-256 digest; its manifest, CLI version, and code-mode host passed validation, and only the obsolete shell requirement failed. The release notes document the shell removal. codex --version shows which runtime remains active.

Fix. With the updated installer, retry only the Codex step:

Terminal window
bash ~/dotfiles/install/codex.sh upgrade
codex --version

The installer accepts packages without patched zsh starting at 0.163.0 and still requires it for older releases. Checksum, manifest, version, code-mode host, ripgrep, and Linux bubblewrap checks remain enforced. Existing sessions keep their current executable until restarted. See native CLI installation for exact-version selection and runtime paths.

Codex disappears after an npm upgrade on NFS

Section titled “Codex disappears after an npm upgrade on NFS”

Symptom. codex resume reports command not found; npm install -g @openai/codex@latest fails with EBUSY while unlinking a .nfs… file.

Root cause. An active Codex process holds a file in the npm package tree open. NFS retains that file under a temporary name; npm cannot remove it during package replacement, and the failed replacement can leave a dangling CLI shim. A shell function named codex still appears in which even when the executable it delegates to is gone.

Fix. The managed installer now owns the native executable and stages and verifies its replacement before atomic activation. Run bash ~/dotfiles/install/codex.sh install, then use its upgrade mode for updates. Start a fresh login shell to remove the retired function and inspect type -a codex. Do not remove open .nfs… files or delete the package tree underneath running clients. Restart affected app-server processes when interruption is acceptable; no prompts or paused queues should be replayed automatically.

Codex reconnect collides with an SSH OAuth tunnel

Section titled “Codex reconnect collides with an SSH OAuth tunnel”

Symptom. Remote restart or reconnect logs report a callback port already in use, sometimes alongside mux session refused.

Root cause. A persistent SSH connection owns a LocalForward, while a separate app-managed SSH connection tries to bind that same port. MCP refresh warnings after reconnect are a separate result of initializing enabled servers.

Fix. Keep callback forwarding on a dedicated workstation-oauth alias with its own ControlPath; keep the ordinary app workstation alias free of callback forwards. See MCP login from SSH. Existing SSH masters retain their original forwards until closed. A task marked interrupted also needs its own event timeline: received interrupt requests, transport failures, and MCP authentication errors are distinct events.

Codex daemon bootstrap reports an unmanaged app server

Section titled “Codex daemon bootstrap reports an unmanaged app server”

Symptom. codex app-server daemon bootstrap fails with app server is running but is not managed by codex app-server daemon.

Root cause. A directly launched codex app-server --listen unix:// already owns the control socket for this CODEX_HOME, but has no daemon process record. Desktop SSH connections can launch this way. In Codex 0.162.1, bootstrap refuses to adopt that process; daemon stop and restart refuse it too. Management uses a PID file, so the absence of a systemd unit is expected.

Confirm. Inspect the current CLI, server version, and exact process:

Terminal window
type -a codex
codex app-server daemon version
ps -u "$(id -u)" -o pid,ppid,args | rg '[c]odex.*app-server'

An unmanaged server can report status: running while omitting backend and reporting managedCodexVersion: null. Check the process environment to confirm it uses the same CODEX_HOME. Do not confuse the SSH app-server proxy process with the server itself.

Fix. Prepare a complete native CLI package before stopping anything. A stale npm launcher may shadow the managed native binary; bootstrap also needs the complete package to create its daemon install. For example, to retain the observed 0.162.1 version:

Terminal window
CODEX_VERSION=rust-v0.162.1 bash ~/dotfiles/install/codex.sh install

Wait for active work to finish and disconnect the desktop’s SSH connection. After checking its identity again, send SIGTERM to the exact unmanaged server PID and wait for it to exit. This disconnects clients; avoid broad pkill commands, deleting the socket, or fabricating a daemon PID file. Then run:

Terminal window
codex app-server daemon bootstrap
codex app-server daemon version
codex app-server daemon start

Expect bootstrapped, then running and alreadyRunning, with backend: pid and the expected managed version. Reconnect the desktop afterward. Plain bootstrap leaves remote control disabled; SSH access does not require enabling it. See the 0.162.1 daemon lifecycle contract.

Codex cannot read sessions: No file descriptors available

Section titled “Codex cannot read sessions: No file descriptors available”

Symptom. Session resume reports thread-store internal error and os error 24. MCP initialization, CA certificate reads, or helper launches may also fail.

Root cause. The app-server has exhausted its per-process soft descriptor limit. Each live MCP child can hold pipe and process-handle descriptors in the parent. A 1,024 limit can be too small for many simultaneously loaded sessions; the transcript itself need not be damaged.

Confirm. Inspect /proc/<daemon-pid>/limits and count entries under /proc/<daemon-pid>/fd. Compare live process handles, pipes, sockets, and files before diagnosing corruption or expired credentials.

Fix. Deploy the shared Linux resource-limit startup policy, start a fresh SSH login, and restart the affected Codex daemon when authorized. Verify its actual soft limit is 65,536 (or the hard cap), then read the affected session and exercise tool execution. Leave the hard limit unchanged. Raising capacity does not repair a missing code-mode-host binary; install the complete matching runtime for that error. Investigate continuing unbounded helper growth separately.