Troubleshooting
Google status reports the wrong quota project
Section titled “Google status reports the wrong quota project”Symptom. gcloud’s selected project differs from the project charged or checked for quota by a client library or Cloud MCP call.
Cause. gcloud config get-value project is the CLI resource default, while
ADC contains its own quota_project_id. Earlier auth status output incorrectly
labeled the CLI default as the ADC quota project. Environment overrides can
select another quota project or another credential file entirely.
Confirm. Run bash ~/dotfiles/install/auth.sh google-status. Compare
gcloud-project, adc-quota-project, and the override labels without dumping
the full credential JSON.
Fix. Update the checkout and use the corrected status command. Change ADC
only when the intended project is known, with
gcloud auth application-default set-quota-project YOUR_PROJECT_ID. Do not
change the CLI project merely to make the two labels agree. See
Gemini and Google APIs.
Gemini discovery works but a model request fails
Section titled “Gemini discovery works but a model request fails”Symptom. df-gemini models lists a model, but generation or Live fails.
Cause. Discovery access does not establish generation quota, model availability, billing, or region access. A 429 can mean an exhausted or zero project quota; a 503 indicates temporary service unavailability. Google-account Pro status does not establish paid Developer API access.
Confirm. Check the same project’s AI Studio rate-limit page
and bash ~/dotfiles/install/auth.sh google-status. The probe returns failure
and a safe error class/status without exposing its key or WebSocket URL.
Fix. For a 503, retry with backoff. For quota/billing failures, select a model with available quota or deliberately configure paid billing. Do not create extra API keys expecting more quota: project keys share the same pool. See the Google API guide.
Fish reports a missing old PLAT env.fish
Section titled “Fish reports a missing old PLAT env.fish”Symptom: Fish starts with source: No such file or directory naming an
env.fish under an old ~/.local/plat_*/bin directory.
Root cause: uv’s standalone installer wrote
~/.config/fish/conf.d/uv.env.fish with the installation’s fixed path. Changing
the runtime layout did not update that snippet; Fish runs conf.d files even
for noninteractive commands.
Confirm: inspect ~/.config/fish/conf.d/uv.env.fish, then run
fish -lc 'command -s uv; printf "%s\n" $_LOCAL_PLAT'.
Fix: update the managed Fish files with
chezmoi apply --exclude=scripts ~/.config/fish/conf.d and open Fish again.
The managed environment resolves the runtime root at startup and replaces the
legacy uv snippet with an empty ownership marker. The Python installer now
uses UV_NO_MODIFY_PATH=1 to prevent future standalone installations from
rewriting shell startup. See Fish setup.
Push validation cannot mount its isolated checkout on NFS
Section titled “Push validation cannot mount its isolated checkout on NFS”Symptom: native tests, documentation, and lint pass, then Docker reports
permission denied for .ci-validation/push.*/repo before container tests start.
Root cause: mktemp -d creates a private directory. On an NFS export with
root squashing, the Docker daemon cannot traverse that directory even though
the invoking user can read the checkout. A restrictive caller umask can also
make cloned files unreadable to the container.
Confirm: inspect the permissions of .ci-validation/ and the push.*
ancestor while validation runs. A bind mount of the same test source succeeds
once the gate-owned ancestors allow traversal and the source is readable.
Fix: use the updated tests/pre-push.sh and retry git push. The gate
creates a readable snapshot with traversable ancestors and no write access
for other users; it leaves the original checkout permissions unchanged. The
checkout’s existing parent directories must also be accessible to Docker.
See validation before publishing.
Docker bootstrap build reports invalid Ubuntu package signatures
Section titled “Docker bootstrap build reports invalid Ubuntu package signatures”Symptom: ./tests/run.sh fails during apt-get update with
At least one invalid signature was encountered, before running tests.
Possible root cause: A full Docker data disk prevents complete package
index writes. On the affected Colima VM, /var/lib/docker was at 100% with
zero available space. This observation does not rule out other signature errors.
Confirm: Use colima ssh -- df -h / /var/lib/docker and docker system df
to distinguish VM storage exhaustion from host free space.
Fix: After choosing an explicitly authorized cleanup or disk expansion,
restore free space and rerun ./tests/run.sh. Do not disable signature checking
or treat this pre-test environment failure as a passing suite.
For an isolated test profile, verify the daemon before running the gate:
env -u DOCKER_HOST DOCKER_CONTEXT=colima-PROFILE docker info --format '{{.Name}}'Replace PROFILE with an existing profile. An inherited DOCKER_HOST can keep
Docker pointed at the default socket despite selecting another context.
Bats stalls before listing tests on macOS
Section titled “Bats stalls before listing tests on macOS”Symptom: Validation prints Fast Bats tests, then stops before the test
count. A bats-preprocess process remains idle.
Observed cause: On macOS with Homebrew Bash 5.3.15, a process sample showed
bats-preprocess blocked in heredoc_write → write while preparing a here-string.
This is a shell redirection stall before test execution; it does not indicate
a failing test.
Confirm: Inspect the command and sample the affected PID:
ps -axo pid,ppid,etime,command | rg bats-preprocessmacprof sample PID 2 -o - --keep-idleWorkaround: Use Bash’s temporary-file redirection behavior for this invocation:
BASH_COMPAT=50 bats --count tests/agents.batsBASH_COMPAT=50 ./tests/ci.sh full# Or apply the same setting to the required outgoing-commit gate:BASH_COMPAT=50 git push origin mainThe count returned 22 on the affected checkout. The full gate still must pass. Keep this setting local to the command; compatibility mode changes other Bash semantics too. See the Bash maintainer’s explanation of here-document compatibility.
Agent shell loops report installed commands as missing
Section titled “Agent shell loops report installed commands as missing”Symptom: rg, curl, or stat works initially, then a generated zsh loop
prints repeated command not found errors. Assigning status can also fail.
Root cause: zsh’s path array is tied to PATH; assigning a file name to
path replaces command search directories. status is a read-only parameter.
Fix: Use variables such as file_path and exit_status. Select Bash
explicitly for scripts written for Bash. This is a shell-variable problem,
not a reason to reinstall the missing-looking commands.
Bundled skill validation fails with missing yaml
Section titled “Bundled skill validation fails with missing yaml”Symptom: quick_validate.py exits with ModuleNotFoundError: No module named 'yaml' before validating a skill.
Root cause: The bundled validator imports PyYAML using the plain python3
environment. PyYAML was absent from the managed interactive library manifest.
Confirm: python3 -c 'import yaml; print(yaml.__version__)'.
Fix: PyYAML is now declared in packages/python.txt; install/python.sh
installs it alongside SymPy and the public runtime verifier checks both imports.
For an isolated invocation using a different Python, run
uv run --with pyyaml /path/to/quick_validate.py /path/to/skill.
Agent test commands retry unsupported quiet or filter arguments
Section titled “Agent test commands retry unsupported quiet or filter arguments”Symptom: cargo nextest run -q rejects -q, or cargo test first second
rejects the second positional filter.
Fix: Follow the project’s test recipe. For nextest, use
cargo nextest run --cargo-quiet --status-level fail --final-status-level fail.
For a selected nextest subset, use its -E 'test(first) | test(second)'
expression. Cargo test accepts one positional substring filter; run separate
commands for independent filters, and run cargo test --doc for doctests.
Keep test exit codes visible when filtering logs.
Large tool responses defeat output compression
Section titled “Large tool responses defeat output compression”Symptom: Returning whole downloads or JSON documents fills agent context even with RTK enabled.
Fix: Save complete downloads/logs to files, return selected fields using
jq or the CLI’s --json/--jq options, and inspect bounded excerpts. RTK’s
estimated savings are not a guarantee that every command is compressed. Batch
independent reads and prefer a completion-aware wait (for example
gh run watch RUN_ID --exit-status) over rapid repeated status polling.
ShellCheck rejects the runtime-selected platform environment
Section titled “ShellCheck rejects the runtime-selected platform environment”Symptom: Quality and macOS smoke fail with SC1090 in install/_lib.sh,
while the Docker bootstrap suites pass.
Root cause: The platform environment path contains a runtime-selected
$PLAT, so ShellCheck cannot resolve it. The Docker suite does not run the
separate CI lint gate.
Confirm: Run shellcheck -S warning bootstrap.sh install/*.sh install/plat/*/.plat_env.sh home/dot_local/bin/executable_git-wt from the repo.
Fix: Mark only the dynamic import with source=/dev/null and lint every
actual platform environment explicitly in both CI jobs. Keep the global lint
severity unchanged; a passing bootstrap suite does not replace this check.
The jobs now share tests/ci.sh; enable the push gate so the
same checks validate the outgoing commit before Git publishes it.
Local secret scanning reports files outside the outgoing branch
Section titled “Local secret scanning reports files outside the outgoing branch”Symptom: The local push gate reports archived transcript paths that do not exist in the outgoing branch, while hosted CI sees no corresponding files.
Root cause: Gitleaks defaults to scanning all local refs. An independent validation clone can also contain unrelated checkpoint branches; tool names in those transcripts can resemble credential formats.
Confirm: Inspect the redacted finding’s commit and run
git for-each-ref --contains COMMIT --format='%(refname)'. Do not print raw
matches or assume every finding is a false positive.
Fix: The shared quality command uses gitleaks git --log-opts=HEAD to scan
the outgoing commit and its full history. It does not bypass secret checks or
ignore credentials removed from earlier commits in that history.
Hosted Codex runtime and backup tests fail while local tests pass
Section titled “Hosted Codex runtime and backup tests fail while local tests pass”Symptom: Several Codex runtime tests fail together, and the history/backup
tests report uv exiting 127 on the hosted Quality runner.
Root cause: These tests use uv to run Python validation. It was present on the developer machine and installed by Docker bootstrap, but absent from the separately provisioned hosted Quality environment.
Confirm: Check for Command not found on uv run in the Bats warnings.
The fast test entrypoint now requires uv before it starts any fixtures.
Fix: Provision uv through the existing SHA-pinned validation-tool action. Its manifest pins uv’s version with the action revision. Keep the dependency check and real runtime/backup tests enabled.
Local fixture tests overwrite a managed tool
Section titled “Local fixture tests overwrite a managed tool”Symptom: Remote-bootstrap tests fail to find their fake chezmoi, while the
real chezmoi --version unexpectedly prints chezmoi version test. Fish profile
fixtures can also resolve live runtime directories instead of their fake home.
Root cause: Setting a fixture’s HOME does not override exported
DF_TOOLS_ROOT, DF_STATE_ROOT, CODEX_HOME, or other managed runtime variables.
The fixture can therefore write into the real installation. A host overlay can
also restore these roots if the test only unsets the variables.
Confirm: Inspect the executable reported by command -v chezmoi with
file; the fixture stub is a short shell script containing the test version.
Inspect path variables without printing credential values.
Fix: Stop the affected test run, preserve the confirmed stub outside the
managed executable path, and restore chezmoi with bash install/chezmoi.sh.
Use the updated tests/ci.sh, whose Bats subprocess clears inherited runtime
and build settings and exports empty root overrides. Writing fixtures also set
their temporary roots explicitly. Rerun ./tests/ci.sh fast; the regression
checks require inherited tool and state directories to remain untouched.
Quality CI fails with chezmoi: command not found
Section titled “Quality CI fails with chezmoi: command not found”Symptom: The fast Bats profile-rendering test exits 127, while the Docker bootstrap jobs pass.
Root cause: The Quality job runs directly on the hosted runner. Unlike the bootstrap container, it did not install chezmoi before rendering login profiles.
Confirm: Check the failed step for tests/profiles.bats and the missing
chezmoi command; this is a test-environment dependency failure.
Fix: The Quality job installs a pinned, checksum-verified chezmoi release
binary into its tool directory before running Bats. It also installs zsh because
the test sources both bash and zsh profiles. Keep these dependencies in
.github/workflows/ci.yml; do not skip the rendering test. Use the release binary
because this chezmoi version’s Go module has exclude directives and cannot be
installed with go install github.com/twpayne/chezmoi/v2@version.
Native OpenCode test cannot find an installed binary
Section titled “Native OpenCode test cannot find an installed binary”Symptom: A fresh Linux bootstrap installs OpenCode successfully, but
tests/opencode-native-argv.bats fails with FileNotFoundError: 'opencode'.
Root cause: The test entrypoint resolved its Node path before bootstrap
installed nvm. Bootstrap runs in a child shell, so its updated PATH cannot
reach the parent Bats runner. Tests that source installer helpers again can
pass while tests using the inherited environment fail.
Confirm: Read the native argv test’s captured exception and check whether
the runner’s PATH contains the installed nvm default’s bin directory.
Fix: tests/entrypoint.sh resolves the nvm default after bootstrap and
adds it to the shared test environment. Keep this setup in the entrypoint so
native commands work without per-test environment repairs.
Local agents default to MLX but localhost:8080 is unavailable
Section titled “Local agents default to MLX but localhost:8080 is unavailable”Symptom: Pi or OpenCode starts with the configured local model but cannot connect to port 8080.
Root cause: Local models are the macOS client defaults, while heavyweight login services are intentionally disabled. A working CLI does not mean its inference backend is loaded.
Confirm: Inspect local-agent status and the client model selection. A
managed default uses mlx/qwen3.6-27b; a different configured backend should be
started through its own workflow.
Fix: Run local-agent pi or local-agent opencode. The helper verifies the
configured model, starts the managed MLX server on demand if the port is free,
and leaves login policy unchanged. It uses cached weights only; run
bash ~/dotfiles/install/local-llm.sh pull-models explicitly when weights are
missing. Logs are in $LOCAL_PLAT/share/mlxserve/on-demand/server.log.
For fresh DF_PROFILE=core bootstraps, local inference is now skipped by default
because its packages belong to the full Python manifest. OpenCode’s config still
reconciles independently.
Upgrade retains an old OpenCode or Ollama package owner
Section titled “Upgrade retains an old OpenCode or Ollama package owner”Symptom: An existing machine still has the OpenCode Homebrew formula or an Ollama.app server after its package registry selects npm OpenCode and formula Ollama.
Root cause: Removing a package declaration does not uninstall an existing package or transfer a running service.
Confirm: Use brew list --versions opencode ollama, command -v opencode,
and curl --noproxy '*' -fsS http://127.0.0.1:11434/api/version.
Fix: Rerun bash ~/dotfiles/install/node.sh for OpenCode and
bash ~/dotfiles/install/macos-services.sh for Ollama. The migrations verify
the replacement, preserve rollback receipts, and retain user configuration and
models. Ollama migration stops when models are loaded, connections are active,
or port ownership cannot be verified; finish those requests before retrying.
Quick reference for when things go wrong. Check here before digging into scripts.
Colima intermittently rewrites or breaks ~/.ssh/config
Section titled “Colima intermittently rewrites or breaks ~/.ssh/config”Symptom: tools that parse ~/.ssh/config fail intermittently, or
chezmoi diff alternates between a clean file and a leading Colima Include.
Root cause: with sshConfig: true, every Colima start may prepend its exact
~/.colima/ssh_config include. A globbed include in the managed template does
not satisfy Colima’s exact-path check, so Colima and chezmoi repeatedly rewrite
the same file. Colima writes the file in place, which can also expose a partial
config to concurrent readers.
Confirm:
grep '^sshConfig:' ~/.colima/*/colima.yamlgrep -n '\.colima/.*ssh_config' ~/.ssh/configchezmoi diff ~/.ssh/configFix: update the dotfiles checkout and rerun bootstrap.sh. The macOS
services installer sets sshConfig: false in Colima’s default template and all
existing profiles, while chezmoi removes the Colima include from the managed
SSH config. Docker is unaffected. Use colima ssh to enter the VM, or use the
separately generated config without loading it globally:
ssh -F ~/.colima/ssh_config colimaTool not found after bootstrap
Section titled “Tool not found after bootstrap”echo "$_PLAT" "$_LOCAL_PLAT" # capability + install rootls "$_LOCAL_PLAT/bin/" # chezmoi, uv, claude should be herels "$_LOCAL_PLAT/cargo/bin/" # fd, sd, zoxide, etc.which fd # should point under $_LOCAL_PLAT$_LOCAL_PLAT is $HOME/.local by default (flat layout) or $HOME/.local/$_PLAT when PLAT isolation is enabled. If $_PLAT or $_LOCAL_PLAT is empty, .zprofile wasn’t sourced. Open a new login shell (zsh -l) or source it:
source ~/.zprofileCodex install fails with marketplace unavailable: openai-bundled
Section titled “Codex install fails with marketplace unavailable: openai-bundled”Symptom: install/codex.sh reports that openai-bundled is unavailable even
though codex login status says the user is authenticated.
Root cause: openai-bundled is a local marketplace owned and registered by
Codex Desktop. Authentication does not expose it to the standalone CLI, whose
built-in marketplace is openai-curated. The CLI-managed
packages/codex-plugins.txt must therefore contain only plugins from
marketplaces reported by codex plugin marketplace list.
Confirm:
codex login statuscodex plugin marketplace list --jsonFix: update the dotfiles checkout and rerun bootstrap.sh. Bundled plugins
remain owned by Codex Desktop; do not register the app’s internal plugin path
manually because that can conflict with the app’s marketplace reconciliation.
Codex plugin healthcheck reports a plugin was dropped upstream
Section titled “Codex plugin healthcheck reports a plugin was dropped upstream”Symptom: install/codex.sh dies with
Declared Codex plugin was dropped upstream: <plugin>@openai-curated, even
though the plugin still appears in the curated marketplace.
Root cause: codex-cli 0.153.0 changed plugin list --json: uninstalled plugins
are omitted unless --available is passed, and an unscoped listing can prefer
a same-named remote plugin over an installed local-marketplace plugin. Older
install/codex.sh code read that incomplete global inventory and mistook the
missing row for an upstream removal.
Confirm using the marketplace-scoped complete inventory:
codex plugin list --marketplace openai-curated --available --json \ | jq -r '.installed[].pluginId, .available[].pluginId'Fix: update the dotfiles checkout and rerun bootstrap.sh. The installer
now queries each declared marketplace with --available, so collisions with
remote plugins do not hide installed curated plugins. If the scoped inventory
really does omit a declared selector, prune or replace that selector in
packages/codex-plugins.txt; declarations remain a hard contract.
The WARNING: failed to clean up stale arg0 temp dirs: Directory not empty line
from codex-cli is unrelated NFS noise (.nfs* files in its temp dir) — harmless.
Claude cannot load required remote managed settings
Section titled “Claude cannot load required remote managed settings”Symptom: install/claude.sh stops at the marketplace inventory with
Your organization requires remote managed settings to load, but they could not be loaded.
Root cause: an organization with forceRemoteSettingsRefresh requires a fresh
policy fetch before Claude starts. A cached policy does not satisfy that
requirement. Authentication commands remain available so the user can sign in
again. See server-managed settings.
Confirm with claude plugin marketplace list. For the fetch cause, use a private
debug log:
umask 077debug_dir=$(mktemp -d "${TMPDIR:-/tmp}/claude-policy.XXXXXX")claude --debug-file "$debug_dir/debug.log" plugin marketplace listInspect the Remote settings lines locally. http_401 means the server rejected
the credential; claude auth status reporting loggedIn: true alone does not
prove the server accepts it. Console API keys are static credentials, while OAuth
profiles support automatic token refresh. See authentication.
Fix for an authentication rejection: refresh the sign-in, verify that the inventory loads, then rerun bootstrap:
claude auth loginclaude plugin marketplace list~/dotfiles/bootstrap.sh upgradeclaude auth login opens the
supported sign-in flow. For network or policy-delivery failures instead, restore
access to api.anthropic.com or contact the organization administrator. Preserve
the managed policy and cache; changing them does not repair rejected credentials.
The installer checks this prerequisite once and stops before dependent plugin
operations if it fails.
Claude plugin fails: Plugin "X" not found in any configured marketplace
Section titled “Claude plugin fails: Plugin "X" not found in any configured marketplace”Symptom: install/claude.sh logs [warn] fail <plugin>: … ✘ Failed to install plugin "<plugin>": Plugin "<plugin>" not found in any configured marketplace, but the plugin visibly exists in the marketplace’s GitHub repo.
Root cause: plugin installs resolve against the local marketplace clones
under ~/.claude/plugins/marketplaces/, and with DISABLE_AUTOUPDATER=1 those
never refresh themselves. A plugin added upstream after the clone date is
invisible (the claude-plugins-official clone once sat 4 months stale while
math-olympiad existed upstream). claude.sh used to refresh catalogs only in
upgrade mode — and even that call was broken, passing a nonexistent --all flag
whose error was silenced by >/dev/null || true, so no mode ever refreshed.
It now refreshes (with the correct no-name form) in every mode.
Confirm — compare the clone date against upstream:
git -C ~/.claude/plugins/marketplaces/<marketplace> log -1 --format=%cdjq -r '.plugins[].name' \ ~/.claude/plugins/marketplaces/<marketplace>/.claude-plugin/marketplace.json | grep <plugin>Fix: update the checkout and rerun bootstrap.sh (or install/claude.sh).
Manual one-off:
claude plugin marketplace update <marketplace>claude plugin install <plugin>@<marketplace>The same VS Code / Cursor extensions report fail on every upgrade run
Section titled “The same VS Code / Cursor extensions report fail on every upgrade run”Symptom: bootstrap.sh upgrade logs fail <ext-id> for a fixed set of
extensions, run after run — yet the extensions are installed and working in the
editor. Install mode never reports them.
Root cause: upgrade mode used to reinstall each declared extension with
--install-extension <id> --force, which re-resolves the ID against the
editor’s marketplace. Two categories can never satisfy that:
- Extensions the editor now bundles. VS Code ships
github.copilot-chatbuilt in (0.59.0); the marketplace copy is older (0.48.1) and the CLI refuses the downgrade outright:is a built-in extension … and cannot be downgraded. - IDs the marketplace doesn’t carry. Cursor resolves against Open VSX, so
Microsoft-proprietary IDs fail with
Extension '<id>' not foundeven when the extension is installed — Cursor imported it from VS Code on first run, a path the CLI can’t reproduce.nvidia.nsight-vscode-editionis refused explicitly:not available in Cursor for the Mac Silicon.
Confirm — run the install by hand to reproduce the marketplace error reported by the now-failing bootstrap step:
code --install-extension <id> --force # or: cursor --install-extension …Fix: update the checkout and rerun. vscode.sh / cursor.sh now upgrade
with a single --update-extensions bulk pass instead of per-extension
--force, which only touches what the editor can actually resolve. Drop
bundled extensions from packages/vscode-extensions.txt, and keep IDs Open VSX
can’t serve out of packages/cursor-extensions.txt (that file’s header lists
the known-unavailable set and the Anysphere forks to use instead).
Cursor reports ENOENT for User/settings.json and ignores user settings
Section titled “Cursor reports ENOENT for User/settings.json and ignores user settings”Symptom: Cursor logs or displays an error such as:
ENOENT: no such file or directory, open '.../Cursor/User/settings.json'The native file is still a symlink, but its managed target is missing or empty:
ls -l "$HOME/Library/Application Support/Cursor/User/settings.json"jq -e 'type == "object"' ~/.config/cursor/settings.jsongit diff -- home/dot_config/cursor/settings.jsonRoot cause: Cursor writes the symlinked settings file non-atomically. The Cursor
agent hook could run after the file was truncated but before the replacement
contents arrived, and chezmoi add then copied the empty file into the repo.
The hook’s jq cleanup also accepted empty input as success, preserving the
damage. A later failed rewrite can leave the native symlink dangling.
Fix: update the checkout. The hook now accepts only a complete settings object or keybindings array, then validates the chezmoi source after import and restores its previous contents if the file changed during the copy. To recover an already-empty source after confirming the diff contains no wanted edits:
git restore --source=HEAD -- home/dot_config/cursor/settings.jsonchezmoi apply ~/.config/cursor/settings.json ~/.cursor/hooks/sync-dotfiles-cursor.shIf an open window still shows defaults after the native symlink resolves, run
Developer: Reload Window from Cursor’s command palette.
Cursor computer use reports Ready: no, or screenshots never appear
Section titled “Cursor computer use reports Ready: no, or screenshots never appear”Symptom: agent worker debug shows a worker, but Computer use Ready: no
and screenshot or click tasks fail. The My Machines worker may be loaded.
Confirm:
launchctl print "gui/$(id -u)/dev.cade.cursor-worker"ls -d "$HOME/.cursor/cursor-computer-use/Cursor Computer Use.app"agent worker debugRoot cause: computer use is implemented by Cursor Computer Use
(co.anysphere.cursor-computer-use), a helper installed on the first
--computer-use start. Accessibility and Screen Recording must be granted to
that app, not Terminal or Cursor.app. Sequoia and later may re-prompt Screen
Recording. agent worker debug does not prove those grants. This repo does not
write TCC.db.
Only one worker daemon can own ~/.local/share/cursor-agent. Cursor’s in-app
worker (often named like church) holds that lock without computer use.
df-cursor-worker passes --wait so the LaunchAgent waits for that process
instead of crash-looping. To take over with computer use, stop the in-app
worker, then bash ~/dotfiles/install/cursor.sh sync-worker.
Fix: open System Settings → Privacy & Security, grant Accessibility and
Screen Recording to Cursor Computer Use, then rerun a screenshot task. If
the helper app is missing, start the worker once (bash ~/dotfiles/install/cursor.sh sync-worker)
so it can install, then grant TCC. To stop the worker:
DF_CURSOR_WORKER=0 bash ~/dotfiles/install/cursor.sh.
Cloud /in-cloud sessions use native VM computer use and dashboard MCP, not
this Mac’s helper app or ~/.cursor/mcp.json.
Cursor CLI default model reverts to Auto
Section titled “Cursor CLI default model reverts to Auto”Symptom: bash install/cursor.sh check fails on exploreSubagentModel, or
jq -r .exploreSubagentModel ~/.cursor/cli-config.json is default after a
Cursor or agent start. permissions.allow may shrink to Shell(ls).
Confirm:
jq '{exploreSubagentModel,permissions}' ~/.cursor/cli-config.jsonbash ~/dotfiles/install/cursor.sh checkRoot cause: ~/.cursor/cli-config.json is app-owned at runtime. The installer
merge-pins Explore to Composer 2.5 and unique-appends Shell(*); the running app
rewrites those keys. Parent selectedModel is intentionally left alone.
Fix: bash ~/dotfiles/install/cursor.sh sync-cli. That restores Explore and
Shell(*) without changing the parent chat picker.
Brew bundle fails: No available formula … This command requires the tap
Section titled “Brew bundle fails: No available formula … This command requires the tap”Symptom: brew bundle errors with No available formula with the name "owner/tap/formula". This command requires the tap owner/tap. If you trust this tap, tap it explicitly and then try again: brew tap owner/tap — even though the
Brewfile has the tap "owner/tap" line and the tap is already trusted.
Root cause: two separate Homebrew gates protect third-party taps — trust
(HOMEBREW_REQUIRE_TAP_TRUST) and the tap actually being cloned. Homebrew no
longer auto-taps from a fully-qualified formula name, and brew bundle can hit
formula resolution before executing the Brewfile’s own tap directive — in
particular the upgrade check for a formula already installed under the same name
from homebrew/core (seen with rtk: core keg installed, rtk-ai/tap/rtk in the
Brewfile, tap trusted but never tapped → resolution error every run).
Confirm:
brew tap # tap missing from the listjq . ~/.homebrew/trust.json # …while already trusted hereFix: update the checkout and rerun — ensure_brewfile_taps() (_lib.sh)
now trusts and taps every tap referenced by the Brewfile before the bundle.
Manual one-off: brew tap owner/tap, then rerun install/homebrew.sh.
Brew bundle appears stuck after Using lines
Section titled “Brew bundle appears stuck after Using lines”Symptom: bootstrap prints its package inventory, ending in a line such as
Using surrealdb/tap/surreal, then produces no visible build output.
Root cause: current Homebrew Bundle lists satisfied and actionable entries
before running one batched install. Without --verbose, it captures that
child’s output and can discard it on success. The last inventory line does not
identify the running package or prove the process is hung.
Confirm: on Linux, inspect the original bootstrap’s descendants:
pgrep -af 'bootstrap.sh'pstree -ap BOOTSTRAP_PIDA compiler beneath brew establishes an active build. A different installer
means Bundle has already returned. If the tree names google-mcp.py, check
Python’s cache lock wait.
Fix: the Linux and macOS package installers now pass --verbose to Bundle,
including Linux’s single verified retry. After the original process exits,
rerun ~/dotfiles/bootstrap.sh upgrade with the updated sources. This exposes
build output; it does not shorten a build or bypass dependency checks.
Sources: Homebrew Bundle,
linux-packages.sh, and
homebrew.sh.
Python setup waits in NFS locking
Section titled “Python setup waits in NFS locking”Symptom: Python setup stops at Preparing the refreshable Google MCP transport; uv run --locked --script ... --check-runtime stays idle.
Root cause: the script environment cache lives under a shared NFS tool
root. uv locks its environment before installing dependencies, and an NFS lock
request can wait in the kernel even though no local process holds the lock.
On Blackwell, observed on 2026-10-12, the cache was on NFSv3 and a uv worker’s
wait channel was nlmclnt_wait. The same locked check completed with its cache
on the host’s ext4 state volume.
Confirm on Linux: select the affected uv PID and inspect its actual cache and worker threads:
pgrep -af 'uv.*google-mcp.py'findmnt -T "$UV_CACHE_DIR" -o TARGET,FSTYPEps -L -p UV_PID -o pid,tid,stat,wchan:30,argsIf the calling shell does not have UV_CACHE_DIR, read only that variable from
/proc/UV_PID/environ; do not print the complete environment.
Fix: configure a persistent host-local DF_STATE_ROOT with
bash ~/dotfiles/install/host.sh configure. Current installers and managed
Google/Gemini launchers then use $DF_STATE_ROOT/uv/cache. An explicit
UV_CACHE_DIR still wins. Stop the stuck bootstrap in its terminal before
starting another; with Blackwell’s existing local state configuration, the
one-invocation recovery is:
UV_CACHE_DIR="$DF_STATE_ROOT/uv/cache" ~/dotfiles/bootstrap.sh upgradeThe first run populates a fresh cache. Existing processes retain the old path;
do not delete a lock file or disable locking. This change does not relocate
installed Python/tool environments or fix project environments stored on NFS.
Sources: uv cache safety,
_runtime-paths.sh,
python.sh, and
package cache defaults.
Brew bundle reports Upgrading X has failed! after installing X
Section titled “Brew bundle reports Upgrading X has failed! after installing X”Symptom: bootstrap.sh upgrade pours and links a formula successfully, then
reports Upgrading <formula> has failed!. Nearby errors name a vanished file in
~/.cache/Homebrew/downloads/, such as No such file or directory @ dir_s_rmdir - ...bottle_manifest.json. Several unrelated formulae can fail this
way in one run.
Root cause: Homebrew Bundle defaults to as many as four package workers. Those
workers launch separate brew install or brew upgrade processes that share
one download cache and run install cleanup against it. One worker can remove a
cache entry after another has inspected it, turning successful installs into
nonzero exits. Homebrew tracks the broader parallel-worker race as
Homebrew/brew#23328; the
Homebrew manpage documents the auto job
default and the sequential override.
Confirm after the original bootstrap process has exited:
formula=treebrew list --versions "$formula"brew outdated --formula "$formula"brew linkage --test "$formula"If the new version is listed, brew outdated prints nothing, and linkage
passes, the install succeeded and only its cleanup path failed.
Graphviz has a second version of this symptom. Netpbm fetches its source and
manual from Subversion. Bundle can queue those SVN fetches before it finishes
installing Graphviz’s Subversion dependency, record You must: brew install svn, then install Subversion, retry both checkouts, and successfully build
Netpbm and Graphviz. The early fetch result still makes Bundle print Upgrading graphviz has failed! after the successful install.
Fix: update the checkout and rerun bootstrap.sh upgrade. The shared
installer environment now disables Bundle package jobs; downloads and each
source build can still run concurrently, but package installs and cleanup are
serialized. On Linux it also installs a working Subversion before a Brewfile
containing Graphviz, then runs brew bundle check after any nonzero Bundle exit.
If that check passes, it retries Bundle once and reports recovery only when the
clean retry exits zero. Manual one-off:
brew bundle install --jobs=1 --file="$HOME/dotfiles/packages/Brewfile"Do not start the retry while the first bootstrap is still running.
Brew cleanup fails with Device or resource busy .../.nfs...
Section titled “Brew cleanup fails with Device or resource busy .../.nfs...”Symptom: an upgrade prints a formula’s beer-mug success line, then fails while cleanup removes an unrelated old keg:
Error: Device or resource busy @ apply2files - .../Cellar/expat/<version>/lib/.nfs...Root cause: NFS renames an unlinked-but-open file to .nfs* and keeps it until
the last process closes it. Any Homebrew executable or shared library can be a
holder: this first appeared with the Bash running bootstrap, then with dozens of
long-lived dbus-daemon processes mapping an old libexpat.so. Homebrew runs
formula cleanup after installs and, every 30 days, a full cleanup. That cleanup
exception changes the command’s exit status after the package succeeds, so
Bundle misleadingly reports Upgrading <formula> has failed! for each later
package too.
Confirm which processes still hold the file:
lsof /path/from/the/error/.nfs...Fix: wait for the original bootstrap to exit, update the checkout, and retry.
On an NFS Homebrew prefix, linux-packages.sh sets the documented
HOMEBREW_NO_INSTALL_CLEANUP
switch for the run. Installs and upgrades still happen, but cleanup cannot turn
their success into failure. The switch does not disable an explicit cleanup;
after every process shown by lsof has exited, reclaim the retired keg with:
brew cleanup expatOld kegs consume some disk until that maintenance succeeds. Do not delete the
.nfs* file manually or terminate unrelated holders just to make cleanup pass.
Ruby upgrade writes outside its new keg during make install
Section titled “Ruby upgrade writes outside its new keg during make install”Symptom: upgrading vim, ccache, or another Ruby dependent builds Ruby and
then fails under the global Homebrew Ruby directory:
Dir.mkdir: Permission denied @ dir_s_mkdir - .../brew/lib/rubyThe same contamination can later surface as Errno::ENOENT under
.../brew/lib/ruby/gems/... during RubyGems setup.
Root cause: the formula adds the versioned [email protected] path as a compatibility
fallback. runruby supplies the build directory through LD_LIBRARY_PATH, but
Homebrew’s GCC emits DT_RPATH, which the dynamic loader searches first. During
make install, the new Cellar lib directory is not populated yet, so the build
executable falls through to the previous keg’s libruby. The source RUBYLIB
also lacks RubyGems’ optional defaults/operating_system.rb; its require can
therefore find the previous keg’s file. Those old Homebrew defaults redirect
Gem.default_dir and Gem.ruby outside the new keg. The Linux filesystem
sandbox correctly rejects that write; the prefix permissions are not broken.
Fix: install/patch-homebrew-ruby.sh patches the local formula before Brew
Bundle runs. It enables new ELF dtags so DT_RUNPATH yields to the build-tree
library path, retains the new keg before the versioned fallback after install,
and adds a build-local empty RubyGems packager-default file so the previous
keg’s override cannot leak in. The formula replaces that empty file with the
current Homebrew configuration after installation.
This formula-local DT_RUNPATH is a deliberate exception to the prefix’s usual
DT_RPATH policy: the build runner must let its temporary LD_LIBRARY_PATH
select the new build-tree libruby.
Do not chmod the prefix or disable Homebrew’s Linux sandbox. Wait for any
active Homebrew process to exit, then rerun ~/dotfiles/bootstrap.sh upgrade so
the formula refresh and patch happen in the intended order.
apache-serf cannot find asm/socket.h
Section titled “apache-serf cannot find asm/socket.h”Symptom: a source build of apache-serf invokes a brewed GCC directly and
fails through Homebrew’s glibc headers:
glibc/include/bits/socket.h: fatal error: asm/socket.h: No such file or directoryRoot cause: Homebrew’s standard build environment already puts the installed
[email protected] include directory in CPATH. Serf’s SConstruct creates a
new SCons child environment that does not inherit that variable, so the direct
GCC command loses the kernel-header path. Adding the path to superenv or CPATH
again does not cross this second environment boundary.
Fix: install/patch-homebrew-apache-serf.sh adds a direct Linux dependency
on [email protected] and passes its stable opt_include path through Serf’s
supported CPPFLAGS SCons variable. The patch fails closed if the formula
structure changes instead of silently starting another known-broken source
build.
A trailing Clang warning about which GCC installation it may prefer is separate
from this GCC compile failure. Wait for any active Homebrew process to exit,
then rerun ~/dotfiles/bootstrap.sh upgrade so the refreshed formula is patched
before Bundle starts.
Gecode patch reports that its configure target moved
Section titled “Gecode patch reports that its configure target moved”Symptom: bootstrap records this degradation even though MiniZinc may still finish installing:
gecode configure patch target not found — formula may have changedRoot cause: Homebrew changed the Gecode formula from Autotools flags such as
--enable-qt to CMake settings such as GECODE_ENABLE_GIST. The dependency
guard could still apply while the old configure anchor no longer existed,
leaving a partial formula edit and a misleading successful patch status.
Fix: install/patch-homebrew-gecode.sh recognizes both formula shapes,
forbids the upstream bottle on Linux, and sets Gist and Qt off while retaining
the macOS bottle and GUI. The bottle gate matters because build flags cannot
change a bottle that already contains libgecodegist and Qt dependencies. The
installer rebuilds an existing Gist-bearing keg from source and requires both
brew linkage --test gecode and the absence of libgecodegist.so; a moved
anchor stops source builds instead of leaving a partial formula edit.
Clang cannot load libz3 during an in-progress upgrade
Section titled “Clang cannot load libz3 during an in-progress upgrade”Symptom: a formula failure ends with a separate loader error such as:
clang: error while loading shared libraries: libz3.so.4.15: cannot open shared object fileRoot cause: Bundle upgraded Z3 before unversioned LLVM. The installed LLVM still
needs Z3’s previous major SONAME, while opt/z3 already selects the new keg.
This is independent of a formula that was compiled with GCC and usually repairs
itself when the same Bundle run reaches LLVM.
The Linux installer reconciles this pair before Bundle: it upgrades an outdated
LLVM in upgrade mode, reinstalls a current keg whose linkage is broken, and
checks both brew linkage and the unversioned Clang executable again after
Bundle.
Do not point the old SONAME at the new major library or repoint opt/z3 during
the active transaction. After all Homebrew processes exit, rerun
~/dotfiles/bootstrap.sh upgrade. For a manual check, run:
"$(brew --prefix)/bin/clang" --versionIf it still reports the old Z3 SONAME, upgrade an outdated LLVM or reinstall a current but broken keg, then verify its linkage:
if [[ -n "$(brew outdated --formula llvm)" ]]; then brew upgrade llvmelse brew reinstall llvmfibrew linkage --test llvm"$(brew --prefix)/bin/clang" --versionOpenSSH upgrade fails with inreplace failed ... sshd_config
Section titled “OpenSSH upgrade fails with inreplace failed ... sshd_config”Symptom: OpenSSH finishes make install, then Homebrew aborts while replacing
its Cellar prefix in the persistent configuration:
Error: inreplace failed.../brew/etc/ssh/sshd_config: expected replacement of ".../Cellar/openssh/<version>" with ".../opt/openssh"Root cause: Homebrew preserves files under etc across upgrades. After the
first install, sshd_config already contains opt/openssh; a later install has
no Cellar path left to replace, but the formula treats that valid no-op as an
error.
Fix: install/patch-homebrew-openssh.sh guards the replacement with a
content check. It leaves an already-normalized configuration untouched, while
the formula’s test still rejects any Cellar path that remains.
Brew has the current keg but still uses an older version
Section titled “Brew has the current keg but still uses an older version”Symptom: brew list --versions glib or another formula lists the current
version, and brew outdated prints nothing, but opt/<formula>, bin/<tool>,
or pkg-config still resolves an older keg. This can follow an interrupted or
failed upgrade.
Inspect both the selected keg and the current keg’s receipt:
formula=glibprefix=$(brew --prefix)brew info --json=v2 "$formula" | jq '.formulae[0] | {linked_keg, installed}'readlink -f "$prefix/opt/$formula"ls "$prefix/Cellar/$formula"/*/INSTALL_RECEIPT.jsonIf the current keg has an install receipt, its direct executable works, and
brew linkage --test "$formula" passes, preview and repair only the links:
brew link --overwrite --dry-run "$formula"brew link --overwrite "$formula"If its receipt is absent or brew info --json=v2 reports a null install time,
the keg is incomplete. Do not force-link it; rebuild it:
HOMEBREW_NO_INSTALL_CLEANUP=1 brew reinstall --build-from-source "$formula"This run found both forms: Fish 4.8.1 was complete but its bin/fish symlink
still named 4.5.0, while GLib 2.88.3 lacked a receipt and had to be rebuilt.
OpenSSL 4 retries collide with an existing manpage
Section titled “OpenSSL 4 retries collide with an existing manpage”On Linux, an interrupted OpenSSL 3-to-4 transition can leave a complete but
unlinked openssl@4 keg while openssl@3 still owns the global links. A retry
can install over leftover output and fail with Errno::EEXIST for
share/man/man1/asn1parse.1ssl.gz: the formula tries to create a compressed
manpage symlink that already exists. A long-prefix bottle warning explains
the source build; it does not explain this file collision.
After all installs using the shared prefix have stopped, apply the receipt,
direct-executable, and linkage checks above to openssl@4. If they pass,
repair the transition without rebuilding the existing keg:
brew unlink openssl@3brew link openssl@4brew postinstall openssl@4HOMEBREW_NO_INSTALL_CLEANUP=1 brew upgrade wgetbrew linkage --test openssl@4 wgetwget --versionwget --spider --timeout=20 --tries=1 https://docs.brew.sh/Unlinking preserves OpenSSL 3’s installed keg and opt/openssl@3 path, but
source-built tools with an undeclared OpenSSL dependency may still need repair.
Check the checksum execution described under
checksum loader failures
before proceeding to other installers. OpenSSL 4 should become globally linked, its post-install
step should create the CA-certificate link, and wget should upgrade without
rebuilding OpenSSL. If the keg fails the checks, use the source-reinstall path
above instead; do not delete individual manpages or lock files.
On blackwell (Linux x86-64, shared NFS prefix, 2026-09-28), this repaired
OpenSSL 4.0.2_1 and installed wget 1.25.0_2; Bundle check and the HTTPS
request passed. The wget linkage audit still exited nonzero for indirect
libunistring linkage: the Linux formula declares it transitively through
dependencies, while this source build links it directly. It reported no
missing libraries; this formula-declaration issue is separate from the
OpenSSL installation failure and remains unresolved.
Brew Bundle reports a circular libtiff, webp dependency
Section titled “Brew Bundle reports a circular libtiff, webp dependency”Symptom: brew bundle check refuses to sort its graph even though both current
formulae are installed:
Formulae dependency graph sorting found a circular dependency: libtiff, webpRoot cause: the installed WebP receipt can retain an old libtiff dependency,
while the current libtiff formula depends on WebP. Generated keg receipts are
installation records; do not hand-edit them. Reinstalling WebP regenerates its
receipt from the current one-way dependency graph:
HOMEBREW_NO_INSTALL_CLEANUP=1 brew reinstall --build-from-source webpbrew bundle check --file="$HOME/dotfiles/packages/Brewfile"Rust fails after Homebrew says its packages are satisfied
Section titled “Rust fails after Homebrew says its packages are satisfied”Symptom: install/rust.sh reports Homebrew rustup not found even though
brew list rustup and brew --prefix rustup succeed.
Root cause: Homebrew’s keg-only rustup formula removed rustup-init. Older
bootstrap runs left ~/.local/cargo/bin/rustup pointing at the removed
/opt/homebrew/bin/rustup-init, and rust.sh incorrectly required that removed
binary before accepting the installed formula.
Confirm:
brew info rustup | grep -E 'keg-only|no longer provides'readlink ~/.local/cargo/bin/rustupls "$(brew --prefix rustup)/bin/rustup"Fix: update the checkout and run bash ~/dotfiles/install/rust.sh. The
installer now links Homebrew’s individual keg wrappers into the managed Cargo
bin directory and initializes stable with rustup toolchain install; it does
not depend on rustup-init.
macOS syntax checks report an unmatched quote in a formula patch
Section titled “macOS syntax checks report an unmatched quote in a formula patch”Symptom: ./tests/ci.sh macos fails in patch-homebrew-mesa.sh or
patch-homebrew-systemd.sh with unexpected EOF while looking for matching
a single quote, although the same scripts pass Linux checks.
Cause: Bash 3.2 misparses the embedded Python heredoc inside command substitution. Quoting the outer assignment does not resolve this parser failure. A newer Bash on Linux does not reproduce it.
Confirm: run /bin/bash -n install/patch-homebrew-mesa.sh and
/bin/bash -n install/patch-homebrew-systemd.sh on macOS. These commands parse
the files without installing packages or changing formulas.
Fix: update the checkout. Each patch now puts its Python heredoc in a shell
function and captures that function’s output. Formula transformations and
Linux-only execution remain the same. The macOS CI mode explicitly validates
with /bin/bash and runs both formula fixture suites under its constrained
system PATH, even when Homebrew Bash is installed.
See the Mesa source, systemd source, and Bash’s heredoc reference.
Mesa PyYAML patch target not found
Section titled “Mesa PyYAML patch target not found”Symptom: Linux bootstrap stops during Homebrew preparation with
mesa pyyaml patch target not found — formula changed.
Cause: Mesa added r.test? to its Python resource filter. The old exact
replacement only recognized the earlier filter, so it rejected the new formula.
Fix: update the checkout and rerun the maintained patch:
bash ~/dotfiles/install/patch-homebrew-mesa.shExpect a successful patch report, then an already-applied report on a repeat
run. The patch preserves the formula’s test/resource exclusions and existing
bindgen compiler workaround, selects the formula’s Python expression, and uses
the declared PyYAML version. Linux requires that version’s binary wheel;
unsupported formula or bindgen shapes fail before changing the file. This
prepares the formula without rebuilding Mesa. install/linux-packages.sh calls
it before Bundle; DF_PATCH_BREW_MESA=0 explicitly skips it.
See the patch source and the Homebrew formula.
systemd patch target not found
Section titled “systemd patch target not found”Symptom: Linux bootstrap reports systemd patch target not found after a
Homebrew formula update.
Cause: the old patch matched a literal "python3.14" interpreter and pinned
lxml to 6.0.2. The formula now passes a python3 variable to
virtualenv_create and declares its own newer lxml resource. Matching only the
new interpreter spelling would still install the wrong lxml version.
Confirm: inspect the interpreter and resource in the local formula:
brew cat systemd | grep -A 5 -E 'resource "lxml"|virtualenv_create'Fix: update the checkout and apply the maintained patch:
bash ~/dotfiles/install/patch-homebrew-systemd.shExpect a successful patch report, or an already-applied report on the next
run. The patch retains the formula’s interpreter expression and obtains lxml’s
version from resource("lxml").version. Linux builds require a wheel for that
exact version and fail if none is available; macOS retains its original
resource install. Unsupported formula shapes produce a visible failure
without being rewritten. DF_PATCH_BREW_SYSTEMD=0 explicitly skips the patch.
This repairs the formula preparation step; it does not rebuild systemd.
install/linux-packages.sh applies the patch before its package pass. See the
patch source and Homebrew’s
formula cookbook.
Cargo source build rejects OpenSSL 4
Section titled “Cargo source build rejects OpenSSL 4”Symptom: rust-docs-mcp or another Cargo source fallback fails in
openssl-sys with version: 4_0_3 and a message that only OpenSSL 1.x or 3
is supported. OPENSSL_DIR unset is diagnostic output, not the failure itself.
Cause: Homebrew’s global pkg-config search finds OpenSSL 4, while the
crate’s locked openssl-sys version rejects that major. An installed OpenSSL 3
keg is insufficient unless the build selects its versioned path. A later
successful doctor check can belong to the previously installed binary; the
failed replacement still makes install/rust.sh fail.
Confirm: compare default discovery with the available OpenSSL 3 headers:
pkg-config --modversion opensslbrew --prefix openssl@3test -f "$(brew --prefix openssl@3)/include/openssl/ssl.h"Fix: update the checkout and retry the Rust installer. If OpenSSL 3 is absent, install its declared Brewfile dependency first:
brew install openssl@3bash ~/dotfiles/install/rust.shThe installer selects opt/openssl@3 for source builds through OPENSSL_DIR,
adds its library directory to Linux runtime search paths, preserves explicit
OpenSSL path overrides, and leaves global Homebrew links
unchanged. Expect the source-build log to name OpenSSL 3 and the final count to
report zero failures. If you explicitly selected OpenSSL 4, remove or correct
that override before retrying. See Cargo package behavior
and the upstream build variables.
Bootstrap upgrade leaves GUI apps outdated
Section titled “Bootstrap upgrade leaves GUI apps outdated”Symptom: formula upgrades complete, but VS Code, ChatGPT, or other casks
remain outdated. The output says Cask upgrades deferred.
Root cause: the cask pass requires cached sudo authentication. Previously,
bootstrap only checked sudo -n true and skipped every cask if it failed, even
when individual apps did not need privileges. A manual sudo -v before starting
was unreliable because lengthy formula upgrades could outlast the timestamp.
Confirm: inspect the bootstrap output for the deferral message and compare Homebrew’s records with the app’s own version:
brew outdated --cask --greedyAuto-updating apps can have stale Homebrew receipts, so this list alone does not establish the version of the app bundle.
Fix: run the updated ~/dotfiles/bootstrap.sh upgrade from an interactive
terminal. Bootstrap authenticates once and maintains the timestamp until exit.
Keep DF_SUDO=auto and the desired cask/MAS upgrade gates enabled. Unattended
runs without authentication report the skipped upgrades in the degradation
summary. See sudo session behavior
for configuration and limitations.
If the log first reports sudo credential renewal failed, the credential was
lost during the run. Updated installers recheck before cask, App Store, and TeX
updates and can visibly reauthenticate in an interactive terminal. The keeper
preserves sudo’s error and retries without prompting. Inspect the effective
policy with sudo -l if renewal repeatedly fails; do not change sudo timeout or
security policy merely to suppress the warning. An unattended run still needs
an interactive follow-up to complete skipped updates.
Native Codex installed but the shell runs an older npm version
Section titled “Native Codex installed but the shell runs an older npm version”Symptom: ~/.local/bin/codex --version is newer than codex --version, or
bootstrap reports Codex PATH conflict after installing the complete runtime.
Root cause: nvm precedes the native bin directory on PATH, and a previous
global npm installation left its codex launcher in the node version’s bin
directory. Restarting the shell recreates the same ordering. Older cleanup code
also skipped the managed nvm directory when ~/.local resolved to scratch
storage outside $HOME, leaving that launcher in place after installation.
Confirm: compare command -v codex and the versions reported by that path
and the managed $LOCAL_PLAT/bin/codex (~/.local/bin/codex without isolation).
Fix: run bash ~/dotfiles/install/codex.sh install, then rehash in zsh
or hash -r in Bash. The installer validates the native runtime, retires known
npm launcher symlinks in the resolved managed root (including scratch storage
and PLAT layouts) and ~/.nvm, and verifies PATH resolution. It preserves package files
for active sessions and refuses to replace unknown launchers. See
native CLI migration.
Cask upgrade fails: It seems there is already an App at '/Applications/X.app'
Section titled “Cask upgrade fails: It seems there is already an App at '/Applications/X.app'”Symptom: bootstrap.sh upgrade reports Some greedy cask upgrades failed, and
brew upgrade --cask --greedy ends with Error: Problems with multiple casks:
naming an app (or a binary, e.g. already a Binary at '/opt/homebrew/bin/dnx').
Homebrew reverts the upgrade, so the same failure repeats every run.
Root cause: Homebrew refuses to overwrite an artifact it doesn’t have a receipt for. Two ways an auto-updating cask gets there:
- Upstream renames the app bundle. The cask’s
appstanza changes name, the self-updater has already written the new bundle, and brew’s receipt still points at the old one — so brew tries to create a file that exists. This is whatcodex-appdid: OpenAI folded the Codex desktop app into ChatGPT and renamedCodex.app→ChatGPT.app(samecom.openai.codexbundle ID). Thecodex-appcask is deprecated withchatgptas its replacement, and the auto-migration leaves an orphanCaskroom/codex-app/directory behind, sobrew list --caskstill shows it whilebrew infosays “Not installed”. - A leftover symlink from a previous install.
dnxpointing into/usr/local/share/dotnet/blocked everydotnet-sdkupgrade.
Confirm:
brew outdated --cask --greedy # which casks are stuckls /opt/homebrew/Caskroom/<cask> # receipt version vs the running app/usr/libexec/PlistBuddy -c "Print :CFBundleIdentifier" /Applications/X.appTwo apps printing the same bundle ID means one is a stale copy under the old name, not a second product.
Fix: brew install --cask <name> --force — it overwrites the unmanaged
artifact, removes the old-named bundle, and re-establishes the receipt at the
current version. Delete any orphan Caskroom/<old-cask>/ directory (it holds
only a symlink and metadata; rm -rf on it does not follow the symlink) and
remove stray binaries before retrying. For a renamed cask, also update
packages/Brewfile to the replacement name.
A Homebrew package is months behind upstream (Linux)
Section titled “A Homebrew package is months behind upstream (Linux)”Symptom: a formula installs at a version far older than upstream stable, and
re-running bootstrap never moves it. Seen with glab, which stuck at 1.89.0
while upstream was 1.109.0 — old enough that glab skills install --global
failed with Unknown command "skills", so install/skills-sync.sh reported
fail glab / missing declared skill: glab on every run.
Root cause: install/linux-packages.sh sets HOMEBREW_NO_AUTO_UPDATE=1 (so an
implicit refresh can’t revert the in-place formula patches) and, until now, never
ran an explicit brew update. The homebrew-core clone therefore froze at
whatever date it was first cloned, and HOMEBREW_NO_INSTALL_FROM_API=1 forces
every lookup through that frozen clone. One machine sat 4.5 months stale with
193 of 336 installed formulae behind upstream.
Confirm:
git -C "$(brew --repo homebrew/core)" log -1 --format=%ci # tap's agebrew info <formula> | head -1 # frozen versionFix: rerun install/linux-packages.sh — it now discards the formula patches,
runs brew update, and re-applies each patch against the fresh formula. This
refreshes definitions only; DF_BREW_UPGRADE still governs whether installed
kegs move, so a plain run leaves working binaries alone.
Upgrade a single package without touching the tap:
env -u HOMEBREW_NO_INSTALL_FROM_API -u HOMEBREW_NO_AUTO_UPDATE brew upgrade <formula>Expect patch anchors to rot across a long refresh — see the patch-anchor gotcha
in .claude/rules/homebrew.md.
Upgrade finishes installing, then the version audit calls newer tools outdated
Section titled “Upgrade finishes installing, then the version audit calls newer tools outdated”Symptom: bootstrap.sh upgrade installs current Homebrew packages, passes runtime
verification, then fails its strict version audit because an installed patch
version is newer than the reported exact baseline.
Root cause: an unpinned package-manager tool was audited as an exact pin. A
Homebrew upgrade could therefore make the machine newer than the repo baseline,
and the audit categorized every exact mismatch as outdated.
Confirm:
bash install/audit-versions.sh | jq '.[] | select(.status != "current")'brew info --json=v2 rustup juliaup | jq '.formulae[] | {name, stable: .versions.stable, installed: .installed[0].version}'Fix: update the repo, then rerun bootstrap.sh upgrade. Unpinned rustup,
juliaup, and unversioned Homebrew zig releases use minimum-version floors;
genuinely pinned toolchains
remain exact.
Quarto follows the same ownership rule: the macOS Homebrew cask uses a minimum
floor of 1.10.18, so an installed 1.10.19 passes. Linux uses an exact release
archive pin, including an explicit DF_QUARTO_VERSION override. An older audit
applied the Linux pin on macOS and incorrectly rejected newer cask releases.
After updating the audit, rerun bash install/audit-versions.sh --strict; a
successful report requires every row to have status: current and exits zero.
Skill upgrade reports both “up to date” and committed-digest drift
Section titled “Skill upgrade reports both “up to date” and committed-digest drift”Symptom: the skills updater says global skills are current, then
install/skills-sync.sh reports content differs from committed digest.
Root cause: the mutable installer receipt and committed review digest serve different purposes. A legitimate upstream update advances the receipt, while the committed digest stays unchanged until the new content is reviewed. Local trees that differ from their installer receipt are preserved instead.
Confirm with the read-only audit:
bash install/skills-sync.sh checkAfter reviewing the changed skill trees, refresh the review baseline with
bash install/skills-sync.sh lock. Do not use npx skills check for this; it
updates installed content. Self-installed skills must declare their own force
flag. The glab row uses --force, because glab skills install otherwise
refuses an existing file but exits successfully.
Cursor skills remain a directory instead of the shared symlink
Section titled “Cursor skills remain a directory instead of the shared symlink”Symptom: readlink ~/.cursor/skills prints nothing, Cursor has a separate
skill tree, or chezmoi apply cannot replace ~/.cursor/skills with its managed
link.
Root cause: an older Cursor installation created ~/.cursor/skills as a real
directory before this repository made it a symlink to ~/.claude/skills. The
pre-chezmoi ownership migration preserves that directory as an adjacent backup,
copies only top-level entries that do not already exist in the shared tree, and
does not overwrite a shared collision.
Confirm the current shape without changing it:
test -d ~/.cursor/skills && ! test -L ~/.cursor/skills && echo 'real directory'readlink ~/.cursor/skillsFix: run the migration, then rerun bootstrap so it records its chezmoi source and creates the managed link:
bash ~/dotfiles/install/skills-sync.sh adopt~/dotfiles/bootstrap.sh updatereadlink ~/.cursor/skillsThe last command should print ../.claude/skills. The original complete tree
remains at ~/.cursor/skills.pre-chezmoi-*; inspect it before deleting anything.
For a name collision, the shared ~/.claude/skills/<name> stays authoritative and
the Cursor version remains only in that backup.
GLIBC_x.y not found from a Homebrew binary (Linux)
Section titled “GLIBC_x.y not found from a Homebrew binary (Linux)”Symptom: a brew-installed binary refuses to start, blaming Homebrew’s own libc:
.../opt/binutils/bin/as: .../opt/glibc/lib/libc.so.6: version `GLIBC_2.38' not foundOnly some binaries are affected — the recently installed ones — and the error
names whichever binary you happened to run, so it reads like a problem with that
package. ldd disagrees and shows the system libc, because it resolves through
the system loader while the binary itself runs under brew/lib/ld.so.
Root cause: the glibc keg is older than the bottles. Homebrew’s Linux bottles
carry the glibc floor of the CI image that built them, and when homebrew-core
moves that image it bumps the glibc formula in the same breath (Ubuntu 22.04 →
24.04, glibc 2.35 → 2.39, July 2026). Nothing upgrades an installed glibc keg on
its own — it isn’t in the Brewfile — so every formula poured after the move lands
with a floor the keg can’t meet. Aug 2026: seven kegs (binutils, gcc@15,
texlab, tinymist, cadical, harper, juliaup) broke at once, all poured
by one brew bundle run days after the builder moved.
Confirm:
brew outdated --formula --verbose glibc # glibc (2.35_2) < 2.39_1ldd --version | head -1 # host glibcjq -r '.built_on.os_version' "$(brew --cellar)"/<formula>/*/INSTALL_RECEIPT.jsonFix: rerun install/linux-packages.sh. It reconciles the keg against the
formula before the bundle, and checks every keg installed since the last run
against what the keg provides. The broken kegs need no reinstall — they were
fine all along; only the loader under them was too old.
Upgrading by hand needs one guard:
HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1 brew upgrade glibcA bare brew upgrade glibc moves the keg in ~3 minutes and then spends hours
source-rebuilding every dependent whose linkage it considers stale, holding a
formula lock the whole time so every other brew command fails with “has already
locked”. The rebuild buys nothing: glibc is backward compatible, and the kegs
built against the old one keep running.
Homebrew refuses to build a glibc newer than the host’s, so a host older than the formula can’t be fixed this way. That combination has no in-place remedy: either the host glibc moves, or the keg goes and every formula is reinstalled against the host loader. The script warns rather than pretending.
Related: brew reinstall glibc fails with Errno::ENOENT ... gcc-13. Reinstall
unlinks the keg before building, which leaves brew/lib/ld.so dangling, and no
brew binary — including the compiler — can start. Upgrades build the new keg
first and are safe; a keg can only be rebuilt at the same version by rebuilding
the prefix.
NODE_MODULE_VERSION mismatch / ERR_DLOPEN_FAILED from an npm tool
Section titled “NODE_MODULE_VERSION mismatch / ERR_DLOPEN_FAILED from an npm tool”Symptom: an npm-installed CLI dies loading a native addon, usually mid-script:
Error: The module '.../node_modules/better-sqlite3/build/Release/better_sqlite3.node'was compiled against a different Node.js version using NODE_MODULE_VERSION 141.This version of Node.js requires NODE_MODULE_VERSION 147.Root cause: the package was installed under one Node major and is being run by
another. Native addons are ABI-locked per major, and the bin shebang is
#!/usr/bin/env node — so whichever Node is first on PATH wins. Two ways in:
- Two Node layers. Globals installed under Homebrew’s node keg and under nvm.
List them:
npm ls -g --depth=0 --prefix "$(brew --prefix)"should show onlynpm. Remove strays withnpm uninstall -g --prefix "$(brew --prefix)" <pkg>— nvm is the layer that owns npm globals. - PATH order.
brew shellenvputs brew’s bin ahead of nvm’s;install/_lib.shrestores nvm-first for install scripts, but an ad-hoc shell can still invert it.
It hides well: only subcommands that actually load the addon fail. qmd collection show works while qmd update aborts, which reads like a broken index rather than a
broken interpreter.
Fix: run it under the Node that installed it (command -v node should be under
$NVM_DIR), or reinstall the package under the current Node (npm install -g <pkg>).
nvm rejects prefix or globalconfig in ~/.npmrc
Section titled “nvm rejects prefix or globalconfig in ~/.npmrc”Symptom: install/node.sh stops during nvm use or after installing a Node
version:
Your user’s .npmrc file (${HOME}/.npmrc)has a `globalconfig` and/or a `prefix` setting, which are incompatible with nvm.Root cause: a legacy prefix=~/.npm sends every global package to one shared
tree. nvm instead gives each Node version its own global prefix under $NVM_DIR;
mixing the two makes package binaries and native addons run under the wrong Node
ABI. globalconfig can redirect npm to another file containing the same
conflict.
Fix: rerun install/node.sh. Before loading nvm it atomically removes an
active prefix from the user .npmrc, preserves the remaining npm policy, and
keeps the file private. If .npmrc sets globalconfig, the installer leaves it
unchanged and stops: copy any needed registry/auth/policy from the referenced
file into ~/.npmrc, remove the globalconfig line, then rerun. nvm owns the
Node/npm prefix; packages/npm.txt owns the global CLI set. Do not export
NPM_CONFIG_PREFIX or add another npm prefix for this setup.
Verify the selected runtime and global tree agree:
source "$NVM_DIR/nvm.sh"nvm use default --silentcommand -v nodenpm prefix -gdirname "$(dirname "$(nvm which default)")"The last two paths must match.
uv update reports a checksum mismatch because sha256sum cannot start
Section titled “uv update reports a checksum mismatch because sha256sum cannot start”Symptom: both uv self-update and its standalone installer print
sha256sum: error while loading shared libraries: libcrypto.so.3, followed by
ERROR: checksum mismatch with an empty got: value. The checksum program
did not run; this output does not establish a corrupt download.
Cause: a source-built Coreutils can auto-detect globally linked OpenSSL
without the Homebrew formula declaring that dependency. Its checksum binaries
then search the global Brew library directory. Transitioning global links from
OpenSSL 3 to 4 removes libcrypto.so.3 from that directory even though the
OpenSSL 3 keg remains installed. Keeping that keg alone does not repair the
binary’s search path.
Confirm the selected binary’s execution and ELF dependencies. If inspecting
runtime resolution, use its own ELF interpreter with --list; system ldd
can mislead by resolving libraries against the system glibc instead:
command -v sha256sumprintf abc | sha256sumreadelf -d "$(readlink -f "$(command -v sha256sum)")"Fix: after all other installs using the prefix have stopped, apply the managed formula patch and rebuild Coreutils:
bash ~/dotfiles/install/patch-homebrew-coreutils.shHOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_FROM_API=1 \ HOMEBREW_NO_INSTALL_CLEANUP=1 brew reinstall --build-from-source coreutilsprintf abc | sha256sumDF_MODE=upgrade bash ~/dotfiles/install/python.shThe digest must be
ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad.
The patch adds an explicit openssl@4 dependency and --with-openssl=yes,
preserving OpenSSL acceleration while giving the build a versioned library
search path. linux-packages.sh checks
the known digest before and after Bundle and rebuilds a broken installed
Coreutils; python.sh preserves the original
self-update failure instead of claiming every failure means an unsupported
uv build.
DF_PATCH_BREW_COREUTILS=0 or DF_PATCH_BREW_ALL=0 disables this Linux
patch and automatic repair. The check covers checksum execution, not every
installed program. Do not skip checksum verification, globally set
LD_LIBRARY_PATH, or relink OpenSSL 3 over OpenSSL 4. The authoritative
patch is patch-homebrew-coreutils.sh;
GNU documents the build option in its
Coreutils manual.
A Homebrew binary can’t find libX.so.N after an upgrade
Section titled “A Homebrew binary can’t find libX.so.N after an upgrade”Symptom: one program stops starting, naming a library version that used to exist:
node: error while loading shared libraries: libllhttp.so.9.3: cannot open shared object fileRoot cause: brew upgrade <formula> doesn’t stop at the formula — it then rebuilds every
dependent whose linkage the upgrade invalidated. Interrupt it in between (Ctrl-C, a
killed terminal, a timeout) and you’re left with the new dependency and the old
dependent: the dependent’s RPATH points at opt/<dep>/lib, which now holds only the new
soname. Both kegs are usually still in the Cellar, so brew list looks healthy.
Confirm:
brew list --versions <dep> # e.g. llhttp 9.3.1 9.4.3readelf -d "$(brew --prefix)/opt/<formula>/bin/<prog>" | grep RPATHFix: rebuild the dependent — brew upgrade <formula> (or reinstall). Don’t relink
the old dependency keg; that just moves the breakage to whatever wanted the new one.
Prevention: pass HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1 when you upgrade something
with many dependents, so nothing half-finishes in the first place.
nvm or node not available in a script
Section titled “nvm or node not available in a script”nvm.sh is lazy-loaded in interactive shells only. Login profiles put the bin
directory selected by nvm’s default alias on PATH. A standalone
non-interactive script should activate that same alias explicitly:
source "$NVM_DIR/nvm.sh"nvm use default --silentchezmoi keeps prompting for name/email
Section titled “chezmoi keeps prompting for name/email”The cached values live in ~/.config/chezmoi/chezmoi.toml. To reset:
chezmoi init --data=falseTo pre-seed without prompting:
chezmoi diff shows unexpected changes
Section titled “chezmoi diff shows unexpected changes”Another program modified a managed file. Common culprits:
uvauto-adds source lines to.zshrc/.bashrcfor itsbin/envfiles- Claude Code updates
~/.claude/settings.jsonwhen plugins are installed - Other tools may modify shell configs without asking
Options:
chezmoi diff # see what changedchezmoi apply --force # overwrite with repo version (safe for shell configs)chezmoi add ~/.claude/settings.json # pull the live version into the repo (for config files)For shell configs (.zshrc, .zprofile, .bash_profile), always use chezmoi apply --force to restore the clean template. These files should never be manually edited.
PATH order is wrong — wrong binary is resolving
Section titled “PATH order is wrong — wrong binary is resolving”Expected priority (highest to lowest). $_LOCAL_PLAT collapses to $HOME/.local in flat-mode (default).
$_LOCAL_PLAT/cargo/bin Rust tools (fd, sd, zoxide, bat, rg, etc.)$_LOCAL_PLAT/nvm/.../bin Node.js (highest installed version)$_LOCAL_PLAT/bin chezmoi, uv, claude, codex, uv-tool entrypoints~/.local/bin arch-neutral scripts (collapses to $_LOCAL_PLAT/bin in flat mode — deduped via typeset -U)/opt/homebrew/bin Homebrew (macOS) — also where rustup lives/opt/homebrew/sbin Homebrew sbin/usr/bin systemDiagnose with:
which <tool> # where it's resolving fromtype -a <tool> # all locations on PATHecho $PATH | tr ':' '\n' # full PATH in orderIf a Homebrew tool is shadowing a cargo tool, check packages/cargo.txt and packages/Brewfile for duplicates — remove the one you don’t want.
The other classic shadowing footgun: legacy binaries at ~/.local/bin/<tool> from before a layout migration. The [[ -x "$ARCH_BIN/<tool>" ]] install checks in current scripts catch most of these, but if <tool> --version shows an unexpectedly old version, check ls ~/.local/bin/<tool>* for backups (*.preplat-bak.* or stale binaries) and delete them.
nsys / ncu resolve to the CUDA toolkit copy, not the standalone install
Section titled “nsys / ncu resolve to the CUDA toolkit copy, not the standalone install”Symptom. You added a newer Nsight Systems / Nsight Compute to
dotfiles-nvidia/packages/{nsys,ncu}-versions.txt, the installer reports ok,
nsys_list / ncu_list show it — but nsys --version still prints the older
version, and which nsys points at $_LOCAL_PLAT/.cuda/bin/nsys.
Root cause chain. The CUDA toolkit bundles its own nsys and ncu. In the
shell profiles the ### CUDA ### block runs before the Nsight blocks, and
cuda_use prepends $CUDA_HOME/bin to PATH. The Nsight auto-activation used to
be guarded on ! command -v nsys (“activate only if not already in PATH”) — by
that point the toolkit had always put one there, so the guard never fired and
the standalone install was permanently shadowed.
A second, independent trap: Nsight Compute ships ncu at the root of its
tree, not under bin/ (Nsight Systems does use bin/). So even when ncu_use
did run, prepending $NCU_HOME/bin added a nonexistent directory.
Confirm.
which nsys ncu # .cuda/bin/... means it's shadowedls "$_LOCAL_PLAT/.ncu" # ncu at top level, no bin/readlink "$_LOCAL_PLAT/.nsys" # which standalone version is activeFix. Both are fixed in the profile templates: activation is now
unconditional whenever $_LOCAL_PLAT/.nsys / .ncu exists (the standalone
prepends after cuda_use, so it wins), and ncu_use falls back to
$NCU_HOME when there is no bin/. Run chezmoi apply and start a new login
shell. If it still resolves wrong, the version symlink is the likely culprit —
cuda.sh/nsys.sh/ncu.sh deliberately never overwrite an existing .nsys /
.ncu / .cuda, so a new install does not become active on its own:
nsys_switch tarball_nsys_2026.1.3.425ncu_switch tarball_ncu_2026.2.1.5Cloudflare Pages build failing
Section titled “Cloudflare Pages build failing”Check the build log via the API:
ACCOUNT="YOUR_CLOUDFLARE_ACCOUNT_ID"TOKEN="..."# List recent deploymentscurl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/pages/projects/dotfiles/deployments" \ -H "Authorization: Bearer $TOKEN" | python3 -m json.tool | grep -E '"id"|"status"'
# Get logs for a specific deploymentDEPLOY_ID="..."curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/pages/projects/dotfiles/deployments/$DEPLOY_ID/history/logs" \ -H "Authorization: Bearer $TOKEN" | python3 -c "import sys, jsonfor e in json.load(sys.stdin)['result']['data']: print(e['line'])"Common causes:
- Unsupported Node version — the handbook requires Node 22.12 or newer; CI selects Node 24. Check the build environment and
site/package.json. - Chromium executable missing — Mermaid renders during the build. Install the matching browser with
npx --prefix site --no-install playwright install chromium; keepPLAYWRIGHT_BROWSERS_PATHidentical for installation and the build. - Invalid link or missing feature coverage — run
./tests/ci.sh docs, then fix the named Markdown link, source path, or explicit entry indocs/_data/features.json. Do not disable the checker. - Build output not found — confirm
destination_dir = "site/dist"ininfra/cloudflare/main.tf. The Astro build is verified before upload.
Two machines fighting over dotfiles on a shared home
Section titled “Two machines fighting over dotfiles on a shared home”This happens when a template renders differently on each machine (e.g. using {{ .chezmoi.arch }}). The rule: templates must be arch-neutral. Arch-specific logic belongs in shell runtime code, not templates.
Check which template is causing the conflict:
chezmoi diff # shows what chezmoi wants to change vs what's on diskThe fix is almost always to replace a template variable with a shell runtime expression. See Managing dotfiles → Shared home safety.
A symlinked ~/.codex or ~/.claude turned back into a real directory
Section titled “A symlinked ~/.codex or ~/.claude turned back into a real directory”Symptom: you hand-symlinked ~/.codex (or ~/.claude) to scratch, and some time
later it is a plain directory again holding only the managed config — while the
scratch copy sits frozen at the date the link died. Nothing logged an error.
Root cause: chezmoi manages files inside both directories (config.toml,
AGENTS.md, hooks.json, profiles, rules/, themes/, agents/ for Codex;
settings.json, skills/, hook scripts for Claude). A source directory means
chezmoi’s target state for that path is “directory” — so on the next chezmoi apply
it removes whatever is there, symlink included, and recreates a real directory with
just the managed files. Everything else is orphaned wherever the link pointed.
Confirm — a stale target next to a fresh $HOME copy is the tell:
chezmoi managed | grep -x '.codex' # non-empty ⇒ chezmoi owns this pathstat -c '%n %y' ~/.codex ~/scratch/.codex # scratch frozen, home currentFix: don’t symlink the managed directory. For Codex on a shared home,
configure a host-local runtime and run bash install/codex.sh sync-runtime.
The scratch installer no longer moves Codex children or SQLite files. For
Claude, it still supports selected unmanaged children. See
Scratch space.
Then reconcile the orphaned copy by hand — the script won’t touch it, because it cannot tell your stale data from a deliberate second install:
du -sh ~/scratch/.codex # what was orphanedThe supported CODEX_HOME boundary now includes managed-configuration sync;
see Codex host-local state.
Codex background server has incompatible feature settings
Section titled “Codex background server has incompatible feature settings”Symptom: interactive Codex requires api_key_model_discovery and offers
Run without daemon this time; restarting is disabled because the server is
not managed by Codex.
Root cause: the CLI and the shared app server have different effective startup features. A desktop/SSH-launched app server can outlive CLI upgrades, and the CLI cannot automatically restart a server it does not manage. A version difference is useful evidence, but does not by itself establish which feature differs.
Confirm: compare the CLI and running server through the native command:
codex --versioncodex app-server daemon versioncodex features listFix for the current invocation: select Run without daemon this time, or:
codex --no-daemonThis preserves the shared server and its existing clients. For a persistent
repair, restart it through the desktop/SSH client that owns it when active and
queued work may be interrupted, using the shared features named by the
compatibility dialog. Changing a file does not update an already running
server’s startup features. Sources: official CLI reference
and the installed CLI’s codex --help.
SSH works but the remote Codex desktop cannot initialize its database
Section titled “SSH works but the remote Codex desktop cannot initialize its database”Symptom: ordinary SSH succeeds, while app-server startup fails with a
damaged-database recovery error and Cross-device link (os error 18).
Root cause: individually symlinked SQLite files and their recovery directory can cross filesystem boundaries. Separately, placing WAL databases on NFS allows incompatible multi-host access. Moving them to another NFS export does not solve that problem. A clean SQLite integrity check does not establish safe WAL locking or successful recovery renames.
Confirm on Linux: inspect the actual filesystem and link targets, not the directory names. Avoid printing authentication files.
findmnt -T ~/.codex -o TARGET,FSTYPEreadlink -f ~/.codex/state_5.sqlitereadlink -f ~/.codex/db-backupstail -n 30 ~/.codex/app-server-control/app-server.logFix: stop writers and make a full verified backup of both the legacy root
and any existing destination first, using codex.sh backup SOURCE DESTINATION
as described in host-local state. Select
persistent host-local state with install/host.sh configure, run
bash install/codex.sh sync-runtime, and reconnect the desktop’s SSH
connection. Verify its daemon received the new CODEX_HOME and that its Unix
socket and databases are local. Keep the old shared tree for explicit history
recovery. Do not delete a database merely because startup called it damaged,
and do not symlink app-server-control: the native Unix listener requires a
real directory. See host-local state.
If the error still names ~/.codex after the desktop is working locally, inspect
the failing launcher’s environment. An old terminal/tmux shell may retain the
previous codex() function, and non-login shells never read login profiles.
Current Bash/Zsh wrappers resolve host policy lazily; reload the shell once
after deployment (exec zsh -l or exec bash -l). A new desktop process using
the local root does not prove an old terminal inherited that same environment.
Duplicate PLAT paths in PATH (both v3 and v4 showing up)
Section titled “Duplicate PLAT paths in PATH (both v3 and v4 showing up)”Only relevant with DF_USE_PLAT=1. Fixed in current versions — .zprofile/.bash_profile resolve ~/.local symlinks before setting _LOCAL_PLAT so PATH entries use the same physical path.
If you upgraded from before that fix:
chezmoi apply ~/.zprofile ~/.bash_profileexec zsh -l # or: exec bash -lecho "$PATH" | tr ':' '\n' | grep plat # all entries should share the same PLAT prefixIn flat mode (DF_USE_PLAT=0, the default), this failure mode doesn’t apply — there’s no $PLAT segment in $_LOCAL_PLAT.
Lost shell history
Section titled “Lost shell history”Zsh history lives at ~/.zsh_history (the conventional default; survives any ~/.local cleanup). Bash history at ~/.bash_history. The bash sidecar command log (richer: timestamps, exit codes, cwd) at ~/.bash_log — search via bash_log_search <pattern>.
If you have history under the old location (~/.local/state/{zsh,bash}/), one-time migrate:
[ -f ~/.local/state/zsh/history ] && mv ~/.local/state/zsh/history ~/.zsh_history[ -f ~/.local/state/bash/history ] && mv ~/.local/state/bash/history ~/.bash_history[ -f ~/.local/state/bash/log ] && mv ~/.local/state/bash/log ~/.bash_logMigrating off PLAT isolation
Section titled “Migrating off PLAT isolation”If you set up with DF_USE_PLAT=1 and want to switch to flat (or vice-versa), the layout in ~/.local/ is stable as long as one mode is active — but switching strands GBs in the unused tree. Decommission tool:
# After setting DF_USE_PLAT=0 (or removing use_plat=true from chezmoi data):bash ~/dotfiles/install/plat-decommission.shRefuses to run if DF_USE_PLAT=1 is currently set (won’t nuke the active install). See PLAT isolation for the full migration story.
Brew zsh tab completion leaves remnant characters (Linux)
Section titled “Brew zsh tab completion leaves remnant characters (Linux)”Symptom: after pressing Tab, stale characters remain on the line instead of being erased.
Root cause chain:
- Brew zsh’s RUNPATH loads Homebrew’s own glibc (
brew/opt/glibc/lib/libc.so.6) - Homebrew’s glibc ships no
lib/locale/data →setlocale()silently falls back toC/ASCII - In the C locale,
wcwidth()returns byte counts instead of display columns - Every cursor-position calculation in ZLE/completion is off → artifacts
Confirm by checking the codeset inside brew zsh:
zsh --no-rcs -c 'zmodload zsh/langinfo; echo $langinfo[CODESET]'# broken: ANSI_X3.4-1968# working: UTF-8Fix: linux-packages.sh generates en_US.UTF-8 and C.UTF-8 locale data for brew’s glibc into
$LOCAL_PLAT/locale/ using brew’s own localedef. The shell profiles export LOCPATH
pointing there so brew zsh picks it up at startup.
Both locales are needed: non-login processes can inherit LC_CTYPE=C.UTF-8
while the profile sets LANG=en_US.UTF-8. Without the category locale, brewed
Perl reports Setting locale failed on each tlmgr invocation. Generating the
matching data preserves the requested UTF-8 locale without suppressing warnings.
If you installed before this fix:
# Regenerate locale databash ~/dotfiles/install/linux-packages.sh
# Apply updated shell profiles (adds LOCPATH export)chezmoi apply ~/.zprofile ~/.bash_profile
# Open a new login shell and verifyexec zsh -lzsh --no-rcs -c 'zmodload zsh/langinfo; echo $langinfo[CODESET]' # UTF-8Test suite: bash ~/dotfiles/tests/test-locale.sh
Copy/paste from a remote SSH session pastes as mojibake (’, Â, é)
Section titled “Copy/paste from a remote SSH session pastes as mojibake (’, Â, é)”Symptom: text copied out of a remote Linux session pastes with latin-1 garbage
where punctuation, accents, or spaces should be: ’ becomes ’, é becomes
é, non-breaking spaces surface as  . Two common shapes:
- plain ssh + tmux — the display itself is garbled, and copies carry it
- VS Code/Cursor Remote-SSH embedded terminal — display may look fine, but
copying agent TUI output (Claude Code renders with padding/NBSP characters)
pastes with stray
Âaccent characters; no tmux involved
The terminal emulator (iTerm2, xterm.js) is innocent: the bytes are already mangled before they reach it.
Root cause chain:
- On macOS,
.zprofileexportsLC_ALL=en_US.UTF-8 - macOS ships
SendEnv LANG LC_*in/etc/ssh/ssh_config, and Linux sshd acceptsLC_*by default — the Mac’sLC_ALLlands in the remote environment. This covers Remote-SSH too: the VS Code/Cursor server is started over that same ssh connection, and every embedded terminal inherits its environment LC_ALLoverridesLANG, defeating the deliberate LANG-only locale setup in the Linux shell profiles (see the entry above). Embedded terminals are hit hardest: they spawn non-login shells, so a profile-only guard never even runs there- On hosts whose system glibc has no
en_US.UTF-8compiled (minimal server images — the brew-glibcLOCPATHdata doesn’t help system binaries),setlocale()falls back to C/ASCII - Anything in that C locale that re-encodes the byte stream (system tmux is the classic offender) treats each UTF-8 byte as a separate latin-1 character — the display, and therefore anything selected and copied from it, is mojibake
Confirm on the remote, inside the garbling session:
locale; echo "LC_ALL=$LC_ALL"; locale -a 2>/dev/null | grep -iE 'en_US|utf'printf 'caf\xc3\xa9 \xe2\x80\x94 \xe2\x80\x9cok\xe2\x80\x9d\n' # should render: café — “ok”Broken looks like: a “cannot change locale” warning or LC_CTYPE="C" in the
locale output, and the printf line rendering as café — “okâ€.
Fix: the locale guard (unset LC_ALL before exporting LANG, from the
locale-env.sh shared partial) runs in the Linux shell profiles AND the
interactive rc files — the rc copy is what protects non-login embedded
terminals. Then:
chezmoi apply ~/.zprofile ~/.bash_profile ~/.zshrc ~/.bashrctmux kill-server # the tmux server caches the locale it started withexec zsh -l # or reconnect / open a fresh embedded terminalNote the fix cleans the encoding; agent TUIs like Claude Code still put
invisible layout characters (padding spaces, hard wraps) into the scrollback,
so terminal-selection copies of long output stay imperfect. For clean text use
/export or copy from the paired web/mobile session instead.
If it’s still garbled, the host has no UTF-8 locale usable by system binaries at
all — check locale -a; export LANG=C.UTF-8 (built into every modern glibc)
is the fallback.
Note the tempting client-side fix does NOT work: SendEnv -LC_* in
~/.ssh/config is a no-op here, because ssh reads the user config before
/etc/ssh/ssh_config and -pattern removals apply at parse time — the system
default adds the patterns after your removal runs.
python is Python 2 or cannot import SymPy
Section titled “python is Python 2 or cannot import SymPy”Symptom: python --version reports 2.7, or python -c 'import sympy'
fails after bootstrap.
Cause: The interactive Python environment or its entrypoint symlink is missing for the active PLAT. This commonly appears after moving between hosts with different CPU capability levels: another PLAT may have Python packages, but its binaries are intentionally not on this host’s PATH.
Fix: Re-run the Python installer for the active platform:
bash ~/dotfiles/install/python.shhash -rpython -c 'import sys, sympy; print(sys.version, sympy.__version__)'The installer owns $LOCAL_PLAT/python and exposes its interpreter as
$LOCAL_PLAT/bin/python; project-specific dependencies still belong in each
project’s uv environment.
WhisperX upgrade fails with SQLAlchemy duplicate normalized extra name
Section titled “WhisperX upgrade fails with SQLAlchemy duplicate normalized extra name”Symptom: uv tool upgrade --all fails while building SQLAlchemy 2.1.0,
reporting duplicate normalized extra name mssql-pymssql.
Root-cause chain: WhisperX depends indirectly on SQLAlchemy through
pyannote and Optuna. SQLAlchemy’s source metadata declares both
mssql-pymssql and the legacy alias mssql_pymssql; uv rejects this collision.
On 2026-09-24, the PyPI release initially lacked a Python 3.12 macOS ARM64
wheel, exposing the source metadata failure. Later that session, the matching
wheel became available and the same installer upgrade succeeded with SQLAlchemy
2.1.0. This bypasses the source-package defect; it does not repair its metadata.
Confirm: inspect the failing package/version and the platform wheels on the SQLAlchemy release page. The extra-name normalization specification explains why underscores and hyphens collide.
Fix: once a compatible wheel is published, retry the managed upgrade:
DF_MODE=upgrade bash ~/dotfiles/install/python.shExpected result: all tool upgrades and the installer’s entrypoint checks pass. Do not edit uv’s cached source or change the default Python solely to bypass this error. Other platforms without a compatible wheel may still encounter it.
[email protected] build fails on Linux (uuid or test_datetime errors)
Section titled “[email protected] build fails on Linux (uuid or test_datetime errors)”Python 3.14 from Homebrew has build issues on some Linux systems:
- UUID module detection failure - configure detects libuuid but the build fails
- test_datetime hangs during PGO - Profile-guided optimization runs the test suite, but
test_datetimehangs on some CPUs (timezone-related)
Fix: Patches are applied automatically by install/patch-homebrew-python.sh during bootstrap. If you need to re-apply manually:
bash ~/dotfiles/install/patch-homebrew-python.shThe patches:
- Set
py_cv_module__uuid=n/ato disable the uuid module - Patch Makefile’s
PROFILE_TASKto skiptest_datetimeduring PGO
Environment variables in .zprofile/.bash_profile prevent Homebrew from auto-updating and overwriting these patches:
HOMEBREW_NO_AUTO_UPDATE=1- prevents tap updatesHOMEBREW_NO_INSTALL_FROM_API=1- forces local formula usage
cass source build fails with rustc 1.94.0 is not supported or E0554
Section titled “cass source build fails with rustc 1.94.0 is not supported or E0554”On a host with glibc < 2.38 (e.g. Ubuntu 22.04) cass has no usable prebuilt, so
memory.sh builds it from source — and you see one of:
rustc 1.94.0 is not supported by the following packages: [email protected] requires rustc 1.95 …# or, on a newer stable:error[E0554]: `#![feature]` may not be used on the stable release channelTwo root causes stacked:
- cass requires nightly. A dependency gates
#![feature(try_trait_v2)]and the repo pinschannel = "nightly". Stable can’t build it — an old stable fails the MSRV check, a new stable failsE0554. - A stray Homebrew
rustshadows rustup. Arustformula (a lingering build dependency — not in the Brewfile, nothing depends on it) putscargo/rustcinbrew/binat an old version. In bootstrap’s PATH that shadows rustup, socargoresolved to brew’s 1.94.0 even afterrust.shupdated rustup’s stable to 1.97.1.
Confirm:
which -a cargo # a brew/bin/cargo at an old version is the smoking gunrustup toolchain list # is `nightly` installed?Fix (already baked into current memory.sh — this is for older checkouts or
manual recovery):
brew uninstall rust # remove the orphan shadow (safe: nothing depends on it)rustup toolchain install nightly --profile minimal$CARGO_HOME/bin/cargo +nightly install --git \ https://github.com/Dicklesworthstone/coding_agent_session_search \ coding-agent-search --bin cass --locked --root "$LOCAL_PLAT"_cass_build_from_source now installs nightly on demand and calls
$CARGO_HOME/bin/cargo +nightly explicitly, so it no longer depends on PATH
resolution or the default toolchain.
cass search misses sessions you know happened
Section titled “cass search misses sessions you know happened”Symptom: history you remember from Codex or Cursor never surfaces, even on exact phrases, while Claude Code sessions from the same week come back fine.
Root cause: cass ingest is append-only per conversation. Once a conversation is
in the canonical DB it is never re-read, so every parser improvement since it was
first indexed only reaches new sessions. Old ones keep whatever subset the parser
of the day extracted. cass index --full does not fix this — it forces a full
scan and a lexical rebuild, then logs skipping historical salvage because canonical database is already populated.
Confirm — compare the DB against a fresh parse of the same files:
sqlite3 -readonly ~/.cass/agent_search.db \ "select a.name, count(m.id) from messages m join conversations c on c.id=m.conversation_id join agents a on a.id=c.agent_id group by 1 order by 2 desc;"wc -l < ~/.codex/sessions/2026/*/*/rollout-*.jsonl # rough upper bound per fileMeasured 2026-08-01: codex held 15,037 messages where the same 88 rollouts parse to 85,488 today, and cursor 3,384 vs 6,949 — 82% and 51% of that history unsearchable.
Fix, one connector at a time. Every step matters:
# 0. verify each source file still exists — forget is only safe if it can come backsqlite3 -readonly ~/.cass/agent_search.db \ "select c.source_path from conversations c join agents a on a.id=c.agent_id where a.name='codex';" | while read -r p; do [ -f "$p" ] || echo "MISSING: $p"; done
# 1. back up via sqlite, NOT cp — a plain copy can tear a live WALsqlite3 ~/.cass/agent_search.db ".backup '$HOME/.cass/agent_search.db.bak'"
# 2. drop the stale rows (dry-run first: omit --apply)cass forget --source-glob "$HOME/.codex/sessions/**" --apply
# 3. reset the connector watermark — --full still honours it, and rollouts dated# months ago never beat a watermark stamped today, so the scan finds nothingsqlite3 ~/.cass/agent_search.db \ "update meta set value='0' where key='last_scan_ts:connector:codex';"
# 4. re-ingest, then clean up after a cass bug: forget leaks tail-state rows keyed# by the deleted conversation_id, and that column is a plain rowid — SQLite# reuses freed ids, so a future conversation would inherit a stale# "ingested through idx N" marker and be silently truncatedcass index --fullsqlite3 ~/.cass/agent_search.db \ "delete from conversation_tail_state where conversation_id not in (select id from conversations);"
# 5. rebuild vectors in bounded batches; repeat until the backlog is emptybash ~/dotfiles/install/memory.sh semanticSkipping step 3 is the usual failure — the run exits 0, having ingested nothing.
cass index fails with graph topology attestation failed
Section titled “cass index fails with graph topology attestation failed”build HNSW index failed: hnsw error: graph topology attestation failed:parallel construction failed (search entry origin 6219 reaches only 92908/97513points at the base layer); serial rebuild also failedTranscripts contain thousands of byte-identical tool stubs — [Tool: apply_patch]
alone repeats 4,383 times — and identical text embeds to identical vectors. Under
DistDot those form zero-distance cliques larger than the layer-0 fanout
(max_nb_connection 16 → ~32 links), so a clique fills every member’s neighbour
list with its own duplicates and nothing outside ever links in. HNSW reachability
is directional, so the whole group is unreachable from the entry point and cass’s
attestation rejects the graph.
It is deterministic: retrying fails identically, which is why the serial rebuild
also failed and why retryable=true in the error is misleading.
Fix: drop --build-hnsw (the manual memory.sh semantic mode does not use
it). HNSW only backs --approximate; exact search over ~100k vectors is
fast enough, and the flag has never once succeeded on this archive — every
semantic_manifest.json here records "hnsw": null. Restore it if cass starts
deduping identical vectors before insert.
Every cass command fails: unable to open database file (but sqlite3 opens it fine)
Section titled “Every cass command fails: unable to open database file (but sqlite3 opens it fine)”opening frankensqlite db readonly at /Users/cade/.cass/agent_search.db:unable to open database file: '/Users/cade/.cass/agent_search.db'cass doctor reports archive-db-unreadable, cass index --full refuses to
run (“index refused to modify an unhealthy canonical archive”), and the
older cass-watch/cass-semantic LaunchAgents may crash-loop with exit 5 — yet
sqlite3 -readonly ~/.cass/agent_search.db "pragma quick_check;" says ok.
frankensqlite pins the database’s file identity — device id + inode — in the
agent_search.db-fsqlite-ns-use sidecar (record: 8-byte FSQLNS01 magic,
1-byte version, then tag/dev/ino big-endian). On macOS, APFS volume device ids
are assigned at mount time and can change across reboots. After the id shifts,
every read-only open compares recorded vs live identity and fails closed with
SQLITE_CANTOPEN. A read-write open would rewrite the record and self-heal,
but cass health-gates every mutating command behind a read-only open first, so
nothing ever reaches the heal path.
Confirm.
stat -f "dev=%d ino=%i" ~/.cass/agent_search.db # live identityhexyl -n 40 ~/.cass/agent_search.db-fsqlite-ns-use# bytes 10..18 = recorded dev (BE), bytes 18..26 = recorded ino (BE)Inode matches, device id doesn’t → this bug. If the inode differs, the db file was actually replaced — stop and investigate before touching anything.
Fix. Stop all cass processes, then patch the recorded dev to the live value
(here only the last byte differed, 0x10 → 0x0d at offset 17):
printf '\x0d' | dd of="$HOME/.cass/agent_search.db-fsqlite-ns-use" \ bs=1 seek=17 count=1 conv=notrunccass doctor # database failure should be gonebash ~/dotfiles/install/memory.sh indexbash ~/dotfiles/install/memory.sh semantic # one bounded vector batchRecurs whenever the Data volume mounts with a different device id. Do not
run cass doctor --fix for this: on 0.6.23 it enters unbounded recursion in
the reconstruct path (observed: 1.6 h at 100% CPU, ~50 GB RSS, no output) —
kill it if started; it only touches lock files before hanging.
macOS still asks: “cass would like to access data from other apps”
Section titled “macOS still asks: “cass would like to access data from other apps””After an upgrade, an old launchd job may still be scanning the session archive.
The current setup never schedules cass indexing: qmd stays warm for Markdown
search, while cass runs only when you invoke memory.sh index or semantic.
Older dotfiles deployed a dev.cade.cass-watch LaunchAgent that ran cass index
every 300 s, and the aider
connector crawls $HOME. Aider histories are project-local
(.aider.chat.history.md in each repo), so discovery walks its root — and that root
defaults to $HOME. The walk enters ~/Pictures, ~/Music, ~/Documents,
~/Desktop, ~/Downloads, and ~/Library, so it asks for Photos, MediaLibrary,
AddressBook, Calendar, AppData — and AllFiles.
It is tempting to blame the connectors that read ~/Library/Application Support
(cursor, chatgpt, copilot) — don’t. Measured 2026-08-05: a scan that opens 13
Cursor state.vscdb files raises zero TCC requests, because Cursor is a
non-sandboxed Electron app with no registered container, so its app-support dir isn’t
protected. Excluding those connectors costs you cross-harness session coverage and
fixes nothing.
“Allow” doesn’t make it stop because cass is ad-hoc signed (codesign -dv →
Signature=adhoc, no Team ID), so a grant is pinned to the binary’s cdhash and is
voided at the next cass upgrade. There is also no System Settings pane for App Data
grants, so a stale one can’t be repaired from the UI.
See exactly what cass is asking for — this is the diagnostic that matters, since the cass logs only record paths it opens deliberately, not what a directory walk touches:
/usr/bin/log show --last 30m --predicate 'process == "tccd"' --info \ | rg 'Sub:\{.*/\.local/bin/cass\}' | rg -o 'kTCCService[A-Za-z]+' | sort | uniq -c(/usr/bin/log explicitly — log is a shell function in this repo’s profiles.)
Fix — remove the old scheduler and bound any manual crawl. First rerun the current memory setup, which unloads the legacy cass jobs. Then keep the aider discovery root bounded for manual indexing:
export CASS_AIDER_DATA_ROOT="$HOME/dev" # aider discovery root, not $HOMEApplied on macOS by the manual install/memory.sh index modes. Aider still
indexes normally — just only under the given root, so aider projects elsewhere
go unindexed.
CASS_AIDER_DATA_ROOT takes a single path, so $HOME is the only “covers everything”
value and it is what causes the problem.
Verify with the log show command above: a scan should now produce no cass entries at
all. Check a scan really ran, or the empty result proves nothing —
rg 'skipping disabled connectors' ~/.local/share/cass/stderr.log | tail -1.
The alternative to all of this is granting ~/.local/bin/cass Full Disk Access
(kTCCServiceSystemPolicyAllFiles is one of the things it asks for), which covers every
service at once — but it must be re-added after every cass upgrade, and it hands a
self-updating ad-hoc-signed binary read access to Mail, Messages, and Safari history.
npm install -g fails with EBUSY … unlink '.nfsXXXX' (qmd upgrade)
Section titled “npm install -g fails with EBUSY … unlink '.nfsXXXX' (qmd upgrade)”bootstrap.sh upgrade (or install/node.sh) dies upgrading a global npm
package — almost always @tobilu/qmd:
npm error code EBUSYnpm error EBUSY: resource busy or locked, unlink'.../@tobilu/qmd/node_modules/sqlite-vec-linux-x64/.nfs000000001f79d0f000015a88'[fail] node.sh failedRoot cause: NFS “silly-rename”. The qmd MCP daemon
(qmd mcp --http --port 8181) keeps native addons (sqlite-vec,
node-llama-cpp, better-sqlite3) mmap’d. When npm deletes the old package
tree to swap in the new one, NFS can’t remove a file the daemon still has open,
so it renames it to .nfsXXXX and keeps it until that fd closes. npm then can’t
unlink the .nfs* file and aborts with EBUSY. Only happens on NFS homes
(the Linux clusters) — macOS local disks unlink open files fine, so this is
gated to Linux.
node.sh now stops the daemon around the qmd upgrade and restarts it (via the
qmd_daemon_* helpers in _lib.sh), so a normal upgrade no longer trips on it.
To recover a checkout that predates the fix, or if you hit it by hand:
pkill -f "qmd[^ ]* mcp --http" # 1. stop the daemon → NFS reaps .nfs* filesnpm install -g @tobilu/qmd@latest # 2. re-run the upgrade (or: bash install/node.sh)qmd mcp --http --daemon & # 3. restart (a new shell also lazy-starts it)A failed swap can also leave a broken husk — a qmd/ dir with only an empty
node_modules/ plus a dangling bin/qmd symlink — in a different npm prefix
than the one which qmd resolves to (nvm’s). Delete the husk; the live copy is
the one on PATH.
qmd is running but its health check cannot connect to 127.0.0.1:8181
Section titled “qmd is running but its health check cannot connect to 127.0.0.1:8181”Recent qmd releases may bind localhost:8181 only on IPv6 (::1). A check
hard-coded to IPv4 then fails even though the daemon and the configured MCP URL
are healthy. Confirm with curl http://localhost:8181/health; it should return
{"status":"ok",...}. The shared health check now uses localhost, matching
the MCP server URL in packages/mcp-servers.txt.
import sage.all / cysignals dies with TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object
Section titled “import sage.all / cysignals dies with TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object”Symptom. Importing cysignals.pysignals (directly, or transitively via
passagemath’s sage.all) raises the TypeError above. Plain import cysignals.signals (and cypari2) works fine.
Root cause. This macOS release (Darwin 25.x) pre-installs C-level fault
handlers (SIGILL, SIGABRT, SIGFPE, SIGBUS, SIGSEGV) in every process.
signal.getsignal() reports a handler Python didn’t install as None, and
cysignals’ pysignals init saves + re-installs existing handlers — re-setting
None is rejected by CPython. Not sandbox-, uv-, or Python-version-specific:
reproduced on uv’s python-build-standalone 3.12/3.13 and Homebrew 3.14.
Confirm.
python3 -c "import signal; print(signal.getsignal(signal.SIGSEGV))" # → NoneFix. Reset the fault handlers from Python before anything imports
cysignals.pysignals (~/dev/math-lab/sagefix.py does exactly this):
import signalfor s in (signal.SIGILL, signal.SIGABRT, signal.SIGFPE, signal.SIGBUS, signal.SIGSEGV): signal.signal(s, signal.SIG_DFL)Related trap: pinning cysignals older than what passagemath wheels were
built against fails later with cysignals.signals does not export expected C function _do_raise_exception — keep the resolver’s cysignals (1.12.x), fix
the handlers instead.
TinyTeX refuses package installation until tlmgr is updated
Section titled “TinyTeX refuses package installation until tlmgr is updated”Symptom: Linux bootstrap stops at the TeX baseline with tlmgr itself needs to be updated, followed by latex.sh failed.
Cause: TeX Live requires critical package-manager updates before it permits
package installation. The installer previously attempted tlmgr install before
its upgrade-mode self-update, so it could never reach the repair command.
Fix: update the checkout and run the installer again:
DF_MODE=upgrade bash ~/dotfiles/install/latex.shThe Linux installer now runs tlmgr update --self --all before installing the
baseline in upgrade mode. Install/update modes run tlmgr update --self first,
leaving broad package upgrades opt-in. It then installs the baseline and checks
pdflatex, latexmk, chktex, texcount, and latexdiff in the managed tool
directory. Expect TeX ready; a failed manager update, missing command, or
failed package operation still stops the installer. This does not bypass TeX
Live’s repository/release compatibility checks.
The implementation is install/latex.sh; see the
upstream tlmgr manual for self-update
and package-update behavior.
pdflatex: command not found on macOS with MacTeX installed
Section titled “pdflatex: command not found on macOS with MacTeX installed”Symptom. brew list --cask shows mactex and
/Library/TeX/texbin/pdflatex exists and is executable, but pdflatex,
latexmk, chktex, and texcount all report “command not found”.
Root cause. MacTeX installs into /Library/TeX/texbin, which is on no
default PATH. Its installer drops a /etc/paths.d/TeX entry, but that only
reaches path_helper-processed shells, and these profiles rebuild PATH
themselves. install/latex.sh verified the binary by absolute path, so the
step reported [okay] while nothing was actually reachable.
Confirm.
ls /Library/TeX/texbin/pdflatex # existscommand -v pdflatex # nothingFix. Handled by both shell profiles:
[ -d /Library/TeX/texbin ] && path=($path /Library/TeX/texbin) # zprofileAppended, not prepended, so Linux’s TinyTeX binaries (symlinked into
$ARCH_BIN by latex.sh) keep priority on a machine with both. Run
chezmoi apply ~/.zprofile ~/.bash_profile and start a new shell.
brew bundle installs nothing new and exits 0
Section titled “brew bundle installs nothing new and exits 0”Symptom. brew bundle install prints only Using <formula> lines and
succeeds. Packages just added to the Brewfile never appear, and no error names
them.
Root cause. A cask-only package declared as brew "..." instead of
cask "...". Homebrew resolves the whole dependency graph before installing
anything, so one unsatisfiable entry aborts the entire run — every other new
package is collateral, which is what makes this read as a no-op rather than a
failure. Hit Aug 2026 with brew "quarto": homebrew-core has no quarto formula
at all, only a cask.
Confirm.
brew bundle check --file=packages/Brewfile --verbose# → Formula quarto needs to be installed or updated.brew info --formula quarto# → Error: No available formula ... Found a cask named "quarto" instead.Fix. Move it into the if OS.mac? block as cask "quarto". Casks are
macOS-only, so a cask-only tool has no Homebrew route on Linux — install it
another way there rather than leaving a brew line that breaks every bundle
run. Check a new entry with brew info --formula <name> before committing.
A GUI app’s config is permanently dirty in chezmoi status
Section titled “A GUI app’s config is permanently dirty in chezmoi status”Symptom. chezmoi status shows MM on an app’s config file every time you
look, even when you changed nothing. Running chezmoi apply “fixes” it, then it
comes back after the app runs. Worse, bootstrap (which runs chezmoi apply --force) silently reverts real in-app settings changes along the way.
Root cause. Two writers on one file. The app owns and rewrites its config,
and a statically chezmoi-managed copy fights it. LinearMouse is the sharpest
case: it stamps "$schema": "https://schema.linearmouse.app/<app version>" into
the file, so the file goes dirty on a timer — every app update produces a
diff with no setting change behind it. That trains you to ignore the dirty
status, which is exactly when a real reverted setting slips past.
Confirm.
chezmoi diff ~/.config/linearmouse/linearmouse.json# - "$schema" : ".../0.11.3" <- what the app wrote# + "$schema" : ".../0.11.2" <- what apply would force backFix. Don’t let chezmoi manage app-owned configs. Use the apply/sync split
(install/linearmouse.sh, install/claude-desktop.sh,
install/codex-desktop.sh): the tracked source lives under install/<app>/,
apply merges it into the live file live-first so app-owned keys survive, and
sync captures in-app changes back. For LinearMouse specifically the tracked
source omits $schema entirely, so only genuine setting changes ever diff.
bash install/linearmouse.sh sync # capture in-app changes → repobash install/linearmouse.sh # push repo settings → app (default: apply)Adding a new app to this pattern means deleting its home/ chezmoi source
(chezmoi then leaves the live file alone), adding the script, and wiring a
DF_DO_* flag in bootstrap.sh.
codex -p <domain> cannot see its expected MCP tools
Section titled “codex -p <domain> cannot see its expected MCP tools”Symptom. A new Codex session using -p browser, creative, or another
domain profile starts, but the expected tools are absent or Codex rejects the
profile.
Root cause. Domain overlays are generated files in the active CODEX_HOME.
Older configurations selected core MCPs only; current Codex sync enables every
managed server unless DF_MCP_PROFILES explicitly narrows that selection.
An older checkout, an incomplete bootstrap, or a registry change can leave
the configuration or overlays stale. Codex accepts one profile per invocation.
Confirm and fix. Regenerate the configuration, then run the same parser check used by bootstrap:
env -u DF_MCP_PROFILES bash ~/dotfiles/install/codex.sh sync-configbash ~/dotfiles/install/codex.sh checkcodex -p browser debug prompt-input "healthcheck"Set DF_MCP_PROFILES=core or a named selection only when a narrower baseline
is wanted. Disabled custom entries outside the registry remain user-controlled.
Start a new Codex task after the check, because
an existing task keeps its initial tool inventory and may retain an old MCP
subprocess environment.
Persistent Playwright profile loses login cookies after close
Section titled “Persistent Playwright profile loses login cookies after close”Symptom. A Playwright CLI 0.1.19 profile reopens with local storage intact, but its persistent login cookie has disappeared.
Root cause. The CLI daemon’s context-close callback reenters graceful shutdown. On this Mac the second shutdown force-kills Chrome before cookies are flushed to disk. A persistent profile directory alone does not verify working login persistence.
Fix. df-browser close includes a guarded compatibility adapter for owned
sessions. It removes only the known duplicate-shutdown listener before normal
close, retaining Playwright’s completion listeners. Unknown implementations
leave the profile open with a clear error. The
browser workspace guide
documents the adapter and the real-browser regression test to run after updates.
Google Cloud MCP stops authenticating during a long session
Section titled “Google Cloud MCP stops authenticating during a long session”Symptom. An official Google Cloud MCP connects successfully, then starts returning authentication errors after its initial access token expires.
Root cause. Older Codex/OpenCode launchers minted GOOGLE_MCP_TOKEN once at
startup, and Claude’s old headersHelper minted it at connection time. Neither
path refreshed the token while the same connection remained active.
Fix. The current registry emits df-google-mcp as a local stdio transport
for all four harnesses. It uses Google’s ADC refresh interface before HTTP
requests. Refresh the installed helpers and affected configuration:
bash ~/dotfiles/install/agent-tools.shdf-google-mcp --check-runtimebash ~/dotfiles/install/codex.sh sync-configbash ~/dotfiles/install/claude.sh sync-mcpbash ~/dotfiles/install/opencode.sh sync-configbash ~/dotfiles/install/cursor.sh sync-mcpRestart the affected harness connection so it starts the relay. Existing ADC continues to work; revoked credentials, disabled services, and missing IAM permissions still need their own fixes. See Google Cloud MCP credentials.
Codex MCP login times out in an SSH shell after browser sign-in
Section titled “Codex MCP login times out in an SSH shell after browser sign-in”Symptom. The provider reports successful sign-in on the laptop, but the
remote Codex CLI waits and eventually reports timed out waiting for OAuth callback. The browser’s localhost callback may show connection refused.
Root cause. Codex’s callback listener is on the workstation. The laptop
browser reaches its own loopback interface unless the same port is forwarded
over SSH. ForwardAgent yes forwards SSH signing access; it does not deliver
an HTTP callback. By default the MCP callback port changes between logins.
Confirm. Read the callback host and port from the login URL’s redirect_uri
parameter. While login is waiting, compare ss -ltnp on the workstation with
lsof -nP -iTCP:<port> -sTCP:LISTEN on the laptop. Keep callback query strings
out of shared logs: they contain authorization codes and state.
Fix. Configure a host-specific DF_CODEX_MCP_CALLBACK_PORT and matching
loopback-only LocalForward, then retry a fresh login. If sign-in already
completed while the CLI is still waiting, establish the matching tunnel and
use the provider’s Finish OAuth button. See
MCP login from SSH for commands,
concurrent-host behavior, and existing SSH connection handling.
Codex MCP OAuth fails: “Authorization server response missing required issuer”
Section titled “Codex MCP OAuth fails: “Authorization server response missing required issuer””Symptom. codex mcp login <server> (or first use of an OAuth MCP server in
Codex) opens the browser, auth succeeds there, then the CLI dies with
failed to handle OAuth callback … Authorization server response missing required issuer: expected <server url>. The same server connects fine from
Claude Code.
Root cause. A Codex regression, not a server or config problem. Codex
0.143.0+ looks for an iss field in the token endpoint’s JSON response body —
where RFC 6749 doesn’t put one — instead of using the RFC 9207 iss callback
parameter it already validated. Spec-compliant authorization servers
(Cloudflare’s among them) fail the check. Tracked in
openai/codex#31573; introduced
via a modelcontextprotocol/rust-sdk change.
Confirm. codex --version ≥ 0.143.0, the issue above still open, and the
server works from another harness. For Cloudflare specifically, prove the
server itself is healthy with a direct handshake:
curl -s -X POST https://mcp.cloudflare.com/mcp \ -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"probe","version":"0.0.0"}}}'# → {"result":{…"serverInfo":{"name":"cloudflare-api"…Fix. Sidestep OAuth in Codex with a static bearer: the
--codex-bearer <ENV_VAR> annotation in packages/mcp-servers.txt makes
install/codex.sh emit bearer_token_env_var = "<ENV_VAR>" for that server
while every other harness keeps OAuth. Cloudflare rides
--codex-bearer CLOUDFLARE_API_TOKEN (from ~/.cloudflare.env,
bash install/auth.sh cloudflare); mcp.cloudflare.com accepts API-token
bearers directly, verified with the handshake above. Caveat: the token’s
scopes bound what the tools can do — OAuth carried your full user grant, so
mint a broader token if a tool call 403s. Remove the annotation once the
upstream fix ships.
Mac App Store reports No downloads initiated
Section titled “Mac App Store reports No downloads initiated”Symptom: bootstrap.sh upgrade reaches Upgrading Mac App Store apps (mas),
then prints one or more messages such as:
Error: No downloads initiated for ADAM ID 682658836Root cause: mas found a newer catalog version, but macOS’s private CommerceKit
API returned no download object and no underlying error. The message alone does
not distinguish App Store sign-in or account ownership problems from an App
Store service or cache failure. Homebrew has already finished by this point, so
this refusal must not turn the whole Homebrew reconciliation into a failure.
Confirm the pending apps and the local App Store configuration:
mas outdatedmas configmas 7 also requires root privileges for updates. The bootstrap only runs it
when sudo is already cached because its password prompt is hidden inside the
logged command. Run sudo -v before bootstrap to enable the automatic attempt.
Fix: open App Store, sign in or accept any pending account terms, and update
the named apps there. Then rerun mas outdated; once it is empty, rerun
~/dotfiles/bootstrap.sh upgrade. The bootstrap preserves mas warnings but
continues after Homebrew itself has reconciled successfully.
Homebrew LLDB installs but fails its runtime checks
Section titled “Homebrew LLDB installs but fails its runtime checks”Symptom: Homebrew successfully pours the lldb bottle, then lldb.sh reports
Homebrew LLDB failed its runtime checks after installation.
Root cause: the runtime probe asked LLDB to load /bin/true. Linux provides
that path, but macOS provides true at /usr/bin/true, so a healthy macOS LLDB
failed the probe before lldb-dap was checked.
Fix: update the dotfiles checkout and rerun:
~/dotfiles/bootstrap.sh upgradeThe probe now loads /bin/sh, which exists on both supported platforms.
Python tools repeatedly report missing entrypoints
Section titled “Python tools repeatedly report missing entrypoints”Symptom: python.sh successfully installs a package and lists its executables,
then immediately reports Installed tool has missing entrypoints; several
large tools may be reinstalled on every bootstrap run.
Root cause: the validator treated every console script advertised by a Python
package as part of the dotfiles contract and gave each script a 10-second cold
start. Those scripts are not equivalent to selected tools. For example,
mlx-lm advertises mlx_lm.evaluate, but that optional command imports
lm-eval, which the base mlx-lm install intentionally does not include.
Heavy ML commands could also exceed the cold-start bound and pass on the next
run after their caches were warm.
Fix: update the dotfiles checkout and rerun the upgrade. Each pip*.txt
declaration now names the command the dotfiles actually require with entry=.
The installer validates only those commands, uses a 30-second Python CLI bound,
and reports the exact package and command when validation fails.
Python bytecode caching stays disabled after updating dotfiles
Section titled “Python bytecode caching stays disabled after updating dotfiles”Symptom: uv run --offline python -c 'import sys; print(sys.dont_write_bytecode)'
prints True, or a terminal still exports CMAKE_CXX_COMPILER_LAUNCHER=ccache.
Root cause: a long-lived terminal, IDE, or agent inherited the former
PYTHONDONTWRITEBYTECODE=1 and compiler-cache policy. Editing the profile cannot
change an already-running parent process, and the old _PROFILE_SOURCED guard
could prevent a nested shell from refreshing it.
Fix: apply the managed dotfiles, then run . ~/.profile in Bash/Zsh or start
a new Fish shell. Cache policy now refreshes before the guard, and Fish imports
explicit unset operations. Python should report False, with bytecode under
~/.cache/python/pycache. A process launched directly by an old IDE can still
inherit old values until that IDE restarts. An explicit python -B remains a
command-specific bypass. See compiler-cache migration.
Login-profile fixture intermittently misses QMD startup
Section titled “Login-profile fixture intermittently misses QMD startup”Symptom: the push gate fails the QMD call-count assertion in tests/profiles.bats,
but rerunning the focused test passes.
Root cause: login profiles detach QMD inside a subshell. A later wait in the
sourcing shell cannot join that grandchild, so the fixture can inspect its log
before the fake QMD process writes it.
Confirm: run bats --filter 'rendered Linux login profiles' tests/profiles.bats.
The fixture uses a fake home and fake QMD; it does not start the real service.
Fix: update to the fixture that polls for both recorded startup calls with a bounded deadline, then checks their exact arguments and the absence of CASS calls. Keep the push gate enabled; changing the real QMD startup behavior is unnecessary.
Home directory is near quota despite install/scratch.sh running
Section titled “Home directory is near quota despite install/scratch.sh running”Symptom: df -H ~ reports the home filesystem close to full even though
.local, .cache, and the other DF_LINKS entries are already symlinked to
scratch space, and disk-dependent tools (e.g. an MCP server’s cache doctor
check) start failing on free-space thresholds.
Root cause: install/scratch.sh only migrates a fixed, named list of
cache-shaped directories (DF_LINKS, plus DF_CLAUDE_LINKS/DF_CODEX_LINKS
one level under the chezmoi-managed ~/.claude/~/.codex). Two other classes
of growth are not covered automatically:
- Ad hoc project/output directories created directly under
~(build trees, bug-repro dumps, scratch clones) —install/dirs.shalready provisions~/dev,~/bones, and~/miscas scratch-backed symlinks for exactly this, but nothing enforces using them; a directory created as~/<name>instead of~/misc/<name>sits on the small quota indefinitely. - New subdirectories an app adds after the
DF_CODEX_LINKS/DF_CONFIG_LINKSdefault list was written — e.g. Codex’sdb-backups/(pre-migration SQLite schema-upgrade snapshots) grew unnoticed under~/.codexfor months because the migration list only knew aboutbackups/, a similarly-named but different directory.
Confirm: du -sh --exclude=.local -- ~/.[!.]* ~/* 2>/dev/null | sort -rh | head -20
to find what’s actually large; for a chezmoi-managed dir like ~/.claude or
~/.codex, run du -sh ~/.codex/* ~/.codex/.[!.]* one level down, since the
directory itself can’t be a symlink (see the “never symlink a chezmoi-managed
directory” gotcha in .claude/rules/install-scripts.md).
Fix: for case 1, move the stray directory under ~/misc/<name> (or
~/dev, ~/bones) and update anything that referenced the old path; going
forward, create new project/output directories there directly rather than in
~. For case 2, add the missing subdir name to the relevant _DEFAULT_*_LINKS
list in install/scratch.sh and rerun it — Codex/Claude Code must not be
running at the time, since scratch.sh refuses to migrate ~/.codex while a
process holds its files open (a live app-server can otherwise keep writing to
an unlinked inode across the migration).
Remote Codex repeatedly warns about MCP authentication
Section titled “Remote Codex repeatedly warns about MCP authentication”Symptom. Startup warnings reappear during ongoing work or after a client
reconnect. Remote logs report failed to read OAuth tokens from keyring or
refusing file fallback, even though .credentials.json exists.
Root cause. Automatic OAuth storage selection can resolve a desktop keyring
that the headless app-server cannot read reliably. Runtime MCP refresh and new
agent initialization surface the connection failures again. Separately,
GitHub’s GH_TOKEN shell-function export is absent when an app starts the Codex
binary directly. Neither condition is fixed by a longer startup timeout.
Confirm and fix. Inspect the active CODEX_HOME, server startup errors, and
credential presence without printing secrets. Update dotfiles and run
bash ~/dotfiles/install/codex.sh sync-config on each remote host. Linux sync
selects file OAuth storage and GitHub uses its connection-time credential
helper. Restart affected clients when idle. Authenticate a missing server with
codex mcp login <server> using the SSH callback tunnel,
then verify a read-only operation. Keep existing credential files private; do not
copy rotating OAuth credentials between hosts or delete them as a reset.
An untrusted-project warning naming the legacy ~/.codex is a separate config
layer warning after moving CODEX_HOME. Inspect the effective config and actual
project root. Do not trust an entire home directory just to silence it. The
warning does not mean that the active host-local global config was disabled.
Codex MCP login fails with EOF parsing .credentials.json
Section titled “Codex MCP login fails with EOF parsing .credentials.json”Symptom. HTTP MCP servers report Unknown authentication and login fails
with failed to parse credentials file and EOF while parsing a value at line 1 column 0. The active $CODEX_HOME/.credentials.json is empty. A second
login can separately report Address in use when the first still owns the
configured OAuth callback port.
Root cause. An empty credential file is invalid JSON, so Codex cannot read
the store to update it. Disk exhaustion can leave a credential write
incomplete. On Blackwell, inspected September 23, 2026, the ext4 credential
file’s modification time was September 16 at 04:58:24.871558 UTC; the Codex
logs_2.sqlite database contained 24 No space left on device errors in the
04:58:20–04:58:30 UTC window. This strongly implicates a failed write during
disk exhaustion; the logs do not identify the credential writer itself.
Freeing space afterward does not repair the file. The
Codex 0.154.0 credential writer
calls set_len(0) before write_all, so a failed replacement write can leave
an empty store. A healthcheck detects this damage; it cannot make upstream
writes atomic.
Confirm. Check the active directory, capacity, and callback listener without printing credentials. On Linux:
runtime_dir="${CODEX_HOME:-$HOME/.codex}"stat -c '%n bytes=%s modified=%y' "$runtime_dir/.credentials.json"df -h "$runtime_dir"df -i "$runtime_dir"ss -ltnp 'sport = :45231' # Blackwell; use the configured port on other hostsbash ~/dotfiles/install/codex.sh checkFix. Resolve capacity exhaustion first. End an outstanding login in its
original terminal before starting another. Preserve the damaged file privately;
if it is confirmed empty, replace it atomically with a mode-0600 file
containing {}. Do not overwrite a nonempty store or silently reset it during
bootstrap. Reauthenticate each affected server and verify a read-only tool
operation. A successful codex mcp list --json with not_logged_in confirms
that parsing is repaired, not that authentication is restored. Legacy NFS
credentials may contain stale rotating refresh tokens; their presence alone
does not establish a safe recovery copy.
Keep active SQLite state host-local; SQLite WAL does not support network filesystems. Relocating the runtime does not fix a failed credential writer; protect runtime storage from build/cache exhaustion and retain private, consistent backups. See Codex runtime storage.
Codex lifecycle hook times out after three seconds
Section titled “Codex lifecycle hook times out after three seconds”Symptom. Hook failed: hook timed out after 3s appears at a lifecycle event,
including after the model finishes a response. The managed command is
~/.local/bin/df-task hook.
Root cause. The deadline includes the shell runner, helper launcher, and
checkpoint write. Older launchers started Bash, which can read .bashrc under
SSH, then resolved host policy and CPU platform again despite an inherited
runtime. The host resolver also started a process for each policy variable.
Older checkpoint code ignored DF_STATE_ROOT, placing records on a shared home
filesystem despite a configured local state directory.
Fix. Update dotfiles and run bash ~/dotfiles/install/codex.sh sync-runtime.
The resolver inspects exported names without per-variable processes. The POSIX
hook launcher reuses an executable resolved Python runtime, and checkpoint
writes honor the host state root. Existing metadata remains readable during
migration. The three-second timeout and 200 ms lock-wait bound remain unchanged.
No daemon restart or prompt replay is needed for these helper changes.
Verify. Run the native hook regression test (bats tests/codex-hooks.bats).
It completes a real app-server turn using a local model fixture and checks its
lifecycle checkpoints. Codex normally provides a non-login environment shell;
$SHELL -lc is its fallback. A manual full login-shell timing can overstate the
normal hook cost. Inspect the active shell path and native hook completion
results before changing a timeout. Interrupt and SessionEnd are capped at
three seconds by Codex, and requesting a larger value also changes the effective
trust hash. See durable work.
Codex upgrade rejects a valid 0.163.0 package
Section titled “Codex upgrade rejects a valid 0.163.0 package”Symptom. install/codex.sh upgrade downloads
codex-package-aarch64-apple-darwin.tar.gz, then fails with is incomplete or reports the wrong version; keeping active runtime.
Root cause. Codex 0.163.0 removed its bundled patched zsh, but the installer
still required codex-resources/zsh/bin/zsh. This rejected an otherwise valid
package before activation. The old runtime stays active. This failure is
independent of sudo, App Store, and optional MCP authentication warnings.
Confirm. On macOS arm64, the official 0.163.0 archive was verified on
October 19, 2026 against its published SHA-256 digest; its manifest, CLI version,
and code-mode host passed validation, and only the obsolete shell requirement
failed. The release notes
document the shell removal. codex --version shows which runtime remains active.
Fix. With the updated installer, retry only the Codex step:
bash ~/dotfiles/install/codex.sh upgradecodex --versionThe installer accepts packages without patched zsh starting at 0.163.0 and
still requires it for older releases. Checksum, manifest, version, code-mode
host, ripgrep, and Linux bubblewrap checks remain enforced. Existing sessions
keep their current executable until restarted. See
native CLI installation
for exact-version selection and runtime paths.
Codex disappears after an npm upgrade on NFS
Section titled “Codex disappears after an npm upgrade on NFS”Symptom. codex resume reports command not found; npm install -g @openai/codex@latest fails with EBUSY while unlinking a .nfs… file.
Root cause. An active Codex process holds a file in the npm package tree
open. NFS retains that file under a temporary name; npm cannot remove it during
package replacement, and the failed replacement can leave a dangling CLI shim.
A shell function named codex still appears in which even when the executable
it delegates to is gone.
Fix. The managed installer now owns the native executable and stages and
verifies its replacement before atomic activation. Run bash ~/dotfiles/install/codex.sh install, then use its upgrade mode for updates.
Start a fresh login shell to remove the retired function and inspect type -a codex. Do not remove open .nfs… files or delete the package tree underneath
running clients. Restart affected app-server processes when interruption is
acceptable; no prompts or paused queues should be replayed automatically.
Codex reconnect collides with an SSH OAuth tunnel
Section titled “Codex reconnect collides with an SSH OAuth tunnel”Symptom. Remote restart or reconnect logs report a callback port already in
use, sometimes alongside mux session refused.
Root cause. A persistent SSH connection owns a LocalForward, while a
separate app-managed SSH connection tries to bind that same port. MCP refresh
warnings after reconnect are a separate result of initializing enabled servers.
Fix. Keep callback forwarding on a dedicated workstation-oauth alias with
its own ControlPath; keep the ordinary app workstation alias free of callback
forwards. See MCP login from SSH. Existing
SSH masters retain their original forwards until closed. A task marked
interrupted also needs its own event timeline: received interrupt requests,
transport failures, and MCP authentication errors are distinct events.
Codex daemon bootstrap reports an unmanaged app server
Section titled “Codex daemon bootstrap reports an unmanaged app server”Symptom. codex app-server daemon bootstrap fails with app server is running but is not managed by codex app-server daemon.
Root cause. A directly launched codex app-server --listen unix:// already
owns the control socket for this CODEX_HOME, but has no daemon process record.
Desktop SSH connections can launch this way. In Codex 0.162.1, bootstrap refuses
to adopt that process; daemon stop and restart refuse it too. Management uses a
PID file, so the absence of a systemd unit is expected.
Confirm. Inspect the current CLI, server version, and exact process:
type -a codexcodex app-server daemon versionps -u "$(id -u)" -o pid,ppid,args | rg '[c]odex.*app-server'An unmanaged server can report status: running while omitting backend and
reporting managedCodexVersion: null. Check the process environment to confirm
it uses the same CODEX_HOME. Do not confuse the SSH app-server proxy process
with the server itself.
Fix. Prepare a complete native CLI package before stopping anything. A stale npm launcher may shadow the managed native binary; bootstrap also needs the complete package to create its daemon install. For example, to retain the observed 0.162.1 version:
CODEX_VERSION=rust-v0.162.1 bash ~/dotfiles/install/codex.sh installWait for active work to finish and disconnect the desktop’s SSH connection.
After checking its identity again, send SIGTERM to the exact unmanaged server
PID and wait for it to exit. This disconnects clients; avoid broad pkill
commands, deleting the socket, or fabricating a daemon PID file. Then run:
codex app-server daemon bootstrapcodex app-server daemon versioncodex app-server daemon startExpect bootstrapped, then running and alreadyRunning, with backend: pid
and the expected managed version. Reconnect the desktop afterward. Plain
bootstrap leaves remote control disabled; SSH access does not require enabling
it. See the 0.162.1 daemon lifecycle contract.
Codex cannot read sessions: No file descriptors available
Section titled “Codex cannot read sessions: No file descriptors available”Symptom. Session resume reports thread-store internal error and os error 24. MCP initialization, CA certificate reads, or helper launches may also fail.
Root cause. The app-server has exhausted its per-process soft descriptor limit. Each live MCP child can hold pipe and process-handle descriptors in the parent. A 1,024 limit can be too small for many simultaneously loaded sessions; the transcript itself need not be damaged.
Confirm. Inspect /proc/<daemon-pid>/limits and count entries under
/proc/<daemon-pid>/fd. Compare live process handles, pipes, sockets, and files
before diagnosing corruption or expired credentials.
Fix. Deploy the shared Linux resource-limit startup policy, start a fresh SSH login, and restart the affected Codex daemon when authorized. Verify its actual soft limit is 65,536 (or the hard cap), then read the affected session and exercise tool execution. Leave the hard limit unchanged. Raising capacity does not repair a missing code-mode-host binary; install the complete matching runtime for that error. Investigate continuing unbounded helper growth separately.