Troubleshooting

Look up issues by symptom. Each entry gives you symptom / likely causes / how to diagnose / how to fix. If your situation isn't listed, open an issue on GitHub.

Daemon can't connect to the server

Symptom: multica daemon's status command shows offline or connection refused; the server logs show no /api/daemon/register or /api/daemon/heartbeat requests. For how the daemon mechanism works, see Daemon and runtimes.

Likely causes:

1. MULTICA_SERVER_URL points at the wrong address — default is ws://localhost:8080/ws; self-host must change it to your server address
2. Network / firewall blocking — the daemon and server aren't on the same network, or outbound traffic is blocked
3. Token expired or invalid — you never ran multica login, or the PAT was revoked
4. Server rejected registration — the account you signed in with isn't in the target workspace (register returns 403)
5. DNS resolution failure — the hostname doesn't resolve on the daemon machine

How to diagnose:

multica daemon logs --lines 100    # look for daemon-side errors
echo $MULTICA_SERVER_URL          # confirm the address is set
curl -i http://<server-host>:8080/health   # hit the server directly
curl -i http://<server-host>:8080/readyz  # include DB + migration readiness
cat ~/.multica/config.json        # verify api_token exists
multica workspace list            # confirm you're a member of the target workspace

How to fix: address each cause above. The two most common fixes are changing MULTICA_SERVER_URL and restarting the daemon (multica daemon restart) and signing in again (multica logout && multica login).

Tasks stuck in queued

Symptom: after assigning an issue to an agent, the issue status flips to in_progress immediately, but a long time passes with no sign of agent execution on the page; multica daemon status shows the daemon online.

Likely causes (ordered by frequency):

1. Agent concurrency limit reached — this agent's max_concurrent_tasks (default 6) is fully occupied by other running tasks
2. Another task from the same agent is still running on the same issue — same agent × same issue is forced to run sequentially (prevents duplicate execution)
3. Agent has been archived — after archival, new tasks still enqueue but can't be claimed, and they time out after 5 minutes (code-issue G-01)
4. Daemon hasn't registered this runtime in the current workspace — restart the daemon or reselect the runtime in the UI
5. Daemon disconnected — no heartbeat in the last 45 seconds. daemon status reporting online may reflect a very recent disconnect

How to diagnose:

multica daemon status --output json       # runtime list + last_seen_at
multica agent list                         # check agent archived state
multica issue show <issue-id>             # inspect task history

On the server side (self-host), grep for "no_tasks" / "no_capacity" to see the claim outcome.

How to fix:

• Concurrency full → wait for running tasks to finish, or multica agent update <id> --max-concurrent-tasks 10 to raise the ceiling
• Same-issue serialization → wait for the previous task to finish, or reassign to a different agent
• Agent archived → multica agent restore <id>
• Runtime not registered → multica daemon restart, and the daemon will re-register

WebSocket can't connect

Symptom: the browser console logs WebSocket is closed; the page doesn't show real-time updates (task progress, comments, inbox), and a refresh is needed to see them; backend tasks still execute.

Likely causes:

1. Origin check failure — your frontend domain isn't in the server's CORS allowlist. The default allowlist only includes localhost:3000/5173/5174; self-hosting on the public internet requires FRONTEND_ORIGIN
2. Protocol mismatch — frontend on https:// needs wss://; HTTP uses ws://
3. Reverse proxy doesn't enable WebSocket upgrade — Nginx / Envoy / HAProxy don't forward the Upgrade header by default
4. JWT cookie expired or missing — no re-sign-in after the 30-day expiry

How to diagnose:

• Browser DevTools → Network → filter by "WS" and check connection state and status code
• Grep server logs for "rejected origin" / "websocket" — an origin issue spells itself out
• curl -i http://<server-host>:8080/ws should return 101 Switching Protocols (with the Upgrade header)

How to fix:

• Wrong origin → set FRONTEND_ORIGIN=https://multica.yourdomain.com in the server's .env (or comma-separated CORS_ALLOWED_ORIGINS) and restart the server
• Protocol mismatch → make sure FRONTEND_ORIGIN's protocol matches the frontend's
• Reverse proxy → in Nginx, add proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade";
• Cookie expired → refresh the page and sign in again

Emails not received

Symptom: after submitting an email during sign-in or invite acceptance, neither the inbox nor the spam folder has the verification code.

Likely causes:

1. RESEND_API_KEY not set — the server silently falls back and writes the code to its own stdout without error. Easy to trip over in production
2. Resend API key invalid / out of quota — server logs show "failed to send verification code"
3. RESEND_FROM_EMAIL's domain not verified in Resend — Resend refuses to send
4. Email was sent but flagged as spam by the recipient's ISP — check the Resend dashboard and the spam folder

How to diagnose:

• Grep server logs for "[DEV] Verification code for" — if present, Resend isn't configured and the code was written to stdout
• Resend dashboard → Emails for send history
• Confirm RESEND_FROM_EMAIL's domain appears in the Resend console's "Verified Domains" list

How to fix:

• Missing API key → follow Sign-in and signup configuration → How email works to configure and restart the server
• Domain not verified → run the DNS verification flow in the Resend console (add SPF / DKIM records)
• In an emergency (internal testing) → copy the code printed under [DEV] from the server logs

Fixed local test code doesn't work

Symptom: on a self-hosted instance, you try to sign in with a fixed local test code such as 888888 and it's rejected with invalid or expired code.

Likely causes (mutually exclusive):

1. MULTICA_DEV_VERIFICATION_CODE is empty — fixed codes are disabled by default
2. APP_ENV=production — this is the correct production configuration; fixed local test codes are ignored in production
3. The configured code is not 6 digits — the shortcut only accepts a 6-digit value

How to diagnose:

cat .env | grep -E 'APP_ENV|MULTICA_DEV_VERIFICATION_CODE'
docker exec <container> env | grep -E 'APP_ENV|MULTICA_DEV_VERIFICATION_CODE'

Check your inbox (including spam) for the real verification code.

How to fix:

• In production, leave MULTICA_DEV_VERIFICATION_CODE empty — configure Resend and use real codes
• For local development or internal testing, either copy the generated code from server logs or set APP_ENV=development plus MULTICA_DEV_VERIFICATION_CODE=888888 — never enable a fixed code on a public instance (see Sign-in and signup configuration → Fixed local testing codes)

Usage dashboard stays at zero

Symptom: agents complete tasks, raw token usage is written to the database, but Settings → Usage and Settings → Runtime show 0 input / output / cost across the board. This is silent — there is no error in the backend logs.

Likely causes:

1. rollup_task_usage_hourly() is never scheduled — the Usage / Runtime dashboards read from the derived task_usage_hourly table, which is populated by that function. The bundled pgvector/pgvector:pg17 image does not include pg_cron, and the backend does not run the rollup in-process either. On a fresh self-host install with no external scheduler, this is the default state.
2. pg_cron is installed but pointing at the wrong database — pg_cron.database_name defaults to postgres; if your Multica database has a different name, the scheduled job never sees rollup_task_usage_hourly().
3. The scheduler is running but the rollup is silently erroring — e.g. wrong DB role / search_path inside the cron entry.

How to diagnose:

-- Confirm raw events exist but the hourly table is empty.
SELECT count(*) AS raw_rows FROM task_usage;
SELECT count(*) AS hourly_rows FROM task_usage_hourly;

-- Confirm pg_cron is (or isn't) available.
SELECT * FROM pg_available_extensions WHERE name = 'pg_cron';
SHOW shared_preload_libraries;

-- If pg_cron is installed, check the schedule + last run.
SELECT jobname, schedule, database, active FROM cron.job;
SELECT jobname, status, return_message, start_time, end_time
  FROM cron.job_run_details ORDER BY start_time DESC LIMIT 10;

-- Watermark — if this is 1970-01-01, the rollup has never run.
SELECT watermark_at FROM task_usage_hourly_rollup_state;

How to fix:

• Call the rollup once by hand to confirm it works: SELECT rollup_task_usage_hourly(); — refresh the dashboard; if numbers appear, the only missing piece is a scheduler.
• Pick one of the supported paths from Self-host quickstart → Schedule the usage rollup: external cron / systemd-timer / Kubernetes CronJob, or swap Postgres for an image with pg_cron.
• If you already have history that pre-dates the schedule, run backfill_task_usage_hourly inside the backend container to seed buckets before the watermark.

Migration 103 fails with refusing to drop legacy daily rollups

Symptom: upgrading from v0.3.4 to v0.3.5+, the backend container fails to start (or migrate up aborts) with:

ERROR: refusing to drop legacy daily rollups:
  task_usage_hourly_rollup_state.watermark_at (1970-01-01 ...) trails
  task_usage latest event (...) by more than 01:00:00 — backfill is
  incomplete or pg_cron is not running. Run cmd/backfill_task_usage_hourly
  (and let pg_cron catch up) before re-running migrate

Likely cause: this is migration 103's fail-closed guard. It refuses to drop the legacy daily rollups until task_usage_hourly has caught up with raw task_usage. The guard fires whenever existing rows are present and the rollup watermark still sits at the epoch — i.e. nothing has rolled history into the hourly table yet.

How to fix:

1. Run the backfill against the same database (idempotent, safe to interrupt, safe to re-run):

   # Docker Compose
   docker compose -f docker-compose.selfhost.yml exec backend \
     ./backfill_task_usage_hourly --sleep-between-slices=2s
   
   # Kubernetes
   kubectl -n multica exec deploy/multica-backend -- \
     ./backfill_task_usage_hourly --sleep-between-slices=2s

2. Re-run the upgrade — restarting the backend container is enough, migrations run on startup. The guard now sees a current watermark and lets 103 apply.

3. Set up an ongoing rollup schedule (cron / pg_cron) so the watermark keeps advancing — see Self-host quickstart → Schedule the usage rollup.

--sleep-between-slices=2s is a polite default on production databases with years of history. Use --months-back N --force-partial if you only want to keep the last N months and are willing to permanently abandon older buckets.

Port conflicts

Symptom: multica server or multica daemon start fails with address already in use.

Likely causes:

1. Server port taken (default 8080)
2. Daemon health port taken (default 19514, offset by a hash per profile)
3. Web dev server port conflict (3000 / 5173)
4. Insufficient privileges for the port (binding a privileged port < 1024 requires sudo)

How to diagnose:

lsof -i :8080        # macOS / Linux
netstat -ano | findstr :8080    # Windows

How to fix:

• Kill the conflicting process (kill -9 <PID>), or change ports via PORT=9000
• To use 80 / 443 → don't bind directly; put a reverse proxy (Nginx / Caddy) in front, forwarding to a high port

Where to find logs

For more detailed daemon logs, move it from background to foreground: multica daemon stop && multica daemon start --foreground.