receipt-manager/.claude/projects/-home-kamaji/memory/MEMORY.md

Project Memory

Receipt Manager (separate from NextSnap)

• Source: compute1:~/ (receipts.html + server.py), Gitea repo kamaji/receipt-manager
• Container: receipt-manager on docker1 (~/receipt-manager/), port 8082
• Deploy: scp receipts.html server.py docker1:~/receipt-manager/ && ssh docker1 'cd ~/receipt-manager && docker-compose down && docker-compose up -d --build'
• Stack: Python stdlib HTTP server, vanilla JS frontend, Nextcloud WebDAV for storage
• NOT part of NextSnap — separate app, separate container, separate repo

NextSnap Architecture

• Flask PWA on docker1 (~/nextsnap), Docker container
• Deploy: ssh docker1 'cd ~/nextsnap && docker-compose down && docker-compose up -d --build'
• Files accessed via SSH (scp to deploy, ssh to run commands)
• Frontend: Vanilla JS, Dexie.js (IndexedDB), Service Worker for offline-first
• Reverse proxy: nginx-reverse host, config at /etc/nginx/sites-available/reverse-proxy.conf

Infrastructure - nginx-reverse

• VM: on host compute1 (migrated from console, 2026-02-22)
• Disk: /1TB/vm/nginx-reverse.qcow2 on compute1
• IP: 192.168.128.19
• Sudo: %sudo group has NOPASSWD:ALL; kamaji is in sudo group
• SSH from compute1: Uses id_rsa key (added to authorized_keys)
• Certbot: Webroot at /var/www/letsencrypt, auto-renewal enabled
• Services: nginx (reverse proxy), iperf3 (speed test), WireGuard (wg0 — direct VPN server)
• WireGuard: Direct VPN server on port 51820, clients connect via 104.52.199.76:51820
• Autostart enabled on compute1

Infrastructure - docker1

• VM: "docker1" on host compute1 (migrated from "docker-box" on console, 2026-02-22)
• Disk: /1TB/vm/docker-box.qcow2 on compute1
• RAM: 8 GB, vCPUs: 8
• IP: 192.168.128.5
• Root disk: 24 GB /dev/vda1 — tight with Docker; monitor usage
• NFS mount: console:/Storage → /Storage (in fstab with _netdev)
• Nextcloud data: symlinked /opt/nextcloud/data → /Storage/nextcloud/data
• Nextcloud compose: /opt/nextcloud/docker-compose.yml (app + redis + mariadb)
• Nextcloud Office: Built-in CODE (richdocumentscode AppImage), runs COOLWSD on port 9983 inside container
• Autostart enabled on compute1

Nextcloud Backup (legacy)

• Script: docker1:~/nextcloud-backup.sh — hourly cron, predates snapback
• Note: Snapback now handles the same backups (nextcloud-data, nextcloud-app, nextcloud-db jobs)
• SSH from docker1→compute1 uses kamaji's ed25519 key; sudo rsync needs explicit SSH opts

Two User Classes

• Admin: NC admin login (username+password), full access including admin panel
• Tech: Field workers, login with username+PIN, access to capture/queue/browser only
• Tech users stored in data/tech_users.json (Docker named volume app_data at /app/data)
• PIN hashed with werkzeug.security; NC password stored plaintext (admin-visible)
• Tech user creation auto-provisions NC account via OCS API
• Session stores NC password (base64) for transparent API calls
• session['user_type'] = 'admin' | 'tech'; session['is_admin'] for nav guards
• Admin panel: tappable user list → detail modal (enable/disable, reset PIN, reset NC password, delete)
• Login page: tabbed UI — "Tech Login" (default) and "Admin Login"

Key Bugs & Lessons

Service Worker Caching (CRITICAL)

• SW was cache-first for ALL routes including HTML pages → stale code served forever
• Fix: Only /static/ uses cache-first; pages + API use network-first
• Always bump SW cache versions when changing any static asset
• Even after SW bump, user needs one hard refresh to pick up new SW

DOM Element Destroyed by innerHTML (queue.html)

• getElementById('empty-state') returned null after innerHTML = '' destroyed it
• Caused silent crash on every subsequent loadQueue() call
• Counters still worked because updateStats() ran independently
• Lesson: Never reference DOM elements inside a container you clear with innerHTML

iOS Safari Blob Eviction

• Blobs stored in IndexedDB are file-backed references that iOS can evict
• instanceof Blob and .size checks pass but reading data throws "Load failed"
• Fix: Store as ArrayBuffer (inline bytes) + navigator.storage.persist()
• Storage.getBlob(photo) converts ArrayBuffer→Blob at read time

iOS Safari Fetch "Load Failed"

• Generic error for any network failure (backgrounding, connectivity)
• Upload POST fails client-side, never reaches server
• Show friendly messages, hide transient errors (< 3 retries) from UI

Nextcloud OCS API 412 Error

• All OCS endpoints require OCS-APIRequest: true header (CSRF protection)
• Without it → 412 Precondition Failed

Docker Volume Permissions

• Named volumes mounted to dirs created in Dockerfile inherit ownership from image
• If volume already exists with root ownership, must docker-compose down -v to recreate
• Container runs as nextsnap (uid 1000) — /app/data must be owned by nextsnap

iOS Safari inputmode="numeric" + type="password"

• Combining these causes browser validation error "string did not match expected pattern"
• Fix: Use type="text" with inputmode="numeric" and validate in JS

Nextcloud CODE "Document loading failed"

• COOLWSD (CODE server) stores temp files in /tmp jails inside the container
• Disk full → "Low disk space" / "Out of storage" in COOLWSD logs → 400/408 on proxy.php
• Memory exhaustion → inconsistent loading (works sometimes), CODE processes killed/timed out
• Fix: Free disk space, increase VM RAM; COOLWSD log at /tmp/coolwsd.*/coolwsd.log inside container
• After fixing, restart container so COOLWSD gets a clean start

MikroTik (cornerLot)

• Model: RB750Gr3, RouterOS 6.48.2, identity cornerLot
• IP: 192.168.88.1 (clients), 192.168.128.1 (servers)
• SSH: Port 65523, user kamaji
• SSH keys: RouterOS 6.x disables password auth once a key is imported; key must work or user is locked out (WinBox recovery only)
• SSH from compute1: Dedicated key ~/.ssh/mikrotik_rsa, SSH config alias mikrotik; requires ssh-rsa algo + DEFAULT:SHA1 crypto policy (RHEL 9 blocks SHA-1 by default)
• NAT rules: Working dstnat rules use dst-address=104.52.199.76, NOT in-interface=ether1
• Hairpin NAT: srcnat masquerade for 192.168.0.0/16 at rule 2
• Forward chain: action=accept with no conditions at rule 3 (accepts all)

Podman SELinux on compute1

• Volume mounts require :z flag (e.g. -v /path:/mount:ro,z)
• Without it: PermissionError: [Errno 13] Permission denied inside container

NFS Mount Not in fstab

• /Storage was NFS-mounted manually (console:/Storage) but not in fstab
• After VM reboot, Docker couldn't start nextcloud-app: broken symlink to /Storage/nextcloud/data
• Fix: Added console:/Storage /Storage nfs defaults,_netdev 0 0 to fstab

Infrastructure - console

• IP: 192.168.88.5 (servers subnet)
• Role: KVM host (libvirt, Debian), NFS server (/Storage)
• VMs remaining: devtest1 (autostart), pihole2, unifi, win10-signed_to_outlook.com
• Migrated off (2026-02-22): docker-box→docker1, docker2, nginx-reverse, pihole, jellyfin2, git1 — all to compute1
• win10-signed_to_outlook.com: moved from compute1 (2026-02-23), /var/lib/libvirt/images/win10.qcow2, 8 GB RAM, 4 vCPUs, autostart enabled, Hyper-V enlightenments enabled (vpindex, synic, stimer, frequencies, reenlightenment, tlbflush — dropped idle CPU from ~32% to ~5%)

Gitea (git1)

• VM: on host compute1 (migrated from console, 2026-02-22)
• Disk: /1TB/vm/git1.qcow2 on compute1
• IP: 192.168.128.23
• URL: https://git.sdanywhere.com/
• Version: 1.25.4
• Config: /etc/gitea/app.ini, runs as kamaji user
• Repo root: /srv/git/
• SSH push: Use kamaji@git1:kamaji/<repo>.git format (Gitea SSH wrapper)
• Do NOT use bare repo paths (git1:/srv/git/...) — pre-receive hook rejects without Gitea env
• authorized_keys: Plain keys (docker1, nginx-reverse, dns1) for shell access; RSA key has Gitea command= wrapper for git operations
• Shell access to git1: Use docker1 (ed25519 key), not this machine (RSA key goes through Gitea wrapper)
• API: Available at https://git.sdanywhere.com/api/v1/, generate tokens via gitea --config /etc/gitea/app.ini admin user generate-access-token
• Repos: kamaji/receipt-manager (Receipt Manager), kamaji/snapback (backup system)

Snapback (compute1)

• Repo: kamaji/snapback on Gitea
• Deployed to: compute1:~/snapback/
• Backup root: /1TB/backups/ on compute1
• Cron: 0 * * * * runs all jobs hourly
• SSH key: id_rsa on compute1 (not ed25519)
• Log: ~/snapback/snapback.log (not /var/log — not writable by kamaji)
• ntfy: Sends to https://ntfy.sdanywhere.com topic backups
• Retention: 24 hourly, 7 daily, 4 weekly
• Jobs: nextcloud-data (rsync), nextcloud-app (rsync+sudo), nextcloud-db (db_dump), gitea-repos (rsync), gitea-data (rsync), gitea-db (db_dump), nginx-config (rsync), nginx-certs (rsync+sudo), nginx-wireguard (rsync+sudo), mikrotik-config (db_dump, /export), mikrotik-backup (db_dump, local, binary .backup)
• Local jobs: local: true on db_dump runs command directly instead of via SSH
• MikroTik SSH: Dedicated key ~/.ssh/mikrotik_rsa on compute1; SSH config alias mikrotik (port 65523, user kamaji, ssh-rsa algo for RouterOS 6.x); requires DEFAULT:SHA1 crypto policy on compute1 (RHEL 9)
• MikroTik helper: ~/snapback/mikrotik-backup.sh — creates backup on router, SCPs it off, outputs to stdout

Snapback Web Browser (compute1)

• Container: snapback-web via podman on compute1, port 8082
• Public URL: https://backups.sdanywhere.com (via nginx-reverse proxy)
• Auth: HTTP Basic Auth (BROWSER_USER/BROWSER_PASSWORD env vars)
• Source: compute1:~/snapback/web/ (Flask + gunicorn, vanilla JS SPA)
• Mounts: /1TB/backups:/backups:ro,z, ~/.ssh:/ssh:ro,z, config.yml:/app/config.yml:ro,z
• Features: Browse snapshots, preview text/images, download files/zips, restore via rsync
• SSH key handling: Copies /ssh/* to /tmp/.ssh/ with correct permissions at startup
• Restore: Only rsync-type jobs; builds rsync command from config job metadata
• SELinux: Podman on compute1 requires :z flag on volume mounts
• Deploy: ssh compute1 'podman build -t snapback-web ~/snapback/web/ && podman stop snapback-web && podman rm snapback-web && podman run -d --name snapback-web -p 8082:8082 -v /1TB/backups:/backups:ro,z -v /home/kamaji/.ssh:/ssh:ro,z -v /home/kamaji/snapback/config.yml:/app/config.yml:ro,z -e BROWSER_USER=kamaji -e BROWSER_PASSWORD=<pw> --restart unless-stopped snapback-web'

Sysmon (compute1)

• Source: devtest1:~/sysmon/, deployed to compute1:~/sysmon/
• Dashboard: Podman container sysmon on compute1, port 8083 (pure aggregator, no /proc)
• Stack: Flask + gunicorn, vanilla JS frontend, dark theme
• Architecture: Single codebase, two modes controlled by SYSMON_SERVERS env var
  • Agent mode (no env var): Reads local /proc + sudo virsh for VMs, serves /api/stats
  • Dashboard mode (env var set): Pure aggregator, no /proc or CPU sampler; serves UI at /
• Config: SYSMON_SERVERS="compute1:http://host.containers.internal:8084,console:http://192.168.88.5:8083"
• Agents: compute1 (port 8084, systemd), console (port 8083, systemd) — both bare installs, python3 -m gunicorn
• UI: Hash-based routing — dashboard summary cards (#) + server detail view (#<name>) with CPU grid, memory, load, uptime, VM table
• VM base info: sudo virsh dominfo per VM (30s cache) — name, state, vcpus, memory, autostart
• VM CPU %: sudo virsh domstats --cpu-total --balloon (5s cache), delta tracking for CPU %
• VM memory: From balloon stats (available - unused); Windows VMs lack guest-side stats, show allocated only
• VM disk: sudo virsh guestinfo --filesystem per VM (30s cache, async background refresh to avoid blocking)
• Dashboard fetch timeout: 8s (cold-cache agent responses can take 2-3s due to dominfo calls)
• Deploy dashboard: scp ~/sysmon/{app.py,templates/index.html,run.sh,Dockerfile} compute1:~/sysmon/ && ssh compute1 'mkdir -p ~/sysmon/templates && mv ~/sysmon/index.html ~/sysmon/templates/ && ~/sysmon/run.sh'
• Deploy agent: ~/sysmon/deploy-agent.sh <host> [port] (default port 8083)
• SELinux note: compute1 (RHEL 9) blocks ~/.local/bin/gunicorn from systemd; use python3 -m gunicorn instead
• Adding servers: Add name:url to SYSMON_SERVERS, deploy agent on new host, rebuild dashboard

iperf3 (nginx-reverse)

• Service: iperf3.service (systemd, installed via apt)
• Port: 5201 (TCP+UDP)
• DNS: speed.sdanywhere.com
• MikroTik NAT: dstnat rules use dst-address=104.52.199.76 (NOT in-interface=ether1 — WAN traffic doesn't match ether1)
• Hairpin NAT: srcnat masquerade for 192.168.0.0/16 (rule 2 in NAT table)

Infrastructure - Pi-hole HA

• pihole1 (primary): 192.168.128.3 on compute1, /1TB/vm/pihole.qcow2, 2 GB RAM, 2 vCPUs
• pihole2 (replica): 192.168.128.2 on console, /var/lib/libvirt/images/pihole2.qcow2, 1 GB RAM, 2 vCPUs
• Both running Pi-hole v6.4, autostart enabled
• Sync: Nebula Sync container (nebula-sync) on docker1, full sync every 30 min (pihole1 → pihole2)
• DHCP DNS: MikroTik hands out both 192.168.128.3,192.168.128.2
• Cross-host redundancy: pihole1 on compute1, pihole2 on console
• Admin password: both set to same password via sudo pihole setpassword
• pihole2 SSH: kamaji user, ed25519 key generated on pihole2, can SSH to pihole1

Infrastructure - unifi

• VM: on host console (moved back from compute1, 2026-02-23)
• Disk: /var/lib/libvirt/images/unifi.qcow2 on console
• IP: 192.168.128.6
• RAM: 1 GB, vCPUs: 2
• Services: UniFi Network Controller
• Autostart enabled on console

Infrastructure - jellyfin2

• VM: on host compute1 (migrated from console, 2026-02-22)
• Disk: /1TB/vm/jellyfin2.qcow2 on compute1
• IP: 192.168.128.4
• RAM: 1 GB, vCPUs: 2
• NFS mount: console:/Storage → /Storage (in fstab with _netdev)
• Services: Jellyfin media server (port 8096)
• Autostart enabled on compute1

Infrastructure - zoneminder

• VM: on host compute1, cloned from debian12-template (2026-02-23)
• Disk: /1TB/vm/zoneminder.qcow2 on compute1
• IP: 192.168.128.20
• RAM: 4 GB, vCPUs: 4
• OS: Debian 12 Bookworm
• Services: ZoneMinder 1.36.33 (CCTV/surveillance), Apache2, MariaDB
• DB: zm database, user zmuser/zmpass
• Web UI: http://192.168.128.20/zm/
• Config: /etc/zm/zm.conf (group www-data), overrides in /etc/zm/conf.d/
• ffmpeg: Configured in /etc/zm/conf.d/03-ffmpeg.conf
• Sudo: kamaji has NOPASSWD:ALL via /etc/sudoers.d/kamaji
• Autostart enabled on compute1

Infrastructure - compute1 (VM host)

• IP: 192.168.88.9 (servers subnet), also reachable as compute1
• OS: RHEL 9 (Rocky), QEMU at /usr/libexec/qemu-kvm
• libvirt: Machine type q35 (RHEL variant), VNC graphics (no SPICE support)
• Networking: Bridge br0 on enp4s0f1 for VM bridged networking
• VM storage: /1TB/vm/ on 932 GB local disk
• VMs hosted: docker1, docker2, nginx-reverse, pihole, jellyfin2, git1, zoneminder, macos/SheepShaver, dos/DOSBox
• Migration notes: Adapted from console (Debian): emulator path, machine type, macvtap→bridge, SPICE→VNC

Retro VMs

See retro-vms.md — DOSBox (192.168.128.31) and SheepShaver/Mac OS 9 (192.168.128.30) on compute1, via Guacamole

Guacamole (docker1)

• Containers: guacamole, guacd, guac-mysql on docker1
• Compose: docker1:~/guacamole/docker-compose.yml
• DB: MySQL 8.0, user guacuser / guacpassword, database guacamole
• Port: 8080 (guacamole webapp)
• Auth: MySQL + LDAP (OpenLDAP, ou=users,dc=sdanywhere,dc=com)
• kamaji entity_id: 2

VPN (Direct WireGuard)

• Server: nginx-reverse, 10.10.10.1/24 (wg0), ListenPort 51820
• Public endpoint: 104.52.199.76:51820 (MikroTik dstnat → 192.168.128.19:51820)
• Config: /etc/wireguard/wg0.conf on nginx-reverse, wg-quick@wg0 enabled
• Split tunnel: Clients route 192.168.128.0/24, 192.168.88.0/24, 10.10.10.0/24 through VPN
• DNS: 192.168.128.3 (Pi-hole primary), 192.168.128.2 (Pi-hole replica) — no public fallback
• Clients: iPhone at 10.10.10.2/32, MikroTik travel router at 10.10.10.3/32, Windows laptop at 10.10.10.4/32, Linux client at 10.10.10.5/32
• Routing: No masquerade — MikroTik has static route 10.10.10.0/24 via 192.168.128.19 so LAN devices can reach VPN clients directly; bidirectional FORWARD rules; MSS clamping for TCP on both enp1s0 and wg0
• MikroTik notrack: Raw prerouting rule dst-address=10.10.10.0/24 action=notrack — required because nginx-reverse delivers VPN→LAN packets directly on L2 (same subnet), creating asymmetric routing; without notrack, MikroTik conntrack marks reply packets as invalid and drops them intermittently
• sdanywhere.com: VPN services disabled (2026-02-22), still active as web server (64.227.104.26, Debian 12, login root)

ntfy (docker2)

• VM: on host compute1 (migrated from console, 2026-02-22)
• Disk: /1TB/vm/docker2.qcow2 on compute1
• IP: 192.168.128.8
• Container: binwiederhier/ntfy on docker2, port 8080; uses docker compose v2
• Deploy: ssh docker2 'cd ~/snapback && docker compose down && docker compose up -d'
• URL: https://ntfy.sdanywhere.com, auth deny-all, admin user kamaji
• Config: docker2:~/snapback/ntfy/server.yml, upstream relay to ntfy.sh for mobile push
• Topic: backups (requires auth)
• Autostart enabled on compute1

NextSnap File Locations

See nextsnap-files.md