From 63a285689a3f8807e3d589604066b0b0328add20 Mon Sep 17 00:00:00 2001 From: mrwhiski Date: Mon, 5 Oct 2026 22:14:45 +0200 Subject: [PATCH] Adds paperless setup --- docker/paperless/.env.example | 3 + docker/paperless/README.md | 259 ++++++++++++++++++++++++++++++++++ docker/paperless/compose.yaml | 60 ++++++++ docs/status.md | 17 ++- docs/todo.md | 9 +- 5 files changed, 339 insertions(+), 9 deletions(-) create mode 100644 docker/paperless/.env.example create mode 100644 docker/paperless/README.md create mode 100644 docker/paperless/compose.yaml diff --git a/docker/paperless/.env.example b/docker/paperless/.env.example new file mode 100644 index 0000000..422999e --- /dev/null +++ b/docker/paperless/.env.example @@ -0,0 +1,3 @@ +# Copy to .env on the OptiPlex and fill in. Never commit .env. +# Generate with: openssl rand -base64 48 +PAPERLESS_SECRET_KEY= diff --git a/docker/paperless/README.md b/docker/paperless/README.md new file mode 100644 index 0000000..c6cadb1 --- /dev/null +++ b/docker/paperless/README.md @@ -0,0 +1,259 @@ +# Paperless-ngx + +Document archive with OCR. Runs on the OptiPlex homeserver (`192.168.30.20`) +in `~/paperless`, reachable only as `https://paperless.home.staffenberger.at`, +at home or through the WireGuard VPN. + +## How the pieces fit + +``` +Brother scanner (IoT) ──SMB──▶ NAS share `scans` ──SMB mount──▶ Paperless consume + │ +NAS share `paperless-export` ◀──nightly document_exporter────────────┘ + │ + └──▶ Hyper Backup to the USB drive (manual, every two weeks) +``` + +- **`scans`** is only an inbox. Paperless polls it every 60 s, imports + each file, and **deletes it from the share** afterwards. The documents + themselves live in `./media` on the OptiPlex. +- **`paperless-export`** gets a full export every night: originals, archived + PDFs and a manifest with tags, correspondents and so on. `document_importer` + can rebuild a fresh instance from it, so this export is the backup. No + extra copy of `./data` or `./media` is needed. +- The database (SQLite) stays on the local disk. A database on a network + share is asking for corruption. + +**Sorting** works through matching rules on tags, correspondents and document +types. The default mode, *Auto*, learns from the documents you've already +assigned. So at first you assign by hand; after a few dozen documents it +starts suggesting assignments. It doesn't sort anything correctly out of the box. + +## NAS (DSM) + +1. **Two shared folders:** `scans` and `paperless-export`. Hide them from + "My Network Places", **no recycle bin on `scans`**. Every imported file + would end up there, and worse: DSM restricts `#recycle` to admins, so + Paperless's recursive watcher crashes on it with `Permission denied + (os error 13)` (hit 2026-10-05). If `#recycle` exists, delete it in + File Station after turning the recycle bin off. +2. **Two users**, each with no other permissions and no app access besides SMB: + - `scanner`: read/write on `scans` only. Its password is stored in the + Brother scanner. + - `paperless`: read/write on `scans` and `paperless-export`. + + Gotchas (set up 2026-10-05): + - The `users` group allows almost every application by default, so new + users start with full app access. Set **Deny** on everything except + **SMB** in each user's Applications tab, and check the Preview column. + Don't deny SMB too, or the user can't log in at all. + - In the shared folder's permissions (Local groups view), leave the + `users` group with **no box ticked**. Its Preview then shows "No Access", + which only means "grants nothing". Ticking **No access** would be an + explicit deny, which always wins in DSM, and would lock out `scanner` + and `paperless`, who are both in `users`. +3. **Hyper Backup:** add `paperless-export` to the existing task's sources. + New shared folders are **not** picked up automatically, even though "all + shared folders" was selected when the task was created. `scans` doesn't + need backing up. + +## Scanner (Brother) → NAS + +The scanner sits on IoT, and IoT Outwards (`IoT → !IoT` deny) blocks it from the +NAS. It needs one narrow exception **above IoT Outwards** (in place since 2026-10-05 as "Printer to NAS SMB"): + +| Policy | Protocol | Source | Destination | +|---|---|---|---| +| Allow | TCP | IP group: scanner's reserved IP | IP-Port group: `192.168.30.10` port `445` | + +Groups are under Settings → Profiles → Groups; the rule goes in the Gateway +ACL, LAN→LAN. (Check the menu paths in Controller 6.3.) + +Then in the scanner's web interface (Web Based Management): Scan → Scan to +FTP/SFTP/Network/SharePoint → set a profile to **Network**, host +`192.168.30.10`, store directory `scans`, user `scanner`, authentication +NTLMv2, file type PDF, 300 dpi. Exact menu names depend on the model. + +The scanner is a **Brother MFC-L3750CDW**. **Working since 2026-10-05** over +SMB with DSM's default minimum of SMB2: Profile 1 = Network, +`\\192.168.30.10\scans`, NTLMv2, user `scanner`, firmware MAIN +`ZJ2606120536` (already the latest). + +- **No SFTP on this model** (the menu is "Scan to FTP/Network/SharePoint"). + If SMB ever breaks, **don't turn SMB1 on in DSM**. Fallback: plain FTP, + port 21 + passive range in the ACL. The `scanner` password would then + cross the network in plain text, which is tolerable only because that user + can do nothing except write to `scans`. +- **Chrome autofill on the printer's web interface:** Chrome filled its saved + printer login into the profile's username/password fields and then + reported a "breached password" on Submit. That was most likely the old + default admin password. Paste the `scanner` login from Enpass into cleared + fields, and don't let Chrome save the printer's passwords. +- Time on the scanner (SNTP) has to be right, or authentication can fail. +- Optional: one profile per person, saving into `scans/David` and + `scans/`. Because of `CONSUMER_SUBDIRS_AS_TAGS`, the documents + then come in tagged with the person's name. + +## OptiPlex + +The shares are mounted on the host with **systemd automount**, not at boot: +after a power cut the OptiPlex comes up long before the NAS, and a normal +mount would just fail and stay that way. The automount connects on first +access and tries again later if the NAS isn't up yet. + +1. `sudo apt install cifs-utils` +2. Credentials file, readable by root only: + + ```bash + sudo install -m 600 /dev/null /etc/nas-paperless.cred + sudoedit /etc/nas-paperless.cred + ``` + + Contents: + + ``` + username=paperless + password=... + ``` + +3. `sudo mkdir -p /mnt/nas/scans /mnt/nas/paperless-export`, then add to + `/etc/fstab`: + + ``` + //192.168.30.10/scans /mnt/nas/scans cifs credentials=/etc/nas-paperless.cred,uid=1000,gid=1000,file_mode=0660,dir_mode=0770,vers=3.0,_netdev,nofail,x-systemd.automount,x-systemd.idle-timeout=0 0 0 + //192.168.30.10/paperless-export /mnt/nas/paperless-export cifs credentials=/etc/nas-paperless.cred,uid=1000,gid=1000,file_mode=0660,dir_mode=0770,vers=3.0,_netdev,nofail,x-systemd.automount,x-systemd.idle-timeout=0 0 0 + ``` + + `uid`/`gid` 1000 is the container's user (`USERMAP_UID`); check that + `id -u` on the OptiPlex is 1000 as well. +4. `sudo systemctl daemon-reload && sudo systemctl restart remote-fs.target`, + then `ls /mnt/nas/scans` (should be empty, no error). + +## Setup + +1. `cp .env.example .env`, fill in the secret key. +2. `docker compose up -d`. The `proxy` network already exists from the + Vaultwarden setup. +3. Check the mount reaches the container: + `docker compose exec webserver ls -la /usr/src/paperless/consume` +4. Admin user (the password stays out of the compose file): + `docker compose exec webserver createsuperuser`. Later, make a normal + user for each person in the web UI. +5. **Pi-hole Local DNS record:** `paperless.home.staffenberger.at` → + `192.168.30.20`. +6. **NPM proxy host:** `paperless.home.staffenberger.at`, scheme `http`, + forward to `paperless` port `8000` (container name, as with Vaultwarden), + **Websockets Support on** (live status of imports), wildcard certificate, + Force SSL, HTTP/2. +7. Scan one test page and check it shows up in Paperless within a minute + and disappears from `scans`. + +Steps 1–7 done 2026-10-05. + +## Users and separating documents + +Each person scans with their own profile on the printer. The file lands in +their subfolder, and a workflow makes them the owner, so only they see it. +Scans from the root folder (profile `NAS`) get no owner and are visible to +both of us. + +``` +Printer profile Folder on the NAS Owner in Paperless +NAS scans/ none → shared +David scans/David/ david +Denise scans/Denise/ denise +``` + +**Not a security boundary:** the superuser sees everything, and so does +anyone with access to the OptiPlex disk or the export on the NAS. It keeps +daily use tidy and private between the two of us, nothing more. + +Steps: + +1. **Group `Household`** (Settings → Users & Groups → Groups). Global + permissions View/Add/Change/Delete on: Documents, Tags, Correspondents, + Document types, Storage paths, Saved views, Custom fields, Notes, Share + links, UI settings. **Not**: Users, Groups, Workflows, Mail accounts/rules, + App config, Admin. +2. **Normal users `david` and `denise`**, both in `Household`, neither + superuser. The admin account is only for administration; otherwise + Denise's documents show up in my daily view. +3. **Subfolders** `scans/David` and `scans/Denise` in File Station. The + printer probably doesn't create folders itself. +4. **Printer profiles** (web interface, Scan to FTP/Network/SharePoint + Profile): Profile 2 `David` → `\\192.168.30.10\scans\David`, Profile 3 + `Denise` → `\\192.168.30.10\scans\Denise`, both set to Network, otherwise + like Profile 1 (paste the `scanner` login into cleared fields, see the + Chrome autofill note above). +5. **Workflows** (Settings → Workflows), one per person: + + | Field | Value | + |---|---| + | Trigger | Consumption Started | + | Sources | Consume Folder | + | Filter path | `*/David/*` (or `*/Denise/*`) | + | Action | Assignment → Owner `david` (or `denise`) | + +6. **Default permissions:** each user, under Settings → Permissions, sets + Default View/Edit to the group `Household`. Otherwise new tags and + correspondents belong only to whoever created them, and the other person + doesn't see them, even on shared documents. +7. **Test:** one scan through each profile, then log in as `david` and as + `denise` and check who sees what. + +### Double-sided documents + +The MFC-L3750CDW's feeder scans **one side only**. Paperless merges two scans +into one document (`PAPERLESS_CONSUMER_ENABLE_COLLATE_DOUBLE_SIDED`): + +1. Stack in the feeder with page 1 first, scan → pages 1, 3, 5. +2. **Turn the stack over** without re-sorting it, scan again → pages 6, 4, 2. +3. Paperless interleaves them into 1–6. The second scan must arrive + **within 30 minutes**, otherwise it counts as a new first half. + +Needs a `double-sided` subfolder (Paperless doesn't create it): +`scans/David/double-sided`, `scans/Denise/double-sided`, and `scans/double-sided` +for shared documents. One printer profile each, e.g. `David 2-sided` → +`\\192.168.30.10\scans\David\double-sided`. Paperless treats +`David/double-sided/` like `David/`. `double-sided` doesn't become a tag. +**Still to test:** whether the owner workflow (`*/David/*`) also applies +to the merged document. + +If the two scans get mixed up (a single-sided document scanned into the +`double-sided` profile), the next scan gets merged with the wrong one. So +use the double-sided profile only for genuinely double-sided documents. + +The subfolders also become tags (`David`, `Denise`) through +`PAPERLESS_CONSUMER_SUBDIRS_AS_TAGS`. That duplicates the owner, but is handy +for filtering. Drop the setting if it's in the way. + +## Backups + +Nightly export, as a crontab entry for `david` on the OptiPlex (`crontab -e`): + +``` +30 2 * * * cd ~/paperless && docker compose exec -T webserver document_exporter ../export --delete --no-progress-bar >> ~/paperless/export.log 2>&1 +``` + +`--delete` removes files from the export that no longer exist in Paperless, +so the export mirrors the current state. Older versions are in Hyper Backup. +Later this moves into Ansible as a systemd timer. + +- The export is **not encrypted**. It's on the NAS (Servers VLAN, share + only readable by `paperless` and admins), and Hyper Backup encrypts it on + the USB drive. +- **Test a restore** once: a throwaway instance, then + `document_importer ../export`. +- Off the OptiPlex, only `.env` matters. `./data` and `./media` are + covered by the export, and `./redisdata` is disposable. + +## Notes + +- Things to verify on first setup: after rebooting the OptiPlex while the NAS is + off, the container should start, and importing should resume on its own + once the NAS is back. If the mount doesn't show up inside the container, + check the `rslave` propagation in the compose file first. +- The image is pinned; check the release notes before bumping. 3.x removed + some database settings, which don't matter with SQLite. +- Office documents and emails need Tika + Gotenberg as extra containers. + Not needed for scans, so add them only if mail import becomes interesting. diff --git a/docker/paperless/compose.yaml b/docker/paperless/compose.yaml new file mode 100644 index 0000000..a5f45df --- /dev/null +++ b/docker/paperless/compose.yaml @@ -0,0 +1,60 @@ +services: + broker: + # Task queue only; holds nothing worth backing up. + image: docker.io/valkey/valkey:9-alpine + container_name: paperless-broker + restart: unless-stopped + volumes: + - ./redisdata:/data + + webserver: + # Pin the exact version; read the release notes before bumping. + # https://github.com/paperless-ngx/paperless-ngx/releases + image: ghcr.io/paperless-ngx/paperless-ngx:3.2.1 + container_name: paperless + restart: unless-stopped + depends_on: + - broker + # No published ports: only reachable through NPM on the shared `proxy` + # network, same as Vaultwarden. `default` reaches the broker. + networks: + - default + - proxy + # PAPERLESS_SECRET_KEY lives here — never commit it. See .env.example. + env_file: .env + environment: + PAPERLESS_REDIS: redis://broker:6379 + PAPERLESS_DBENGINE: sqlite + PAPERLESS_URL: "https://paperless.home.staffenberger.at" + PAPERLESS_TIME_ZONE: Europe/Vienna + PAPERLESS_OCR_LANGUAGE: deu+eng + # The consume folder is an SMB share: no inotify over the network, so poll. + PAPERLESS_CONSUMER_POLLING_INTERVAL: 60 + # scans//file.pdf gets the tag . + PAPERLESS_CONSUMER_RECURSIVE: "true" + PAPERLESS_CONSUMER_SUBDIRS_AS_TAGS: "true" + # The MFC-L3750CDW can't scan both sides. Two scans into a + # `double-sided` subfolder (front sides, then the flipped stack) + # are merged into one document. See README. + PAPERLESS_CONSUMER_ENABLE_COLLATE_DOUBLE_SIDED: "true" + volumes: + # Database, search index and the documents themselves. Local on the + # NVMe; the nightly export to the NAS is the backup. + - ./data:/usr/src/paperless/data + - ./media:/usr/src/paperless/media + # NAS shares, mounted on the host by systemd automount (see README). + # rslave lets the mount that automount creates show up in the container. + - type: bind + source: /mnt/nas/scans + target: /usr/src/paperless/consume + bind: + propagation: rslave + - type: bind + source: /mnt/nas/paperless-export + target: /usr/src/paperless/export + bind: + propagation: rslave + +networks: + proxy: + external: true diff --git a/docs/status.md b/docs/status.md index 2c1b89d..741e776 100644 --- a/docs/status.md +++ b/docs/status.md @@ -68,20 +68,22 @@ Linux PC · Windows gaming PC · partner's PC · printer (optional) · Synology - ACL direction for alerts/control is **Servers → IoT** (Home Assistant reaching in to poll/control devices), not the reverse — IoT devices never need to initiate connections out to notify anyone. - TV casting only needs **Trusted → IoT** allowed (the casting source connects out to the TV); return traffic flows back automatically as part of that established connection, no reverse rule needed. Discovery is handled by the ER605's mDNS repeater. - **ACL rules built** (Omada ACL types: `Network` = exact match, `!Network` = any network except the one picked; Direction `LAN-LAN` vs `LAN-WAN` vs `WAN-IN` are separate scopes — a `LAN-LAN` rule has zero effect on internet access): - Gateway ACLs as of 2026-10-04 (LAN→LAN, evaluated top to bottom): + Gateway ACLs as of 2026-10-05 (LAN→LAN, evaluated top to bottom): | # | Name | Policy | Source | Destination | |---|---|---|---|---| - | 1 | IoT Outwards | Deny | IoT | `!IoT` (everything except IoT) | - | 2 | Guest Outwards | Deny | Guest | `!Guest` (everything except Guest) | - | 3 | Server Outwards | Deny | Servers | Management, Guest | - | 4 | Trusted Outwards | Deny | Trusted | Guest | + | 1 | Printer to NAS SMB | Permit (TCP) | IP group `Printer` | IP-Port group `NAS SMB` (`192.168.30.10:445`) — scan-to-network for Paperless, see [`../docker/paperless/`](../docker/paperless/README.md) | + | 2 | IoT Outwards | Deny | IoT | `!IoT` (everything except IoT) | + | 3 | Guest Outwards | Deny | Guest | `!Guest` (everything except Guest) | + | 4 | Server Outwards | Deny | Servers | Management, Guest | + | 5 | Trusted Outwards | Deny | Trusted | Guest | + | 6 | Controller - Management | Permit, **disabled** | IP group `Omada Controller` | Management | - Internet access is untouched by all of these, since WAN isn't in scope for a `LAN-LAN` rule. - Everything else (Trusted↔Servers, Trusted→Management, Servers→IoT, Trusted→IoT) relies on Omada's allow-all-by-default behavior. - - The ACLs are **stateful**: replies to allowed connections pass, so the controller on Servers reaches VLAN 99 devices without an extra rule even though rule 3 denies Servers → Management (the devices open the connection to the controller). + - The ACLs are **stateful**: replies to allowed connections pass, so the controller on Servers reaches VLAN 99 devices without an extra rule even though Server Outwards denies Servers → Management (the devices open the connection to the controller). - Printer discovery from Trusted uses an Omada **Bonjour (mDNS) rule**, IoT → Trusted. - - Planned tightening (rule 3 also covering Trusted/Default, DNS exceptions, locking down Default/Management): see [todo.md](todo.md). + - Planned tightening (Server Outwards also covering Trusted/Default, DNS exceptions, locking down Default/Management): see [todo.md](todo.md). - **WAN-IN rules generally aren't needed** — NAT already blocks all unsolicited inbound traffic by default; WAN-IN only matters for scoping something you've already port-forwarded (e.g. the VPN port). - ER605 supports full per-port 802.1Q tagging/untagging/PVID; no practical VLAN-count limit for this setup. - ER605's built-in **mDNS repeater** bridges Bonjour/mDNS discovery across VLANs (relevant for reaching OctoPrint/HA by local hostname). @@ -182,6 +184,7 @@ Planned: - Portainer — behind HTTPS, for viewing and restarts only; the compose files stay the source of truth - Vaultwarden, with backups to the NAS +- Paperless-ngx — scanner drops into a NAS share, Paperless imports from there; data local on the OptiPlex, nightly `document_exporter` to the NAS is the backup (DB on a network share rejected) - Filament spool manager - Self-written Docker tools - Everything set up by hand so far; bring it under Ansible (see todo.md) diff --git a/docs/todo.md b/docs/todo.md index 5439375..ee1d293 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -21,8 +21,8 @@ Set up 2026-10-04, see [homeserver.md](homeserver.md). **Network** - [ ] **Switch DHCP DNS** in Omada from `192.168.30.10` (NAS Pi-hole) to `192.168.30.20` on every VLAN that uses Pi-hole, renew leases, then **turn off the NAS Pi-hole**. Keep no secondary DNS outside Pi-hole (see status.md, Local DNS). -- [ ] **Narrow ACL exceptions for DNS** to Pi-hole (`192.168.30.20`, port 53) from IoT and Guest, placed above rules 1 and 2. Replaces the old "should IoT use Pi-hole" question below. -- [ ] Add **Trusted and Default** to the destinations of rule 3 (Server Outwards). +- [ ] **Narrow ACL exceptions for DNS** to Pi-hole (`192.168.30.20`, port 53) from IoT and Guest, placed above IoT Outwards and Guest Outwards. Replaces the old "should IoT use Pi-hole" question below. +- [ ] Add **Trusted and Default** to the destinations of Server Outwards. - [ ] **Lock down the Default network**: allow only the controller ports 29810–29817 to the OptiPlex, block internet. - [ ] Optional: restrict Management to controller, DNS and internet only. - [ ] Check the **DDNS and LAN DNS settings** in the controller after the migration. @@ -43,6 +43,11 @@ Set up 2026-10-04, see [homeserver.md](homeserver.md). - [ ] **Trial phase:** a few non-critical passwords kept in both Enpass and Vaultwarden. Decide whether to switch. - [ ] If yes: partner's account (then `SIGNUPS_ALLOWED: "false"`), Organization "Home", backup + test restore, then both import their old vaults. - [ ] Every phone's WireGuard tunnel needs the OptiPlex Pi-hole as DNS server — edited by hand in the app (changing it in the ER605 only affects newly generated configs). +- [ ] Paperless-ngx: Brother scanner → NAS share `scans` → Paperless, nightly export to the NAS — setup in [`../docker/paperless/`](../docker/paperless/README.md). NAS shares + users, ACL exception (Printer to NAS SMB), Hyper Backup, scanner → NAS and Paperless itself (behind NPM) done 2026-10-05. Still open: + - [ ] **Users and separating documents**: group `Household`, users `david` + `denise`, printer profiles + subfolders per person, owner workflows, default permissions — steps in the README's "Users and separating documents" section. Includes `double-sided` subfolders + printer profiles (the scanner can't do duplex, Paperless merges two scans). Copy the updated `compose.yaml` to the OptiPlex and `docker compose up -d` first. + - [ ] **Nightly export** cron job (README → Backups), then check the first export lands in `paperless-export`. + - [ ] Test a restore from the export onto a throwaway instance. + - [ ] Reboot test: OptiPlex reboots while the NAS is off → importing resumes once the NAS is back. - [ ] Own Docker tools. **Home Assistant**