Skip to content

Music Sweeper

k8s: heezy namespace, one CronJob. No web UI, no Service, no hostname. Source: heezy-containers/dockerfiles/music-sweeper/ Manifests: heezy-k8s/apps/music-sweeper/ Image: 025066240222.dkr.ecr.us-east-2.amazonaws.com/heezy-music-sweeper Schedule: */30 * * * * Last Updated: 2026-08-27

Why It Exists

soularr is the only thing in the media pipeline that moves files, and it only imports albums it grabbed inside its own run, tracked in memory. Three things fall through:

  • Manual slskd searches. Nothing scans the downloads directory, and Lidarr has no slskd download client configured, so a hand-grabbed album has no path to /music at all.
  • Downloads that outlive the run that started them. soularr moves on; the transfer completes into a folder nobody revisits.
  • Failed imports. These land in failed_imports/ and stay.

Measured on 2026-08-14, before the sweeper existed:

Count
Folders in the downloads PVC 6,559
Still carrying raw remote names (never handed to Lidarr) 6,239
Carrying soularr's Artist - Album (Year) form 320
In failed_imports/ 129
Name-identical to a folder already under /music 252

The worked example was Chris Hülsbeck: a monitored artist in Lidarr with all four Turrican Soundtrack Anthology volumes listed as monitored albums, 2.5GB of FLAC sitting in the PVC, and zero mentions of "Turrican" in soularr's logs. Hand-grabbed, so nothing was ever going to import it.

What One Run Does

  1. Ask slskd which folders it is currently writing into. If slskd cannot be reached the run stops here — "no answer" is not "nothing active", and moving a folder mid-transfer corrupts it.
  2. Take up to MAX_ALBUMS (25) folders that are not in flight, not modified in the last QUIET_MINUTES (15), and actually contain audio.
  3. Delete slskd's ticks-suffixed re-grabs, keeping the largest of each collision.
  4. Identify the album from tags, falling back to parsing the folder name.
  5. If the library already holds it, delete the download.
  6. Otherwise rename to Artist - Album (Year) and hand it to Lidarr via DownloadedAlbumsScan. Success is measured by the audio being gone afterwards, not by the command status, which reports completed even when it imported nothing.
  7. If Lidarr declines, move the folder into /music directly.
  8. Refresh only the Plex paths that changed, then post a summary to Discord.

Steps 6 and 7 are the reason both destinations exist. Lidarr-imported albums land organized under /music/lists/<Artist>/; everything else lands flat at /music/<Artist> - <Album> (<Year>)/. Both are inside Plex's library.

The slskd Ticks Suffix

When a file arrives whose name already exists in the destination, slskd appends .NET ticks rather than overwriting:

01_Shoot or Die.flac
01_Shoot or Die_639223320842621518.flac

These are re-grabs of the same track, and an album carrying them will fail a Lidarr import because the track count no longer matches the release. Turrican Vol. 1 and Vol. 2 had 15 each.

The sweeper groups files by their ticks-stripped name and keeps the largest, tie-breaking to the unsuffixed original. A group with no ticks-suffixed member is two genuinely different files and is left alone.

Safety Properties

These are deliberate and worth preserving through any edit.

Property Mechanism
Unidentifiable albums are never deleted album_key() returns None unless both artist and album normalize to something non-empty
A partial in the library cannot delete a full download Library copy must have ≥ 90% as many tracks (COMPLETENESS_RATIO)
Destinations are never overwritten An existing path gets a timestamp suffix
Blast radius is bounded MAX_ALBUMS per run
Manual runs cannot collide with scheduled ones Lock file under .sweeper/, stale after 3h
A pulled image cannot do damage Dockerfile CMD passes --dry-run
One bad folder does not end the batch Per-folder exception handling, reported in the summary

Configuration

Full table in the app README. The ones that matter operationally:

Variable Default Notes
MAX_ALBUMS 25 Folders per run. Raise to drain the backlog faster
QUIET_MINUTES 15 Ignore recently-modified folders
DRY_RUN false Set true to report without changing anything
DELETE_DUPLICATES true Set false to report duplicates and keep them
LIDARR_DOWNLOADS_DIR /slskd-downloads The downloads PVC as Lidarr mounts it

LIDARR_DOWNLOADS_DIR is the same trap soularr documents in its ConfigMap. The sweeper mounts the PVC at /downloads, Lidarr at /slskd-downloads, and Lidarr is handed a string it must resolve itself. Change one mount, change both.

The Library Index

Tag-reading a 40TB library every 30 minutes is not viable over NFS, so the normalized artist|album index is cached to $STATE_DIR/library-index.json (default /downloads/.sweeper/) and invalidated per folder by mtime. The first run is slow; later runs re-read only what changed.

Force a rebuild:

kubectl exec -n heezy deploy/lidarr -- rm /slskd-downloads/.sweeper/library-index.json

Two Dedupe Passes

Neither runs on the schedule. Both are dry run unless given --apply.

app.dedupe_library app.dedupe_tracks
Unit a folder a file
Deletes when every album in the folder is kept elsewhere the same recording sits twice in one folder
Reads library-index.json library.db
Catches the same record under two folder names the same record duplicated inside one

The split is the whole point. dedupe_library never opens a folder to remove a file, so a folder holding every one of its tracks twice is a folder it correctly calls healthy. dedupe_tracks only ever groups files within a single folder, so it can never move an album's tracks between them.

The worked example is /music/Beyond the Flow of Time - Music from Chrono Trigger (2021): 36 files of an 18-track album, every track present twice and byte-identical, one copy of each carrying the slskd ticks suffix.

... - 08 - Frog's Theme (Arrangement).mp3                     5,941,704 bytes
... - 08 - Frog's Theme (Arrangement)_639079837294224296.mp3  5,941,704 bytes

The sweeper's own ticks check would have caught it, but that check only runs on a folder while it sits in the downloads PVC, on its way past. A folder copied into /music by hand, or moved there before that check existed, is never looked at again by anything.

The Per-File Inventory

inventory.py writes one row per audio file to $STATE_DIR/library.db: tags, disc and track position, size, mtime, the normalized album and track keys, and whether the filename carries a ticks suffix. The JSON index is shaped for one question — "do we already hold this album?" — and collapses each folder to a count and a byte total, so it can say nothing about an individual file. This is the flat, queryable other half.

The walk is uncapped in depth, unlike album_folders(). That stops at 2 because nothing deeper is an album folder; a file the inventory never lists is a file no dedupe can reason about, and copied torrent dumps put audio four levels down.

SQLite is never opened on the NFS export. The database is copied to local disk, worked on, and copied back only on a clean exit — byte-range locking over NFS is unreliable, and a per-INSERT round trip would dominate a 46,000-file scan.

Rescans are incremental by size and mtime. --no-rescan skips the NFS walk entirely and plans against what is stored.

python3 -m app.dedupe_tracks --csv /downloads/.sweeper/library.csv --no-notify

Removing Duplicate Tracks

Evidence is graded, and only the top grade deletes by default.

Tier Match Inference
ticks same folder, same filename once _<18-digit ticks> is stripped, at least one member carrying it none
tagged same folder, album, disc, track number and title, with a real title tag on every member tags are correct
cross the same track in two different folders reported only, never deleted

cross is a report and not a delete path because choosing between folders belongs to dedupe_library, which has the album completeness rules to do it safely. Deleting file by file across folders would hollow out one copy of an album without ever removing the folder.

The keeper is ranked lossless first, then largest, then the unsuffixed name, then the shorter one. Quality outranks everything for the same reason it does in dedupe_library.choose_keeper(): deleting the only FLAC to keep an mp3 is a loss no tidiness pays for.

The tagged tier refuses any file missing a disc or track number rather than reading absent as zero. A double album whose two discs both open with an "Intro" is exactly how a title-only match deletes a different song.

Flag Effect
--apply actually delete. Absent, everything is a dry run
--tier repeatable, ticks (default) or tagged
--include-managed also consider /music/lists, which Lidarr owns and tracks in its own database
--sidecars also drop ticks re-grabs of .lrc, .jpg, .cue, .nfo
--limit N only the N largest
--report PATH write the full plan, keeper named beside every drop
--csv PATH dump the whole inventory
--no-rescan plan against the stored inventory without walking NFS

Operations

Trigger a run:

kubectl create job -n heezy --from=cronjob/music-sweeper sweep-manual-$(date +%s)

The Discord embed is the intended interface — counts per outcome, what moved, what was deleted and what it already had. Logs are the fallback:

kubectl logs -n heezy -l job-name=sweep-manual-<ts> --tail=50

Either dedupe pass runs as a one-off Job off the same CronJob template. Nothing deletes without --apply:

kubectl create job -n heezy --from=cronjob/music-sweeper dedupe-tracks-$(date +%s) \
  --dry-run=client -o json \
  | jq '.spec.template.spec.containers[0].command = ["python3","-m","app.dedupe_tracks","--report","/downloads/.sweeper/plan.md"]' \
  | kubectl apply -f -

Read the plan it wrote before applying anything:

kubectl exec -n heezy deploy/lidarr -- head -50 /slskd-downloads/.sweeper/plan.md

It reports "slskd unreachable"

Expected during a gluetun VPN reconnect. Nothing was swept and nothing was harmed. If it persists, see Media Pipeline.

It deleted something it should not have

Every deletion names the library folder it matched against. If the match was wrong, the fix is DELETE_DUPLICATES=false while the normalization in app/naming.py is corrected — the matching rules are shared with media-audit.py, so a fix belongs in both.

The backlog is not shrinking

Check the still queued count in the embed. At 25 albums per run on a half-hourly schedule the ceiling is 1,200 a day, so a 6,000-folder backlog takes about five days. Raise MAX_ALBUMS to move faster, at the cost of more NFS traffic per tick.