Development guide¶

Start here to work on genplis: set up a checkout, run the checks, and find the code responsible for your change. For installation and normal use, see the user documentation.

Setup¶

You need Python 3.12 or later and uv:

git clone https://github.com/haplo/genplis.git
cd genplis
uv sync
uv run pre-commit install

uv sync installs runtime and development dependencies into .venv. The pre-commit hook runs checks automatically when you commit.

Running¶

Run from the checkout with uv run:

uv run genplis ~/Music
uv run genplis --verbose /path/to/song.mp3

See Usage for playlist commands and Filter debugging for rule trees and traces.

Tests¶

Testing patterns¶

• Reuse the file_mp3, file_ogg, and file_m3ug session fixtures from tests/conftest.py. Copy them to tmp_path before writing playlists or changing modification times; don't modify tests/files/.
• Use genplis_db for a temporary database and make_args to build core.Args without invoking argparse.
• Use retain_tags=True with process_directory only when a test needs the returned all_tags map. Normal runs release tags after filtering.
• Monkeypatch genplis.tags.TinyTag for synthetic tag data; see _patch_tinytag in tests/test_tags.py.
• Cover success, missing values, and error paths. tests/test_m3ug.py exercises rule semantics; the suite also validates examples/config.ini.

Linting, formatting, and typing¶

uv run ruff check --fix .
uv run ruff format .
uv run ty check

pyproject.toml configures Ruff and ty for Python 3.12. Keep function signatures typed, imports ordered, and use the shared TagValue and Tags aliases from src/genplis/types.py rather than ad-hoc types or Any. See AGENTS.md for the full coding guidelines.

Ruff and ty are intentionally unpinned in the development dependencies and pre-commit configuration. [tool.uv] upgrade-package lets lockfile updates refresh them; keep the seven-day dependency cutoff used by CI.

Pre-commit¶

After staging changes, run the hooks before committing:

uv run pre-commit run

Fix failures, re-stage affected files, and run the hooks again. Repeat this after staging more changes. When hooks or configuration change, also run:

uv run pre-commit run --all-files

The hooks cover whitespace, TOML, large files, the uv lockfile, Ruff, and ty. The ty hook checks the whole project even when only a few files are staged. Definitions live in .pre-commit-config.yaml.

CI¶

Pull requests and pushes to main run:

See ci.yml and pages.yml. Both workflows use UV_EXCLUDE_NEWER="7 days" to match the lockfile policy.

Project layout¶

Application modules live under src/genplis/:

Tests and media fixtures live in tests/; copyable filters and the annotated config live in examples/. Website sources are covered below.

Data flow¶

A normal directory run in core.process_directory:

1. Uses the resolved CLI/config settings and the main process's database connection.
2. Discovers and parses .m3ug files, or uses the filters passed explicitly through filters=. Excluded directories are pruned.
3. Walks audio files: applies filters immediately to cache hits and queues missing or stale files for parsing.
4. Parses the queue, writes cache updates, and evaluates each result against the filters.
5. Sorts matches, removes duplicates, applies count/per-tag/duration limits, and writes playlists with selected entries. @NO-PLAYLIST files are skipped.

Each Match retains only the playlist entry, walk order, selection keys, and duration—not the whole tag map. retain_tags=True is an opt-in exception for tests and debugging. See the selection rules for user-visible behavior.

CLI modes¶

resolve_args chooses one core.Mode: PROCESS, TAG_STATS, SHOW_RULES, TRACE, or TRACE_FULL.

• --install-skill is a standalone early exit before loading configuration or resolving collection arguments; it never accesses the database.
• SHOW_RULES branches before database setup.
• Trace modes load the target through the cache. Full traces use dry_run and SelectionTrace to record selection decisions without writing playlists; cache updates still happen.
• Statistics reuse the walk/cache pipeline with an empty filter map. TagStatsCollector keeps aggregate counts, not song mappings. Explicitly requested tags have uncapped distinct-value counts.
• --clean is a post-processing flag, not a mode: with a collection path, normal processing happens before cleaning.

Parsing pipeline¶

Cache misses and stale files are queued during the walk so progress reporting knows the parsing total. With --jobs 1, core.process_file handles that queue in the main process. Otherwise, make_executor chooses a SequentialExecutor for small queues or a ProcessExecutor for larger ones. See executors.py for batching and worker thresholds.

Keep these invariants when changing the pipeline:

• Only the main process accesses SQLite. Workers parse tags and return ParseResult values with structured diagnostics. M3UG parsing, filtering, selection, and output also stay in the main process.
• Do not inherit database connections. Worker contexts use forkserver on POSIX and spawn elsewhere, never plain fork.
• Preserve ordering. Worker results are consumed in submission order; each file's walk index remains the final tie-breaker for playlist sorting. Changing the worker count must not change deterministic playlist results.
• Release tags after use. Keep Match records, not an all_tags map, unless retention was explicitly requested.
• Flush on exit. db.WriteBatch commits every core.COMMIT_EVERY writes and flushes at the end of parsing, including errors and interrupts. Interrupted executors cancel pending work and shut down without waiting.

Benchmarking¶

Use a test copy of a collection on a real filesystem, not tmpfs. Choose a new database path for a cold run:

/usr/bin/time -f '%e s, %M KiB' \
    uv run genplis --db ./benchmark-sequential.db --jobs 1 /path/to/test-collection

Repeat the command against that database for a warm run. Compare with --jobs 0 using a separate fresh database. Run each case three times and take the median. Compare playlist checksums too; use fixed random seeds and identical inputs/settings so byte-for-byte comparisons are meaningful.

Filter engine¶

The rule engine lives in m3ug.py. These are the main structures to understand before changing it:

Tag lookup and comparison caching¶

EvaluationTags provides a read-only per-song view shared by filtering and selection-key extraction. It lazily resolves names, calculated values, and normalized rule inputs. Don't mutate the underlying tags while the view is in use. Exact lowercase names win over mixed-case aliases; otherwise the first matching spelling wins. The view is discarded after the song.

PredicateCache shares pure built-in comparisons across equivalent rules for one scan. Its keys distinguish rule types, tag names, value types, expected values, and case sensitivity. Custom rule checks aren't cached. The cache is bounded—by default 16,384 results total, at most 1,024 per predicate—and bypasses oversized keys and unsupported values. Admission checks live alongside the class; don't remove them when changing caching.

Neither cache retains whole song mappings across the scan. Short-circuiting, per-rule warnings, and distinctions such as booleans versus numbers must stay unchanged. Standalone rules and traces also support ordinary tag mappings.

Includes and diagnostics¶

parse_m3ug recursively resolves includes in the main process, detects cycles and depth errors, then merges included defaults before local directives. include_loader lets tests supply included content without files. @NO-PLAYLIST belongs to M3UGFile, not Directives, so it isn't inherited. See Includes for path and precedence rules.

own_expression and ordered IncludeInfo records retain each file's boundaries alongside the merged expression. diagnostics.py renders this metadata and trace results. RuleNode.explain records optional TagLookup details; normal apply avoids that trace-recording overhead.

Output and statistics¶

Use Output and Palette from output.py rather than emitting ANSI codes in parsers or workers. Helpers default to Output.plain(); workers return TagDiagnostic records for the main process to render. Theme choice and color enablement are separate decisions.

Statistics renderers measure plain-text widths before coloring cells, so table alignment doesn't depend on escape sequences. Keep collection of statistics separate from rendering and avoid retaining per-file tags.

Cache internals¶

The disposable SQLite cache is implemented in db.py:

• files(path TEXT PRIMARY KEY, last_modified TIMESTAMP, tags JSONB) stores one row per audio file. Tags are JSON-serialized Tags values.
• db.get_db_key normalizes absolute paths and collapses . / .., but doesn't resolve symlinks.
• Freshness compares whole-second modification times. A file whose timestamp is no newer than the cached value is a hit. M3UG files aren't cached.
• cache_meta stores tag_version, built from CACHE_SCHEMA_VERSION:tinytag_version:max_size:ignored-tags with ignored names sorted. A version change makes create_files_table rebuild files.
• Cleaning removes rows for missing files and runs VACUUM only when rows were removed.

Bump CACHE_SCHEMA_VERSION when changing tag extraction or storage in a way that makes old rows incompatible. For location and maintenance commands, see Cleaning the cache.

Extending the code¶

Adding an operator¶

1. Subclass RuleNode, define OPERATOR for diagnostics, and follow the existing OPERATOR_NAME naming convention. Set IS_TEXT = True for text operators so boolean-tag warnings are preserved. Implement check() and override check_params() when argument types need validation. Raise GenplisM3UGException for invalid syntax or values.
2. Register the symbol in RuleNode.build and the operator sets used during parsing. Decide whether it is eligible for PredicateCache.
3. Test text, numbers, lists, missing values, invalid inputs, and cached versus uncached behavior in tests/test_m3ug.py.
4. Document its semantics in M3UG.

Adding a rating source¶

Extend get_rating() in m3ug.py and return a CalculatedRating with the normalized 0–5 value, actual source key, and diagnostic note. Keep source-tag knowledge inside this function; lookup and trace code consume its result without knowing which rating sources are supported. Use a None value when no usable source is found.

Add rating tests in tests/test_m3ug.py and trace tests in tests/test_trace.py. Check explicit *rating, raw-rating precedence, zero ratings, unusable values, and source provenance. Update Tags.

Changing ordering¶

Start with sorting.py for grammar, normalization, and comparison, then the selection helpers in core.py. Preserve the distinction between an absent sort (None, use the configured default) and an empty sort (walk order).

Test direction, numeric strings, case folding, missing and multi-valued keys, random seeds, dedupe, limits, and worker-count stability. See Ordering for the behavior to preserve.

Changing CLI or config behavior¶

Update core.setup_argparse() / resolve_args(), config.py's known settings and parsing, and tests for CLI-over-config and default behavior. Keep Usage, Configuration, and examples/config.ini consistent with the change.

Documentation¶

User guides are Markdown files under docs/; the documentation index lists their subjects. Keep each fact in one guide and link to it elsewhere, rather than duplicating settings or language rules.

• Use relative Markdown links between guides, such as [Tags](tags.md). The website build resolves them to published routes.
• Use absolute GitHub links in README.md and for repository files outside docs/. Keep the README short and link to detailed guides.
• Edit examples in examples/, not copies in documentation. Keep filters and config.ini valid.
• Check incoming links when renaming headings, then run the strict website build to validate local links and anchors.

Packaged agent skill¶

Edit agent-instructions.md, not the generated src/genplis/skills/genplis/SKILL.md. The canonical language, tag, and usage guides supply its offline references. After changing those documents or the package version, regenerate the checked-in package resources:

uv run python scripts/sync_skill.py
uv run python -m pytest tests/test_skill.py

The generator records the package version, adjusts instruction links to the bundled references, and points links to unbundled guides at GitHub. A test rejects drift between documentation and packaged resources. Hatch includes the resources in the wheel and source distribution without custom build hooks. The installed-distribution smoke test also installs both skill targets into a temporary home directory.

Website development¶

The website uses Zensical. Its dependencies live in the separate docs group and aren't needed to run genplis.

Run the website locally¶

uv run --group docs python scripts/website.py serve

Open http://localhost:8000/genplis/, or the printed preview URL. Source edits in docs/ and examples/ regenerate the site and reload the preview. Stop with Ctrl+C; restart after editing scripts/website.py itself.

Build and validate¶

Run the strict build before submitting documentation changes:

uv run --group docs python scripts/website.py build

It writes site/ and validates local links, assets, and anchors under the /genplis/ prefix. To validate existing output without rebuilding, use:

uv run --group docs python scripts/website.py check

Try browser search in the preview after changing navigation or examples. Search is local, covers the guides and examples, and excludes the landing page.

Sources and deployment¶

Edit original sources, not .website-src/ or site/. Both are ignored build outputs. The preparation step rewrites documentation links and adds source-edit links pointing back to the originals.

The gallery reads examples/*.m3ug in filename order and provides copyable code and byte-for-byte downloads. Its ZIP preserves companion include files. Don't create a separate docs/examples.md or maintain duplicate examples.

pages.yml builds on pull requests, main pushes, and manual dispatches; only non-PR runs on main deploy through a Pages artifact and the github-pages environment. Set the repository's Settings → Pages source to GitHub Actions. No gh-pages branch or personal token is needed.

Release¶

Versions live in pyproject.toml; genplis --version reads the installed distribution metadata. Use Semantic Versioning for the CLI, configuration, and documented M3UG language, not internal Python APIs or the disposable cache format.

Keep notable user-visible changes under [Unreleased] in the root CHANGELOG.md. Use Keep a Changelog categories (Added, Changed, Deprecated, Removed, Fixed, Security) only when needed. Internal refactors and CI-only changes need no entry. At release time, move entries into a dated version section, leave an empty [Unreleased], and update the comparison links. GitHub Release notes are extracted from this file, not generated from commit messages.

One-time publishing setup¶

1. Enable 2FA on the PyPI account that will own genplis.
2. Before the first upload, add a pending trusted publisher on PyPI: project genplis, GitHub owner haplo, repository genplis, workflow filename release.yml, environment pypi. After the first upload, this becomes the project's trusted publisher. No API token is stored in GitHub.
3. Create the GitHub repository environment pypi. Restrict deployments to release tags and require approval if available for the repository/account.
4. Protect v* tags against deletion or updates. Restrict who can create them.

Prepare and verify a release¶

Finish unrelated work first. Prepare release changes on a branch and merge them into main after checks pass. The first release is 1.0.0, directly to PyPI, without an RC or TestPyPI release.

1. Run uv version 1.0.0 --no-sync (substitute the next version for later releases). Review the pyproject.toml and uv.lock diff; avoid unrelated dependency upgrades.
2. Date the changelog entry using the intended release day (YYYY-MM-DD), and verify installation instructions and links.
3. Validate the tag/version/changelog combination and run checks:

uv run --locked python scripts/release.py v1.0.0
uv run --locked pre-commit run --all-files
uv run --locked python -m pytest

1. Build into a new, empty directory to avoid uploading old artifacts:

uv build --no-sources --out-dir /tmp/opencode/genplis-1.0.0-dist
uvx twine check --strict /tmp/opencode/genplis-1.0.0-dist/*
uv run --isolated --no-project --with /tmp/opencode/genplis-1.0.0-dist/*.whl python scripts/smoke_release.py 1.0.0
uv run --isolated --no-project --with /tmp/opencode/genplis-1.0.0-dist/*.tar.gz python scripts/smoke_release.py 1.0.0
uv publish --dry-run /tmp/opencode/genplis-1.0.0-dist/*

Use an appropriate empty temporary path on other machines. The smoke test runs against installed artifacts, not src/, and checks both entry points plus sequential, multi-process, and warm-cache playlist generation using copies of the test media. Inspect the wheel and sdist contents too: the wheel must include py.typed, the license, entry points, and package modules, not local caches or benchmark outputs. The sdist must build without git metadata.

1. Stage only the release changes, run uv run pre-commit run after staging, fix/re-stage if needed, and commit. Merge and wait for CI and website checks on main before creating the annotated tag:

git switch main
git pull --ff-only github main
git tag -a v1.0.0 -m "genplis 1.0.0"
git push github v1.0.0

Automated publishing and verification¶

.github/workflows/release.yml runs on stable vX.Y.Z tag pushes. It reuses the full CI checks (including Python 3.12–3.15 tests), validates the tag and dated changelog entry, builds the wheel and sdist, checks metadata/README, and smoke-tests both installed artifacts. Artifacts are built only once.

The separate publish job downloads these verified artifacts and uses PyPI Trusted Publishing (OIDC), with id-token: write restricted to that job and the pypi environment. Only after publishing succeeds does a separate job create the GitHub Release with curated notes and the same wheel/sdist.

Verify the workflow, PyPI README/links, and GitHub Release notes/assets, then test an index installation outside the checkout:

uv run --isolated --no-project --refresh-package genplis --with genplis==1.0.0 genplis --version

Leave the package version at the released version until preparing the next release. If a job fails, prefer re-running failed jobs to preserve the original build artifacts. PyPI accepts retries of identical files, but does not allow replacing an uploaded distribution. Never move a published tag: fix a broken release with a new patch version (and yank the bad version on PyPI if necessary). The GitHub Release job can be retried independently.

Roadmap¶

• Parse tags of all files in music collection
• Cache tags to local DB
• Implement parser for filter grammar
• Implement filtering logic
• Filter files in collection
• Generate playlists (MVP complete!)
• Display tags when receiving a file as parameter
• --exclude option
• Install as genplis command
• pre-commit
• Tests
• PR check
• Run pre-commit on whole project
• Test on every supported Python version
• Documentation
• Example cron and systemd files and instructions on how to run periodically
• Improve M3U generation
• Include original M3UG content as comment
• Support basic extended M3U tags: #EXTM3U and #EXTINF
• Parallel parsing of files
• Config file support
• Improve M3UG grammar
• Command for DB cleaning
• Optimize memory usage
• Optimize DB space
• Display all available tags, with stats
• Debug M3UG
• Color output
• Website
• Improve Windows support (help wanted).
• Improve macOS support (help wanted).
• AI agent skill.
• Publish to PyPI.

Suggestions and contributions are welcome in GitHub issues.