feat(output): Implement incremental save functionality for packs (fixes #512) by Stoupy51 · Pull Request #513 · mcbeet/beet

Stoupy51 · 2026-05-09T16:05:28Z

Fixes #512

A new incremental option (default to false) can be given to the output plugin.
either OutputOptions or in beet.yml:

meta:
  output:
    incremental: true

When enabled:

Files no longer in the pack are deleted individually instead of wiping the whole directory
Possible empty directories left over are pruned bottom-up
Files already on disk and unchanged are skipped
Only new files (or modified) are written

File watchers will now be less in pain each build.
On my tests, I get faster build time, idk if you want to make it True by default so I put at False.

Bonus: Optimized the list_files() function that was building a Path to compute relative time for each file:
8f4d701

mcbeet#512)

Stoupy51 · 2026-05-09T16:10:55Z

Damn, didn't thought copilot review would start automatically on another repo beside mines

Copilot

Pull request overview

This PR adds an incremental output mode to the output contrib plugin to avoid wiping and rewriting entire pack directories on every build, addressing performance issues for large projects (issue #512).

Changes:

Add incremental: bool to OutputOptions (default False).
Implement incremental_save() to delete removed files, prune empty directories, and skip writing unchanged files.
Update output() to use incremental saving for non-zipped packs when enabled.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…management

…elative root

…ifest

Stoupy51 · 2026-09-01T08:55:50Z

should meta.output.incremental option be true by default?

rx-dev · 2026-09-01T19:21:31Z

should meta.output.incremental option be true by default?

Honestly, yes imo. it's an improvement.

Stoupy51 · 2026-09-01T19:27:19Z

should meta.output.incremental option be true by default?

Honestly, yes imo. it's an improvement.

I was just on a call with airdox, and he says it's not worth it on linux but windows would be good.
We should test that. I can't personally now, I can try at work if I find time tho

Stoupy51 · 2026-09-01T19:36:39Z

AirDox just tested from my branch on linux:

top: meta.output.incremental = False/None | 174ms
bottom: meta.output.incremental = True | 31ms
(Tested multiple time to be sure and got even better (error margin))

rx-dev

Some things were nitpicks, i realize the original code here used os than I realized though I really do think the pathlib apis are a bit cleaner and has better patterns which would cleanup more of the use-cases here.

There's a lot of functions written out here. Some is good organization, but some feels like over-doing it. Though, each one is described well so perhaps it would be good for future maintenance.

rx-dev · 2026-09-01T19:21:50Z

 from beet.core.utils import FileSystemPath, log_time_scope
+from beet.library.base import Pack, PackFile
+
+ManifestEntry = tuple[int, int, str]


Suggested change

ManifestEntry = tuple[int, int, str]

type ManifestEntry = tuple[int, int, str]

rx-dev · 2026-09-01T20:05:41Z

+    result: dict[str, tuple[int, int]] = {}
+
+    def walk(path: str, prefix: str) -> None:
+        with os.scandir(path) as entries:


We should generally use pathlib to walk the tree.

rx-dev · 2026-09-01T20:05:54Z

+    for dirpath, _dirnames, _filenames in os.walk(directory, topdown=False):
+        if dirpath != str(directory):
+            try:
+                os.rmdir(dirpath)


same here, this code pattern is a bit cleaner with pathlib

rx-dev · 2026-09-01T20:06:36Z

+
+
+def content_signature(pack_file: PackFile) -> str:
+    """ Return a signature that changes whenever the bytes the pack file would write change.


the formatting of these docstrings should be consistent with the codebase (no leading spaces).

rx-dev · 2026-09-01T20:07:43Z

+    Args:
+        pack_file (PackFile): The file to sign.
+    Returns:
+        str: Either `source:<size>:<mtime>` or `sha1:<digest>`.


I don't think we use this expanded style for docstrings in the beet repo.

rx-dev · 2026-09-01T20:08:43Z

+            encoding: str = getattr(pack_file, "encoding", None) or "utf-8"
+            return disk_path.read_text(encoding=encoding) != serialized
+        return disk_path.read_bytes() != serialized
+    except Exception:


This captures way to many states here, what specific exceptions are you trying to guard here?

i would also use with contextlib.suppress pattern for this and specify the specific Exception types that u want to guard

rx-dev · 2026-09-01T20:09:39Z

+            if len(value) == 3
+        }
+    except (OSError, ValueError, TypeError, KeyError, IndexError):
+        return {}


same here, not a big fan of how this is laid out tbh. this is a big hard to read, i would refactor this even if it's a bit more verbose.

rx-dev · 2026-09-01T20:11:40Z

+    if incremental is None:
+        meta_opts = ctx.meta.get("output")
+        if isinstance(meta_opts, dict):
+            incremental = bool(meta_opts.get("incremental", False))


hmmm, i don't think u should be reading ctx.meta here, the way that @configurable works in the first place should auto-fill the opts with stuff from meta.

rx-dev · 2026-09-01T20:13:33Z

+                        target: Path = Path(path) / pack.name
+                        incremental_save(pack, target, cache.get_path(f"manifest:{target}"))


Suggested change

target: Path = Path(path) / pack.name

incremental_save(pack, target, cache.get_path(f"manifest:{target}"))

incremental_save(path / pack.name, target, cache.get_path(f"manifest:{target}"))

path should already be a Path

rx-dev · 2026-09-01T20:14:23Z

+            incremental = bool(meta_opts.get("incremental", False))
+
+    paths: list[Path] = [ctx.directory / path for path in opts.directory.entries()]
+    packs: list[Pack] = list(filter(None, ctx.packs))


Suggested change

packs: list[Pack] = list(filter(None, ctx.packs))

packs: list[Pack] = [pack for pack in ctx.packs if pack is not None]

matches the symmetry of the above line a bit better imo. (nitpick)

feat(output): Implement incremental save functionality for packs (fixes

83e9969

mcbeet#512)

Copilot AI review requested due to automatic review settings May 9, 2026 16:05

Copilot started reviewing on behalf of Stoupy51 May 9, 2026 16:06 View session

Copilot AI reviewed May 9, 2026

View reviewed changes

Comment thread src/beet/contrib/output.py

Comment thread src/beet/contrib/output.py

Comment thread src/beet/contrib/output.py Outdated

Comment thread src/beet/contrib/output.py

Stoupy51 and others added 6 commits May 9, 2026 18:32

fix(output): Update incremental option handling and ensure directory …

5febfcb

…management

fix(output): Fixed serialized issues with incremental mode

803f421

Merge branch 'mcbeet:main' into main

63ea065

Merge branch 'mcbeet:main' into main

b24ce54

perf(output): Optimized list_files() function with precomputing the r…

8f4d701

…elative root

perf(output): Skip reading output files back with a written-files man…

a300747

…ifest

perf(output): Incremental save by default

ec1ebbd

rx-dev requested changes Sep 1, 2026

View reviewed changes

	ManifestEntry = tuple[int, int, str]
	type ManifestEntry = tuple[int, int, str]



		def content_signature(pack_file: PackFile) -> str:
		""" Return a signature that changes whenever the bytes the pack file would write change.

		target: Path = Path(path) / pack.name
		incremental_save(pack, target, cache.get_path(f"manifest:{target}"))

	target: Path = Path(path) / pack.name
	incremental_save(pack, target, cache.get_path(f"manifest:{target}"))
	incremental_save(path / pack.name, target, cache.get_path(f"manifest:{target}"))

	packs: list[Pack] = list(filter(None, ctx.packs))
	packs: list[Pack] = [pack for pack in ctx.packs if pack is not None]

Uh oh!

Uh oh!

Conversation

Stoupy51 commented May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Stoupy51 commented May 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Stoupy51 commented Sep 1, 2026

Uh oh!

rx-dev commented Sep 1, 2026

Uh oh!

Stoupy51 commented Sep 1, 2026

Uh oh!

Stoupy51 commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rx-dev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rx-dev Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rx-dev Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Stoupy51 commented May 9, 2026 •

edited

Loading

Stoupy51 commented Sep 1, 2026 •

edited

Loading

rx-dev Sep 1, 2026 •

edited

Loading

rx-dev Sep 1, 2026 •

edited

Loading