getdeps: vendor subcommand and --vendor-dir for offline builds - #491
Closed
michel-slm wants to merge 3 commits into
Closed
michel-slm wants to merge 3 commits into
michel-slm wants to merge 3 commits into
Conversation
meta-cla
Bot
added
the
CLA Signed
Contributor
|
@michel-slm has imported this pull request. If you are a Meta employee, you can view this in D120204081. |
`getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to DIR/<name>, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook#488 land that drops to the seven projects Fedora does not package. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Michel Lind <salimma@fedoraproject.org>
`getdeps.py --vendor-dir DIR build project` takes every third-party dependency from DIR/<project>, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook#488, facebook#489 and facebook#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Michel Lind <salimma@fedoraproject.org>
michel-slm
force-pushed
the
getdeps-vendor
branch
from
September 15, 2026 19:32
3f4ba2f to
8f865f6
Compare
|
@michel-slm has updated the pull request. You must reimport the pull request before landing. |
Cover the new behaviour with focused unit tests in the style of the existing builder tests (MagicMock loader and build options, real ManifestParser objects): - `vendor` copies each non-system dependency's fetched tree to <output-dir>/<name>, skips dependencies that resolve to a SystemPackageFetcher, skips the project itself, drops .git, follows symlinks so the result is self-contained, and writes getdeps-vendor.txt with the fetcher hash of every vendored project; - `vendor` replaces a stale tree already present in the output dir; - with --vendor-dir set, a manifest with a download URL resolves to a LocalDirFetcher on <vendor-dir>/<name>; - with --vendor-dir set and the project absent from it, fetcher creation fails naming the missing project instead of falling back to a network fetcher; - without --vendor-dir the normal ArchiveFetcher is still chosen. Run with `python3 -m unittest getdeps.test.vendor_test` from build/fbcode_builder: 5 tests, all passing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Michel Lind <salimma@fedoraproject.org>
|
@michel-slm has updated the pull request. You must reimport the pull request before landing. |
meta-codesync Bot
pushed a commit
to facebook/openr
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/BGP
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/fbthrift
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/wangle
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebookexperimental/rust-shed
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebookexperimental/edencommon
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
Contributor
|
@michel-slm merged this pull request in 72f00ef. |
meta-codesync Bot
pushed a commit
to facebook/mvfst
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/fboss
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebookincubator/katran
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/fb303
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/proxygen
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebookexperimental/moxygen
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/hhvm
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebookincubator/fizz
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebookincubator/hsthrift
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/rebalancer
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/watchman
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/folly
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
meta-codesync Bot
pushed a commit
to facebook/sapling
that referenced
this pull request
Sep 16, 2026
Summary: Three commits, best read in order. Together they give getdeps the equivalent of `cargo vendor` plus `--offline`, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep. ## getdeps: add a vendor subcommand `getdeps.py vendor --output-dir DIR project` fetches every third-party dependency of the project and copies its source tree to `DIR/<name>`, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256). This is the getdeps analogue of `cargo vendor` / `go mod vendor`: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will let `build` consume such a directory offline. Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder). Tested on Fedora 44: `getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelib` produced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from facebook/CacheLib#488 land that drops to the seven projects Fedora does not package. ## getdeps: add --vendor-dir to build from vendored sources offline `getdeps.py --vendor-dir DIR build project` takes every third-party dependency from `DIR/<project>`, as populated by `getdeps.py vendor`, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed. The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for. --free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths). Known limitation: patchfiles are applied with `git apply` from the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up. Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB): ``` unshare -rn python3 build/fbcode_builder/getdeps.py \ --allow-system-packages --vendor-dir /var/tmp/cachelib-vendor \ --scratch-path /var/tmp/getdeps-offline-scratch --num-jobs 2 \ --extra-cmake-defines '{"CMAKE_POLICY_VERSION_MINIMUM":"3.5"}' \ build --free-up-disk --no-tests --src-dir=. cachelib ``` `unshare -rn` puts the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833]; `nsenter -n ip -brief link` inside it showed only `lo` DOWN, `getent hosts github.com` failed (exit 2), and `ss -tunap` listed no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, and `show-source-dir --recursive` resolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB. ## getdeps: add tests for vendor and --vendor-dir `getdeps/test/vendor_test.py`, in the style of the existing builder tests (MagicMock loader and build options, real `ManifestParser` objects), 5 tests: `vendor` copies a non-system dependency's tree, skips a dependency that resolves to a `SystemPackageFetcher`, skips the project itself, drops `.git`, materialises symlinks and writes `getdeps-vendor.txt`; `vendor` replaces a stale tree in the output dir; `--vendor-dir` resolves a download-URL manifest to a `LocalDirFetcher` on `<vendor-dir>/<name>`; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without `--vendor-dir` the normal `ArchiveFetcher` is still chosen. Run from `build/fbcode_builder` with `python3 -m unittest getdeps.test.vendor_test`. Verified together with facebook/CacheLib#488, facebook/CacheLib#489 and facebook/CacheLib#490 applied; each of those stands alone and none of them is required for this change to apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) X-link: facebook/CacheLib#491 Reviewed By: aleivag, likewhatevs Differential Revision: D120204081 Pulled By: michel-slm fbshipit-source-id: f877310e93bcddbf6ff6f81f72e2738e23ecdfa5
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three commits, best read in order. Together they give getdeps the equivalent of
cargo vendorplus--offline, which is what Fedora's vendored-dependency guidelines need for packaging cachelib without maintaining separate folly/fizz/wangle/mvfst/fbthrift packages in lockstep.getdeps: add a vendor subcommand
getdeps.py vendor --output-dir DIR projectfetches every third-party dependency of the project and copies its source tree toDIR/<name>, skipping deps satisfied by system packages and the project itself. It also writes DIR/getdeps-vendor.txt listing each vendored project with its fetcher hash (the pinned git revision or the archive sha256).This is the getdeps analogue of
cargo vendor/go mod vendor: distro packagers can ship the result as a vendor tarball next to the project source, as Fedora's vendored-dependency guidelines require, and a follow-up change will letbuildconsume such a directory offline.Extracted trees rather than the original archives and clones are vendored so the result is self-contained, reviewable and license-scannable; .git directories are dropped and symlinks are followed for the same reason (subproject fetchers symlink into the scratch dir, which does not exist on an offline builder).
Tested on Fedora 44:
getdeps.py --allow-system-packages vendor --no-tests --output-dir /var/tmp/cachelib-vendor cachelibproduced eleven trees (about 330 MB, fbthrift being 216 MB of that) plus getdeps-vendor.txt, with no .git directories or symlinks left behind. Once the Fedora rpm mappings from #488 land that drops to the seven projects Fedora does not package.getdeps: add --vendor-dir to build from vendored sources offline
getdeps.py --vendor-dir DIR build projecttakes every third-party dependency fromDIR/<project>, as populated bygetdeps.py vendor, instead of cloning or downloading it. The lookup sits in the manifest's fetcher selection after the system-package check, so a dependency that --allow-system-packages resolves to an installed package still wins, and anything not found in DIR raises an error naming the missing project rather than falling back to the network. That single check is the offline guarantee: no GitFetcher or ArchiveFetcher is ever constructed.The vendored tree is wrapped in the existing LocalDirFetcher, the same mechanism --src-dir uses, so no fetcher code changes. Its hash is fixed and it always reports the sources as changed, which means repeated builds against a vendor dir reconfigure their dependencies each time; that is acceptable for the one-shot distro builds this is meant for.
--free-up-disk only removed build trees whose sources live under the scratch dir, a guard meant to protect a user's own --src-dir checkout. Vendored sources are ours to clean up after, so the guard now also accepts the vendor dir (stored realpath'd so the prefix comparison matches LocalDirFetcher's realpath'd source paths).
Known limitation: patchfiles are applied with
git applyfrom the enclosing git top-level, so a vendor dir placed inside another git checkout would mis-apply patches for the (few) manifests that carry them. A distro build directory is not a git checkout, so this does not affect the intended use; making the patch step independent of the surrounding repository is left for a follow-up.Tested on Fedora 44 (aarch64, 4 cores) on a tree that also carried #488, #489 and #490, against the seven-project vendor dir produced by the previous commit (magic_enum, sparsemap, folly, fizz, wangle, mvfst, fbthrift; 276 MB):
unshare -rnputs the build in its own network namespace. Inspected from outside while it ran, the build process was in net:[4026532485] versus the shell's net:[4026531833];nsenter -n ip -brief linkinside it showed onlyloDOWN,getent hosts github.comfailed (exit 2), andss -tunaplisted no sockets. The build finished in 72 minutes with exit 0, all eight projects installed, no "Download with" or "Cloning" line in the log, and cachebench linking the system glog, liboqs, libaio and libnuma. Requesting a project that is neither vendored nor allowed from system packages fails with the new error, andshow-source-dir --recursiveresolves every vendored project into the vendor dir. The --free-up-disk fix was verified separately: an offline sparsemap build with the flag left no build tree behind, where the 72-minute run (made before the fix) had left 11 GB.getdeps: add tests for vendor and --vendor-dir
getdeps/test/vendor_test.py, in the style of the existing builder tests (MagicMock loader and build options, realManifestParserobjects), 5 tests:vendorcopies a non-system dependency's tree, skips a dependency that resolves to aSystemPackageFetcher, skips the project itself, drops.git, materialises symlinks and writesgetdeps-vendor.txt;vendorreplaces a stale tree in the output dir;--vendor-dirresolves a download-URL manifest to aLocalDirFetcheron<vendor-dir>/<name>; a project missing from the vendor dir fails naming the project rather than constructing a network fetcher; and without--vendor-dirthe normalArchiveFetcheris still chosen. Run frombuild/fbcode_builderwithpython3 -m unittest getdeps.test.vendor_test.Verified together with #488, #489 and #490 applied; each of those stands alone and none of them is required for this change to apply.
🤖 Generated with Claude Code