gix backend - #2265
Closed
Byron wants to merge 14 commits into
Closed
gix backend#2265Byron wants to merge 14 commits into
Byron wants to merge 14 commits into
Conversation
GitPython's Python implementations of repository storage couple its behavior to on-disk formats and object-ID widths. Delegate repository discovery, configuration, references, reflogs, object storage, tree construction, index operations, and revision parsing to `git.cmd.Git` so the default backend works with SHA-1/SHA-256 objects and files/reftable references. Require Git 2.52 or newer, retain the deprecated `GitDB` as an explicit choice, and preserve raw `repo.git` access. Use existing unsafe option/protocol primitives together with operand validation, NUL-framed records, and protected option ordering. Suppress implicit hooks, filesystem monitors, maintenance, lazy fetching, external diffs, and default text conversion in managed plumbing. Explicit commit hooks remain supported; signing and custom archive commands require opt-in. Preserve the established clean/smudge-filter behavior of existing worktree operations. Keep common workflows and map recognizable native failures to established exceptions. Remove or restrict low-level binary index/tree, raw reflog, precompressed object, and direct storage mutation APIs that cannot be exposed faithfully through Git. Preserve semantic index edits through private native indexes, discover worktree storage through Git, and reconnect retained submodule metadata without assuming its object or reference backend. Document the API and CLI changes in `doc/source/changes.rst`, add format and injection coverage, update minimum-Git CI and fuzz tooling, and bound the existing throughput benchmarks for subprocess-based operations. This work is planned for GitPython 3.4 some weeks before Git 3's official release, allowing users to test this branch and report compatibility feedback beforehand. Validation: the full pytest run produced 1,328 passes, 79 skips, one expected failure, and three submodule failures that were resolved and passed reruns. A fresh submodule/offline run had 224 passes; its two metadata-alias failures were fixed and retested, followed by eight passing retained-metadata checks. The Git 2.52 matrix passed 127 tests, plus four later quoted-branch checks. Ruff, mypy, basedpyright, Sphinx with warnings as errors, Python 3.8 package smoke tests, and deterministic fuzz-harness/version-guard checks pass. The updated fuzzing Docker image was not built and no long fuzz campaign was run.
The Cygwin performance job failed all six setup phases because the Cygwin package currently supplies Git 2.51.0, below the new Git 2.52 minimum. The same mismatch prevents the regular Cygwin suite from opening repositories. Build upstream Git `v2.52.0` with Cygwin tools and install it in `/usr/local` before preparing the fixtures. Include the HTTPS development dependencies so the existing clone and remote tests retain network transport support. Verify that the selected `git` is the built version before continuing. Validation: the workflow parses as YAML, the added shell block passes `bash -n` and ShellCheck, and `git diff --check` passes. The native Cygwin build and test suites require the Windows CI runner.
The macOS Python 3.8 and 3.14 CI jobs each passed 1,348 tests but failed `test_refresh_with_good_relative_git_path_arg`. Installing the supported Homebrew Git exposes `/opt/homebrew/opt/git/bin` on `PATH`; changing into that directory resolves it to the versioned Cellar directory. Compute the expected relative executable path from the current directory after changing into it. This preserves the documented `Git.refresh()` behavior and executable symlinks while removing the test's assumption that `shutil.which()` and `os.getcwd()` retain the same directory spelling. Validation: reproduced the failure using a symlinked directory on `PATH`. All 36 refresh tests pass with both ordinary and symlinked `PATH` values. Ruff lint, Ruff format, and `git diff --check` pass.
The Windows Python 3.8 CI job reported 42 failures and 179 setup errors. Most submodule failures shared a discovery bug: Git resolves relative gitfile targets using forward slashes even when the caller supplies a native Windows path. Normalize Git-facing discovery operands with the existing platform helper, and account for native separators in Git's worktree registry output. Use matching `surrogateescape` codecs for Git protocol paths so undecodable tree names round-trip without Windows filesystem encoding changing their bytes. Verify path/stage, mode, and object ID after materializing a private index: `git update-index --index-info` can exit successfully while dropping Windows-incompatible names. Raise `ValueError` before publishing such an index and document the platform restriction in `changes.rst`. Keep unusual names in object-only tests when the host cannot represent them in a checkout. Use native-valid paths for worktree tests, assert rejection of unsupported index names and quoted file references, and keep full quoted reference coverage with reftable. Fix separator and LF assumptions in config, URL, and packed-reference fixtures. Close test-owned repositories before submodule removal and make the fake Windows Git executable discoverable without shell execution. Also restore root paths for `Repo.tree()` results resolved directly from tree IDs while preserving explicit subtree paths. Validation: 116 repository tests and 10 focused submodule tests passed; index/helper tests passed 93 with 2 platform skips; the focused format/safety run passed 72; config/reference/remote tests passed 57 with 24 subtests; 36 refresh tests and 2 revision regressions passed. The Git 2.52 targeted index matrix passed 19 tests. Ruff, mypy, basedpyright, Sphinx with warnings as errors, and `git diff --check` pass. Native Windows validation awaits CI.
The partial Cygwin fast-suite log exposed a failure in `test_valid_unusual_index_names_round_trip`: native Git omitted a literal backslash filename. Cygwin Git recognizes Windows separators and applies NTFS path protection, even though Python reports a POSIX platform. Move that case into the existing unsupported-name assertions for Cygwin. Check that GitPython raises `ValueError`, preserves the published index, and removes its lock. Other POSIX systems retain the round-trip case; Windows retains the control-character and colon rejection cases. Document the Cygwin restriction in `changes.rst`. Validation: three focused index tests pass locally. The Cygwin and Windows test branches pass with only Git's ignored-record behavior simulated; native Cygwin verification awaits CI. Ruff lint, formatting, and `git diff --check` pass.
The Windows Python 3.8 partial CI log showed two failures in `test_submodule_allows_existing_metadata_symlinks`: preparing update and move aliases raised `PermissionError` before invoking GitPython. Git marks the submodule's `.git` file hidden; Python's `write_text()` attempts to recreate it, which Windows rejects for an existing hidden file. Open that fixture file with `r+`, write the replacement target, and truncate it. This preserves the hidden attribute while replacing the full contents. The separate fixture that creates a previously absent gitfile is unchanged. Validation: all six native/windows37 update, move, and remove alias modes pass locally. Ruff lint, formatting, and `git diff --check` pass. The Windows-specific file-attribute behavior will be verified by CI.
The Windows Python 3.15 CI job failed to set up twelve missing-submodule cases because `shutil.rmtree()` cannot remove read-only loose Git objects. The fixture deliberately removes retained metadata to model an absent submodule, so use the existing `git.util.rmtree()` helper, which clears read-only attributes when retrying Windows deletions. All twelve affected cases pass locally, along with Ruff lint and format checks. Production behavior and test coverage are unchanged.
The Windows Python 3.15 CI job failed to remove submodule checkouts because persistent `cat-file` processes still used them as working directories. `Submodule.update()` relied on collection of its temporary `Repo`, but captured log records retained a `Head` argument and therefore the repository and its process. Recursive updates also opened an extra unbounded repository. Close the owned repository after updates and on errors, and reuse it for recursion with final cleanup. This preserves `keep_going` behavior while releasing processes even when logs or callbacks retain repository objects. The compatibility test now scopes its own repository and closes it before removal; allocation tracing identified those separate caller-owned handles. Four regressions retain real logging arguments and verify process cleanup for normal, failing, recursive, and recursive `keep_going` updates. Both previously failing tests pass with tracing asserting no live checkout processes at each removal. Ruff, mypy, basedpyright, and `git diff --check` pass locally. Native Windows validation will run in CI.
The Cygwin full suite reached 1,372 passing tests but failed its native-Git detection check. The new source build installed Git into `/usr/local`, while GitPython's existing detector expects `uname` beside the selected Git executable. Cygwin installs `uname` in its normal `/usr/bin` directory. Install Git 2.52 under `/usr`, replacing the older packaged Git and preserving that standard layout. Verify `Git.is_cygwin()` immediately after installing Python dependencies so a setup regression fails before the long test suite. The detector and its missing-`uname` behavior remain unchanged. YAML parsing, extracted Bash syntax, ShellCheck, and `git diff --check` pass locally. The preceding Cygwin performance suite also passed all six tests; native validation of the corrected installation runs in CI.
…ckout Exercise a current high-download GitPython consumer with its unchanged upstream tests. Add a shared `uv` runner that resolves the latest PyPI release, retrieves verified source, installs a private environment, and replaces the released GitPython dependency with this editable checkout. Verify the imported `git` module before testing and retain source provenance, frozen requirements, and JUnit results for diagnosis and reproduction. Clear inherited Git repository/configuration settings and Python import paths so upstream commits and resets use their own fixtures. Use pytest's long `--override-ini` spelling because Bandit's CLI tests interpret `-o` as a forbidden Bandit output option. Reject successful runs with no passing tests, including entirely skipped suites. Add a CI job for the latest Bandit release, with Git 2.52 or newer, and local usage and download-ranking documentation. Bandit 1.9.4 passes all 12 selected tests against this checkout on Python 3.12 and Git 2.54, even with deliberately invalid inherited Git directory, index, and config settings. Ruff, workflow YAML parsing, and whitespace checks also pass. No GitPython compatibility changes were needed.
Add MLflow's released Git project and context tests to the shared local
runner and CI matrix. Rank the project by `mlflow-skinny` downloads without
summing overlapping distributions, resolve its current PyPI release, and
check out the matching `v{version}` source tag for unchanged upstream tests.
Install the matching full `mlflow` package because upstream global fixtures
need its server and SQLite support. Clear `CI` and `GITHUB_ACTIONS` inside
the isolated run to avoid unrelated upstream wheel builds and conda
cleanup. Disable telemetry and keep the venv first on `PATH` so project
subprocesses use the same editable GitPython checkout.
The shared runner passes all 47 selected tests for MLflow 3.16.1 on Python
3.12 and Git 2.54, including 31 repository/project/model-versioning cases and 16 Git
context or credential-redaction contract cases. Validation started with
both CI variables set, exercising the CI isolation. Public example clones
and a localhost HTTP server are required; no cloud services or models are
needed. The model-versioning cases also cover staged/unstaged diffs and
dirty-state handling. No GitPython compatibility changes were necessary.
Include `langchain-community` as a current runtime GitPython user: its
published `GitLoader` requests a manual GitPython installation even though
it is absent from `Requires-Dist`. Its September download count ranks
above the other selected consumers.
Resolve the latest PyPI release and run unchanged tests from the matching
`libs/community/v{version}` tag against the editable GitPython checkout.
Use upstream test-plugin ranges and `--only-extended` so missing integration
dependencies fail collection instead of silently skipping the tests.
Add the same profile to CI and document the runtime-use selection.
The shared `uv` runner passes both GitLoader tests for release 0.4.2 on
Python 3.12 and Git 2.54. They exercise real local clones, commits,
checkout, tree traversal, ignored files, repeated loads, and remote URL
validation. No network services are used during testing and no GitPython
compatibility changes were required.
SWE-bench is a current high-download GitPython user, but its latest 5.0.2 release has no tests for the GitPython inference helpers and no matching Git release tag. Retrieve the verified PyPI source and explicitly document that this profile runs a GitPython-authored supplemental integration test, rather than misrepresent unrelated upstream tests as compatibility coverage. Import the real `AutoContextManager` with only its required `chardet` and GitPython dependencies. Exercise clone, commit checkout, reset, untracked cleanup, directory restoration, and clone reuse for SHA-1/SHA-256 with files/reftable. Route its normal URL to a local fixture and permit only file transport during the test. Do not patch production modules or mock GitPython. BM25's Java/Pyserini helpers remain outside this focused check. Add the profile to the local runner and CI. Expand pytest-option paths, cut off unrelated ancestor conftests, and use importlib mode so the supplemental filename cannot shadow the installed upstream package. All four cases pass through the shared runner on Python 3.12 and Git 2.54; they also passed separately on minimum Git 2.52. Ruff and whitespace checks pass. No GitPython compatibility changes were required.
Complete the five-current-user compatibility matrix with `acryl-datahub`. Resolve the latest PyPI release and retrieve its tag from `acryldata/datahub`, which publishes patch tags missing from the repository named by package metadata. Run the unchanged Git integration file against the installed release and this editable GitPython checkout. Exclude unrelated SQL/docker conftests, disable telemetry, and clear the private SSH test credential variable. The selected tests still exercise a real public GitLab clone and fixed-commit checkout, a localhost SSH timeout, GitCommandError handling, password redaction, and source configuration. The upstream private-clone test retains its credential-dependent skip. Document all five current users and their September 2026 download ranking, including optional runtime integrations, distribution deduplication, and projects whose latest releases dropped GitPython. Add an all-project local command and the fifth CI matrix entry. The shared runner passes 7 tests with 1 upstream skip for DataHub 1.7.0.14 on Python 3.12 and Git 2.54. All five profiles have now passed locally, with 68 upstream passes plus 4 SWE-bench supplemental cases. Ruff, Python syntax, workflow YAML/matrix consistency, Bash syntax, ShellCheck, and whitespace checks pass. No GitPython compatibility fixes were required. Also clear inherited pytest options and plugins: a caller's `-k` filter could otherwise leave only unrelated contract tests and yield a misleading pass. All four SWE-bench cases still pass with a deliberately nonmatching inherited filter and a nonexistent plugin, verifying their removal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a candidate for GitPython release 3.3 or 3.4, depending, to have a speedy alternative to the CLI.