forked from git/git
-
Notifications
You must be signed in to change notification settings - Fork 196
Objects treated as missing despite being present, due to race with geometric repacking #2207
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
newren
wants to merge
4
commits into
gitgitgadget:ps/odb-generic-corrupt-objects
Choose a base branch
from
newren:midx-removed-pack-recovery
base: ps/odb-generic-corrupt-objects
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
36bf2ce
replay: fail gracefully when a merge input is unreadable
newren 3f3b756
mktree: plug per-tree leak in --batch mode
newren fc98f48
packfile: recover object lookups racing a concurrent repack
newren eacf6ba
packfile: recover when a multi-pack-index names a removed pack
newren File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1786,7 +1786,7 @@ static int want_object_in_pack_mtime(const struct object_id *oid, | |
| struct multi_pack_index *m = get_multi_pack_index(files->packed); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Jeff King wrote on the Git mailing list (how to reply to this email): On Tue, Aug 25, 2026 at 07:00:29PM +0000, Elijah Newren via GitGitGadget wrote:
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index 399acd0f22..30ad7d822c 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -1786,7 +1786,7 @@ static int want_object_in_pack_mtime(const struct object_id *oid,
> struct multi_pack_index *m = get_multi_pack_index(files->packed);
> struct pack_entry e;
>
> - if (m && fill_midx_entry(m, oid, &e, NULL)) {
> + if (m && fill_midx_entry(m, oid, &e, NULL) == MIDX_FILL_HIT) {
> want = want_object_in_pack_one(e.p, oid, exclude, found_pack, found_offset, found_mtime);
> if (want != -1)
> return want;
We've changed the return value semantics without changing the signature
(or name). So we need to make sure we adjust all callers, as here.
That's _probably_ OK in practice for such a specialized function. But we
could also rename it if we wanted to be paranoid (especially about
new callers added on parallel branches).
> +enum midx_fill_result fill_midx_entry(struct multi_pack_index *m,
> + const struct object_id *oid,
> + struct pack_entry *e,
> + struct packed_git **bad_pack)
OK, so this is our tri-state fix. Mostly looks as expected, though:
> if (prepare_midx_pack(m, pack_int_id))
> - return 0;
> + goto owner_unavailable;
I'd have expected just "return MIDX_FILL_OWNER_UNAVAILABLE" here. But
then, I'm not sure I buy the need for this stale_packs_detected stuff
from patch 3.
> p = m->packs[pack_int_id - m->num_packs_in_base];
>
> - /*
> - * We are about to tell the caller where they can locate the
> - * requested object. We better make sure the packfile is
> - * still here and can be accessed before supplying that
> - * answer, as it may have been deleted since the MIDX was
> - * loaded!
> - */
> + /* Make sure the pack is still present before pointing at it. */
> if (!is_pack_valid(p))
> - return 0;
> + goto owner_unavailable;
This comment rewrite seems superfluous at best. Can we try to keep such
patch fluff to a minimum?
> + /*
> + * Recovery for a concurrent-repack race: a stale MIDX may still name a
> + * vanished owning pack even though the object survives in another pack
> + * the same MIDX covers. The regular fallback above skips MIDX-covered
> + * packs, and repreparing the on-disk pack set does not reload the
> + * borrowed, cached MIDX, so scan its packs directly for the survivor.
> + *
> + * Do this only on the second read, by which point repreparing packs has
> + * already had a chance to find an object merely relocated into a new,
> + * uncovered pack; only a genuine hidden duplicate reaches here.
> + */
> + if (midx_result == MIDX_FILL_OWNER_UNAVAILABLE &&
> + (flags & OBJECT_INFO_SECOND_READ)) {
> + struct multi_pack_index *m = store->midx;
> + uint32_t i;
> +
> + for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {
> + struct packed_git *p;
> +
> + if (prepare_midx_pack(m, i))
> + continue;
> + p = nth_midxed_pack(m, i);
> + if (p && packfile_fill_entry(p, oid, e, bad_pack))
> + return 1;
> + }
> + }
OK, and this is as-before but now gated on the SECOND_READ flag. As
expected in this revision.
-Peff |
||
| struct pack_entry e; | ||
|
|
||
| if (m && fill_midx_entry(m, oid, &e, NULL)) { | ||
| if (m && fill_midx_entry(m, oid, &e, NULL) == MIDX_FILL_HIT) { | ||
| want = want_object_in_pack_one(e.p, oid, exclude, found_pack, found_offset, found_mtime); | ||
| if (want != -1) | ||
| return want; | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -475,6 +475,12 @@ int prepare_midx_pack(struct multi_pack_index *m, | |
|
|
||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Jeff King wrote on the Git mailing list (how to reply to this email): On Tue, Aug 25, 2026 at 07:00:28PM +0000, Elijah Newren via GitGitGadget wrote:
> 1. open_pack_index() fails, so we print
>
> error: packfile <path> index unavailable
>
> and report the pack as unusable, even though the object still lives
> in the replacement pack.
>
> 2. A normal lookup recovers: odb_read_object_info_extended() issues a
> second read that reloads the on-disk pack state and finds the object
> in its new home, making the message above mere noise. But an
> OBJECT_INFO_QUICK lookup deliberately skips that second read to stay
> fast on a genuine miss, so it does *not* recover: it reports the
> object as absent even though it still lives in the replacement pack.
> A resident reader that resolves objects with a QUICK lookup -- such
> as the `git mktree --batch` process the tests below drive -- then
> produces wrong results. Even where a spurious miss is not fatal it
> is not harmless: `git upload-pack` checks a client's "have" lines
> with a QUICK lookup, and a dropped "have" removes a common object
> from the negotiation, so the client is sent more than it needs.
Maybe I am still being dense, but this description does not make any
sense to me at all.
The _point_ of QUICK is to accept those false negatives. It is the right
thing for upload-pack to do, to avoid re-scans for objects which we
simply don't have (and don't necessarily expect to have).
It sounds like mktree is wrong to be using QUICK at all. It comes from
817b0f6027 (mktree: do not check type of remote objects, 2022-06-21)
which rewrote a call to vanilla oid_object_info(). From the description
there it probably should be using SKIP_FETCH_OBJECT but not QUICK. Or
possibly it should use neither unless --missing is given.
So I don't see QUICK itself here violating any contract (even if it
_could_ find the object in some cases with just a little more work, as
in the case that we were discussing for v1).
The much more interesting case is the non-QUICK one that Patrick
outlined earlier in the thread. Where we say "nope, we don't have that
object" even though we could find it with a little more work. But that
doesn't seem to be described here either. But I think that is not even
what this patch is about; that's in patch 4.
If the "error:" message is scary and gross (especially because we may
retry and correct it anyway) and happens due to routine races, we might
consider suppressing it.
> + /*
> + * Set when a lookup finds that a pack we already know about has
> + * vanished -- its ".idx" or ".pack" removed out from under us, the
> + * signature of a concurrent "git repack". It tells
> + * odb_read_object_info_extended() to reprepare and retry even for an
> + * OBJECT_INFO_QUICK lookup, which normally skips that rescan to stay
> + * fast on a genuine miss. Reset when the packfiles are reprepared
> + * (see odb_source_packed_prepare()).
> + */
> + unsigned stale_packs_detected : 1;
So this is a way of hackily triggering SECOND_READ for QUICK queries,
even though the point of QUICK is to suppress that second read! Again,
maybe I'm just being dense, but I don't get it.
> @@ -535,8 +550,20 @@ static int open_packed_git_1(struct packed_git *p)
> ssize_t read_result;
> const unsigned hashsz = p->repo->hash_algo->rawsz;
>
> - if (open_pack_index(p))
> + if (open_pack_index(p)) {
> + /*
> + * A concurrent repack may have removed this pack, deleting its
> + * ".idx" before its ".pack" (see unlink_pack_path()). If the
> + * index simply vanished, note the stale pack set and stay
> + * quiet; the pack is still reported unusable. Only a
> + * still-present but unreadable index is worth an error.
> + */
> + if (pack_index_is_missing(p)) {
> + p->repo->objects->stale_packs_detected = 1;
> + return -1;
> + }
> return error("packfile %s index unavailable", p->pack_name);
> + }
And this seems racy. We might catch the .idx but miss the .pack file.
That would cause a failed read, but not trigger sale_packs_detected.
-Peff |
||
| if (!p) { | ||
| m->packs[pack_int_id] = MIDX_PACK_ERROR; | ||
| /* | ||
| * The midx names a pack we can no longer open (its files | ||
| * vanished, e.g. a concurrent repack replaced it). Record the | ||
| * stale pack set (see stale_packs_detected). | ||
| */ | ||
| packed->base.odb->stale_packs_detected = 1; | ||
| return 1; | ||
| } | ||
|
|
||
|
|
@@ -589,46 +595,50 @@ uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos) | |
| (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH); | ||
| } | ||
|
|
||
| int fill_midx_entry(struct multi_pack_index *m, | ||
| const struct object_id *oid, | ||
| struct pack_entry *e, | ||
| struct packed_git **bad_pack) | ||
| enum midx_fill_result fill_midx_entry(struct multi_pack_index *m, | ||
| const struct object_id *oid, | ||
| struct pack_entry *e, | ||
| struct packed_git **bad_pack) | ||
| { | ||
| uint32_t pos; | ||
| uint32_t pack_int_id; | ||
| struct packed_git *p; | ||
|
|
||
| if (!bsearch_midx(oid, m, &pos)) | ||
| return 0; | ||
| return MIDX_FILL_MISS; | ||
|
|
||
| midx_for_object(&m, pos); | ||
| pack_int_id = nth_midxed_pack_int_id(m, pos); | ||
|
|
||
| if (prepare_midx_pack(m, pack_int_id)) | ||
| return 0; | ||
| goto owner_unavailable; | ||
| p = m->packs[pack_int_id - m->num_packs_in_base]; | ||
|
|
||
| /* | ||
| * We are about to tell the caller where they can locate the | ||
| * requested object. We better make sure the packfile is | ||
| * still here and can be accessed before supplying that | ||
| * answer, as it may have been deleted since the MIDX was | ||
| * loaded! | ||
| */ | ||
| /* Make sure the pack is still present before pointing at it. */ | ||
| if (!is_pack_valid(p)) | ||
| return 0; | ||
| goto owner_unavailable; | ||
|
|
||
| if (oidset_size(&p->bad_objects) && | ||
| oidset_contains(&p->bad_objects, oid)) { | ||
| if (bad_pack && !*bad_pack) | ||
| *bad_pack = p; | ||
| return 0; | ||
| return MIDX_FILL_MISS; | ||
| } | ||
|
|
||
| e->offset = nth_midxed_offset(m, pos); | ||
| e->p = p; | ||
|
|
||
| return 1; | ||
| return MIDX_FILL_HIT; | ||
|
|
||
| owner_unavailable: | ||
| /* | ||
| * Re-arm stale_packs_detected on every such lookup, not just the | ||
| * first: prepare_midx_pack() caches the failure, so without this a | ||
| * later lookup of the same vanished pack would leave the flag clear | ||
| * and a QUICK reader would skip its recovering second read. | ||
| */ | ||
| m->source->base.odb->stale_packs_detected = 1; | ||
| return MIDX_FILL_OWNER_UNAVAILABLE; | ||
| } | ||
|
|
||
| /* Match "foo.idx" against either "foo.pack" _or_ "foo.idx". */ | ||
|
|
@@ -1032,7 +1042,7 @@ int verify_midx_file(struct odb_source_packed *source, unsigned flags) | |
|
|
||
| nth_midxed_object_oid(&oid, m, pairs[i].pos); | ||
|
|
||
| if (!fill_midx_entry(m, &oid, &e, NULL)) { | ||
| if (fill_midx_entry(m, &oid, &e, NULL) != MIDX_FILL_HIT) { | ||
| midx_report(_("failed to load pack entry for oid[%d] = %s"), | ||
| pairs[i].pos, oid_to_hex(&oid)); | ||
| continue; | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Jeff King wrote on the Git mailing list (how to reply to this email):