Restore per-coefficient bound substitutions before untransforming a path cut (#3171) - #3176
Closed
EamonHetherton wants to merge 2 commits into
Closed
Restore per-coefficient bound substitutions before untransforming a path cut (#3171)#3176EamonHetherton wants to merge 2 commits into
EamonHetherton wants to merge 2 commits into
Conversation
…anged substitution The model is solved incorrectly under default options: HiGHS reports Optimal at a zero gap with objective 42332.2356068, while a feasible integral point with objective 42215.5250005 exists. Fixing every integer column to the values of that better point and solving the remaining LP confirms it, so the optimum does not rest on trusting the branch and bound search. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ath cut HighsPathSeparator builds a path-mixing cut from several base rows, each transformed by its own HighsTransformedLp::transform() call, and maps the result back with a single untransform(). The chosen substitution per column lives in the shared boundTypes array, and untransform() reads whatever is in there when it runs. A base row is transformed before its right-hand side is checked for the required monotonicity, and dropped only afterwards, so a discarded row's mutation of boundTypes survives. Since the substitution chosen for a binary depends on the sign of its coefficient in the row being transformed, a discarded row can flip a binary's complementation, and untransform() then resubstitutes a different variable than the coefficients were computed against, producing a cut that is not globally valid. Record the substitution used for each index when its cut coefficient is computed and restore it immediately before untransform(). Refuse to build the cut when two base rows of one path disagree about a shared column, since then no single resubstitution is correct for all of the coefficients. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
|
Please see #3179 for fixing this issue. Interested to see if the AI agrees with the fix |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #3171.
The first commit adds the reproducer as a failing regression test, on its own, so the defect is
visible without any fix applied. The second commit contains the fix.
The model
check/instances/3171-1.mps— 85 rows, 99 columns, 49 binaries, 300 nonzeros. It is solvedincorrectly under default options, with no option pinning: HiGHS reports
OptimalatGap 0%with objective
42332.2356068against an optimum of42215.5250005, suboptimal by 116.71, arelative error of 0.276% — about 28x the default relative gap tolerance, so not a tolerance
artefact. It is decided at the root, in zero nodes.
The optimum does not rest on trusting the branch and bound search, or on trusting a patched build:
fixing all 49 integer columns to the values of the better point and solving the remaining LP with
unmodified HiGHS returns
Optimal 42215.5250005. So a feasible integral point 116.71 betterthan the reported optimum demonstrably exists.
--presolve offreturns42215.5250005as well,which means the contradiction is visible within a single unmodified binary.
The failure is seed-dependent — it occurs on seeds 0 and 5 of the first 8 — so the test pins
random_seedto 0 (the default) along withthreadsandparallel, to keep it off the scheduler.The defect
HighsPathSeparator's path-mixing cut is assembled from several base rows, each transformed by itsown
HighsTransformedLp::transform()call, and mapped back by a singleuntransform()at the end.HighsTransformedLpkeeps the chosen substitution per column in the sharedboundTypesarray, anduntransform()reads whatever is in there when it runs.The transform loop transforms base row
kbefore testing whether its right-hand side keeps therequired monotonicity, and drops the row only afterwards:
The discarded row's mutation survives. Since the substitution chosen for a binary depends on the
sign of its coefficient in the row being transformed, a discarded row can flip a binary's
complementation, and
untransform()then resubstitutes a different variable than the coefficientswere computed against. Nothing in the current code ties the two together.
The existing assertions in that loop cannot catch it:
solvalislbDist[col]for bothkSimpleLbandkVariableLbandubDist[col]for bothkSimpleUbandkVariableUb, andupperis
ub - lbin every case, so neither value changes when the substitution flips.The change
Record the substitution used for each index when its cut coefficient is computed, and restore those
substitutions immediately before calling
untransform(). Also refuse to build the cut when twobase rows of the same path disagree about a shared column, since then no single resubstitution is
correct for all of the coefficients.
This needs
BoundTypemade public onHighsTransformedLpplus aboundType()/setBoundType()accessor pair; the separator then records
transLp.boundType(index)alongsidesolval/upperwhen an index is first seen, flags a disagreement on later occurrences, and restores the recorded
values before untransforming.
Evidence on a production model
Besides the attached reproducer, this fires on a real 10404 x 18899 MIP, where it costs 2978.48 on
the objective. That model still returns the wrong answer with the fixes for #3170 and #3173 both
applied, so the three defects are independent:
presolve=onlatest-12370583.0955latest+ #3170 + #3173-12370583.0955latest+ #3170 + #3173 + this-12373561.5756presolve=offreturns the correct-12373561.5756throughout. The node count rises from 103 to376 with the fix, which is the search doing its job: without it an invalid cut was closing the tree
early.
Instrumenting that model gives the mismatch directly. Instrumenting the
separator to record the substitution used when each coefficient was computed, and compare it
against the one in force at
untransform(), gives exactly one mismatched column:Evaluating that cut in the transformed space at the known optimum gives violation
0; afteruntransform()it gives violation1. The arithmetic is otherwise sound — the entire error is theone flipped complementation. With the change, that model returns the correct optimum.
I can supply the model, or the instrumentation patch that makes the mismatch observable in a single
solve, if either would help.
What it took to find the model, and why that matters for review
Delta-debugging the large model does not work: deleting a single row with zero LP dual, which
provably cannot change the root LP optimum, is already enough to destroy the trigger, because the
path the separator walks depends on row and column indices and on hash-table iteration order.
Generating models is what worked, but only after the search was aimed at the actual mechanism.
Roughly 100,000 unshaped random MIPs produced no trigger at all. Three conditions turn out to gate
it, all readable from the code:
transform()is reached only whensimpleLbDistandsimpleUbDisttie, so any other LP valuepicks the bound type by distance and no two rows can disagree;
coefficient has to change sign between the row that records it and the row that is discarded;
Generating for those — odd cycles with equal rewards to pin binaries at 0.5, and alternating
near-cancelling coefficients so the running sum crosses zero — produced 5 substitution flips in
about 250,000 models, of which this is the one that also loses the optimum. The other four generate
the invalid cut without it cutting anything off, which is worth knowing: the defect fires more
often than it changes an answer.
That rarity is a caveat on the test, not on the defect. The test is one seed on one model, and the
conjunction it depends on is narrow, so it is a witness rather than broad coverage. The
inconsistentBoundTypesguard added here is what makes the general situation detectable ratherthan silent.
Adverse effects
Measured on
latest:check/instancesThe change can only suppress a cut, never produce a different one, so the risk is losing cuts that
were previously generated and valid. On the models above none is lost. I have not benchmarked
against MIPLIB or anything at production scale, so I cannot rule out a cut-quality cost where paths
of length two or more are common.
What this does not close
bestVub/bestVlbare shared mutable state as well, andtransform()may tighten them in placevia
cleanupVub()/cleanupVlb(), so a coefficient computed from an earlier base row can beuntransformed against a tightened bound. This restores the bound type but not the bound values.
I have no reproducer for that case and did not want to widen the change on speculation.
Both are the same design issue: the cut is untransformed by reading state back out of a shared,
still-mutating
HighsTransformedLp, rather than carrying the transformation it was built with. Ifyou would prefer a structural change along those lines, that would close this, the
bestVubcase,and #3170's class as well, and this PR is probably not the right starting point for it.
Testing
issue-3171test fails onlatestat the first commit and passes at the second;presolvesettings once fixed;🤖 Generated with Claude Code