Roman Lebedev 360691947c [PassManager] Run additional LICM before LoopRotate ... Loop rotation often has to perform code duplication from header into preheader, which introduces PHI nodes. >>! In D99204, @thopre wrote: > > With loop peeling, it is important that unnecessary PHIs be avoided or > it will leads to spurious peeling. One source of such PHIs is loop > rotation which creates PHIs for invariant loads. Those PHIs are > particularly problematic since loop peeling is now run as part of simple > loop unrolling before GVN is run, and are thus a source of spurious > peeling. > > Note that while some of the load can be hoisted and eventually > eliminated by instruction combine, this is not always possible due to > alignment issue. In particular, the motivating example [1] was a load > inside a class instance which cannot be hoisted because the `this' > pointer has an alignment of 1. > > [1] http://lists.llvm.org/pipermail/llvm-dev/attachments/20210312/4ce73c47/attachment.cpp Now, we could enhance LoopRotate to avoid duplicating code when not needed, but instead hoist loop-invariant code, but isn't that a code duplication? (*sic*) We have LICM, and in fact we already run it right after LoopRotation. We could try to move it to before LoopRotation, that is basically free from compile-time perspective: https://llvm-compile-time-tracker.com/compare.php?from=6c93eb4477d88af046b915bc955c03693b2cbb58&to=a4bee6d07732b1184c436da489040b912f0dc271&stat=instructions But, looking at stats, i think it isn't great that we would no longer do LICM after LoopRotation, in particular: | statistic name | LoopRotate-LICM | LICM-LoopRotate | Δ | % | abs(%) | | asm-printer.EmittedInsts | 9015930 | 9015799 | -131 | 0.00% | 0.00% | | indvars.NumElimCmp | 3536 | 3544 | 8 | 0.23% | 0.23% | | indvars.NumElimExt | 36725 | 36580 | -145 | -0.39% | 0.39% | | indvars.NumElimIV | 1197 | 1187 | -10 | -0.84% | 0.84% | | indvars.NumElimIdentity | 143 | 136 | -7 | -4.90% | 4.90% | | indvars.NumElimRem | 4 | 5 | 1 | 25.00% | 25.00% | | indvars.NumLFTR | 29842 | 29890 | 48 | 0.16% | 0.16% | | indvars.NumReplaced | 2293 | 2227 | -66 | -2.88% | 2.88% | | indvars.NumSimplifiedSDiv | 6 | 8 | 2 | 33.33% | 33.33% | | indvars.NumWidened | 26438 | 26329 | -109 | -0.41% | 0.41% | | instcount.TotalBlocks | 1178338 | 1173840 | -4498 | -0.38% | 0.38% | | instcount.TotalFuncs | 111825 | 111829 | 4 | 0.00% | 0.00% | | instcount.TotalInsts | 9905442 | 9896139 | -9303 | -0.09% | 0.09% | | lcssa.NumLCSSA | 425871 | 423961 | -1910 | -0.45% | 0.45% | | licm.NumHoisted | 378357 | 378753 | 396 | 0.10% | 0.10% | | licm.NumMovedCalls | 2193 | 2208 | 15 | 0.68% | 0.68% | | licm.NumMovedLoads | 35899 | 31821 | -4078 | -11.36% | 11.36% | | licm.NumPromoted | 11178 | 11154 | -24 | -0.21% | 0.21% | | licm.NumSunk | 13359 | 13587 | 228 | 1.71% | 1.71% | | loop-delete.NumDeleted | 8547 | 8402 | -145 | -1.70% | 1.70% | | loop-instsimplify.NumSimplified | 12876 | 11890 | -986 | -7.66% | 7.66% | | loop-peel.NumPeeled | 1008 | 925 | -83 | -8.23% | 8.23% | | loop-rotate.NumNotRotatedDueToHeaderSize | 368 | 365 | -3 | -0.82% | 0.82% | | loop-rotate.NumRotated | 42015 | 42003 | -12 | -0.03% | 0.03% | | loop-simplifycfg.NumLoopBlocksDeleted | 240 | 242 | 2 | 0.83% | 0.83% | | loop-simplifycfg.NumLoopExitsDeleted | 497 | 20 | -477 | -95.98% | 95.98% | | loop-simplifycfg.NumTerminatorsFolded | 618 | 336 | -282 | -45.63% | 45.63% | | loop-unroll.NumCompletelyUnrolled | 11028 | 11032 | 4 | 0.04% | 0.04% | | loop-unroll.NumUnrolled | 12608 | 12529 | -79 | -0.63% | 0.63% | | mem2reg.NumDeadAlloca | 10222 | 10221 | -1 | -0.01% | 0.01% | | mem2reg.NumPHIInsert | 192110 | 192106 | -4 | 0.00% | 0.00% | | mem2reg.NumSingleStore | 637650 | 637643 | -7 | 0.00% | 0.00% | | scalar-evolution.NumBruteForceTripCountsComputed | 814 | 812 | -2 | -0.25% | 0.25% | | scalar-evolution.NumTripCountsComputed | 283108 | 282934 | -174 | -0.06% | 0.06% | | scalar-evolution.NumTripCountsNotComputed | 106712 | 106718 | 6 | 0.01% | 0.01% | | simple-loop-unswitch.NumBranches | 5178 | 4752 | -426 | -8.23% | 8.23% | | simple-loop-unswitch.NumCostMultiplierSkipped | 914 | 503 | -411 | -44.97% | 44.97% | | simple-loop-unswitch.NumSwitches | 20 | 18 | -2 | -10.00% | 10.00% | | simple-loop-unswitch.NumTrivial | 183 | 95 | -88 | -48.09% | 48.09% | ... but that actually regresses LICM (-12% `licm.NumMovedLoads`), loop-simplifycfg (`NumLoopExitsDeleted`, `NumTerminatorsFolded`), simple-loop-unswitch (`NumTrivial`). What if we instead have LICM both before and after LoopRotate? | statistic name | LoopRotate-LICM | LICM-LoopRotate-LICM | Δ | % | abs(%) | | asm-printer.EmittedInsts | 9015930 | 9014474 | -1456 | -0.02% | 0.02% | | indvars.NumElimCmp | 3536 | 3546 | 10 | 0.28% | 0.28% | | indvars.NumElimExt | 36725 | 36681 | -44 | -0.12% | 0.12% | | indvars.NumElimIV | 1197 | 1185 | -12 | -1.00% | 1.00% | | indvars.NumElimIdentity | 143 | 146 | 3 | 2.10% | 2.10% | | indvars.NumElimRem | 4 | 5 | 1 | 25.00% | 25.00% | | indvars.NumLFTR | 29842 | 29899 | 57 | 0.19% | 0.19% | | indvars.NumReplaced | 2293 | 2299 | 6 | 0.26% | 0.26% | | indvars.NumSimplifiedSDiv | 6 | 8 | 2 | 33.33% | 33.33% | | indvars.NumWidened | 26438 | 26404 | -34 | -0.13% | 0.13% | | instcount.TotalBlocks | 1178338 | 1173652 | -4686 | -0.40% | 0.40% | | instcount.TotalFuncs | 111825 | 111829 | 4 | 0.00% | 0.00% | | instcount.TotalInsts | 9905442 | 9895452 | -9990 | -0.10% | 0.10% | | lcssa.NumLCSSA | 425871 | 425373 | -498 | -0.12% | 0.12% | | licm.NumHoisted | 378357 | 383352 | 4995 | 1.32% | 1.32% | | licm.NumMovedCalls | 2193 | 2204 | 11 | 0.50% | 0.50% | | licm.NumMovedLoads | 35899 | 35755 | -144 | -0.40% | 0.40% | | licm.NumPromoted | 11178 | 11163 | -15 | -0.13% | 0.13% | | licm.NumSunk | 13359 | 14321 | 962 | 7.20% | 7.20% | | loop-delete.NumDeleted | 8547 | 8538 | -9 | -0.11% | 0.11% | | loop-instsimplify.NumSimplified | 12876 | 12041 | -835 | -6.48% | 6.48% | | loop-peel.NumPeeled | 1008 | 924 | -84 | -8.33% | 8.33% | | loop-rotate.NumNotRotatedDueToHeaderSize | 368 | 365 | -3 | -0.82% | 0.82% | | loop-rotate.NumRotated | 42015 | 42005 | -10 | -0.02% | 0.02% | | loop-simplifycfg.NumLoopBlocksDeleted | 240 | 241 | 1 | 0.42% | 0.42% | | loop-simplifycfg.NumTerminatorsFolded | 618 | 619 | 1 | 0.16% | 0.16% | | loop-unroll.NumCompletelyUnrolled | 11028 | 11029 | 1 | 0.01% | 0.01% | | loop-unroll.NumUnrolled | 12608 | 12525 | -83 | -0.66% | 0.66% | | mem2reg.NumPHIInsert | 192110 | 192073 | -37 | -0.02% | 0.02% | | mem2reg.NumSingleStore | 637650 | 637652 | 2 | 0.00% | 0.00% | | scalar-evolution.NumTripCountsComputed | 283108 | 282998 | -110 | -0.04% | 0.04% | | scalar-evolution.NumTripCountsNotComputed | 106712 | 106691 | -21 | -0.02% | 0.02% | | simple-loop-unswitch.NumBranches | 5178 | 5185 | 7 | 0.14% | 0.14% | | simple-loop-unswitch.NumCostMultiplierSkipped | 914 | 925 | 11 | 1.20% | 1.20% | | simple-loop-unswitch.NumTrivial | 183 | 179 | -4 | -2.19% | 2.19% | | simple-loop-unswitch.NumBranches | 5178 | 4752 | -426 | -8.23% | 8.23% | | simple-loop-unswitch.NumCostMultiplierSkipped | 914 | 503 | -411 | -44.97% | 44.97% | | simple-loop-unswitch.NumSwitches | 20 | 18 | -2 | -10.00% | 10.00% | | simple-loop-unswitch.NumTrivial | 183 | 95 | -88 | -48.09% | 48.09% | I.e. we end up with less instructions, less peeling, more LICM activity, also note how none of those 4 regressions are here. Namely: | statistic name | LICM-LoopRotate | LICM-LoopRotate-LICM | Δ | % | abs(%) | | asm-printer.EmittedInsts | 9015799 | 9014474 | -1325 | -0.01% | 0.01% | | indvars.NumElimCmp | 3544 | 3546 | 2 | 0.06% | 0.06% | | indvars.NumElimExt | 36580 | 36681 | 101 | 0.28% | 0.28% | | indvars.NumElimIV | 1187 | 1185 | -2 | -0.17% | 0.17% | | indvars.NumElimIdentity | 136 | 146 | 10 | 7.35% | 7.35% | | indvars.NumLFTR | 29890 | 29899 | 9 | 0.03% | 0.03% | | indvars.NumReplaced | 2227 | 2299 | 72 | 3.23% | 3.23% | | indvars.NumWidened | 26329 | 26404 | 75 | 0.28% | 0.28% | | instcount.TotalBlocks | 1173840 | 1173652 | -188 | -0.02% | 0.02% | | instcount.TotalInsts | 9896139 | 9895452 | -687 | -0.01% | 0.01% | | lcssa.NumLCSSA | 423961 | 425373 | 1412 | 0.33% | 0.33% | | licm.NumHoisted | 378753 | 383352 | 4599 | 1.21% | 1.21% | | licm.NumMovedCalls | 2208 | 2204 | -4 | -0.18% | 0.18% | | licm.NumMovedLoads | 31821 | 35755 | 3934 | 12.36% | 12.36% | | licm.NumPromoted | 11154 | 11163 | 9 | 0.08% | 0.08% | | licm.NumSunk | 13587 | 14321 | 734 | 5.40% | 5.40% | | loop-delete.NumDeleted | 8402 | 8538 | 136 | 1.62% | 1.62% | | loop-instsimplify.NumSimplified | 11890 | 12041 | 151 | 1.27% | 1.27% | | loop-peel.NumPeeled | 925 | 924 | -1 | -0.11% | 0.11% | | loop-rotate.NumRotated | 42003 | 42005 | 2 | 0.00% | 0.00% | | loop-simplifycfg.NumLoopBlocksDeleted | 242 | 241 | -1 | -0.41% | 0.41% | | loop-simplifycfg.NumLoopExitsDeleted | 20 | 497 | 477 | 2385.00% | 2385.00% | | loop-simplifycfg.NumTerminatorsFolded | 336 | 619 | 283 | 84.23% | 84.23% | | loop-unroll.NumCompletelyUnrolled | 11032 | 11029 | -3 | -0.03% | 0.03% | | loop-unroll.NumUnrolled | 12529 | 12525 | -4 | -0.03% | 0.03% | | mem2reg.NumDeadAlloca | 10221 | 10222 | 1 | 0.01% | 0.01% | | mem2reg.NumPHIInsert | 192106 | 192073 | -33 | -0.02% | 0.02% | | mem2reg.NumSingleStore | 637643 | 637652 | 9 | 0.00% | 0.00% | | scalar-evolution.NumBruteForceTripCountsComputed | 812 | 814 | 2 | 0.25% | 0.25% | | scalar-evolution.NumTripCountsComputed | 282934 | 282998 | 64 | 0.02% | 0.02% | | scalar-evolution.NumTripCountsNotComputed | 106718 | 106691 | -27 | -0.03% | 0.03% | | simple-loop-unswitch.NumBranches | 4752 | 5185 | 433 | 9.11% | 9.11% | | simple-loop-unswitch.NumCostMultiplierSkipped | 503 | 925 | 422 | 83.90% | 83.90% | | simple-loop-unswitch.NumSwitches | 18 | 20 | 2 | 11.11% | 11.11% | | simple-loop-unswitch.NumTrivial | 95 | 179 | 84 | 88.42% | 88.42% | {F15983613} {F15983615} {F15983616} (this is vanilla llvm testsuite + rawspeed + darktable) As an example of the code where early LICM only is bad, see: https://godbolt.org/z/GzEbacs4K This does have an observable compile-time regression of +~0.5% geomean https://llvm-compile-time-tracker.com/compare.php?from=7c5222e4d1a3a14f029e5f614c9aefd0fa505f1e&to=5d81826c3411982ca26e46b9d0aff34c80577664&stat=instructions but i think that's basically nothing, and there's potential that it might be avoidable in the future by fixing clang to produce alignment information on function arguments, thus making the second run unneeded. Differential Revision: https://reviews.llvm.org/D99249		2021-04-02 11:11:42 +03:00
..
AMDGPU
ARM	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
NVPTX
X86	[PassManager] Run additional LICM before LoopRotate	2021-04-02 11:11:42 +03:00
2002-09-09-PointerIndVar.ll
2003-04-16-ExprAnalysis.ll
2003-09-23-NotAtTop.ll
2003-12-10-RemoveInstrCrash.ll
2003-12-15-Crash.ll
2004-03-10-PHIInsertionBug.ll
2004-04-05-InvokeCastCrash.ll
2004-04-07-ScalarEvolutionCrash.ll
2005-02-11-InvokeCrash.ll
2005-02-17-TruncateExprCrash.ll
2005-02-26-ExitValueCompute.ll
2005-06-15-InstMoveCrash.ll
2005-11-18-Crash.ll
2006-03-31-NegativeStride.ll
2006-06-16-Indvar-LCSSA-Crash.ll
2006-09-20-LFTR-Crash.ll
2006-12-10-BitCast.ll
2007-01-06-TripCount.ll
2007-06-06-DeleteDanglesPtr.ll
2007-11-23-BitcastCrash.ll
2008-06-15-SCEVExpanderBug.ll
2008-09-02-IVType.ll
2008-10-03-CouldNotCompute.ll
2008-11-25-APFloatAssert.ll
2009-04-22-IndvarCrash.ll
2009-04-27-Floating.ll
2009-05-24-useafterfree.ll
2011-09-10-widen-nsw.ll
2011-09-19-vectoriv.ll
2011-09-27-hoistsext.ll
2011-10-27-lftrnull.ll	[SCEV] Add false->any implication	2021-03-19 11:29:48 +07:00
2011-11-01-lftrptr.ll	Reland [SCEV] Improve modelling for (null) pointer constants	2021-03-13 16:05:34 +03:00
2011-11-17-selfphi.ll
2012-10-19-congruent-constant.ll
2014-06-21-congruent-constant.ll
2020-12-15-trunc-bug-expensive-range-inference.ll	[SCEV] Add missing type check into getRangeForAffineNoSelfWrappingAR	2020-12-15 14:50:32 +07:00
ada-loops.ll
addrec_no_exec_on_every_iteration.ll
ashr-tripcount.ll
avoid-i0.ll
backedge-on-min-max.ll
bec-cmp.ll
canonicalize-cmp.ll
casted-argument.ll
checks_against_min_value.ll	[IndVars] Provide eliminateIVComparison with context	2021-03-19 12:28:22 +07:00
const_phi.ll
constant_result.ll
constant-fold.ll
crash.ll
dangling-use.ll
divide-pointer.ll
dont-recompute.ll
drop-exact.ll
elim-extend.ll	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
eliminate-comparison.ll	[IndVars] Sharpen context in eliminateIVComparison	2021-03-22 11:55:57 +07:00
eliminate-exit-no-dl.ll
eliminate-exit.ll
eliminate-max.ll
eliminate-overflow-modified.ll
eliminate-rem.ll
eliminate-sat.ll
exit_value_test2.ll	[LoopDeletion] Break backedge of outermost loops when known not taken	2021-01-10 16:02:33 -08:00
exit_value_test3.ll
exit_value_tests.ll	[SimplifyCFG] MergeBlockIntoPredecessor() already knows how to preserve DomTree	2020-12-17 01:03:49 +03:00
floating-point-iv.ll
full_widening.ll	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
indirectbr.ll
infer-poison-flags.ll
interesting-invoke-use.ll
iterationCount_zext_or_trunc.ll
iv-fold.ll
iv-sext.ll
iv-widen-elim-ext.ll
iv-zext.ll
lcssa-preservation.ll	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
lftr_disabled.ll
lftr-address-space-pointers.ll
lftr-dead-ivs.ll
lftr-multi-exit.ll	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
lftr-pr20680.ll
lftr-pr31181.ll
lftr-pr41998.ll
lftr-promote.ll
lftr-reuse.ll	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
lftr.ll
loop_evaluate7.ll
loop_evaluate8.ll
loop_evaluate9.ll
loop_evaluate11.ll
loop_evaluate_1.ll	[SimplifyCFG] Teach FoldTwoEntryPHINode() to preserve DomTree	2020-12-20 00:18:33 +03:00
loop_evaluate_2.ll	[SimplifyCFG] Teach FoldTwoEntryPHINode() to preserve DomTree	2020-12-20 00:18:33 +03:00
loop_evaluate_3.ll
loop_evaluate_4.ll
loop_evaluate_5.ll
loop_evaluate_6.ll
loop-predication.ll	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
lrev-existing-umin.ll
masked-iv.ll
monotonic_checks.ll
no-iv-rewrite.ll	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
phi-uses-value-multiple-times.ll
post-inc-range.ll
pr18223.ll
pr22222.ll
pr24783.ll
pr24952.ll
pr25060.ll
pr25578.ll
pr26974.ll
pr28705.ll
pr32045.ll
pr38674.ll
pr38855.ll
pr39673.ll
pr40454.ll
pr45835.ll
predicated_ranges.ll
preserve-memoryssa.ll
preserve-signed-wrap.ll
promote-iv-to-eliminate-casts.ll
replace-loop-exit-folds.ll	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
replace-sdiv-by-udiv.ll
replace-srem-by-urem.ll
rewrite-loop-exit-value.ll
rewrite-loop-exit-values-phi.ll
rlev-add-me.ll
scev-expander-preserve-lcssa.ll	[test] Fix scev-expander-preserve-lcssa.ll under NPM	2020-12-10 09:46:08 -08:00
scevexpander-phi-base-case.ll
sentinel.ll	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
shared-exit-between-nested-loop.ll
sharpen-range.ll
shrunk-constant.ll
signed-trip-count.ll
simplify-pointer-arithmetic.ll	[IndVars] Add test cases inspired by PR48965.	2021-02-25 15:54:18 +00:00
single-element-range.ll
sink-alloca.ll
sink-from-preheader.ll
sink-trapping.ll
strengthen-overflow.ll
tripcount_compute.ll
tripcount_infinite.ll	[NFCI][SimplifyCFG] Mark all the SimplifyCFG tests that already don't invalidate DomTree as such	2020-12-17 01:03:49 +03:00
trivial-checks.ll
trivial-guard.ll	[SCEV] Add false->any implication	2021-03-19 11:29:48 +07:00
udiv-invariant-but-traps.ll
udiv.ll
ult-sub-to-eq.ll
use-range-metadata.ll
variable-stride-ivs-0.ll
widen-i32-i8ptr.ll	Reland [SCEV] Improve modelling for (null) pointer constants	2021-03-13 16:05:34 +03:00
widen-loop-comp.ll	[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions	2021-03-06 21:52:46 +03:00
zext-nuw.ll