Anna Thomas b71aa89415 [LV] Teach about non header phis that have uses outside the loop ...

Summary:
This patch teaches the loop vectorizer to vectorize loops with non
header phis that have have outside uses.  This is because the iteration
dependence distance for these phis can be widened upto VF (similar to
how we do for induction/reduction) if they do not have a cyclic
dependence with header phis. When identifying reduction/induction/first
order recurrence header phis, we already identify if there are any cyclic
dependencies that prevents vectorization.

The vectorizer is taught to extract the last element from the vectorized
phi and update the scalar loop exit block phi to contain this extracted
element from the vector loop.

This patch can be extended to vectorize loops where instructions other
than phis have outside uses.

Reviewers: Ayal, mkuper, mssimpso, efriedma

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D50579

llvm-svn: 339703

2018-08-14 18:22:19 +00:00

..

AArch64

[AArch64] Add custom lowering for v4i8 trunc store

2018-06-27 13:58:46 +00:00

AMDGPU

LV: Don't insert runtime ptr checks on divergent targets

2017-08-02 21:43:08 +00:00

ARM

[LV] Preserve inbounds on created GEPs

2018-05-01 15:35:08 +00:00

Hexagon

[LV] Introduce TTI::getMinimumVF

2018-04-13 20:16:32 +00:00

PowerPC

[ppc] Correctly compute the cost of loading 32/64 bit memory into VSR

2016-12-03 00:41:43 +00:00

SystemZ

[LV] Fix for PR38110, LV encountered llvm_unreachable()

2018-07-24 22:30:31 +00:00

X86

Revert "[InstCombine] Delay foldICmpUsingKnownBits until simple transforms are done"

2018-07-06 04:04:13 +00:00

XCore

…

12-12-11-if-conv.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

2012-10-20-infloop.ll

…

2012-10-22-isconsec.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

2016-07-27-loop-vec.ll

Fix the assertion error in collectLoopUniforms caused by empty Worklist before expanding.

2016-07-27 23:53:58 +00:00

align.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

bsd_regex.ll

…

bzip_reverse_loops.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

calloc.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

cast-induction.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

conditional-assignment.ll

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

2018-05-09 02:40:45 +00:00

consec_no_gep.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

consecutive-ptr-uniforms.ll

[LV] Preserve inbounds on created GEPs

2018-05-01 15:35:08 +00:00

control-flow.ll

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

2018-05-09 02:40:45 +00:00

cpp-new-array.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

dbg.value.ll

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

2018-05-09 02:40:45 +00:00

dead_instructions.ll

[LV] Avoid emitting trivially dead instructions

2016-10-19 19:22:02 +00:00

debugloc.ll

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

2018-05-09 02:40:45 +00:00

diag-missing-instr-debug-loc.ll

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

2018-05-09 02:40:45 +00:00

diag-with-hotness-info-2.ll

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

2018-05-09 02:40:45 +00:00

diag-with-hotness-info.ll

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

2018-05-09 02:40:45 +00:00

discriminator.ll

Do not add discriminator encoding for debug intrinsics.

2017-10-26 21:20:52 +00:00

ee-crash.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

exact.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

explicit_outer_detection.ll

[LV][VPlan] Detect outer loops for explicit vectorization.

2018-04-24 17:04:17 +00:00

explicit_outer_nonuniform_inner.ll

[LV][VPlan] Detect outer loops for explicit vectorization.

2018-04-24 17:04:17 +00:00

explicit_outer_uniform_diverg_branch.ll

[LV][VPlan] Detect outer loops for explicit vectorization.

2018-04-24 17:04:17 +00:00

fcmp-vectorize.ll

[LoopVectorize] Fix assertion failure in Fcmp vectorization

2017-08-08 18:07:44 +00:00

first-order-recurrence.ll

[LV] Fix PR34711 - widen instruction ranges when sinking casts

2017-10-05 12:41:49 +00:00

flags.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

float-induction.ll

[LV] Preserve inbounds on created GEPs

2018-05-01 15:35:08 +00:00

float-reduction.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

funcall.ll

…

gcc-examples.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

gep_with_bitcast.ll

[LV] Don't widen trivial induction variables

2016-07-06 14:26:59 +00:00

global_alias.ll

[LV] Scalarize instructions marked scalar after vectorization

2016-09-26 17:08:37 +00:00

hints-trans.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

hoist-loads.ll

[LV] Move isLegalMasked* functions from Legality to CostModel

2018-02-26 11:06:36 +00:00

i8-induction.ll

[DebugInfo][LoopVectorize] Preserve DL in induction PHI and Add

2018-07-10 13:29:50 +00:00

icmp-uniforms.ll

[LV] Don't mark multi-use branch conditions uniform

2016-10-07 15:20:13 +00:00

if-conv-crash.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

if-conversion-edgemasks.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

if-conversion-nest.ll

[LV] Model masking in VPlan, introducing VPInstructions

2017-11-20 12:01:47 +00:00

if-conversion-reduction.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

if-conversion.ll

[LV] Fix PR34523 - avoid generating redundant selects

2017-09-13 06:28:37 +00:00

if-pred-non-void.ll

[LV] Fix PR34248 - recommit D32871 after revert r311304

2017-08-27 12:55:46 +00:00

if-pred-not-when-safe.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

if-pred-stores.ll

[LV] Avoid redundant operations manipulating masks

2017-07-31 13:21:42 +00:00

incorrect-dom-info.ll

…

increment.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

induction_plus.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

induction-step.ll

[LV] Preserve inbounds on created GEPs

2018-05-01 15:35:08 +00:00

induction.ll

[LV] Preserve inbounds on created GEPs

2018-05-01 15:35:08 +00:00

infiniteloop.ll

…

int_sideeffect.ll

Add an @llvm.sideeffect intrinsic

2017-11-08 21:59:51 +00:00

interleaved-accesses-1.ll

Second attempt at r285517.

2016-10-31 13:17:31 +00:00

interleaved-accesses-2.ll

Second attempt at r285517.

2016-10-31 13:17:31 +00:00

interleaved-accesses-3.ll

Second attempt at r285517.

2016-10-31 13:17:31 +00:00

interleaved-accesses-alias.ll

[LV] Interleaved access vectorization: fix computing new alias info

2017-12-06 22:42:24 +00:00

interleaved-accesses-pred-stores.ll

[LV] Test once if vector trip count is zero, instead of twice

2017-07-19 05:16:39 +00:00

interleaved-accesses.ll

[LV] Preserve inbounds on created GEPs

2018-05-01 15:35:08 +00:00

interleaved-acess-with-remarks.ll

[LV] Fix analyzeInterleaving when -pass-remarks enabled

2018-02-10 00:07:45 +00:00

intrinsic.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

iv_outside_user.ll

[LV] Test once if vector trip count is zero, instead of twice

2017-07-19 05:16:39 +00:00

lcssa-crash.ll

[LV] Fix an issue where forming LCSSA in the place that we did would

2017-01-26 10:41:09 +00:00

legal_preheader_check.ll

[LV] Remove unnecessary DoExtraAnalysis guard (silent bug)

2017-12-20 13:28:38 +00:00

lifetime.ll

Add address space mangling to lifetime intrinsics

2017-04-10 20:18:21 +00:00

loop-form.ll

…

loop-scalars.ll

Reapply r298620: [LV] Vectorize GEPs

2017-04-07 14:15:34 +00:00

loop-vect-memdep.ll

…

memdep.ll

[LV] Fix maximum legal VF calculation

2017-09-14 07:40:02 +00:00

metadata-unroll.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

metadata-width.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

metadata.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

miniters.ll

[LV] Test once if vector trip count is zero, instead of twice

2017-07-19 05:16:39 +00:00

minmax_reduction.ll

[InstCombine] remove extract-of-select vector transform (2nd try)

2017-09-25 20:30:53 +00:00

multi-use-reduction-bug.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

multiple-address-spaces.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

multiple-strides-vectorization.ll

[Loop Vectorize] Added a separate metadata

2017-08-20 10:32:41 +00:00

no_array_bounds.ll

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

2018-05-09 02:40:45 +00:00

no_idiv_reduction.ll

…

no_int_induction.ll

…

no_outside_user.ll

[LV] Teach about non header phis that have uses outside the loop

2018-08-14 18:22:19 +00:00

no_switch.ll

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

2018-05-09 02:40:45 +00:00

noalias-md-licm.ll

…

noalias-md.ll

…

nofloat.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

non-const-n.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

nontemporal.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

nsw-crash.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

opt.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

optsize.ll

…

partial-lcssa.ll

[LV] Run loop-simplify and LCSSA explicitly instead of "requiring" them

2017-01-19 00:42:28 +00:00

phi-cost.ll

[InstCombine] Teach select01 helper of foldSelectIntoOp to handle vector splats

2017-08-28 22:00:27 +00:00

phi-hang.ll

…

pr25281.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

pr28541.ll

[LoopVectorize] Detect loops in the innermost loop before creating InnerLoopVectorizer

2016-08-12 22:47:13 +00:00

pr30654-phiscev-sext-trunc.ll

Revert r325687 (workaround for PR36032).

2018-03-22 22:04:39 +00:00

pr31098.ll

[LV/LoopAccess] Check statically if an unknown dependence distance can be

2017-02-12 09:32:53 +00:00

pr31190.ll

[LV] Run loop-simplify and LCSSA explicitly instead of "requiring" them

2017-01-19 00:42:28 +00:00

pr32859.ll

[LoopOptimizer][Fix]PR32859, PR24738

2017-05-13 13:25:57 +00:00

pr33706.ll

[LV] Don't allow outside uses of IVs if the SCEV is predicated on loop conditions.

2017-07-12 19:53:55 +00:00

pr34681.ll

[LV/LAA] Avoid specializing a loop for stride=1 when this predicate implies a

2017-11-05 16:53:15 +00:00

pr35743.ll

[NFC] Add tests for PR35743

2018-02-05 08:09:49 +00:00

pr35773.ll

Revert r325687 (workaround for PR36032).

2018-03-22 22:04:39 +00:00

pr36311.ll

[LV] Adding test for r327109

2018-03-09 18:02:36 +00:00

pr36983.ll

[LV] Fix PR36983. For a given recurrence, fix all phis in exit block

2018-06-08 08:21:20 +00:00

pr37248.ll

[LV] Fix for PR37248, Broadcast codegen incorrectly assumed vector loop body is single basic block

2018-05-08 18:57:34 +00:00

preserve-dbg-loc-and-loop-metadata.ll

[DebugInfo][LoopVectorize] Preserve DL in generated phi instruction

2018-07-04 10:16:55 +00:00

ptr_loops.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

ptr-induction.ll

Reverted patch 273864

2016-06-29 10:01:06 +00:00

read-only.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

reduction-small-size.ll

[LV] Use Demanded Bits and ValueTracking for reduction type-shrinking

2018-02-04 15:42:24 +00:00

reduction.ll

NFC: Add a test to LV showing that reduction is not possible when reduction var is reset in the loop

2018-08-13 19:55:25 +00:00

reverse_induction.ll

[LV] Don't emit unused scalars for uniform instructions

2016-09-21 16:50:24 +00:00

reverse_iter.ll

Reapply "[LV] Extend trunc optimization to all IVs with constant integer steps"

2017-02-14 16:28:32 +00:00

runtime-check-address-space.ll

…

runtime-check-readonly-address-space.ll

…

runtime-check-readonly.ll

[LV] Test once if vector trip count is zero, instead of twice

2017-07-19 05:16:39 +00:00

runtime-check.ll

[LoopVectorize] regenerate full checks; NFC

2018-06-21 16:54:32 +00:00

runtime-limit.ll

…

safegep.ll

…

same-base-access.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

scalar_after_vectorization.ll

[LV] Preserve inbounds on created GEPs

2018-05-01 15:35:08 +00:00

scalar-select.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

scev-exitlim-crash.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

simple-unroll.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

small-loop.ll

[LV] Optimize for size when vectorizing loops with tiny trip count

2017-06-30 08:02:35 +00:00

start-non-zero.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

store-shuffle-bug.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

struct_access.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

tbaa-nodep.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

tripcount.ll

Verify profile data confirms large loop trip counts.

2018-02-07 23:29:52 +00:00

undef-inst-bug.ll

…

unroll_novec.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

unroll-novec-memcheck-metadata.ll

[LV] Add regression test for r297610

2017-03-23 20:02:23 +00:00

unroll.ll

…

unsafe-dep-remark.ll

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

2018-05-09 02:40:45 +00:00

unsized-pointee-crash.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

value-ptr-bug.ll

…

vect-phiscev-sext-trunc.ll

Revert r325687 (workaround for PR36032).

2018-03-22 22:04:39 +00:00

vect.omp.persistence.ll

[tests] Cleanup vect.omp.persistence.ll test.

2017-07-25 10:35:16 +00:00

vect.stats.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

vector-geps.ll

Reapply r298620: [LV] Vectorize GEPs

2017-04-07 14:15:34 +00:00

vectorize-once.ll

[Loop Vectorize] Added a separate metadata

2017-08-20 10:32:41 +00:00

version-mem-access.ll

[LV/LAA] Avoid specializing a loop for stride=1 when this predicate implies a

2017-11-05 16:53:15 +00:00

vplan_hcfg_stress_test.ll

[VPlan] Reland r332654 and silence unused func warning

2018-05-21 18:14:23 +00:00

write-only.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00

zero-sized-pointee-crash.ll

[LV] Remove triples from target-independent vectorizer tests. NFC.

2016-10-06 23:57:25 +00:00