2016-02-18 22:38:19 +01:00
|
|
|
//===-------- LoopDataPrefetch.cpp - Loop Data Prefetching Pass -----------===//
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
//
|
2019-01-19 09:50:56 +01:00
|
|
|
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
|
|
|
// See https://llvm.org/LICENSE.txt for license information.
|
|
|
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
//
|
|
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
|
//
|
|
|
|
|
// This file implements a Loop Data Prefetching Pass.
|
|
|
|
|
//
|
|
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
|
|
2016-08-13 06:11:27 +02:00
|
|
|
#include "llvm/Transforms/Scalar/LoopDataPrefetch.h"
|
Sink all InitializePasses.h includes
This file lists every pass in LLVM, and is included by Pass.h, which is
very popular. Every time we add, remove, or rename a pass in LLVM, it
caused lots of recompilation.
I found this fact by looking at this table, which is sorted by the
number of times a file was changed over the last 100,000 git commits
multiplied by the number of object files that depend on it in the
current checkout:
recompiles touches affected_files header
342380 95 3604 llvm/include/llvm/ADT/STLExtras.h
314730 234 1345 llvm/include/llvm/InitializePasses.h
307036 118 2602 llvm/include/llvm/ADT/APInt.h
213049 59 3611 llvm/include/llvm/Support/MathExtras.h
170422 47 3626 llvm/include/llvm/Support/Compiler.h
162225 45 3605 llvm/include/llvm/ADT/Optional.h
158319 63 2513 llvm/include/llvm/ADT/Triple.h
140322 39 3598 llvm/include/llvm/ADT/StringRef.h
137647 59 2333 llvm/include/llvm/Support/Error.h
131619 73 1803 llvm/include/llvm/Support/FileSystem.h
Before this change, touching InitializePasses.h would cause 1345 files
to recompile. After this change, touching it only causes 550 compiles in
an incremental rebuild.
Reviewers: bkramer, asbirlea, bollu, jdoerfert
Differential Revision: https://reviews.llvm.org/D70211
2019-11-13 22:15:01 +01:00
|
|
|
#include "llvm/InitializePasses.h"
|
2016-08-13 06:11:27 +02:00
|
|
|
|
2015-04-10 17:05:02 +02:00
|
|
|
#include "llvm/ADT/DepthFirstIterator.h"
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
#include "llvm/ADT/Statistic.h"
|
2016-12-19 09:22:17 +01:00
|
|
|
#include "llvm/Analysis/AssumptionCache.h"
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
#include "llvm/Analysis/CodeMetrics.h"
|
|
|
|
|
#include "llvm/Analysis/LoopInfo.h"
|
2017-10-10 01:19:02 +02:00
|
|
|
#include "llvm/Analysis/OptimizationRemarkEmitter.h"
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
#include "llvm/Analysis/ScalarEvolution.h"
|
|
|
|
|
#include "llvm/Analysis/ScalarEvolutionExpressions.h"
|
|
|
|
|
#include "llvm/Analysis/TargetTransformInfo.h"
|
|
|
|
|
#include "llvm/IR/CFG.h"
|
|
|
|
|
#include "llvm/IR/Dominators.h"
|
|
|
|
|
#include "llvm/IR/Function.h"
|
|
|
|
|
#include "llvm/IR/Module.h"
|
|
|
|
|
#include "llvm/Support/CommandLine.h"
|
|
|
|
|
#include "llvm/Support/Debug.h"
|
2016-07-23 00:53:12 +02:00
|
|
|
#include "llvm/Transforms/Scalar.h"
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
#include "llvm/Transforms/Utils/BasicBlockUtils.h"
|
2020-05-20 11:08:08 +02:00
|
|
|
#include "llvm/Transforms/Utils/ScalarEvolutionExpander.h"
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
#include "llvm/Transforms/Utils/ValueMapper.h"
|
2021-05-30 11:13:48 +02:00
|
|
|
|
|
|
|
|
#define DEBUG_TYPE "loop-data-prefetch"
|
|
|
|
|
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
using namespace llvm;
|
|
|
|
|
|
|
|
|
|
// By default, we limit this to creating 16 PHIs (which is a little over half
|
|
|
|
|
// of the allocatable register set).
|
|
|
|
|
static cl::opt<bool>
|
2016-02-18 22:37:12 +01:00
|
|
|
PrefetchWrites("loop-prefetch-writes", cl::Hidden, cl::init(false),
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
cl::desc("Prefetch write addresses"));
|
|
|
|
|
|
2016-03-30 01:45:52 +02:00
|
|
|
static cl::opt<unsigned>
|
|
|
|
|
PrefetchDistance("prefetch-distance",
|
|
|
|
|
cl::desc("Number of instructions to prefetch ahead"),
|
|
|
|
|
cl::Hidden);
|
|
|
|
|
|
|
|
|
|
static cl::opt<unsigned>
|
|
|
|
|
MinPrefetchStride("min-prefetch-stride",
|
|
|
|
|
cl::desc("Min stride to add prefetches"), cl::Hidden);
|
|
|
|
|
|
|
|
|
|
static cl::opt<unsigned> MaxPrefetchIterationsAhead(
|
|
|
|
|
"max-prefetch-iters-ahead",
|
|
|
|
|
cl::desc("Max number of iterations to prefetch ahead"), cl::Hidden);
|
|
|
|
|
|
2016-03-09 06:33:21 +01:00
|
|
|
STATISTIC(NumPrefetches, "Number of prefetches inserted");
|
|
|
|
|
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
namespace {
|
|
|
|
|
|
2016-08-13 06:11:27 +02:00
|
|
|
/// Loop prefetch implementation class.
|
|
|
|
|
class LoopDataPrefetch {
|
|
|
|
|
public:
|
2019-10-31 16:05:58 +01:00
|
|
|
LoopDataPrefetch(AssumptionCache *AC, DominatorTree *DT, LoopInfo *LI,
|
|
|
|
|
ScalarEvolution *SE, const TargetTransformInfo *TTI,
|
2016-08-13 06:11:27 +02:00
|
|
|
OptimizationRemarkEmitter *ORE)
|
2019-10-31 16:05:58 +01:00
|
|
|
: AC(AC), DT(DT), LI(LI), SE(SE), TTI(TTI), ORE(ORE) {}
|
2016-08-13 06:11:27 +02:00
|
|
|
|
|
|
|
|
bool run();
|
|
|
|
|
|
|
|
|
|
private:
|
|
|
|
|
bool runOnLoop(Loop *L);
|
|
|
|
|
|
2018-05-01 17:54:18 +02:00
|
|
|
/// Check if the stride of the accesses is large enough to
|
2016-08-13 06:11:27 +02:00
|
|
|
/// warrant a prefetch.
|
2019-10-31 16:05:58 +01:00
|
|
|
bool isStrideLargeEnough(const SCEVAddRecExpr *AR, unsigned TargetMinStride);
|
2016-08-13 06:11:27 +02:00
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
unsigned getMinPrefetchStride(unsigned NumMemAccesses,
|
|
|
|
|
unsigned NumStridedMemAccesses,
|
|
|
|
|
unsigned NumPrefetches,
|
|
|
|
|
bool HasCall) {
|
2016-08-13 06:11:27 +02:00
|
|
|
if (MinPrefetchStride.getNumOccurrences() > 0)
|
|
|
|
|
return MinPrefetchStride;
|
2019-10-31 16:05:58 +01:00
|
|
|
return TTI->getMinPrefetchStride(NumMemAccesses, NumStridedMemAccesses,
|
|
|
|
|
NumPrefetches, HasCall);
|
2016-08-13 06:11:27 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
unsigned getPrefetchDistance() {
|
|
|
|
|
if (PrefetchDistance.getNumOccurrences() > 0)
|
|
|
|
|
return PrefetchDistance;
|
|
|
|
|
return TTI->getPrefetchDistance();
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
unsigned getMaxPrefetchIterationsAhead() {
|
|
|
|
|
if (MaxPrefetchIterationsAhead.getNumOccurrences() > 0)
|
|
|
|
|
return MaxPrefetchIterationsAhead;
|
|
|
|
|
return TTI->getMaxPrefetchIterationsAhead();
|
|
|
|
|
}
|
|
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
bool doPrefetchWrites() {
|
|
|
|
|
if (PrefetchWrites.getNumOccurrences() > 0)
|
|
|
|
|
return PrefetchWrites;
|
|
|
|
|
return TTI->enableWritePrefetching();
|
|
|
|
|
}
|
|
|
|
|
|
2016-12-19 09:22:17 +01:00
|
|
|
AssumptionCache *AC;
|
2019-10-31 16:05:58 +01:00
|
|
|
DominatorTree *DT;
|
2016-08-13 06:11:27 +02:00
|
|
|
LoopInfo *LI;
|
|
|
|
|
ScalarEvolution *SE;
|
|
|
|
|
const TargetTransformInfo *TTI;
|
|
|
|
|
OptimizationRemarkEmitter *ORE;
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
/// Legacy class for inserting loop data prefetches.
|
|
|
|
|
class LoopDataPrefetchLegacyPass : public FunctionPass {
|
|
|
|
|
public:
|
|
|
|
|
static char ID; // Pass ID, replacement for typeid
|
|
|
|
|
LoopDataPrefetchLegacyPass() : FunctionPass(ID) {
|
|
|
|
|
initializeLoopDataPrefetchLegacyPassPass(*PassRegistry::getPassRegistry());
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void getAnalysisUsage(AnalysisUsage &AU) const override {
|
2016-12-19 09:22:17 +01:00
|
|
|
AU.addRequired<AssumptionCacheTracker>();
|
2019-10-31 16:05:58 +01:00
|
|
|
AU.addRequired<DominatorTreeWrapperPass>();
|
2016-08-13 06:11:27 +02:00
|
|
|
AU.addPreserved<DominatorTreeWrapperPass>();
|
|
|
|
|
AU.addRequired<LoopInfoWrapperPass>();
|
|
|
|
|
AU.addPreserved<LoopInfoWrapperPass>();
|
|
|
|
|
AU.addRequired<OptimizationRemarkEmitterWrapperPass>();
|
|
|
|
|
AU.addRequired<ScalarEvolutionWrapperPass>();
|
[LoopDataPrefetch][AArch64FalkorHWPFFix] Preserve ScalarEvolution
Summary:
Mark LoopDataPrefetch and AArch64FalkorHWPFFix passes as preserving
ScalarEvolution since they do not alter loop structure and should not
alter any SCEV values (though LoopDataPrefetch may introduce new
instructions that won't have cached SCEV values yet).
This can result in slight code differences, mainly w.r.t. nsw/nuw flags
on SCEVs, since these are computed somewhat lazily when a zext/sext
instruction is encountered. As a result, passes after the modified
passes may see SCEVs with more nsw/nuw flags present.
Reviewers: sanjoy, anemet
Subscribers: aemerson, rengolin, mzolotukhin, javed.absar, kristof.beyls, mcrosier, llvm-commits
Differential Revision: https://reviews.llvm.org/D36716
llvm-svn: 311032
2017-08-16 21:03:16 +02:00
|
|
|
AU.addPreserved<ScalarEvolutionWrapperPass>();
|
2016-08-13 06:11:27 +02:00
|
|
|
AU.addRequired<TargetTransformInfoWrapperPass>();
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
bool runOnFunction(Function &F) override;
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
};
|
2015-06-23 11:49:53 +02:00
|
|
|
}
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
|
2016-08-13 06:11:27 +02:00
|
|
|
char LoopDataPrefetchLegacyPass::ID = 0;
|
|
|
|
|
INITIALIZE_PASS_BEGIN(LoopDataPrefetchLegacyPass, "loop-data-prefetch",
|
2016-02-18 22:37:12 +01:00
|
|
|
"Loop Data Prefetch", false, false)
|
2016-12-19 09:22:17 +01:00
|
|
|
INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
|
|
|
|
|
INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass)
|
2016-07-23 00:53:17 +02:00
|
|
|
INITIALIZE_PASS_DEPENDENCY(OptimizationRemarkEmitterWrapperPass)
|
[PM] Port ScalarEvolution to the new pass manager.
This change makes ScalarEvolution a stand-alone object and just produces
one from a pass as needed. Making this work well requires making the
object movable, using references instead of overwritten pointers in
a number of places, and other refactorings.
I've also wired it up to the new pass manager and added a RUN line to
a test to exercise it under the new pass manager. This includes basic
printing support much like with other analyses.
But there is a big and somewhat scary change here. Prior to this patch
ScalarEvolution was never *actually* invalidated!!! Re-running the pass
just re-wired up the various other analyses and didn't remove any of the
existing entries in the SCEV caches or clear out anything at all. This
might seem OK as everything in SCEV that can uses ValueHandles to track
updates to the values that serve as SCEV keys. However, this still means
that as we ran SCEV over each function in the module, we kept
accumulating more and more SCEVs into the cache. At the end, we would
have a SCEV cache with every value that we ever needed a SCEV for in the
entire module!!! Yowzers. The releaseMemory routine would dump all of
this, but that isn't realy called during normal runs of the pipeline as
far as I can see.
To make matters worse, there *is* actually a key that we don't update
with value handles -- there is a map keyed off of Loop*s. Because
LoopInfo *does* release its memory from run to run, it is entirely
possible to run SCEV over one function, then over another function, and
then lookup a Loop* from the second function but find an entry inserted
for the first function! Ouch.
To make matters still worse, there are plenty of updates that *don't*
trip a value handle. It seems incredibly unlikely that today GVN or
another pass that invalidates SCEV can update values in *just* such
a way that a subsequent run of SCEV will incorrectly find lookups in
a cache, but it is theoretically possible and would be a nightmare to
debug.
With this refactoring, I've fixed all this by actually destroying and
recreating the ScalarEvolution object from run to run. Technically, this
could increase the amount of malloc traffic we see, but then again it is
also technically correct. ;] I don't actually think we're suffering from
tons of malloc traffic from SCEV because if we were, the fact that we
never clear the memory would seem more likely to have come up as an
actual problem before now. So, I've made the simple fix here. If in fact
there are serious issues with too much allocation and deallocation,
I can work on a clever fix that preserves the allocations (while
clearing the data) between each run, but I'd prefer to do that kind of
optimization with a test case / benchmark that shows why we need such
cleverness (and that can test that we actually make it faster). It's
possible that this will make some things faster by making the SCEV
caches have higher locality (due to being significantly smaller) so
until there is a clear benchmark, I think the simple change is best.
Differential Revision: http://reviews.llvm.org/D12063
llvm-svn: 245193
2015-08-17 04:08:17 +02:00
|
|
|
INITIALIZE_PASS_DEPENDENCY(ScalarEvolutionWrapperPass)
|
2016-08-13 06:11:27 +02:00
|
|
|
INITIALIZE_PASS_END(LoopDataPrefetchLegacyPass, "loop-data-prefetch",
|
2016-02-18 22:37:12 +01:00
|
|
|
"Loop Data Prefetch", false, false)
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
|
2016-08-13 06:11:27 +02:00
|
|
|
FunctionPass *llvm::createLoopDataPrefetchPass() {
|
|
|
|
|
return new LoopDataPrefetchLegacyPass();
|
|
|
|
|
}
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
bool LoopDataPrefetch::isStrideLargeEnough(const SCEVAddRecExpr *AR,
|
|
|
|
|
unsigned TargetMinStride) {
|
2016-03-18 01:27:38 +01:00
|
|
|
// No need to check if any stride goes.
|
|
|
|
|
if (TargetMinStride <= 1)
|
|
|
|
|
return true;
|
|
|
|
|
|
|
|
|
|
const auto *ConstStride = dyn_cast<SCEVConstant>(AR->getStepRecurrence(*SE));
|
|
|
|
|
// If MinStride is set, don't prefetch unless we can ensure that stride is
|
|
|
|
|
// larger.
|
|
|
|
|
if (!ConstStride)
|
|
|
|
|
return false;
|
|
|
|
|
|
|
|
|
|
unsigned AbsStride = std::abs(ConstStride->getAPInt().getSExtValue());
|
|
|
|
|
return TargetMinStride <= AbsStride;
|
|
|
|
|
}
|
|
|
|
|
|
2016-08-13 06:11:27 +02:00
|
|
|
PreservedAnalyses LoopDataPrefetchPass::run(Function &F,
|
|
|
|
|
FunctionAnalysisManager &AM) {
|
2019-10-31 16:05:58 +01:00
|
|
|
DominatorTree *DT = &AM.getResult<DominatorTreeAnalysis>(F);
|
2016-08-13 06:11:27 +02:00
|
|
|
LoopInfo *LI = &AM.getResult<LoopAnalysis>(F);
|
|
|
|
|
ScalarEvolution *SE = &AM.getResult<ScalarEvolutionAnalysis>(F);
|
2016-12-19 09:22:17 +01:00
|
|
|
AssumptionCache *AC = &AM.getResult<AssumptionAnalysis>(F);
|
2016-08-13 06:11:27 +02:00
|
|
|
OptimizationRemarkEmitter *ORE =
|
|
|
|
|
&AM.getResult<OptimizationRemarkEmitterAnalysis>(F);
|
|
|
|
|
const TargetTransformInfo *TTI = &AM.getResult<TargetIRAnalysis>(F);
|
|
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
LoopDataPrefetch LDP(AC, DT, LI, SE, TTI, ORE);
|
2016-08-13 06:11:27 +02:00
|
|
|
bool Changed = LDP.run();
|
|
|
|
|
|
|
|
|
|
if (Changed) {
|
|
|
|
|
PreservedAnalyses PA;
|
|
|
|
|
PA.preserve<DominatorTreeAnalysis>();
|
|
|
|
|
PA.preserve<LoopAnalysis>();
|
|
|
|
|
return PA;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return PreservedAnalyses::all();
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
bool LoopDataPrefetchLegacyPass::runOnFunction(Function &F) {
|
2016-05-04 00:32:30 +02:00
|
|
|
if (skipFunction(F))
|
|
|
|
|
return false;
|
|
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
DominatorTree *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
|
2016-08-13 06:11:27 +02:00
|
|
|
LoopInfo *LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
|
|
|
|
|
ScalarEvolution *SE = &getAnalysis<ScalarEvolutionWrapperPass>().getSE();
|
2016-12-19 09:22:17 +01:00
|
|
|
AssumptionCache *AC =
|
|
|
|
|
&getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
|
2016-08-13 06:11:27 +02:00
|
|
|
OptimizationRemarkEmitter *ORE =
|
|
|
|
|
&getAnalysis<OptimizationRemarkEmitterWrapperPass>().getORE();
|
|
|
|
|
const TargetTransformInfo *TTI =
|
|
|
|
|
&getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
|
|
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
LoopDataPrefetch LDP(AC, DT, LI, SE, TTI, ORE);
|
2016-08-13 06:11:27 +02:00
|
|
|
return LDP.run();
|
|
|
|
|
}
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
|
2016-08-13 06:11:27 +02:00
|
|
|
bool LoopDataPrefetch::run() {
|
2016-03-07 19:35:42 +01:00
|
|
|
// If PrefetchDistance is not set, don't run the pass. This gives an
|
|
|
|
|
// opportunity for targets to run this pass for selected subtargets only
|
|
|
|
|
// (whose TTI sets PrefetchDistance).
|
2016-03-30 01:45:52 +02:00
|
|
|
if (getPrefetchDistance() == 0)
|
2016-03-07 19:35:42 +01:00
|
|
|
return false;
|
2016-01-21 19:28:36 +01:00
|
|
|
assert(TTI->getCacheLineSize() && "Cache line size is not set for target");
|
|
|
|
|
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
bool MadeChange = false;
|
|
|
|
|
|
2016-06-26 14:28:59 +02:00
|
|
|
for (Loop *I : *LI)
|
|
|
|
|
for (auto L = df_begin(I), LE = df_end(I); L != LE; ++L)
|
2015-04-12 19:18:56 +02:00
|
|
|
MadeChange |= runOnLoop(*L);
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
|
|
|
|
|
return MadeChange;
|
|
|
|
|
}
|
|
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
/// A record for a potential prefetch made during the initial scan of the
|
|
|
|
|
/// loop. This is used to let a single prefetch target multiple memory accesses.
|
|
|
|
|
struct Prefetch {
|
|
|
|
|
/// The address formula for this prefetch as returned by ScalarEvolution.
|
|
|
|
|
const SCEVAddRecExpr *LSCEVAddRec;
|
|
|
|
|
/// The point of insertion for the prefetch instruction.
|
|
|
|
|
Instruction *InsertPt;
|
|
|
|
|
/// True if targeting a write memory access.
|
|
|
|
|
bool Writes;
|
|
|
|
|
/// The (first seen) prefetched instruction.
|
|
|
|
|
Instruction *MemI;
|
|
|
|
|
|
2020-05-06 11:22:31 +02:00
|
|
|
/// Constructor to create a new Prefetch for \p I.
|
2019-10-31 16:05:58 +01:00
|
|
|
Prefetch(const SCEVAddRecExpr *L, Instruction *I)
|
|
|
|
|
: LSCEVAddRec(L), InsertPt(nullptr), Writes(false), MemI(nullptr) {
|
|
|
|
|
addInstruction(I);
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
/// Add the instruction \param I to this prefetch. If it's not the first
|
|
|
|
|
/// one, 'InsertPt' and 'Writes' will be updated as required.
|
|
|
|
|
/// \param PtrDiff the known constant address difference to the first added
|
|
|
|
|
/// instruction.
|
|
|
|
|
void addInstruction(Instruction *I, DominatorTree *DT = nullptr,
|
|
|
|
|
int64_t PtrDiff = 0) {
|
|
|
|
|
if (!InsertPt) {
|
|
|
|
|
MemI = I;
|
|
|
|
|
InsertPt = I;
|
|
|
|
|
Writes = isa<StoreInst>(I);
|
|
|
|
|
} else {
|
|
|
|
|
BasicBlock *PrefBB = InsertPt->getParent();
|
|
|
|
|
BasicBlock *InsBB = I->getParent();
|
|
|
|
|
if (PrefBB != InsBB) {
|
|
|
|
|
BasicBlock *DomBB = DT->findNearestCommonDominator(PrefBB, InsBB);
|
|
|
|
|
if (DomBB != PrefBB)
|
|
|
|
|
InsertPt = DomBB->getTerminator();
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (isa<StoreInst>(I) && PtrDiff == 0)
|
|
|
|
|
Writes = true;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
};
|
|
|
|
|
|
2016-02-18 22:37:12 +01:00
|
|
|
bool LoopDataPrefetch::runOnLoop(Loop *L) {
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
bool MadeChange = false;
|
|
|
|
|
|
|
|
|
|
// Only prefetch in the inner-most loop
|
2020-09-22 22:28:00 +02:00
|
|
|
if (!L->isInnermost())
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
return MadeChange;
|
|
|
|
|
|
|
|
|
|
SmallPtrSet<const Value *, 32> EphValues;
|
2016-12-19 09:22:17 +01:00
|
|
|
CodeMetrics::collectEphemeralValues(L, AC, EphValues);
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
|
|
|
|
|
// Calculate the number of iterations ahead to prefetch
|
|
|
|
|
CodeMetrics Metrics;
|
2019-10-31 16:05:58 +01:00
|
|
|
bool HasCall = false;
|
2016-09-08 19:08:20 +02:00
|
|
|
for (const auto BB : L->blocks()) {
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
// If the loop already has prefetches, then assume that the user knows
|
2016-06-10 22:06:03 +02:00
|
|
|
// what they are doing and don't add any more.
|
2019-10-31 16:05:58 +01:00
|
|
|
for (auto &I : *BB) {
|
|
|
|
|
if (isa<CallInst>(&I) || isa<InvokeInst>(&I)) {
|
2020-04-24 07:00:10 +02:00
|
|
|
if (const Function *F = cast<CallBase>(I).getCalledFunction()) {
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
if (F->getIntrinsicID() == Intrinsic::prefetch)
|
|
|
|
|
return MadeChange;
|
2019-10-31 16:05:58 +01:00
|
|
|
if (TTI->isLoweredToCall(F))
|
|
|
|
|
HasCall = true;
|
|
|
|
|
} else { // indirect call.
|
|
|
|
|
HasCall = true;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
2016-09-08 19:08:20 +02:00
|
|
|
Metrics.analyzeBasicBlock(BB, *TTI, EphValues);
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
}
|
|
|
|
|
unsigned LoopSize = Metrics.NumInsts;
|
|
|
|
|
if (!LoopSize)
|
|
|
|
|
LoopSize = 1;
|
|
|
|
|
|
2016-03-30 01:45:52 +02:00
|
|
|
unsigned ItersAhead = getPrefetchDistance() / LoopSize;
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
if (!ItersAhead)
|
|
|
|
|
ItersAhead = 1;
|
|
|
|
|
|
2016-03-30 01:45:52 +02:00
|
|
|
if (ItersAhead > getMaxPrefetchIterationsAhead())
|
2016-03-18 01:27:43 +01:00
|
|
|
return MadeChange;
|
|
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
unsigned ConstantMaxTripCount = SE->getSmallConstantMaxTripCount(L);
|
|
|
|
|
if (ConstantMaxTripCount && ConstantMaxTripCount < ItersAhead + 1)
|
|
|
|
|
return MadeChange;
|
2016-03-09 06:33:21 +01:00
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
unsigned NumMemAccesses = 0;
|
|
|
|
|
unsigned NumStridedMemAccesses = 0;
|
|
|
|
|
SmallVector<Prefetch, 16> Prefetches;
|
|
|
|
|
for (const auto BB : L->blocks())
|
2016-09-08 19:08:20 +02:00
|
|
|
for (auto &I : *BB) {
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
Value *PtrValue;
|
|
|
|
|
Instruction *MemI;
|
|
|
|
|
|
2016-09-08 19:08:20 +02:00
|
|
|
if (LoadInst *LMemI = dyn_cast<LoadInst>(&I)) {
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
MemI = LMemI;
|
|
|
|
|
PtrValue = LMemI->getPointerOperand();
|
2016-09-08 19:08:20 +02:00
|
|
|
} else if (StoreInst *SMemI = dyn_cast<StoreInst>(&I)) {
|
2019-10-31 16:05:58 +01:00
|
|
|
if (!doPrefetchWrites()) continue;
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
MemI = SMemI;
|
|
|
|
|
PtrValue = SMemI->getPointerOperand();
|
|
|
|
|
} else continue;
|
|
|
|
|
|
|
|
|
|
unsigned PtrAddrSpace = PtrValue->getType()->getPointerAddressSpace();
|
|
|
|
|
if (PtrAddrSpace)
|
|
|
|
|
continue;
|
2019-10-31 16:05:58 +01:00
|
|
|
NumMemAccesses++;
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
if (L->isLoopInvariant(PtrValue))
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
const SCEV *LSCEV = SE->getSCEV(PtrValue);
|
|
|
|
|
const SCEVAddRecExpr *LSCEVAddRec = dyn_cast<SCEVAddRecExpr>(LSCEV);
|
|
|
|
|
if (!LSCEVAddRec)
|
|
|
|
|
continue;
|
2019-10-31 16:05:58 +01:00
|
|
|
NumStridedMemAccesses++;
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
// We don't want to double prefetch individual cache lines. If this
|
|
|
|
|
// access is known to be within one cache line of some other one that
|
|
|
|
|
// has already been prefetched, then don't prefetch this one as well.
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
bool DupPref = false;
|
2019-10-31 16:05:58 +01:00
|
|
|
for (auto &Pref : Prefetches) {
|
|
|
|
|
const SCEV *PtrDiff = SE->getMinusSCEV(LSCEVAddRec, Pref.LSCEVAddRec);
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
if (const SCEVConstant *ConstPtrDiff =
|
|
|
|
|
dyn_cast<SCEVConstant>(PtrDiff)) {
|
2015-03-09 21:20:16 +01:00
|
|
|
int64_t PD = std::abs(ConstPtrDiff->getValue()->getSExtValue());
|
2016-01-21 19:28:36 +01:00
|
|
|
if (PD < (int64_t) TTI->getCacheLineSize()) {
|
2019-10-31 16:05:58 +01:00
|
|
|
Pref.addInstruction(MemI, DT, PD);
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
DupPref = true;
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
2019-10-31 16:05:58 +01:00
|
|
|
if (!DupPref)
|
|
|
|
|
Prefetches.push_back(Prefetch(LSCEVAddRec, MemI));
|
|
|
|
|
}
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
unsigned TargetMinStride =
|
|
|
|
|
getMinPrefetchStride(NumMemAccesses, NumStridedMemAccesses,
|
|
|
|
|
Prefetches.size(), HasCall);
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
LLVM_DEBUG(dbgs() << "Prefetching " << ItersAhead
|
|
|
|
|
<< " iterations ahead (loop size: " << LoopSize << ") in "
|
|
|
|
|
<< L->getHeader()->getParent()->getName() << ": " << *L);
|
|
|
|
|
LLVM_DEBUG(dbgs() << "Loop has: "
|
|
|
|
|
<< NumMemAccesses << " memory accesses, "
|
|
|
|
|
<< NumStridedMemAccesses << " strided memory accesses, "
|
|
|
|
|
<< Prefetches.size() << " potential prefetch(es), "
|
|
|
|
|
<< "a minimum stride of " << TargetMinStride << ", "
|
|
|
|
|
<< (HasCall ? "calls" : "no calls") << ".\n");
|
|
|
|
|
|
|
|
|
|
for (auto &P : Prefetches) {
|
|
|
|
|
// Check if the stride of the accesses is large enough to warrant a
|
|
|
|
|
// prefetch.
|
|
|
|
|
if (!isStrideLargeEnough(P.LSCEVAddRec, TargetMinStride))
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
const SCEV *NextLSCEV = SE->getAddExpr(P.LSCEVAddRec, SE->getMulExpr(
|
|
|
|
|
SE->getConstant(P.LSCEVAddRec->getType(), ItersAhead),
|
|
|
|
|
P.LSCEVAddRec->getStepRecurrence(*SE)));
|
|
|
|
|
if (!isSafeToExpand(NextLSCEV, *SE))
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
BasicBlock *BB = P.InsertPt->getParent();
|
|
|
|
|
Type *I8Ptr = Type::getInt8PtrTy(BB->getContext(), 0/*PtrAddrSpace*/);
|
|
|
|
|
SCEVExpander SCEVE(*SE, BB->getModule()->getDataLayout(), "prefaddr");
|
|
|
|
|
Value *PrefPtrValue = SCEVE.expandCodeFor(NextLSCEV, I8Ptr, P.InsertPt);
|
|
|
|
|
|
|
|
|
|
IRBuilder<> Builder(P.InsertPt);
|
|
|
|
|
Module *M = BB->getParent()->getParent();
|
|
|
|
|
Type *I32 = Type::getInt32Ty(BB->getContext());
|
|
|
|
|
Function *PrefetchFunc = Intrinsic::getDeclaration(
|
|
|
|
|
M, Intrinsic::prefetch, PrefPtrValue->getType());
|
|
|
|
|
Builder.CreateCall(
|
|
|
|
|
PrefetchFunc,
|
|
|
|
|
{PrefPtrValue,
|
|
|
|
|
ConstantInt::get(I32, P.Writes),
|
|
|
|
|
ConstantInt::get(I32, 3), ConstantInt::get(I32, 1)});
|
|
|
|
|
++NumPrefetches;
|
|
|
|
|
LLVM_DEBUG(dbgs() << " Access: "
|
|
|
|
|
<< *P.MemI->getOperand(isa<LoadInst>(P.MemI) ? 0 : 1)
|
|
|
|
|
<< ", SCEV: " << *P.LSCEVAddRec << "\n");
|
|
|
|
|
ORE->emit([&]() {
|
|
|
|
|
return OptimizationRemark(DEBUG_TYPE, "Prefetched", P.MemI)
|
|
|
|
|
<< "prefetched memory access";
|
2017-10-11 19:12:59 +02:00
|
|
|
});
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
|
2019-10-31 16:05:58 +01:00
|
|
|
MadeChange = true;
|
[PowerPC] Loop Data Prefetching for the BG/Q
The IBM BG/Q supercomputer's A2 cores have a hardware prefetching unit, the
L1P, but it does not prefetch directly into the A2's L1 cache. Instead, it
prefetches into its own L1P buffer, and the latency to access that buffer is
significantly higher than that to the L1 cache (although smaller than the
latency to the L2 cache). As a result, especially when multiple hardware
threads are not actively busy, explicitly prefetching data into the L1 cache is
advantageous.
I've been using this pass out-of-tree for data prefetching on the BG/Q for well
over a year, and it has worked quite well. It is enabled by default only for
the BG/Q, but can be enabled for other cores as well via a command-line option.
Eventually, we might want to add some TTI interfaces and move this into
Transforms/Scalar (there is nothing particularly target dependent about it,
although only machines like the BG/Q will benefit from its simplistic
strategy).
llvm-svn: 229966
2015-02-20 06:08:21 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return MadeChange;
|
|
|
|
|
}
|
