RPCS3/llvm-mirror
1
0
Fork 0
mirror of https://github.com/RPCS3/llvm-mirror.git synced 2024-11-22 10:42:39 +01:00
Code Issues Projects Releases Wiki Activity
8c87b66158
llvm-mirror/lib/Analysis/BranchProbabilityInfo.cpp

1299 lines
47 KiB
C++
Raw Normal View History

[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-21 23:37:46 +02:00
//===- BranchProbabilityInfo.cpp - Branch Probability Analysis ------------===//
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
//
Update the file headers across all of the LLVM projects in the monorepo to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
2019-01-19 09:50:56 +01:00
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
//
//===----------------------------------------------------------------------===//
//
// Loops should be simplified before this analysis.
//
//===----------------------------------------------------------------------===//
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-03 17:50:05 +01:00
#include "llvm/Analysis/BranchProbabilityInfo.h"
#include "llvm/ADT/PostOrderIterator.h"
[BranchProbabilityInfo] Handle irreducible loops. Summary: Compute the strongly connected components of the CFG and fall back to use these for blocks that are in loops that are not detected by LoopInfo when computing loop back-edge and exit branch probabilities. Reviewers: dexonsmith, davidxl Subscribers: mcrosier, llvm-commits Differential Revision: https://reviews.llvm.org/D39385 llvm-svn: 317094
2017-11-01 16:16:50 +01:00
#include "llvm/ADT/SCCIterator.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-21 23:37:46 +02:00
#include "llvm/ADT/STLExtras.h"
#include "llvm/ADT/SmallVector.h"
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-03 17:50:05 +01:00
#include "llvm/Analysis/LoopInfo.h"
Reland "b19ec1eb3d0c [BPI] Improve unreachable/ColdCall heurstics to handle loops." Summary: b19ec1eb3d0c has been reverted because of the test failures with PowerPC targets. This patch addresses the issues from the previous commit. Test Plan: ninja check-all. Confirmed that CodeGen/PowerPC/pr36292.ll and CodeGen/PowerPC/sms-cpy-1.ll pass Subscribers: llvm-commits
2019-12-02 19:15:22 +01:00
#include "llvm/Analysis/PostDominators.h"
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 11:44:40 +02:00
#include "llvm/Analysis/TargetLibraryInfo.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-21 23:37:46 +02:00
#include "llvm/IR/Attributes.h"
#include "llvm/IR/BasicBlock.h"
[Modules] Move CFG.h to the IR library as it defines graph traits over IR types. llvm-svn: 202827
2014-03-04 12:45:46 +01:00
#include "llvm/IR/CFG.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 12:36:10 +01:00
#include "llvm/IR/Constants.h"
Require DominatorTree when requiring/preserving LoopInfo in the old pass manager Summary: Require DominatorTree when requiring/preserving LoopInfo in the old pass manager BreakCriticalEdges tries to keep LoopInfo and DominatorTree updated if they exist. However, since commit r321653 and r321805, to update LoopInfo we must have a DominatorTree, or we will hit an assert. To fix this we now make a couple of passes that only required/preserved LoopInfo also require DominatorTree. This solves PR37334. Reviewers: eli.friedman, efriedma Reviewed By: efriedma Subscribers: efriedma, llvm-commits Differential Revision: https://reviews.llvm.org/D46829 llvm-svn: 332583
2018-05-17 11:05:40 +02:00
#include "llvm/IR/Dominators.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 12:36:10 +01:00
#include "llvm/IR/Function.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-21 23:37:46 +02:00
#include "llvm/IR/InstrTypes.h"
#include "llvm/IR/Instruction.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 12:36:10 +01:00
#include "llvm/IR/Instructions.h"
#include "llvm/IR/LLVMContext.h"
#include "llvm/IR/Metadata.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-21 23:37:46 +02:00
#include "llvm/IR/PassManager.h"
#include "llvm/IR/Type.h"
#include "llvm/IR/Value.h"
Sink all InitializePasses.h includes This file lists every pass in LLVM, and is included by Pass.h, which is very popular. Every time we add, remove, or rename a pass in LLVM, it caused lots of recompilation. I found this fact by looking at this table, which is sorted by the number of times a file was changed over the last 100,000 git commits multiplied by the number of object files that depend on it in the current checkout: recompiles touches affected_files header 342380 95 3604 llvm/include/llvm/ADT/STLExtras.h 314730 234 1345 llvm/include/llvm/InitializePasses.h 307036 118 2602 llvm/include/llvm/ADT/APInt.h 213049 59 3611 llvm/include/llvm/Support/MathExtras.h 170422 47 3626 llvm/include/llvm/Support/Compiler.h 162225 45 3605 llvm/include/llvm/ADT/Optional.h 158319 63 2513 llvm/include/llvm/ADT/Triple.h 140322 39 3598 llvm/include/llvm/ADT/StringRef.h 137647 59 2333 llvm/include/llvm/Support/Error.h 131619 73 1803 llvm/include/llvm/Support/FileSystem.h Before this change, touching InitializePasses.h would cause 1345 files to recompile. After this change, touching it only causes 550 compiles in an incremental rebuild. Reviewers: bkramer, asbirlea, bollu, jdoerfert Differential Revision: https://reviews.llvm.org/D70211
2019-11-13 22:15:01 +01:00
#include "llvm/InitializePasses.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-21 23:37:46 +02:00
#include "llvm/Pass.h"
#include "llvm/Support/BranchProbability.h"
#include "llvm/Support/Casting.h"
Add missing includes needed to prune LLVMContext.h include, NFC These are a pre-requisite to removing #include "llvm/Support/Options.h" from LLVMContext.h: https://reviews.llvm.org/D70280
2019-11-15 00:15:48 +01:00
#include "llvm/Support/CommandLine.h"
Branch profiling: floating-point avoidance. Patch by: Jakub Staszak! Introduces BranchProbability. Changes unsigned to uint32_t all over and uint64_t only when overflow is expected. llvm-svn: 132867
2011-06-11 03:05:22 +02:00
#include "llvm/Support/Debug.h"
Purge unused includes throughout libSupport. NFC. llvm-svn: 232976
2015-03-23 19:07:13 +01:00
#include "llvm/Support/raw_ostream.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-21 23:37:46 +02:00
#include <cassert>
#include <cstdint>
#include <iterator>
#include <utility>
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
using namespace llvm;
[Modules] Fix potential ODR violations by sinking the DEBUG_TYPE definition below all the header #include lines, lib/Analysis/... edition. This one has a bit extra as there were *other* #define's before #include lines in addition to DEBUG_TYPE. I've sunk all of them as a block. llvm-svn: 206843
2014-04-22 04:48:03 +02:00
#define DEBUG_TYPE "branch-prob"
Add options to dump block frequency/branch probability info in text. Summary: Add options -print-bfi/-print-bpi that dump block frequency and branch probability info like -view-block-freq-propagation-dags and -view-machine-block-freq-propagation-dags do but in text. This is useful when the graph is very large and complex (the dot command crashes, lines/edges too close to tell apart, hard to navigate without textual search) or simply when text is preferred. Reviewers: davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D37165 llvm-svn: 311822
2017-08-26 02:31:00 +02:00
static cl::opt<bool> PrintBranchProb(
"print-bpi", cl::init(false), cl::Hidden,
cl::desc("Print the branch probability info."));
cl::opt<std::string> PrintBranchProbFuncName(
"print-bpi-func-name", cl::Hidden,
cl::desc("The option to specify the name of the function "
"whose branch probability info is printed."));
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
INITIALIZE_PASS_BEGIN(BranchProbabilityInfoWrapperPass, "branch-prob",
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
"Branch Probability Analysis", false, true)
[PM] Split the LoopInfo object apart from the legacy pass, creating a LoopInfoWrapperPass to wire the object up to the legacy pass manager. This switches all the clients of LoopInfo over and paves the way to port LoopInfo to the new pass manager. No functionality change is intended with this iteration. llvm-svn: 226373
2015-01-17 15:16:18 +01:00
INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass)
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 11:44:40 +02:00
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
[BPI][NFC] Reuse post dominantor tree from analysis manager when available Summary: Currenlty BPI unconditionally creates post dominator tree each time. While this is not incorrect we can save compile time by reusing existing post dominator tree (when it's valid) provided by analysis manager. Reviewers: skatkov, taewookoh, yrouban Reviewed By: skatkov Subscribers: hiraditya, steven_wu, dexonsmith, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D78987
2020-04-28 11:31:20 +02:00
INITIALIZE_PASS_DEPENDENCY(PostDominatorTreeWrapperPass)
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
INITIALIZE_PASS_END(BranchProbabilityInfoWrapperPass, "branch-prob",
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
"Branch Probability Analysis", false, true)
Sink all InitializePasses.h includes This file lists every pass in LLVM, and is included by Pass.h, which is very popular. Every time we add, remove, or rename a pass in LLVM, it caused lots of recompilation. I found this fact by looking at this table, which is sorted by the number of times a file was changed over the last 100,000 git commits multiplied by the number of object files that depend on it in the current checkout: recompiles touches affected_files header 342380 95 3604 llvm/include/llvm/ADT/STLExtras.h 314730 234 1345 llvm/include/llvm/InitializePasses.h 307036 118 2602 llvm/include/llvm/ADT/APInt.h 213049 59 3611 llvm/include/llvm/Support/MathExtras.h 170422 47 3626 llvm/include/llvm/Support/Compiler.h 162225 45 3605 llvm/include/llvm/ADT/Optional.h 158319 63 2513 llvm/include/llvm/ADT/Triple.h 140322 39 3598 llvm/include/llvm/ADT/StringRef.h 137647 59 2333 llvm/include/llvm/Support/Error.h 131619 73 1803 llvm/include/llvm/Support/FileSystem.h Before this change, touching InitializePasses.h would cause 1345 files to recompile. After this change, touching it only causes 550 compiles in an incremental rebuild. Reviewers: bkramer, asbirlea, bollu, jdoerfert Differential Revision: https://reviews.llvm.org/D70211
2019-11-13 22:15:01 +01:00
BranchProbabilityInfoWrapperPass::BranchProbabilityInfoWrapperPass()
: FunctionPass(ID) {
initializeBranchProbabilityInfoWrapperPassPass(
*PassRegistry::getPassRegistry());
}
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
char BranchProbabilityInfoWrapperPass::ID = 0;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
Simplify the design of BranchProbabilityInfo by collapsing it into a single class. Previously it was split between two classes, one internal and one external. The concern seemed to center around exposing the weights used, but those can remain confined to the implementation file. Having a single class to maintain the state and analyses in use will also simplify several of the enhancements I want to make to our static heuristics. llvm-svn: 142783
2011-10-24 03:40:45 +02:00
// Weights are for internal use only. They are used by heuristics to help to
// estimate edges' probability. Example:
//
// Using "Loop Branch Heuristics" we predict weights of edges for the
// block BB2.
// ...
// |
// V
// BB1<-+
// | |
// | | (Weight = 124)
// V |
// BB2--+
// |
// | (Weight = 4)
// V
// BB3
//
// Probability of the edge BB2->BB1 = 124 / (124 + 4) = 0.96875
// Probability of the edge BB2->BB3 = 4 / (124 + 4) = 0.03125
static const uint32_t LBH_TAKEN_WEIGHT = 124;
static const uint32_t LBH_NONTAKEN_WEIGHT = 4;
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
// Unlikely edges within a loop are half as likely as other edges
static const uint32_t LBH_UNLIKELY_WEIGHT = 62;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
Remove \brief commands from doxygen comments. We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
2018-05-01 17:54:18 +02:00
/// Unreachable-terminating branch taken probability.
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 14:01:08 +02:00
///
[BPI] Reduce the probability of unreachable edge to minimal value greater than 0 The probability of edge coming to unreachable block should be as low as possible. The change reduces the probability to minimal value greater than zero. The bug https://bugs.llvm.org/show_bug.cgi?id=32214 show the example when the probability of edge coming to unreachable block is greater than for edge coming to out of the loop and it causes incorrect loop rotation. Please note that with this change the behavior of unreachable heuristic is a bit different than others. Specifically, before this change the sum of probabilities coming to unreachable blocks have the same weight for all branches (it was just split over all edges of this block coming to unreachable blocks). With this change it might be slightly different but not to much due to probability of taken branch to unreachable block is really small. Reviewers: chandlerc, sanjoy, vsk, congh, junbuml, davidxl, dexonsmith Reviewed By: chandlerc, dexonsmith Subscribers: reames, llvm-commits Differential Revision: https://reviews.llvm.org/D30633 llvm-svn: 303327
2017-05-18 08:11:56 +02:00
/// This is the probability for a branch being taken to a block that terminates
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 14:01:08 +02:00
/// (eventually) in unreachable. These are predicted as unlikely as possible.
[BrachProbablityInfo] Proportional distribution of reachable probabilities When fixing probability of unreachable edges in BranchProbabilityInfo::calcMetadataWeights() proportionally distribute remainder probability over the reachable edges. The old implementation distributes the remainder probability evenly. See examples in the fixed tests. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D80611
2020-06-02 06:28:12 +02:00
/// All reachable probability will proportionally share the remaining part.
[BPI] Reduce the probability of unreachable edge to minimal value greater than 0 The probability of edge coming to unreachable block should be as low as possible. The change reduces the probability to minimal value greater than zero. The bug https://bugs.llvm.org/show_bug.cgi?id=32214 show the example when the probability of edge coming to unreachable block is greater than for edge coming to out of the loop and it causes incorrect loop rotation. Please note that with this change the behavior of unreachable heuristic is a bit different than others. Specifically, before this change the sum of probabilities coming to unreachable blocks have the same weight for all branches (it was just split over all edges of this block coming to unreachable blocks). With this change it might be slightly different but not to much due to probability of taken branch to unreachable block is really small. Reviewers: chandlerc, sanjoy, vsk, congh, junbuml, davidxl, dexonsmith Reviewed By: chandlerc, dexonsmith Subscribers: reames, llvm-commits Differential Revision: https://reviews.llvm.org/D30633 llvm-svn: 303327
2017-05-18 08:11:56 +02:00
static const BranchProbability UR_TAKEN_PROB = BranchProbability::getRaw(1);
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
Remove \brief commands from doxygen comments. We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
2018-05-01 17:54:18 +02:00
/// Weight for a branch taken going into a cold block.
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 14:26:52 +02:00
///
/// This is the weight for a branch taken toward a block marked
/// cold. A block is marked cold if it's postdominated by a
/// block containing a call to a cold function. Cold functions
/// are those marked with attribute 'cold'.
static const uint32_t CC_TAKEN_WEIGHT = 4;
Remove \brief commands from doxygen comments. We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
2018-05-01 17:54:18 +02:00
/// Weight for a branch not-taken into a cold block.
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 14:26:52 +02:00
///
/// This is the weight for a branch not taken toward a block marked
/// cold.
static const uint32_t CC_NONTAKEN_WEIGHT = 64;
Simplify the design of BranchProbabilityInfo by collapsing it into a single class. Previously it was split between two classes, one internal and one external. The concern seemed to center around exposing the weights used, but those can remain confined to the implementation file. Having a single class to maintain the state and analyses in use will also simplify several of the enhancements I want to make to our static heuristics. llvm-svn: 142783
2011-10-24 03:40:45 +02:00
static const uint32_t PH_TAKEN_WEIGHT = 20;
static const uint32_t PH_NONTAKEN_WEIGHT = 12;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
Revert "[BPI] Improve static heuristics for integer comparisons" This reverts commit 50c743fa713002fe4e0c76d23043e6c1f9e9fe6f. Patch will be split to smaller ones.
2020-08-17 20:42:57 +02:00
static const uint32_t ZH_TAKEN_WEIGHT = 20;
static const uint32_t ZH_NONTAKEN_WEIGHT = 12;
Simplify the design of BranchProbabilityInfo by collapsing it into a single class. Previously it was split between two classes, one internal and one external. The concern seemed to center around exposing the weights used, but those can remain confined to the implementation file. Having a single class to maintain the state and analyses in use will also simplify several of the enhancements I want to make to our static heuristics. llvm-svn: 142783
2011-10-24 03:40:45 +02:00
static const uint32_t FPH_TAKEN_WEIGHT = 20;
static const uint32_t FPH_NONTAKEN_WEIGHT = 12;
[BPI] Adjust the probability for floating point unordered comparison Since NaN is very rare in normal programs, so the probability for floating point unordered comparison should be extremely small. Current probability is 3/8, it is too large, this patch changes it to a tiny number. Differential Revision: https://reviews.llvm.org/D65303 llvm-svn: 371541
2019-09-10 19:25:11 +02:00
/// This is the probability for an ordered floating point comparison.
static const uint32_t FPH_ORD_WEIGHT = 1024 * 1024 - 1;
/// This is the probability for an unordered floating point comparison, it means
/// one or two of the operands are NaN. Usually it is used to test for an
/// exceptional case, so the result is unlikely.
static const uint32_t FPH_UNO_WEIGHT = 1;
Remove \brief commands from doxygen comments. We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
2018-05-01 17:54:18 +02:00
/// Invoke-terminating normal branch taken weight
Set the branch probability of branching to the 'normal' destination of an invoke instruction to something absurdly high, while setting the probability of branching to the 'unwind' destination to the bare minimum. This should set cause the normal destination's invoke blocks to be moved closer to the invoke. PR13612 llvm-svn: 161944
2012-08-15 14:22:35 +02:00
///
/// This is the weight for branching to the normal destination of an invoke
/// instruction. We expect this to happen most of the time. Set the weight to an
/// absurdly high value so that nested loops subsume it.
static const uint32_t IH_TAKEN_WEIGHT = 1024 * 1024 - 1;
Remove \brief commands from doxygen comments. We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
2018-05-01 17:54:18 +02:00
/// Invoke-terminating normal branch not-taken weight.
Set the branch probability of branching to the 'normal' destination of an invoke instruction to something absurdly high, while setting the probability of branching to the 'unwind' destination to the bare minimum. This should set cause the normal destination's invoke blocks to be moved closer to the invoke. PR13612 llvm-svn: 161944
2012-08-15 14:22:35 +02:00
///
/// This is the weight for branching to the unwind destination of an invoke
/// instruction. This is essentially never taken.
static const uint32_t IH_NONTAKEN_WEIGHT = 1;
[BPI][NFC] Consolidate code to deal with SCCs under a dedicated data structure. In order to facilitate review of D79485 here is a small NFC change which restructures code around handling of SCCs in BPI. Reviewed By: davidxl Differential Revision: https://reviews.llvm.org/D84514
2020-07-24 13:57:10 +02:00
BranchProbabilityInfo::SccInfo::SccInfo(const Function &F) {
// Record SCC numbers of blocks in the CFG to identify irreducible loops.
// FIXME: We could only calculate this if the CFG is known to be irreducible
// (perhaps cache this info in LoopInfo if we can easily calculate it there?).
int SccNum = 0;
for (scc_iterator<const Function *> It = scc_begin(&F); !It.isAtEnd();
++It, ++SccNum) {
// Ignore single-block SCCs since they either aren't loops or LoopInfo will
// catch them.
const std::vector<const BasicBlock *> &Scc = *It;
if (Scc.size() == 1)
continue;
LLVM_DEBUG(dbgs() << "BPI: SCC " << SccNum << ":");
for (const auto *BB : Scc) {
LLVM_DEBUG(dbgs() << " " << BB->getName());
SccNums[BB] = SccNum;
calculateSccBlockType(BB, SccNum);
}
LLVM_DEBUG(dbgs() << "\n");
}
}
int BranchProbabilityInfo::SccInfo::getSCCNum(const BasicBlock *BB) const {
auto SccIt = SccNums.find(BB);
if (SccIt == SccNums.end())
return -1;
return SccIt->second;
}
void BranchProbabilityInfo::SccInfo::getSccEnterBlocks(
int SccNum, SmallVectorImpl<BasicBlock *> &Enters) const {
for (auto MapIt : SccBlocks[SccNum]) {
const auto *BB = MapIt.first;
if (isSCCHeader(BB, SccNum))
for (const auto *Pred : predecessors(BB))
if (getSCCNum(Pred) != SccNum)
Enters.push_back(const_cast<BasicBlock *>(BB));
}
}
void BranchProbabilityInfo::SccInfo::getSccExitBlocks(
int SccNum, SmallVectorImpl<BasicBlock *> &Exits) const {
for (auto MapIt : SccBlocks[SccNum]) {
const auto *BB = MapIt.first;
if (isSCCExitingBlock(BB, SccNum))
for (const auto *Succ : successors(BB))
if (getSCCNum(Succ) != SccNum)
Exits.push_back(const_cast<BasicBlock *>(BB));
}
}
uint32_t BranchProbabilityInfo::SccInfo::getSccBlockType(const BasicBlock *BB,
int SccNum) const {
assert(getSCCNum(BB) == SccNum);
assert(SccBlocks.size() > static_cast<unsigned>(SccNum) && "Unknown SCC");
const auto &SccBlockTypes = SccBlocks[SccNum];
auto It = SccBlockTypes.find(BB);
if (It != SccBlockTypes.end()) {
return It->second;
}
return Inner;
}
void BranchProbabilityInfo::SccInfo::calculateSccBlockType(const BasicBlock *BB,
int SccNum) {
assert(getSCCNum(BB) == SccNum);
uint32_t BlockType = Inner;
if (llvm::any_of(make_range(pred_begin(BB), pred_end(BB)),
[&](const BasicBlock *Pred) {
// Consider any block that is an entry point to the SCC as
// a header.
return getSCCNum(Pred) != SccNum;
}))
BlockType |= Header;
if (llvm::any_of(
make_range(succ_begin(BB), succ_end(BB)),
[&](const BasicBlock *Succ) { return getSCCNum(Succ) != SccNum; }))
BlockType |= Exiting;
// Lazily compute the set of headers for a given SCC and cache the results
// in the SccHeaderMap.
if (SccBlocks.size() <= static_cast<unsigned>(SccNum))
SccBlocks.resize(SccNum + 1);
auto &SccBlockTypes = SccBlocks[SccNum];
if (BlockType != Inner) {
bool IsInserted;
std::tie(std::ignore, IsInserted) =
SccBlockTypes.insert(std::make_pair(BB, BlockType));
assert(IsInserted && "Duplicated block in SCC");
}
}
[BPI][NFC] Unify handling of normal and SCC based loops This is one more NFC part extracted from D79485. Normal and SCC based loops have very different representation and have to be handled separatly each time we deal with loops. D79485 is going to introduce much more extensive use of loops what will be problematic with out this change. Reviewed By: davidxl Differential Revision: https://reviews.llvm.org/D84838
2020-07-29 14:19:00 +02:00
BranchProbabilityInfo::LoopBlock::LoopBlock(const BasicBlock *BB,
const LoopInfo &LI,
const SccInfo &SccI)
: BB(BB) {
LD.first = LI.getLoopFor(BB);
if (!LD.first) {
LD.second = SccI.getSCCNum(BB);
}
}
bool BranchProbabilityInfo::isLoopEnteringEdge(const LoopEdge &Edge) const {
const auto &SrcBlock = Edge.first;
const auto &DstBlock = Edge.second;
return (DstBlock.getLoop() &&
!DstBlock.getLoop()->contains(SrcBlock.getLoop())) ||
// Assume that SCCs can't be nested.
(DstBlock.getSccNum() != -1 &&
SrcBlock.getSccNum() != DstBlock.getSccNum());
}
bool BranchProbabilityInfo::isLoopExitingEdge(const LoopEdge &Edge) const {
return isLoopEnteringEdge({Edge.second, Edge.first});
}
bool BranchProbabilityInfo::isLoopEnteringExitingEdge(
const LoopEdge &Edge) const {
return isLoopEnteringEdge(Edge) || isLoopExitingEdge(Edge);
}
bool BranchProbabilityInfo::isLoopBackEdge(const LoopEdge &Edge) const {
const auto &SrcBlock = Edge.first;
const auto &DstBlock = Edge.second;
return SrcBlock.belongsToSameLoop(DstBlock) &&
((DstBlock.getLoop() &&
DstBlock.getLoop()->getHeader() == DstBlock.getBlock()) ||
(DstBlock.getSccNum() != -1 &&
SccI->isSCCHeader(DstBlock.getBlock(), DstBlock.getSccNum())));
}
void BranchProbabilityInfo::getLoopEnterBlocks(
const LoopBlock &LB, SmallVectorImpl<BasicBlock *> &Enters) const {
if (LB.getLoop()) {
auto *Header = LB.getLoop()->getHeader();
Enters.append(pred_begin(Header), pred_end(Header));
} else {
assert(LB.getSccNum() != -1 && "LB doesn't belong to any loop?");
SccI->getSccEnterBlocks(LB.getSccNum(), Enters);
}
}
void BranchProbabilityInfo::getLoopExitBlocks(
const LoopBlock &LB, SmallVectorImpl<BasicBlock *> &Exits) const {
if (LB.getLoop()) {
LB.getLoop()->getExitBlocks(Exits);
} else {
assert(LB.getSccNum() != -1 && "LB doesn't belong to any loop?");
SccI->getSccExitBlocks(LB.getSccNum(), Exits);
}
}
Reland "b19ec1eb3d0c [BPI] Improve unreachable/ColdCall heurstics to handle loops." Summary: b19ec1eb3d0c has been reverted because of the test failures with PowerPC targets. This patch addresses the issues from the previous commit. Test Plan: ninja check-all. Confirmed that CodeGen/PowerPC/pr36292.ll and CodeGen/PowerPC/sms-cpy-1.ll pass Subscribers: llvm-commits
2019-12-02 19:15:22 +01:00
static void UpdatePDTWorklist(const BasicBlock *BB, PostDominatorTree *PDT,
SmallVectorImpl<const BasicBlock *> &WorkList,
SmallPtrSetImpl<const BasicBlock *> &TargetSet) {
SmallVector<BasicBlock *, 8> Descendants;
SmallPtrSet<const BasicBlock *, 16> NewItems;
PDT->getDescendants(const_cast<BasicBlock *>(BB), Descendants);
for (auto *BB : Descendants)
if (TargetSet.insert(BB).second)
for (pred_iterator PI = pred_begin(BB), E = pred_end(BB); PI != E; ++PI)
if (!TargetSet.count(*PI))
NewItems.insert(*PI);
WorkList.insert(WorkList.end(), NewItems.begin(), NewItems.end());
}
Simplify the design of BranchProbabilityInfo by collapsing it into a single class. Previously it was split between two classes, one internal and one external. The concern seemed to center around exposing the weights used, but those can remain confined to the implementation file. Having a single class to maintain the state and analyses in use will also simplify several of the enhancements I want to make to our static heuristics. llvm-svn: 142783
2011-10-24 03:40:45 +02:00
Reland "b19ec1eb3d0c [BPI] Improve unreachable/ColdCall heurstics to handle loops." Summary: b19ec1eb3d0c has been reverted because of the test failures with PowerPC targets. This patch addresses the issues from the previous commit. Test Plan: ninja check-all. Confirmed that CodeGen/PowerPC/pr36292.ll and CodeGen/PowerPC/sms-cpy-1.ll pass Subscribers: llvm-commits
2019-12-02 19:15:22 +01:00
/// Compute a set of basic blocks that are post-dominated by unreachables.
void BranchProbabilityInfo::computePostDominatedByUnreachable(
const Function &F, PostDominatorTree *PDT) {
SmallVector<const BasicBlock *, 8> WorkList;
for (auto &BB : F) {
const Instruction *TI = BB.getTerminator();
if (TI->getNumSuccessors() == 0) {
if (isa<UnreachableInst>(TI) ||
// If this block is terminated by a call to
// @llvm.experimental.deoptimize then treat it like an unreachable
// since the @llvm.experimental.deoptimize call is expected to
// practically never execute.
BB.getTerminatingDeoptimizeCall())
UpdatePDTWorklist(&BB, PDT, WorkList, PostDominatedByUnreachable);
}
[BPI] Improve unreachable/ColdCall heurstics to handle loops. Summary: While updatePostDominatedByUnreachable attemps to find basic blocks that are post-domianted by unreachable blocks, it currently cannot handle loops precisely, because it doesn't use the actual post dominator tree analysis but relies on heuristics of visiting basic blocks in post-order. More precisely, when the entire loop is post-dominated by the unreachable block, current algorithm fails to detect the entire loop as post-dominated by the unreachable because when the algorithm reaches to the loop latch it fails to tell all its successors (including the loop header) will "eventually" be post-domianted by the unreachable block, because the algorithm hasn't visited the loop header yet. This makes BPI for the loop latch to assume that loop backedges are taken with 100% of probability. And because of this, block frequency info sometimes marks virtually dead loops (which are post dominated by unreachable blocks) super hot, because 100% backedge-taken probability makes the loop iteration count the max value. updatePostDominatedByColdCall has the exact same problem as well. To address this problem, this patch makes PostDominatedByUnreachable/PostDominatedByColdCall to be computed with the actual post-dominator tree. Reviewers: skatkov, chandlerc, manmanren Reviewed By: skatkov Subscribers: manmanren, vsk, apilipenko, Carrot, qcolombet, hiraditya, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D70104
2019-11-27 19:18:01 +01:00
}
Revert b19ec1eb3d0c Summary: This reverts commit b19ec1eb3d0c as it fails powerpc tests Subscribers: llvm-commits
2019-11-27 20:17:10 +01:00
Reland "b19ec1eb3d0c [BPI] Improve unreachable/ColdCall heurstics to handle loops." Summary: b19ec1eb3d0c has been reverted because of the test failures with PowerPC targets. This patch addresses the issues from the previous commit. Test Plan: ninja check-all. Confirmed that CodeGen/PowerPC/pr36292.ll and CodeGen/PowerPC/sms-cpy-1.ll pass Subscribers: llvm-commits
2019-12-02 19:15:22 +01:00
while (!WorkList.empty()) {
const BasicBlock *BB = WorkList.pop_back_val();
if (PostDominatedByUnreachable.count(BB))
continue;
// If the terminator is an InvokeInst, check only the normal destination
// block as the unwind edge of InvokeInst is also very unlikely taken.
if (auto *II = dyn_cast<InvokeInst>(BB->getTerminator())) {
if (PostDominatedByUnreachable.count(II->getNormalDest()))
UpdatePDTWorklist(BB, PDT, WorkList, PostDominatedByUnreachable);
}
// If all the successors are unreachable, BB is unreachable as well.
else if (!successors(BB).empty() &&
llvm::all_of(successors(BB), [this](const BasicBlock *Succ) {
return PostDominatedByUnreachable.count(Succ);
}))
UpdatePDTWorklist(BB, PDT, WorkList, PostDominatedByUnreachable);
}
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 07:42:14 +02:00
}
Reland "b19ec1eb3d0c [BPI] Improve unreachable/ColdCall heurstics to handle loops." Summary: b19ec1eb3d0c has been reverted because of the test failures with PowerPC targets. This patch addresses the issues from the previous commit. Test Plan: ninja check-all. Confirmed that CodeGen/PowerPC/pr36292.ll and CodeGen/PowerPC/sms-cpy-1.ll pass Subscribers: llvm-commits
2019-12-02 19:15:22 +01:00
/// compute a set of basic blocks that are post-dominated by ColdCalls.
void BranchProbabilityInfo::computePostDominatedByColdCall(
const Function &F, PostDominatorTree *PDT) {
SmallVector<const BasicBlock *, 8> WorkList;
for (auto &BB : F)
for (auto &I : BB)
if (const CallInst *CI = dyn_cast<CallInst>(&I))
if (CI->hasFnAttr(Attribute::Cold))
UpdatePDTWorklist(&BB, PDT, WorkList, PostDominatedByColdCall);
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 07:42:14 +02:00
Reland "b19ec1eb3d0c [BPI] Improve unreachable/ColdCall heurstics to handle loops." Summary: b19ec1eb3d0c has been reverted because of the test failures with PowerPC targets. This patch addresses the issues from the previous commit. Test Plan: ninja check-all. Confirmed that CodeGen/PowerPC/pr36292.ll and CodeGen/PowerPC/sms-cpy-1.ll pass Subscribers: llvm-commits
2019-12-02 19:15:22 +01:00
while (!WorkList.empty()) {
const BasicBlock *BB = WorkList.pop_back_val();
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 07:42:14 +02:00
Reland "b19ec1eb3d0c [BPI] Improve unreachable/ColdCall heurstics to handle loops." Summary: b19ec1eb3d0c has been reverted because of the test failures with PowerPC targets. This patch addresses the issues from the previous commit. Test Plan: ninja check-all. Confirmed that CodeGen/PowerPC/pr36292.ll and CodeGen/PowerPC/sms-cpy-1.ll pass Subscribers: llvm-commits
2019-12-02 19:15:22 +01:00
// If the terminator is an InvokeInst, check only the normal destination
// block as the unwind edge of InvokeInst is also very unlikely taken.
if (auto *II = dyn_cast<InvokeInst>(BB->getTerminator())) {
if (PostDominatedByColdCall.count(II->getNormalDest()))
UpdatePDTWorklist(BB, PDT, WorkList, PostDominatedByColdCall);
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 07:42:14 +02:00
}
Reland "b19ec1eb3d0c [BPI] Improve unreachable/ColdCall heurstics to handle loops." Summary: b19ec1eb3d0c has been reverted because of the test failures with PowerPC targets. This patch addresses the issues from the previous commit. Test Plan: ninja check-all. Confirmed that CodeGen/PowerPC/pr36292.ll and CodeGen/PowerPC/sms-cpy-1.ll pass Subscribers: llvm-commits
2019-12-02 19:15:22 +01:00
// If all of successor are post dominated then BB is also done.
else if (!successors(BB).empty() &&
llvm::all_of(successors(BB), [this](const BasicBlock *Succ) {
return PostDominatedByColdCall.count(Succ);
}))
UpdatePDTWorklist(BB, PDT, WorkList, PostDominatedByColdCall);
}
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 07:42:14 +02:00
}
Remove \brief commands from doxygen comments. We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
2018-05-01 17:54:18 +02:00
/// Calculate edge weights for successors lead to unreachable.
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 07:42:14 +02:00
///
/// Predict that a successor which leads necessarily to an
/// unreachable-terminated block as extremely unlikely.
bool BranchProbabilityInfo::calcUnreachableHeuristics(const BasicBlock *BB) {
[TI removal] Make variables declared as `TerminatorInst` and initialized by `getTerminator()` calls instead be declared as `Instruction`. This is the biggest remaining chunk of the usage of `getTerminator()` that insists on the narrow type and so is an easy batch of updates. Several files saw more extensive updates where this would cascade to requiring API updates within the file to use `Instruction` instead of `TerminatorInst`. All of these were trivial in nature (pervasively using `Instruction` instead just worked). llvm-svn: 344502
2018-10-15 12:04:59 +02:00
const Instruction *TI = BB->getTerminator();
[BPI] Apply invoke heuristic before loop branch heuristic Currently the loop branch heuristic is applied before the invoke heuristic which makes us overestimate the probability of the unwind destination of invokes inside loops. This in turn makes us grossly underestimate the frequencies of loops with invokes. Reviewed By: skatkov, vsk Differential Revision: https://reviews.llvm.org/D47371 llvm-svn: 334285
2018-06-08 15:03:21 +02:00
(void) TI;
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 08:39:47 +02:00
assert(TI->getNumSuccessors() > 1 && "expected more than one successor!");
[BPI] Apply invoke heuristic before loop branch heuristic Currently the loop branch heuristic is applied before the invoke heuristic which makes us overestimate the probability of the unwind destination of invokes inside loops. This in turn makes us grossly underestimate the frequencies of loops with invokes. Reviewed By: skatkov, vsk Differential Revision: https://reviews.llvm.org/D47371 llvm-svn: 334285
2018-06-08 15:03:21 +02:00
assert(!isa<InvokeInst>(TI) &&
"Invokes should have already been handled by calcInvokeHeuristics");
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 07:42:14 +02:00
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-24 20:14:27 +02:00
SmallVector<unsigned, 4> UnreachableEdges;
SmallVector<unsigned, 4> ReachableEdges;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
[CFG/BasicBlock] Rename succ_const to const_succ. [NFC] Summary: Rename `succ_const_iterator` to `const_succ_iterator` and `succ_const_range` to `const_succ_range` for consistency with the predecessor iterators, and the corresponding iterators in MachineBasicBlock. Reviewers: nicholas, dblaikie, nlewycky Subscribers: hiraditya, bmahjour, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D75952
2020-03-10 19:33:02 +01:00
for (const_succ_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I)
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 14:01:08 +02:00
if (PostDominatedByUnreachable.count(*I))
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-24 20:14:27 +02:00
UnreachableEdges.push_back(I.getSuccessorIndex());
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 14:01:08 +02:00
else
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-24 20:14:27 +02:00
ReachableEdges.push_back(I.getSuccessorIndex());
Simplify the design of BranchProbabilityInfo by collapsing it into a single class. Previously it was split between two classes, one internal and one external. The concern seemed to center around exposing the weights used, but those can remain confined to the implementation file. Having a single class to maintain the state and analyses in use will also simplify several of the enhancements I want to make to our static heuristics. llvm-svn: 142783
2011-10-24 03:40:45 +02:00
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 08:39:47 +02:00
// Skip probabilities if all were reachable.
if (UnreachableEdges.empty())
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 07:42:14 +02:00
return false;
Enhance BranchProbabilityInfo::calcUnreachableHeuristics for InvokeInst This is recommit of r256028 with minor fixes in unittests: CodeGen/Mips/eh.ll CodeGen/Mips/insn-zero-size-bb.ll Original commit message: When identifying blocks post-dominated by an unreachable-terminated block in BranchProbabilityInfo, consider only the edge to the normal destination block if the terminator is InvokeInst and let calcInvokeHeuristics() decide edge weights for the InvokeInst. llvm-svn: 256202
2015-12-21 23:00:51 +01:00
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
SmallVector<BranchProbability, 4> EdgeProbabilities(
BB->getTerminator()->getNumSuccessors(), BranchProbability::getUnknown());
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
if (ReachableEdges.empty()) {
BranchProbability Prob(1, UnreachableEdges.size());
for (unsigned SuccIdx : UnreachableEdges)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
EdgeProbabilities[SuccIdx] = Prob;
setEdgeProbability(BB, EdgeProbabilities);
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 14:01:08 +02:00
return true;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
}
[BPI] Reduce the probability of unreachable edge to minimal value greater than 0 The probability of edge coming to unreachable block should be as low as possible. The change reduces the probability to minimal value greater than zero. The bug https://bugs.llvm.org/show_bug.cgi?id=32214 show the example when the probability of edge coming to unreachable block is greater than for edge coming to out of the loop and it causes incorrect loop rotation. Please note that with this change the behavior of unreachable heuristic is a bit different than others. Specifically, before this change the sum of probabilities coming to unreachable blocks have the same weight for all branches (it was just split over all edges of this block coming to unreachable blocks). With this change it might be slightly different but not to much due to probability of taken branch to unreachable block is really small. Reviewers: chandlerc, sanjoy, vsk, congh, junbuml, davidxl, dexonsmith Reviewed By: chandlerc, dexonsmith Subscribers: reames, llvm-commits Differential Revision: https://reviews.llvm.org/D30633 llvm-svn: 303327
2017-05-18 08:11:56 +02:00
auto UnreachableProb = UR_TAKEN_PROB;
auto ReachableProb =
(BranchProbability::getOne() - UR_TAKEN_PROB * UnreachableEdges.size()) /
ReachableEdges.size();
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
for (unsigned SuccIdx : UnreachableEdges)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
EdgeProbabilities[SuccIdx] = UnreachableProb;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
for (unsigned SuccIdx : ReachableEdges)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
EdgeProbabilities[SuccIdx] = ReachableProb;
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 14:01:08 +02:00
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
setEdgeProbability(BB, EdgeProbabilities);
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 14:01:08 +02:00
return true;
}
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
Teach the BranchProbabilityInfo analysis pass to read any metadata encoding of probabilities. In the absense of metadata, it continues to fall back on static heuristics. This allows __builtin_expect, after lowering through llvm.expect a branch instruction's metadata, to actually enter the branch probability model. This is one component of resolving PR2577. llvm-svn: 142492
2011-10-19 12:30:30 +02:00
// Propagate existing explicit probabilities from either profile data or
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
// 'expect' intrinsic processing. Examine metadata against unreachable
// heuristic. The probability of the edge coming to unreachable block is
// set to min of metadata and unreachable heuristic.
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-07 23:59:28 +02:00
bool BranchProbabilityInfo::calcMetadataWeights(const BasicBlock *BB) {
[TI removal] Make variables declared as `TerminatorInst` and initialized by `getTerminator()` calls instead be declared as `Instruction`. This is the biggest remaining chunk of the usage of `getTerminator()` that insists on the narrow type and so is an easy batch of updates. Several files saw more extensive updates where this would cascade to requiring API updates within the file to use `Instruction` instead of `TerminatorInst`. All of these were trivial in nature (pervasively using `Instruction` instead just worked). llvm-svn: 344502
2018-10-15 12:04:59 +02:00
const Instruction *TI = BB->getTerminator();
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 08:39:47 +02:00
assert(TI->getNumSuccessors() > 1 && "expected more than one successor!");
Extend InvokeInst !prof branch_weights metadata to unwind branches Allow InvokeInst to have the second optional prof branch weight for its unwind branch. InvokeInst is a terminator with two successors. It might have its unwind branch taken many times. If so the BranchProbabilityInfo unwind branch heuristic can be inaccurate. This patch allows a higher accuracy calculated with both branch weights set. Changes: - A new section about InvokeInst is added to the BranchWeightMetadata page. It states the old information that missed in the doc and adds new about the second branch weight. - Verifier is changed to allow either 1 or 2 branch weights for InvokeInst. - A new test is written for BranchProbabilityInfo to demonstrate the main improvement of the simple fix in calcMetadataWeights(). - Several new testcases are created for Inliner. Those check that both weights are accounted for invoke instruction weight calculation. - PGOUseFunc::setBranchWeights() is fixed to be applicable to InvokeInst. Reviewers: davidxl, reames, xur, yamauchi Tags: #llvm Differential Revision: https://reviews.llvm.org/D80618
2020-06-04 10:34:14 +02:00
if (!(isa<BranchInst>(TI) || isa<SwitchInst>(TI) || isa<IndirectBrInst>(TI) ||
isa<InvokeInst>(TI)))
Teach the BranchProbabilityInfo analysis pass to read any metadata encoding of probabilities. In the absense of metadata, it continues to fall back on static heuristics. This allows __builtin_expect, after lowering through llvm.expect a branch instruction's metadata, to actually enter the branch probability model. This is one component of resolving PR2577. llvm-svn: 142492
2011-10-19 12:30:30 +02:00
return false;
Revert "IR: MDNode => Value" Instead, we're going to separate metadata from the Value hierarchy. See PR21532. This reverts commit r221375. This reverts commit r221373. This reverts commit r221359. This reverts commit r221167. This reverts commit r221027. This reverts commit r221024. This reverts commit r221023. This reverts commit r220995. This reverts commit r220994. llvm-svn: 221711
2014-11-11 22:30:22 +01:00
MDNode *WeightsNode = TI->getMetadata(LLVMContext::MD_prof);
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 12:32:19 +02:00
if (!WeightsNode)
Teach the BranchProbabilityInfo analysis pass to read any metadata encoding of probabilities. In the absense of metadata, it continues to fall back on static heuristics. This allows __builtin_expect, after lowering through llvm.expect a branch instruction's metadata, to actually enter the branch probability model. This is one component of resolving PR2577. llvm-svn: 142492
2011-10-19 12:30:30 +02:00
return false;
Fix information loss in branch probability computation. Summary: This addresses PR 22718. When branch weights are too large, they were being clamped to the range [1, MaxWeightForBB]. But this clamping is only applied to edges that go outside the range, so it distorts the relative branch probabilities. This patch changes the weight calculation to scale every branch so the relative probabilities are preserved. The scaling is done differently now. First, all the branch weights are added up, and if the sum exceeds 32 bits, it computes an integer scale to bring all the weights within the range. The patch fixes an existing test that had slightly wrong branch probabilities due to the previous clamping. It now gets branch weights scaled accordingly. Reviewers: dexonsmith Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9442 llvm-svn: 236750
2015-05-07 19:22:06 +02:00
// Check that the number of successors is manageable.
assert(TI->getNumSuccessors() < UINT32_MAX && "Too many successors");
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 12:32:19 +02:00
// Ensure there are weights for all of the successors. Note that the first
// operand to the metadata node is a name, not a weight.
if (WeightsNode->getNumOperands() != TI->getNumSuccessors() + 1)
Teach the BranchProbabilityInfo analysis pass to read any metadata encoding of probabilities. In the absense of metadata, it continues to fall back on static heuristics. This allows __builtin_expect, after lowering through llvm.expect a branch instruction's metadata, to actually enter the branch probability model. This is one component of resolving PR2577. llvm-svn: 142492
2011-10-19 12:30:30 +02:00
return false;
Fix information loss in branch probability computation. Summary: This addresses PR 22718. When branch weights are too large, they were being clamped to the range [1, MaxWeightForBB]. But this clamping is only applied to edges that go outside the range, so it distorts the relative branch probabilities. This patch changes the weight calculation to scale every branch so the relative probabilities are preserved. The scaling is done differently now. First, all the branch weights are added up, and if the sum exceeds 32 bits, it computes an integer scale to bring all the weights within the range. The patch fixes an existing test that had slightly wrong branch probabilities due to the previous clamping. It now gets branch weights scaled accordingly. Reviewers: dexonsmith Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9442 llvm-svn: 236750
2015-05-07 19:22:06 +02:00
// Build up the final weights that will be used in a temporary buffer.
Revert "Use uint64_t for branch weights instead of uint32_t" This reverts commit 10f2a0d662d8d72eaac48d3e9b31ca8dc90df5a4. More uint64_t overflows.
2020-10-31 08:15:46 +01:00
// Compute the sum of all weights to later decide whether they need to
// be scaled to fit in 32 bits.
uint64_t WeightSum = 0;
SmallVector<uint32_t, 2> Weights;
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
SmallVector<unsigned, 2> UnreachableIdxs;
SmallVector<unsigned, 2> ReachableIdxs;
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 12:32:19 +02:00
Weights.reserve(TI->getNumSuccessors());
[BrachProbablityInfo] Rename loop variables. NFC
2020-06-02 05:55:27 +02:00
for (unsigned I = 1, E = WeightsNode->getNumOperands(); I != E; ++I) {
IR: Split Metadata from Value Split `Metadata` away from the `Value` class hierarchy, as part of PR21532. Assembly and bitcode changes are in the wings, but this is the bulk of the change for the IR C++ API. I have a follow-up patch prepared for `clang`. If this breaks other sub-projects, I apologize in advance :(. Help me compile it on Darwin I'll try to fix it. FWIW, the errors should be easy to fix, so it may be simpler to just fix it yourself. This breaks the build for all metadata-related code that's out-of-tree. Rest assured the transition is mechanical and the compiler should catch almost all of the problems. Here's a quick guide for updating your code: - `Metadata` is the root of a class hierarchy with three main classes: `MDNode`, `MDString`, and `ValueAsMetadata`. It is distinct from the `Value` class hierarchy. It is typeless -- i.e., instances do *not* have a `Type`. - `MDNode`'s operands are all `Metadata *` (instead of `Value *`). - `TrackingVH<MDNode>` and `WeakVH` referring to metadata can be replaced with `TrackingMDNodeRef` and `TrackingMDRef`, respectively. If you're referring solely to resolved `MDNode`s -- post graph construction -- just use `MDNode*`. - `MDNode` (and the rest of `Metadata`) have only limited support for `replaceAllUsesWith()`. As long as an `MDNode` is pointing at a forward declaration -- the result of `MDNode::getTemporary()` -- it maintains a side map of its uses and can RAUW itself. Once the forward declarations are fully resolved RAUW support is dropped on the ground. This means that uniquing collisions on changing operands cause nodes to become "distinct". (This already happened fairly commonly, whenever an operand went to null.) If you're constructing complex (non self-reference) `MDNode` cycles, you need to call `MDNode::resolveCycles()` on each node (or on a top-level node that somehow references all of the nodes). Also, don't do that. Metadata cycles (and the RAUW machinery needed to construct them) are expensive. - An `MDNode` can only refer to a `Constant` through a bridge called `ConstantAsMetadata` (one of the subclasses of `ValueAsMetadata`). As a side effect, accessing an operand of an `MDNode` that is known to be, e.g., `ConstantInt`, takes three steps: first, cast from `Metadata` to `ConstantAsMetadata`; second, extract the `Constant`; third, cast down to `ConstantInt`. The eventual goal is to introduce `MDInt`/`MDFloat`/etc. and have metadata schema owners transition away from using `Constant`s when the type isn't important (and they don't care about referring to `GlobalValue`s). In the meantime, I've added transitional API to the `mdconst` namespace that matches semantics with the old code, in order to avoid adding the error-prone three-step equivalent to every call site. If your old code was: MDNode *N = foo(); bar(isa <ConstantInt>(N->getOperand(0))); baz(cast <ConstantInt>(N->getOperand(1))); bak(cast_or_null <ConstantInt>(N->getOperand(2))); bat(dyn_cast <ConstantInt>(N->getOperand(3))); bay(dyn_cast_or_null<ConstantInt>(N->getOperand(4))); you can trivially match its semantics with: MDNode *N = foo(); bar(mdconst::hasa <ConstantInt>(N->getOperand(0))); baz(mdconst::extract <ConstantInt>(N->getOperand(1))); bak(mdconst::extract_or_null <ConstantInt>(N->getOperand(2))); bat(mdconst::dyn_extract <ConstantInt>(N->getOperand(3))); bay(mdconst::dyn_extract_or_null<ConstantInt>(N->getOperand(4))); and when you transition your metadata schema to `MDInt`: MDNode *N = foo(); bar(isa <MDInt>(N->getOperand(0))); baz(cast <MDInt>(N->getOperand(1))); bak(cast_or_null <MDInt>(N->getOperand(2))); bat(dyn_cast <MDInt>(N->getOperand(3))); bay(dyn_cast_or_null<MDInt>(N->getOperand(4))); - A `CallInst` -- specifically, intrinsic instructions -- can refer to metadata through a bridge called `MetadataAsValue`. This is a subclass of `Value` where `getType()->isMetadataTy()`. `MetadataAsValue` is the *only* class that can legally refer to a `LocalAsMetadata`, which is a bridged form of non-`Constant` values like `Argument` and `Instruction`. It can also refer to any other `Metadata` subclass. (I'll break all your testcases in a follow-up commit, when I propagate this change to assembly.) llvm-svn: 223802
2014-12-09 19:38:53 +01:00
ConstantInt *Weight =
[BrachProbablityInfo] Rename loop variables. NFC
2020-06-02 05:55:27 +02:00
mdconst::dyn_extract<ConstantInt>(WeightsNode->getOperand(I));
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 12:32:19 +02:00
if (!Weight)
return false;
Revert "Use uint64_t for branch weights instead of uint32_t" This reverts commit 10f2a0d662d8d72eaac48d3e9b31ca8dc90df5a4. More uint64_t overflows.
2020-10-31 08:15:46 +01:00
assert(Weight->getValue().getActiveBits() <= 32 &&
"Too many bits for uint32_t");
Weights.push_back(Weight->getZExtValue());
WeightSum += Weights.back();
[BrachProbablityInfo] Rename loop variables. NFC
2020-06-02 05:55:27 +02:00
if (PostDominatedByUnreachable.count(TI->getSuccessor(I - 1)))
UnreachableIdxs.push_back(I - 1);
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
else
[BrachProbablityInfo] Rename loop variables. NFC
2020-06-02 05:55:27 +02:00
ReachableIdxs.push_back(I - 1);
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 12:32:19 +02:00
}
assert(Weights.size() == TI->getNumSuccessors() && "Checked above");
Fix information loss in branch probability computation. Summary: This addresses PR 22718. When branch weights are too large, they were being clamped to the range [1, MaxWeightForBB]. But this clamping is only applied to edges that go outside the range, so it distorts the relative branch probabilities. This patch changes the weight calculation to scale every branch so the relative probabilities are preserved. The scaling is done differently now. First, all the branch weights are added up, and if the sum exceeds 32 bits, it computes an integer scale to bring all the weights within the range. The patch fixes an existing test that had slightly wrong branch probabilities due to the previous clamping. It now gets branch weights scaled accordingly. Reviewers: dexonsmith Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9442 llvm-svn: 236750
2015-05-07 19:22:06 +02:00
Revert "Use uint64_t for branch weights instead of uint32_t" This reverts commit 10f2a0d662d8d72eaac48d3e9b31ca8dc90df5a4. More uint64_t overflows.
2020-10-31 08:15:46 +01:00
// If the sum of weights does not fit in 32 bits, scale every weight down
// accordingly.
uint64_t ScalingFactor =
(WeightSum > UINT32_MAX) ? WeightSum / UINT32_MAX + 1 : 1;
if (ScalingFactor > 1) {
WeightSum = 0;
for (unsigned I = 0, E = TI->getNumSuccessors(); I != E; ++I) {
Weights[I] /= ScalingFactor;
WeightSum += Weights[I];
}
}
assert(WeightSum <= UINT32_MAX &&
"Expected weights to scale down to 32 bits");
[BPI] Fix two potential divide-by-zero operations that are introduced in r256263. llvm-svn: 256303
2015-12-23 00:45:55 +01:00
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
if (WeightSum == 0 || ReachableIdxs.size() == 0) {
[BrachProbablityInfo] Rename loop variables. NFC
2020-06-02 05:55:27 +02:00
for (unsigned I = 0, E = TI->getNumSuccessors(); I != E; ++I)
Weights[I] = 1;
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
WeightSum = TI->getNumSuccessors();
[BPI] Fix two potential divide-by-zero operations that are introduced in r256263. llvm-svn: 256303
2015-12-23 00:45:55 +01:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
// Set the probability.
SmallVector<BranchProbability, 2> BP;
[BrachProbablityInfo] Rename loop variables. NFC
2020-06-02 05:55:27 +02:00
for (unsigned I = 0, E = TI->getNumSuccessors(); I != E; ++I)
Revert "Use uint64_t for branch weights instead of uint32_t" This reverts commit 10f2a0d662d8d72eaac48d3e9b31ca8dc90df5a4. More uint64_t overflows.
2020-10-31 08:15:46 +01:00
BP.push_back({ Weights[I], static_cast<uint32_t>(WeightSum) });
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
// Examine the metadata against unreachable heuristic.
// If the unreachable heuristic is more strong then we use it for this edge.
[BrachProbablityInfo] Proportional distribution of reachable probabilities When fixing probability of unreachable edges in BranchProbabilityInfo::calcMetadataWeights() proportionally distribute remainder probability over the reachable edges. The old implementation distributes the remainder probability evenly. See examples in the fixed tests. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D80611
2020-06-02 06:28:12 +02:00
if (UnreachableIdxs.size() == 0 || ReachableIdxs.size() == 0) {
setEdgeProbability(BB, BP);
return true;
}
auto UnreachableProb = UR_TAKEN_PROB;
for (auto I : UnreachableIdxs)
if (UnreachableProb < BP[I]) {
BP[I] = UnreachableProb;
}
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
[BrachProbablityInfo] Proportional distribution of reachable probabilities When fixing probability of unreachable edges in BranchProbabilityInfo::calcMetadataWeights() proportionally distribute remainder probability over the reachable edges. The old implementation distributes the remainder probability evenly. See examples in the fixed tests. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D80611
2020-06-02 06:28:12 +02:00
// Sum of all edge probabilities must be 1.0. If we modified the probability
// of some edges then we must distribute the introduced difference over the
// reachable blocks.
//
// Proportional distribution: the relation between probabilities of the
// reachable edges is kept unchanged. That is for any reachable edges i and j:
// newBP[i] / newBP[j] == oldBP[i] / oldBP[j] =>
// newBP[i] / oldBP[i] == newBP[j] / oldBP[j] == K
// Where K is independent of i,j.
// newBP[i] == oldBP[i] * K
// We need to find K.
// Make sum of all reachables of the left and right parts:
// sum_of_reachable(newBP) == K * sum_of_reachable(oldBP)
// Sum of newBP must be equal to 1.0:
// sum_of_reachable(newBP) + sum_of_unreachable(newBP) == 1.0 =>
// sum_of_reachable(newBP) = 1.0 - sum_of_unreachable(newBP)
// Where sum_of_unreachable(newBP) is what has been just changed.
// Finally:
// K == sum_of_reachable(newBP) / sum_of_reachable(oldBP) =>
// K == (1.0 - sum_of_unreachable(newBP)) / sum_of_reachable(oldBP)
BranchProbability NewUnreachableSum = BranchProbability::getZero();
for (auto I : UnreachableIdxs)
NewUnreachableSum += BP[I];
BranchProbability NewReachableSum =
BranchProbability::getOne() - NewUnreachableSum;
BranchProbability OldReachableSum = BranchProbability::getZero();
for (auto I : ReachableIdxs)
OldReachableSum += BP[I];
if (OldReachableSum != NewReachableSum) { // Anything to dsitribute?
if (OldReachableSum.isZero()) {
// If all oldBP[i] are zeroes then the proportional distribution results
// in all zero probabilities and the error stays big. In this case we
// evenly spread NewReachableSum over the reachable edges.
BranchProbability PerEdge = NewReachableSum / ReachableIdxs.size();
[BrachProbablityInfo] Rename loop variables. NFC
2020-06-02 05:55:27 +02:00
for (auto I : ReachableIdxs)
[BrachProbablityInfo] Proportional distribution of reachable probabilities When fixing probability of unreachable edges in BranchProbabilityInfo::calcMetadataWeights() proportionally distribute remainder probability over the reachable edges. The old implementation distributes the remainder probability evenly. See examples in the fixed tests. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D80611
2020-06-02 06:28:12 +02:00
BP[I] = PerEdge;
} else {
for (auto I : ReachableIdxs) {
// We use uint64_t to avoid double rounding error of the following
// calculation: BP[i] = BP[i] * NewReachableSum / OldReachableSum
// The formula is taken from the private constructor
// BranchProbability(uint32_t Numerator, uint32_t Denominator)
uint64_t Mul = static_cast<uint64_t>(NewReachableSum.getNumerator()) *
BP[I].getNumerator();
uint32_t Div = static_cast<uint32_t>(
divideNearest(Mul, OldReachableSum.getNumerator()));
BP[I] = BranchProbability::getRaw(Div);
}
}
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
}
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
setEdgeProbability(BB, BP);
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
Teach the BranchProbabilityInfo analysis pass to read any metadata encoding of probabilities. In the absense of metadata, it continues to fall back on static heuristics. This allows __builtin_expect, after lowering through llvm.expect a branch instruction's metadata, to actually enter the branch probability model. This is one component of resolving PR2577. llvm-svn: 142492
2011-10-19 12:30:30 +02:00
return true;
}
Remove \brief commands from doxygen comments. We've been running doxygen with the autobrief option for a couple of years now. This makes the \brief markers into our comments redundant. Since they are a visual distraction and we don't want to encourage more \brief markers in new code either, this patch removes them all. Patch produced by for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done Differential Revision: https://reviews.llvm.org/D46290 llvm-svn: 331272
2018-05-01 17:54:18 +02:00
/// Calculate edge weights for edges leading to cold blocks.
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 14:26:52 +02:00
///
/// A cold block is one post-dominated by a block with a call to a
/// cold function. Those edges are unlikely to be taken, so we give
/// them relatively low weight.
///
/// Return true if we could compute the weights for cold edges.
/// Return false, otherwise.
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-07 23:59:28 +02:00
bool BranchProbabilityInfo::calcColdCallHeuristics(const BasicBlock *BB) {
[TI removal] Make variables declared as `TerminatorInst` and initialized by `getTerminator()` calls instead be declared as `Instruction`. This is the biggest remaining chunk of the usage of `getTerminator()` that insists on the narrow type and so is an easy batch of updates. Several files saw more extensive updates where this would cascade to requiring API updates within the file to use `Instruction` instead of `TerminatorInst`. All of these were trivial in nature (pervasively using `Instruction` instead just worked). llvm-svn: 344502
2018-10-15 12:04:59 +02:00
const Instruction *TI = BB->getTerminator();
[BPI] Apply invoke heuristic before loop branch heuristic Currently the loop branch heuristic is applied before the invoke heuristic which makes us overestimate the probability of the unwind destination of invokes inside loops. This in turn makes us grossly underestimate the frequencies of loops with invokes. Reviewed By: skatkov, vsk Differential Revision: https://reviews.llvm.org/D47371 llvm-svn: 334285
2018-06-08 15:03:21 +02:00
(void) TI;
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 08:39:47 +02:00
assert(TI->getNumSuccessors() > 1 && "expected more than one successor!");
[BPI] Apply invoke heuristic before loop branch heuristic Currently the loop branch heuristic is applied before the invoke heuristic which makes us overestimate the probability of the unwind destination of invokes inside loops. This in turn makes us grossly underestimate the frequencies of loops with invokes. Reviewed By: skatkov, vsk Differential Revision: https://reviews.llvm.org/D47371 llvm-svn: 334285
2018-06-08 15:03:21 +02:00
assert(!isa<InvokeInst>(TI) &&
"Invokes should have already been handled by calcInvokeHeuristics");
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 14:26:52 +02:00
// Determine which successors are post-dominated by a cold block.
SmallVector<unsigned, 4> ColdEdges;
SmallVector<unsigned, 4> NormalEdges;
[CFG/BasicBlock] Rename succ_const to const_succ. [NFC] Summary: Rename `succ_const_iterator` to `const_succ_iterator` and `succ_const_range` to `const_succ_range` for consistency with the predecessor iterators, and the corresponding iterators in MachineBasicBlock. Reviewers: nicholas, dblaikie, nlewycky Subscribers: hiraditya, bmahjour, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D75952
2020-03-10 19:33:02 +01:00
for (const_succ_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I)
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 14:26:52 +02:00
if (PostDominatedByColdCall.count(*I))
ColdEdges.push_back(I.getSuccessorIndex());
else
NormalEdges.push_back(I.getSuccessorIndex());
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 08:39:47 +02:00
// Skip probabilities if no cold edges.
if (ColdEdges.empty())
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 14:26:52 +02:00
return false;
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
SmallVector<BranchProbability, 4> EdgeProbabilities(
BB->getTerminator()->getNumSuccessors(), BranchProbability::getUnknown());
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
if (NormalEdges.empty()) {
BranchProbability Prob(1, ColdEdges.size());
for (unsigned SuccIdx : ColdEdges)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
EdgeProbabilities[SuccIdx] = Prob;
setEdgeProbability(BB, EdgeProbabilities);
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 14:26:52 +02:00
return true;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
}
Retry: [BPI] Use a safer constructor to calculate branch probabilities BPI may trigger signed overflow UB while computing branch probabilities for cold calls or to unreachables. For example, with our current choice of weights, we'll crash if there are >= 2^12 branches to an unreachable. Use a safer BranchProbability constructor which is better at handling fractions with large denominators. Changes since the initial commit: - Use explicit casts to ensure that multiplication operands are 64-bit ints. rdar://problem/29368161 Differential Revision: https://reviews.llvm.org/D27862 llvm-svn: 290022
2016-12-17 02:02:08 +01:00
auto ColdProb = BranchProbability::getBranchProbability(
CC_TAKEN_WEIGHT,
(CC_TAKEN_WEIGHT + CC_NONTAKEN_WEIGHT) * uint64_t(ColdEdges.size()));
auto NormalProb = BranchProbability::getBranchProbability(
CC_NONTAKEN_WEIGHT,
(CC_TAKEN_WEIGHT + CC_NONTAKEN_WEIGHT) * uint64_t(NormalEdges.size()));
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
for (unsigned SuccIdx : ColdEdges)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
EdgeProbabilities[SuccIdx] = ColdProb;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
for (unsigned SuccIdx : NormalEdges)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
EdgeProbabilities[SuccIdx] = NormalProb;
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 14:26:52 +02:00
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
setEdgeProbability(BB, EdgeProbabilities);
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 14:26:52 +02:00
return true;
}
Fix more spelling mistakes in comments of LLVM Analysis passes Patch by Reshabh Sharma! Differential Revision: https://reviews.llvm.org/D43939 llvm-svn: 326601
2018-03-02 19:57:02 +01:00
// Calculate Edge Weights using "Pointer Heuristics". Predict a comparison
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
// between two pointer or pointer and NULL will fail.
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-07 23:59:28 +02:00
bool BranchProbabilityInfo::calcPointerHeuristics(const BasicBlock *BB) {
const BranchInst *BI = dyn_cast<BranchInst>(BB->getTerminator());
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
if (!BI || !BI->isConditional())
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-28 23:45:07 +02:00
return false;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
Value *Cond = BI->getCondition();
ICmpInst *CI = dyn_cast<ICmpInst>(Cond);
Fix pointer heuristic. Check whether predicator is ICMP_NE instead of if it is not isEquality(). llvm-svn: 135296
2011-07-15 22:51:06 +02:00
if (!CI || !CI->isEquality())
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-28 23:45:07 +02:00
return false;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
Value *LHS = CI->getOperand(0);
if (!LHS->getType()->isPointerTy())
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-28 23:45:07 +02:00
return false;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
Fold assert-only-used variable into the assert. llvm-svn: 132620
2011-06-04 04:07:10 +02:00
assert(CI->getOperand(1)->getType()->isPointerTy());
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
BranchProbability TakenProb(PH_TAKEN_WEIGHT,
PH_TAKEN_WEIGHT + PH_NONTAKEN_WEIGHT);
BranchProbability UntakenProb(PH_NONTAKEN_WEIGHT,
PH_TAKEN_WEIGHT + PH_NONTAKEN_WEIGHT);
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
// p != 0 -> isProb = true
// p == 0 -> isProb = false
// p != q -> isProb = true
// p == q -> isProb = false;
Fix pointer heuristic. Check whether predicator is ICMP_NE instead of if it is not isEquality(). llvm-svn: 135296
2011-07-15 22:51:06 +02:00
bool isProb = CI->getPredicate() == ICmpInst::ICMP_NE;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
if (!isProb)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
std::swap(TakenProb, UntakenProb);
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
setEdgeProbability(
BB, SmallVector<BranchProbability, 2>({TakenProb, UntakenProb}));
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-28 23:45:07 +02:00
return true;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
}
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
// Compute the unlikely successors to the block BB in the loop L, specifically
// those that are unlikely because this is a loop, and add them to the
// UnlikelyBlocks set.
static void
computeUnlikelySuccessors(const BasicBlock *BB, Loop *L,
SmallPtrSetImpl<const BasicBlock*> &UnlikelyBlocks) {
// Sometimes in a loop we have a branch whose condition is made false by
// taking it. This is typically something like
// int n = 0;
// while (...) {
// if (++n >= MAX) {
// n = 0;
// }
// }
// In this sort of situation taking the branch means that at the very least it
// won't be taken again in the next iteration of the loop, so we should
// consider it less likely than a typical branch.
//
// We detect this by looking back through the graph of PHI nodes that sets the
// value that the condition depends on, and seeing if we can reach a successor
// block which can be determined to make the condition false.
//
// FIXME: We currently consider unlikely blocks to be half as likely as other
// blocks, but if we consider the example above the likelyhood is actually
// 1/MAX. We could therefore be more precise in how unlikely we consider
// blocks to be, but it would require more careful examination of the form
// of the comparison expression.
const BranchInst *BI = dyn_cast<BranchInst>(BB->getTerminator());
if (!BI || !BI->isConditional())
return;
// Check if the branch is based on an instruction compared with a constant
CmpInst *CI = dyn_cast<CmpInst>(BI->getCondition());
if (!CI || !isa<Instruction>(CI->getOperand(0)) ||
!isa<Constant>(CI->getOperand(1)))
return;
// Either the instruction must be a PHI, or a chain of operations involving
// constants that ends in a PHI which we can then collapse into a single value
// if the PHI value is known.
Instruction *CmpLHS = dyn_cast<Instruction>(CI->getOperand(0));
PHINode *CmpPHI = dyn_cast<PHINode>(CmpLHS);
Constant *CmpConst = dyn_cast<Constant>(CI->getOperand(1));
// Collect the instructions until we hit a PHI
[BPI] Remove unnecessary std::list vector is sufficient here. No functionality change intended. llvm-svn: 334865
2018-06-15 23:06:43 +02:00
SmallVector<BinaryOperator *, 1> InstChain;
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
while (!CmpPHI && CmpLHS && isa<BinaryOperator>(CmpLHS) &&
isa<Constant>(CmpLHS->getOperand(1))) {
// Stop if the chain extends outside of the loop
if (!L->contains(CmpLHS))
return;
[BPI] Remove unnecessary std::list vector is sufficient here. No functionality change intended. llvm-svn: 334865
2018-06-15 23:06:43 +02:00
InstChain.push_back(cast<BinaryOperator>(CmpLHS));
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
CmpLHS = dyn_cast<Instruction>(CmpLHS->getOperand(0));
if (CmpLHS)
CmpPHI = dyn_cast<PHINode>(CmpLHS);
}
if (!CmpPHI || !L->contains(CmpPHI))
return;
// Trace the phi node to find all values that come from successors of BB
SmallPtrSet<PHINode*, 8> VisitedInsts;
SmallVector<PHINode*, 8> WorkList;
WorkList.push_back(CmpPHI);
VisitedInsts.insert(CmpPHI);
while (!WorkList.empty()) {
PHINode *P = WorkList.back();
WorkList.pop_back();
for (BasicBlock *B : P->blocks()) {
// Skip blocks that aren't part of the loop
if (!L->contains(B))
continue;
Value *V = P->getIncomingValueForBlock(B);
// If the source is a PHI add it to the work list if we haven't
// already visited it.
if (PHINode *PN = dyn_cast<PHINode>(V)) {
if (VisitedInsts.insert(PN).second)
WorkList.push_back(PN);
continue;
}
// If this incoming value is a constant and B is a successor of BB, then
// we can constant-evaluate the compare to see if it makes the branch be
// taken or not.
Constant *CmpLHSConst = dyn_cast<Constant>(V);
Use llvm::is_contained where appropriate (NFC) Use llvm::is_contained where appropriate (NFC) Reviewed By: kazu Differential Revision: https://reviews.llvm.org/D85083
2020-08-02 06:49:38 +02:00
if (!CmpLHSConst || !llvm::is_contained(successors(BB), B))
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
continue;
// First collapse InstChain
[BPI] Remove unnecessary std::list vector is sufficient here. No functionality change intended. llvm-svn: 334865
2018-06-15 23:06:43 +02:00
for (Instruction *I : llvm::reverse(InstChain)) {
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
CmpLHSConst = ConstantExpr::get(I->getOpcode(), CmpLHSConst,
[BPI] Remove unnecessary std::list vector is sufficient here. No functionality change intended. llvm-svn: 334865
2018-06-15 23:06:43 +02:00
cast<Constant>(I->getOperand(1)), true);
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
if (!CmpLHSConst)
break;
}
if (!CmpLHSConst)
continue;
// Now constant-evaluate the compare
Constant *Result = ConstantExpr::getCompare(CI->getPredicate(),
CmpLHSConst, CmpConst, true);
// If the result means we don't branch to the block then that block is
// unlikely.
if (Result &&
((Result->isZeroValue() && B == BI->getSuccessor(0)) ||
(Result->isOneValue() && B == BI->getSuccessor(1))))
UnlikelyBlocks.insert(B);
}
}
}
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
// Calculate Edge Weights using "Loop Branch Heuristics". Predict backedges
// as taken, exiting edges as not-taken.
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-07 23:59:28 +02:00
bool BranchProbabilityInfo::calcLoopBranchHeuristics(const BasicBlock *BB,
[BPI][NFC] Consolidate code to deal with SCCs under a dedicated data structure. In order to facilitate review of D79485 here is a small NFC change which restructures code around handling of SCCs in BPI. Reviewed By: davidxl Differential Revision: https://reviews.llvm.org/D84514
2020-07-24 13:57:10 +02:00
const LoopInfo &LI) {
[BPI][NFC] Unify handling of normal and SCC based loops This is one more NFC part extracted from D79485. Normal and SCC based loops have very different representation and have to be handled separatly each time we deal with loops. D79485 is going to introduce much more extensive use of loops what will be problematic with out this change. Reviewed By: davidxl Differential Revision: https://reviews.llvm.org/D84838
2020-07-29 14:19:00 +02:00
LoopBlock LB(BB, LI, *SccI.get());
if (!LB.belongsToLoop())
return false;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
SmallPtrSet<const BasicBlock*, 8> UnlikelyBlocks;
[BPI][NFC] Unify handling of normal and SCC based loops This is one more NFC part extracted from D79485. Normal and SCC based loops have very different representation and have to be handled separatly each time we deal with loops. D79485 is going to introduce much more extensive use of loops what will be problematic with out this change. Reviewed By: davidxl Differential Revision: https://reviews.llvm.org/D84838
2020-07-29 14:19:00 +02:00
if (LB.getLoop())
computeUnlikelySuccessors(BB, LB.getLoop(), UnlikelyBlocks);
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-24 20:14:27 +02:00
SmallVector<unsigned, 8> BackEdges;
SmallVector<unsigned, 8> ExitingEdges;
SmallVector<unsigned, 8> InEdges; // Edges from header to the loop.
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
SmallVector<unsigned, 8> UnlikelyEdges;
Add InEdges (edges from header to the loop) in Loop Branch Heuristics, so there is no frequency difference whether condition is in the header or in the latch. llvm-svn: 136398
2011-07-28 23:33:46 +02:00
[CFG/BasicBlock] Rename succ_const to const_succ. [NFC] Summary: Rename `succ_const_iterator` to `const_succ_iterator` and `succ_const_range` to `const_succ_range` for consistency with the predecessor iterators, and the corresponding iterators in MachineBasicBlock. Reviewers: nicholas, dblaikie, nlewycky Subscribers: hiraditya, bmahjour, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D75952
2020-03-10 19:33:02 +01:00
for (const_succ_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) {
[BPI][NFC] Unify handling of normal and SCC based loops This is one more NFC part extracted from D79485. Normal and SCC based loops have very different representation and have to be handled separatly each time we deal with loops. D79485 is going to introduce much more extensive use of loops what will be problematic with out this change. Reviewed By: davidxl Differential Revision: https://reviews.llvm.org/D84838
2020-07-29 14:19:00 +02:00
LoopBlock SuccLB(*I, LI, *SccI.get());
LoopEdge Edge(LB, SuccLB);
bool IsUnlikelyEdge =
LB.getLoop() && (UnlikelyBlocks.find(*I) != UnlikelyBlocks.end());
if (IsUnlikelyEdge)
UnlikelyEdges.push_back(I.getSuccessorIndex());
else if (isLoopExitingEdge(Edge))
ExitingEdges.push_back(I.getSuccessorIndex());
else if (isLoopBackEdge(Edge))
BackEdges.push_back(I.getSuccessorIndex());
else {
InEdges.push_back(I.getSuccessorIndex());
[BranchProbabilityInfo] Handle irreducible loops. Summary: Compute the strongly connected components of the CFG and fall back to use these for blocks that are in loops that are not detected by LoopInfo when computing loop back-edge and exit branch probabilities. Reviewers: dexonsmith, davidxl Subscribers: mcrosier, llvm-commits Differential Revision: https://reviews.llvm.org/D39385 llvm-svn: 317094
2017-11-01 16:16:50 +01:00
}
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
}
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
if (BackEdges.empty() && ExitingEdges.empty() && UnlikelyEdges.empty())
Fix a bug in which BranchProbabilityInfo wasn't setting branch weights of basic blocks inside loops correctly. Previously, BranchProbabilityInfo::calcLoopBranchHeuristics would determine the weights of basic blocks inside loops even when it didn't have enough information to estimate the branch probabilities correctly. This patch fixes the function to exit early if it doesn't see any exit edges or back edges and let the later heuristics determine the weights. This fixes PR18705 and <rdar://problem/15991090>. Differential Revision: http://reviews.llvm.org/D3363 llvm-svn: 206194
2014-04-14 18:56:19 +02:00
return false;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
// Collect the sum of probabilities of back-edges/in-edges/exiting-edges, and
// normalize them so that they sum up to one.
unsigned Denom = (BackEdges.empty() ? 0 : LBH_TAKEN_WEIGHT) +
(InEdges.empty() ? 0 : LBH_TAKEN_WEIGHT) +
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
(UnlikelyEdges.empty() ? 0 : LBH_UNLIKELY_WEIGHT) +
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
(ExitingEdges.empty() ? 0 : LBH_NONTAKEN_WEIGHT);
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
SmallVector<BranchProbability, 4> EdgeProbabilities(
BB->getTerminator()->getNumSuccessors(), BranchProbability::getUnknown());
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
if (uint32_t numBackEdges = BackEdges.size()) {
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
BranchProbability TakenProb = BranchProbability(LBH_TAKEN_WEIGHT, Denom);
auto Prob = TakenProb / numBackEdges;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
for (unsigned SuccIdx : BackEdges)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
EdgeProbabilities[SuccIdx] = Prob;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
}
Add InEdges (edges from header to the loop) in Loop Branch Heuristics, so there is no frequency difference whether condition is in the header or in the latch. llvm-svn: 136398
2011-07-28 23:33:46 +02:00
if (uint32_t numInEdges = InEdges.size()) {
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
BranchProbability TakenProb = BranchProbability(LBH_TAKEN_WEIGHT, Denom);
auto Prob = TakenProb / numInEdges;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
for (unsigned SuccIdx : InEdges)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
EdgeProbabilities[SuccIdx] = Prob;
Add InEdges (edges from header to the loop) in Loop Branch Heuristics, so there is no frequency difference whether condition is in the header or in the latch. llvm-svn: 136398
2011-07-28 23:33:46 +02:00
}
Fix the API usage in loop probability heuristics. It was incorrectly classifying many edges as exiting which were in fact not. These mainly formed edges into sub-loops. It was also not correctly classifying all returning edges out of loops as leaving the loop. With this match most of the loop heuristics are more rational. Several serious regressions on loop-intesive benchmarks like perlbench's loop tests when built with -enable-block-placement are fixed by these updated heuristics. Unfortunately they in turn uncover some other regressions. There are still several improvemenst that should be made to loop heuristics including trip-count, and early back-edge management. llvm-svn: 142917
2011-10-25 11:47:41 +02:00
if (uint32_t numExitingEdges = ExitingEdges.size()) {
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
BranchProbability NotTakenProb = BranchProbability(LBH_NONTAKEN_WEIGHT,
Denom);
auto Prob = NotTakenProb / numExitingEdges;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
for (unsigned SuccIdx : ExitingEdges)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
EdgeProbabilities[SuccIdx] = Prob;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
}
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-28 23:45:07 +02:00
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
if (uint32_t numUnlikelyEdges = UnlikelyEdges.size()) {
BranchProbability UnlikelyProb = BranchProbability(LBH_UNLIKELY_WEIGHT,
Denom);
auto Prob = UnlikelyProb / numUnlikelyEdges;
for (unsigned SuccIdx : UnlikelyEdges)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
EdgeProbabilities[SuccIdx] = Prob;
[BPI] Detect branches in loops that make themselves not taken If we have a loop like this: int n = 0; while (...) { if (++n >= MAX) { n = 0; } } then the body of the 'if' statement will only be executed once every MAX iterations. Detect this by looking for branches in loops where taking the branch makes the branch condition evaluate to 'not taken' in the next iteration of the loop, and reduce the probability of such branches. This slightly improves EEMBC benchmarks on cortex-m4/cortex-m33 due to making better choices in if-conversion, but has no effect on any other cpu/benchmark that I could detect. Differential Revision: https://reviews.llvm.org/D35804 llvm-svn: 325925
2018-02-23 18:17:31 +01:00
}
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
setEdgeProbability(BB, EdgeProbabilities);
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-28 23:45:07 +02:00
return true;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
}
Revert "[BPI] Improve static heuristics for integer comparisons" This reverts commit 50c743fa713002fe4e0c76d23043e6c1f9e9fe6f. Patch will be split to smaller ones.
2020-08-17 20:42:57 +02:00
bool BranchProbabilityInfo::calcZeroHeuristics(const BasicBlock *BB,
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 11:44:40 +02:00
const TargetLibraryInfo *TLI) {
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-07 23:59:28 +02:00
const BranchInst *BI = dyn_cast<BranchInst>(BB->getTerminator());
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 05:27:24 +02:00
if (!BI || !BI->isConditional())
return false;
Value *Cond = BI->getCondition();
ICmpInst *CI = dyn_cast<ICmpInst>(Cond);
if (!CI)
return false;
[BPI] Look through bitcasts in calcZeroHeuristic Constant hoisting may have hidden a constant behind a bitcast so that it isn't folded into its users. However, this prevents BPI from calculating some of its heuristics that are based upon constant values. So, I've added a simple helper function to look through these casts. Differential Revision: https://reviews.llvm.org/D58166 llvm-svn: 354119
2019-02-15 12:50:21 +01:00
auto GetConstantInt = [](Value *V) {
if (auto *I = dyn_cast<BitCastInst>(V))
return dyn_cast<ConstantInt>(I->getOperand(0));
return dyn_cast<ConstantInt>(V);
};
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 05:27:24 +02:00
Value *RHS = CI->getOperand(1);
[BPI] Look through bitcasts in calcZeroHeuristic Constant hoisting may have hidden a constant behind a bitcast so that it isn't folded into its users. However, this prevents BPI from calculating some of its heuristics that are based upon constant values. So, I've added a simple helper function to look through these casts. Differential Revision: https://reviews.llvm.org/D58166 llvm-svn: 354119
2019-02-15 12:50:21 +01:00
ConstantInt *CV = GetConstantInt(RHS);
Revert "[BPI] Improve static heuristics for integer comparisons" This reverts commit 50c743fa713002fe4e0c76d23043e6c1f9e9fe6f. Patch will be split to smaller ones.
2020-08-17 20:42:57 +02:00
if (!CV)
return false;
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 05:27:24 +02:00
Re-apply r234898 and fix tests. This commit makes LLVM not estimate branch probabilities when doing a single bit bitmask tests. The code that originally made me discover this is: if ((a & 0x1) == 0x1) { .. } In this case we don't actually have any branch probability information and should not assume to have any. LLVM transforms this into: %and = and i32 %a, 1 %tobool = icmp eq i32 %and, 0 So, in this case, the result of a bitwise and is compared against 0, but nevertheless, we should not assume to have probability information. CodeGen/ARM/2013-10-11-select-stalls.ll started failing because the changed probabilities changed the results of ARMBaseInstrInfo::isProfitableToIfCvt() and led to an Ifcvt of the diamond in the test. AFAICT, the test was never meant to test this and thus changing the test input slightly to not change the probabilities seems like the best way to preserve the meaning of the test. llvm-svn: 234979
2015-04-15 08:24:07 +02:00
// If the LHS is the result of AND'ing a value with a single bit bitmask,
// we don't have information about probabilities.
if (Instruction *LHS = dyn_cast<Instruction>(CI->getOperand(0)))
if (LHS->getOpcode() == Instruction::And)
if (ConstantInt *AndRHS = dyn_cast<ConstantInt>(LHS->getOperand(1)))
[ConstantInt] Use ConstantInt::getValue instead of Constant::getUniqueInteger in a few places where we obviously have a ConstantInt. NFC getUniqueInteger will ultimately call ConstantInt::getValue, but calling ConstantInt::getValue should be inlined. llvm-svn: 310069
2017-08-04 18:59:29 +02:00
if (AndRHS->getValue().isPowerOf2())
Re-apply r234898 and fix tests. This commit makes LLVM not estimate branch probabilities when doing a single bit bitmask tests. The code that originally made me discover this is: if ((a & 0x1) == 0x1) { .. } In this case we don't actually have any branch probability information and should not assume to have any. LLVM transforms this into: %and = and i32 %a, 1 %tobool = icmp eq i32 %and, 0 So, in this case, the result of a bitwise and is compared against 0, but nevertheless, we should not assume to have probability information. CodeGen/ARM/2013-10-11-select-stalls.ll started failing because the changed probabilities changed the results of ARMBaseInstrInfo::isProfitableToIfCvt() and led to an Ifcvt of the diamond in the test. AFAICT, the test was never meant to test this and thus changing the test input slightly to not change the probabilities seems like the best way to preserve the meaning of the test. llvm-svn: 234979
2015-04-15 08:24:07 +02:00
return false;
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 11:44:40 +02:00
// Check if the LHS is the return value of a library function
LibFunc Func = NumLibFuncs;
if (TLI)
if (CallInst *Call = dyn_cast<CallInst>(CI->getOperand(0)))
if (Function *CalledFn = Call->getCalledFunction())
TLI->getLibFunc(*CalledFn, Func);
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 05:27:24 +02:00
bool isProb;
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 11:44:40 +02:00
if (Func == LibFunc_strcasecmp ||
Func == LibFunc_strcmp ||
Func == LibFunc_strncasecmp ||
Func == LibFunc_strncmp ||
[BPI] Teach BPI about bcmp function bcmp is similar to memcmp
2020-08-11 20:42:58 +02:00
Func == LibFunc_memcmp ||
Func == LibFunc_bcmp) {
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 11:44:40 +02:00
// strcmp and similar functions return zero, negative, or positive, if the
// first string is equal, less, or greater than the second. We consider it
// likely that the strings are not equal, so a comparison with zero is
// probably false, but also a comparison with any other number is also
// probably false given that what exactly is returned for nonzero values is
// not specified. Any kind of comparison other than equality we know
// nothing about.
switch (CI->getPredicate()) {
case CmpInst::ICMP_EQ:
isProb = false;
break;
case CmpInst::ICMP_NE:
isProb = true;
break;
default:
return false;
}
} else if (CV->isZero()) {
Use canonical forms for the branch probability zero heutistic. - Drop support for X >u 0, it's equivalent to X != 0 and should be canonicalized into the latter. - Add X < 1 -> unlikely, which is what instcombine canonicalizes X <= 0 into. - Add X > -1 -> likely, which is what instcombine canonicalizes X >= 0 into. llvm-svn: 139110
2011-09-05 01:53:04 +02:00
switch (CI->getPredicate()) {
case CmpInst::ICMP_EQ:
// X == 0 -> Unlikely
isProb = false;
break;
case CmpInst::ICMP_NE:
// X != 0 -> Likely
isProb = true;
break;
case CmpInst::ICMP_SLT:
// X < 0 -> Unlikely
isProb = false;
break;
case CmpInst::ICMP_SGT:
// X > 0 -> Likely
isProb = true;
break;
default:
return false;
}
} else if (CV->isOne() && CI->getPredicate() == CmpInst::ICMP_SLT) {
// InstCombine canonicalizes X <= 0 into X < 1.
// X <= 0 -> Unlikely
Do not handle case where LHS is equal to zero, because InstCombiner always moves it to RHS anyway. llvm-svn: 136586
2011-07-31 06:47:20 +02:00
isProb = false;
[Constants] If we already have a ConstantInt*, prefer to use isZero/isOne/isMinusOne instead of isNullValue/isOneValue/isAllOnesValue inherited from Constant. NFCI Going through the Constant methods requires redetermining that the Constant is a ConstantInt and then calling isZero/isOne/isMinusOne. llvm-svn: 307292
2017-07-06 20:39:47 +02:00
} else if (CV->isMinusOne()) {
Consider (x == -1) unlikely in BranchProbabilityInfo This adds another heuristic to BPI, similar to the existing heuristic that considers (x == 0) unlikely to be true. As suggested in the PACT'98 paper by Deitrich, Cheng, and Hwu, -1 is often used to indicate an invalid index, and equality comparisons with -1 are also unlikely to succeed. Local experimentation supports this hypothesis: This yields a 1-2% speedup in the test-suite sqlite benchmark on the PPC A2 core, with no significant regressions. llvm-svn: 193855
2013-11-01 11:58:22 +01:00
switch (CI->getPredicate()) {
case CmpInst::ICMP_EQ:
// X == -1 -> Unlikely
isProb = false;
break;
case CmpInst::ICMP_NE:
// X != -1 -> Likely
isProb = true;
break;
case CmpInst::ICMP_SGT:
// InstCombine canonicalizes X >= 0 into X > -1.
// X >= 0 -> Likely
isProb = true;
break;
default:
return false;
}
Use canonical forms for the branch probability zero heutistic. - Drop support for X >u 0, it's equivalent to X != 0 and should be canonicalized into the latter. - Add X < 1 -> unlikely, which is what instcombine canonicalizes X <= 0 into. - Add X > -1 -> likely, which is what instcombine canonicalizes X >= 0 into. llvm-svn: 139110
2011-09-05 01:53:04 +02:00
} else {
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 05:27:24 +02:00
return false;
Use canonical forms for the branch probability zero heutistic. - Drop support for X >u 0, it's equivalent to X != 0 and should be canonicalized into the latter. - Add X < 1 -> unlikely, which is what instcombine canonicalizes X <= 0 into. - Add X > -1 -> likely, which is what instcombine canonicalizes X >= 0 into. llvm-svn: 139110
2011-09-05 01:53:04 +02:00
}
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 05:27:24 +02:00
Revert "[BPI] Improve static heuristics for integer comparisons" This reverts commit 50c743fa713002fe4e0c76d23043e6c1f9e9fe6f. Patch will be split to smaller ones.
2020-08-17 20:42:57 +02:00
BranchProbability TakenProb(ZH_TAKEN_WEIGHT,
ZH_TAKEN_WEIGHT + ZH_NONTAKEN_WEIGHT);
BranchProbability UntakenProb(ZH_NONTAKEN_WEIGHT,
ZH_TAKEN_WEIGHT + ZH_NONTAKEN_WEIGHT);
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
if (!isProb)
std::swap(TakenProb, UntakenProb);
setEdgeProbability(
BB, SmallVector<BranchProbability, 2>({TakenProb, UntakenProb}));
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 05:27:24 +02:00
return true;
}
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-07 23:59:28 +02:00
bool BranchProbabilityInfo::calcFloatingPointHeuristics(const BasicBlock *BB) {
const BranchInst *BI = dyn_cast<BranchInst>(BB->getTerminator());
BranchProbabilityInfo: floating point equality is unlikely. This is from the same paper from Ball and Larus as the rest of the currently implemented heuristics. llvm-svn: 142677
2011-10-21 22:12:47 +02:00
if (!BI || !BI->isConditional())
return false;
Value *Cond = BI->getCondition();
FCmpInst *FCmp = dyn_cast<FCmpInst>(Cond);
Extend the floating point heuristic to consider NaN checks unlikely. llvm-svn: 142687
2011-10-21 23:13:47 +02:00
if (!FCmp)
BranchProbabilityInfo: floating point equality is unlikely. This is from the same paper from Ball and Larus as the rest of the currently implemented heuristics. llvm-svn: 142677
2011-10-21 22:12:47 +02:00
return false;
[BPI] Adjust the probability for floating point unordered comparison Since NaN is very rare in normal programs, so the probability for floating point unordered comparison should be extremely small. Current probability is 3/8, it is too large, this patch changes it to a tiny number. Differential Revision: https://reviews.llvm.org/D65303 llvm-svn: 371541
2019-09-10 19:25:11 +02:00
uint32_t TakenWeight = FPH_TAKEN_WEIGHT;
uint32_t NontakenWeight = FPH_NONTAKEN_WEIGHT;
Extend the floating point heuristic to consider NaN checks unlikely. llvm-svn: 142687
2011-10-21 23:13:47 +02:00
bool isProb;
if (FCmp->isEquality()) {
// f1 == f2 -> Unlikely
// f1 != f2 -> Likely
isProb = !FCmp->isTrueWhenEqual();
} else if (FCmp->getPredicate() == FCmpInst::FCMP_ORD) {
// !isnan -> Likely
isProb = true;
[BPI] Adjust the probability for floating point unordered comparison Since NaN is very rare in normal programs, so the probability for floating point unordered comparison should be extremely small. Current probability is 3/8, it is too large, this patch changes it to a tiny number. Differential Revision: https://reviews.llvm.org/D65303 llvm-svn: 371541
2019-09-10 19:25:11 +02:00
TakenWeight = FPH_ORD_WEIGHT;
NontakenWeight = FPH_UNO_WEIGHT;
Extend the floating point heuristic to consider NaN checks unlikely. llvm-svn: 142687
2011-10-21 23:13:47 +02:00
} else if (FCmp->getPredicate() == FCmpInst::FCMP_UNO) {
// isnan -> Unlikely
isProb = false;
[BPI] Adjust the probability for floating point unordered comparison Since NaN is very rare in normal programs, so the probability for floating point unordered comparison should be extremely small. Current probability is 3/8, it is too large, this patch changes it to a tiny number. Differential Revision: https://reviews.llvm.org/D65303 llvm-svn: 371541
2019-09-10 19:25:11 +02:00
TakenWeight = FPH_ORD_WEIGHT;
NontakenWeight = FPH_UNO_WEIGHT;
Extend the floating point heuristic to consider NaN checks unlikely. llvm-svn: 142687
2011-10-21 23:13:47 +02:00
} else {
return false;
}
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
BranchProbability TakenProb(TakenWeight, TakenWeight + NontakenWeight);
BranchProbability UntakenProb(NontakenWeight, TakenWeight + NontakenWeight);
Extend the floating point heuristic to consider NaN checks unlikely. llvm-svn: 142687
2011-10-21 23:13:47 +02:00
if (!isProb)
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
std::swap(TakenProb, UntakenProb);
BranchProbabilityInfo: floating point equality is unlikely. This is from the same paper from Ball and Larus as the rest of the currently implemented heuristics. llvm-svn: 142677
2011-10-21 22:12:47 +02:00
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
setEdgeProbability(
BB, SmallVector<BranchProbability, 2>({TakenProb, UntakenProb}));
BranchProbabilityInfo: floating point equality is unlikely. This is from the same paper from Ball and Larus as the rest of the currently implemented heuristics. llvm-svn: 142677
2011-10-21 22:12:47 +02:00
return true;
}
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 05:27:24 +02:00
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-07 23:59:28 +02:00
bool BranchProbabilityInfo::calcInvokeHeuristics(const BasicBlock *BB) {
const InvokeInst *II = dyn_cast<InvokeInst>(BB->getTerminator());
Set the branch probability of branching to the 'normal' destination of an invoke instruction to something absurdly high, while setting the probability of branching to the 'unwind' destination to the bare minimum. This should set cause the normal destination's invoke blocks to be moved closer to the invoke. PR13612 llvm-svn: 161944
2012-08-15 14:22:35 +02:00
if (!II)
return false;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
BranchProbability TakenProb(IH_TAKEN_WEIGHT,
IH_TAKEN_WEIGHT + IH_NONTAKEN_WEIGHT);
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
setEdgeProbability(
BB, SmallVector<BranchProbability, 2>({TakenProb, TakenProb.getCompl()}));
Set the branch probability of branching to the 'normal' destination of an invoke instruction to something absurdly high, while setting the probability of branching to the 'unwind' destination to the bare minimum. This should set cause the normal destination's invoke blocks to be moved closer to the invoke. PR13612 llvm-svn: 161944
2012-08-15 14:22:35 +02:00
return true;
}
Add BranchProbabilityInfo::releaseMemory to clear the Weights field. BranchProbabilityInfo was leaking 3MB of memory when running 'opt -O2 verify-uselistorder.lto.bc'. This was due to the Weights member not being cleared once the pass is no longer needed. This adds the releaseMemory override to clear that field. The other fields are cleared at the end of runOnFunction so can stay there. llvm-svn: 238462
2015-05-28 21:43:06 +02:00
void BranchProbabilityInfo::releaseMemory() {
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
Probs.clear();
[BranchProbabilityInfo] Fix eraseBlock This patch ensures that BranchProbabilityInfo::eraseBlock(BB) deletes all entries in Probs associated with with BB. Without this patch, stale entries for BB may remain in Probs after eraseBlock(BB), leading to a situation where a newly created basic block has an edge probability associated with it even before the pass responsible for creating the basic block adds any edge probability to it. Consider the current implementation of eraseBlock(BB): for (const_succ_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) { auto MapI = Probs.find(std::make_pair(BB, I.getSuccessorIndex())); if (MapI != Probs.end()) Probs.erase(MapI); } Notice that it uses succ_begin(BB) and succ_end(BB), which are based on BB->getTerminator(). This means that if the terminator changes between calls to setEdgeProbability and eraseBlock, then we may not examine all pairs associated with BB. This is exactly what happens in MaybeMergeBasicBlockIntoOnlyPred, which merges basic blocks A into B if A is the sole predecessor of B, and B is the sole successor of A. It replaces the terminator of A with UnreachableInst before (indirectly) calling eraseBlock(A). The patch fixes the problem by keeping track of all edge probablities entered with setEdgeProbability in a map from BasicBlock* to a successor index. Differential Revision: https://reviews.llvm.org/D90272
2020-10-28 00:14:25 +01:00
MaxSuccIdx.clear();
[BPI] Clear handles when releasing memory (NFC) This reduces max-rss of sqlite compilation by 2.5%.
2020-04-07 21:21:30 +02:00
Handles.clear();
Add BranchProbabilityInfo::releaseMemory to clear the Weights field. BranchProbabilityInfo was leaking 3MB of memory when running 'opt -O2 verify-uselistorder.lto.bc'. This was due to the Weights member not being cleared once the pass is no longer needed. This adds the releaseMemory override to clear that field. The other fields are cleared at the end of runOnFunction so can stay there. llvm-svn: 238462
2015-05-28 21:43:06 +02:00
}
[BrachProbablityInfo] Add invalidate method. Summary: Add invalidate method for BrachProbablityInfo. Reviewers: Eugene.Zelenko, chandlerc Subscribers: hiraditya, sanjoy.google, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D72815
2020-01-15 23:02:33 +01:00
bool BranchProbabilityInfo::invalidate(Function &, const PreservedAnalyses &PA,
FunctionAnalysisManager::Invalidator &) {
// Check whether the analysis, all analyses on functions, or the function's
// CFG have been preserved.
auto PAC = PA.getChecker<BranchProbabilityAnalysis>();
return !(PAC.preserved() || PAC.preservedSet<AllAnalysesOn<Function>>() ||
PAC.preservedSet<CFGAnalyses>());
}
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
void BranchProbabilityInfo::print(raw_ostream &OS) const {
Teach the BranchProbabilityInfo pass to print its results, and use that to bring it under direct test instead of merely indirectly testing it in the BlockFrequencyInfo pass. The next step is to start adding tests for the various heuristics employed, and to start fixing those heuristics once they're under test. llvm-svn: 142778
2011-10-23 23:21:50 +02:00
OS << "---- Branch Probabilities ----\n";
// We print the probabilities from the last function the analysis ran over,
// or the function it is currently running over.
assert(LastF && "Cannot print prior to running over a function");
Analysis: Remove implicit ilist iterator conversions Remove implicit ilist iterator conversions from LLVMAnalysis. I came across something really scary in `llvm::isKnownNotFullPoison()` which relied on `Instruction::getNextNode()` being completely broken (not surprising, but scary nevertheless). This function is documented (and coded to) return `nullptr` when it gets to the sentinel, but with an `ilist_half_node` as a sentinel, the sentinel check looks into some other memory and we don't recognize we've hit the end. Rooting out these scary cases is the reason I'm removing the implicit conversions before doing anything else with `ilist`; I'm not at all surprised that clients rely on badness. I found another scary case -- this time, not relying on badness, just bad (but I guess getting lucky so far) -- in `ObjectSizeOffsetEvaluator::compute_()`. Here, we save out the insertion point, do some things, and then restore it. Previously, we let the iterator auto-convert to `Instruction*`, and then set it back using the `Instruction*` version: Instruction *PrevInsertPoint = Builder.GetInsertPoint(); /* Logic that may change insert point */ if (PrevInsertPoint) Builder.SetInsertPoint(PrevInsertPoint); The check for `PrevInsertPoint` doesn't protect correctly against bad accesses. If the insertion point has been set to the end of a basic block (i.e., `SetInsertPoint(SomeBB)`), then `GetInsertPoint()` returns an iterator pointing at the list sentinel. The version of `SetInsertPoint()` that's getting called will then call `PrevInsertPoint->getParent()`, which explodes horribly. The only reason this hasn't blown up is that it's fairly unlikely the builder is adding to the end of the block; usually, we're adding instructions somewhere before the terminator. llvm-svn: 249925
2015-10-10 02:53:03 +02:00
for (const auto &BI : *LastF) {
[CFG/BasicBlock] Rename succ_const to const_succ. [NFC] Summary: Rename `succ_const_iterator` to `const_succ_iterator` and `succ_const_range` to `const_succ_range` for consistency with the predecessor iterators, and the corresponding iterators in MachineBasicBlock. Reviewers: nicholas, dblaikie, nlewycky Subscribers: hiraditya, bmahjour, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D75952
2020-03-10 19:33:02 +01:00
for (const_succ_iterator SI = succ_begin(&BI), SE = succ_end(&BI); SI != SE;
Analysis: Remove implicit ilist iterator conversions Remove implicit ilist iterator conversions from LLVMAnalysis. I came across something really scary in `llvm::isKnownNotFullPoison()` which relied on `Instruction::getNextNode()` being completely broken (not surprising, but scary nevertheless). This function is documented (and coded to) return `nullptr` when it gets to the sentinel, but with an `ilist_half_node` as a sentinel, the sentinel check looks into some other memory and we don't recognize we've hit the end. Rooting out these scary cases is the reason I'm removing the implicit conversions before doing anything else with `ilist`; I'm not at all surprised that clients rely on badness. I found another scary case -- this time, not relying on badness, just bad (but I guess getting lucky so far) -- in `ObjectSizeOffsetEvaluator::compute_()`. Here, we save out the insertion point, do some things, and then restore it. Previously, we let the iterator auto-convert to `Instruction*`, and then set it back using the `Instruction*` version: Instruction *PrevInsertPoint = Builder.GetInsertPoint(); /* Logic that may change insert point */ if (PrevInsertPoint) Builder.SetInsertPoint(PrevInsertPoint); The check for `PrevInsertPoint` doesn't protect correctly against bad accesses. If the insertion point has been set to the end of a basic block (i.e., `SetInsertPoint(SomeBB)`), then `GetInsertPoint()` returns an iterator pointing at the list sentinel. The version of `SetInsertPoint()` that's getting called will then call `PrevInsertPoint->getParent()`, which explodes horribly. The only reason this hasn't blown up is that it's fairly unlikely the builder is adding to the end of the block; usually, we're adding instructions somewhere before the terminator. llvm-svn: 249925
2015-10-10 02:53:03 +02:00
++SI) {
printEdgeProbability(OS << " ", &BI, *SI);
Revert "[C++11] Add predecessors(BasicBlock *) / successors(BasicBlock *) iterator ranges." This reverts commit r213474 (and r213475), which causes a miscompile on a stage2 LTO build. I'll reply on the list in a moment. llvm-svn: 213562
2014-07-21 19:06:51 +02:00
}
}
Teach the BranchProbabilityInfo pass to print its results, and use that to bring it under direct test instead of merely indirectly testing it in the BlockFrequencyInfo pass. The next step is to start adding tests for the various heuristics employed, and to start fixing those heuristics once they're under test. llvm-svn: 142778
2011-10-23 23:21:50 +02:00
}
Add more constantness in BranchProbabilityInfo. llvm-svn: 136502
2011-07-29 21:30:00 +02:00
bool BranchProbabilityInfo::
isEdgeHot(const BasicBlock *Src, const BasicBlock *Dst) const {
Branch profiling: floating-point avoidance. Patch by: Jakub Staszak! Introduces BranchProbability. Changes unsigned to uint32_t all over and uint64_t only when overflow is expected. llvm-svn: 132867
2011-06-11 03:05:22 +02:00
// Hot probability is at least 4/5 = 80%
Add compare operators to BranchProbability and use it to determine if an edge is hot. llvm-svn: 142751
2011-10-23 13:19:14 +02:00
// FIXME: Compare against a static "hot" BranchProbability.
return getEdgeProbability(Src, Dst) > BranchProbability(4, 5);
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
}
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-07 23:59:28 +02:00
const BasicBlock *
BranchProbabilityInfo::getHotSucc(const BasicBlock *BB) const {
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
auto MaxProb = BranchProbability::getZero();
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-07 23:59:28 +02:00
const BasicBlock *MaxSucc = nullptr;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
[CFG/BasicBlock] Rename succ_const to const_succ. [NFC] Summary: Rename `succ_const_iterator` to `const_succ_iterator` and `succ_const_range` to `const_succ_range` for consistency with the predecessor iterators, and the corresponding iterators in MachineBasicBlock. Reviewers: nicholas, dblaikie, nlewycky Subscribers: hiraditya, bmahjour, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D75952
2020-03-10 19:33:02 +01:00
for (const_succ_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) {
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-07 23:59:28 +02:00
const BasicBlock *Succ = *I;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
auto Prob = getEdgeProbability(BB, Succ);
if (Prob > MaxProb) {
MaxProb = Prob;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
MaxSucc = Succ;
}
}
Add compare operators to BranchProbability and use it to determine if an edge is hot. llvm-svn: 142751
2011-10-23 13:19:14 +02:00
// Hot probability is at least 4/5 = 80%
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
if (MaxProb > BranchProbability(4, 5))
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
return MaxSucc;
[C++11] More 'nullptr' conversion. In some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 206243
2014-04-15 06:59:12 +02:00
return nullptr;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
/// Get the raw edge probability for the edge. If can't find it, return a
/// default probability 1/N where N is the number of successors. Here an edge is
/// specified using PredBlock and an
/// index to the successors.
BranchProbability
BranchProbabilityInfo::getEdgeProbability(const BasicBlock *Src,
unsigned IndexInSuccessors) const {
auto I = Probs.find(std::make_pair(Src, IndexInSuccessors));
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
if (I != Probs.end())
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
return I->second;
[STLExtras] Add distance() for ranges, pred_size(), and succ_size() This commit adds a wrapper for std::distance() which works with ranges. As it would be a common case to write `distance(predecessors(BB))`, this also introduces `pred_size()` and `succ_size()` helpers to make that easier to write. Differential Revision: https://reviews.llvm.org/D46668 llvm-svn: 332057
2018-05-11 01:01:54 +02:00
return {1, static_cast<uint32_t>(succ_size(Src))};
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
BranchProbability
BranchProbabilityInfo::getEdgeProbability(const BasicBlock *Src,
[CFG/BasicBlock] Rename succ_const to const_succ. [NFC] Summary: Rename `succ_const_iterator` to `const_succ_iterator` and `succ_const_range` to `const_succ_range` for consistency with the predecessor iterators, and the corresponding iterators in MachineBasicBlock. Reviewers: nicholas, dblaikie, nlewycky Subscribers: hiraditya, bmahjour, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D75952
2020-03-10 19:33:02 +01:00
const_succ_iterator Dst) const {
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
return getEdgeProbability(Src, Dst.getSuccessorIndex());
[block-freq] Teach branch probability how to return the edge weight in between a BasicBlock and one of its successors. IMHO At some point BasicBlock should be refactored along the lines of MachineBasicBlock so that successors/weights are actually embedded within the block. Now is not that time though. llvm-svn: 197303
2013-12-14 03:24:25 +01:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
/// Get the raw edge probability calculated for the block pair. This returns the
/// sum of all raw edge probabilities from Src to Dst.
BranchProbability
BranchProbabilityInfo::getEdgeProbability(const BasicBlock *Src,
const BasicBlock *Dst) const {
auto Prob = BranchProbability::getZero();
bool FoundProb = false;
[BPI] Incorrect probability reported in case of mulptiple edges. Summary: By design 'BranchProbabilityInfo:: getEdgeProbability(const BasicBlock *Src, const BasicBlock *Dst) const' should return sum of probabilities over all edges from Src to Dst. Current implementation is buggy and returns 1/num_of_successors if probabilities are not explicitly set. Note current implementation of BPI printing has an issue as well and annotates each edge with sum of probabilities over all ages from one basic block to another. That's why 30% probability reported (instead of 10%) in the lit test. This is not urgent issue since only printing is affected. Note also current implementation assumes that either all or none edges have probabilities set. This is not the only place which uses such assumption. At least we should assert that in verifier. In addition we can think on a more robust API of BPI which would prevent situations. Reviewers: skatkov, yrouban, taewookoh Reviewed By: skatkov Subscribers: hiraditya, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D79071
2020-04-29 09:08:01 +02:00
uint32_t EdgeCount = 0;
[CFG/BasicBlock] Rename succ_const to const_succ. [NFC] Summary: Rename `succ_const_iterator` to `const_succ_iterator` and `succ_const_range` to `const_succ_range` for consistency with the predecessor iterators, and the corresponding iterators in MachineBasicBlock. Reviewers: nicholas, dblaikie, nlewycky Subscribers: hiraditya, bmahjour, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D75952
2020-03-10 19:33:02 +01:00
for (const_succ_iterator I = succ_begin(Src), E = succ_end(Src); I != E; ++I)
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-24 20:14:27 +02:00
if (*I == Dst) {
[BPI] Incorrect probability reported in case of mulptiple edges. Summary: By design 'BranchProbabilityInfo:: getEdgeProbability(const BasicBlock *Src, const BasicBlock *Dst) const' should return sum of probabilities over all edges from Src to Dst. Current implementation is buggy and returns 1/num_of_successors if probabilities are not explicitly set. Note current implementation of BPI printing has an issue as well and annotates each edge with sum of probabilities over all ages from one basic block to another. That's why 30% probability reported (instead of 10%) in the lit test. This is not urgent issue since only printing is affected. Note also current implementation assumes that either all or none edges have probabilities set. This is not the only place which uses such assumption. At least we should assert that in verifier. In addition we can think on a more robust API of BPI which would prevent situations. Reviewers: skatkov, yrouban, taewookoh Reviewed By: skatkov Subscribers: hiraditya, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D79071
2020-04-29 09:08:01 +02:00
++EdgeCount;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
auto MapI = Probs.find(std::make_pair(Src, I.getSuccessorIndex()));
if (MapI != Probs.end()) {
FoundProb = true;
Prob += MapI->second;
Allow 0-weight branches in BranchProbabilityInfo. Summary: When computing branch weights in BPI, we used to disallow branches with weight 0. This is a minor nuisance, because a branch with weight 0 is different to "don't have information". In the context of instrumentation, it may mean "never executed", in the context of sampling, it means "never or seldom executed". In allowing 0 weight branches, I ran into issues with the switch expansion code in selection DAG. It is currently hardwired to not handle branches with weight 0. To maintain the current behaviour, I changed it to use 1 when it finds 0, but perhaps the algorithm needs changes to tolerate branches with weight zero. Reviewers: hansw Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9533 llvm-svn: 236617
2015-05-06 19:55:11 +02:00
}
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-24 20:14:27 +02:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
uint32_t succ_num = std::distance(succ_begin(Src), succ_end(Src));
[BPI] Incorrect probability reported in case of mulptiple edges. Summary: By design 'BranchProbabilityInfo:: getEdgeProbability(const BasicBlock *Src, const BasicBlock *Dst) const' should return sum of probabilities over all edges from Src to Dst. Current implementation is buggy and returns 1/num_of_successors if probabilities are not explicitly set. Note current implementation of BPI printing has an issue as well and annotates each edge with sum of probabilities over all ages from one basic block to another. That's why 30% probability reported (instead of 10%) in the lit test. This is not urgent issue since only printing is affected. Note also current implementation assumes that either all or none edges have probabilities set. This is not the only place which uses such assumption. At least we should assert that in verifier. In addition we can think on a more robust API of BPI which would prevent situations. Reviewers: skatkov, yrouban, taewookoh Reviewed By: skatkov Subscribers: hiraditya, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D79071
2020-04-29 09:08:01 +02:00
return FoundProb ? Prob : BranchProbability(EdgeCount, succ_num);
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-24 20:14:27 +02:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-22 19:56:14 +01:00
/// Set the edge probability for a given edge specified by PredBlock and an
/// index to the successors.
void BranchProbabilityInfo::setEdgeProbability(const BasicBlock *Src,
unsigned IndexInSuccessors,
BranchProbability Prob) {
Probs[std::make_pair(Src, IndexInSuccessors)] = Prob;
Re-submit r272891 "Prevent dangling pointer problems in BranchProbabilityInfo" Most possibly problem was caused by the same reason as PR28400. This change bypasses it by using CallbackVH instead of AssertingVH. Differential Revision: https://reviews.llvm.org/D20957 llvm-svn: 275563
2016-07-15 16:31:16 +02:00
Handles.insert(BasicBlockCallbackVH(Src, this));
Rename DEBUG macro to LLVM_DEBUG. The DEBUG() macro is very generic so it might clash with other projects. The renaming was done as follows: - git grep -l 'DEBUG' | xargs sed -i 's/\bDEBUG\s\?(/LLVM_DEBUG(/g' - git diff -U0 master | ../clang/tools/clang-format/clang-format-diff.py -i -p1 -style LLVM - Manual change to APInt - Manually chage DOCS as regex doesn't match it. In the transition period the DEBUG() macro is still present and aliased to the LLVM_DEBUG() one. Differential Revision: https://reviews.llvm.org/D43624 llvm-svn: 332240
2018-05-14 14:53:11 +02:00
LLVM_DEBUG(dbgs() << "set edge " << Src->getName() << " -> "
<< IndexInSuccessors << " successor probability to " << Prob
<< "\n");
[BranchProbabilityInfo] Fix eraseBlock This patch ensures that BranchProbabilityInfo::eraseBlock(BB) deletes all entries in Probs associated with with BB. Without this patch, stale entries for BB may remain in Probs after eraseBlock(BB), leading to a situation where a newly created basic block has an edge probability associated with it even before the pass responsible for creating the basic block adds any edge probability to it. Consider the current implementation of eraseBlock(BB): for (const_succ_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) { auto MapI = Probs.find(std::make_pair(BB, I.getSuccessorIndex())); if (MapI != Probs.end()) Probs.erase(MapI); } Notice that it uses succ_begin(BB) and succ_end(BB), which are based on BB->getTerminator(). This means that if the terminator changes between calls to setEdgeProbability and eraseBlock, then we may not examine all pairs associated with BB. This is exactly what happens in MaybeMergeBasicBlockIntoOnlyPred, which merges basic blocks A into B if A is the sole predecessor of B, and B is the sole successor of A. It replaces the terminator of A with UnreachableInst before (indirectly) calling eraseBlock(A). The patch fixes the problem by keeping track of all edge probablities entered with setEdgeProbability in a map from BasicBlock* to a successor index. Differential Revision: https://reviews.llvm.org/D90272
2020-10-28 00:14:25 +01:00
[BranchProbabilityInfo] Make MaxSuccIdx[Src] efficient and add a comment about the subtle eraseBlock. NFC Follow-up to D90272.
2020-10-28 00:29:10 +01:00
unsigned &SuccIdx = MaxSuccIdx[Src];
SuccIdx = std::max(SuccIdx, IndexInSuccessors);
Replace all weight-based interfaces in MBB with probability-based interfaces, and update all uses of old interfaces. (This is the second attempt to submit this patch. The first caused two assertion failures and was reverted. See https://llvm.org/bugs/show_bug.cgi?id=25687) The patch in http://reviews.llvm.org/D13745 is broken into four parts: 1. New interfaces without functional changes (http://reviews.llvm.org/D13908). 2. Use new interfaces in SelectionDAG, while in other passes treat probabilities as weights (http://reviews.llvm.org/D14361). 3. Use new interfaces in all other passes. 4. Remove old interfaces. This patch is 3+4 above. In this patch, MBB won't provide weight-based interfaces any more, which are totally replaced by probability-based ones. The interface addSuccessor() is redesigned so that the default probability is unknown. We allow unknown probabilities but don't allow using it together with known probabilities in successor list. That is to say, we either have a list of successors with all known probabilities, or all unknown probabilities. In the latter case, we assume each successor has 1/N probability where N is the number of successors. An assertion checks if the user is attempting to add a successor with the disallowed mixed use as stated above. This can help us catch many misuses. All uses of weight-based interfaces are now updated to use probability-based ones. Differential revision: http://reviews.llvm.org/D14973 llvm-svn: 254377
2015-12-01 06:29:22 +01:00
}
[BrachProbablityInfo] Set edge probabilities at once and fix calcMetadataWeights() Hide the method that allows setting probability for particular edge and introduce a public method that sets probabilities for all outgoing edges at once. Setting individual edge probability is error prone. More over it is difficult to check that the total probability is 1.0 because there is no easy way to know when the user finished setting all the probabilities. Related bug is fixed in BranchProbabilityInfo::calcMetadataWeights(). Changing unreachable branch probabilities to raw(1) and distributing the rest (oldProbability - raw(1)) over the reachable branches could introduce total probability inaccuracy bigger than 1/numOfBranches. Reviewers: yamauchi, ebrevnov Tags: #llvm Differential Revision: https://reviews.llvm.org/D79396
2020-05-21 06:49:11 +02:00
/// Set the edge probability for all edges at once.
void BranchProbabilityInfo::setEdgeProbability(
const BasicBlock *Src, const SmallVectorImpl<BranchProbability> &Probs) {
assert(Src->getTerminator()->getNumSuccessors() == Probs.size());
if (Probs.size() == 0)
return; // Nothing to set.
uint64_t TotalNumerator = 0;
for (unsigned SuccIdx = 0; SuccIdx < Probs.size(); ++SuccIdx) {
setEdgeProbability(Src, SuccIdx, Probs[SuccIdx]);
TotalNumerator += Probs[SuccIdx].getNumerator();
}
// Because of rounding errors the total probability cannot be checked to be
// 1.0 exactly. That is TotalNumerator == BranchProbability::getDenominator.
// Instead, every single probability in Probs must be as accurate as possible.
// This results in error 1/denominator at most, thus the total absolute error
// should be within Probs.size / BranchProbability::getDenominator.
assert(TotalNumerator <= BranchProbability::getDenominator() + Probs.size());
assert(TotalNumerator >= BranchProbability::getDenominator() - Probs.size());
}
Branch profiling: floating-point avoidance. Patch by: Jakub Staszak! Introduces BranchProbability. Changes unsigned to uint32_t all over and uint64_t only when overflow is expected. llvm-svn: 132867
2011-06-11 03:05:22 +02:00
raw_ostream &
Teach the BranchProbabilityInfo pass to print its results, and use that to bring it under direct test instead of merely indirectly testing it in the BlockFrequencyInfo pass. The next step is to start adding tests for the various heuristics employed, and to start fixing those heuristics once they're under test. llvm-svn: 142778
2011-10-23 23:21:50 +02:00
BranchProbabilityInfo::printEdgeProbability(raw_ostream &OS,
const BasicBlock *Src,
const BasicBlock *Dst) const {
Introduce MachineBranchProbabilityInfo class, which has similar API to BranchProbabilityInfo (expect setEdgeWeight which is not available here). Branch Weights are kept in MachineBasicBlocks. To turn off this analysis set -use-mbpi=false. llvm-svn: 133184
2011-06-16 22:22:37 +02:00
const BranchProbability Prob = getEdgeProbability(Src, Dst);
Remove all remaining uses of Value::getNameStr(). llvm-svn: 144648
2011-11-15 17:27:03 +01:00
OS << "edge " << Src->getName() << " -> " << Dst->getName()
Branch profiling: floating-point avoidance. Patch by: Jakub Staszak! Introduces BranchProbability. Changes unsigned to uint32_t all over and uint64_t only when overflow is expected. llvm-svn: 132867
2011-06-11 03:05:22 +02:00
<< " probability is " << Prob
<< (isEdgeHot(Src, Dst) ? " [HOT edge]\n" : "\n");
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 03:16:30 +02:00
return OS;
}
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
Re-submit r272891 "Prevent dangling pointer problems in BranchProbabilityInfo" Most possibly problem was caused by the same reason as PR28400. This change bypasses it by using CallbackVH instead of AssertingVH. Differential Revision: https://reviews.llvm.org/D20957 llvm-svn: 275563
2016-07-15 16:31:16 +02:00
void BranchProbabilityInfo::eraseBlock(const BasicBlock *BB) {
[BranchProbabilityInfo] Make MaxSuccIdx[Src] efficient and add a comment about the subtle eraseBlock. NFC Follow-up to D90272.
2020-10-28 00:29:10 +01:00
// Note that we cannot use successors of BB because the terminator of BB may
// have changed when eraseBlock is called as a BasicBlockCallbackVH callback.
[BranchProbabilityInfo] Fix eraseBlock This patch ensures that BranchProbabilityInfo::eraseBlock(BB) deletes all entries in Probs associated with with BB. Without this patch, stale entries for BB may remain in Probs after eraseBlock(BB), leading to a situation where a newly created basic block has an edge probability associated with it even before the pass responsible for creating the basic block adds any edge probability to it. Consider the current implementation of eraseBlock(BB): for (const_succ_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) { auto MapI = Probs.find(std::make_pair(BB, I.getSuccessorIndex())); if (MapI != Probs.end()) Probs.erase(MapI); } Notice that it uses succ_begin(BB) and succ_end(BB), which are based on BB->getTerminator(). This means that if the terminator changes between calls to setEdgeProbability and eraseBlock, then we may not examine all pairs associated with BB. This is exactly what happens in MaybeMergeBasicBlockIntoOnlyPred, which merges basic blocks A into B if A is the sole predecessor of B, and B is the sole successor of A. It replaces the terminator of A with UnreachableInst before (indirectly) calling eraseBlock(A). The patch fixes the problem by keeping track of all edge probablities entered with setEdgeProbability in a map from BasicBlock* to a successor index. Differential Revision: https://reviews.llvm.org/D90272
2020-10-28 00:14:25 +01:00
auto It = MaxSuccIdx.find(BB);
if (It == MaxSuccIdx.end())
return;
for (unsigned I = 0, E = It->second; I <= E; ++I) {
auto MapI = Probs.find(std::make_pair(BB, I));
[BPI] Compile time improvement when erasing blocks (NFC) Summary: eraseBlock is trying to erase all probability info for the given BB. This info is stored in a DenseMap organized like so: using Edge = std::pair<const BasicBlock *, unsigned>; DenseMap<Edge, BranchProbability> Probs; where the unsigned in the Edge key is the successor id. It was walking through every single map entry, checking if the BB in the key's pair matched the given BB. Much more efficient is to do what another method (getEdgeProbability) was already doing, which is to walk the successors of the BB, and simply do a map lookup on the key formed from each <BB, successor id> pair. Doing this dropped the overall compile time for a file containing a very large function by around 32%. Reviewers: davidxl, xur Subscribers: llvm-commits, hiraditya Tags: #llvm Differential Revision: https://reviews.llvm.org/D83596
2020-07-11 00:25:27 +02:00
if (MapI != Probs.end())
Probs.erase(MapI);
Re-submit r272891 "Prevent dangling pointer problems in BranchProbabilityInfo" Most possibly problem was caused by the same reason as PR28400. This change bypasses it by using CallbackVH instead of AssertingVH. Differential Revision: https://reviews.llvm.org/D20957 llvm-svn: 275563
2016-07-15 16:31:16 +02:00
}
[BranchProbabilityInfo] Fix eraseBlock This patch ensures that BranchProbabilityInfo::eraseBlock(BB) deletes all entries in Probs associated with with BB. Without this patch, stale entries for BB may remain in Probs after eraseBlock(BB), leading to a situation where a newly created basic block has an edge probability associated with it even before the pass responsible for creating the basic block adds any edge probability to it. Consider the current implementation of eraseBlock(BB): for (const_succ_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) { auto MapI = Probs.find(std::make_pair(BB, I.getSuccessorIndex())); if (MapI != Probs.end()) Probs.erase(MapI); } Notice that it uses succ_begin(BB) and succ_end(BB), which are based on BB->getTerminator(). This means that if the terminator changes between calls to setEdgeProbability and eraseBlock, then we may not examine all pairs associated with BB. This is exactly what happens in MaybeMergeBasicBlockIntoOnlyPred, which merges basic blocks A into B if A is the sole predecessor of B, and B is the sole successor of A. It replaces the terminator of A with UnreachableInst before (indirectly) calling eraseBlock(A). The patch fixes the problem by keeping track of all edge probablities entered with setEdgeProbability in a map from BasicBlock* to a successor index. Differential Revision: https://reviews.llvm.org/D90272
2020-10-28 00:14:25 +01:00
MaxSuccIdx.erase(BB);
Re-submit r272891 "Prevent dangling pointer problems in BranchProbabilityInfo" Most possibly problem was caused by the same reason as PR28400. This change bypasses it by using CallbackVH instead of AssertingVH. Differential Revision: https://reviews.llvm.org/D20957 llvm-svn: 275563
2016-07-15 16:31:16 +02:00
}
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 11:44:40 +02:00
void BranchProbabilityInfo::calculate(const Function &F, const LoopInfo &LI,
[BPI][NFC] Reuse post dominantor tree from analysis manager when available Summary: Currenlty BPI unconditionally creates post dominator tree each time. While this is not incorrect we can save compile time by reusing existing post dominator tree (when it's valid) provided by analysis manager. Reviewers: skatkov, taewookoh, yrouban Reviewed By: skatkov Subscribers: hiraditya, steven_wu, dexonsmith, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D78987
2020-04-28 11:31:20 +02:00
const TargetLibraryInfo *TLI,
PostDominatorTree *PDT) {
Rename DEBUG macro to LLVM_DEBUG. The DEBUG() macro is very generic so it might clash with other projects. The renaming was done as follows: - git grep -l 'DEBUG' | xargs sed -i 's/\bDEBUG\s\?(/LLVM_DEBUG(/g' - git diff -U0 master | ../clang/tools/clang-format/clang-format-diff.py -i -p1 -style LLVM - Manual change to APInt - Manually chage DOCS as regex doesn't match it. In the transition period the DEBUG() macro is still present and aliased to the LLVM_DEBUG() one. Differential Revision: https://reviews.llvm.org/D43624 llvm-svn: 332240
2018-05-14 14:53:11 +02:00
LLVM_DEBUG(dbgs() << "---- Branch Probability Info : " << F.getName()
<< " ----\n\n");
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
LastF = &F; // Store the last function we ran on for printing.
assert(PostDominatedByUnreachable.empty());
assert(PostDominatedByColdCall.empty());
[BPI][NFC] Consolidate code to deal with SCCs under a dedicated data structure. In order to facilitate review of D79485 here is a small NFC change which restructures code around handling of SCCs in BPI. Reviewed By: davidxl Differential Revision: https://reviews.llvm.org/D84514
2020-07-24 13:57:10 +02:00
SccI = std::make_unique<SccInfo>(F);
[BranchProbabilityInfo] Handle irreducible loops. Summary: Compute the strongly connected components of the CFG and fall back to use these for blocks that are in loops that are not detected by LoopInfo when computing loop back-edge and exit branch probabilities. Reviewers: dexonsmith, davidxl Subscribers: mcrosier, llvm-commits Differential Revision: https://reviews.llvm.org/D39385 llvm-svn: 317094
2017-11-01 16:16:50 +01:00
[BPI][NFC] Reuse post dominantor tree from analysis manager when available Summary: Currenlty BPI unconditionally creates post dominator tree each time. While this is not incorrect we can save compile time by reusing existing post dominator tree (when it's valid) provided by analysis manager. Reviewers: skatkov, taewookoh, yrouban Reviewed By: skatkov Subscribers: hiraditya, steven_wu, dexonsmith, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D78987
2020-04-28 11:31:20 +02:00
std::unique_ptr<PostDominatorTree> PDTPtr;
if (!PDT) {
PDTPtr = std::make_unique<PostDominatorTree>(const_cast<Function &>(F));
PDT = PDTPtr.get();
}
computePostDominatedByUnreachable(F, PDT);
computePostDominatedByColdCall(F, PDT);
Reland "b19ec1eb3d0c [BPI] Improve unreachable/ColdCall heurstics to handle loops." Summary: b19ec1eb3d0c has been reverted because of the test failures with PowerPC targets. This patch addresses the issues from the previous commit. Test Plan: ninja check-all. Confirmed that CodeGen/PowerPC/pr36292.ll and CodeGen/PowerPC/sms-cpy-1.ll pass Subscribers: llvm-commits
2019-12-02 19:15:22 +01:00
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
// Walk the basic blocks in post-order so that we can build up state about
// the successors of a block iteratively.
for (auto BB : post_order(&F.getEntryBlock())) {
Rename DEBUG macro to LLVM_DEBUG. The DEBUG() macro is very generic so it might clash with other projects. The renaming was done as follows: - git grep -l 'DEBUG' | xargs sed -i 's/\bDEBUG\s\?(/LLVM_DEBUG(/g' - git diff -U0 master | ../clang/tools/clang-format/clang-format-diff.py -i -p1 -style LLVM - Manual change to APInt - Manually chage DOCS as regex doesn't match it. In the transition period the DEBUG() macro is still present and aliased to the LLVM_DEBUG() one. Differential Revision: https://reviews.llvm.org/D43624 llvm-svn: 332240
2018-05-14 14:53:11 +02:00
LLVM_DEBUG(dbgs() << "Computing probabilities for " << BB->getName()
<< "\n");
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 08:39:47 +02:00
// If there is no at least two successors, no sense to set probability.
if (BB->getTerminator()->getNumSuccessors() < 2)
continue;
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
if (calcMetadataWeights(BB))
continue;
[BPI] Apply invoke heuristic before loop branch heuristic Currently the loop branch heuristic is applied before the invoke heuristic which makes us overestimate the probability of the unwind destination of invokes inside loops. This in turn makes us grossly underestimate the frequencies of loops with invokes. Reviewed By: skatkov, vsk Differential Revision: https://reviews.llvm.org/D47371 llvm-svn: 334285
2018-06-08 15:03:21 +02:00
if (calcInvokeHeuristics(BB))
continue;
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 06:33:04 +02:00
if (calcUnreachableHeuristics(BB))
continue;
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
if (calcColdCallHeuristics(BB))
continue;
[BPI][NFC] Consolidate code to deal with SCCs under a dedicated data structure. In order to facilitate review of D79485 here is a small NFC change which restructures code around handling of SCCs in BPI. Reviewed By: davidxl Differential Revision: https://reviews.llvm.org/D84514
2020-07-24 13:57:10 +02:00
if (calcLoopBranchHeuristics(BB, LI))
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
continue;
if (calcPointerHeuristics(BB))
continue;
Revert "[BPI] Improve static heuristics for integer comparisons" This reverts commit 50c743fa713002fe4e0c76d23043e6c1f9e9fe6f. Patch will be split to smaller ones.
2020-08-17 20:42:57 +02:00
if (calcZeroHeuristics(BB, TLI))
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
continue;
if (calcFloatingPointHeuristics(BB))
continue;
}
PostDominatedByUnreachable.clear();
PostDominatedByColdCall.clear();
[BPI] Fix memory leak reported by sanitizer bots There is a silly mistake where release() is used instead of reset() for free resources of unique pointer. Reviewed By: ebrevnov Differential Revision: https://reviews.llvm.org/D84747
2020-07-28 14:50:40 +02:00
SccI.reset();
Add options to dump block frequency/branch probability info in text. Summary: Add options -print-bfi/-print-bpi that dump block frequency and branch probability info like -view-block-freq-propagation-dags and -view-machine-block-freq-propagation-dags do but in text. This is useful when the graph is very large and complex (the dot command crashes, lines/edges too close to tell apart, hard to navigate without textual search) or simply when text is preferred. Reviewers: davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D37165 llvm-svn: 311822
2017-08-26 02:31:00 +02:00
if (PrintBranchProb &&
(PrintBranchProbFuncName.empty() ||
F.getName().equals(PrintBranchProbFuncName))) {
print(dbgs());
}
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
}
void BranchProbabilityInfoWrapperPass::getAnalysisUsage(
AnalysisUsage &AU) const {
Require DominatorTree when requiring/preserving LoopInfo in the old pass manager Summary: Require DominatorTree when requiring/preserving LoopInfo in the old pass manager BreakCriticalEdges tries to keep LoopInfo and DominatorTree updated if they exist. However, since commit r321653 and r321805, to update LoopInfo we must have a DominatorTree, or we will hit an assert. To fix this we now make a couple of passes that only required/preserved LoopInfo also require DominatorTree. This solves PR37334. Reviewers: eli.friedman, efriedma Reviewed By: efriedma Subscribers: efriedma, llvm-commits Differential Revision: https://reviews.llvm.org/D46829 llvm-svn: 332583
2018-05-17 11:05:40 +02:00
// We require DT so it's available when LI is available. The LI updating code
// asserts that DT is also present so if we don't make sure that we have DT
// here, that assert will trigger.
AU.addRequired<DominatorTreeWrapperPass>();
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
AU.addRequired<LoopInfoWrapperPass>();
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 11:44:40 +02:00
AU.addRequired<TargetLibraryInfoWrapperPass>();
[BPI][NFC] Reuse post dominantor tree from analysis manager when available Summary: Currenlty BPI unconditionally creates post dominator tree each time. While this is not incorrect we can save compile time by reusing existing post dominator tree (when it's valid) provided by analysis manager. Reviewers: skatkov, taewookoh, yrouban Reviewed By: skatkov Subscribers: hiraditya, steven_wu, dexonsmith, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D78987
2020-04-28 11:31:20 +02:00
AU.addRequired<PostDominatorTreeWrapperPass>();
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
AU.setPreservesAll();
}
bool BranchProbabilityInfoWrapperPass::runOnFunction(Function &F) {
const LoopInfo &LI = getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
Change TargetLibraryInfo analysis passes to always require Function Summary: This is the first change to enable the TLI to be built per-function so that -fno-builtin* handling can be migrated to use function attributes. See discussion on D61634 for background. This is an enabler for fixing handling of these options for LTO, for example. This change should not affect behavior, as the provided function is not yet used to build a specifically per-function TLI, but rather enables that migration. Most of the changes were very mechanical, e.g. passing a Function to the legacy analysis pass's getTLI interface, or in Module level cases, adding a callback. This is similar to the way the per-function TTI analysis works. There was one place where we were looking for builtins but not in the context of a specific function. See FindCXAAtExit in lib/Transforms/IPO/GlobalOpt.cpp. I'm somewhat concerned my workaround could provide the wrong behavior in some corner cases. Suggestions welcome. Reviewers: chandlerc, hfinkel Subscribers: arsenm, dschuff, jvesely, nhaehnle, mehdi_amini, javed.absar, sbc100, jgravelle-google, eraman, aheejin, steven_wu, george.burgess.iv, dexonsmith, jfb, asbirlea, gchatelet, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D66428 llvm-svn: 371284
2019-09-07 05:09:36 +02:00
const TargetLibraryInfo &TLI =
getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F);
[BPI][NFC] Reuse post dominantor tree from analysis manager when available Summary: Currenlty BPI unconditionally creates post dominator tree each time. While this is not incorrect we can save compile time by reusing existing post dominator tree (when it's valid) provided by analysis manager. Reviewers: skatkov, taewookoh, yrouban Reviewed By: skatkov Subscribers: hiraditya, steven_wu, dexonsmith, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D78987
2020-04-28 11:31:20 +02:00
PostDominatorTree &PDT =
getAnalysis<PostDominatorTreeWrapperPass>().getPostDomTree();
BPI.calculate(F, LI, &TLI, &PDT);
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 00:48:29 +02:00
return false;
}
void BranchProbabilityInfoWrapperPass::releaseMemory() { BPI.releaseMemory(); }
void BranchProbabilityInfoWrapperPass::print(raw_ostream &OS,
const Module *) const {
BPI.print(OS);
}
[PM] Port Branch Probability Analysis pass to the new pass manager. Differential Revision: http://reviews.llvm.org/D19839 llvm-svn: 268601
2016-05-05 04:59:57 +02:00
[PM] Change the static object whose address is used to uniquely identify analyses to have a common type which is enforced rather than using a char object and a `void *` type when used as an identifier. This has a number of advantages. First, it at least helps some of the confusion raised in Justin Lebar's code review of why `void *` was being used everywhere by having a stronger type that connects to documentation about this. However, perhaps more importantly, it addresses a serious issue where the alignment of these pointer-like identifiers was unknown. This made it hard to use them in pointer-like data structures. We were already dodging this in dangerous ways to create the "all analyses" entry. In a subsequent patch I attempted to use these with TinyPtrVector and things fell apart in a very bad way. And it isn't just a compile time or type system issue. Worse than that, the actual alignment of these pointer-like opaque identifiers wasn't guaranteed to be a useful alignment as they were just characters. This change introduces a type to use as the "key" object whose address forms the opaque identifier. This both forces the objects to have proper alignment, and provides type checking that we get it right everywhere. It also makes the types somewhat less mysterious than `void *`. We could go one step further and introduce a truly opaque pointer-like type to return from the `ID()` static function rather than returning `AnalysisKey *`, but that didn't seem to be a clear win so this is just the initial change to get to a reliably typed and aligned object serving is a key for all the analyses. Thanks to Richard Smith and Justin Lebar for helping pick plausible names and avoid making this refactoring many times. =] And thanks to Sean for the super fast review! While here, I've tried to move away from the "PassID" nomenclature entirely as it wasn't really helping and is overloaded with old pass manager constructs. Now we have IDs for analyses, and key objects whose address can be used as IDs. Where possible and clear I've shortened this to just "ID". In a few places I kept "AnalysisID" to make it clear what was being identified. Differential Revision: https://reviews.llvm.org/D27031 llvm-svn: 287783
2016-11-23 18:53:26 +01:00
AnalysisKey BranchProbabilityAnalysis::Key;
[PM] Port Branch Probability Analysis pass to the new pass manager. Differential Revision: http://reviews.llvm.org/D19839 llvm-svn: 268601
2016-05-05 04:59:57 +02:00
BranchProbabilityInfo
Consistently use FunctionAnalysisManager Besides a general consistently benefit, the extra layer of indirection allows the mechanical part of https://reviews.llvm.org/D23256 that requires touching every transformation and analysis to be factored out cleanly. Thanks to David for the suggestion. llvm-svn: 278077
2016-08-09 02:28:15 +02:00
BranchProbabilityAnalysis::run(Function &F, FunctionAnalysisManager &AM) {
[PM] Port Branch Probability Analysis pass to the new pass manager. Differential Revision: http://reviews.llvm.org/D19839 llvm-svn: 268601
2016-05-05 04:59:57 +02:00
BranchProbabilityInfo BPI;
[BPI][NFC] Reuse post dominantor tree from analysis manager when available Summary: Currenlty BPI unconditionally creates post dominator tree each time. While this is not incorrect we can save compile time by reusing existing post dominator tree (when it's valid) provided by analysis manager. Reviewers: skatkov, taewookoh, yrouban Reviewed By: skatkov Subscribers: hiraditya, steven_wu, dexonsmith, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D78987
2020-04-28 11:31:20 +02:00
BPI.calculate(F, AM.getResult<LoopAnalysis>(F),
&AM.getResult<TargetLibraryAnalysis>(F),
&AM.getResult<PostDominatorTreeAnalysis>(F));
[PM] Port Branch Probability Analysis pass to the new pass manager. Differential Revision: http://reviews.llvm.org/D19839 llvm-svn: 268601
2016-05-05 04:59:57 +02:00
return BPI;
}
PreservedAnalyses
Consistently use FunctionAnalysisManager Besides a general consistently benefit, the extra layer of indirection allows the mechanical part of https://reviews.llvm.org/D23256 that requires touching every transformation and analysis to be factored out cleanly. Thanks to David for the suggestion. llvm-svn: 278077
2016-08-09 02:28:15 +02:00
BranchProbabilityPrinterPass::run(Function &F, FunctionAnalysisManager &AM) {
[PM] Port Branch Probability Analysis pass to the new pass manager. Differential Revision: http://reviews.llvm.org/D19839 llvm-svn: 268601
2016-05-05 04:59:57 +02:00
OS << "Printing analysis results of BPI for function "
<< "'" << F.getName() << "':"
<< "\n";
AM.getResult<BranchProbabilityAnalysis>(F).print(OS);
return PreservedAnalyses::all();
}
Reference in New Issue Copy Permalink