RPCS3/llvm-mirror
1
0
Fork 0
mirror of https://github.com/RPCS3/llvm-mirror.git synced 2024-11-21 18:22:53 +01:00
Code Issues Projects Releases Wiki Activity

[CSSPGO] Pseudo probe encoding and emission.

Browse Source 
This change implements pseudo probe encoding and emission for CSSPGO. Please see RFC here for more context: https://groups.google.com/g/llvm-dev/c/1p1rdYbL93s

Pseudo probes are in the form of intrinsic calls on IR/MIR but they do not turn into any machine instructions. Instead they are emitted into the binary as a piece of data in standalone sections.  The probe-specific sections are not needed to be loaded into memory at execution time, thus they do not incur a runtime overhead. 

**ELF object emission**

The binary data to emit are organized as two ELF sections, i.e, the `.pseudo_probe_desc` section and the `.pseudo_probe` section. The `.pseudo_probe_desc` section stores a function descriptor for each function and the `.pseudo_probe` section stores the actual probes, each fo which corresponds to an IR basic block or an IR function callsite. A function descriptor is stored as a module-level metadata during the compilation and is serialized into the object file during object emission.

Both the probe descriptors and pseudo probes can be emitted into a separate ELF section per function to leverage the linker for deduplication.  A `.pseudo_probe` section shares the same COMDAT group with the function code so that when the function is dead, the probes are dead and disposed too. On the contrary, a `.pseudo_probe_desc` section has its own COMDAT group. This is because even if a function is dead, its probes may be inlined into other functions and its descriptor is still needed by the profile generation tool.

The format of `.pseudo_probe_desc` section looks like:

```
.section   .pseudo_probe_desc,"",@progbits
.quad   6309742469962978389  // Func GUID
.quad   4294967295           // Func Hash
.byte   9                    // Length of func name
.ascii  "_Z5funcAi"          // Func name
.quad   7102633082150537521
.quad   138828622701
.byte   12
.ascii  "_Z8funcLeafi"
.quad   446061515086924981
.quad   4294967295
.byte   9
.ascii  "_Z5funcBi"
.quad   -2016976694713209516
.quad   72617220756
.byte   7
.ascii  "_Z3fibi"
```

For each `.pseudoprobe` section, the encoded binary data consists of a single function record corresponding to an outlined function (i.e, a function with a code entry in the `.text` section). A function record has the following format :

```
FUNCTION BODY (one for each outlined function present in the text section)
    GUID (uint64)
        GUID of the function
    NPROBES (ULEB128)
        Number of probes originating from this function.
    NUM_INLINED_FUNCTIONS (ULEB128)
        Number of callees inlined into this function, aka number of
        first-level inlinees
    PROBE RECORDS
        A list of NPROBES entries. Each entry contains:
          INDEX (ULEB128)
          TYPE (uint4)
            0 - block probe, 1 - indirect call, 2 - direct call
          ATTRIBUTE (uint3)
            reserved
          ADDRESS_TYPE (uint1)
            0 - code address, 1 - address delta
          CODE_ADDRESS (uint64 or ULEB128)
            code address or address delta, depending on ADDRESS_TYPE
    INLINED FUNCTION RECORDS
        A list of NUM_INLINED_FUNCTIONS entries describing each of the inlined
        callees.  Each record contains:
          INLINE SITE
            GUID of the inlinee (uint64)
            ID of the callsite probe (ULEB128)
          FUNCTION BODY
            A FUNCTION BODY entry describing the inlined function.
```

To support building a context-sensitive profile, probes from inlinees are grouped by their inline contexts. An inline context is logically a call path through which a callee function lands in a caller function. The probe emitter builds an inline tree based on the debug metadata for each outlined function in the form of a trie tree. A tree root is the outlined function. Each tree edge stands for a callsite where inlining happens. Pseudo probes originating from an inlinee function are stored in a tree node and the tree path starting from the root all the way down to the tree node is the inline context of the probes. The emission happens on the whole tree top-down recursively. Probes of a tree node will be emitted altogether with their direct parent edge. Since a pseudo probe corresponds to a real code address, for size savings, the address is encoded as a delta from the previous probe except for the first probe. Variant-sized integer encoding, aka LEB128, is used for address delta and probe index.

**Assembling**

Pseudo probes can be printed as assembly directives alternatively. This allows for good assembly code readability and also provides a view of how optimizations and pseudo probes affect each other, especially helpful for diff time assembly analysis.

A pseudo probe directive has the following operands in order: function GUID, probe index, probe type, probe attributes and inline context. The directive is generated by the compiler and can be parsed by the assembler to form an encoded `.pseudoprobe` section in the object file.

A example assembly looks like:

```
foo2: # @foo2
# %bb.0: # %bb0
pushq %rax
testl %edi, %edi
.pseudoprobe 837061429793323041 1 0 0
je .LBB1_1
# %bb.2: # %bb2
.pseudoprobe 837061429793323041 6 2 0
callq foo
.pseudoprobe 837061429793323041 3 0 0
.pseudoprobe 837061429793323041 4 0 0
popq %rax
retq
.LBB1_1: # %bb1
.pseudoprobe 837061429793323041 5 1 0
callq *%rsi
.pseudoprobe 837061429793323041 2 0 0
.pseudoprobe 837061429793323041 4 0 0
popq %rax
retq
# -- End function
.section .pseudo_probe_desc,"",@progbits
.quad 6699318081062747564
.quad 72617220756
.byte 3
.ascii "foo"
.quad 837061429793323041
.quad 281547593931412
.byte 4
.ascii "foo2"
```

With inlining turned on, the assembly may look different around %bb2 with an inlined probe:

```
# %bb.2:                                # %bb2
.pseudoprobe    837061429793323041 3 0
.pseudoprobe    6699318081062747564 1 0 @ 837061429793323041:6
.pseudoprobe    837061429793323041 4 0
popq    %rax
retq
```

**Disassembling**

We have a disassembling tool (llvm-profgen) that can display disassembly alongside with pseudo probes. So far it only supports ELF executable file.

An example disassembly looks like:

```
00000000002011a0 <foo2>:
  2011a0: 50                    push   rax
  2011a1: 85 ff                 test   edi,edi
  [Probe]:  FUNC: foo2  Index: 1  Type: Block
  2011a3: 74 02                 je     2011a7 <foo2+0x7>
  [Probe]:  FUNC: foo2  Index: 3  Type: Block
  [Probe]:  FUNC: foo2  Index: 4  Type: Block
  [Probe]:  FUNC: foo   Index: 1  Type: Block  Inlined: @ foo2:6
  2011a5: 58                    pop    rax
  2011a6: c3                    ret
  [Probe]:  FUNC: foo2  Index: 2  Type: Block
  2011a7: bf 01 00 00 00        mov    edi,0x1
  [Probe]:  FUNC: foo2  Index: 5  Type: IndirectCall
  2011ac: ff d6                 call   rsi
  [Probe]:  FUNC: foo2  Index: 4  Type: Block
  2011ae: 58                    pop    rax
  2011af: c3                    ret
```

Reviewed By: wmi

Differential Revision: https://reviews.llvm.org/D91878
This commit is contained in:
Hongtao Yu 2020-12-08 15:37:32 -08:00
parent d4257740dd
commit 85e4f6f241
29 changed files with 1048 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
include/llvm/CodeGen/AsmPrinter.h
View File

@ -66,6 +66,7 @@ class MCSymbol;
class MCTargetOptions;
class MDNode;
class Module;
class PseudoProbeHandler;
class raw_ostream;
class StackMaps;
class StringRef;
@ -206,6 +207,10 @@ private:
/// If the target supports dwarf debug info, this pointer is non-null.
DwarfDebug *DD = nullptr;
/// A handler that supports pseudo probe emission with embedded inline
/// context.
PseudoProbeHandler *PP = nullptr;
/// If the current module uses dwarf CFI annotations strictly for debugging.
bool isCFIMoveForDebugging = false;
@ -360,6 +365,8 @@ public:
void emitBBAddrMapSection(const MachineFunction &MF);
void emitPseudoProbe(const MachineInstr &MI);
void emitRemarksSection(remarks::RemarkStreamer &RS);
enum CFIMoveType { CFI_M_None, CFI_M_EH, CFI_M_Debug };

3
include/llvm/IR/MDBuilder.h
View File

@ -76,6 +76,9 @@ public:
/// Return metadata containing the section prefix for a function.
MDNode *createFunctionSectionPrefix(StringRef Prefix);
/// Return metadata containing the pseudo probe descriptor for a function.
MDNode *createPseudoProbeDesc(uint64_t GUID, uint64_t Hash, Function *F);
//===------------------------------------------------------------------===//
// Range metadata.
//===------------------------------------------------------------------===//

2
include/llvm/IR/PseudoProbe.h
View File

@ -18,6 +18,8 @@
namespace llvm {
constexpr const char *PseudoProbeDescMetadataName = "llvm.pseudo_probe_desc";
enum class PseudoProbeType { Block = 0, IndirectCall, DirectCall };
struct PseudoProbeDwarfDiscriminator {

1
include/llvm/MC/MCAssembler.h
View File

@ -202,6 +202,7 @@ private:
bool relaxCVInlineLineTable(MCAsmLayout &Layout,
MCCVInlineLineTableFragment &DF);
bool relaxCVDefRange(MCAsmLayout &Layout, MCCVDefRangeFragment &DF);
bool relaxPseudoProbeAddr(MCAsmLayout &Layout, MCPseudoProbeAddrFragment &DF);
/// finishLayout - Finalize a layout, including fragment lowering.
void finishLayout(MCAsmLayout &Layout);

6
include/llvm/MC/MCContext.h
View File

@ -22,6 +22,7 @@
#include "llvm/BinaryFormat/XCOFF.h"
#include "llvm/MC/MCAsmMacro.h"
#include "llvm/MC/MCDwarf.h"
#include "llvm/MC/MCPseudoProbe.h"
#include "llvm/MC/MCSubtargetInfo.h"
#include "llvm/MC/MCTargetOptions.h"
#include "llvm/MC/SectionKind.h"
@ -199,6 +200,9 @@ namespace llvm {
/// The Compile Unit ID that we are currently processing.
unsigned DwarfCompileUnitID = 0;
/// A collection of MCPseudoProbe in the current module
MCPseudoProbeTable PseudoProbeTable;
// Sections are differentiated by the quadruple (section_name, group_name,
// unique_id, link_to_symbol_name). Sections sharing the same quadruple are
// combined into one section.
@ -749,6 +753,8 @@ namespace llvm {
}
void undefineMacro(StringRef Name) { MacroMap.erase(Name); }
MCPseudoProbeTable &getMCPseudoProbeTable() { return PseudoProbeTable; }
};
} // end namespace llvm

19
include/llvm/MC/MCFragment.h
View File

@ -47,6 +47,7 @@ public:
FT_SymbolId,
FT_CVInlineLines,
FT_CVDefRange,
FT_PseudoProbe,
FT_Dummy
};
@ -147,6 +148,7 @@ public:
case MCFragment::FT_Data:
case MCFragment::FT_Dwarf:
case MCFragment::FT_DwarfFrame:
case MCFragment::FT_PseudoProbe:
return true;
}
}
@ -591,6 +593,23 @@ public:
return F->getKind() == MCFragment::FT_BoundaryAlign;
}
};
class MCPseudoProbeAddrFragment : public MCEncodedFragmentWithFixups<8, 1> {
/// The expression for the difference of the two symbols that
/// make up the address delta between two .pseudoprobe directives.
const MCExpr *AddrDelta;
public:
MCPseudoProbeAddrFragment(const MCExpr *AddrDelta, MCSection *Sec = nullptr)
: MCEncodedFragmentWithFixups<8, 1>(FT_PseudoProbe, false, Sec),
AddrDelta(AddrDelta) {}
const MCExpr &getAddrDelta() const { return *AddrDelta; }
static bool classof(const MCFragment *F) {
return F->getKind() == MCFragment::FT_PseudoProbe;
}
};
} // end namespace llvm
#endif // LLVM_MC_MCFRAGMENT_H

8
include/llvm/MC/MCObjectFileInfo.h
View File

@ -174,6 +174,10 @@ protected:
/// Section containing metadata on function stack sizes.
MCSection *StackSizesSection = nullptr;
/// Section for pseudo probe information used by AutoFDO
MCSection *PseudoProbeSection = nullptr;
MCSection *PseudoProbeDescSection = nullptr;
// ELF specific sections.
MCSection *DataRelROSection = nullptr;
MCSection *MergeableConst4Section = nullptr;
@ -340,6 +344,10 @@ public:
MCSection *getBBAddrMapSection(const MCSection &TextSec) const;
MCSection *getPseudoProbeSection(const MCSection *TextSec) const;
MCSection *getPseudoProbeDescSection(StringRef FuncName) const;
// ELF specific sections.
MCSection *getDataRelROSection() const { return DataRelROSection; }
const MCSection *getMergeableConst4Section() const {

178
include/llvm/MC/MCPseudoProbe.h Normal file
View File

@ -0,0 +1,178 @@
//===- MCPseudoProbe.h - Pseudo probe encoding support ---------*- C++ -*-===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// This file contains the declaration of the MCPseudoProbe to support the pseudo
// probe encoding for AutoFDO. Pseudo probes together with their inline context
// are encoded in a DFS recursive way in the .pseudoprobe sections. For each
// .pseudoprobe section, the encoded binary data consist of a single or mutiple
// function records each for one outlined function. A function record has the
// following format :
//
// FUNCTION BODY (one for each outlined function present in the text section)
// GUID (uint64)
// GUID of the function
// NPROBES (ULEB128)
// Number of probes originating from this function.
// NUM_INLINED_FUNCTIONS (ULEB128)
// Number of callees inlined into this function, aka number of
// first-level inlinees
// PROBE RECORDS
// A list of NPROBES entries. Each entry contains:
// INDEX (ULEB128)
// TYPE (uint4)
// 0 - block probe, 1 - indirect call, 2 - direct call
// ATTRIBUTE (uint3)
// reserved
// ADDRESS_TYPE (uint1)
// 0 - code address, 1 - address delta
// CODE_ADDRESS (uint64 or ULEB128)
// code address or address delta, depending on ADDRESS_TYPE
// INLINED FUNCTION RECORDS
// A list of NUM_INLINED_FUNCTIONS entries describing each of the inlined
// callees. Each record contains:
// INLINE SITE
// GUID of the inlinee (uint64)
// ID of the callsite probe (ULEB128)
// FUNCTION BODY
// A FUNCTION BODY entry describing the inlined function.
//===----------------------------------------------------------------------===//
#ifndef LLVM_MC_MCPSEUDOPROBE_H
#define LLVM_MC_MCPSEUDOPROBE_H
#include "llvm/ADT/MapVector.h"
#include "llvm/MC/MCSection.h"
#include <functional>
#include <map>
#include <vector>
namespace llvm {
class MCStreamer;
class MCSymbol;
class MCObjectStreamer;
enum class MCPseudoProbeFlag {
// If set, indicates that the probe is encoded as an address delta
// instead of a real code address.
AddressDelta = 0x1,
};
/// Instances of this class represent a pseudo probe instance for a pseudo probe
/// table entry, which is created during a machine instruction is assembled and
/// uses an address from a temporary label created at the current address in the
/// current section.
class MCPseudoProbe {
MCSymbol *Label;
uint64_t Guid;
uint64_t Index;
uint8_t Type;
uint8_t Attributes;
public:
MCPseudoProbe(MCSymbol *Label, uint64_t Guid, uint64_t Index, uint64_t Type,
uint64_t Attributes)
: Label(Label), Guid(Guid), Index(Index), Type(Type),
Attributes(Attributes) {
assert(Type <= 0xFF && "Probe type too big to encode, exceeding 2^8");
assert(Attributes <= 0xFF &&
"Probe attributes too big to encode, exceeding 2^16");
}
MCSymbol *getLabel() const { return Label; }
uint64_t getGuid() const { return Guid; }
uint64_t getIndex() const { return Index; }
uint8_t getType() const { return Type; }
uint8_t getAttributes() const { return Attributes; }
void emit(MCObjectStreamer *MCOS, const MCPseudoProbe *LastProbe) const;
};
// An inline frame has the form <Guid, ProbeID>
using InlineSite = std::tuple<uint64_t, uint32_t>;
using MCPseudoProbeInlineStack = SmallVector<InlineSite, 8>;
// A Tri-tree based data structure to group probes by inline stack.
// A tree is allocated for a standalone .text section. A fake
// instance is created as the root of a tree.
// A real instance of this class is created for each function, either an
// unlined function that has code in .text section or an inlined function.
class MCPseudoProbeInlineTree {
uint64_t Guid;
// Set of probes that come with the function.
std::vector<MCPseudoProbe> Probes;
// Use std::map for a deterministic output.
std::map<InlineSite, MCPseudoProbeInlineTree *> Inlinees;
// Root node has a GUID 0.
bool isRoot() { return Guid == 0; }
MCPseudoProbeInlineTree *getOrAddNode(InlineSite Site);
public:
MCPseudoProbeInlineTree() = default;
MCPseudoProbeInlineTree(uint64_t Guid) : Guid(Guid) {}
~MCPseudoProbeInlineTree();
void addPseudoProbe(const MCPseudoProbe &Probe,
const MCPseudoProbeInlineStack &InlineStack);
void emit(MCObjectStreamer *MCOS, const MCPseudoProbe *&LastProbe);
};
/// Instances of this class represent the pseudo probes inserted into a compile
/// unit.
class MCPseudoProbeSection {
public:
void addPseudoProbe(MCSection *Sec, const MCPseudoProbe &Probe,
const MCPseudoProbeInlineStack &InlineStack) {
MCProbeDivisions[Sec].addPseudoProbe(Probe, InlineStack);
}
// TODO: Sort by getOrdinal to ensure a determinstic section order
using MCProbeDivisionMap = std::map<MCSection *, MCPseudoProbeInlineTree>;
private:
// A collection of MCPseudoProbe for each text section. The MCPseudoProbes
// are grouped by GUID of the functions where they are from and will be
// encoded by groups. In the comdat scenario where a text section really only
// contains the code of a function solely, the probes associated with a comdat
// function are still grouped by GUIDs due to inlining that can bring probes
// from different functions into one function.
MCProbeDivisionMap MCProbeDivisions;
public:
const MCProbeDivisionMap &getMCProbes() const { return MCProbeDivisions; }
bool empty() const { return MCProbeDivisions.empty(); }
void emit(MCObjectStreamer *MCOS);
};
class MCPseudoProbeTable {
// A collection of MCPseudoProbe in the current module grouped by text
// sections. MCPseudoProbes will be encoded into a corresponding
// .pseudoprobe section. With functions emitted as separate comdats,
// a text section really only contains the code of a function solely, and the
// probes associated with the text section will be emitted into a standalone
// .pseudoprobe section that shares the same comdat group with the function.
MCPseudoProbeSection MCProbeSections;
public:
static void emit(MCObjectStreamer *MCOS);
MCPseudoProbeSection &getProbeSections() { return MCProbeSections; }
#ifndef NDEBUG
static int DdgPrintIndent;
#endif
};
} // end namespace llvm
#endif // LLVM_MC_MCPSEUDOPROBE_H

6
include/llvm/MC/MCStreamer.h
View File

@ -21,6 +21,7 @@
#include "llvm/ADT/StringRef.h"
#include "llvm/MC/MCDirectives.h"
#include "llvm/MC/MCLinkerOptimizationHint.h"
#include "llvm/MC/MCPseudoProbe.h"
#include "llvm/MC/MCSymbol.h"
#include "llvm/MC/MCWinEH.h"
#include "llvm/Support/Error.h"
@ -1049,6 +1050,11 @@ public:
/// Emit the given \p Instruction into the current section.
virtual void emitInstruction(const MCInst &Inst, const MCSubtargetInfo &STI);
/// Emit the a pseudo probe into the current section.
virtual void emitPseudoProbe(uint64_t Guid, uint64_t Index, uint64_t Type,
uint64_t Attr,
const MCPseudoProbeInlineStack &InlineStack);
/// Set the bundle alignment mode from now on in the section.
/// The argument is the power of 2 to which the alignment is set. The
/// value 0 means turn the bundle alignment off.

2
include/llvm/Passes/PassBuilder.h
View File

@ -64,7 +64,7 @@ struct PGOOptions {
assert(this->Action != NoAction || this->CSAction != NoCSAction ||
this->DebugInfoForProfiling || this->PseudoProbeForProfiling);
// Pseudo probe emission does work with -fdebug-info-for-profiling since
// Pseudo probe emission does not work with -fdebug-info-for-profiling since
// they both use the discriminator field of debug lines but for different
// purposes.
if (this->DebugInfoForProfiling && this->PseudoProbeForProfiling) {

11
include/llvm/Transforms/IPO/SampleProfileProbe.h
View File

@ -37,18 +37,27 @@ enum class PseudoProbeReservedId { Invalid = 0, Last = Invalid };
class SampleProfileProber {
public:
// Give an empty module id when the prober is not used for instrumentation.
SampleProfileProber(Function &F);
SampleProfileProber(Function &F, const std::string &CurModuleUniqueId);
void instrumentOneFunc(Function &F, TargetMachine *TM);
private:
Function *getFunction() const { return F; }
uint64_t getFunctionHash() const { return FunctionHash; }
uint32_t getBlockId(const BasicBlock *BB) const;
uint32_t getCallsiteId(const Instruction *Call) const;
void computeCFGHash();
void computeProbeIdForBlocks();
void computeProbeIdForCallsites();
Function *F;
/// The current module ID that is used to name a static object as a comdat
/// group.
std::string CurModuleUniqueId;
/// A CFG hash code used to identify a function code changes.
uint64_t FunctionHash;
/// Map basic blocks to the their pseudo probe ids.
BlockIdMap BlockProbeIds;

24
lib/CodeGen/AsmPrinter/AsmPrinter.cpp
View File

@ -14,6 +14,7 @@
#include "CodeViewDebug.h"
#include "DwarfDebug.h"
#include "DwarfException.h"
#include "PseudoProbePrinter.h"
#include "WasmException.h"
#include "WinCFGuard.h"
#include "WinException.h"
@ -77,6 +78,7 @@
#include "llvm/IR/Metadata.h"
#include "llvm/IR/Module.h"
#include "llvm/IR/Operator.h"
#include "llvm/IR/PseudoProbe.h"
#include "llvm/IR/Type.h"
#include "llvm/IR/Value.h"
#include "llvm/MC/MCAsmInfo.h"
@ -146,6 +148,10 @@ const char CFGuardName[] = "Control Flow Guard";
const char CFGuardDescription[] = "Control Flow Guard";
const char CodeViewLineTablesGroupName[] = "linetables";
const char CodeViewLineTablesGroupDescription[] = "CodeView Line Tables";
const char PPTimerName[] = "emit";
const char PPTimerDescription[] = "Pseudo Probe Emission";
const char PPGroupName[] = "pseudo probe";
const char PPGroupDescription[] = "Pseudo Probe Emission";
STATISTIC(EmittedInsts, "Number of machine instrs printed");
@ -333,6 +339,12 @@ bool AsmPrinter::doInitialization(Module &M) {
}
}
if (M.getNamedMetadata(PseudoProbeDescMetadataName)) {
PP = new PseudoProbeHandler(this, &M);
Handlers.emplace_back(std::unique_ptr<PseudoProbeHandler>(PP), PPTimerName,
PPTimerDescription, PPGroupName, PPGroupDescription);
}
switch (MAI->getExceptionHandlingType()) {
case ExceptionHandling::SjLj:
case ExceptionHandling::DwarfCFI:
@ -1086,6 +1098,15 @@ void AsmPrinter::emitBBAddrMapSection(const MachineFunction &MF) {
OutStreamer->PopSection();
}
void AsmPrinter::emitPseudoProbe(const MachineInstr &MI) {
auto GUID = MI.getOperand(0).getImm();
auto Index = MI.getOperand(1).getImm();
auto Type = MI.getOperand(2).getImm();
auto Attr = MI.getOperand(3).getImm();
DILocation *DebugLoc = MI.getDebugLoc();
PP->emitPseudoProbe(GUID, Index, Type, Attr, DebugLoc);
}
void AsmPrinter::emitStackSizeSection(const MachineFunction &MF) {
if (!MF.getTarget().Options.EmitStackSizeSection)
return;
@ -1219,6 +1240,9 @@ void AsmPrinter::emitFunctionBody() {
case TargetOpcode::KILL:
if (isVerbose()) emitKill(&MI, *this);
break;
case TargetOpcode::PSEUDO_PROBE:
emitPseudoProbe(MI);
break;
default:
emitInstruction(&MI);
if (CanDoExtraAnalysis) {

1
lib/CodeGen/AsmPrinter/CMakeLists.txt
View File

@ -21,6 +21,7 @@ add_llvm_component_library(LLVMAsmPrinter
EHStreamer.cpp
ErlangGCPrinter.cpp
OcamlGCPrinter.cpp
PseudoProbePrinter.cpp
WinCFGuard.cpp
WinException.cpp
CodeViewDebug.cpp

84
lib/CodeGen/AsmPrinter/PseudoProbePrinter.cpp Normal file
View File

@ -0,0 +1,84 @@
//===- llvm/CodeGen/PseudoProbePrinter.cpp - Pseudo Probe Emission -------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// This file contains support for writing pseudo probe info into asm files.
//
//===----------------------------------------------------------------------===//
#include "PseudoProbePrinter.h"
#include "llvm/CodeGen/AsmPrinter.h"
#include "llvm/IR/DebugInfoMetadata.h"
#include "llvm/IR/Module.h"
#include "llvm/IR/PseudoProbe.h"
#include "llvm/MC/MCPseudoProbe.h"
#include "llvm/MC/MCStreamer.h"
using namespace llvm;
#define DEBUG_TYPE "pseudoprobe"
PseudoProbeHandler::~PseudoProbeHandler() = default;
PseudoProbeHandler::PseudoProbeHandler(AsmPrinter *A, Module *M) : Asm(A) {
NamedMDNode *FuncInfo = M->getNamedMetadata(PseudoProbeDescMetadataName);
assert(FuncInfo && "Pseudo probe descriptors are missing");
for (const auto *Operand : FuncInfo->operands()) {
const auto *MD = cast<MDNode>(Operand);
auto GUID =
mdconst::dyn_extract<ConstantInt>(MD->getOperand(0))->getZExtValue();
auto Name = cast<MDString>(MD->getOperand(2))->getString();
// We may see pairs with same name but different GUIDs here in LTO mode, due
// to static same-named functions inlined from other modules into this
// module. Function profiles with the same name will be merged no matter
// whether they are collected on the same function. Therefore we just pick
// up the last <Name, GUID> pair here to represent the same-named function
// collection and all probes from the collection will be merged into a
// single profile eventually.
Names[Name] = GUID;
}
LLVM_DEBUG(dump());
}
void PseudoProbeHandler::emitPseudoProbe(uint64_t Guid, uint64_t Index,
uint64_t Type, uint64_t Attr,
const DILocation *DebugLoc) {
// Gather all the inlined-at nodes.
// When it's done ReversedInlineStack looks like ([66, B], [88, A])
// which means, Function A inlines function B at calliste with a probe id 88,
// and B inlines C at probe 66 where C is represented by Guid.
SmallVector<InlineSite, 8> ReversedInlineStack;
auto *InlinedAt = DebugLoc ? DebugLoc->getInlinedAt() : nullptr;
while (InlinedAt) {
const DISubprogram *SP = InlinedAt->getScope()->getSubprogram();
// Use linkage name for C++ if possible.
auto Name = SP->getLinkageName();
if (Name.empty())
Name = SP->getName();
assert(Names.count(Name) && "Pseudo probe descriptor missing for function");
uint64_t CallerGuid = Names[Name];
uint64_t CallerProbeId = PseudoProbeDwarfDiscriminator::extractProbeIndex(
InlinedAt->getDiscriminator());
ReversedInlineStack.emplace_back(CallerGuid, CallerProbeId);
InlinedAt = InlinedAt->getInlinedAt();
}
SmallVector<InlineSite, 8> InlineStack(ReversedInlineStack.rbegin(),
ReversedInlineStack.rend());
Asm->OutStreamer->emitPseudoProbe(Guid, Index, Type, Attr, InlineStack);
}
#ifndef NDEBUG
void PseudoProbeHandler::dump() const {
dbgs() << "\n=============================\n";
dbgs() << "\nFunction Name to GUID map:\n";
dbgs() << "\n=============================\n";
for (const auto &Item : Names)
dbgs() << "Func: " << Item.first << " GUID: " << Item.second << "\n";
}
#endif

53
lib/CodeGen/AsmPrinter/PseudoProbePrinter.h Normal file
View File

@ -0,0 +1,53 @@
//===- PseudoProbePrinter.h - Pseudo probe encoding support -----*- C++ -*-===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// This file contains support for writing pseudo probe info into asm files.
//
//===----------------------------------------------------------------------===//
#ifndef LLVM_LIB_CODEGEN_ASMPRINTER_PSEUDOPROBEPRINTER_H
#define LLVM_LIB_CODEGEN_ASMPRINTER_PSEUDOPROBEPRINTER_H
#include "llvm/ADT/DenseMap.h"
#include "llvm/CodeGen/AsmPrinterHandler.h"
namespace llvm {
class AsmPrinter;
class MCStreamer;
class Module;
class DILocation;
class PseudoProbeHandler : public AsmPrinterHandler {
// Target of pseudo probe emission.
AsmPrinter *Asm;
// Name to GUID map
DenseMap<StringRef, uint64_t> Names;
public:
PseudoProbeHandler(AsmPrinter *A, Module *M);
~PseudoProbeHandler() override;
void emitPseudoProbe(uint64_t Guid, uint64_t Index, uint64_t Type,
uint64_t Attr, const DILocation *DebugLoc);
// Unused.
void setSymbolSize(const MCSymbol *Sym, uint64_t Size) override {}
void endModule() override {}
void beginFunction(const MachineFunction *MF) override {}
void endFunction(const MachineFunction *MF) override {}
void beginInstruction(const MachineInstr *MI) override {}
void endInstruction() override {}
#ifndef NDEBUG
void dump() const;
#endif
};
} // namespace llvm
#endif // LLVM_LIB_CODEGEN_ASMPRINTER_PSEUDOPROBEPRINTER_H

24
lib/CodeGen/TargetLoweringObjectFileImpl.cpp
View File

@ -40,6 +40,7 @@
#include "llvm/IR/Mangler.h"
#include "llvm/IR/Metadata.h"
#include "llvm/IR/Module.h"
#include "llvm/IR/PseudoProbe.h"
#include "llvm/IR/Type.h"
#include "llvm/MC/MCAsmInfo.h"
#include "llvm/MC/MCContext.h"
@ -316,6 +317,29 @@ void TargetLoweringObjectFileELF::emitModuleMetadata(MCStreamer &Streamer,
}
}
if (NamedMDNode *FuncInfo = M.getNamedMetadata(PseudoProbeDescMetadataName)) {
// Emit a descriptor for every function including functions that have an
// available external linkage. We may not want this for imported functions
// that has code in another thinLTO module but we don't have a good way to
// tell them apart from inline functions defined in header files. Therefore
// we put each descriptor in a separate comdat section and rely on the
// linker to deduplicate.
for (const auto *Operand : FuncInfo->operands()) {
const auto *MD = cast<MDNode>(Operand);
auto *GUID = mdconst::dyn_extract<ConstantInt>(MD->getOperand(0));
auto *Hash = mdconst::dyn_extract<ConstantInt>(MD->getOperand(1));
auto *Name = cast<MDString>(MD->getOperand(2));
auto *S = C.getObjectFileInfo()->getPseudoProbeDescSection(
TM->getFunctionSections() ? Name->getString() : StringRef());
Streamer.SwitchSection(S);
Streamer.emitInt64(GUID->getZExtValue());
Streamer.emitInt64(Hash->getZExtValue());
Streamer.emitULEB128IntValue(Name->getString().size());
Streamer.emitBytes(Name->getString());
}
}
unsigned Version = 0;
unsigned Flags = 0;
StringRef Section;

10
lib/IR/MDBuilder.cpp
View File

@ -305,3 +305,13 @@ MDNode *MDBuilder::createIrrLoopHeaderWeight(uint64_t Weight) {
};
return MDNode::get(Context, Vals);
}
MDNode *MDBuilder::createPseudoProbeDesc(uint64_t GUID, uint64_t Hash,
Function *F) {
auto *Int64Ty = Type::getInt64Ty(Context);
SmallVector<Metadata *, 3> Ops(3);
Ops[0] = createConstant(ConstantInt::get(Int64Ty, GUID));
Ops[1] = createConstant(ConstantInt::get(Int64Ty, Hash));
Ops[2] = createString(F->getName());
return MDNode::get(Context, Ops);
}

1
lib/MC/CMakeLists.txt
View File

@ -32,6 +32,7 @@ add_llvm_component_library(LLVMMC
MCObjectFileInfo.cpp
MCObjectStreamer.cpp
MCObjectWriter.cpp
MCPseudoProbe.cpp
MCRegisterInfo.cpp
MCSchedule.cpp
MCSection.cpp

17
lib/MC/MCAsmStreamer.cpp
View File

@ -24,6 +24,7 @@
#include "llvm/MC/MCInstPrinter.h"
#include "llvm/MC/MCObjectFileInfo.h"
#include "llvm/MC/MCObjectWriter.h"
#include "llvm/MC/MCPseudoProbe.h"
#include "llvm/MC/MCRegister.h"
#include "llvm/MC/MCRegisterInfo.h"
#include "llvm/MC/MCSectionMachO.h"
@ -350,6 +351,10 @@ public:
void emitInstruction(const MCInst &Inst, const MCSubtargetInfo &STI) override;
void emitPseudoProbe(uint64_t Guid, uint64_t Index, uint64_t Type,
uint64_t Attr,
const MCPseudoProbeInlineStack &InlineStack) override;
void emitBundleAlignMode(unsigned AlignPow2) override;
void emitBundleLock(bool AlignToEnd) override;
void emitBundleUnlock() override;
@ -2122,6 +2127,18 @@ void MCAsmStreamer::emitInstruction(const MCInst &Inst,
EmitEOL();
}
void MCAsmStreamer::emitPseudoProbe(
uint64_t Guid, uint64_t Index, uint64_t Type, uint64_t Attr,
const MCPseudoProbeInlineStack &InlineStack) {
OS << "\t.pseudoprobe\t" << Guid << " " << Index << " " << Type << " "
<< Attr;
// Emit inline stack like
// @ GUIDmain:3 @ GUIDCaller:1 @ GUIDDirectCaller:11
for (const auto &Site : InlineStack)
OS << " @ " << std::get<0>(Site) << ":" << std::get<1>(Site);
EmitEOL();
}
void MCAsmStreamer::emitBundleAlignMode(unsigned AlignPow2) {
OS << "\t.bundle_align_mode " << AlignPow2;
EmitEOL();

36
lib/MC/MCAssembler.cpp
View File

@ -383,6 +383,8 @@ uint64_t MCAssembler::computeFragmentSize(const MCAsmLayout &Layout,
return cast<MCCVInlineLineTableFragment>(F).getContents().size();
case MCFragment::FT_CVDefRange:
return cast<MCCVDefRangeFragment>(F).getContents().size();
case MCFragment::FT_PseudoProbe:
return cast<MCPseudoProbeAddrFragment>(F).getContents().size();
case MCFragment::FT_Dummy:
llvm_unreachable("Should not have been added");
}
@ -704,6 +706,11 @@ static void writeFragment(raw_ostream &OS, const MCAssembler &Asm,
OS << DRF.getContents();
break;
}
case MCFragment::FT_PseudoProbe: {
const MCPseudoProbeAddrFragment &PF = cast<MCPseudoProbeAddrFragment>(F);
OS << PF.getContents();
break;
}
case MCFragment::FT_Dummy:
llvm_unreachable("Should not have been added");
}
@ -915,6 +922,12 @@ void MCAssembler::layout(MCAsmLayout &Layout) {
Contents = DF.getContents();
break;
}
case MCFragment::FT_PseudoProbe: {
MCPseudoProbeAddrFragment &PF = cast<MCPseudoProbeAddrFragment>(Frag);
Fixups = PF.getFixups();
Contents = PF.getContents();
break;
}
}
for (const MCFixup &Fixup : Fixups) {
uint64_t FixedValue;
@ -1170,6 +1183,27 @@ bool MCAssembler::relaxCVDefRange(MCAsmLayout &Layout,
return OldSize != F.getContents().size();
}
bool MCAssembler::relaxPseudoProbeAddr(MCAsmLayout &Layout,
MCPseudoProbeAddrFragment &PF) {
uint64_t OldSize = PF.getContents().size();
int64_t AddrDelta;
bool Abs = PF.getAddrDelta().evaluateKnownAbsolute(AddrDelta, Layout);
assert(Abs && "We created a pseudo probe with an invalid expression");
(void)Abs;
SmallVectorImpl<char> &Data = PF.getContents();
Data.clear();
raw_svector_ostream OSE(Data);
PF.getFixups().clear();
// Relocations should not be needed in general except on RISC-V which we are
// not targeted for now.
assert(!getBackend().requiresDiffExpressionRelocations() &&
"cannot relax relocations");
// AddrDelta is a signed integer
encodeSLEB128(AddrDelta, OSE, OldSize);
return OldSize != Data.size();
}
bool MCAssembler::relaxFragment(MCAsmLayout &Layout, MCFragment &F) {
switch(F.getKind()) {
default:
@ -1191,6 +1225,8 @@ bool MCAssembler::relaxFragment(MCAsmLayout &Layout, MCFragment &F) {
return relaxCVInlineLineTable(Layout, cast<MCCVInlineLineTableFragment>(F));
case MCFragment::FT_CVDefRange:
return relaxCVDefRange(Layout, cast<MCCVDefRangeFragment>(F));
case MCFragment::FT_PseudoProbe:
return relaxPseudoProbeAddr(Layout, cast<MCPseudoProbeAddrFragment>(F));
}
}

12
lib/MC/MCFragment.cpp
View File

@ -309,6 +309,9 @@ void MCFragment::destroy() {
case FT_CVDefRange:
delete cast<MCCVDefRangeFragment>(this);
return;
case FT_PseudoProbe:
delete cast<MCPseudoProbeAddrFragment>(this);
return;
case FT_Dummy:
delete cast<MCDummyFragment>(this);
return;
@ -351,6 +354,9 @@ LLVM_DUMP_METHOD void MCFragment::dump() const {
case MCFragment::FT_SymbolId: OS << "MCSymbolIdFragment"; break;
case MCFragment::FT_CVInlineLines: OS << "MCCVInlineLineTableFragment"; break;
case MCFragment::FT_CVDefRange: OS << "MCCVDefRangeTableFragment"; break;
case MCFragment::FT_PseudoProbe:
OS << "MCPseudoProbe";
break;
case MCFragment::FT_Dummy: OS << "MCDummyFragment"; break;
}
@ -484,6 +490,12 @@ LLVM_DUMP_METHOD void MCFragment::dump() const {
}
break;
}
case MCFragment::FT_PseudoProbe: {
const auto *OF = cast<MCPseudoProbeAddrFragment>(this);
OS << "\n ";
OS << " AddrDelta:" << OF->getAddrDelta();
break;
}
case MCFragment::FT_Dummy:
break;
}

42
lib/MC/MCObjectFileInfo.cpp
View File

@ -497,6 +497,10 @@ void MCObjectFileInfo::initELFMCObjectFileInfo(const Triple &T, bool Large) {
Ctx->getELFSection(".eh_frame", EHSectionType, EHSectionFlags);
StackSizesSection = Ctx->getELFSection(".stack_sizes", ELF::SHT_PROGBITS, 0);
PseudoProbeSection = Ctx->getELFSection(".pseudo_probe", DebugSecType, 0);
PseudoProbeDescSection =
Ctx->getELFSection(".pseudo_probe_desc", DebugSecType, 0);
}
void MCObjectFileInfo::initCOFFMCObjectFileInfo(const Triple &T) {
@ -1025,3 +1029,41 @@ MCObjectFileInfo::getBBAddrMapSection(const MCSection &TextSec) const {
Flags, 0, GroupName, ElfSec.getUniqueID(),
cast<MCSymbolELF>(TextSec.getBeginSymbol()));
}
MCSection *
MCObjectFileInfo::getPseudoProbeSection(const MCSection *TextSec) const {
if (Env == IsELF) {
const auto *ElfSec = static_cast<const MCSectionELF *>(TextSec);
// Create a separate section for probes that comes with a comdat function.
if (const MCSymbol *Group = ElfSec->getGroup()) {
auto *S = static_cast<MCSectionELF *>(PseudoProbeSection);
auto Flags = S->getFlags() | ELF::SHF_GROUP;
return Ctx->getELFSection(S->getName(), S->getType(), Flags,
S->getEntrySize(), Group->getName());
}
}
return PseudoProbeSection;
}
MCSection *
MCObjectFileInfo::getPseudoProbeDescSection(StringRef FuncName) const {
if (Env == IsELF) {
// Create a separate comdat group for each function's descriptor in order
// for the linker to deduplicate. The duplication, must be from different
// tranlation unit, can come from:
// 1. Inline functions defined in header files;
// 2. ThinLTO imported funcions;
// 3. Weak-linkage definitions.
// Use a concatenation of the section name and the function name as the
// group name so that descriptor-only groups won't be folded with groups of
// code.
if (TT.supportsCOMDAT() && !FuncName.empty()) {
auto *S = static_cast<MCSectionELF *>(PseudoProbeDescSection);
auto Flags = S->getFlags() | ELF::SHF_GROUP;
return Ctx->getELFSection(S->getName(), S->getType(), Flags,
S->getEntrySize(),
S->getName() + "_" + FuncName);
}
}
return PseudoProbeDescSection;
}

3
lib/MC/MCObjectStreamer.cpp
View File

@ -852,6 +852,9 @@ void MCObjectStreamer::finishImpl() {
// Dump out the dwarf file & directory tables and line tables.
MCDwarfLineTable::Emit(this, getAssembler().getDWARFLinetableParams());
// Emit pseudo probes for the current module.
MCPseudoProbeTable::emit(this);
// Update any remaining pending labels with empty data fragments.
flushPendingLabels();

70
lib/MC/MCParser/AsmParser.cpp
View File

@ -515,6 +515,7 @@ private:
DK_PRINT,
DK_ADDRSIG,
DK_ADDRSIG_SYM,
DK_PSEUDO_PROBE,
DK_END
};
@ -678,6 +679,9 @@ private:
// .print <double-quotes-string>
bool parseDirectivePrint(SMLoc DirectiveLoc);
// .pseudoprobe
bool parseDirectivePseudoProbe();
// Directives to support address-significance tables.
bool parseDirectiveAddrsig();
bool parseDirectiveAddrsigSym();
@ -2202,6 +2206,8 @@ bool AsmParser::parseStatement(ParseStatementInfo &Info,
return parseDirectiveAddrsig();
case DK_ADDRSIG_SYM:
return parseDirectiveAddrsigSym();
case DK_PSEUDO_PROBE:
return parseDirectivePseudoProbe();
}
return Error(IDLoc, "unknown directive");
@ -5520,6 +5526,7 @@ void AsmParser::initializeDirectiveKindMap() {
DirectiveKindMap[".print"] = DK_PRINT;
DirectiveKindMap[".addrsig"] = DK_ADDRSIG;
DirectiveKindMap[".addrsig_sym"] = DK_ADDRSIG_SYM;
DirectiveKindMap[".pseudoprobe"] = DK_PSEUDO_PROBE;
}
MCAsmMacro *AsmParser::parseMacroLikeBody(SMLoc DirectiveLoc) {
@ -5775,6 +5782,69 @@ bool AsmParser::parseDirectiveAddrsigSym() {
return false;
}
bool AsmParser::parseDirectivePseudoProbe() {
int64_t Guid;
int64_t Index;
int64_t Type;
int64_t Attr;
if (getLexer().is(AsmToken::Integer)) {
if (parseIntToken(Guid, "unexpected token in '.pseudoprobe' directive"))
return true;
}
if (getLexer().is(AsmToken::Integer)) {
if (parseIntToken(Index, "unexpected token in '.pseudoprobe' directive"))
return true;
}
if (getLexer().is(AsmToken::Integer)) {
if (parseIntToken(Type, "unexpected token in '.pseudoprobe' directive"))
return true;
}
if (getLexer().is(AsmToken::Integer)) {
if (parseIntToken(Attr, "unexpected token in '.pseudoprobe' directive"))
return true;
}
// Parse inline stack like @ GUID:11:12 @ GUID:1:11 @ GUID:3:21
MCPseudoProbeInlineStack InlineStack;
while (getLexer().is(AsmToken::At)) {
// eat @
Lex();
int64_t CallerGuid = 0;
if (getLexer().is(AsmToken::Integer)) {
if (parseIntToken(CallerGuid,
"unexpected token in '.pseudoprobe' directive"))
return true;
}
// eat colon
if (getLexer().is(AsmToken::Colon))
Lex();
int64_t CallerProbeId = 0;
if (getLexer().is(AsmToken::Integer)) {
if (parseIntToken(CallerProbeId,
"unexpected token in '.pseudoprobe' directive"))
return true;
}
InlineSite Site(CallerGuid, CallerProbeId);
InlineStack.push_back(Site);
}
if (parseToken(AsmToken::EndOfStatement,
"unexpected token in '.pseudoprobe' directive"))
return true;
getStreamer().emitPseudoProbe(Guid, Index, Type, Attr, InlineStack);
return false;
}
// We are comparing pointers, but the pointers are relative to a single string.
// Thus, this should always be deterministic.
static int rewritesSort(const AsmRewrite *AsmRewriteA,

213
lib/MC/MCPseudoProbe.cpp Normal file
View File

@ -0,0 +1,213 @@
//===- lib/MC/MCPseudoProbe.cpp - Pseudo probe encoding support ----------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
#include "llvm/MC/MCPseudoProbe.h"
#include "llvm/MC/MCAsmInfo.h"
#include "llvm/MC/MCContext.h"
#include "llvm/MC/MCObjectFileInfo.h"
#include "llvm/MC/MCObjectStreamer.h"
#include "llvm/MC/MCStreamer.h"
#define DEBUG_TYPE "mcpseudoprobe"
using namespace llvm;
#ifndef NDEBUG
int MCPseudoProbeTable::DdgPrintIndent = 0;
#endif
static const MCExpr *buildSymbolDiff(MCObjectStreamer *MCOS, const MCSymbol *A,
const MCSymbol *B) {
MCContext &Context = MCOS->getContext();
MCSymbolRefExpr::VariantKind Variant = MCSymbolRefExpr::VK_None;
const MCExpr *ARef = MCSymbolRefExpr::create(A, Variant, Context);
const MCExpr *BRef = MCSymbolRefExpr::create(B, Variant, Context);
const MCExpr *AddrDelta =
MCBinaryExpr::create(MCBinaryExpr::Sub, ARef, BRef, Context);
return AddrDelta;
}
void MCPseudoProbe::emit(MCObjectStreamer *MCOS,
const MCPseudoProbe *LastProbe) const {
// Emit Index
MCOS->emitULEB128IntValue(Index);
// Emit Type and the flag:
// Type (bit 0 to 3), with bit 4 to 6 for attributes.
// Flag (bit 7, 0 - code address, 1 - address delta). This indicates whether
// the following field is a symbolic code address or an address delta.
assert(Type <= 0xF && "Probe type too big to encode, exceeding 15");
assert(Attributes <= 0x7 &&
"Probe attributes too big to encode, exceeding 7");
uint8_t PackedType = Type | (Attributes << 4);
uint8_t Flag = LastProbe ? ((int8_t)MCPseudoProbeFlag::AddressDelta << 7) : 0;
MCOS->emitInt8(Flag | PackedType);
if (LastProbe) {
// Emit the delta between the address label and LastProbe.
const MCExpr *AddrDelta =
buildSymbolDiff(MCOS, Label, LastProbe->getLabel());
int64_t Delta;
if (AddrDelta->evaluateAsAbsolute(Delta, MCOS->getAssemblerPtr())) {
MCOS->emitSLEB128IntValue(Delta);
} else {
MCOS->insert(new MCPseudoProbeAddrFragment(AddrDelta));
}
} else {
// Emit label as a symbolic code address.
MCOS->emitSymbolValue(
Label, MCOS->getContext().getAsmInfo()->getCodePointerSize());
}
LLVM_DEBUG({
dbgs().indent(MCPseudoProbeTable::DdgPrintIndent);
dbgs() << "Probe: " << Index << "\n";
});
}
MCPseudoProbeInlineTree::~MCPseudoProbeInlineTree() {
for (auto &Inlinee : Inlinees)
delete Inlinee.second;
}
MCPseudoProbeInlineTree *
MCPseudoProbeInlineTree::getOrAddNode(InlineSite Site) {
auto Iter = Inlinees.find(Site);
if (Iter == Inlinees.end()) {
auto *Node = new MCPseudoProbeInlineTree(std::get<0>(Site));
Inlinees[Site] = Node;
return Node;
} else {
return Iter->second;
}
}
void MCPseudoProbeInlineTree::addPseudoProbe(
const MCPseudoProbe &Probe, const MCPseudoProbeInlineStack &InlineStack) {
// The function should not be called on the root.
assert(isRoot() && "Should not be called on root");
// When it comes here, the input look like:
// Probe: GUID of C, ...
// InlineStack: [88, A], [66, B]
// which means, Function A inlines function B at call site with a probe id of
// 88, and B inlines C at probe 66. The tri-tree expects a tree path like {[0,
// A], [88, B], [66, C]} to locate the tree node where the probe should be
// added. Note that the edge [0, A] means A is the top-level function we are
// emitting probes for.
// Make a [0, A] edge.
// An empty inline stack means the function that the probe originates from
// is a top-level function.
InlineSite Top;
if (InlineStack.empty()) {
Top = InlineSite(Probe.getGuid(), 0);
} else {
Top = InlineSite(std::get<0>(InlineStack.front()), 0);
}
auto *Cur = getOrAddNode(Top);
// Make interior edges by walking the inline stack. Once it's done, Cur should
// point to the node that the probe originates from.
if (!InlineStack.empty()) {
auto Iter = InlineStack.begin();
auto Index = std::get<1>(*Iter);
Iter++;
for (; Iter != InlineStack.end(); Iter++) {
// Make an edge by using the previous probe id and current GUID.
Cur = Cur->getOrAddNode(InlineSite(std::get<0>(*Iter), Index));
Index = std::get<1>(*Iter);
}
Cur = Cur->getOrAddNode(InlineSite(Probe.getGuid(), Index));
}
Cur->Probes.push_back(Probe);
}
void MCPseudoProbeInlineTree::emit(MCObjectStreamer *MCOS,
const MCPseudoProbe *&LastProbe) {
LLVM_DEBUG({
dbgs().indent(MCPseudoProbeTable::DdgPrintIndent);
dbgs() << "Group [\n";
MCPseudoProbeTable::DdgPrintIndent += 2;
});
// Emit probes grouped by GUID.
if (Guid != 0) {
LLVM_DEBUG({
dbgs().indent(MCPseudoProbeTable::DdgPrintIndent);
dbgs() << "GUID: " << Guid << "\n";
});
// Emit Guid
MCOS->emitInt64(Guid);
// Emit number of probes in this node
MCOS->emitULEB128IntValue(Probes.size());
// Emit number of direct inlinees
MCOS->emitULEB128IntValue(Inlinees.size());
// Emit probes in this group
for (const auto &Probe : Probes) {
Probe.emit(MCOS, LastProbe);
LastProbe = &Probe;
}
} else {
assert(Probes.empty() && "Root should not have probes");
}
// Emit descendent
for (const auto &Inlinee : Inlinees) {
if (Guid) {
// Emit probe index
MCOS->emitULEB128IntValue(std::get<1>(Inlinee.first));
LLVM_DEBUG({
dbgs().indent(MCPseudoProbeTable::DdgPrintIndent);
dbgs() << "InlineSite: " << std::get<1>(Inlinee.first) << "\n";
});
}
// Emit the group
Inlinee.second->emit(MCOS, LastProbe);
}
LLVM_DEBUG({
MCPseudoProbeTable::DdgPrintIndent -= 2;
dbgs().indent(MCPseudoProbeTable::DdgPrintIndent);
dbgs() << "]\n";
});
}
void MCPseudoProbeSection::emit(MCObjectStreamer *MCOS) {
MCContext &Ctx = MCOS->getContext();
for (auto &ProbeSec : MCProbeDivisions) {
const MCPseudoProbe *LastProbe = nullptr;
if (auto *S =
Ctx.getObjectFileInfo()->getPseudoProbeSection(ProbeSec.first)) {
// Switch to the .pseudoprobe section or a comdat group.
MCOS->SwitchSection(S);
// Emit probes grouped by GUID.
ProbeSec.second.emit(MCOS, LastProbe);
}
}
}
//
// This emits the pseudo probe tables.
//
void MCPseudoProbeTable::emit(MCObjectStreamer *MCOS) {
MCContext &Ctx = MCOS->getContext();
auto &ProbeTable = Ctx.getMCPseudoProbeTable();
// Bail out early so we don't switch to the pseudo_probe section needlessly
// and in doing so create an unnecessary (if empty) section.
auto &ProbeSections = ProbeTable.getProbeSections();
if (ProbeSections.empty())
return;
LLVM_DEBUG(MCPseudoProbeTable::DdgPrintIndent = 0);
// Put out the probe.
ProbeSections.emit(MCOS);
}

20
lib/MC/MCStreamer.cpp
View File

@ -22,6 +22,7 @@
#include "llvm/MC/MCInst.h"
#include "llvm/MC/MCInstPrinter.h"
#include "llvm/MC/MCObjectFileInfo.h"
#include "llvm/MC/MCPseudoProbe.h"
#include "llvm/MC/MCRegister.h"
#include "llvm/MC/MCRegisterInfo.h"
#include "llvm/MC/MCSection.h"
@ -1042,6 +1043,25 @@ void MCStreamer::emitInstruction(const MCInst &Inst, const MCSubtargetInfo &) {
visitUsedExpr(*Inst.getOperand(i).getExpr());
}
void MCStreamer::emitPseudoProbe(uint64_t Guid, uint64_t Index, uint64_t Type,
uint64_t Attr,
const MCPseudoProbeInlineStack &InlineStack) {
auto &Context = getContext();
// Create a symbol at in the current section for use in the probe.
MCSymbol *ProbeSym = Context.createTempSymbol();
// Set the value of the symbol to use for the MCPseudoProbe.
emitLabel(ProbeSym);
// Create a (local) probe entry with the symbol.
MCPseudoProbe Probe(ProbeSym, Guid, Index, Type, Attr);
// Add the probe entry to this section's entries.
Context.getMCPseudoProbeTable().getProbeSections().addPseudoProbe(
getCurrentSectionOnly(), Probe, InlineStack);
}
void MCStreamer::emitAbsoluteSymbolDiff(const MCSymbol *Hi, const MCSymbol *Lo,
unsigned Size) {
// Get the Hi-Lo expression.

74
lib/Transforms/IPO/SampleProfileProbe.cpp
View File

@ -35,12 +35,46 @@ using namespace llvm;
STATISTIC(ArtificialDbgLine,
"Number of probes that have an artificial debug line");
SampleProfileProber::SampleProfileProber(Function &Func) : F(&Func) {
SampleProfileProber::SampleProfileProber(Function &Func,
const std::string &CurModuleUniqueId)
: F(&Func), CurModuleUniqueId(CurModuleUniqueId) {
BlockProbeIds.clear();
CallProbeIds.clear();
LastProbeId = (uint32_t)PseudoProbeReservedId::Last;
computeProbeIdForBlocks();
computeProbeIdForCallsites();
computeCFGHash();
}
// Compute Hash value for the CFG: the lower 32 bits are CRC32 of the index
// value of each BB in the CFG. The higher 32 bits record the number of edges
// preceded by the number of indirect calls.
// This is derived from FuncPGOInstrumentation<Edge, BBInfo>::computeCFGHash().
void SampleProfileProber::computeCFGHash() {
std::vector<uint8_t> Indexes;
JamCRC JC;
for (auto &BB : *F) {
auto *TI = BB.getTerminator();
for (unsigned I = 0, E = TI->getNumSuccessors(); I != E; ++I) {
auto *Succ = TI->getSuccessor(I);
auto Index = getBlockId(Succ);
for (int J = 0; J < 4; J++)
Indexes.push_back((uint8_t)(Index >> (J * 8)));
}
}
JC.update(Indexes);
FunctionHash = (uint64_t)CallProbeIds.size() << 48 |
(uint64_t)Indexes.size() << 32 | JC.getCRC();
// Reserve bit 60-63 for other information purpose.
FunctionHash &= 0x0FFFFFFFFFFFFFFF;
assert(FunctionHash && "Function checksum should not be zero");
LLVM_DEBUG(dbgs() << "\nFunction Hash Computation for " << F->getName()
<< ":\n"
<< " CRC = " << JC.getCRC() << ", Edges = "
<< Indexes.size() << ", ICSites = " << CallProbeIds.size()
<< ", Hash = " << FunctionHash << "\n");
}
void SampleProfileProber::computeProbeIdForBlocks() {
@ -150,14 +184,50 @@ void SampleProfileProber::instrumentOneFunc(Function &F, TargetMachine *TM) {
Call->setDebugLoc(DIL);
}
}
// Create module-level metadata that contains function info necessary to
// synthesize probe-based sample counts, which are
// - FunctionGUID
// - FunctionHash.
// - FunctionName
auto Hash = getFunctionHash();
auto *MD = MDB.createPseudoProbeDesc(Guid, Hash, &F);
auto *NMD = M->getNamedMetadata(PseudoProbeDescMetadataName);
assert(NMD && "llvm.pseudo_probe_desc should be pre-created");
NMD->addOperand(MD);
// Preserve a comdat group to hold all probes materialized later. This
// allows that when the function is considered dead and removed, the
// materialized probes are disposed too.
// Imported functions are defined in another module. They do not need
// the following handling since same care will be taken for them in their
// original module. The pseudo probes inserted into an imported functions
// above will naturally not be emitted since the imported function is free
// from object emission. However they will be emitted together with the
// inliner functions that the imported function is inlined into. We are not
// creating a comdat group for an import function since it's useless anyway.
if (!F.isDeclarationForLinker()) {
if (TM) {
auto Triple = TM->getTargetTriple();
if (Triple.supportsCOMDAT() && TM->getFunctionSections()) {
GetOrCreateFunctionComdat(F, Triple, CurModuleUniqueId);
}
}
}
}
PreservedAnalyses SampleProfileProbePass::run(Module &M,
ModuleAnalysisManager &AM) {
auto ModuleId = getUniqueModuleId(&M);
// Create the pseudo probe desc metadata beforehand.
// Note that modules with only data but no functions will require this to
// be set up so that they will be known as probed later.
M.getOrInsertNamedMetadata(PseudoProbeDescMetadataName);
for (auto &F : M) {
if (F.isDeclaration())
continue;
SampleProfileProber ProbeManager(F);
SampleProfileProber ProbeManager(F, ModuleId);
ProbeManager.instrumentOneFunc(F, TM);
}

95
test/Transforms/SampleProfile/pseudo-probe-emit-inline.ll Normal file
View File

@ -0,0 +1,95 @@
; REQUIRES: x86_64-linux
; RUN: opt < %s -passes='pseudo-probe,cgscc(inline)' -function-sections -mtriple=x86_64-unknown-linux-gnu -S -o %t
; RUN: FileCheck %s < %t --check-prefix=CHECK-IL
; RUN: llc -pseudo-probe-for-profiling -function-sections <%t -filetype=asm -o %t1
; RUN: FileCheck %s < %t1 --check-prefix=CHECK-ASM
; RUN: llc -pseudo-probe-for-profiling -function-sections <%t -filetype=obj -o %t2
; RUN: llvm-objdump --section-headers %t2 | FileCheck %s --check-prefix=CHECK-OBJ
; RUN: llvm-mc -filetype=asm <%t1 -o %t3
; RUN: FileCheck %s < %t3 --check-prefix=CHECK-ASM
; RUN: llvm-mc -filetype=obj <%t1 -o %t4
; RUN: llvm-objdump --section-headers %t4 | FileCheck %s --check-prefix=CHECK-OBJ
define dso_local void @foo2() !dbg !7 {
; CHECK-IL: call void @llvm.pseudoprobe(i64 [[#GUID1:]], i64 1, i32 0), !dbg ![[#]]
; CHECK-ASM: .pseudoprobe [[#GUID1:]] 1 0 0
ret void, !dbg !10
}
define dso_local void @foo() #0 !dbg !11 {
; CHECK-IL: call void @llvm.pseudoprobe(i64 [[#GUID2:]], i64 1, i32 0), !dbg ![[#]]
; CHECK-IL: call void @llvm.pseudoprobe(i64 [[#GUID1]], i64 1, i32 0), !dbg ![[#DL1:]]
; CHECK-ASM: .pseudoprobe [[#GUID2:]] 1 0 0
; CHECK-ASM: .pseudoprobe [[#GUID1]] 1 0 0 @ [[#GUID2]]:2
call void @foo2(), !dbg !12
ret void, !dbg !13
}
define dso_local i32 @entry() !dbg !14 {
; CHECK-IL: call void @llvm.pseudoprobe(i64 [[#GUID3:]], i64 1, i32 0), !dbg ![[#]]
; CHECK-IL: call void @llvm.pseudoprobe(i64 [[#GUID2]], i64 1, i32 0), !dbg ![[#DL2:]]
; CHECK-IL: call void @llvm.pseudoprobe(i64 [[#GUID1]], i64 1, i32 0), !dbg ![[#DL3:]]
; CHECK-ASM: .pseudoprobe [[#GUID3:]] 1 0 0
; CHECK-ASM: .pseudoprobe [[#GUID2]] 1 0 0 @ [[#GUID3]]:2
; CHECK-ASM: .pseudoprobe [[#GUID1]] 1 0 0 @ [[#GUID3]]:2 @ [[#GUID2]]:2
call void @foo(), !dbg !18
ret i32 0, !dbg !19
}
; CHECK-IL: ![[#SCOPE1:]] = distinct !DISubprogram(name: "foo2"
; CHECK-IL: ![[#SCOPE2:]] = distinct !DISubprogram(name: "foo"
; CHECK-IL: ![[#DL1]] = !DILocation(line: 3, column: 1, scope: ![[#SCOPE1]], inlinedAt: ![[#INL1:]])
; CHECK-IL: ![[#INL1]] = distinct !DILocation(line: 7, column: 3, scope: ![[#BL1:]])
;; A discriminator of 134217751 which is 0x8000017 in hexdecimal, stands for a direct call probe
;; with an index of 2.
; CHECK-IL: ![[#BL1]] = !DILexicalBlockFile(scope: ![[#SCOPE2]], file: !1, discriminator: 134217751)
; CHECK-IL: ![[#SCOPE3:]] = distinct !DISubprogram(name: "entry"
; CHECK-IL: ![[#DL2]] = !DILocation(line: 7, column: 3, scope: ![[#SCOPE2]], inlinedAt: ![[#INL2:]])
; CHECK-IL: ![[#INL2]] = distinct !DILocation(line: 11, column: 3, scope: ![[#BL2:]])
; CHECK-IL: ![[#BL2]] = !DILexicalBlockFile(scope: ![[#SCOPE3]], file: !1, discriminator: 134217751)
; CHECK-IL: ![[#DL3]] = !DILocation(line: 3, column: 1, scope: ![[#SCOPE1]], inlinedAt: ![[#INL3:]])
; CHECK-IL: ![[#INL3]] = distinct !DILocation(line: 7, column: 3, scope: ![[#BL1]], inlinedAt: ![[#INL2]])
; Check the generation of .pseudo_probe_desc section
; CHECK-ASM: .section .pseudo_probe_desc,"G",@progbits,.pseudo_probe_desc_foo2,comdat
; CHECK-ASM-NEXT: .quad [[#GUID1]]
; CHECK-ASM-NEXT: .quad [[#HASH1:]]
; CHECK-ASM-NEXT: .byte 4
; CHECK-ASM-NEXT: .ascii "foo2"
; CHECK-ASM-NEXT: .section .pseudo_probe_desc,"G",@progbits,.pseudo_probe_desc_foo,comdat
; CHECK-ASM-NEXT: .quad [[#GUID2]]
; CHECK-ASM-NEXT: .quad [[#HASH2:]]
; CHECK-ASM-NEXT: .byte 3
; CHECK-ASM-NEXT: .ascii "foo"
; CHECK-ASM-NEXT: .section .pseudo_probe_desc,"G",@progbits,.pseudo_probe_desc_entry,comdat
; CHECK-ASM-NEXT: .quad [[#GUID3]]
; CHECK-ASM-NEXT: .quad [[#HASH3:]]
; CHECK-ASM-NEXT: .byte 5
; CHECK-ASM-NEXT: .ascii "entry"
; CHECK-OBJ: .pseudo_probe_desc
; CHECK-OBJ: .pseudo_probe
!llvm.dbg.cu = !{!0}
!llvm.module.flags = !{!3, !4}
!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, emissionKind: FullDebug)
!1 = !DIFile(filename: "foo.c", directory: "any")
!2 = !{}
!3 = !{i32 2, !"Dwarf Version", i32 4}
!4 = !{i32 2, !"Debug Info Version", i32 3}
!7 = distinct !DISubprogram(name: "foo2", scope: !1, file: !1, line: 2, type: !8, scopeLine: 2, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !2)
!8 = !DISubroutineType(types: !9)
!9 = !{null}
!10 = !DILocation(line: 3, column: 1, scope: !7)
!11 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 6, type: !8, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !2)
!12 = !DILocation(line: 7, column: 3, scope: !11)
!13 = !DILocation(line: 8, column: 1, scope: !11)
!14 = distinct !DISubprogram(name: "entry", scope: !1, file: !1, line: 10, type: !15, scopeLine: 10, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !2)
!15 = !DISubroutineType(types: !16)
!16 = !{!17}
!17 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
!18 = !DILocation(line: 11, column: 3, scope: !14)
!19 = !DILocation(line: 12, column: 3, scope: !14)

32
test/Transforms/SampleProfile/pseudo-probe-emit.ll
View File

@ -1,8 +1,14 @@
; REQUIRES: x86-registered-target
; REQUIRES: x86_64-linux
; RUN: opt < %s -passes=pseudo-probe -function-sections -S -o %t
; RUN: FileCheck %s < %t --check-prefix=CHECK-IL
; RUN: llc %t -pseudo-probe-for-profiling -stop-after=pseudo-probe-inserter -o - | FileCheck %s --check-prefix=CHECK-MIR
;
; RUN: llc %t -pseudo-probe-for-profiling -function-sections -filetype=asm -o %t1
; RUN: FileCheck %s < %t1 --check-prefix=CHECK-ASM
; RUN: llc %t -pseudo-probe-for-profiling -function-sections -filetype=obj -o %t2
; RUN: llvm-objdump --section-headers %t2 | FileCheck %s --check-prefix=CHECK-OBJ
; RUN: llvm-mc %t1 -filetype=obj -o %t3
; RUN: llvm-objdump --section-headers %t3 | FileCheck %s --check-prefix=CHECK-OBJ
;; Check the generation of pseudoprobe intrinsic call.
define void @foo(i32 %x) !dbg !3 {
@ -10,18 +16,23 @@ bb0:
%cmp = icmp eq i32 %x, 0
; CHECK-IL: call void @llvm.pseudoprobe(i64 [[#GUID:]], i64 1, i32 0), !dbg ![[#FAKELINE:]]
; CHECK-MIR: PSEUDO_PROBE [[#GUID:]], 1, 0, 0
; CHECK-ASM: .pseudoprobe [[#GUID:]] 1 0 0
br i1 %cmp, label %bb1, label %bb2
bb1:
; CHECK-IL: call void @llvm.pseudoprobe(i64 [[#GUID:]], i64 2, i32 0), !dbg ![[#FAKELINE]]
; CHECK-MIR: PSEUDO_PROBE [[#GUID]], 3, 0, 0
; CHECK-MIR: PSEUDO_PROBE [[#GUID]], 4, 0, 0
; CHECK-ASM: .pseudoprobe [[#GUID]] 3 0 0
; CHECK-ASM: .pseudoprobe [[#GUID]] 4 0 0
br label %bb3
bb2:
; CHECK-IL: call void @llvm.pseudoprobe(i64 [[#GUID:]], i64 3, i32 0), !dbg ![[#FAKELINE]]
; CHECK-MIR: PSEUDO_PROBE [[#GUID]], 2, 0, 0
; CHECK-MIR: PSEUDO_PROBE [[#GUID]], 4, 0, 0
; CHECK-ASM: .pseudoprobe [[#GUID]] 2 0 0
; CHECK-ASM: .pseudoprobe [[#GUID]] 4 0 0
br label %bb3
bb3:
@ -35,13 +46,16 @@ define internal void @foo2(void (i32)* %f) !dbg !4 {
entry:
; CHECK-IL: call void @llvm.pseudoprobe(i64 [[#GUID2:]], i64 1, i32 0)
; CHECK-MIR: PSEUDO_PROBE [[#GUID2:]], 1, 0, 0
; CHECK-ASM: .pseudoprobe [[#GUID2:]] 1 0 0
; Check pseudo_probe metadata attached to the indirect call instruction.
; CHECK-IL: call void %f(i32 1), !dbg ![[#PROBE0:]]
; CHECK-MIR: PSEUDO_PROBE [[#GUID2]], 2, 1, 0
; CHECK-ASM: .pseudoprobe [[#GUID2]] 2 1 0
call void %f(i32 1), !dbg !13
; Check pseudo_probe metadata attached to the direct call instruction.
; CHECK-IL: call void @bar(i32 1), !dbg ![[#PROBE1:]]
; CHECK-MIR: PSEUDO_PROBE [[#GUID2]], 3, 2, 0
; CHECK-ASM: .pseudoprobe [[#GUID2]] 3 2 0
call void @bar(i32 1)
ret void
}
@ -58,6 +72,20 @@ entry:
;; with an index of 3.
; CHECK-IL: ![[#SCOPE1]] = !DILexicalBlockFile(scope: ![[#]], file: ![[#]], discriminator: 134217759)
; Check the generation of .pseudo_probe_desc section
; CHECK-ASM: .section .pseudo_probe_desc,"G",@progbits,.pseudo_probe_desc_foo,comdat
; CHECK-ASM-NEXT: .quad [[#GUID]]
; CHECK-ASM-NEXT: .quad [[#HASH:]]
; CHECK-ASM-NEXT: .byte 3
; CHECK-ASM-NEXT: .ascii "foo"
; CHECK-ASM-NEXT: .section .pseudo_probe_desc,"G",@progbits,.pseudo_probe_desc_foo2,comdat
; CHECK-ASM-NEXT: .quad [[#GUID2]]
; CHECK-ASM-NEXT: .quad [[#HASH2:]]
; CHECK-ASM-NEXT: .byte 4
; CHECK-ASM-NEXT: .ascii "foo2"
; CHECK-OBJ-COUNT-2: .pseudo_probe_desc
; CHECK-OBJ-COUNT-2: .pseudo_probe
!llvm.dbg.cu = !{!0}
!llvm.module.flags = !{!9, !10}