mirror of
https://github.com/RPCS3/llvm-mirror.git
synced 2024-11-22 18:54:02 +01:00
85e4f6f241
This change implements pseudo probe encoding and emission for CSSPGO. Please see RFC here for more context: https://groups.google.com/g/llvm-dev/c/1p1rdYbL93s Pseudo probes are in the form of intrinsic calls on IR/MIR but they do not turn into any machine instructions. Instead they are emitted into the binary as a piece of data in standalone sections. The probe-specific sections are not needed to be loaded into memory at execution time, thus they do not incur a runtime overhead. **ELF object emission** The binary data to emit are organized as two ELF sections, i.e, the `.pseudo_probe_desc` section and the `.pseudo_probe` section. The `.pseudo_probe_desc` section stores a function descriptor for each function and the `.pseudo_probe` section stores the actual probes, each fo which corresponds to an IR basic block or an IR function callsite. A function descriptor is stored as a module-level metadata during the compilation and is serialized into the object file during object emission. Both the probe descriptors and pseudo probes can be emitted into a separate ELF section per function to leverage the linker for deduplication. A `.pseudo_probe` section shares the same COMDAT group with the function code so that when the function is dead, the probes are dead and disposed too. On the contrary, a `.pseudo_probe_desc` section has its own COMDAT group. This is because even if a function is dead, its probes may be inlined into other functions and its descriptor is still needed by the profile generation tool. The format of `.pseudo_probe_desc` section looks like: ``` .section .pseudo_probe_desc,"",@progbits .quad 6309742469962978389 // Func GUID .quad 4294967295 // Func Hash .byte 9 // Length of func name .ascii "_Z5funcAi" // Func name .quad 7102633082150537521 .quad 138828622701 .byte 12 .ascii "_Z8funcLeafi" .quad 446061515086924981 .quad 4294967295 .byte 9 .ascii "_Z5funcBi" .quad -2016976694713209516 .quad 72617220756 .byte 7 .ascii "_Z3fibi" ``` For each `.pseudoprobe` section, the encoded binary data consists of a single function record corresponding to an outlined function (i.e, a function with a code entry in the `.text` section). A function record has the following format : ``` FUNCTION BODY (one for each outlined function present in the text section) GUID (uint64) GUID of the function NPROBES (ULEB128) Number of probes originating from this function. NUM_INLINED_FUNCTIONS (ULEB128) Number of callees inlined into this function, aka number of first-level inlinees PROBE RECORDS A list of NPROBES entries. Each entry contains: INDEX (ULEB128) TYPE (uint4) 0 - block probe, 1 - indirect call, 2 - direct call ATTRIBUTE (uint3) reserved ADDRESS_TYPE (uint1) 0 - code address, 1 - address delta CODE_ADDRESS (uint64 or ULEB128) code address or address delta, depending on ADDRESS_TYPE INLINED FUNCTION RECORDS A list of NUM_INLINED_FUNCTIONS entries describing each of the inlined callees. Each record contains: INLINE SITE GUID of the inlinee (uint64) ID of the callsite probe (ULEB128) FUNCTION BODY A FUNCTION BODY entry describing the inlined function. ``` To support building a context-sensitive profile, probes from inlinees are grouped by their inline contexts. An inline context is logically a call path through which a callee function lands in a caller function. The probe emitter builds an inline tree based on the debug metadata for each outlined function in the form of a trie tree. A tree root is the outlined function. Each tree edge stands for a callsite where inlining happens. Pseudo probes originating from an inlinee function are stored in a tree node and the tree path starting from the root all the way down to the tree node is the inline context of the probes. The emission happens on the whole tree top-down recursively. Probes of a tree node will be emitted altogether with their direct parent edge. Since a pseudo probe corresponds to a real code address, for size savings, the address is encoded as a delta from the previous probe except for the first probe. Variant-sized integer encoding, aka LEB128, is used for address delta and probe index. **Assembling** Pseudo probes can be printed as assembly directives alternatively. This allows for good assembly code readability and also provides a view of how optimizations and pseudo probes affect each other, especially helpful for diff time assembly analysis. A pseudo probe directive has the following operands in order: function GUID, probe index, probe type, probe attributes and inline context. The directive is generated by the compiler and can be parsed by the assembler to form an encoded `.pseudoprobe` section in the object file. A example assembly looks like: ``` foo2: # @foo2 # %bb.0: # %bb0 pushq %rax testl %edi, %edi .pseudoprobe 837061429793323041 1 0 0 je .LBB1_1 # %bb.2: # %bb2 .pseudoprobe 837061429793323041 6 2 0 callq foo .pseudoprobe 837061429793323041 3 0 0 .pseudoprobe 837061429793323041 4 0 0 popq %rax retq .LBB1_1: # %bb1 .pseudoprobe 837061429793323041 5 1 0 callq *%rsi .pseudoprobe 837061429793323041 2 0 0 .pseudoprobe 837061429793323041 4 0 0 popq %rax retq # -- End function .section .pseudo_probe_desc,"",@progbits .quad 6699318081062747564 .quad 72617220756 .byte 3 .ascii "foo" .quad 837061429793323041 .quad 281547593931412 .byte 4 .ascii "foo2" ``` With inlining turned on, the assembly may look different around %bb2 with an inlined probe: ``` # %bb.2: # %bb2 .pseudoprobe 837061429793323041 3 0 .pseudoprobe 6699318081062747564 1 0 @ 837061429793323041:6 .pseudoprobe 837061429793323041 4 0 popq %rax retq ``` **Disassembling** We have a disassembling tool (llvm-profgen) that can display disassembly alongside with pseudo probes. So far it only supports ELF executable file. An example disassembly looks like: ``` 00000000002011a0 <foo2>: 2011a0: 50 push rax 2011a1: 85 ff test edi,edi [Probe]: FUNC: foo2 Index: 1 Type: Block 2011a3: 74 02 je 2011a7 <foo2+0x7> [Probe]: FUNC: foo2 Index: 3 Type: Block [Probe]: FUNC: foo2 Index: 4 Type: Block [Probe]: FUNC: foo Index: 1 Type: Block Inlined: @ foo2:6 2011a5: 58 pop rax 2011a6: c3 ret [Probe]: FUNC: foo2 Index: 2 Type: Block 2011a7: bf 01 00 00 00 mov edi,0x1 [Probe]: FUNC: foo2 Index: 5 Type: IndirectCall 2011ac: ff d6 call rsi [Probe]: FUNC: foo2 Index: 4 Type: Block 2011ae: 58 pop rax 2011af: c3 ret ``` Reviewed By: wmi Differential Revision: https://reviews.llvm.org/D91878
218 lines
8.6 KiB
C++
218 lines
8.6 KiB
C++
//===---- llvm/MDBuilder.h - Builder for LLVM metadata ----------*- C++ -*-===//
|
|
//
|
|
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
// See https://llvm.org/LICENSE.txt for license information.
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
//
|
|
//===----------------------------------------------------------------------===//
|
|
//
|
|
// This file defines the MDBuilder class, which is used as a convenient way to
|
|
// create LLVM metadata with a consistent and simplified interface.
|
|
//
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
#ifndef LLVM_IR_MDBUILDER_H
|
|
#define LLVM_IR_MDBUILDER_H
|
|
|
|
#include "llvm/ADT/DenseSet.h"
|
|
#include "llvm/ADT/StringRef.h"
|
|
#include "llvm/IR/Constants.h"
|
|
#include "llvm/IR/GlobalValue.h"
|
|
#include "llvm/Support/DataTypes.h"
|
|
#include <utility>
|
|
|
|
namespace llvm {
|
|
|
|
class APInt;
|
|
template <typename T> class ArrayRef;
|
|
class LLVMContext;
|
|
class Constant;
|
|
class ConstantAsMetadata;
|
|
class MDNode;
|
|
class MDString;
|
|
class Metadata;
|
|
|
|
class MDBuilder {
|
|
LLVMContext &Context;
|
|
|
|
public:
|
|
MDBuilder(LLVMContext &context) : Context(context) {}
|
|
|
|
/// Return the given string as metadata.
|
|
MDString *createString(StringRef Str);
|
|
|
|
/// Return the given constant as metadata.
|
|
ConstantAsMetadata *createConstant(Constant *C);
|
|
|
|
//===------------------------------------------------------------------===//
|
|
// FPMath metadata.
|
|
//===------------------------------------------------------------------===//
|
|
|
|
/// Return metadata with the given settings. The special value 0.0
|
|
/// for the Accuracy parameter indicates the default (maximal precision)
|
|
/// setting.
|
|
MDNode *createFPMath(float Accuracy);
|
|
|
|
//===------------------------------------------------------------------===//
|
|
// Prof metadata.
|
|
//===------------------------------------------------------------------===//
|
|
|
|
/// Return metadata containing two branch weights.
|
|
MDNode *createBranchWeights(uint32_t TrueWeight, uint32_t FalseWeight);
|
|
|
|
/// Return metadata containing a number of branch weights.
|
|
MDNode *createBranchWeights(ArrayRef<uint32_t> Weights);
|
|
|
|
/// Return metadata specifying that a branch or switch is unpredictable.
|
|
MDNode *createUnpredictable();
|
|
|
|
/// Return metadata containing the entry \p Count for a function, a boolean
|
|
/// \Synthetic indicating whether the counts were synthetized, and the
|
|
/// GUIDs stored in \p Imports that need to be imported for sample PGO, to
|
|
/// enable the same inlines as the profiled optimized binary
|
|
MDNode *createFunctionEntryCount(uint64_t Count, bool Synthetic,
|
|
const DenseSet<GlobalValue::GUID> *Imports);
|
|
|
|
/// Return metadata containing the section prefix for a function.
|
|
MDNode *createFunctionSectionPrefix(StringRef Prefix);
|
|
|
|
/// Return metadata containing the pseudo probe descriptor for a function.
|
|
MDNode *createPseudoProbeDesc(uint64_t GUID, uint64_t Hash, Function *F);
|
|
|
|
//===------------------------------------------------------------------===//
|
|
// Range metadata.
|
|
//===------------------------------------------------------------------===//
|
|
|
|
/// Return metadata describing the range [Lo, Hi).
|
|
MDNode *createRange(const APInt &Lo, const APInt &Hi);
|
|
|
|
/// Return metadata describing the range [Lo, Hi).
|
|
MDNode *createRange(Constant *Lo, Constant *Hi);
|
|
|
|
//===------------------------------------------------------------------===//
|
|
// Callees metadata.
|
|
//===------------------------------------------------------------------===//
|
|
|
|
/// Return metadata indicating the possible callees of indirect
|
|
/// calls.
|
|
MDNode *createCallees(ArrayRef<Function *> Callees);
|
|
|
|
//===------------------------------------------------------------------===//
|
|
// Callback metadata.
|
|
//===------------------------------------------------------------------===//
|
|
|
|
/// Return metadata describing a callback (see llvm::AbstractCallSite).
|
|
MDNode *createCallbackEncoding(unsigned CalleeArgNo, ArrayRef<int> Arguments,
|
|
bool VarArgsArePassed);
|
|
|
|
/// Merge the new callback encoding \p NewCB into \p ExistingCallbacks.
|
|
MDNode *mergeCallbackEncodings(MDNode *ExistingCallbacks, MDNode *NewCB);
|
|
|
|
//===------------------------------------------------------------------===//
|
|
// AA metadata.
|
|
//===------------------------------------------------------------------===//
|
|
|
|
protected:
|
|
/// Return metadata appropriate for a AA root node (scope or TBAA).
|
|
/// Each returned node is distinct from all other metadata and will never
|
|
/// be identified (uniqued) with anything else.
|
|
MDNode *createAnonymousAARoot(StringRef Name = StringRef(),
|
|
MDNode *Extra = nullptr);
|
|
|
|
public:
|
|
/// Return metadata appropriate for a TBAA root node. Each returned
|
|
/// node is distinct from all other metadata and will never be identified
|
|
/// (uniqued) with anything else.
|
|
MDNode *createAnonymousTBAARoot() {
|
|
return createAnonymousAARoot();
|
|
}
|
|
|
|
/// Return metadata appropriate for an alias scope domain node.
|
|
/// Each returned node is distinct from all other metadata and will never
|
|
/// be identified (uniqued) with anything else.
|
|
MDNode *createAnonymousAliasScopeDomain(StringRef Name = StringRef()) {
|
|
return createAnonymousAARoot(Name);
|
|
}
|
|
|
|
/// Return metadata appropriate for an alias scope root node.
|
|
/// Each returned node is distinct from all other metadata and will never
|
|
/// be identified (uniqued) with anything else.
|
|
MDNode *createAnonymousAliasScope(MDNode *Domain,
|
|
StringRef Name = StringRef()) {
|
|
return createAnonymousAARoot(Name, Domain);
|
|
}
|
|
|
|
/// Return metadata appropriate for a TBAA root node with the given
|
|
/// name. This may be identified (uniqued) with other roots with the same
|
|
/// name.
|
|
MDNode *createTBAARoot(StringRef Name);
|
|
|
|
/// Return metadata appropriate for an alias scope domain node with
|
|
/// the given name. This may be identified (uniqued) with other roots with
|
|
/// the same name.
|
|
MDNode *createAliasScopeDomain(StringRef Name);
|
|
|
|
/// Return metadata appropriate for an alias scope node with
|
|
/// the given name. This may be identified (uniqued) with other scopes with
|
|
/// the same name and domain.
|
|
MDNode *createAliasScope(StringRef Name, MDNode *Domain);
|
|
|
|
/// Return metadata for a non-root TBAA node with the given name,
|
|
/// parent in the TBAA tree, and value for 'pointsToConstantMemory'.
|
|
MDNode *createTBAANode(StringRef Name, MDNode *Parent,
|
|
bool isConstant = false);
|
|
|
|
struct TBAAStructField {
|
|
uint64_t Offset;
|
|
uint64_t Size;
|
|
MDNode *Type;
|
|
TBAAStructField(uint64_t Offset, uint64_t Size, MDNode *Type) :
|
|
Offset(Offset), Size(Size), Type(Type) {}
|
|
};
|
|
|
|
/// Return metadata for a tbaa.struct node with the given
|
|
/// struct field descriptions.
|
|
MDNode *createTBAAStructNode(ArrayRef<TBAAStructField> Fields);
|
|
|
|
/// Return metadata for a TBAA struct node in the type DAG
|
|
/// with the given name, a list of pairs (offset, field type in the type DAG).
|
|
MDNode *
|
|
createTBAAStructTypeNode(StringRef Name,
|
|
ArrayRef<std::pair<MDNode *, uint64_t>> Fields);
|
|
|
|
/// Return metadata for a TBAA scalar type node with the
|
|
/// given name, an offset and a parent in the TBAA type DAG.
|
|
MDNode *createTBAAScalarTypeNode(StringRef Name, MDNode *Parent,
|
|
uint64_t Offset = 0);
|
|
|
|
/// Return metadata for a TBAA tag node with the given
|
|
/// base type, access type and offset relative to the base type.
|
|
MDNode *createTBAAStructTagNode(MDNode *BaseType, MDNode *AccessType,
|
|
uint64_t Offset, bool IsConstant = false);
|
|
|
|
/// Return metadata for a TBAA type node in the TBAA type DAG with the
|
|
/// given parent type, size in bytes, type identifier and a list of fields.
|
|
MDNode *createTBAATypeNode(MDNode *Parent, uint64_t Size, Metadata *Id,
|
|
ArrayRef<TBAAStructField> Fields =
|
|
ArrayRef<TBAAStructField>());
|
|
|
|
/// Return metadata for a TBAA access tag with the given base type,
|
|
/// final access type, offset of the access relative to the base type, size of
|
|
/// the access and flag indicating whether the accessed object can be
|
|
/// considered immutable for the purposes of the TBAA analysis.
|
|
MDNode *createTBAAAccessTag(MDNode *BaseType, MDNode *AccessType,
|
|
uint64_t Offset, uint64_t Size,
|
|
bool IsImmutable = false);
|
|
|
|
/// Return mutable version of the given mutable or immutable TBAA
|
|
/// access tag.
|
|
MDNode *createMutableTBAAAccessTag(MDNode *Tag);
|
|
|
|
/// Return metadata containing an irreducible loop header weight.
|
|
MDNode *createIrrLoopHeaderWeight(uint64_t Weight);
|
|
};
|
|
|
|
} // end namespace llvm
|
|
|
|
#endif
|