On z/OS, the following error message is not matched correctly in lit tests.
```
EDC5129I No such file or directory.
```
This patch uses a lit config substitution to check for platform specific error messages.
Reviewed By: muiez, jhenderson
Differential Revision: https://reviews.llvm.org/D95246
On z/OS, the following error message is not matched correctly in lit tests. This patch updates the CHECK expression to match successfully.
```
EDC5129I No such file or directory.
```
Reviewed By: muiez
Differential Revision: https://reviews.llvm.org/D94239
When we produce an YAML output, we also print leading zeroes currently.
An output might look like this:
```
- Name: .dynsym
Type: SHT_DYNSYM
Address: 0x0000000000001000
EntSize: 0x0000000000000018
```
There are probably no reason to print leading zeroes.
It just makes harder to read values. This patch stops printing them.
The output becomes like:
```
- Name: .dynsym
Type: SHT_DYNSYM
Address: 0x1000
EntSize: 0x18
```
This affects obj2yaml mostly, but also dsymutil and llvm-xray tools output.
Differential revision: https://reviews.llvm.org/D90930
Imagine we have a YAML declaration of few sections: `foo1`, `<unnamed 2>`, `foo3`, `foo4`.
To put them into segment we can do (1*):
```
Sections:
- Section: foo1
- Section: foo4
```
or we can use (2*):
```
Sections:
- Section: foo1
- Section: foo3
- Section: foo4
```
or (3*) :
```
Sections:
- Section: foo1
## "(index 2)" here is a name that we automatically created for a unnamed section.
- Section: (index 2)
- Section: foo3
- Section: foo4
```
It looks really confusing that we don't have to list all of sections.
At first I've tried to make this rule stricter and report an error when there is a gap
(i.e. when a section is included into segment, but not listed explicitly).
This did not work perfect, because such approach conflicts with unnamed sections/fills (see (3*)).
This patch drops "Sections" key and introduces 2 keys instead: `FirstSec` and `LastSec`.
Both are optional.
Differential revision: https://reviews.llvm.org/D90458
Summary:
Recursion detection can be non-trivial. Currently, the state-of-the-art for LLVM,
as far as i'm concerned, is D72362 `[clang-tidy] misc-no-recursion: a new check`.
However, it is quite limited:
* It does very basic call-graph based analysis, in the sense it will report even dynamically-unreachable recursion.
* It is inherently limited to a single TU
* It is hard to gauge how problematic each recursion is in practice.
Some of that can be addressed by adding clang analyzer-based check,
then it would at least support multiple TU's.
However, we can approach this problem from another angle - dynamic run-time analysis.
We already have means to capture a run-time callgraph (XRay, duh),
and there are already means to reconstruct it within `llvm-xray` tool.
This proposes to add a `-recursive-calls-only` switch to the `account` tool.
When the switch is on, when re-constructing callgraph for latency reconstruction,
each time we enter/leave some function, we increment/decrement an entry for the function
in a "recursion depth" map. If, when we leave the function, said entry was at `1`,
then that means the function didn't call itself, however if it is at `2` or more,
then that means the function (possibly indirectly) called itself.
If the depth is 1, we don't account the time spent there,
unless within this call stack the function already recursed into itself.
Note that we don't pay for recursion depth tracking when `recursive-calls-only` is not on,
and the perf impact is insignificant (+0.3% regression)
The overhead of the option is actually negative, around -5.26% user time on a medium-sized (3.5G) XRay log.
As a practical example, that 3.5G log is a capture of the entire middle-end opt pipeline
at `-O3` for RawSpeed unity build. There are total of `5500` functions in the log,
however `-recursive-calls-only` says that `269`, or 5%, are recursive.
Having this functionality could be helpful for recursion eradication.
Reviewers: dberris, mboerger
Reviewed By: dberris
Subscribers: llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D84582
This patch extends D58439 (`llvm/test/{yaml2obj,obj2yaml}/**/*.yaml`) and runs all
`llvm/test/**/*.yaml`
Many directories have configured `.yaml` (see the deleted lit.local.cfg
files). Yet still some don't configure .yaml and have caused stale tests:
* 8c5825befb7bbb2e76f7eccedc6d3bf26e9b2a6a test/llvm-readobj
* bdc3134e237737dd46b51cd1ecd41ecbbe9f921a test/ExecutionEngine
Just hoist .yaml to `llvm/test/lit.cfg.py`. Also delete .cxx which is
not used. The number of tests running on my machine increases from 38304 to 38309.
The list of new tests:
```
ExecutionEngine/RuntimeDyld/X86/ELF_x86-64_none.yaml
Object/archive-error-tmp.txt
tools/llvm-ar/coff-weak.yaml
tools/llvm-readobj/ELF/verneed-flags.yaml
tools/obj2yaml/COFF/bss.s
```
Reviewed By: grimar, jhenderson, rupprecht
Differential Revision: https://reviews.llvm.org/D83350
Summary:
XRay works on 32-bit ARM but extract didn't support it.
See also another previous attempt in D77858.
Reviewers: MaskRay, dberris, johnislarry
Subscribers: kristof.beyls, hiraditya, danielkiss, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D80185
Rename extract-instrmap-aarch64.test to extract-instrmap.test because
the path component `AArch64` conveys the target name clearly.
Additionally, adopt a convention we start to use in LLVM binary
utilities: prepend `#` to CHECK/RUN lines and `##` to comment lines even
if the file contains no code. The notation makes CHECK/RUN/comments
stand out.
Reviewed By: dberris
Differential Revision: https://reviews.llvm.org/D77883
This adds an additional cli flag for the llvm-xray extract tool. This
is useful if you're more interested in consuming the mangled symbol
name, instead of the default now which is demangled.
Differential Revision: https://reviews.llvm.org/D72804
When the instrumented binary is linked as PIE, we need to apply the
relative relocations to sleds. This is handled by the dynamic linker
at runtime, but when processing the file we have to do it ourselves.
Differential Revision: https://reviews.llvm.org/D55542
llvm-svn: 349120
Summary:
This change cuts across LLVM and compiler-rt to add support for
rendering custom events in the XRayRecord type, to allow for including
user-provided annotations in the output YAML (as raw bytes).
This work enables us to add custom event and typed event records into
the `llvm::xray::Trace` type for user-provided events. This can then be
programmatically handled through the C++ API and can be included in some
of the tooling as well. For now we support printing the raw data we
encounter in the custom events in the converted output.
Future work will allow us to start interpreting these custom and typed
events through a yet-to-be-defined API for extending the trace analysis
library.
Reviewers: mboerger
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D54139
llvm-svn: 346214
Summary:
This more correctly reflects the data written by the FDR mode runtime.
This is a continuation of the work in D50441.
Reviewers: mboerger, eizan
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51911
llvm-svn: 341905
Summary:
In this change, we implement a `BlockPrinter` which orders records in a
Block that's been indexed by the `BlockIndexer`. This is used in the
`llvm-xray fdr-dump` tool which ties together the various types and
utilities we've been working on, to allow for inspection of XRay FDR
mode traces both with and without verification.
This change is the final step of the refactoring of D50441.
Reviewers: mboerger, eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51846
llvm-svn: 341887
Summary:
llvm-xray changes:
- account-mode - process-id {...} shows after thread-id
- convert-mode - process {...} shows after thread
- parses FDR and basic mode pid entries
- Checks version number for FDR log parsing.
Basic logging changes:
- Update header version from 2 -> 3
FDR logging changes:
- Update header version from 2 -> 3
- in writeBufferPreamble, there is an additional PID Metadata record (after thread id record and tsc record)
Test cases changes:
- fdr-mode.cc, fdr-single-thread.cc, fdr-thread-order.cc modified to catch process id output in the log.
Reviewers: dberris
Reviewed By: dberris
Subscribers: hiraditya, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D49153
llvm-svn: 336974
Summary:
The arg1 logging handler changed in compiler-rt to start writing a
different type for entries encountered when logging the first argument
of XRay-instrumented functions. This change allows the trace loader to
support reading these record types as well as prepare for when the
basic (naive) mode implementation starts writing down the argument
payloads.
Without this change, binaries with arg1 logging support enabled start
writing unreadable logs for any of the XRay tracing tools.
Reviewers: pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D38550
llvm-svn: 314967
Summary:
A new FDR metadata record will support logging a function call argument;
appending multiple metadata records will represent a sequence of arguments
meaning that "holes" are not representable by the buffer format. Each
call argument is currently a 64-bit value (useful for "this" pointers and
synchronization objects).
If present, we put this argument to the function call "entry" record it
belongs to, and alter its type to notify the user of its presence.
Reviewers: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D32840
llvm-svn: 314269
Summary:
This change adds support for explicit tail-exit records to be written by
the XRay runtime. This lets us differentiate the tail exit
records/events in the log, and allows us to treat those exit events
especially in the future. For now we allow printing those out in YAML
(and reading them in).
Reviewers: kpw, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D37964
llvm-svn: 313514
We already uses pipefail to detect failure of a redirected command, so
the "|| echo failure" construct was unnecessary.
These tests run and pass on Windows now.
llvm-svn: 312747
The tests are filechecking against stderr and use some magic to make stdout go
away and pipe stderr to FileCheck. This broke bots on windows.
llvm-svn: 312739
Second try after fixing a code san problem with iterator reference types.
This change introduces a subcommand to the llvm-xray tool called
"stacks" which allows for analysing XRay traces provided as inputs and
accounting time to stacks instead of just individual functions. This
gives us a more precise view of where in a program the latency is
actually attributed.
The tool uses a trie data structure to keep track of the caller-callee
relationships as we process the XRay traces. In particular, we keep
track of the function call stack as we enter functions. While we're
doing this we're adding nodes in a trie and indicating a "calls"
relatinship between the caller (current top of the stack) and the callee
(the new top of the stack). When we push function ids onto the stack, we
keep track of the timestamp (TSC) for the enter event.
When exiting functions, we are able to account the duration by getting
the difference between the timestamp of the exit event and the
corresponding entry event in the stack. This works even if we somehow
miss the exit events for intermediary functions (i.e. if the exit event
is not cleanly associated with the enter event at the top of the stack).
The output of the tool currently provides just the top N leaf functions
that contribute the most latency, and the top N stacks that have the
most frequency. In the future we can provide more sophisticated query
mechanisms and potentially an export to database feature to make offline
analysis of the stack traces possible with existing tools.
Differential revision: D34863
llvm-svn: 312733
This change introduces a subcommand to the llvm-xray tool called
"stacks" which allows for analysing XRay traces provided as inputs and
accounting time to stacks instead of just individual functions. This
gives us a more precise view of where in a program the latency is
actually attributed.
The tool uses a trie data structure to keep track of the caller-callee
relationships as we process the XRay traces. In particular, we keep
track of the function call stack as we enter functions. While we're
doing this we're adding nodes in a trie and indicating a "calls"
relatinship between the caller (current top of the stack) and the callee
(the new top of the stack). When we push function ids onto the stack, we
keep track of the timestamp (TSC) for the enter event.
When exiting functions, we are able to account the duration by getting
the difference between the timestamp of the exit event and the
corresponding entry event in the stack. This works even if we somehow
miss the exit events for intermediary functions (i.e. if the exit event
is not cleanly associated with the enter event at the top of the stack).
The output of the tool currently provides just the top N leaf functions
that contribute the most latency, and the top N stacks that have the
most frequency. In the future we can provide more sophisticated query
mechanisms and potentially an export to database feature to make offline
analysis of the stack traces possible with existing tools.
llvm-svn: 312426
Summary:
Before this patch, llvm-xray account will assume that thread stacks will
not be empty. Unfortunately there are cases where an instrumented
function will see a call to `fork()` which will cause the child process
to not see the start of the function, but only see the end of the
function. The tooling cannot assume that threads will always have
perfect stacks, and so we change it to support this reality.
Reviewers: dblaikie
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D31870
llvm-svn: 312204
Summary:
When extracting the instrumentation map from a binary, we should be able
to recognize the new kinds of instrumentation sleds we've been emitting
with the compiler using -fxray-instrument. This change adds a test for
all the kinds of sleds we currently support (sans the tail-call sled,
which is a bit harder to force in a simple prebuilt input).
Reviewers: kpw, dblaikie
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36819
llvm-svn: 311305
Summary:
This is a tool for comparing the function graphs produced by the
llvm-xray graph too. It takes the form of a new subcommand of the
llvm-xray tool 'graph-diff'.
This initial version of the patch is very rough, but it is close to
feature complete.
Depends on D29363
Reviewers: dblaikie, dberris
Reviewed By: dberris
Subscribers: mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D29320
llvm-svn: 301160
Summary:
This allows us to, if the symbol names are available in the binary, be
able to provide the function name in the YAML output.
Reviewers: dblaikie, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D32153
llvm-svn: 300624
Summary:
Assertions assuming that function calls may not have zero durations do
not seem to hold in the wild. There are valid cases where the conversion
of the tsc counters end up becoming zero-length durations. These
assertions don't really hold and the algorithms don't need those to be
true for them to work.
Reviewers: dblaikie, echristo
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D31519
llvm-svn: 299150
Summary:
It is problematic for this reader that it expects to read data from
several threads, but the header or message format does not define
framing. Since the buffers are reused, we can't rely on skipping
zeroed out data as a synchronization method either.
There is an argument that this is not version compatible with the format
the reader expected previously. I argue that since the writer wrote garbage
past the end of buffer record, there is no currently working reader to
compromise.
The corresponding writer change is posted to D31384.
Reviewers: dberris, pelikan
Reviewed By: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D31385
llvm-svn: 298983
Summary:
Currently the llvm-xray commandline tool fails to handle the case for
when no subcommand is provided in a graceful manner. This fixes that to
print the help message explaining the subcommands and the available
options.
Reviewers: pcc, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D31409
llvm-svn: 298975
Summary:
In Preparation for graph comparison, this patch breaks out the color
choice code from xray-graph into a library and adds polynomials for
the Sequential and Difference sets from ColorBrewer.
Depends on D29005
Reviewers: dblaikie, chandlerc, dberris
Reviewed By: dberris
Subscribers: chandlerc, llvm-commits, mgorny
Differential Revision: https://reviews.llvm.org/D29363
llvm-svn: 296210
Summary:
The file type packs function trace data onto disk from potentially multiple
threads that are aggregated and flushed during the course of an instrumented
program's runtime.
It is named FDR mode or Flight Data recorder as an analogy to plane
blackboxes, which instrument a running system without access to IO.
The writer code is defined in compiler-rt in xray_fdr_logging.h/cc
Reviewers: rSerge, kcc, dberris
Reviewed By: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D29697
llvm-svn: 295397
Summary:
The YAML output produced by llvm-xray is supposed to be wrapped at the
arbitrary default of 70 columns set by `yaml:Output`. Unfortunately,
the wrapping is rather unpredictable, and can easily go past the set
number of columns, depending on the execution environment.
To make the YAML output environment-independent, disable wrapping
instead.
Reviewers: dberris
Reviewed By: dberris
Subscribers: fhahn, llvm-commits
Differential Revision: https://reviews.llvm.org/D29962
llvm-svn: 295116
Summary:
This change implements the instrumentation map loading library which can
understand both YAML-defined instrumentation maps, and ELF 64-bit object
files that have the XRay instrumentation map section. We break it out
into a library on its own to allow for other applications to deal with
the XRay instrumentation map defined in XRay-instrumented binaries.
This type provides both raw access to the logical representation of the
instrumentation map entries as well as higher level functions for
converting a function ID into a function address.
At this point we only support ELF64 binaries and YAML-defined XRay
instrumentation maps. Future changes should extend this to support
32-bit ELF binaries, as well as other binary formats (like MachO).
As part of this change we also migrate all uses of the extraction logic
that used to be defined in tools/llvm-xray/ to use this new type and
interface for loading from files. We also remove the flag from the
`llvm-xray` tool that required users to specify the type of the
instrumentation map file being provided to instead make the library
auto-detect the file type.
Reviewers: dblaikie
Subscribers: mgorny, varno, llvm-commits
Differential Revision: https://reviews.llvm.org/D29319
llvm-svn: 293721
Summary:
A patch to enable the llvm-xray graph subcommand to color edges and
vertices based on statistics and to annotate vertices with statistics.
Depends on D27243
Reviewers: dblaikie, dberris
Reviewed By: dberris
Subscribers: mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D28225
llvm-svn: 293031
Here we define the `graph` subcommand which generates a graph from the function
call information and uses it to present the call information graphically with
additional annotations.
Reviewers: dblaikie, dberris
Differential Revision: https://reviews.llvm.org/D27243
llvm-svn: 292156
Summary:
This is the third of a multi-part change to implement subcommands for
the `llvm-xray` tool.
Here we define the `account` subcommand which does simple function call
accounting, generating basic statistics on function calls we find in an
XRay log/trace. We support text output and csv output for this
subcommand.
This change also supports sorting, summing, and filtering the top N
results.
Part of this tool will later be turned into a library that could be used
for basic function call accounting.
Depends on D24376.
Reviewers: dblaikie, echristo
Subscribers: mehdi_amini, dberris, beanz, llvm-commits
Differential Revision: https://reviews.llvm.org/D24377
llvm-svn: 291749
Summary:
In this change we move the definition of the log reading routines from
the tools directory in LLVM to {include/llvm,lib}/XRay. We improve the
documentation a little bit for the publicly accessible headers, and
adjust the top-matter. This also leads to some refactoring and cleanup
in the tooling code.
In particular, we do the following:
- Rename the class from LogReader to Trace, as it better represents
the logical set of records as opposed to a log.
- Use file type detection instead of asking the user to say what
format the input file is. This allows us to keep the interface
simple and encapsulate the logic of loading the data appropriately.
In future changes we increase the API surface and write dedicated unit
tests for the XRay library.
Depends on D24376.
Reviewers: dblaikie, echristo
Subscribers: mehdi_amini, mgorny, llvm-commits, varno
Differential Revision: https://reviews.llvm.org/D28345
llvm-svn: 291652
This is the second part of a multi-part change to define additional
subcommands to the `llvm-xray` tool.
This change defines a conversion subcommand to take XRay log files, and
turns them from one format to another (binary or YAML). This currently
only supports the first version of the log file format, defined in the
compiler-rt runtime.
Depends on D21987.
Reviewers: dblaikie, echristo
Subscribers: mehdi_amini, dberris, beanz, llvm-commits
Differential Revision: https://reviews.llvm.org/D24376
llvm-svn: 291529
