Add support for generating a dsymutil reproducer. The result is a folder
containing all the object files for linking.
When --gen-reproducer is passed, dsymutil uses a FileCollectorFileSystem
which keeps track of all the files used by dsymutil. These files are
copied into a temporary directory when dsymutil exists.
When this path is passed to --use-reproducer, dsymutil uses a
RedirectingFileSystem that will use the files from the reproducer
directory instead of the actual paths. This means you don't need to mess
with the OSO path prefix.
Differential revision: https://reviews.llvm.org/D79398
This patch adds statistics about the contribution of each object file to
the linked debug info. When --statistics is passed to dsymutil, it
prints a table after linking as illustrated below.
It lists the object file name, the size of the debug info in the object
file in bytes, and the absolute size contribution to the linked dSYM and
the percentage difference. The table is sorted by the output size, so
the object files contributing the most to the link are listed first.
.debug_info section size (in bytes)
-------------------------------------------------------------------------------
Filename Object dSYM Change
-------------------------------------------------------------------------------
basic2.macho.x86_64.o 210b 165b -24.00%
basic3.macho.x86_64.o 177b 150b -16.51%
basic1.macho.x86_64.o 125b 129b 3.15%
-------------------------------------------------------------------------------
Total 512b 444b -14.23%
-------------------------------------------------------------------------------
Differential revision: https://reviews.llvm.org/D79513
This patch threads the virtual file system through dsymutil.
Currently there is no good way to find out exactly what files are
necessary in order to reproduce a dsymutil link, at least not without
knowledge of how dsymutil's internals. My motivation for this change is
to add lightweight "reproducers" that automatically gather the input
object files through the FileCollectorFileSystem. The files together
with the YAML mapping will allow us to transparently reproduce a
dsymutil link, even without having to mess with the OSO path prefix.
Differential revision: https://reviews.llvm.org/D79376
As reported here: https://reviews.llvm.org/D75153#1987272
Before, each instance of llvm-cov was creating one thread per hardware core, which wasn't needed probably because the number of inputs were small. This was probably causing a thread rlimit issue on large core count systems.
After this patch, the previous behavior is restored (to what was before rG8404aeb5):
If --num-threads is not specified, we create one thread per input, up to num.cores.
When specified, --num-threads indicates any number of threads, with no upper limit.
Differential Revision: https://reviews.llvm.org/D78408
to remap object file paths (but no source paths) before
processing. This is meant to be used for Clang objects where the
module cache location was remapped using ``-fdebug-prefix-map``; to
help dsymutil find the Clang module cache.
<rdar://problem/55685132>
Differential Revision: https://reviews.llvm.org/D76391
The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.
== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.
By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.
This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.
== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".
== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).
When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.
When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.
Differential Revision: https://reviews.llvm.org/D71775
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
This adds support to dsymutil for linking remark files and placing them
in the final .dSYM bundle.
The result will be placed in:
* a.out.dSYM/Contents/Resources/Remarks/a.out
or
* a.out.dSYM/Contents/Resources/Remarks/a.out-<arch> for universal binaries
When multi-threaded, this runs a third thread which loops over all the
object files and parses remarks as it finds __remarks sections.
Testing this involves running dsymutil on pre-built binaries and object
files, then running llvm-bcanalyzer on the final result to check for
remarks.
Differential Revision: https://reviews.llvm.org/D69142
After changing dsymutil to use libOption, we lost error reporting for
missing required arguments (input files). Additionally, we stopped
complaining about unknown arguments. This patch fixes both and adds a
test.
llvm-svn: 375044
The lambda is taking the stack-allocated Verify boolean by reference and
it would go out of scope on the next iteration. Moving it out of the
loop should fix the issue.
Fixes https://bugs.llvm.org/show_bug.cgi?id=43549
llvm-svn: 373683
The dsymutil implementation file has a using-directive for the llvm
namespace. This patch just removes redundant namespace qualifiers.
llvm-svn: 373623
This patch reimplements command line option parsing in dsymutil with
Tablegen and libOption. The main motivation for this change is to
prevent clashes with other cl::opt options defined in llvm. Although
it's a bit more heavyweight, it has some nice advantages such as no
global static initializers and better separation between the code and
the option definitions.
I also used this opportunity to improve how dsymutil deals with
incompatible options. Instead of having checks spread across the code,
everything is now grouped together in verifyOptions. The fact that the
options are no longer global means that we need to pass them around a
bit more, but I think it's worth the trade-off.
Differential revision: https://reviews.llvm.org/D68361
llvm-svn: 373622
In r367348, I changed dsymutil to pass the LinkOptions by value isntead
of by const reference. However, the options were still captured by
reference in the LinkLambda. This patch fixes that by passing them in by
value.
llvm-svn: 367635
Before this patch we used either a single thread, or the number of
hardware threads available, effectively ignoring the number of threads
specified on the command line.
llvm-svn: 362815
When a Swift module built with debug info imports a library without
debug info from a textual interface, the textual interface is
necessary to reconstruct types defined in the library's interface. By
recording the Swift interface files in DWARF dsymutil can collect them
and LLDB can find them.
This patch teaches dsymutil to look for DW_TAG_imported_modules and
records all references to parseable Swift ingterfrace files and copies
them to
a.out.dSYM/Contents/Resources/<Arch>/<ModuleName>.swiftinterface
<rdar://problem/49751748>
llvm-svn: 358921
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
The unobufscation support for BCSymbolMaps was the last piece of code
that hasn't been upstreamed yet. This patch contains a reworked version
of the existing code and relevant tests.
Differential revision: https://reviews.llvm.org/D56346
llvm-svn: 350580
Before this patch we were returning an empty string in case we couldn't
create the output file. Now we return an expected string so we can
return and print the proper issue. We now return errors instead of bools
and defer printing to the call site.
llvm-svn: 344983
Dsymutil's update functionality was broken on Windows because we tried
to rename a file while we're holding open handles to that file. TempFile
provides a solution for this through its keep(Twine) method. This patch
changes dsymutil to make use of that functionality.
Differential revision: https://reviews.llvm.org/D49860
llvm-svn: 338216
This patch add support for emitting DWARF5 accelerator tables
(.debug_names) from dsymutil. Just as with the Apple style accelerator
tables, it's possible to update existing dSYMs. This patch includes a
test that show how you can convert back and forth between the two types.
If no kind of table is specified, dsymutil will default to generating
Apple-style accelerator tables whenever it finds those in its input. The
same is true when there are no accelerator tables at all. Finally, in
the remaining case, where there's at least one DWARF v5 table and no
Apple ones, the output will contains a DWARF accelerator tables
(.debug_names).
Differential revision: https://reviews.llvm.org/D49137
llvm-svn: 337980
When implementing the DWARF accelerator tables in dsymutil I ran into an
assertion in the assembler. Debugging these kind of issues is a lot
easier when looking at the assembly instead of debugging the assembler
itself. Since it's only a matter of creating an AsmStreamer instead of a
MCObjectStreamer it made sense to turn this into a (hidden) dsymutil
feature.
Differential revision: https://reviews.llvm.org/D49079
llvm-svn: 336561
The original binary holder has an optimization where it caches a static
library (archive) between consecutive calls to GetObjects. However, the
actual memory buffer wasn't cached between calls.
This made sense when dsymutil was processing objects one after each
other, but when processing them in parallel, several binaries have to be
in memory at the same time. For this reason, every link context
contained a binary holder.
Having one binary holder per context is problematic, because the same
static archive was cached for every object file. Luckily, when the file
is mmap'ed, this was only costing us virtual memory.
This patch introduces a new BinaryHolder variant that is fully cached,
for all the object files it load, as well as the static archives. This
way, we don't have to give up on this optimization of bypassing the
file system.
Differential revision: https://reviews.llvm.org/D48501
llvm-svn: 335990
This patch splits off some abstractions used by dsymutil's dwarf linker
and moves them into separate header and implementation files. This
almost halves the number of LOC in DwarfLinker.cpp and makes it a lot
easier to understand what functionality lives where.
Differential revision: https://reviews.llvm.org/D48647
llvm-svn: 335749
As noted by Adrian on llvm-commits, PrintHTMLEscaped and PrintEscaped in
StringExtras did not conform to the LLVM coding guidelines. This commit
rectifies that.
llvm-svn: 333669
When printing string in the Plist, we weren't escaping the characters
which lead to invalid XML. This patch adds the escape logic to
StringExtras.
rdar://39785334
llvm-svn: 333565
This commit adds a color category so tools can document this option and
enables it for dwarfdump and dsymuttil.
rdar://problem/40498996
llvm-svn: 333176
Create convenience functions for printing error, warning and note to
stdout. Previously we had similar functions being used in dsymutil, but
given that this pattern is so common it makes sense to make it available
globally.
llvm-svn: 330091
We have a few functions that virtually all command wants to run on
process startup/shutdown. This patch adds InitLLVM class to do that
all at once, so that we don't need to copy-n-paste boilerplate code
to each llvm command's main() function.
Differential Revision: https://reviews.llvm.org/D45602
llvm-svn: 330046
The DwarfLinker can have some very deep recursion that can max out the
(significantly smaller) stack when using threads. We don't want this
limitation when we only have a single thread. We already have this
workaround for the architecture-related threading. This patch applies
the same workaround to the parallel analysis and cloning.
Differential revision: https://reviews.llvm.org/D45172
llvm-svn: 329093
When running dsymutil as part of your build system, it can be desirable
for warnings to be part of the end product, rather than just being
emitted to the output stream. This patch upstreams that functionality.
Differential revision: https://reviews.llvm.org/D44639
llvm-svn: 328965
This is a follow-up to r327137 where we unified error handling for the
DwarfLinker. This replaces calls to errs() and outs() with the
appropriate ostream wrapper everywhere in dsymutil.
llvm-svn: 327411
When processing a dSYM bundle, use llvm::sys::path to join the different
path components instead of using a string with hard coded forward
slashes as separators.
llvm-svn: 324622
This commit attempts to re-land the r324480 which was reverted in
r324493 because it broke the Windows bots. For now I disabled the two
update tests on Windows until I'm able to debug this.
Differential revision: https://reviews.llvm.org/D42880
llvm-svn: 324592
Now that dsymutil can generate accelerator tables, we can upstream the
update logic that, as the name implies, updates the accelerator tables
in an existing dSYM bundle. In combination with `-minimize` this can be
used to remove redundant .debug_(inlines|pubtypes|pubnames).
Differential revision: https://reviews.llvm.org/D42880
llvm-svn: 324480
Passing -minimize to dsymutil prevents the emission of .debug_inlines,
.debug_pubnames, and .debug_pubtypes in favor of the Apple accelerator
tables.
The actual check in the DWARF linker was added in r323655. This patch
simply enables it.
Differential revision: https://reviews.llvm.org/D42688
llvm-svn: 323812
