This re-lands e5553b9a6ab9 with two small fixes to the tests:
- Don't touch the source directory in debug-map-parsing.test but
instead copy everything over in a temporary directory in
timestamp-mismatch.test.
- Don't redirect stderr to stdout to avoid the output getting
intertwined in extern-alias.test.
This reverts commit e5553b9a6ab9f02f382a31cc5117b52c3bfaf77a.
Tests are not allowed to modify the source. Please figure out a way to
use %t rather than dynamically modifying the inputs.
Currently dsymutil will silently fail when processing binaries with
Dwarf 5 debug info. This patch adds rudimentary support for Dwarf 5 in
dsymutil.
- Recognize relocations in the debug_addr section.
- Recognize (a subset of) Dwarf 5 form values.
- Emits valid Dwarf 5 compile unit header chains.
To simplify things (and avoid having to emit indexed sections) I decided
to emit the relocated addresses directly in the debug info section.
- DW_FORM_strx gets relocated and rewritten to DW_FORM_strp
- DW_FORM_addrx gets relocated and rewritten to DW_FORM_addr
Obviously there's a lot of work left, but this should be a step in the
right direction.
rdar://62345491
Differential revision: https://reviews.llvm.org/D94323
Copy over the __eh_frame from the binary into the dSYM. This helps
kernel developers that are working with only dSYMs (i.e. no binaries)
when debugging a core file. This only kicks in when the __eh_frame
exists in the linked binary. Most of the time ld64 will remove the
section in favor of compact unwind info. When it is emitted, it's
generally small enough and should not bloat the dSYM.
rdar://69774935
Differential revision: https://reviews.llvm.org/D94460
Add a warning when the timestmap doesn't match between the object file
and the debug map entry. We were already emitting such warnings for
archive members and swift interface files. This patch also unifies the
warning across all three.
rdar://65614640
Differential revision: https://reviews.llvm.org/D94536
Current interface of AddressMap assumes that relocations exist.
That is correct for not-linked object file but is not correct
for linked executable. This patch changes interface in such way
that AddressMap could be used not only with not-linked object files:
hasValidRelocationAt()
replaced with:
hasLiveMemoryLocation()
hasLiveAddressRange()
Differential Revision: https://reviews.llvm.org/D87723
In f9fb9da36c34d2745b74dc30e6c26f7d3f48466c we fixed spurious warnings
caused by aliases to private extern symbols. This patch generalizes that
to regular external symbols as well.
No longer rely on an external tool to build the llvm component layout.
Instead, leverage the existing `add_llvm_componentlibrary` cmake function and
introduce `add_llvm_component_group` to accurately describe component behavior.
These function store extra properties in the created targets. These properties
are processed once all components are defined to resolve library dependencies
and produce the header expected by llvm-config.
Differential Revision: https://reviews.llvm.org/D90848
Make these types conform to the LLVM Coding Standards:
> Type names (including classes, structs, enums, typedefs, etc) should
> be nouns and start with an upper-case letter.
dsymutil was incorrectly ignoring aliases to private extern symbols in
the MachODebugMapParser. This resulted in spurious warnings about not
being able to find symbols.
rdar://49652389
Differential revision: https://reviews.llvm.org/D89444
Rename the DwarfFile class in DWARFLinker to DWARFFile. This is
consistent with the other DWARF classes and avoids a ODR violation with
the DwarfFile class in AsmPrinter.
- Fix a memory leak accidentally introduced yesterday by using CodeGen's
existing mangling context instead of creating a new context afresh.
- Move GNU-runtime ObjC method mangling into the AST mangler; this will
eventually be necessary to support direct methods there, but is also
just the right architecture.
- Make the Apple-runtime method mangling work properly when given an
interface declaration, fixing a bug (which had solidified into a test)
where mangling a category method from the interface could cause it to
be mangled as if the category name was a class name. (Category names
are namespaced within their class and have no global meaning.)
- Fix a code cross-reference in dsymutil.
Based on a patch by Ellis Hoag.
Treat N_AST symbol table entries like other debug entries and don't emit
them in the linked binary.
Differential revision: https://reviews.llvm.org/D81205
Reproducer.cpp:70:12: error: could not convert ‘Repro’ from
‘std::unique_ptr<llvm::dsymutil::ReproducerGenerate,
std::default_delete<llvm::dsymutil::ReproducerGenerate> >’ to
‘llvm::Expected<std::unique_ptr<llvm::dsymutil::Reproducer> >’
Add support for generating a dsymutil reproducer. The result is a folder
containing all the object files for linking.
When --gen-reproducer is passed, dsymutil uses a FileCollectorFileSystem
which keeps track of all the files used by dsymutil. These files are
copied into a temporary directory when dsymutil exists.
When this path is passed to --use-reproducer, dsymutil uses a
RedirectingFileSystem that will use the files from the reproducer
directory instead of the actual paths. This means you don't need to mess
with the OSO path prefix.
Differential revision: https://reviews.llvm.org/D79398
This patch adds statistics about the contribution of each object file to
the linked debug info. When --statistics is passed to dsymutil, it
prints a table after linking as illustrated below.
It lists the object file name, the size of the debug info in the object
file in bytes, and the absolute size contribution to the linked dSYM and
the percentage difference. The table is sorted by the output size, so
the object files contributing the most to the link are listed first.
.debug_info section size (in bytes)
-------------------------------------------------------------------------------
Filename Object dSYM Change
-------------------------------------------------------------------------------
basic2.macho.x86_64.o 210b 165b -24.00%
basic3.macho.x86_64.o 177b 150b -16.51%
basic1.macho.x86_64.o 125b 129b 3.15%
-------------------------------------------------------------------------------
Total 512b 444b -14.23%
-------------------------------------------------------------------------------
Differential revision: https://reviews.llvm.org/D79513
This patch threads the virtual file system through dsymutil.
Currently there is no good way to find out exactly what files are
necessary in order to reproduce a dsymutil link, at least not without
knowledge of how dsymutil's internals. My motivation for this change is
to add lightweight "reproducers" that automatically gather the input
object files through the FileCollectorFileSystem. The files together
with the YAML mapping will allow us to transparently reproduce a
dsymutil link, even without having to mess with the OSO path prefix.
Differential revision: https://reviews.llvm.org/D79376
Summary:
In D77860, we have changed `getSymbolFlags()` return type to `Expected<uint32_t>`.
This change helps bubble the error further up the stack.
Reviewers: jhenderson, grimar, JDevlieghere, MaskRay
Reviewed By: jhenderson
Subscribers: hiraditya, MaskRay, rupprecht, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D79075
As reported here: https://reviews.llvm.org/D75153#1987272
Before, each instance of llvm-cov was creating one thread per hardware core, which wasn't needed probably because the number of inputs were small. This was probably causing a thread rlimit issue on large core count systems.
After this patch, the previous behavior is restored (to what was before rG8404aeb5):
If --num-threads is not specified, we create one thread per input, up to num.cores.
When specified, --num-threads indicates any number of threads, with no upper limit.
Differential Revision: https://reviews.llvm.org/D78408
For implementing "remove obsolete debug info in lld", it is neccesary
to have DWARF generation code implementation. dsymutil uses DwarfStreamer
for that purpose. DwarfStreamer uses AsmPrinter. It is considered OK
to use AsmPrinter based code in lld(D74169). This patch moves
DwarfStreamer implementation into DWARFLinker, so that it could be reused
from lld.
Generally, a better place for such a common DWARF generation code would be
not DWARFLinker but an additional separate library. Such a library could
contain a single version of DWARF generation routines and could also
be independent of AsmPrinter. At the current moment, DwarfStreamer
does not pretend to be such a general implementation of DWARF generation.
So I decided to put it into DWARFLinker since it is the only user
of DwarfStreamer.
Testing: it passes "check-all" lit testing. MD5 checksum for clang .dSYM
bundle matches for the dsymutil with/without that patch.
Reviewed By: JDevlieghere
Differential revision: https://reviews.llvm.org/D77169
to remap object file paths (but no source paths) before
processing. This is meant to be used for Clang objects where the
module cache location was remapped using ``-fdebug-prefix-map``; to
help dsymutil find the Clang module cache.
<rdar://problem/55685132>
Differential Revision: https://reviews.llvm.org/D76391
MCTargetOptionsCommandFlags.inc and CommandFlags.inc are headers which contain
cl::opt with static storage.
These headers are meant to be incuded by tools to make it easier to parametrize
codegen/mc.
However, these headers are also included in at least two libraries: lldCommon
and handle-llvm. As a result, when creating DYLIB, clang-cpp holds a reference
to the options, and lldCommon holds another reference. Linking the two in a
single executable, as zig does[0], results in a double registration.
This patch explores an other approach: the .inc files are moved to regular
files, and the registration happens on-demand through static declaration of
options in the constructor of a static object.
[0] https://bugzilla.redhat.com/show_bug.cgi?id=1756977#c5
Differential Revision: https://reviews.llvm.org/D75579
Summary: The wrong variable was being checked for an error, which mean a llvm::Error went unchecked and crashes dsymutil. Discovered this when trying to feed an ELF file to "dsymutil --update" and running into the crash.
Reviewers: aprantl, JDevlieghere
Subscribers: llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D75777
Summary:
DWARFContext has all the required information to access source debug info.
It is not necessary to use "const object::ObjectFile" to create DWARFContext.
Thus this patch removes all usages of "const object::ObjectFile"
from DWARFLinker. Instead, already created DWARFContext is passed
to DWARFLinker. The purpose is to not depend on "const object::ObjectFile".
The patch looks big, but most of changes are renamings and movements.
Testing: it passes "check-all" lit testing. MD5 checksum for clang .dSYM bundle
matches for the dsymutil with/without that patch.
Reviewers: JDevlieghere, friss, dblaikie, aprantl
Reviewed By: JDevlieghere
Subscribers: hiraditya, llvm-commits
Tags: #llvm, #debug-info
Differential Revision: https://reviews.llvm.org/D75029
The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.
== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.
By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.
This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.
== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".
== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).
When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.
When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.
Differential Revision: https://reviews.llvm.org/D71775
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Summary:
The primary goal of this refactoring is to separate DWARF optimizing part.
So that it could be reused by linker or by any other client.
There was a thread on llvm-dev discussing the necessity of such a refactoring:
http://lists.llvm.org/pipermail/llvm-dev/2019-September/135068.html.
This is a final part from series of patches for dsymutil.
Previous patches : D71068, D71839, D72476. This patch:
1. Creates lib/DWARFLinker interface :
void addObjectFile(DwarfLinkerObjFile &ObjFile);
bool link();
void setOptions;
1. Moves all linking logic from tools/dsymutil/DwarfLinkerForBinary
into lib/DWARFLinker.
2. Renames RelocationManager into AddressesManager.
3. Remarks creation logic moved from separate parallel execution
into object file loading routine.
Testing: it passes "check-all" lit testing. MD5 checksum for clang .dSYM bundle
matches for the dsymutil with/without that patch.
Reviewers: JDevlieghere, friss, dblaikie, aprantl, jdoerfert
Reviewed By: JDevlieghere
Subscribers: merge_guards_bot, hiraditya, jfb, llvm-commits, probinson, thegameg
Tags: #llvm, #debug-info
Differential Revision: https://reviews.llvm.org/D72915
