Summary:
Several changes were required for ThinLTO links involving bitcode
archive static libraries. With this patch clang/llvm bootstraps with
ThinLTO and gold.
The first is that the gold callbacks get_input_file and
release_input_file can normally be used to get file information for
each constituent bitcode file within an archive. However, these
interfaces lock the underlying file and can't be for each archive
constituent for ThinLTO backends where we get all the input files up
front and don't release any until after the backend threads complete.
However, it is sufficient to only get and release once per file, and
then each consituent bitcode file can be accessed via get_view. This
required saving some information to identify which file handle is the
"leader" for each claimed file sharing the same file descriptor, and
other information so that get_input_file isn't necessary later when
processing the backends.
Second, the module paths in the index need to distinguish between
different constituent bitcode files within the same archive file,
otherwise they will all end up with the same archive file path.
Do this by appending the offset within the archive for the start of the
bitcode file, returned by get_input_file when we claim each bitcode file,
and saving that along with the file handle.
Third, rather than have the function importer try to load a file based
on the module path identifier (which now contains a suffix to
distinguish different bitcode files within an archive), use a custom
module loader. This is the same approach taken in libLTO, and I am using
the support refactored into the new LTO.h header in r270509. The module
loader parses the bitcode files out of the memory buffers returned from
gold via the get_view callback and saved in a map. This also means that
we call the function importer directly, rather than add it to the pass
pipeline (which was in the plan to do already for other reasons).
Reviewers: pcc, joker.eph
Subscribers: llvm-commits, joker.eph
Differential Revision: http://reviews.llvm.org/D20559
llvm-svn: 270814
This is a support COFF feature. Ensure that we can display the weak externals
auxiliary symbol. It contains useful information (such as the default binding
and how to resolve the symbol).
This reapplies the previous patch with a modification which hopefully should fix
the endianness issues. The variadic call would promote the ulittle32_t to a
uint32_t which would lose the byte-swapping behaviour desired.
llvm-svn: 270813
Richard Smith identified this in post commit review of r270466. The
string sections in particular (in the future, possibly all sections - so
I'm not going to bother pulling out just the string sections for the
extra lifetime handling right now) need to remain valid during
processing of all inputs so that elements of the DWPStringPool can be
looked up repeatedly without having to make in-memory copies of string
contents in the noncompressed case (more common in dwp+dwp merge steps
where the memory is a bigger problem because the files are larger).
Using the SmallVector (or any vector) a reallocation on push_back could
cause any of the nested SmallStrings in small mode to move in memory and
invalid pointers to their contents. Using a deque the SmallStrings will
never move around since no elements are removed from the container.
llvm-svn: 270797
We have need to reuse this functionality, including making
additional generic stream types that are smarter about how and
when they copy memory versus referencing the original memory.
So all of these structures belong in the common library
rather than being pdb specific.
llvm-svn: 270751
This should actually address PR27855. This results in adding references to the system libs inside generated dylibs so that they get correctly pulled in when linking against the dylib.
llvm-svn: 270723
searching for external symbols, and fall back to the SymbolResolver::findSymbol
method if the former returns null.
This makes RuntimeDyld behave more like a static linker: Symbol definitions
from within the current module's "logical dylib" will be preferred to
external definitions. We can build on this behavior in the future to properly
support weak symbol handling.
Custom symbol resolvers that override the findSymbolInLogicalDylib method may
notice changes due to this patch. Clients who have not overridden this method
should generally be unaffected, however users of the OrcMCJITReplacement class
may notice changes.
llvm-svn: 270716
This is a support COFF feature. Ensure that we can display the weak externals
auxiliary symbol. It contains useful information (such as the default binding
and how to resolve the symbol).
llvm-svn: 270648
This adds support for parsing and dumping the following
symbol types:
S_LPROCREF
S_ENVBLOCK
S_COMPILE2
S_REGISTER
S_COFFGROUP
S_SECTION
S_THUNK32
S_TRAMPOLINE
As of this patch, the test PDB files no longer have any unknown
symbol types.
llvm-svn: 270628
When dumping huge PDB files, too many of the options were grouped
together so you would get neverending spew of output. This patch
introduces more granular display options so you can only dump the
fields you actually care about.
llvm-svn: 270607
This makes use of the newly introduced `CVSymbolVisitor` to dump details
of each type of symbol record in the symbol streams. Future patches will
bring this visitor based dumping to the publics stream, as well as
creating a `SymbolDumpDelegate` to print more information about
relocations etc.
Differential Revision: http://reviews.llvm.org/D20545
Reviewed By: ruiu
llvm-svn: 270585
to llvm-objdump. This section is created with -fembed-bitcode option.
This requires the use of libxar and the Cmake and lit support were crafted by
Chris Bieneman!
rdar://26202242
llvm-svn: 270491
This will pave the way to introduce a full fledged symbol visitor
similar to how we have a type visitor, thus allowing the same
dumping code to be used in llvm-readobj and llvm-pdbdump.
Differential Revision: http://reviews.llvm.org/D20384
Reviewed By: rnk
llvm-svn: 270475
Now that the string pool is referential rather than maintaining its own
copy of string data, compressed sections (well, technically only the
debug_str section*) need to be preserved for the lifetime of the pool to
match.
* I'm not currently optimizing for memory footprint with compressed
input - the major memory limit I'm hitting is on dwp+dwp merge steps
and we aren't currently compressing contents in dwp files, just in the
.dwo inputs.
llvm-svn: 270462
Main problem here was that SHF_COMPRESSED has the same value with
XCORE_SHF_CP_SECTION, which was included as standart (common) flag.
As far I understand xCore is a family of controllers and it that
means it's constant should be processed separately,
only if e_machine == EM_XCORE, otherwise llvm-readobj would output
different constants twice for compressed section:
Flags [
..
SHF_COMPRESSED (0x800)
..
XCORE_SHF_CP_SECTION (0x800)
..
]
what probably does not make sence if you're not working with xcore file.
Differential revision: http://reviews.llvm.org/D20273
llvm-svn: 270320
This fills section data with 0xDEADBEEF and segment data not inside a section with 0xBAADDA7A. This results in yaml2obj generating a matching size object file. Any additional bytes in the file are zero'd.
This is a starting point for populating the remaining segment data, and provides a hex viewable file that you can easily see the missing data in.
llvm-svn: 270286
In addition to clarifying the warning message this contains a minor functional
change in that it now warns if the *immediate* parent directory in which the
missing PCM is expected to be isn't found.
This patch also includes a more comprehensive testcase.
rdar://problem/25860711
llvm-svn: 270269
DBI stream contains a stream number of the symbol record stream.
Symbol record streams is an array of length-type-value members.
Each member represents one symbol.
Publics stream contains offsets to the symbol record stream.
This patch is to print out all symbols that are referenced by
the publics stream.
Note that even with this patch, llvm-pdbdump cannot dump all the
information in a publics stream since it contains more information
than symbol names. I'll improve it in followup patches.
Differential Revision: http://reviews.llvm.org/D20480
llvm-svn: 270262
Now that MachO load command fields are fully covered we can fill unaccounted for bytes with 0. That allows us to sparsely specify YAML to simplify tests.
Simplifying load_commands test accordingly.
llvm-svn: 270158
This removes the subclasses of ProfileSummary, moves the members of the derived classes to the base class.
Differential Revision: http://reviews.llvm.org/D20390
llvm-svn: 270143
This splits ProfileSummary into two classes: a ProfileSummary class that has methods to convert from/to metadata and a ProfileSummaryBuilder class that computes the profiles summary which is in ProfileData.
Differential Revision: http://reviews.llvm.org/D20314
llvm-svn: 270136
This re-applies r270115.
Many of the MachO load commands can have data appended after the command structure. This data is frequently strings, but can actually be anything. This patch adds support for three optional fields on load command yaml descriptions.
The new PayloadString YAML field is populated with the data after load commands known to have strings as extra data.
The new ZeroPadBytes YAML field is a count of zero'd bytes after the end of the load command structure before the next command. This can apply anywhere in the file. MachO2YAML verifies that bytes are zero before populating this field, and YAML2MachO will add zero'd bytes.
The new PayloadBytes YAML field stores all bytes after the end of the load command structure before the next command if they are non-zero. This is a catch all for all unhandled bytes. If MachO2Yaml populates PayloadBytes it will not populate ZeroPadBytes, instead zero'd bytes will be in the PayloadBytes structure.
llvm-svn: 270124
Many of the MachO load commands can have data appended after the command structure. This data is frequently strings, but can actually be anything. This patch adds support for three optional fields on load command yaml descriptions.
The new PayloadString YAML field is populated with the data after load commands known to have strings as extra data.
The new ZeroPadBytes YAML field is a count of zero'd bytes after the end of the load command structure before the next command. This can apply anywhere in the file. MachO2YAML verifies that bytes are zero before populating this field, and YAML2MachO will add zero'd bytes.
The new PayloadBytes YAML field stores all bytes after the end of the load command structure before the next command if they are non-zero. This is a catch all for all unhandled bytes. If MachO2Yaml populates PayloadBytes it will not populate ZeroPadBytes, instead zero'd bytes will be in the PayloadBytes structure.
llvm-svn: 270115
Transition InstrProf and Coverage over to the stricter Error/Expected
interface.
Changes since the initial commit:
- Fix error message printing in llvm-profdata.
- Check errors in loadTestingFormat() + annotateAllFunctions().
- Defer error handling in InstrProfIterator to InstrProfReader.
- Remove the base ProfError class to work around an MSVC ICE.
Differential Revision: http://reviews.llvm.org/D19901
llvm-svn: 270020
This refactoring is to reduce code duplication between the 32-bit and 64-bit code paths. This refactoring will also make the special casing for other data after load commands cleaner.
llvm-svn: 270001
Having an enum member named Default is quite confusing: Is it distinct
from the others?
This patch removes that member and instead uses Optional<Reloc> in
places where we have a user input that still hasn't been maped to the
default value, which is now clear has no be one of the remaining 3
options.
llvm-svn: 269988
MC only needs to know if the output is PIC or not. It never has to
decide about creating GOTs and PLTs for example. The only thing that
MC itself uses this information for is expanding "macros" in sparc and
mips. The rest I am pretty sure could be moved to CodeGen.
This is a cleanup and isolates the code from future changes to
Reloc::Model.
llvm-svn: 269909
Coverage mapping data is organized in a sequence of blocks, each of which is expected
to be aligned by 8 bytes. This feature is used when reading those blocks, see
VersionedCovMapFuncRecordReader::readFunctionRecords(). If a misaligned covearge
mapping data has more than one block, it causes llvm-cov to fail.
Differential Revision: http://reviews.llvm.org/D20285
llvm-svn: 269887
* Reworks the CVSymbolTypes.def to work similarly to TypeRecords.def.
* Moves some enums from SymbolRecords.h to CodeView.h to maintain
consistency with how we do type records.
* Generalize a few simple things like the record prefix
* Define the leaf enum and the kind enum similar to how we do with tyep
records.
Differential Revision: http://reviews.llvm.org/D20342
Reviewed By: amccarth, rnk
llvm-svn: 269867
I don't yet fully understand the meaning of these data strcutures,
but at least it seems that their sizes and types are correct.
With this change, we can read publics streams till end.
Differential Revision: http://reviews.llvm.org/D20343
llvm-svn: 269861
This adds support for all the MachO *_command structures. The load_command payloads still are not represented, but that will come next.
llvm-svn: 269808
Summary:
Colons can appear in Windows paths after drive letters. Both colon and
semicolon are valid characters in filenames, but neither are very
common. Semicolon seems just as good, and makes the test pass on
Windows.
Reviewers: tejohnson
Subscribers: joker.eph, llvm-commits
Differential Revision: http://reviews.llvm.org/D20332
llvm-svn: 269798
when the object is in an archive to use something like libx.a(foo.o) as part of
the error message.
Also changed llvm-objdump and llvm-size to be like llvm-nm and ignore non-object
files in archives and not produce any error message.
To do this Archive::Child::getAsBinary() was changed from ErrorOr<...> to
Expected<...> then that was threaded up to its users.
Converting this interface to Expected<> from ErrorOr<> does involve
touching a number of places. To contain the changes for now the use of
errorToErrorCode() is still used in one place yet to be fully converted.
Again there some were bugs in the existing code that did not deal with the
old ErrorOr<> return values. So now with Expected<> since they must be
checked and the error handled, I added a TODO and a comments for those.
llvm-svn: 269784
This adds support for all the MachO *_command structures. The load_command payloads still are not represented, but that will come next.
llvm-svn: 269782
Summary:
Add support to control where files for a distributed backend (the
individual index files and optional imports files) are created.
This is invoked with a new thinlto-prefix-replace option in the gold
plugin and llvm-lto. If specified, expects a string of the form
"oldprefix:newprefix", and instead of generating these files in the
same directory path as the corresponding bitcode file, will use a path
formed by replacing the bitcode file's path prefix matching oldprefix
with newprefix.
Also add a new replace_path_prefix helper to Path.h in libSupport.
Depends on D19636.
Reviewers: joker.eph
Subscribers: llvm-commits, joker.eph
Differential Revision: http://reviews.llvm.org/D19644
llvm-svn: 269771
This diagnostic could be improved by adding the name of the input file
containing the invalid data and/or some information about how to
identify the specific offending attribute/tag in the input. But that's
not an immediate priority as these corner cases of invalid input
shouldn't come up too often.
llvm-svn: 269727
The diagnostic could be improved a bit to include information about
which input file had the mistake (& which unit (counted, since the name
of the unit won't be accessible) within the input).
llvm-svn: 269723
Transition InstrProf and Coverage over to the stricter Error/Expected
interface.
Changes since the initial commit:
- Address undefined-var-template warning.
- Fix error message printing in llvm-profdata.
- Check errors in loadTestingFormat() + annotateAllFunctions().
- Defer error handling in InstrProfIterator to InstrProfReader.
Differential Revision: http://reviews.llvm.org/D19901
llvm-svn: 269694
Actually use the error return path rather than printing the duplicate
information then a separate error. But also just tidy up/deduplicate
some of the code for generating the diagnostic text.
llvm-svn: 269692
Without a diagnostic handler installed, llc's behaviour is to exit on the first
error that it encounters. This is very different from the behaviour of clang
and other front ends, which try to gather as many errors as possible before
exiting.
This commit adds a diagnostic handler to llc, allowing it to find and report
more than one error. The old behaviour is preserved under a flag (-exit-on-error).
Some of the tests fail with the new diagnostic handler, so they have to use the
new flag in order to run under the previous behaviour. Some of these are known
bugs, others need further investigation. Ideally, we should fix the tests and
remove the flag at some point in the future.
Reapplied after fixing the LLDB build that was broken due to the new
DiagnosticSeverity in LLVMContext.h, and fixed an UB in the new change.
Patch by Diana Picus.
llvm-svn: 269655
Without a diagnostic handler installed, llc's behaviour is to exit on the first
error that it encounters. This is very different from the behaviour of clang
and other front ends, which try to gather as many errors as possible before
exiting.
This commit adds a diagnostic handler to llc, allowing it to find and report
more than one error. The old behaviour is preserved under a flag (-exit-on-error).
Some of the tests fail with the new diagnostic handler, so they have to use the
new flag in order to run under the previous behaviour. Some of these are known
bugs, others need further investigation. Ideally, we should fix the tests and
remove the flag at some point in the future.
Reapplied after fixing the LLDB build that was broken due to the new
DiagnosticSeverity in LLVMContext.h.
Patch by Diana Picus.
llvm-svn: 269563
This reverts commit r269538 and r269542.
"rename()" is expected to fail across filesystems, will handle this.
From: Mehdi Amini <mehdi.amini@apple.com>
llvm-svn: 269543
Summary:
This code is intended to be used as part of LLD's PDB writing. Until
that exists, this is exposed via llvm-readobj for testing purposes.
Type stream merging uses the following algorithm:
- Begin with a new empty stream, and a new empty hash table that maps
from type record contents to new type index.
- For each new type stream, maintain a map from source type index to
destination type index.
- For each record, copy it and rewrite its type indices to be valid in
the destination type stream.
- If the new type record is not already present in the destination
stream hash table, append it to the destination type stream, assign it
the next type index, and update the two hash tables.
- If the type record already exists in the destination stream, discard
it and update the type index map to forward the source type index to
the existing destination type index.
Reviewers: zturner, ruiu
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D20122
llvm-svn: 269521
Transition InstrProf and Coverage over to the stricter Error/Expected
interface.
Changes since the initial commit:
- Fix error message printing in llvm-profdata.
- Check errors in loadTestingFormat() + annotateAllFunctions().
- Defer error handling in InstrProfIterator to InstrProfReader.
Differential Revision: http://reviews.llvm.org/D19901
llvm-svn: 269491
Publics stream seems to contain information as to public symbols.
It actually contains a serialized hash table along with fixed-sized
headers. This patch is not complete. It scans only till the end of
the stream and dump the header information. I'll write code to
de-serialize the hash table later.
Reviewers: zturner
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D20256
llvm-svn: 269484
Transition InstrProf and Coverage over to the stricter Error/Expected
interface.
Differential Revision: http://reviews.llvm.org/D19901
llvm-svn: 269462
This patch adds basic support for MachO::load_command. Load command types and sizes are encoded in the YAML and expanded back into MachO.
The YAML doesn't yet support load command structs, that is coming next. In the meantime as a temporary measure when writing MachO files the load commands are padded with zeros so that the generated binary is valid.
llvm-svn: 269442
This reverts commit r269428, as it breaks the LLDB build. We need to
understand how to change LLDB in the same way as LLC before landing this
again.
llvm-svn: 269432
Without a diagnostic handler installed, llc's behaviour is to exit on the first
error that it encounters. This is very different from the behaviour of clang
and other front ends, which try to gather as many errors as possible before
exiting.
This commit adds a diagnostic handler to llc, allowing it to find and report
more than one error. The old behaviour is preserved under a flag (-exit-on-error).
Some of the tests fail with the new diagnostic handler, so they have to use the
new flag in order to run under the previous behaviour. Some of these are known
bugs, others need further investigation. Ideally, we should fix the tests and
remove the flag at some point in the future.
Patch by Diana Picus.
llvm-svn: 269428
This change is the gold side of the change made in D17115 and clang
patch r261045 to add a ThinLTO specific pipeline that moves more of
the optimization to the backends.
llvm-svn: 269386
In verbose mode, we emit a warning if the DWOId of a skeleton CU
mismatches the DWOId of the referenced module. This patch updates the
cached DWOId after a module has been loaded to the DWOId of the module
on disk (instead of storing the DWOId we expected to load). This
allows us to correctly emit the mismatch warning for all subsequent
object files that want to import the same module. This patch also
ensures both warnings are only emitted in verbose mode.
rdar://problem/26214027
llvm-svn: 269383
