Bitcode writer does not flush buffer until the end by default. This is
fine to small bitcode files. When -flto,--plugin-opt=emit-llvm,-gmlt are
used, the final bitcode file is large, for example, >8G. Keeping all
data in memory consumes a lot of memory.
This change allows bitcode writer flush data to disk early when buffered
data size is above some threshold. This is only enabled when lld emits
LLVM bitcode.
One issue to address is backpatching bitcode: subblock length, function
body indexes, meta data indexes need to backfill. If buffer can be
flushed partially, we introduced raw_fd_stream that supports
read/seek/write, and enables backpatching bitcode flushed in disk.
Reviewed-by: tejohnson, MaskRay
Differential Revision: https://reviews.llvm.org/D86905
This happens when generating a huge file by LTO, for example, with -gmlt.
When BitNo is > 2^35, ByteNo is overflowed, and an incorrect output offset is overwritten.
This generates ill-formed bitcodes.
Reviewed-by: tejohnson, vitalybuka
Differential Revision: https://reviews.llvm.org/D86645
Lots of headers pass around MemoryBuffer objects, but very few open
them. Let those that do include FileSystem.h.
Saves ~250 includes of Chrono.h & FileSystem.h:
$ diff -u thedeps-before.txt thedeps-after.txt | grep '^[-+] ' | sort | uniq -c | sort -nr
254 - ../llvm/include/llvm/Support/FileSystem.h
253 - ../llvm/include/llvm/Support/Chrono.h
237 - ../llvm/include/llvm/Support/NativeFormatting.h
237 - ../llvm/include/llvm/Support/FormatProviders.h
192 - ../llvm/include/llvm/ADT/StringSwitch.h
190 - ../llvm/include/llvm/Support/FormatVariadicDetails.h
...
This requires duplicating the file_t typedef, which is unfortunate. I
sunk the choice of mapping mode down into the cpp file using variable
template specializations instead of class members in headers.
The bitstream remark serializer landed in r367372.
This adds a bitstream remark parser that parser bitstream remark files
to llvm::remarks::Remark objects through the RemarkParser interface.
A few interesting things to point out:
* There are parsing helpers to parse the different types of blocks
* The main parsing helper allows us to parse remark metadata and open an
external file containing the encoded remarks
* This adds a dependency from the Remarks library to the BitstreamReader
library
* The testing strategy is to create a remark entry through YAML, parse
it, serialize it to bitstream, parse that back and compare the objects.
* There are close to no tests for malformed bitstream remarks, due to
the lack of textual format for the bitstream format.
* This adds a new C API for parsing bitstream remarks:
LLVMRemarkParserCreateBitstream.
* This bumps the REMARKS_API_VERSION to 1.
Differential Revision: https://reviews.llvm.org/D67134
llvm-svn: 371429
At the moment, bitcode files with invalid forward reference can easily
cause the bitcode reader to run out of memory, by creating a forward
reference with a very high index.
We can use the size of the bitcode file as an upper bound, because a
valid bitcode file can never contain more records. This should be
sufficient to fail early in most cases. The only exception is large
files with invalid forward references close to the file size.
There are a couple of clusterfuzz runs that fail with out-of-memory
because of very high forward references and they should be fixed by this
patch.
A concrete example for this is D64507, which causes out-of-memory on
systems with low memory, like the hexagon upstream bots.
Reviewers: t.p.northover, thegameg, jfb, efriedma, hfinkel
Reviewed By: jfb
Differential Revision: https://reviews.llvm.org/D64577
llvm-svn: 366017
Consolidate llvm::BWH_* statics into an enum to fix
module build issues. This fixes the LLVM_Bitcode module,
getting rid of -Wmodules-ambiguous-internal-linkage.
Differential Revision: https://reviews.llvm.org/D64469
llvm-svn: 365594
This moves Bitcode/Bitstream*, Bitcode/BitCodes.h to Bitstream/.
This is needed to avoid a circular dependency when using the bitstream
code for parsing optimization remarks.
Since Bitcode uses Core for the IR part:
libLLVMRemarks -> Bitcode -> Core
and Core uses libLLVMRemarks to generate remarks (see
IR/RemarkStreamer.cpp):
Core -> libLLVMRemarks
we need to separate the Bitstream and Bitcode part.
For clang-doc, it seems that it doesn't need the whole bitcode layer, so
I updated the CMake to only use the bitstream part.
Differential Revision: https://reviews.llvm.org/D63899
llvm-svn: 365091
