RPCS3/llvm-mirror
1
0
Fork 0
mirror of https://github.com/RPCS3/llvm-mirror.git synced 2024-10-19 11:02:59 +02:00
Code Issues Projects Releases Wiki Activity
171,960 Commits 4 Branches 5 Tags 704 MiB
cc3d32505c
Commit Graph

6876 Commits

Author SHA1 Message Date
Simon Pilgrim
391491b370 [TableGen] Remove unnecessary TypeSetByHwMode -> ValueTypeByHwMode -> TypeSetByHwMode conversions in getPatternSize 
I noticed this during profiling of tablegen (PR28222) that we were calling Child->getType(0) which creates a ValueTypeByHwMode on the fly from the requested internal TypeSetByHwMode type and returns it by value, we then treat it as a TypeSetByHwMode reference which involves constructing a new TypeSetByHwMode on the stack with a large amount of std::map iterating/copying all along the way.

I am not an expert on tablegen, but AFAICT this is all unnecessary and we should be calling Child->getExtType(0) which returns the original TypeSetByHwMode by reference.

This gives me a 90sec reduction in msvc debug builds of x86 -gen-dag-isel.

Differential Revision: https://reviews.llvm.org/D50789

llvm-svn: 339812
 2018-08-15 20:41:19 +00:00
Andrea Di Biagio
8add466d59 [Tablegen][MCInstPredicate] Removed redundant template argument from class TIIPredicate, and implemented verification rules for TIIPredicates. 
This patch removes redundant template argument `TargetName` from TIIPredicate.
Tablegen can always infer the target name from the context. So we don't need to
force users of TIIPredicate to always specify it.

This allows us to better modularize the tablegen class hierarchy for the
so-called "function predicates". class FunctionPredicateBase has been added; it
is currently used as a building block for TIIPredicates. However, I plan to
reuse that class to model other function predicate classes too (i.e. not just
TIIPredicates). For example, this can be a first step towards implementing
proper support for dependency breaking instructions in tablegen.

This patch also adds a verification step on TIIPredicates in tablegen.
We cannot have multiple TIIPredicates with the same name. Otherwise, this will
cause build errors later on, when tablegen'd .inc files are included by cpp
files and then compiled.

Differential Revision: https://reviews.llvm.org/D50708

llvm-svn: 339706
 2018-08-14 18:36:54 +00:00
Simon Pilgrim
8e3f64cd95 [TableGen] Pass string/vector types by const reference (PR37666). NFCI 
llvm-svn: 339670
 2018-08-14 11:17:38 +00:00
Craig Topper
3cb36bdb91 [X86] Don't ignore 0x66 prefix on relative jumps in 64-bit mode. Fix opcode selection of relative jumps in 16-bit mode. Treat jno/jo like other jcc instructions. 
The behavior in 64-bit mode is different between Intel and AMD CPUs. Intel ignores the 0x66 prefix. AMD does not. objump doesn't ignore the 0x66 prefix. Since LLVM aims to match objdump behavior, we should do the same.

While I was trying to fix this I had change brtarget16/32 to use ENCODING_IW/ID instead of ENCODING_Iv to get the 0x66+REX.W case to act sort of sanely. It's still wrong, but that's a problem for another day.

The change in encoding exposed the fact that 16-bit mode disassembly of relative jumps was creating JMP_4 with a 2 byte immediate. It should have been JMP_2. From just printing you can't tell the difference, but if you dumped the encoding it wouldn't have matched what we started with.

While fixing that, it exposed that jo/jno opcodes were missing from the switch that this patch deleted and there were no test cases for them.

Fixes PR38537.

llvm-svn: 339622
 2018-08-13 22:06:28 +00:00
Andrea Di Biagio
0ccb2dee28 [Tablegen] Replace uses of formatted_raw_ostream with raw_ostream in the predicate expander. NFCI 
This is a follow-up of r339552.

As pointed out by Craig in D50566, we don't need a formatted_raw_ostream to
indent strings. We can use instead raw_ostream::indent().

Internally, class PredicateExpander already keeps track of the current
indentation level. Also, the grammar for predicates is well parenthesized, and
therefore we don't need to use a formatted_raw_ostream to continuously track the
column number. Instead we can safely replace all the uses of
formatted_raw_ostream::PadToColumn() with uses of raw_ostream::indent().

By replacing formatted_raw_ostream with a simpler raw_ostream, we also avoid the
implicit check on the newline character on every print to stream.

No functional change intended.

llvm-svn: 339577
 2018-08-13 15:13:35 +00:00
Andrea Di Biagio
e84b02f47e [Tablegen][SubtargetEmitter] Improve expansion of predicates of a variant scheduling class. 
This patch refactors the logic that expands predicates of a variant scheduling
class.

The idea is to improve the readability of the auto-generated code by removing
redundant parentheses around predicate expressions, and by removing redundant
if(true) statements.

This patch replaces the definition of NoSchedPred in TargetSchedule.td with an
instance of MCSchedPredicate. The new definition is sematically equivalent to
the previous one. The main difference is that now SubtargetEmitter knows that it
represents predicate "true".

Before this patch, we always generated an if (true) for the default transition
of a variant scheduling class.

Example (taken from AArch64GenSubtargetInfo.inc) :

```
if (SchedModel->getProcessorID() == 3) { // CycloneModel
  if ((TII->isScaledAddr(*MI)))
    return 927; // (WriteIS_WriteLD)_ReadBaseRS
  if ((true))
    return 928; // WriteLD_ReadDefault
}
```

Extra parentheses were also generated around the predicate expressions.

With this patch, we get the following auto-generated checks:

```
if (SchedModel->getProcessorID() == 3) { // CycloneModel
  if (TII->isScaledAddr(*MI))
    return 927; // (WriteIS_WriteLD)_ReadBaseRS
  return 928; // WriteLD_ReadDefault
}
```

The new auto-generated code behaves exactly the same as before. So, technically
this is a non functional change.

Differential revision: https://reviews.llvm.org/D50566

llvm-svn: 339552
 2018-08-13 11:09:04 +00:00
Daniel Sanders
f4eea0c506 [globalisel] Remove dead code from GlobalISelEmitter 
Summary: Found by GCC's -Wunused-function.

Patch by Kim Gräsman

Reviewers: ab, dsanders, llvm-commits

Reviewed By: dsanders

Subscribers: rovka, kristof.beyls

Differential Revision: https://reviews.llvm.org/D50611

llvm-svn: 339528
 2018-08-12 21:49:42 +00:00
Andrea Di Biagio
49eac3ddbf [Tablegen][SubtargetEmitter] refactor method emitSchedModelHelpersImpl(). NFCI 
Part of the logic has been moved to helper functions to (hopefully) improve
readability.
Added a few code comments to better describe how the algorithm works.

No functional change intended.

llvm-svn: 339421
 2018-08-10 10:43:43 +00:00
Andrea Di Biagio
6413a2dd47 [MC][PredicateExpander] Extend the grammar to support simple switch and return statements. 
This patch introduces tablegen class MCStatement.

Currently, an MCStatement can be either a return statement, or a switch
statement.

```
MCStatement:
   MCReturnStatement
   MCOpcodeSwitchStatement
```

A MCReturnStatement expands to a return statement, and the boolean expression
associated with the return statement is described by a MCInstPredicate.

An MCOpcodeSwitchStatement is a switch statement where the condition is a check
on the machine opcode. It allows the definition of multiple checks, as well as a
default case. More details on the grammar implemented by these two new
constructs can be found in the diff for TargetInstrPredicates.td.

This patch makes it easier to read the body of auto-generated TargetInstrInfo
predicates.

In future, I plan to reuse/extend the MCStatement grammar to describe more
complex target hooks. For now, this is just a first step (mostly a minor
cosmetic change to polish the new predicates framework).

Differential Revision: https://reviews.llvm.org/D50457

llvm-svn: 339352
 2018-08-09 15:32:48 +00:00
Bjorn Pettersson
a3c197394f [MC] Remove PhysRegSize from MCRegisterClass 
Summary:
The interface to get size and spill size of a register
was moved from MCRegisterInfo to TargetRegisterInfo over
a year ago. Afaik the old interface has bee around
to give out-of-tree targets a chance to adapt to the
new interface.

One problem with the old MCRegisterClass::PhysRegSize was that
it represented the size of a register as "size in bits" / 8.
So a register had to be a multiple of eight bits wide for the
size to be correct (and the byte size for the target needed to
be eight bits).

Reviewers: kparzysz, qcolombet

Reviewed By: kparzysz

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D47199

llvm-svn: 339350
 2018-08-09 15:19:07 +00:00
Petr Hosek
1cfc2ea35f [CMake] Use normalized Windows target triples 
Changes the default Windows target triple returned by
GetHostTriple.cmake from the old environment names (which we wanted to
move away from) to newer, normalized ones. This also requires updating
all tests to use the new systems names in constraints.

Differential Revision: https://reviews.llvm.org/D47381

llvm-svn: 339307
 2018-08-09 02:16:18 +00:00
Daniel Sanders
3d06a1da65 [tablegen] Improve performance of -gen-register-info by replacing barely-necessary std::map with a sorted vector 
Summary:
This particular map is hardly ever queried and has a phased usage pattern (insert,
iterate, query, insert, iterate) so it's a good candidate for a sorted vector and
std::lower_bound.

This significantly reduces the run time of runTargetDesc() in some circumstances.
One llvm-tblgen invocation in my build improves the time spent in runTargetDesc()
from 9.86s down to 0.80s (~92%) without changing the output. The same invocation
also has 2GB less allocation churn.

Reviewers: bogner, rtereshin, aditya_nandakumar, volkan

Reviewed By: rtereshin

Subscribers: mgrang, dexonsmith, llvm-commits

Differential Revision: https://reviews.llvm.org/D50272

llvm-svn: 339208
 2018-08-08 00:19:59 +00:00
Aditya Nandakumar
92fa33bc88 Refactor FileCheck to make it usable as an API 
https://reviews.llvm.org/D50283
reviewed by bogner

This patch refactors FileCheck's implementation into support so it can
be used from C++ in other places (Unit tests).

llvm-svn: 339192
 2018-08-07 21:58:49 +00:00
Stella Stamenova
f98614d66f [lit] Disable shtest-timeout on Windows 
This test passes on Windows when using Python 3 but fails when using Python 2, so it needs more investigation before it can be enabled as the bots use Python 2.

llvm-svn: 339184
 2018-08-07 21:21:30 +00:00
Stella Stamenova
e43106cf1b [lit, python3] Update lit error logging to work correctly in python3 and other test fixes 
Summary:
In Python2 'unicode' is a distinct type from 'str', but in Python3 'unicode' does not exist and instead all 'str' objects are Unicode string. This change updates the logic in the test logging for lit to correctly process each of the types, and more importantly, to not just fail in Python3.

This change also reverses the use of quotes in several of the cfg files. By using '""' we are guaranteeing that the resulting path will work correctly on Windows while "''" only works correctly sometimes. This also fixes one of the failing tests.

Reviewers: asmith, zturner

Subscribers: stella.stamenova, delcypher, llvm-commits

Differential Revision: https://reviews.llvm.org/D50397

llvm-svn: 339179
 2018-08-07 20:54:38 +00:00
Stella Stamenova
89cdfaa61e [lit, tests] Fix failing lit test: shtest-format.py 
Summary:
The problem here is that on windows double quotes are used for paths (usually) while single quotes are not. This is not generally a problem for the tests because the lit infrastructure tends to treat both the same. One (and possibly only) exception is when some tests are run in an external shell such as some of the shtest-format tests. In this case on windows the path to python was not created correctly because it had single quotes and the test failed.

This same test is already failing with python 3 which is why our testing missed the new failure. This patch will take care of the immediate failure with python 2 and I'll send a follow up for the python 3 failure.

Reviewers: asmith, zturner

Subscribers: delcypher, llvm-commits

Differential Revision: https://reviews.llvm.org/D50373

llvm-svn: 339091
 2018-08-07 04:08:46 +00:00
Stella Stamenova
f5a62c8af5 [lit, python] Always add quotes around the python path in lit 
Summary:
The issue with the python path is that the path to python on Windows can contain spaces. To make the tests always work, the path to python needs to be surrounded by quotes.

This change updates several configuration files which specify the path to python as a substitution and also remove quotes from existing tests.

Reviewers: asmith, zturner, alexshap, jakehehrlich

Reviewed By: zturner, alexshap, jakehehrlich

Subscribers: mehdi_amini, nemanjai, eraman, kbarton, jakehehrlich, steven_wu, dexonsmith, stella.stamenova, delcypher, llvm-commits

Differential Revision: https://reviews.llvm.org/D50206

llvm-svn: 339073
 2018-08-06 22:37:44 +00:00
Daniil Fukalov
8b5f88897d Fix typo in the MSVC Visualizer for SmallVector class 
llvm-svn: 339029
 2018-08-06 16:47:24 +00:00
Hans Wennborg
a28e486aa2 build_llvm_package.bat: Add OpenMP back 
After r338721, it builds again.

llvm-svn: 338823
 2018-08-03 07:00:08 +00:00
Andrew Savonichev
c0f1a2b3f6 [emacs] Indent statement continuation to match clang-format 
Summary:
Was:

  int LongVariableName =
    veryLongFunctionNameThatExceeds80ColumnsRule(SomeParameter);

  int ShortVar = veryLongFunctionNameThatStillFitsIntoOneLine(SomeParameter)
    .nowThisDoesntFit()
    .andThis()

Now:

  int LongVariableName =
      veryLongFunctionNameThatExceeds80ColumnsRule(SomeParameter);

  int ShortVar = veryLongFunctionNameThatStillFitsIntoOneLine(SomeParameter)
                     .nowThisDoesntFit()
                     .andThis()


Reviewers: espindola, MaskRay

Reviewed By: MaskRay

Subscribers: greened, llvm-commits

Differential Revision: https://reviews.llvm.org/D49726

llvm-svn: 338686
 2018-08-02 08:50:41 +00:00
Hans Wennborg
5e7bf22541 utils/release/tag.sh: add debuginfo-tests to project list 
llvm-svn: 338682
 2018-08-02 08:10:34 +00:00
Hans Wennborg
1e673b607d lit: bump version number 
llvm-svn: 338677
 2018-08-02 07:48:44 +00:00
Hans Wennborg
6b8cb0376e Bump the trunk version to 8.0.0svn 
llvm-svn: 338537
 2018-08-01 13:25:30 +00:00
Ulrich Weigand
322eb17255 Fix build bot after r338521 
llvm-svn: 338522
 2018-08-01 12:07:32 +00:00
Ulrich Weigand
45f5d3d85c [SystemZ, TableGen] Fix shift count handling 
The DAG combiner logic to simplify AND masks in shift counts is invalid.
While it is true that the SystemZ shift instructions ignore all but the
low 6 bits of the shift count, it is still invalid to simplify the AND
masks while the DAG still uses the standard shift operators (which are
*not* defined to match the SystemZ instruction behavior).

Instead, this patch performs equivalent operations during instruction
selection. For completely removing the AND, this now happens via
additional DAG match patterns implemented by a multi-alternative
PatFrags. For simplifying a 32-bit AND to a 16-bit AND, the existing DAG
patterns were already mostly OK, they just needed an output XForm to
actually truncate the immediate value.

Unfortunately, the latter change also exposed a bug in TableGen: it
seems XForms are currently only handled correctly for direct operands of
the outermost operation node. This patch also fixes that bug by simply
recurring through the whole pattern. This should be NFC for all other
targets.

Differential Revision: https://reviews.llvm.org/D50096

llvm-svn: 338521
 2018-08-01 11:57:58 +00:00
Simon Pilgrim
6a5f232f8f Revert r338365: [X86] Improved sched models for X86 BT*rr instructions. 
https://reviews.llvm.org/D49243

Contains WIP code that should not have been included.

llvm-svn: 338369
 2018-07-31 13:00:51 +00:00
Andrew V. Tischenko
fac48f4efe [X86] Improved sched models for X86 BT*rr instructions. 
https://reviews.llvm.org/D49243

llvm-svn: 338365
 2018-07-31 12:33:48 +00:00
Zachary Turner
5466461455 Add VS natvis support for LLVMDemangle's StringView. 
llvm-svn: 338202
 2018-07-28 17:25:42 +00:00
Stella Stamenova
7214eaf562 [windows] Don't inline fieldFromInstruction on Windows 
Summary:
The VS compiler (on Windows) has a bug which results in fieldFromInstruction being optimized out in some circumstances. This only happens in *release no debug info* builds that have assertions *turned off* - in all other situations the function is not inlined, so the functionality is correct. All of the bots have assertions turned on, so this path is not regularly tested. The workaround is to not inline the function on Windows - if the bug is fixed in a later release of the VS compiler, the noinline specification can be removed.

The test that consistently reproduces this is Lanai v11.txt test.

Reviewers: asmith, labath, zturner

Subscribers: dblaikie, stella.stamenova, aprantl, JDevlieghere, llvm-commits

Differential Revision: https://reviews.llvm.org/D49753

llvm-svn: 337942
 2018-07-25 17:33:20 +00:00
Azharuddin Mohammed
964ca0d2f0 [docker] Fix LLVM_EXTERNAL_PROJECTS cmake variable value 
Summary:
LLVM_ENABLE_PROJECTS expects a semicolon separated project list.

Fixes PR38158.

Reviewers: ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D49712

llvm-svn: 337842
 2018-07-24 18:34:13 +00:00
Vedant Kumar
90dde3eaff [utils] Fix the llvm::Optional data formatter 
The llvm::Optional data formatter needs to look through the `Storage`
container if it's present.

Before:

   220    if (Op && Op->getOp() != dwarf::DW_OP_LLVM_fragment)
-> 221      HasComplexExpression = true;
   222
   223    // If the register can only be described by a complex expression (i.e.,
   224    // multiple subregisters) it doesn't safely compose with another complex
Target 0: (llc) stopped.
(lldb) p Op
(llvm::Optional<llvm::DIExpression::ExprOperand>) $0 = None

After:

(lldb) p Op
(llvm::Optional<llvm::DIExpression::ExprOperand>) $0 =
(llvm::DIExpression::ExprOperand) storage = {
  Op = 0x000000010603d460
}

llvm-svn: 337752
 2018-07-23 21:59:06 +00:00
Fangrui Song
5432b93b98 [gdb] Fix SmallVector pretty printer after r337514 
llvm-svn: 337747
 2018-07-23 21:33:51 +00:00
Reid Kleckner
0a9a1968f9 Re-land r335297 "[X86] Implement more of x86-64 large and medium PIC code models" 
Don't try to generate large PIC code for non-ELF targets. Neither COFF
nor MachO have relocations for large position independent code, and
users have been using "large PIC" code models to JIT 64-bit code for a
while now. With this change, if they are generating ELF code, their
JITed code will truly be PIC, but if they target MachO or COFF, it will
contain 64-bit immediates that directly reference external symbols. For
a JIT, that's perfectly fine.

llvm-svn: 337740
 2018-07-23 21:14:35 +00:00
Justin Bogner
586bd2098d [lit] Move the shtest-xunit-output check lines into shtest-format 
These two tests are operating on the same test suite, which causes
them to be racy about writing temporary files and can cause spurious
failures. Merge them into one test to avoid the issue.

llvm-svn: 337718
 2018-07-23 18:08:36 +00:00
Zachary Turner
1e18ab12db Fix the MSVC Visualizers for SmallVector classes. 
Recent changes to the internal structure of SmallVector<> broke
all of the MSVC visualizers.  This fixes them.

llvm-svn: 337644
 2018-07-21 15:38:47 +00:00
George Karpenkov
56b2870853 [FileCheck] Provide an option for FileCheck to dump original input to stderr on failure 
The option can be either set using environment variable (e.g. env
FILECHECK_DUMP_INPUT_ON_FAILURE=1 ninja check-fuzzer) or with a
FileCheck flag.

This can be extremely useful for debugging, cf.
https://groups.google.com/forum/#!topic/llvm-dev/kLrzg8OM_h8 for
discussion.

Differential Revision: https://reviews.llvm.org/D49328

llvm-svn: 337609
 2018-07-20 20:21:57 +00:00
Joel E. Denny
9cdb8f503d [FileCheck] Fix search ranges for DAG-NOT-DAG 
A DAG-NOT-DAG is a CHECK-DAG group, X, followed by a CHECK-NOT group,
N, followed by a CHECK-DAG group, Y.  Let y be the initial directive
of Y.  This patch makes the following changes to the behavior:

    1. Directives in N can no longer match within part of Y's match
       range just because y happens not to be the earliest match from
       Y.  Specifically, this patch withdraws N's search range end
       from y's match range start to Y's match range start.

    2. y can no longer match within X's match range, where a y match
       produced a reordering complaint, which is thus no longer
       possible.  Specifically, this patch withdraws y's search range
       start from X's permitted range start to X's match range end,
       which was already the search range start for other members of
       Y.

Both of these changes can only increase the number of test passes: #1
constrains the ability of CHECK-NOTs to match, and #2 expands the
ability of CHECK-DAGs to match without complaints.

These changes are based on discussions at:

   <http://lists.llvm.org/pipermail/llvm-dev/2018-May/123550.html>
   <https://reviews.llvm.org/D47106>

which conclude that:

    1. These changes simplify the FileCheck conceptual model.  First,
       it makes search ranges for DAG-NOT-DAG more consistent with
       other cases.  Second, it was confusing that y was treated
       differently from the rest of Y.

    2. These changes add theoretical use cases for DAG-NOT-DAG that
       had no obvious means to be expressed otherwise.  We can justify
       the first half of this assertion with the observation that
       these changes can only increase the number of test passes.

    3. Reordering detection for DAG-NOT-DAG had no obvious real
       benefit.

We don't have evidence from real uses cases to help us debate
conclusions #2 and #3, but #1 at least seems intuitive.

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D48986

llvm-svn: 337605
 2018-07-20 20:09:56 +00:00
Roman Lebedev
99b7f809a2 Add (very partial) Kate syntax highlighting definition for TableGen 
This is very clearly not very good, and is very partial.
But this is better than nothing at all, and shouldn't
hurt those who don't need it.

If there are others interested in this functionality,
it will be great to further improve this.

{F6253091}

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D47080

llvm-svn: 337415
 2018-07-18 18:35:27 +00:00
Andrea Di Biagio
ef7e76afcf [Tablegen][PredicateExpander] Add the ability to define checks for invalid registers. 
This was discussed in review D49436.

llvm-svn: 337378
 2018-07-18 11:03:22 +00:00
Andrea Di Biagio
e8c44066cc [Tablegen][PredicateExpander] Fix a bug in expandCheckImmOperand. 
Function `expandCheckImmOperand` should always check if the input machine
instruction is passed by reference before calling method `getOperand()` on it.

Found while working on a patch that relies on `expandCheckImmOperand` to expand
a scheduling predicate.

llvm-svn: 337294
 2018-07-17 16:11:37 +00:00
Craig Topper
55125047c8 [TableGen] std::move vectors into TreePatternNode. 
llvm-svn: 337121
 2018-07-15 06:52:49 +00:00
Craig Topper
c7738f70b7 [TableGen] Remove what seems to be an unnecessary std::map copy. 
The comment says the copy was made so it could be destroyed in the following loop, but the original map wasn't used after the loop.

llvm-svn: 337120
 2018-07-15 06:52:48 +00:00
Craig Topper
08fff11614 [TableGen] Add some std::move to the PatternToMatch constructor. 
The are two vectors passed by value to the constructor. We should be able to move them into the object.

llvm-svn: 337114
 2018-07-15 01:10:28 +00:00
Ulrich Weigand
ef5d1f837e [TableGen] Suppress type validation when parsing pattern fragments 
Currently, any attempt to define a PatFrag involving any floating-point
only (or vector only) node causes a hard assertion failure in TableGen
if the current target does not have any floating-point (or vector)
types.

This is annoying if you want to provide convenience fragments in common
code (e.g. include/llvm/Target/TargetSelectionDAG.td) that is parsed on
all platforms, including those that miss such types.

But really, there's no reason not accept this when parsing the fragment
-- of course it would be an error for such a target to actually *use*
such a fragment anywhere, but as long as it doesn't, I think TableGen
shouldn't error out.

The immediate cause of the assertion failure is the test inside the
ValidateOnExit destructor. This patch simply disables that check while
infering types during parsing of pattern fragments (only).

Reviewed By: hfinkel, kparzysz

Differential Revision: https://reviews.llvm.org/D48887

llvm-svn: 337023
 2018-07-13 16:42:15 +00:00
Marcello Maggioni
ef4dbb7d5f [Tablegen] Optimize isSubsetOf() in AsmMatcherEmitter.cpp. NFC 
isSubsetOf() could be very slow if the hierarchy of the RegisterClasses
of the target is very complicated.
This is mainly caused by the fact that isSubset() is called
multiple times over the same SuperClass of a register class
if this ends up being the super class of a register class
from multiple paths.

Differential Revision: https://reviews.llvm.org/D49124

llvm-svn: 337020
 2018-07-13 16:36:14 +00:00
Joel Galenson
9249622410 [cfi-verify] Support AArch64. 
This patch adds support for AArch64 to cfi-verify.

This required three changes to cfi-verify.  First, it generalizes checking if an instruction is a trap by adding a new isTrap flag to TableGen (and defining it for x86 and AArch64).  Second, the code that ensures that the operand register is not clobbered between the CFI check and the indirect call needs to allow a single dereference (in x86 this happens as part of the jump instruction).  Third, we needed to ensure that return instructions are not counted as indirect branches.  Technically, returns are indirect branches and can be covered by CFI, but LLVM's forward-edge CFI does not protect them, and x86 does not consider them, so we keep that behavior.

In addition, we had to improve AArch64's code to evaluate the branch target of a MCInst to handle calls where the destination is not the first operand (which it often is not).

Differential Revision: https://reviews.llvm.org/D48836

llvm-svn: 337007
 2018-07-13 15:19:33 +00:00
Ulrich Weigand
535942804d [TableGen] Support multi-alternative pattern fragments 
A TableGen instruction record usually contains a DAG pattern that will
describe the SelectionDAG operation that can be implemented by this
instruction. However, there will be cases where several different DAG
patterns can all be implemented by the same instruction. The way to
represent this today is to write additional patterns in the Pattern
(or usually Pat) class that map those extra DAG patterns to the
instruction. This usually also works fine.

However, I've noticed cases where the current setup seems to require
quite a bit of extra (and duplicated) text in the target .td files.
For example, in the SystemZ back-end, there are quite a number of
instructions that can implement an "add-with-overflow" operation.
The same instructions also need to be used to implement just plain
addition (simply ignoring the extra overflow output). The current
solution requires creating extra Pat pattern for every instruction,
duplicating the information about which particular add operands
map best to which particular instruction.

This patch enhances TableGen to support a new PatFrags class, which
can be used to encapsulate multiple alternative patterns that may
all match to the same instruction.  It operates the same way as the
existing PatFrag class, except that it accepts a list of DAG patterns
to match instead of just a single one.  As an example, we can now define
a PatFrags to match either an "add-with-overflow" or a regular add
operation:

  def z_sadd : PatFrags<(ops node:$src1, node:$src2),
                        [(z_saddo node:$src1, node:$src2),
                         (add node:$src1, node:$src2)]>;

and then use this in the add instruction pattern:

  defm AR : BinaryRRAndK<"ar", 0x1A, 0xB9F8, z_sadd, GR32, GR32>;

These SystemZ target changes are implemented here as well.


Note that PatFrag is now defined as a subclass of PatFrags, which
means that some users of internals of PatFrag need to be updated.
(E.g. instead of using PatFrag.Fragment you now need to use
!head(PatFrag.Fragments).)


The implementation is based on the following main ideas:
- InlinePatternFragments may now replace each original pattern
  with several result patterns, not just one.
- parseInstructionPattern delays calling InlinePatternFragments
  and InferAllTypes.  Instead, it extracts a single DAG match
  pattern from the main instruction pattern.
- Processing of the DAG match pattern part of the main instruction
  pattern now shares most code with processing match patterns from
  the Pattern class.
- Direct use of main instruction patterns in InferFromPattern and
  EmitResultInstructionAsOperand is removed; everything now operates
  solely on DAG match patterns.


Reviewed by: hfinkel

Differential Revision: https://reviews.llvm.org/D48545

llvm-svn: 336999
 2018-07-13 13:18:00 +00:00
Chandler Carruth
916457672c [UpdateTestChecks] Teach the x86 asm parser to skip over the function 
begin label emitted for some routines with personality functions and
such.

Without this, we don't even recognize such functions as appearing in the
output and so don't attach any assertions to them. Happy to tweak this
or improve it if folks w/ deeper knowledge of the asm sequences that
show up here want.

llvm-svn: 336987
 2018-07-13 10:29:23 +00:00
Joel E. Denny
460b0069e3 [FileCheck] Implement -v and -vv for tracing matches 
-v prints all directive pattern matches.

-vv additionally prints info that might be noise to users but that can
be helpful to FileCheck developers.

To maximize code reuse and to make diagnostics more consistent, this
patch also adjusts and extends some of the existing diagnostics.
CHECK-NOT failures now report variables uses.  Many more diagnostics
now report the check prefix and kind of directive.

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D47114

llvm-svn: 336967
 2018-07-13 03:08:23 +00:00
Joel E. Denny
a6c7ce37b0 [FileCheck] Don't permit overlapping CHECK-DAG 
That is, make CHECK-DAG skip matches that overlap the matches of any
preceding consecutive CHECK-DAG directives.  This change makes
CHECK-DAG more consistent with other directives, and there is evidence
it makes CHECK-DAG more intuitive and less error-prone.  See the RFC
discussion starting at:

  http://lists.llvm.org/pipermail/llvm-dev/2018-May/123010.html

Moreover, this behavior enables CHECK-DAG groups for unordered,
non-unique strings or patterns.  For example, it is useful for
verifying output or logs from a parallel program, such as the OpenMP
runtime.

This patch also implements the command-line option
-allow-deprecated-dag-overlap, which reverts CHECK-DAG to the old
overlapping behavior.  This option should not be used in new tests.
It is meant only for the existing tests that are broken by this change
and that need time to update.

See the following bugzilla issue for tracking of such tests:

  https://bugs.llvm.org/show_bug.cgi?id=37532

Patches to add -allow-deprecated-dag-overlap to those tests will
follow immediately.

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D47106

llvm-svn: 336847
 2018-07-11 20:27:27 +00:00
Joel E. Denny
71e616812c Revert r336830: [FileCheck] Don't permit overlapping CHECK-DAG 
Companion patches are failing to commit, and this patch alone breaks
many tests.

llvm-svn: 336833
 2018-07-11 19:03:00 +00:00
Joel E. Denny
e3515ea4e8 [FileCheck] Don't permit overlapping CHECK-DAG 
That is, make CHECK-DAG skip matches that overlap the matches of any
preceding consecutive CHECK-DAG directives.  This change makes
CHECK-DAG more consistent with other directives, and there is evidence
it makes CHECK-DAG more intuitive and less error-prone.  See the RFC
discussion starting at:

  http://lists.llvm.org/pipermail/llvm-dev/2018-May/123010.html

Moreover, this behavior enables CHECK-DAG groups for unordered,
non-unique strings or patterns.  For example, it is useful for
verifying output or logs from a parallel program, such as the OpenMP
runtime.

This patch also implements the command-line option
-allow-deprecated-dag-overlap, which reverts CHECK-DAG to the old
overlapping behavior.  This option should not be used in new tests.
It is meant only for the existing tests that are broken by this change
and that need time to update.

See the following bugzilla issue for tracking of such tests:

  https://bugs.llvm.org/show_bug.cgi?id=37532

Patches to add -allow-deprecated-dag-overlap to those tests will
follow immediately.

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D47106

llvm-svn: 336830
 2018-07-11 18:42:58 +00:00
Simon Tatham
7aeb5f145e [TableGen] Add a general-purpose JSON backend. 
The aim of this backend is to output everything TableGen knows about
the record set, similarly to the default -print-records backend. But
where -print-records produces output in TableGen's input syntax
(convenient for humans to read), this backend produces it as
structured JSON data, which is convenient for loading into standard
scripting languages such as Python, in order to extract information
from the data set in an automated way.

The output data contains a JSON representation of the variable
definitions in output 'def' records, and a few pieces of metadata such
as which of those definitions are tagged with the 'field' prefix and
which defs are derived from which classes. It doesn't dump out
absolutely every piece of knowledge it _could_ produce, such as type
information and complicated arithmetic operator nodes in abstract
superclasses; the main aim is to allow consumers of this JSON dump to
essentially act as new backends, and backends don't generally need to
depend on that kind of data.

The new backend is implemented as an EmitJSON() function similar to
all of llvm-tblgen's other EmitFoo functions, except that it lives in
lib/TableGen instead of utils/TableGen on the basis that I'm expecting
to add it to clang-tblgen too in a future patch.

To test it, I've written a Python script that loads the JSON output
and tests properties of it based on comments in the .td source - more
or less like FileCheck, except that the CHECK: lines have Python
expressions after them instead of textual pattern matches.

Reviewers: nhaehnle

Reviewed By: nhaehnle

Subscribers: arichardson, labath, mgorny, llvm-commits

Differential Revision: https://reviews.llvm.org/D46054

llvm-svn: 336771
 2018-07-11 08:40:19 +00:00
Craig Topper
57ffa072f5 [TableGen] Fix some bad formatting. NFC 
llvm-svn: 336751
 2018-07-11 01:01:55 +00:00
Philip Pfaffe
7f47d18278 [Utils] Fix gdb pretty printers to work with Python 3. 
Reiterate D23202 for container printers added after the change landed.

Differential Revision: https://reviews.llvm.org/D46578

llvm-svn: 336580
 2018-07-09 18:51:50 +00:00
Stefan Pintilie
781674a1db [Power9] Add __float128 builtins for Round To Odd 
GCC has builtins for these round to odd instructions:

__float128 __builtin_sqrtf128_round_to_odd (__float128)
__float128 __builtin_{add,sub,mul,div}f128_round_to_odd (__float128, __float128)
__float128 __builtin_fmaf128_round_to_odd (__float128, __float128, __float128)

Differential Revision: https://reviews.llvm.org/D47550

llvm-svn: 336578
 2018-07-09 18:50:06 +00:00
Sander de Smalen
432f7965f4 [TableGen] Increase the number of supported decoder fix-ups. 
The vast number of added instructions for SVE causes TableGen to fail with an assertion:

  Assertion `Delta < 65536U && "disassembler decoding table too large!"'

This patch increases the number of supported decoder fix-ups.

Reviewers: dmgreen, stoklund, petpav01

Reviewed By: dmgreen

Differential Revision: https://reviews.llvm.org/D48937

llvm-svn: 336334
 2018-07-05 10:39:15 +00:00
Hans Wennborg
24eaa76313 build_llvm_package.bat: Re-try the build steps 
The build on Windows has been extra flaky recently; retrying helps.

llvm-svn: 336192
 2018-07-03 11:30:01 +00:00
Krzysztof Parzyszek
2a0fcfa9bf [X86] Add phony registers for high halves of regs with low halves 
Add registers still missing after r328016 (D43353):
- for bits 15-8  of SI, DI, BP, SP (*H), and R8-R15 (*BH),
- for bits 31-16 of R8-R15 (*WH).

Thanks to Craig Topper for pointing it out.

llvm-svn: 336134
 2018-07-02 19:05:09 +00:00
Kristof Beyls
95c680fb11 Make email options of find_interesting_reviews more flexible. 
This enables a few requested improvements on the original review of this
script at https://reviews.llvm.org/D46192.

This introduces 2 new command line options:

* --email-report: This option enables specifying who to email the generated
  report to. This also enables not sending any email and only printing out
  the report on stdout by not specifying this option on the command line.
* --sender: this allows specifying the email address that will be used in
  the "From" email header.

I believe that with these options the script starts having the basic
features needed to run it well on a regular basis for a group of
developers.

Differential Revision: https://reviews.llvm.org/D47930

llvm-svn: 335948
 2018-06-29 07:16:27 +00:00
Jonas Devlieghere
c40fc8755a Revert "Re-land r335297 "[X86] Implement more of x86-64 large and medium PIC code models"" 
Reverting because this is causing failures in the LLDB test suite on
GreenDragon.

  LLVM ERROR: unsupported relocation with subtraction expression, symbol
  '__GLOBAL_OFFSET_TABLE_' can not be undefined in a subtraction
  expression

llvm-svn: 335894
 2018-06-28 17:56:43 +00:00
Zachary Turner
829455b0ec 2 VS natvis improvements. 
Optional<T> was broken due to a change in the class's internals.
That is fixed, and additionally a visualizer is added for
Expected<T>.

llvm-svn: 335892
 2018-06-28 17:55:54 +00:00
James Henderson
de9948f983 [FileCheck] Add CHECK-EMPTY directive for checking for blank lines 
Prior to this change, there was no clean way of getting FileCheck to
check that a line is completely empty. The expected way of using
"CHECK: {{^$}}" does not work because the '^' matches the end of the
previous match (this behaviour may be desirable in certain instances).
For the same reason, "CHECK-NEXT: {{^$}}" will fail when the previous
match was at the end of the line, as the pattern will match there.
Using the recommended [[:space:]] to match an explicit new line could
also match a space, and thus is not always desired. Literal '\n'
matches also do not work. A workaround was suggested in the review, but
it is a little clunky.

This change adds a new directive that behaves the same as CHECK-NEXT,
except that it only matches against empty lines (nothing, not even
whitespace, is allowed). As with CHECK-NEXT, it will fail if more than
one newline occurs before the next blank line. Example usage:
; test.txt
foo

bar
; CHECK: foo
; CHECK-EMPTY:
; CHECK-NEXT: bar

Differential Revision: https://reviews.llvm.org/D28896

Reviewed by: probinson

llvm-svn: 335613
 2018-06-26 15:15:45 +00:00
Andrei Elovikov
35344b129a [NFC] Prefer (void) to LLVM_ATTRIBUTE_UNUSED for unused var in GlobalISElemitter.cpp. 
Reviewers: dsanders, craig.topper

Reviewed By: dsanders

Subscribers: rovka, kristof.beyls, llvm-commits

Differential Revision: https://reviews.llvm.org/D48534

llvm-svn: 335581
 2018-06-26 07:05:08 +00:00
Fangrui Song
3bab5f9eff [gdb] Escape unprintable bytes in SmallString and StringRef 
llvm-svn: 335561
 2018-06-26 00:41:49 +00:00
Fangrui Song
758bcd038b [gdb] Add pretty printer for Expected 
Reviewers: dblaikie

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D48512

llvm-svn: 335554
 2018-06-25 23:38:48 +00:00
Vlad Tsyrklevich
7a3628df0a UBSan blacklist workaround for bot timeouts 
Summary: Workaround for PR37929

Reviewers: eugenis, vitalybuka

Reviewed By: eugenis

Subscribers: llvm-commits, kcc

Differential Revision: https://reviews.llvm.org/D48566

llvm-svn: 335525
 2018-06-25 21:28:35 +00:00
Reid Kleckner
d0145262e2 Re-land r335297 "[X86] Implement more of x86-64 large and medium PIC code models" 
The large code model allows code and data segments to exceed 2GB, which
means that some symbol references may require a displacement that cannot
be encoded as a displacement from RIP. The large PIC model even relaxes
the assumption that the GOT itself is within 2GB of all code. Therefore,
we need a special code sequence to materialize it:
  .LtmpN:
    leaq .LtmpN(%rip), %rbx
    movabsq $_GLOBAL_OFFSET_TABLE_-.LtmpN, %rax # Scratch
    addq %rax, %rbx # GOT base reg

From that, non-local references go through the GOT base register instead
of being PC-relative loads. Local references typically use GOTOFF
symbols, like this:
    movq extern_gv@GOT(%rbx), %rax
    movq local_gv@GOTOFF(%rbx), %rax

All calls end up being indirect:
    movabsq $local_fn@GOTOFF, %rax
    addq %rbx, %rax
    callq *%rax

The medium code model retains the assumption that the code segment is
less than 2GB, so calls are once again direct, and the RIP-relative
loads can be used to access the GOT. Materializing the GOT is easy:
    leaq _GLOBAL_OFFSET_TABLE_(%rip), %rbx # GOT base reg

DSO local data accesses will use it:
    movq local_gv@GOTOFF(%rbx), %rax

Non-local data accesses will use RIP-relative addressing, which means we
may not always need to materialize the GOT base:
    movq extern_gv@GOTPCREL(%rip), %rax

Direct calls are basically the same as they are in the small code model:
They use direct, PC-relative addressing, and the PLT is used for calls
to non-local functions.

This patch adds reasonably comprehensive testing of LEA, but there are
lots of interesting folding opportunities that are unimplemented.

I restricted the MCJIT/eh-lg-pic.ll test to Linux, since the large PIC
code model is not implemented for MachO yet.

Differential Revision: https://reviews.llvm.org/D47211

llvm-svn: 335508
 2018-06-25 18:16:27 +00:00
Reid Kleckner
2a8c0506f2 [IR] Split Intrinsics.inc into enums and implementations 
Implements PR34259

Intrinsics.h is a very popular header. Most LLVM TUs care about things
like dbg_value, but they don't care how they are implemented. After I
split these out, IntrinsicImpl.inc is 1.7 MB, so this saves each LLVM TU
from scanning 1.7 MB of source that gets pre-processed away.

It also means we can modify intrinsic properties without triggering a
full rebuild, but that's probably less of a win.

I think the next best thing to do would be to split out the target
intrinsics into their own header. Very, very few TUs care about
target-specific intrinsics. It's very hard to split up the target
independent intrinsics like llvm.expect, assume, and dbg.value, though.

llvm-svn: 335407
 2018-06-23 02:02:38 +00:00
Fangrui Song
c7c7f4565b [gdb] Use Latin-1 to decode StringRef 
llvm-svn: 335387
 2018-06-22 20:29:42 +00:00
Fangrui Song
aefccfdf5d [gdb] Update llvm::Optional 
Reviewers: dblaikie

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D48461

llvm-svn: 335303
 2018-06-21 22:34:29 +00:00
Reid Kleckner
c1eeade8d2 Revert r335297 "[X86] Implement more of x86-64 large and medium PIC code models" 
MCJIT can't handle R_X86_64_GOT64 yet.

llvm-svn: 335300
 2018-06-21 22:19:05 +00:00
Reid Kleckner
6435d99a79 [X86] Implement more of x86-64 large and medium PIC code models 
Summary:
The large code model allows code and data segments to exceed 2GB, which
means that some symbol references may require a displacement that cannot
be encoded as a displacement from RIP. The large PIC model even relaxes
the assumption that the GOT itself is within 2GB of all code. Therefore,
we need a special code sequence to materialize it:
  .LtmpN:
    leaq .LtmpN(%rip), %rbx
    movabsq $_GLOBAL_OFFSET_TABLE_-.LtmpN, %rax # Scratch
    addq %rax, %rbx # GOT base reg

From that, non-local references go through the GOT base register instead
of being PC-relative loads. Local references typically use GOTOFF
symbols, like this:
    movq extern_gv@GOT(%rbx), %rax
    movq local_gv@GOTOFF(%rbx), %rax

All calls end up being indirect:
    movabsq $local_fn@GOTOFF, %rax
    addq %rbx, %rax
    callq *%rax

The medium code model retains the assumption that the code segment is
less than 2GB, so calls are once again direct, and the RIP-relative
loads can be used to access the GOT. Materializing the GOT is easy:
    leaq _GLOBAL_OFFSET_TABLE_(%rip), %rbx # GOT base reg

DSO local data accesses will use it:
    movq local_gv@GOTOFF(%rbx), %rax

Non-local data accesses will use RIP-relative addressing, which means we
may not always need to materialize the GOT base:
    movq extern_gv@GOTPCREL(%rip), %rax

Direct calls are basically the same as they are in the small code model:
They use direct, PC-relative addressing, and the PLT is used for calls
to non-local functions.

This patch adds reasonably comprehensive testing of LEA, but there are
lots of interesting folding opportunities that are unimplemented.

Reviewers: chandlerc, echristo

Subscribers: hiraditya, llvm-commits

Differential Revision: https://reviews.llvm.org/D47211

llvm-svn: 335297
 2018-06-21 21:55:08 +00:00
Nicolai Haehnle
f2f87b751f TableGen/SearchableTables: Support more generic enums and tables 
Summary:
This is essentially a rewrite of the backend which introduces TableGen
base classes GenericEnum, GenericTable, and SearchIndex. They allow
generating custom enums and tables with lookup functions using
separately defined records as the underlying database.

Also added as part of this change:

- Lookup functions may use indices composed of multiple fields.

- Instruction fields are supported similar to Intrinsic fields.

- When the lookup key has contiguous numeric values, the lookup
  function will directly index into the table instead of using a binary
  search.

The existing SearchableTable functionality is internally mapped to the
new primitives.

Change-Id: I444f3490fa1dbfb262d7286a1660a2c4308e9932

Reviewers: arsenm, tra, t.p.northover

Subscribers: wdng, llvm-commits

Differential Revision: https://reviews.llvm.org/D48013

llvm-svn: 335225
 2018-06-21 13:36:22 +00:00
Craig Topper
150ec2ae44 [X86] Add the ability to force an EVEX2VEX mapping table entry from the .td files. Remove remaining manual table entries from the tablegen emitter. 
This adds an EVEX2VEXOverride string to the X86 instruction class in X86InstrFormats.td. If this field is set it will add manual entry in the EVEX->VEX tables that doesn't check the encoding information.

Then use this mechanism to map VMOVDU/A8/16, 128-bit VALIGN, and VPSHUFF/I instructions to VEX instructions.

Finally, remove the manual table from the emitter.

This has the bonus of fully sorting the autogenerated EVEX->VEX tables by their EVEX instruction enum value. We may be able to use this to do a binary search for the conversion and get rid of the need to create a DenseMap.

llvm-svn: 335018
 2018-06-19 04:24:44 +00:00
Craig Topper
7043d25ccf [X86] Add a new VEX_WPrefix encoding to tag EVEX instruction that have VEX.W==1, but can be converted to their VEX equivalent that uses VEX.W==0. 
EVEX makes heavy use of the VEX.W bit to indicate 64-bit element vs 32-bit elements. Many of the VEX instructions were split into 2 versions with different masking granularity.

The EVEX->VEX table generate can collapse the two versions if the VEX version uses is tagged as VEX_WIG. But if the VEX version is instead marked VEX.W==0 we can't combine them because we don't know if there is also a VEX version with VEX.W==1.

This patch adds a new VEX_W1X tag that indicates the EVEX instruction encodes with VEX.W==1, but is safe to convert to a VEX instruction with VEX.W==0.

This allows us to remove a bunch of manual EVEX->VEX table entries. We may want to look into splitting up the VEX_WPrefix field which would simplify the disassembler.

llvm-svn: 335017
 2018-06-19 04:24:42 +00:00
Craig Topper
96f30be3bf [X86] Encode the EVEX2VEX exception list information in .td files instead of the emitter source. 
Rather than having an exclusion list in tablegen sources, add a flag to the X86 instruction records that can be used to suppress checking for convertibility.

llvm-svn: 334971
 2018-06-18 18:47:07 +00:00
Craig Topper
19bb50c6a9 [TableGen] Make TiedAsmOperandTable in the AsmMatcher 'static' since its at file scope. 
llvm-svn: 334957
 2018-06-18 16:17:46 +00:00
Craig Topper
987002011e [TableGen] Remove unused member variable. 
I think this became unused after r324196.

llvm-svn: 334956
 2018-06-18 16:17:45 +00:00
Sander de Smalen
df252ecda0 [TableGen][AsmMatcherEmitter] Allow tied operands of different classes in aliases. 
Allow a tied operand of a different operand class in InstAliases,
so that the operand can be printed (and added to the MC instruction)
as the appropriate register. For example, 'GPR64as32', which would
be printed/parsed as a 32bit register and should match a tied 64bit
register operand, where the former is a sub-register of the latter.

This patch also generalizes the constraint checking to an overrideable
method in MCTargetAsmParser, so that target asmparsers can specify
whether a given operand satisfies the tied register constraint.

Reviewers: olista01, rengolin, fhahn, SjoerdMeijer, samparker, dsanders, craig.topper

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D47714

llvm-svn: 334942
 2018-06-18 13:39:29 +00:00
Craig Topper
94258135d2 [TableGen] Prevent double flattening of InstAlias asm strings in the asm matcher emitter. 
Unlike CodeGenInstruction, CodeGenInstAlias was flatting asm strings in its constructor. For instructions it was the users responsibility to flatten the string.

AsmMatcherEmitter didn't know this and treated them the same. This caused double flattening of InstAliases. This is mostly harmless unless the desired assembly string contains curly braces. The second flattening wouldn't know to ignore these and would remove the curly braces. And for variant 1 it would remove the contents of them as well.

To mitigate this, this patch makes removes the flattening from the CodeGenIntAlias constructor and modifies AsmWriterEmitter to account for the flattening not having been done.

llvm-svn: 334919
 2018-06-18 01:28:01 +00:00
Craig Topper
29015a63e6 [X86] More additions to the load folding tables based on the autogenerated tables. 
Including more additions for NotMemoryFoldable to remove some entries from the autogenerated table.

llvm-svn: 334898
 2018-06-16 23:25:50 +00:00
Daniel Sanders
50735a1161 [globalisel][tablegen] Add support for C++ predicates on PatFrags and use it to support BFC on ARM. 
So far, we've only handled special cases of PatFrag like ImmLeaf. This patch
adds support for the remaining cases using similar mechanisms.

Like most C++ code from SelectionDAG, GISel and DAGISel expect to operate on
different types and representations and as such the code is not compatible
between the two. It's therefore necessary to add an alternative implementation
in the GISelPredicateCode field.

The target test for this feature could easily be done with IntImmLeaf and this
would save on a little boilerplate. The reason I've chosen to implement this
using PatFrag.GISelPredicateCode and not IntImmLeaf is because I was unable to
find a rule that was blocked solely by lack of support for PatFrag predicates. I
found that the ones I investigated as being likely candidates for the test
were further blocked by other things.

llvm-svn: 334871
 2018-06-15 23:13:43 +00:00
Roman Lebedev
f74c82875a [NFC] chmod +x utils/update_analyze_test_checks.py 
Looks like a simple oversight.

llvm-svn: 334825
 2018-06-15 12:41:50 +00:00
Craig Topper
5584c8b9f5 [X86] Add 'Z' to the internal names of various EVEX instructions for overall consistency. 
llvm-svn: 334785
 2018-06-15 04:42:54 +00:00
Florian Hahn
88222e23c3 Revert r334764, as it breaks some bots 
llvm-svn: 334767
 2018-06-14 20:32:58 +00:00
Florian Hahn
e1f114367e [TableGen] Make TreePatternNode::getChild return a reference (NFC) 
The return value of TreePatternNode::getChild is never null. This patch also
updates various places that use return values of getChild to also use
references. Those changes were suggested post-commit for D47463.

llvm-svn: 334764
 2018-06-14 20:23:48 +00:00
Florian Hahn
bd074cfa10 [TableGen] Move some shared_ptrs to avoid unnecessary copies (NFC). 
Those changes were suggested post-commit for D47463.

llvm-svn: 334706
 2018-06-14 11:56:19 +00:00
Florian Hahn
34430c5d61 [TableGen] Make getOnlyTree return a const ref (NFC) 
This avoids some unnecessary copies of shared_ptrs.
Those changes were suggested post-commit for D47463.

llvm-svn: 334656
 2018-06-13 20:59:53 +00:00
Clement Courbet
e3e2fa9c0a [TableGen] Emit a fatal error on inconsistencies in resource units vs cycles. 
Summary:
For targets I'm not familiar with, I've automatically made the "default to 1 for each resource" behaviour explicit in the td files.
For more obvious cases, I've ventured a fix.

Some notes:
 - Exynos is especially fishy.
 - AArch64SchedThunderX2T99.td had some truncated entries. If I understand correctly, the person who wrote that interpreted the ResourceCycle as a range. I made the decision to use the upper/lower bound for consistency with the 'Latency' value. I'm sure there is a better choice.
 - The change to X86ScheduleBtVer2.td is an NFC, it just makes values more explicit.

Also see PR37310.

Reviewers: RKSimon, craig.topper, javed.absar

Subscribers: kristof.beyls, llvm-commits

Differential Revision: https://reviews.llvm.org/D46356

llvm-svn: 334586
 2018-06-13 09:41:49 +00:00
Tom Stellard
fa1ab0b0a2 utils/release: Add merge-git.sh 
Summary:
This script allows you to use git to backport a commit to a stable
branch while generating the exact same commit message (ignoring
whitespace) that you would get from using the merge.sh script with svn.

Reviewers: hansw

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D47760

llvm-svn: 334568
 2018-06-13 05:14:10 +00:00
Craig Topper
73c6ebc8bf [X86] Mark all instructions that have masked store semantics with NotMemoryFoldable. Remove dependency on SchedRW from memory table autogenerator. 
Previously we were whitelisting in instructions based on their SchedRW value. With the masked store instructions explicitly removed via NotMemoryFoldable, we don't seem to need this check anymore.

llvm-svn: 334563
 2018-06-13 00:04:08 +00:00
Craig Topper
72d5015e27 [X86] Remove unnecessary include from one of the tblgen emitters. 
llvm-svn: 334530
 2018-06-12 19:59:10 +00:00
Zachary Turner
0b859bfff5 Refactor ExecuteAndWait to take StringRefs. 
This simplifies some code which had StringRefs to begin with, and
makes other code more complicated which had const char* to begin
with.

In the end, I think this makes for a more idiomatic and platform
agnostic API.  Not all platforms launch process with null terminated
c-string arrays for the environment pointer and argv, but the api
was designed that way because it allowed easy pass-through for
posix-based platforms.  There's a little additional overhead now
since on posix based platforms we'll be takign StringRefs which
were constructed from null terminated strings and then copying
them to null terminate them again, but from a readability and
usability standpoint of the API user, I think this API signature
is strictly better.

llvm-svn: 334518
 2018-06-12 17:43:52 +00:00
Craig Topper
ed7773b234 [X86] Don't add stores to the autogenerated load folding tables if the register and memory operands have different widths. 
This can cause the amount of the memory written to be changed which would be bad.

llvm-svn: 334480
 2018-06-12 07:32:18 +00:00
Craig Topper
01a2362d18 [X86] Update folding table generator to properly detect RMW arithmetic instructions. 
The RMW instructions are detected by reading the SchedRW data, but the RMW instructions have had their SchedRW changed in recent months. This broke the expectation.

We probably should fix this to use the mayLoad/mayStore flags if possible.

llvm-svn: 334478
 2018-06-12 07:32:15 +00:00
Craig Topper
058160bce4 [X86] Enforce agreement of AdSize field when autogenerating load folding table entries. 
llvm-svn: 334473
 2018-06-12 04:34:58 +00:00
David Blaikie
e580ca2231 TableGen: Change some pointer parameters to references since they're never null anyway 
llvm-svn: 334446
 2018-06-11 22:14:43 +00:00
Roman Lebedev
955034baea [Utils] update_llc_test_checks.py: support AMDGPU backend: AMDGCN, r600 triples 
Summary:
Lack of that support has taken me by surprise.
I need to add (or at least look at) some tests for https://reviews.llvm.org/D47980#1127615,
and i don't really fancy doing that by hand.

The asm pattern is quite similar to that of x86:
https://godbolt.org/g/hfgeds
just with `#` replaced with `;`

Reviewers: spatel, RKSimon, MaskRay, tstellar, arsenm

Reviewed By: arsenm

Subscribers: arsenm, kzhuravl, wdng, yaxunl, dstuttard, tpr, t-tye, rampitec, bogner, mareko, llvm-commits

Tags: #amdgpu

Differential Revision: https://reviews.llvm.org/D48001

llvm-svn: 334396
 2018-06-11 09:20:21 +00:00
Craig Topper
82c8244468 [TableGen] Make better use of std::map::emplace and emplace construct the object in the map rather than moving it into it. Remove a use std::map::find by remembering the return from emplace. 
llvm-svn: 334380
 2018-06-10 23:15:49 +00:00
Craig Topper
491da98404 [TableGen] Combine two constructors by taking vectors by value instead of trying to support combininations for rvalue and lvalue references. 
llvm-svn: 334379
 2018-06-10 23:15:48 +00:00
Florian Hahn
72dee1bd0b Revert r334374 [TableGen] Move some shared_ptrs to avoid unnecessary copies (NFC). 
This breaks some builders.

llvm-svn: 334376
 2018-06-10 21:06:24 +00:00
Florian Hahn
fd5d41cdb9 [TableGen] Move some shared_ptrs to avoid unnecessary copies (NFC). 
Those changes were suggested post-commit for D47463.

llvm-svn: 334374
 2018-06-10 20:38:52 +00:00
Daniel Sanders
1799831af2 [tablegen] Improve performance on *GenRegisterInfo.inc by replacing SparseVector with BitVector. NFC 
Summary: Generating X86GenRegisterInfo.inc and AArch64GenRegisterInfo.inc is 8-9% faster on my build.

Reviewers: bogner, javed.absar

Reviewed By: bogner

Subscribers: llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D47907

llvm-svn: 334337
 2018-06-08 23:12:29 +00:00
Florian Hahn
cc1e276c98 [TableGen] Make DAGInstruction own Pattern to avoid leaking it. 
Reviewers: dsanders, craig.topper, stoklund, nhaehnle

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D47525

llvm-svn: 334275
 2018-06-08 09:54:04 +00:00
Craig Topper
e36da260b7 [X86] Rename vy512mem->vy512xmem and vz256xmem->vz256mem. 
The index size is represented by the letter after the 'v'. The number represents the memory size. If an 'x' appears after the number its means the index register can be from VR128X/VR256X instead of VR128/VR256.

As vy512mem uses a VR256X index it should have an x.
And vz256mem uses a VR512 index so it shouldn't have an x.

I admit these names kind of suck and are confusing.

llvm-svn: 334120
 2018-06-06 19:15:12 +00:00
Hans Wennborg
f9ea32fcfd Relax shtest-run-at-line.py 
The test was failing on Windows machines which had bash.exe on PATH (but
not in the so called lit tools dir, containing cmp.exe, grep.exe etc.).

The problem was that the outer lit invocation would load LLVMConfig
from utils/lit/lit/llvm/config.py, which looks up the tools path with
getToolsPath(). That has a surprising side effect of also setting
bashPath, in our case setting it to empty.

The outer lit invocation would thus configure the pdbg0 and pdbg1
substitutions based on not running with bash.

But the inner lit invocation would not load LLVMConfig, so bash
would be found on PATH, that would be used as external shell,
and so the output wouldn't match pdbg0 and pdbg1.

It seems weird to me that getBashPath() will return different results
depending on whether getToolsPath() has been called before, but I
also don't know how to fix it properly.

This commit just relaxes the test case, because there doesn't seem
to be much point in testing for the exact syntax of the run file
as long as it works.

(See https://crbug.com/850023)

llvm-svn: 334100
 2018-06-06 14:53:03 +00:00
Kristof Beyls
211bc0b9fa Avoid UnicodeEncodeError on non-ascii reviewer names 
... by using unicode instead of byte strings where non-ascii strings can be
formatted in.

llvm-svn: 334098
 2018-06-06 14:19:58 +00:00
Vlad Tsyrklevich
48f4d3ae20 [Analyzer] Fix the Z3 lit test config 
Summary:
The '%analyze' extra_args config argument seems to have been erroneously
deleted in r315627 disabling Z3 tests for the clang analyzer. Add the
flag back.

Reviewers: george.karpenkov, NoQ, ddcc

Reviewed By: george.karpenkov

Subscribers: xazax.hun, szepet, delcypher, a.sidorin, llvm-commits

Differential Revision: https://reviews.llvm.org/D47722

llvm-svn: 334066
 2018-06-06 06:25:37 +00:00
Greg Bedwell
d0b4720ec4 [UpdateTestChecks] Error if --llvm-mca-binary gets an empty string 
If the command line was mistyped like:
./update_mca_test_checks.py --llvm-mca-binary= /path/to/llvm-mca *.s
                                              ^-- extra whitespace

then /path/to/llvm-mca would get treated by argparse as a test-path
pattern and could actually be opened in write mode and overwritten.

llvm-svn: 334029
 2018-06-05 17:16:19 +00:00
Nicolai Haehnle
7eea83d527 TableGen/DAGPatterns: Allow bit constants in addition to int constants 
Summary:
Implicit casting is a simple quality of life improvement.

Change-Id: I3d2b31b8b8f12cbb1e84f691e359fa713a9c4b42

Reviewers: tra, simon_tatham, craig.topper, MartinO, arsenm

Subscribers: wdng, llvm-commits

Differential Revision: https://reviews.llvm.org/D47432

llvm-svn: 333904
 2018-06-04 14:45:12 +00:00
Greg Bedwell
75a3de70a1 [llvm-mca][UpdateTestChecks] Prevent an IndexError being raised when given empty input 
llvm-svn: 333894
 2018-06-04 12:30:10 +00:00
Roman Lebedev
98a53476a7 [llvm-mca] Make sure not to end the test files with an empty line. 
Summary:
It's super irritating.

[properly configured] git client then complains about that double-newline,
and you have to use `--force` to ignore the warning, since even if you
fix it manually, it will be reintroduced the very next runtime :/

Reviewers: RKSimon, andreadb, courbet, craig.topper, javed.absar, gbedwell

Reviewed By: gbedwell

Subscribers: javed.absar, tschuett, gbedwell, llvm-commits

Differential Revision: https://reviews.llvm.org/D47697

llvm-svn: 333887
 2018-06-04 11:48:46 +00:00
Simon Pilgrim
2196dd6cfa [Utils][X86] Help update_llc_test_checks.py to recognise retl/retq to reduce CHECK duplication (PR35003) 
This patch replaces the --x86_extra_scrub command line argument to automatically support a second level of regex-scrubbing if it improves the matching of nearly-identical code patterns. The argument '--extra_scrub' is there now to force extra matching if required.

This is mostly useful to help us share 32-bit/64-bit x86 vector tests which only differs by retl/retq instructions, but any scrubber can now technically support this, meaning test checks don't have to be needlessly obfuscated.

I've updated some of the existing checks that had been manually run with --x86_extra_scrub, to demonstrate the extra "ret{{[l|q]}}" scrub now only happens when useful, and re-run the sse42-intrinsics file to show extra matches - most sse/avx intrinsics files should be able to now share 32/64 checks.

Tested with the opt/analysis scripts as well which share common code - AFAICT the other update scripts use their own versions.

Differential Revision: https://reviews.llvm.org/D47485

llvm-svn: 333749
 2018-06-01 13:37:01 +00:00
Andrea Di Biagio
09b5879706 [MCSchedule] Add the ability to compute the latency and throughput information for MCInst. 
This patch extends the MCSchedModel API with new methods that can be used to
obtain the latency and reciprocal througput information for an MCInst.

Scheduling models have recently gained the ability to resolve variant scheduling
classes associated with MCInst objects. Before, models were only able to resolve
a variant scheduling class from a MachineInstr object.

This patch is mainly required by D47374 to avoid regressing a pair of x86
specific -print-schedule tests for btver2. Patch D47374 introduces a new variant
class to teach the btver scheduling model (x86 target) how to correctly compute
the latency profile for some zero-idioms using the new scheduling predicates.

The new methods added by this patch would be mainly used by llc when flag
-print-schedule is specified. In particular, tests that contain inline assembly
require that code is parsed at code emission stage into a sequence of MCInst.
That forces the print-schedule functionality to query the latency/rthroughput
information for MCInst instructions too. If we don't expose this new API, then
we lose "-print-schedule" test coverage as soon as variant scheduling classes
are added to the x86 models.

The tablegen SubtargetEmitter changes teaches how to query latency profile
information using a object that derives from TargetSubtargetInfo. Note that this
should really have been part of r333286. To avoid code duplication, the logic
that "resolves" variant scheduling classes for MCInst, has been moved to a
common place in MC. That logic is used by the "resolveVariantSchedClass" methods
redefined in override by the tablegen'd GenSubtargetInfo classes.

Differential Revision: https://reviews.llvm.org/D47536

llvm-svn: 333650
 2018-05-31 13:30:42 +00:00
Joel E. Denny
a04841e104 [lit] Fix windows cmd.exe test config for r333620 
llvm-svn: 333630
 2018-05-31 05:48:33 +00:00
Joel E. Denny
bd389c40fd [lit] Terminate ": RUN at line N" with ";" not "&&" 
This fixes projects/compiler-rt/test/fuzzer/sigusr.test, which was
broken by r333614.  The trouble was that "&&" changes the command for
which "$!" gives the pid.

llvm-svn: 333620
 2018-05-31 03:40:37 +00:00
Joel E. Denny
3204e36001 [lit] Report line number for failed RUN command 
(Relands r333584, reverted in 333592.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed.  For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files.  (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)

Reviewed By: delcypher,	asmith, stella.stamenova, jmorse, lebedev.ri, rnk

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 333614
 2018-05-31 00:55:32 +00:00
Joel E. Denny
664a77c58b Revert r333584: [lit] Report line number for failed RUN command 
It breaks test-suite.

llvm-svn: 333592
 2018-05-30 21:07:27 +00:00
Florian Hahn
207dd7516e [TableGen] Avoid leaking TreePatternNodes by using shared_ptr. 
By using std::shared_ptr for TreePatternNode, we can avoid leaking them.

Reviewers: craig.topper, dsanders, stoklund, tstellar, zturner

Reviewed By: dsanders

Differential Revision: https://reviews.llvm.org/D47463

llvm-svn: 333591
 2018-05-30 21:00:18 +00:00
Joel E. Denny
f5a15f4aa3 [lit] Report line number for failed RUN command 
(Relands r330755 (reverted in r330848) with fix for PR37239.)

When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed.  For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.

When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.

To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.

To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files.  (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)

Reviewed By: delcypher,	asmith, stella.stamenova, jmorse, lebedev.ri, rnk

Differential Revision: https://reviews.llvm.org/D44598

llvm-svn: 333584
 2018-05-30 19:42:27 +00:00
Florian Hahn
7618648e0d [TableGen] Use explicit constructor for InstMemo 
This should fix a few buildbot failures with old
GCC versions.

llvm-svn: 333448
 2018-05-29 18:34:42 +00:00
Florian Hahn
c24f7d921e [TableGen] Fix leaking of PhysRegInputs. 
Instead of dynamically allocating the vector for PhysRegs, we can
allocate it on the stack and move it into InstructionMemo.

Reviewers: mcrosier, craig.topper, RKSimon, dsanders

Reviewed By: dsanders

Differential Revision: https://reviews.llvm.org/D47461

llvm-svn: 333438
 2018-05-29 17:40:03 +00:00
Florian Hahn
b1baaee29e [TableGen] Fix leaking synthesized registers. 
By keeping track of unique_ptrs to the synthesized definitions in
CodeGenRegBank we avoid leaking them.

Reviewers: dsanders, kparzysz, stoklund

Reviewed By: dsanders

Differential Revision: https://reviews.llvm.org/D47462

llvm-svn: 333434
 2018-05-29 16:55:06 +00:00
Andrea Di Biagio
8e881f054d [Tablegen] Avoid generating empty switch statements. NFC 
This fixes an MSVC warning (warning C4065: switch statement contains 'default'
but no 'case' labels) introduced with revision 333293.

llvm-svn: 333363
 2018-05-27 19:08:12 +00:00
Andrea Di Biagio
f8f1287b06 [Tablegen][SubtargetEmitter] Add a default case to the auto-generated switch in MCSubtargetInfo::resolveVariantSchedClass(). NFC 
llvm-svn: 333293
 2018-05-25 18:25:54 +00:00
Andrea Di Biagio
d69daf9e61 [RFC][Patch 2/3] Add a MCSubtargetInfo hook to resolve variant scheduling classes. 
This patch is the second of a sequence of three patches related to LLVM-dev RFC
"MC support for varinat scheduling classes".
https://lists.llvm.org/pipermail/llvm-dev/2018-May/123181.html

The goal of this patch is to enable the resolution of variant classes in MC with
the help of a new method named `MCSubtargetInfo::resolveVariantSchedClass()`.

This patch also teaches the SubtargetEmitter how to automatically generate the
definition of method resolveVariantSchedClass().  That definition is emitted
within a sub-class of MCSubtargetInfo named XXXGenMCSubtargetInfo (where XXX is
the name of the Target).

Differential Revision: https://reviews.llvm.org/D47077

llvm-svn: 333286
 2018-05-25 16:02:43 +00:00
Andrea Di Biagio
c26fd3adc6 [RFC][Patch 1/3] Add a new class of predicates for variant scheduling classes. 
This patch is the first of a sequence of three patches described by the LLVM-dev
RFC "MC support for variant scheduling classes".
http://lists.llvm.org/pipermail/llvm-dev/2018-May/123181.html

The goal of this patch is to introduce a new class of scheduling predicates for
SchedReadVariant and SchedWriteVariant.

An MCSchedPredicate can be used instead of a normal SchedPredicate to model
checks on the instruction (either a MachineInstr or a MCInst).
Internally, an MCSchedPredicate encapsulates an MCInstPredicate definition.
MCInstPredicate allows the definition of expressions with a well-known semantic,
that can be used to generate code for both MachineInstr and MCInst.

This is the first step toward teaching to tools like lllvm-mca how to resolve
variant scheduling classes.

Differential Revision: https://reviews.llvm.org/D46695

llvm-svn: 333282
 2018-05-25 15:55:37 +00:00
Chris Matthews
0b61b91c62 Use quoteattr to ensure we make well formed attributes 
We were making malformed XML on tests with ' in the name.  Switch to
using saxutils to set all of our attributes, so it can handle quotes
etc correctly.

llvm-svn: 333249
 2018-05-25 00:31:36 +00:00
Greg Bedwell
fbe9943602 [UpdateTestChecks] Improved update_mca_test_checks block analysis 
Previously update_mca_test_checks worked entirely at "block" level where
a block is some sequence of lines delimited by at least one empty line.
This generally worked well, but could sometimes lead to excessive
repetition of check lines for various prefixes if some block was almost
identical between prefixes, but not quite (for example, due to a
different dispatch width in the otherwise identical summary views).

This new analyis attempts to split blocks further in the case where the
following conditions are met:
  a) There is some prefix common to every RUN line (typically 'ALL').
  b) The first line of the block is common to the output with every prefix.
  c) The block has the same number of lines for the output with every prefix.

Also, regenerated all llvm-mca test files with the following command:
update_mca_test_checks.py "../test/tools/llvm-mca/*/*.s" "../test/tools/llvm-mca/*/*/*.s"

The new analysis showed a "multiple lines not disambiguated by prefixes" warning
for test "AArch64/Exynos/scheduler-queue-usage.s" so I've also added some
explicit prefixes to each of the RUN lines in that test.

Differential Revision: https://reviews.llvm.org/D47321

llvm-svn: 333204
 2018-05-24 16:36:44 +00:00
Roman Tereshin
3066add2d0 [GlobalISel][InstructionSelect] Switching over root LLTs, perf patch 10 
This patch continues a series of patches started by r332907 (reapplied
as r332917).

In this commit we introduce new matching opcode for the MatchTable:
GIM_SwitchType, similar to GIM_SwitchOpcode, and use it to switch over
LLTs of def operands of root instructions on the 2nd level of the
MatchTable within GIM_SwitchOpcode's cases.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by about 6.5% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64 (cross-compile on x86).

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333146
 2018-05-24 00:24:15 +00:00
Roman Tereshin
14fdf4cba1 [GlobalISel][InstructionSelect] Moving Reg Bank Checks forward, perf patch 9 
This patch continues a series of patches started by r332907 (reapplied
as r332917).

In this commit we move register bank checks back from epilogue of
every rule matcher to a position locally close to the rest of the
checks for a particular (nested) instruction.

This increases the number of common conditions within 2nd level
groups.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by about 2% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64 (cross-compile on x86).

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333144
 2018-05-23 23:58:10 +00:00
Roman Tereshin
8c9a32911c [GlobalISel][InstructionSelect] Maximizing # of Group's common conditions, perf patch 8 
This patch continues a series of patches started by r332907 (reapplied
as r332917).

In this commit we greedily stuff 2nd level GroupMatcher's common
conditions with as many predicates as possible. This is purely
post-processing and it doesn't change which rules are put into the
groups in the first place: that decision is made by looking at the
first common predicate only.

The compile time improvements are minor and well within error margin,
however, it's highly improbable that this transformation could
pessimize performance, thus I'm still committing it for potential
gains for targets not implementing GlobalISel yet and out of tree
targets.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333139
 2018-05-23 22:50:53 +00:00
Roman Tereshin
0d915067bd [GlobalISel][Tablegen] Assign small opcodes to pseudos 
Sort pseudo instructions first while emitting enum's for target
instructions info. That puts them close to each other and to generic
G_* opcodes for GlobalISel. This makes it easier to build small jump
tables over opcodes that could be directly embedded into MatchTable's
Tablegen'erated for GlobalISel's InstructionSelect.

Reviewed By: bogner

Differential Revision: https://reviews.llvm.org/D47240

llvm-svn: 333135
 2018-05-23 22:10:21 +00:00
Roman Tereshin
069717c1b2 [GlobalISel][InstructionSelect] Sorting MatchTable's 2nd level by root LLT, perf patch 7 
This patch continues a series of patches started by r332907 (reapplied
as r332917).

In this commit we sort rules within their 2nd level by the type check
on def operand of the root instruction, which allows for better
nesting grouping on the level.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by roughly 22% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64 (cross-compile on x86).

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333131
 2018-05-23 21:30:16 +00:00
Roman Tereshin
70950ef979 [Tablegen] Tidying up InstRegexOp a little, NFC 
Differential Review: https://reviews.llvm.org/D47240

llvm-svn: 333121
 2018-05-23 20:45:43 +00:00
Roman Tereshin
00a81de35d [GlobalISel][InstructionSelect] Moving type checks forward, perf patch 6 
This patch continues a series of patches started by r332907 (reapplied
as r332917)

In this commit we sort type checks towards the beginning of every rule
within the MatchTable as they fail often and it's best to fail early.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by roughly 7% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64. The amalgamation is a large single-file C-source that makes
compiler backend performance improvements to stand out from frontend.
It's also a part of CTMark.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333114
 2018-05-23 19:16:59 +00:00
Petar Jovanovic
8c311a61bc [X86][MIPS][ARM] New machine instruction property 'isMoveReg' 
This property is needed in order to follow values movement between
registers. This property is used in TII to implement method that
returns true if simple copy like instruction is recognized, along
with source and destination machine operands.

Patch by Nikola Prica.

Differential Revision: https://reviews.llvm.org/D45204

llvm-svn: 333093
 2018-05-23 15:28:28 +00:00
Nicola Zaghen
80a94d5002 Remove DEBUG macro. 
Now that the LLVM_DEBUG() macro landed on the various sub-projects
the DEBUG macro can be removed.
Also change the new uses of DEBUG to LLVM_DEBUG.

Differential Revision: https://reviews.llvm.org/D46952

llvm-svn: 333091
 2018-05-23 15:09:29 +00:00
Roman Tereshin
3e1ac7e466 [GlobalISel][InstructionSelect] MatchTable second level grouping, perf patch 5 
This patch continues a series of patches started by r332907 (reapplied
as r332917)

In this commit we start grouping rules with common first condition on
the second level of the table.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by roughly 13% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333053
 2018-05-23 02:04:19 +00:00
Roman Tereshin
16be0df7da [GlobalISel][InstructionSelect] Switching MatchTable over opcodes, perf patch 4 
This patch continues a series of patches started by r332907 (reapplied
as r332917)

In this commit we introduce a new matching opcode GIM_SwitchOpcode
that implements a jump table over opcodes and start emitting them for
root instructions.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by roughly 20% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64.

To some degree, we assume here that the opcodes form a dense set,
which is true at the moment for all upstream targets given the
limitations of our rule importing mechanism.

It might not be true for out of tree targets, specifically due to
pseudo's. If so, we might noticeably increase the size of the
MatchTable with this patch due to padding zeros. This will be
addressed later.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333017
 2018-05-22 19:37:59 +00:00
Roman Tereshin
98360ae694 [GlobalISel][InstructionSelect] Sorting MatchTable's first level by opcodes and num operands, perf patch 3 
This patch continues a series of patches started by r332907 (reapplied as r332917)

In this commit we start sorting the rules by the opcode first, and if
the same, by the number of operands of the root instructions. This
allows better grouping and safe as patterns with different opcodes are
mutually exclusive.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by roughly 18% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64.

I'm also removing RuleMatcher::getFirstConditionAsRootType() function
here and moving it to a later patch within the series as it's not used
yet and was causing a warning on sanitizer-ppc64le-linux bot.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333001
 2018-05-22 16:54:27 +00:00
Roman Tereshin
9f288d1be9 Reverting 332999 to get it a proper commit message 
llvm-svn: 333000
 2018-05-22 16:53:42 +00:00
Roman Tereshin
0b04e29b2c diff --git a/utils/TableGen/GlobalISelEmitter.cpp b/utils/TableGen/GlobalISelEmitter.cpp 
index cdc9df7bf6b..be08165a200 100644
--- a/utils/TableGen/GlobalISelEmitter.cpp
+++ b/utils/TableGen/GlobalISelEmitter.cpp
@@ -1,4531 +1,4539 @@
 //===- GlobalISelEmitter.cpp - Generate an instruction selector -----------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 /// \file
 /// This tablegen backend emits code for use by the GlobalISel instruction
 /// selector. See include/llvm/CodeGen/TargetGlobalISel.td.
 ///
 /// This file analyzes the patterns recognized by the SelectionDAGISel tablegen
 /// backend, filters out the ones that are unsupported, maps
 /// SelectionDAG-specific constructs to their GlobalISel counterpart
 /// (when applicable: MVT to LLT;  SDNode to generic Instruction).
 ///
 /// Not all patterns are supported: pass the tablegen invocation
 /// "-warn-on-skipped-patterns" to emit a warning when a pattern is skipped,
 /// as well as why.
 ///
 /// The generated file defines a single method:
 ///     bool <Target>InstructionSelector::selectImpl(MachineInstr &I) const;
 /// intended to be used in InstructionSelector::select as the first-step
 /// selector for the patterns that don't require complex C++.
 ///
 /// FIXME: We'll probably want to eventually define a base
 /// "TargetGenInstructionSelector" class.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "CodeGenDAGPatterns.h"
 #include "SubtargetFeatureInfo.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Support/CodeGenCoverage.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/LowLevelTypeImpl.h"
 #include "llvm/Support/MachineValueType.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "llvm/TableGen/Error.h"
 #include "llvm/TableGen/Record.h"
 #include "llvm/TableGen/TableGenBackend.h"
 #include <numeric>
 #include <string>
 using namespace llvm;
 
 #define DEBUG_TYPE "gisel-emitter"
 
 STATISTIC(NumPatternTotal, "Total number of patterns");
 STATISTIC(NumPatternImported, "Number of patterns imported from SelectionDAG");
 STATISTIC(NumPatternImportsSkipped, "Number of SelectionDAG imports skipped");
 STATISTIC(NumPatternsTested, "Number of patterns executed according to coverage information");
 STATISTIC(NumPatternEmitted, "Number of patterns emitted");
 
 cl::OptionCategory GlobalISelEmitterCat("Options for -gen-global-isel");
 
 static cl::opt<bool> WarnOnSkippedPatterns(
     "warn-on-skipped-patterns",
     cl::desc("Explain why a pattern was skipped for inclusion "
              "in the GlobalISel selector"),
     cl::init(false), cl::cat(GlobalISelEmitterCat));
 
 static cl::opt<bool> GenerateCoverage(
     "instrument-gisel-coverage",
     cl::desc("Generate coverage instrumentation for GlobalISel"),
     cl::init(false), cl::cat(GlobalISelEmitterCat));
 
 static cl::opt<std::string> UseCoverageFile(
     "gisel-coverage-file", cl::init(""),
     cl::desc("Specify file to retrieve coverage information from"),
     cl::cat(GlobalISelEmitterCat));
 
 static cl::opt<bool> OptimizeMatchTable(
     "optimize-match-table",
     cl::desc("Generate an optimized version of the match table"),
     cl::init(true), cl::cat(GlobalISelEmitterCat));
 
 namespace {
 //===- Helper functions ---------------------------------------------------===//
 
 /// Get the name of the enum value used to number the predicate function.
 std::string getEnumNameForPredicate(const TreePredicateFn &Predicate) {
   return "GIPFP_" + Predicate.getImmTypeIdentifier().str() + "_" +
          Predicate.getFnName();
 }
 
 /// Get the opcode used to check this predicate.
 std::string getMatchOpcodeForPredicate(const TreePredicateFn &Predicate) {
   return "GIM_Check" + Predicate.getImmTypeIdentifier().str() + "ImmPredicate";
 }
 
 /// This class stands in for LLT wherever we want to tablegen-erate an
 /// equivalent at compiler run-time.
 class LLTCodeGen {
 private:
   LLT Ty;
 
 public:
   LLTCodeGen() = default;
   LLTCodeGen(const LLT &Ty) : Ty(Ty) {}
 
   std::string getCxxEnumValue() const {
     std::string Str;
     raw_string_ostream OS(Str);
 
     emitCxxEnumValue(OS);
     return OS.str();
   }
 
   void emitCxxEnumValue(raw_ostream &OS) const {
     if (Ty.isScalar()) {
       OS << "GILLT_s" << Ty.getSizeInBits();
       return;
     }
     if (Ty.isVector()) {
       OS << "GILLT_v" << Ty.getNumElements() << "s" << Ty.getScalarSizeInBits();
       return;
     }
     if (Ty.isPointer()) {
       OS << "GILLT_p" << Ty.getAddressSpace();
       if (Ty.getSizeInBits() > 0)
         OS << "s" << Ty.getSizeInBits();
       return;
     }
     llvm_unreachable("Unhandled LLT");
   }
 
   void emitCxxConstructorCall(raw_ostream &OS) const {
     if (Ty.isScalar()) {
       OS << "LLT::scalar(" << Ty.getSizeInBits() << ")";
       return;
     }
     if (Ty.isVector()) {
       OS << "LLT::vector(" << Ty.getNumElements() << ", "
          << Ty.getScalarSizeInBits() << ")";
       return;
     }
     if (Ty.isPointer() && Ty.getSizeInBits() > 0) {
       OS << "LLT::pointer(" << Ty.getAddressSpace() << ", "
          << Ty.getSizeInBits() << ")";
       return;
     }
     llvm_unreachable("Unhandled LLT");
   }
 
   const LLT &get() const { return Ty; }
 
   /// This ordering is used for std::unique() and llvm::sort(). There's no
   /// particular logic behind the order but either A < B or B < A must be
   /// true if A != B.
   bool operator<(const LLTCodeGen &Other) const {
     if (Ty.isValid() != Other.Ty.isValid())
       return Ty.isValid() < Other.Ty.isValid();
     if (!Ty.isValid())
       return false;
 
     if (Ty.isVector() != Other.Ty.isVector())
       return Ty.isVector() < Other.Ty.isVector();
     if (Ty.isScalar() != Other.Ty.isScalar())
       return Ty.isScalar() < Other.Ty.isScalar();
     if (Ty.isPointer() != Other.Ty.isPointer())
       return Ty.isPointer() < Other.Ty.isPointer();
 
     if (Ty.isPointer() && Ty.getAddressSpace() != Other.Ty.getAddressSpace())
       return Ty.getAddressSpace() < Other.Ty.getAddressSpace();
 
     if (Ty.isVector() && Ty.getNumElements() != Other.Ty.getNumElements())
       return Ty.getNumElements() < Other.Ty.getNumElements();
 
     return Ty.getSizeInBits() < Other.Ty.getSizeInBits();
   }
 
   bool operator==(const LLTCodeGen &B) const { return Ty == B.Ty; }
 };
 
 // Track all types that are used so we can emit the corresponding enum.
 std::set<LLTCodeGen> KnownTypes;
 
 class InstructionMatcher;
 /// Convert an MVT to an equivalent LLT if possible, or the invalid LLT() for
 /// MVTs that don't map cleanly to an LLT (e.g., iPTR, *any, ...).
 static Optional<LLTCodeGen> MVTToLLT(MVT::SimpleValueType SVT) {
   MVT VT(SVT);
 
   if (VT.isVector() && VT.getVectorNumElements() != 1)
     return LLTCodeGen(
         LLT::vector(VT.getVectorNumElements(), VT.getScalarSizeInBits()));
 
   if (VT.isInteger() || VT.isFloatingPoint())
     return LLTCodeGen(LLT::scalar(VT.getSizeInBits()));
   return None;
 }
 
 static std::string explainPredicates(const TreePatternNode *N) {
   std::string Explanation = "";
   StringRef Separator = "";
   for (const auto &P : N->getPredicateFns()) {
     Explanation +=
         (Separator + P.getOrigPatFragRecord()->getRecord()->getName()).str();
     Separator = ", ";
 
     if (P.isAlwaysTrue())
       Explanation += " always-true";
     if (P.isImmediatePattern())
       Explanation += " immediate";
 
     if (P.isUnindexed())
       Explanation += " unindexed";
 
     if (P.isNonExtLoad())
       Explanation += " non-extload";
     if (P.isAnyExtLoad())
       Explanation += " extload";
     if (P.isSignExtLoad())
       Explanation += " sextload";
     if (P.isZeroExtLoad())
       Explanation += " zextload";
 
     if (P.isNonTruncStore())
       Explanation += " non-truncstore";
     if (P.isTruncStore())
       Explanation += " truncstore";
 
     if (Record *VT = P.getMemoryVT())
       Explanation += (" MemVT=" + VT->getName()).str();
     if (Record *VT = P.getScalarMemoryVT())
       Explanation += (" ScalarVT(MemVT)=" + VT->getName()).str();
 
     if (P.isAtomicOrderingMonotonic())
       Explanation += " monotonic";
     if (P.isAtomicOrderingAcquire())
       Explanation += " acquire";
     if (P.isAtomicOrderingRelease())
       Explanation += " release";
     if (P.isAtomicOrderingAcquireRelease())
       Explanation += " acq_rel";
     if (P.isAtomicOrderingSequentiallyConsistent())
       Explanation += " seq_cst";
     if (P.isAtomicOrderingAcquireOrStronger())
       Explanation += " >=acquire";
     if (P.isAtomicOrderingWeakerThanAcquire())
       Explanation += " <acquire";
     if (P.isAtomicOrderingReleaseOrStronger())
       Explanation += " >=release";
     if (P.isAtomicOrderingWeakerThanRelease())
       Explanation += " <release";
   }
   return Explanation;
 }
 
 std::string explainOperator(Record *Operator) {
   if (Operator->isSubClassOf("SDNode"))
     return (" (" + Operator->getValueAsString("Opcode") + ")").str();
 
   if (Operator->isSubClassOf("Intrinsic"))
     return (" (Operator is an Intrinsic, " + Operator->getName() + ")").str();
 
   if (Operator->isSubClassOf("ComplexPattern"))
     return (" (Operator is an unmapped ComplexPattern, " + Operator->getName() +
             ")")
         .str();
 
   if (Operator->isSubClassOf("SDNodeXForm"))
     return (" (Operator is an unmapped SDNodeXForm, " + Operator->getName() +
             ")")
         .str();
 
   return (" (Operator " + Operator->getName() + " not understood)").str();
 }
 
 /// Helper function to let the emitter report skip reason error messages.
 static Error failedImport(const Twine &Reason) {
   return make_error<StringError>(Reason, inconvertibleErrorCode());
 }
 
 static Error isTrivialOperatorNode(const TreePatternNode *N) {
   std::string Explanation = "";
   std::string Separator = "";
 
   bool HasUnsupportedPredicate = false;
   for (const auto &Predicate : N->getPredicateFns()) {
     if (Predicate.isAlwaysTrue())
       continue;
 
     if (Predicate.isImmediatePattern())
       continue;
 
     if (Predicate.isNonExtLoad() || Predicate.isAnyExtLoad() ||
         Predicate.isSignExtLoad() || Predicate.isZeroExtLoad())
       continue;
 
     if (Predicate.isNonTruncStore())
       continue;
 
     if (Predicate.isLoad() && Predicate.getMemoryVT())
       continue;
 
     if (Predicate.isLoad() || Predicate.isStore()) {
       if (Predicate.isUnindexed())
         continue;
     }
 
     if (Predicate.isAtomic() && Predicate.getMemoryVT())
       continue;
 
     if (Predicate.isAtomic() &&
         (Predicate.isAtomicOrderingMonotonic() ||
          Predicate.isAtomicOrderingAcquire() ||
          Predicate.isAtomicOrderingRelease() ||
          Predicate.isAtomicOrderingAcquireRelease() ||
          Predicate.isAtomicOrderingSequentiallyConsistent() ||
          Predicate.isAtomicOrderingAcquireOrStronger() ||
          Predicate.isAtomicOrderingWeakerThanAcquire() ||
          Predicate.isAtomicOrderingReleaseOrStronger() ||
          Predicate.isAtomicOrderingWeakerThanRelease()))
       continue;
 
     HasUnsupportedPredicate = true;
     Explanation = Separator + "Has a predicate (" + explainPredicates(N) + ")";
     Separator = ", ";
     Explanation += (Separator + "first-failing:" +
                     Predicate.getOrigPatFragRecord()->getRecord()->getName())
                        .str();
     break;
   }
 
   if (!HasUnsupportedPredicate)
     return Error::success();
 
   return failedImport(Explanation);
 }
 
 static Record *getInitValueAsRegClass(Init *V) {
   if (DefInit *VDefInit = dyn_cast<DefInit>(V)) {
     if (VDefInit->getDef()->isSubClassOf("RegisterOperand"))
       return VDefInit->getDef()->getValueAsDef("RegClass");
     if (VDefInit->getDef()->isSubClassOf("RegisterClass"))
       return VDefInit->getDef();
   }
   return nullptr;
 }
 
 std::string
 getNameForFeatureBitset(const std::vector<Record *> &FeatureBitset) {
   std::string Name = "GIFBS";
   for (const auto &Feature : FeatureBitset)
     Name += ("_" + Feature->getName()).str();
   return Name;
 }
 
 //===- MatchTable Helpers -------------------------------------------------===//
 
 class MatchTable;
 
 /// A record to be stored in a MatchTable.
 ///
 /// This class represents any and all output that may be required to emit the
 /// MatchTable. Instances  are most often configured to represent an opcode or
 /// value that will be emitted to the table with some formatting but it can also
 /// represent commas, comments, and other formatting instructions.
 struct MatchTableRecord {
   enum RecordFlagsBits {
     MTRF_None = 0x0,
     /// Causes EmitStr to be formatted as comment when emitted.
     MTRF_Comment = 0x1,
     /// Causes the record value to be followed by a comma when emitted.
     MTRF_CommaFollows = 0x2,
     /// Causes the record value to be followed by a line break when emitted.
     MTRF_LineBreakFollows = 0x4,
     /// Indicates that the record defines a label and causes an additional
     /// comment to be emitted containing the index of the label.
     MTRF_Label = 0x8,
     /// Causes the record to be emitted as the index of the label specified by
     /// LabelID along with a comment indicating where that label is.
     MTRF_JumpTarget = 0x10,
     /// Causes the formatter to add a level of indentation before emitting the
     /// record.
     MTRF_Indent = 0x20,
     /// Causes the formatter to remove a level of indentation after emitting the
     /// record.
     MTRF_Outdent = 0x40,
   };
 
   /// When MTRF_Label or MTRF_JumpTarget is used, indicates a label id to
   /// reference or define.
   unsigned LabelID;
   /// The string to emit. Depending on the MTRF_* flags it may be a comment, a
   /// value, a label name.
   std::string EmitStr;
 
 private:
   /// The number of MatchTable elements described by this record. Comments are 0
   /// while values are typically 1. Values >1 may occur when we need to emit
   /// values that exceed the size of a MatchTable element.
   unsigned NumElements;
 
 public:
   /// A bitfield of RecordFlagsBits flags.
   unsigned Flags;
 
   /// The actual run-time value, if known
   int64_t RawValue;
 
   MatchTableRecord(Optional<unsigned> LabelID_, StringRef EmitStr,
                    unsigned NumElements, unsigned Flags,
                    int64_t RawValue = std::numeric_limits<int64_t>::min())
       : LabelID(LabelID_.hasValue() ? LabelID_.getValue() : ~0u),
         EmitStr(EmitStr), NumElements(NumElements), Flags(Flags),
         RawValue(RawValue) {
 
     assert((!LabelID_.hasValue() || LabelID != ~0u) &&
            "This value is reserved for non-labels");
   }
   MatchTableRecord(const MatchTableRecord &Other) = default;
   MatchTableRecord(MatchTableRecord &&Other) = default;
 
   /// Useful if a Match Table Record gets optimized out
   void turnIntoComment() {
     Flags |= MTRF_Comment;
     Flags &= ~MTRF_CommaFollows;
     NumElements = 0;
   }
 
   /// For Jump Table generation purposes
   bool operator<(const MatchTableRecord &Other) const {
     return RawValue < Other.RawValue;
   }
   int64_t getRawValue() const { return RawValue; }
 
   void emit(raw_ostream &OS, bool LineBreakNextAfterThis,
             const MatchTable &Table) const;
   unsigned size() const { return NumElements; }
 };
 
 class Matcher;
 
 /// Holds the contents of a generated MatchTable to enable formatting and the
 /// necessary index tracking needed to support GIM_Try.
 class MatchTable {
   /// An unique identifier for the table. The generated table will be named
   /// MatchTable${ID}.
   unsigned ID;
   /// The records that make up the table. Also includes comments describing the
   /// values being emitted and line breaks to format it.
   std::vector<MatchTableRecord> Contents;
   /// The currently defined labels.
   DenseMap<unsigned, unsigned> LabelMap;
   /// Tracks the sum of MatchTableRecord::NumElements as the table is built.
   unsigned CurrentSize = 0;
   /// A unique identifier for a MatchTable label.
   unsigned CurrentLabelID = 0;
   /// Determines if the table should be instrumented for rule coverage tracking.
   bool IsWithCoverage;
 
 public:
   static MatchTableRecord LineBreak;
   static MatchTableRecord Comment(StringRef Comment) {
     return MatchTableRecord(None, Comment, 0, MatchTableRecord::MTRF_Comment);
   }
   static MatchTableRecord Opcode(StringRef Opcode, int IndentAdjust = 0) {
     unsigned ExtraFlags = 0;
     if (IndentAdjust > 0)
       ExtraFlags |= MatchTableRecord::MTRF_Indent;
     if (IndentAdjust < 0)
       ExtraFlags |= MatchTableRecord::MTRF_Outdent;
 
     return MatchTableRecord(None, Opcode, 1,
                             MatchTableRecord::MTRF_CommaFollows | ExtraFlags);
   }
   static MatchTableRecord NamedValue(StringRef NamedValue) {
     return MatchTableRecord(None, NamedValue, 1,
                             MatchTableRecord::MTRF_CommaFollows);
   }
   static MatchTableRecord NamedValue(StringRef NamedValue, int64_t RawValue) {
     return MatchTableRecord(None, NamedValue, 1,
                             MatchTableRecord::MTRF_CommaFollows, RawValue);
   }
   static MatchTableRecord NamedValue(StringRef Namespace,
                                      StringRef NamedValue) {
     return MatchTableRecord(None, (Namespace + "::" + NamedValue).str(), 1,
                             MatchTableRecord::MTRF_CommaFollows);
   }
   static MatchTableRecord NamedValue(StringRef Namespace, StringRef NamedValue,
                                      int64_t RawValue) {
     return MatchTableRecord(None, (Namespace + "::" + NamedValue).str(), 1,
                             MatchTableRecord::MTRF_CommaFollows, RawValue);
   }
   static MatchTableRecord IntValue(int64_t IntValue) {
     return MatchTableRecord(None, llvm::to_string(IntValue), 1,
                             MatchTableRecord::MTRF_CommaFollows);
   }
   static MatchTableRecord Label(unsigned LabelID) {
     return MatchTableRecord(LabelID, "Label " + llvm::to_string(LabelID), 0,
                             MatchTableRecord::MTRF_Label |
                                 MatchTableRecord::MTRF_Comment |
                                 MatchTableRecord::MTRF_LineBreakFollows);
   }
   static MatchTableRecord JumpTarget(unsigned LabelID) {
     return MatchTableRecord(LabelID, "Label " + llvm::to_string(LabelID), 1,
                             MatchTableRecord::MTRF_JumpTarget |
                                 MatchTableRecord::MTRF_Comment |
                                 MatchTableRecord::MTRF_CommaFollows);
   }
 
   static MatchTable buildTable(ArrayRef<Matcher *> Rules, bool WithCoverage);
 
   MatchTable(bool WithCoverage, unsigned ID = 0)
       : ID(ID), IsWithCoverage(WithCoverage) {}
 
   bool isWithCoverage() const { return IsWithCoverage; }
 
   void push_back(const MatchTableRecord &Value) {
     if (Value.Flags & MatchTableRecord::MTRF_Label)
       defineLabel(Value.LabelID);
     Contents.push_back(Value);
     CurrentSize += Value.size();
   }
 
   unsigned allocateLabelID() { return CurrentLabelID++; }
 
   void defineLabel(unsigned LabelID) {
     LabelMap.insert(std::make_pair(LabelID, CurrentSize));
   }
 
   unsigned getLabelIndex(unsigned LabelID) const {
     const auto I = LabelMap.find(LabelID);
     assert(I != LabelMap.end() && "Use of undeclared label");
     return I->second;
   }
 
   void emitUse(raw_ostream &OS) const { OS << "MatchTable" << ID; }
 
   void emitDeclaration(raw_ostream &OS) const {
     unsigned Indentation = 4;
     OS << "  constexpr static int64_t MatchTable" << ID << "[] = {";
     LineBreak.emit(OS, true, *this);
     OS << std::string(Indentation, ' ');
 
     for (auto I = Contents.begin(), E = Contents.end(); I != E;
          ++I) {
       bool LineBreakIsNext = false;
       const auto &NextI = std::next(I);
 
       if (NextI != E) {
         if (NextI->EmitStr == "" &&
             NextI->Flags == MatchTableRecord::MTRF_LineBreakFollows)
           LineBreakIsNext = true;
       }
 
       if (I->Flags & MatchTableRecord::MTRF_Indent)
         Indentation += 2;
 
       I->emit(OS, LineBreakIsNext, *this);
       if (I->Flags & MatchTableRecord::MTRF_LineBreakFollows)
         OS << std::string(Indentation, ' ');
 
       if (I->Flags & MatchTableRecord::MTRF_Outdent)
         Indentation -= 2;
     }
     OS << "};\n";
   }
 };
 
 MatchTableRecord MatchTable::LineBreak = {
     None, "" /* Emit String */, 0 /* Elements */,
     MatchTableRecord::MTRF_LineBreakFollows};
 
 void MatchTableRecord::emit(raw_ostream &OS, bool LineBreakIsNextAfterThis,
                             const MatchTable &Table) const {
   bool UseLineComment =
       LineBreakIsNextAfterThis | (Flags & MTRF_LineBreakFollows);
   if (Flags & (MTRF_JumpTarget | MTRF_CommaFollows))
     UseLineComment = false;
 
   if (Flags & MTRF_Comment)
     OS << (UseLineComment ? "// " : "/*");
 
   OS << EmitStr;
   if (Flags & MTRF_Label)
     OS << ": @" << Table.getLabelIndex(LabelID);
 
   if (Flags & MTRF_Comment && !UseLineComment)
     OS << "*/";
 
   if (Flags & MTRF_JumpTarget) {
     if (Flags & MTRF_Comment)
       OS << " ";
     OS << Table.getLabelIndex(LabelID);
   }
 
   if (Flags & MTRF_CommaFollows) {
     OS << ",";
     if (!LineBreakIsNextAfterThis && !(Flags & MTRF_LineBreakFollows))
       OS << " ";
   }
 
   if (Flags & MTRF_LineBreakFollows)
     OS << "\n";
 }
 
 MatchTable &operator<<(MatchTable &Table, const MatchTableRecord &Value) {
   Table.push_back(Value);
   return Table;
 }
 
 //===- Matchers -----------------------------------------------------------===//
 
 class OperandMatcher;
 class MatchAction;
 class PredicateMatcher;
 class RuleMatcher;
 
 class Matcher {
 public:
   virtual ~Matcher() = default;
   virtual void optimize() {}
   virtual void emit(MatchTable &Table) = 0;
 
   virtual bool hasFirstCondition() const = 0;
   virtual const PredicateMatcher &getFirstCondition() const = 0;
   virtual std::unique_ptr<PredicateMatcher> popFirstCondition() = 0;
 };
 
 MatchTable MatchTable::buildTable(ArrayRef<Matcher *> Rules,
                                   bool WithCoverage) {
   MatchTable Table(WithCoverage);
   for (Matcher *Rule : Rules)
     Rule->emit(Table);
 
   return Table << MatchTable::Opcode("GIM_Reject") << MatchTable::LineBreak;
 }
 
 class GroupMatcher final : public Matcher {
   /// Conditions that form a common prefix of all the matchers contained.
   SmallVector<std::unique_ptr<PredicateMatcher>, 1> Conditions;
 
   /// All the nested matchers, sharing a common prefix.
   std::vector<Matcher *> Matchers;
 
   /// An owning collection for any auxiliary matchers created while optimizing
   /// nested matchers contained.
   std::vector<std::unique_ptr<Matcher>> MatcherStorage;
 
 public:
   /// Add a matcher to the collection of nested matchers if it meets the
   /// requirements, and return true. If it doesn't, do nothing and return false.
   ///
   /// Expected to preserve its argument, so it could be moved out later on.
   bool addMatcher(Matcher &Candidate);
 
   /// Mark the matcher as fully-built and ensure any invariants expected by both
   /// optimize() and emit(...) methods. Generally, both sequences of calls
   /// are expected to lead to a sensible result:
   ///
   /// addMatcher(...)*; finalize(); optimize(); emit(...); and
   /// addMatcher(...)*; finalize(); emit(...);
   ///
   /// or generally
   ///
   /// addMatcher(...)*; finalize(); { optimize()*; emit(...); }*
   ///
   /// Multiple calls to optimize() are expected to be handled gracefully, though
   /// optimize() is not expected to be idempotent. Multiple calls to finalize()
   /// aren't generally supported. emit(...) is expected to be non-mutating and
   /// producing the exact same results upon repeated calls.
   ///
   /// addMatcher() calls after the finalize() call are not supported.
   ///
   /// finalize() and optimize() are both allowed to mutate the contained
   /// matchers, so moving them out after finalize() is not supported.
   void finalize();
   void optimize() override {}
   void emit(MatchTable &Table) override;
 
   /// Could be used to move out the matchers added previously, unless finalize()
   /// has been already called. If any of the matchers are moved out, the group
   /// becomes safe to destroy, but not safe to re-use for anything else.
   iterator_range<std::vector<Matcher *>::iterator> matchers() {
     return make_range(Matchers.begin(), Matchers.end());
   }
   size_t size() const { return Matchers.size(); }
   bool empty() const { return Matchers.empty(); }
 
   std::unique_ptr<PredicateMatcher> popFirstCondition() override {
     assert(!Conditions.empty() &&
            "Trying to pop a condition from a condition-less group");
     std::unique_ptr<PredicateMatcher> P = std::move(Conditions.front());
     Conditions.erase(Conditions.begin());
     return P;
   }
   const PredicateMatcher &getFirstCondition() const override {
     assert(!Conditions.empty() &&
            "Trying to get a condition from a condition-less group");
     return *Conditions.front();
   }
   bool hasFirstCondition() const override { return !Conditions.empty(); }
 
 private:
   /// See if a candidate matcher could be added to this group solely by
   /// analyzing its first condition.
   bool candidateConditionMatches(const PredicateMatcher &Predicate) const;
 };
 
 /// Generates code to check that a match rule matches.
 class RuleMatcher : public Matcher {
 public:
   using ActionList = std::list<std::unique_ptr<MatchAction>>;
   using action_iterator = ActionList::iterator;
 
 protected:
   /// A list of matchers that all need to succeed for the current rule to match.
   /// FIXME: This currently supports a single match position but could be
   /// extended to support multiple positions to support div/rem fusion or
   /// load-multiple instructions.
   using MatchersTy = std::vector<std::unique_ptr<InstructionMatcher>> ;
   MatchersTy Matchers;
 
   /// A list of actions that need to be taken when all predicates in this rule
   /// have succeeded.
   ActionList Actions;
 
   using DefinedInsnVariablesMap = std::map<InstructionMatcher *, unsigned>;
 
   /// A map of instruction matchers to the local variables
   DefinedInsnVariablesMap InsnVariableIDs;
 
   using MutatableInsnSet = SmallPtrSet<InstructionMatcher *, 4>;
 
   // The set of instruction matchers that have not yet been claimed for mutation
   // by a BuildMI.
   MutatableInsnSet MutatableInsns;
 
   /// A map of named operands defined by the matchers that may be referenced by
   /// the renderers.
   StringMap<OperandMatcher *> DefinedOperands;
 
   /// ID for the next instruction variable defined with implicitlyDefineInsnVar()
   unsigned NextInsnVarID;
 
   /// ID for the next output instruction allocated with allocateOutputInsnID()
   unsigned NextOutputInsnID;
 
   /// ID for the next temporary register ID allocated with allocateTempRegID()
   unsigned NextTempRegID;
 
   std::vector<Record *> RequiredFeatures;
   std::vector<std::unique_ptr<PredicateMatcher>> EpilogueMatchers;
 
   ArrayRef<SMLoc> SrcLoc;
 
   typedef std::tuple<Record *, unsigned, unsigned>
       DefinedComplexPatternSubOperand;
   typedef StringMap<DefinedComplexPatternSubOperand>
       DefinedComplexPatternSubOperandMap;
   /// A map of Symbolic Names to ComplexPattern sub-operands.
   DefinedComplexPatternSubOperandMap ComplexSubOperands;
 
   uint64_t RuleID;
   static uint64_t NextRuleID;
 
 public:
   RuleMatcher(ArrayRef<SMLoc> SrcLoc)
       : Matchers(), Actions(), InsnVariableIDs(), MutatableInsns(),
         DefinedOperands(), NextInsnVarID(0), NextOutputInsnID(0),
         NextTempRegID(0), SrcLoc(SrcLoc), ComplexSubOperands(),
         RuleID(NextRuleID++) {}
   RuleMatcher(RuleMatcher &&Other) = default;
   RuleMatcher &operator=(RuleMatcher &&Other) = default;
 
   uint64_t getRuleID() const { return RuleID; }
 
   InstructionMatcher &addInstructionMatcher(StringRef SymbolicName);
   void addRequiredFeature(Record *Feature);
   const std::vector<Record *> &getRequiredFeatures() const;
 
   template <class Kind, class... Args> Kind &addAction(Args &&... args);
   template <class Kind, class... Args>
   action_iterator insertAction(action_iterator InsertPt, Args &&... args);
 
   /// Define an instruction without emitting any code to do so.
   unsigned implicitlyDefineInsnVar(InstructionMatcher &Matcher);
 
   unsigned getInsnVarID(InstructionMatcher &InsnMatcher) const;
   DefinedInsnVariablesMap::const_iterator defined_insn_vars_begin() const {
     return InsnVariableIDs.begin();
   }
   DefinedInsnVariablesMap::const_iterator defined_insn_vars_end() const {
     return InsnVariableIDs.end();
   }
   iterator_range<typename DefinedInsnVariablesMap::const_iterator>
   defined_insn_vars() const {
     return make_range(defined_insn_vars_begin(), defined_insn_vars_end());
   }
 
   MutatableInsnSet::const_iterator mutatable_insns_begin() const {
     return MutatableInsns.begin();
   }
   MutatableInsnSet::const_iterator mutatable_insns_end() const {
     return MutatableInsns.end();
   }
   iterator_range<typename MutatableInsnSet::const_iterator>
   mutatable_insns() const {
     return make_range(mutatable_insns_begin(), mutatable_insns_end());
   }
   void reserveInsnMatcherForMutation(InstructionMatcher *InsnMatcher) {
     bool R = MutatableInsns.erase(InsnMatcher);
     assert(R && "Reserving a mutatable insn that isn't available");
     (void)R;
   }
 
   action_iterator actions_begin() { return Actions.begin(); }
   action_iterator actions_end() { return Actions.end(); }
   iterator_range<action_iterator> actions() {
     return make_range(actions_begin(), actions_end());
   }
 
   void defineOperand(StringRef SymbolicName, OperandMatcher &OM);
 
   void defineComplexSubOperand(StringRef SymbolicName, Record *ComplexPattern,
                                unsigned RendererID, unsigned SubOperandID) {
     assert(ComplexSubOperands.count(SymbolicName) == 0 && "Already defined");
     ComplexSubOperands[SymbolicName] =
         std::make_tuple(ComplexPattern, RendererID, SubOperandID);
   }
   Optional<DefinedComplexPatternSubOperand>
   getComplexSubOperand(StringRef SymbolicName) const {
     const auto &I = ComplexSubOperands.find(SymbolicName);
     if (I == ComplexSubOperands.end())
       return None;
     return I->second;
   }
 
   InstructionMatcher &getInstructionMatcher(StringRef SymbolicName) const;
   const OperandMatcher &getOperandMatcher(StringRef Name) const;
 
   void optimize() override;
   void emit(MatchTable &Table) override;
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   bool isHigherPriorityThan(const RuleMatcher &B) const;
 
   /// Report the maximum number of temporary operands needed by the rule
   /// matcher.
   unsigned countRendererFns() const;
 
   std::unique_ptr<PredicateMatcher> popFirstCondition() override;
   const PredicateMatcher &getFirstCondition() const override;
-  LLTCodeGen getFirstConditionAsRootType();
   bool hasFirstCondition() const override;
   unsigned getNumOperands() const;
   StringRef getOpcode() const;
 
   // FIXME: Remove this as soon as possible
   InstructionMatcher &insnmatchers_front() const { return *Matchers.front(); }
 
   unsigned allocateOutputInsnID() { return NextOutputInsnID++; }
   unsigned allocateTempRegID() { return NextTempRegID++; }
 
   iterator_range<MatchersTy::iterator> insnmatchers() {
     return make_range(Matchers.begin(), Matchers.end());
   }
   bool insnmatchers_empty() const { return Matchers.empty(); }
   void insnmatchers_pop_front() { Matchers.erase(Matchers.begin()); }
 };
 
 uint64_t RuleMatcher::NextRuleID = 0;
 
 using action_iterator = RuleMatcher::action_iterator;
 
 template <class PredicateTy> class PredicateListMatcher {
 private:
   /// Template instantiations should specialize this to return a string to use
   /// for the comment emitted when there are no predicates.
   std::string getNoPredicateComment() const;
 
 protected:
   using PredicatesTy = std::deque<std::unique_ptr<PredicateTy>>;
   PredicatesTy Predicates;
 
   /// Track if the list of predicates was manipulated by one of the optimization
   /// methods.
   bool Optimized = false;
 
 public:
   /// Construct a new predicate and add it to the matcher.
   template <class Kind, class... Args>
   Optional<Kind *> addPredicate(Args &&... args);
 
   typename PredicatesTy::iterator predicates_begin() {
     return Predicates.begin();
   }
   typename PredicatesTy::iterator predicates_end() {
     return Predicates.end();
   }
   iterator_range<typename PredicatesTy::iterator> predicates() {
     return make_range(predicates_begin(), predicates_end());
   }
   typename PredicatesTy::size_type predicates_size() const {
     return Predicates.size();
   }
   bool predicates_empty() const { return Predicates.empty(); }
 
   std::unique_ptr<PredicateTy> predicates_pop_front() {
     std::unique_ptr<PredicateTy> Front = std::move(Predicates.front());
     Predicates.pop_front();
     Optimized = true;
     return Front;
   }
 
   void prependPredicate(std::unique_ptr<PredicateTy> &&Predicate) {
     Predicates.push_front(std::move(Predicate));
   }
 
   void eraseNullPredicates() {
     const auto NewEnd =
         std::stable_partition(Predicates.begin(), Predicates.end(),
                               std::logical_not<std::unique_ptr<PredicateTy>>());
     if (NewEnd != Predicates.begin()) {
       Predicates.erase(Predicates.begin(), NewEnd);
       Optimized = true;
     }
   }
 
   /// Emit MatchTable opcodes that tests whether all the predicates are met.
   template <class... Args>
   void emitPredicateListOpcodes(MatchTable &Table, Args &&... args) {
     if (Predicates.empty() && !Optimized) {
       Table << MatchTable::Comment(getNoPredicateComment())
             << MatchTable::LineBreak;
       return;
     }
 
     for (const auto &Predicate : predicates())
       Predicate->emitPredicateOpcodes(Table, std::forward<Args>(args)...);
   }
 };
 
 class PredicateMatcher {
 public:
   /// This enum is used for RTTI and also defines the priority that is given to
   /// the predicate when generating the matcher code. Kinds with higher priority
   /// must be tested first.
   ///
   /// The relative priority of OPM_LLT, OPM_RegBank, and OPM_MBB do not matter
   /// but OPM_Int must have priority over OPM_RegBank since constant integers
   /// are represented by a virtual register defined by a G_CONSTANT instruction.
   ///
   /// Note: The relative priority between IPM_ and OPM_ does not matter, they
   /// are currently not compared between each other.
   enum PredicateKind {
     IPM_Opcode,
     IPM_NumOperands,
     IPM_ImmPredicate,
     IPM_AtomicOrderingMMO,
     IPM_MemoryLLTSize,
     IPM_MemoryVsLLTSize,
     OPM_SameOperand,
     OPM_ComplexPattern,
     OPM_IntrinsicID,
     OPM_Instruction,
     OPM_Int,
     OPM_LiteralInt,
     OPM_LLT,
     OPM_PointerToAny,
     OPM_RegBank,
     OPM_MBB,
   };
 
 protected:
   PredicateKind Kind;
   unsigned InsnVarID;
   unsigned OpIdx;
 
 public:
   PredicateMatcher(PredicateKind Kind, unsigned InsnVarID, unsigned OpIdx = ~0)
       : Kind(Kind), InsnVarID(InsnVarID), OpIdx(OpIdx) {}
 
   unsigned getInsnVarID() const { return InsnVarID; }
   unsigned getOpIdx() const { return OpIdx; }
 
   virtual ~PredicateMatcher() = default;
   /// Emit MatchTable opcodes that check the predicate for the given operand.
   virtual void emitPredicateOpcodes(MatchTable &Table,
                                     RuleMatcher &Rule) const = 0;
 
   PredicateKind getKind() const { return Kind; }
 
   virtual bool isIdentical(const PredicateMatcher &B) const {
     return B.getKind() == getKind() && InsnVarID == B.InsnVarID &&
            OpIdx == B.OpIdx;
   }
 
   virtual bool isIdenticalDownToValue(const PredicateMatcher &B) const {
     return hasValue() && PredicateMatcher::isIdentical(B);
   }
 
   virtual MatchTableRecord getValue() const {
     assert(hasValue() && "Can not get a value of a value-less predicate!");
     llvm_unreachable("Not implemented yet");
   }
   virtual bool hasValue() const { return false; }
 
   /// Report the maximum number of temporary operands needed by the predicate
   /// matcher.
   virtual unsigned countRendererFns() const { return 0; }
 };
 
 /// Generates code to check a predicate of an operand.
 ///
 /// Typical predicates include:
 /// * Operand is a particular register.
 /// * Operand is assigned a particular register bank.
 /// * Operand is an MBB.
 class OperandPredicateMatcher : public PredicateMatcher {
 public:
   OperandPredicateMatcher(PredicateKind Kind, unsigned InsnVarID,
                           unsigned OpIdx)
       : PredicateMatcher(Kind, InsnVarID, OpIdx) {}
   virtual ~OperandPredicateMatcher() {}
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   virtual bool isHigherPriorityThan(const OperandPredicateMatcher &B) const;
 };
 
 template <>
 std::string
 PredicateListMatcher<OperandPredicateMatcher>::getNoPredicateComment() const {
   return "No operand predicates";
 }
 
 /// Generates code to check that a register operand is defined by the same exact
 /// one as another.
 class SameOperandMatcher : public OperandPredicateMatcher {
   std::string MatchingName;
 
 public:
   SameOperandMatcher(unsigned InsnVarID, unsigned OpIdx, StringRef MatchingName)
       : OperandPredicateMatcher(OPM_SameOperand, InsnVarID, OpIdx),
         MatchingName(MatchingName) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_SameOperand;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override;
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            MatchingName == cast<SameOperandMatcher>(&B)->MatchingName;
   }
 };
 
 /// Generates code to check that an operand is a particular LLT.
 class LLTOperandMatcher : public OperandPredicateMatcher {
 protected:
   LLTCodeGen Ty;
 
 public:
   static std::map<LLTCodeGen, unsigned> TypeIDValues;
 
   static void initTypeIDValuesMap() {
     TypeIDValues.clear();
 
     unsigned ID = 0;
     for (const LLTCodeGen LLTy : KnownTypes)
       TypeIDValues[LLTy] = ID++;
   }
 
   LLTOperandMatcher(unsigned InsnVarID, unsigned OpIdx, const LLTCodeGen &Ty)
       : OperandPredicateMatcher(OPM_LLT, InsnVarID, OpIdx), Ty(Ty) {
     KnownTypes.insert(Ty);
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_LLT;
   }
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            Ty == cast<LLTOperandMatcher>(&B)->Ty;
   }
   MatchTableRecord getValue() const override {
     const auto VI = TypeIDValues.find(Ty);
     if (VI == TypeIDValues.end())
       return MatchTable::NamedValue(getTy().getCxxEnumValue());
     return MatchTable::NamedValue(getTy().getCxxEnumValue(), VI->second);
   }
   bool hasValue() const override {
     if (TypeIDValues.size() != KnownTypes.size())
       initTypeIDValuesMap();
     return TypeIDValues.count(Ty);
   }
 
   LLTCodeGen getTy() const { return Ty; }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckType") << MatchTable::Comment("MI")
           << MatchTable::IntValue(InsnVarID) << MatchTable::Comment("Op")
           << MatchTable::IntValue(OpIdx) << MatchTable::Comment("Type")
           << getValue() << MatchTable::LineBreak;
   }
 };
 
 std::map<LLTCodeGen, unsigned> LLTOperandMatcher::TypeIDValues;
 
 /// Generates code to check that an operand is a pointer to any address space.
 ///
 /// In SelectionDAG, the types did not describe pointers or address spaces. As a
 /// result, iN is used to describe a pointer of N bits to any address space and
 /// PatFrag predicates are typically used to constrain the address space. There's
 /// no reliable means to derive the missing type information from the pattern so
 /// imported rules must test the components of a pointer separately.
 ///
 /// If SizeInBits is zero, then the pointer size will be obtained from the
 /// subtarget.
 class PointerToAnyOperandMatcher : public OperandPredicateMatcher {
 protected:
   unsigned SizeInBits;
 
 public:
   PointerToAnyOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                              unsigned SizeInBits)
       : OperandPredicateMatcher(OPM_PointerToAny, InsnVarID, OpIdx),
         SizeInBits(SizeInBits) {}
 
   static bool classof(const OperandPredicateMatcher *P) {
     return P->getKind() == OPM_PointerToAny;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckPointerToAny")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::Comment("SizeInBits")
           << MatchTable::IntValue(SizeInBits) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that an operand is a particular target constant.
 class ComplexPatternOperandMatcher : public OperandPredicateMatcher {
 protected:
   const OperandMatcher &Operand;
   const Record &TheDef;
 
   unsigned getAllocatedTemporariesBaseID() const;
 
 public:
   bool isIdentical(const PredicateMatcher &B) const override { return false; }
 
   ComplexPatternOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                                const OperandMatcher &Operand,
                                const Record &TheDef)
       : OperandPredicateMatcher(OPM_ComplexPattern, InsnVarID, OpIdx),
         Operand(Operand), TheDef(TheDef) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_ComplexPattern;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     unsigned ID = getAllocatedTemporariesBaseID();
     Table << MatchTable::Opcode("GIM_CheckComplexPattern")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::Comment("Renderer") << MatchTable::IntValue(ID)
           << MatchTable::NamedValue(("GICP_" + TheDef.getName()).str())
           << MatchTable::LineBreak;
   }
 
   unsigned countRendererFns() const override {
     return 1;
   }
 };
 
 /// Generates code to check that an operand is in a particular register bank.
 class RegisterBankOperandMatcher : public OperandPredicateMatcher {
 protected:
   const CodeGenRegisterClass &RC;
 
 public:
   RegisterBankOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                              const CodeGenRegisterClass &RC)
       : OperandPredicateMatcher(OPM_RegBank, InsnVarID, OpIdx), RC(RC) {}
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            RC.getDef() == cast<RegisterBankOperandMatcher>(&B)->RC.getDef();
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_RegBank;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckRegBankForClass")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::Comment("RC")
           << MatchTable::NamedValue(RC.getQualifiedName() + "RegClassID")
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that an operand is a basic block.
 class MBBOperandMatcher : public OperandPredicateMatcher {
 public:
   MBBOperandMatcher(unsigned InsnVarID, unsigned OpIdx)
       : OperandPredicateMatcher(OPM_MBB, InsnVarID, OpIdx) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_MBB;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckIsMBB") << MatchTable::Comment("MI")
           << MatchTable::IntValue(InsnVarID) << MatchTable::Comment("Op")
           << MatchTable::IntValue(OpIdx) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that an operand is a G_CONSTANT with a particular
 /// int.
 class ConstantIntOperandMatcher : public OperandPredicateMatcher {
 protected:
   int64_t Value;
 
 public:
   ConstantIntOperandMatcher(unsigned InsnVarID, unsigned OpIdx, int64_t Value)
       : OperandPredicateMatcher(OPM_Int, InsnVarID, OpIdx), Value(Value) {}
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            Value == cast<ConstantIntOperandMatcher>(&B)->Value;
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_Int;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckConstantInt")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::IntValue(Value) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that an operand is a raw int (where MO.isImm() or
 /// MO.isCImm() is true).
 class LiteralIntOperandMatcher : public OperandPredicateMatcher {
 protected:
   int64_t Value;
 
 public:
   LiteralIntOperandMatcher(unsigned InsnVarID, unsigned OpIdx, int64_t Value)
       : OperandPredicateMatcher(OPM_LiteralInt, InsnVarID, OpIdx),
         Value(Value) {}
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            Value == cast<LiteralIntOperandMatcher>(&B)->Value;
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_LiteralInt;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckLiteralInt")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::IntValue(Value) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that an operand is an intrinsic ID.
 class IntrinsicIDOperandMatcher : public OperandPredicateMatcher {
 protected:
   const CodeGenIntrinsic *II;
 
 public:
   IntrinsicIDOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                             const CodeGenIntrinsic *II)
       : OperandPredicateMatcher(OPM_IntrinsicID, InsnVarID, OpIdx), II(II) {}
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            II == cast<IntrinsicIDOperandMatcher>(&B)->II;
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_IntrinsicID;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckIntrinsicID")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::NamedValue("Intrinsic::" + II->EnumName)
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that a set of predicates match for a particular
 /// operand.
 class OperandMatcher : public PredicateListMatcher<OperandPredicateMatcher> {
 protected:
   InstructionMatcher &Insn;
   unsigned OpIdx;
   std::string SymbolicName;
 
   /// The index of the first temporary variable allocated to this operand. The
   /// number of allocated temporaries can be found with
   /// countRendererFns().
   unsigned AllocatedTemporariesBaseID;
 
 public:
   OperandMatcher(InstructionMatcher &Insn, unsigned OpIdx,
                  const std::string &SymbolicName,
                  unsigned AllocatedTemporariesBaseID)
       : Insn(Insn), OpIdx(OpIdx), SymbolicName(SymbolicName),
         AllocatedTemporariesBaseID(AllocatedTemporariesBaseID) {}
 
   bool hasSymbolicName() const { return !SymbolicName.empty(); }
   const StringRef getSymbolicName() const { return SymbolicName; }
   void setSymbolicName(StringRef Name) {
     assert(SymbolicName.empty() && "Operand already has a symbolic name");
     SymbolicName = Name;
   }
 
   /// Construct a new operand predicate and add it to the matcher.
   template <class Kind, class... Args>
   Optional<Kind *> addPredicate(Args &&... args) {
     if (isSameAsAnotherOperand())
       return None;
     Predicates.emplace_back(llvm::make_unique<Kind>(
         getInsnVarID(), getOpIdx(), std::forward<Args>(args)...));
     return static_cast<Kind *>(Predicates.back().get());
   }
 
   unsigned getOpIdx() const { return OpIdx; }
   unsigned getInsnVarID() const;
 
   std::string getOperandExpr(unsigned InsnVarID) const {
     return "State.MIs[" + llvm::to_string(InsnVarID) + "]->getOperand(" +
            llvm::to_string(OpIdx) + ")";
   }
 
   InstructionMatcher &getInstructionMatcher() const { return Insn; }
 
   Error addTypeCheckPredicate(const TypeSetByHwMode &VTy,
                               bool OperandIsAPointer);
 
   /// Emit MatchTable opcodes that test whether the instruction named in
   /// InsnVarID matches all the predicates and all the operands.
   void emitPredicateOpcodes(MatchTable &Table, RuleMatcher &Rule) {
     if (!Optimized) {
       std::string Comment;
       raw_string_ostream CommentOS(Comment);
       CommentOS << "MIs[" << getInsnVarID() << "] ";
       if (SymbolicName.empty())
         CommentOS << "Operand " << OpIdx;
       else
         CommentOS << SymbolicName;
       Table << MatchTable::Comment(CommentOS.str()) << MatchTable::LineBreak;
     }
 
     emitPredicateListOpcodes(Table, Rule);
   }
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   bool isHigherPriorityThan(OperandMatcher &B) {
     // Operand matchers involving more predicates have higher priority.
     if (predicates_size() > B.predicates_size())
       return true;
     if (predicates_size() < B.predicates_size())
       return false;
 
     // This assumes that predicates are added in a consistent order.
     for (auto &&Predicate : zip(predicates(), B.predicates())) {
       if (std::get<0>(Predicate)->isHigherPriorityThan(*std::get<1>(Predicate)))
         return true;
       if (std::get<1>(Predicate)->isHigherPriorityThan(*std::get<0>(Predicate)))
         return false;
     }
 
     return false;
   };
 
   /// Report the maximum number of temporary operands needed by the operand
   /// matcher.
   unsigned countRendererFns() {
     return std::accumulate(
         predicates().begin(), predicates().end(), 0,
         [](unsigned A,
            const std::unique_ptr<OperandPredicateMatcher> &Predicate) {
           return A + Predicate->countRendererFns();
         });
   }
 
   unsigned getAllocatedTemporariesBaseID() const {
     return AllocatedTemporariesBaseID;
   }
 
   bool isSameAsAnotherOperand() {
     for (const auto &Predicate : predicates())
       if (isa<SameOperandMatcher>(Predicate))
         return true;
     return false;
   }
 };
 
 Error OperandMatcher::addTypeCheckPredicate(const TypeSetByHwMode &VTy,
                                             bool OperandIsAPointer) {
   if (!VTy.isMachineValueType())
     return failedImport("unsupported typeset");
 
   if (VTy.getMachineValueType() == MVT::iPTR && OperandIsAPointer) {
     addPredicate<PointerToAnyOperandMatcher>(0);
     return Error::success();
   }
 
   auto OpTyOrNone = MVTToLLT(VTy.getMachineValueType().SimpleTy);
   if (!OpTyOrNone)
     return failedImport("unsupported type");
 
   if (OperandIsAPointer)
     addPredicate<PointerToAnyOperandMatcher>(OpTyOrNone->get().getSizeInBits());
   else
     addPredicate<LLTOperandMatcher>(*OpTyOrNone);
   return Error::success();
 }
 
 unsigned ComplexPatternOperandMatcher::getAllocatedTemporariesBaseID() const {
   return Operand.getAllocatedTemporariesBaseID();
 }
 
 /// Generates code to check a predicate on an instruction.
 ///
 /// Typical predicates include:
 /// * The opcode of the instruction is a particular value.
 /// * The nsw/nuw flag is/isn't set.
 class InstructionPredicateMatcher : public PredicateMatcher {
 public:
   InstructionPredicateMatcher(PredicateKind Kind, unsigned InsnVarID)
       : PredicateMatcher(Kind, InsnVarID) {}
   virtual ~InstructionPredicateMatcher() {}
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   virtual bool
   isHigherPriorityThan(const InstructionPredicateMatcher &B) const {
     return Kind < B.Kind;
   };
 };
 
 template <>
 std::string
 PredicateListMatcher<PredicateMatcher>::getNoPredicateComment() const {
   return "No instruction predicates";
 }
 
 /// Generates code to check the opcode of an instruction.
 class InstructionOpcodeMatcher : public InstructionPredicateMatcher {
 protected:
   const CodeGenInstruction *I;
 
   static DenseMap<const CodeGenInstruction *, unsigned> OpcodeValues;
 
 public:
   static void initOpcodeValuesMap(const CodeGenTarget &Target) {
     OpcodeValues.clear();
 
     unsigned OpcodeValue = 0;
     for (const CodeGenInstruction *I : Target.getInstructionsByEnumValue())
       OpcodeValues[I] = OpcodeValue++;
   }
 
   InstructionOpcodeMatcher(unsigned InsnVarID, const CodeGenInstruction *I)
       : InstructionPredicateMatcher(IPM_Opcode, InsnVarID), I(I) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_Opcode;
   }
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
            I == cast<InstructionOpcodeMatcher>(&B)->I;
   }
   MatchTableRecord getValue() const override {
     const auto VI = OpcodeValues.find(I);
     if (VI != OpcodeValues.end())
       return MatchTable::NamedValue(I->Namespace, I->TheDef->getName(),
                                     VI->second);
     return MatchTable::NamedValue(I->Namespace, I->TheDef->getName());
   }
   bool hasValue() const override { return OpcodeValues.count(I); }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckOpcode") << MatchTable::Comment("MI")
           << MatchTable::IntValue(InsnVarID) << getValue()
           << MatchTable::LineBreak;
   }
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   bool
   isHigherPriorityThan(const InstructionPredicateMatcher &B) const override {
     if (InstructionPredicateMatcher::isHigherPriorityThan(B))
       return true;
     if (B.InstructionPredicateMatcher::isHigherPriorityThan(*this))
       return false;
 
     // Prioritize opcodes for cosmetic reasons in the generated source. Although
     // this is cosmetic at the moment, we may want to drive a similar ordering
     // using instruction frequency information to improve compile time.
     if (const InstructionOpcodeMatcher *BO =
             dyn_cast<InstructionOpcodeMatcher>(&B))
       return I->TheDef->getName() < BO->I->TheDef->getName();
 
     return false;
   };
 
   bool isConstantInstruction() const {
     return I->TheDef->getName() == "G_CONSTANT";
   }
 
   StringRef getOpcode() const { return I->TheDef->getName(); }
   unsigned getNumOperands() const { return I->Operands.size(); }
 
   StringRef getOperandType(unsigned OpIdx) const {
     return I->Operands[OpIdx].OperandType;
   }
 };
 
 DenseMap<const CodeGenInstruction *, unsigned>
     InstructionOpcodeMatcher::OpcodeValues;
 
 class InstructionNumOperandsMatcher final : public InstructionPredicateMatcher {
   unsigned NumOperands = 0;
 
 public:
   InstructionNumOperandsMatcher(unsigned InsnVarID, unsigned NumOperands)
       : InstructionPredicateMatcher(IPM_NumOperands, InsnVarID),
         NumOperands(NumOperands) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_NumOperands;
   }
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
            NumOperands == cast<InstructionNumOperandsMatcher>(&B)->NumOperands;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckNumOperands")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Expected")
           << MatchTable::IntValue(NumOperands) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that this instruction is a constant whose value
 /// meets an immediate predicate.
 ///
 /// Immediates are slightly odd since they are typically used like an operand
 /// but are represented as an operator internally. We typically write simm8:$src
 /// in a tablegen pattern, but this is just syntactic sugar for
 /// (imm:i32)<<P:Predicate_simm8>>:$imm which more directly describes the nodes
 /// that will be matched and the predicate (which is attached to the imm
 /// operator) that will be tested. In SelectionDAG this describes a
 /// ConstantSDNode whose internal value will be tested using the simm8 predicate.
 ///
 /// The corresponding GlobalISel representation is %1 = G_CONSTANT iN Value. In
 /// this representation, the immediate could be tested with an
 /// InstructionMatcher, InstructionOpcodeMatcher, OperandMatcher, and a
 /// OperandPredicateMatcher-subclass to check the Value meets the predicate but
 /// there are two implementation issues with producing that matcher
 /// configuration from the SelectionDAG pattern:
 /// * ImmLeaf is a PatFrag whose root is an InstructionMatcher. This means that
 ///   were we to sink the immediate predicate to the operand we would have to
 ///   have two partial implementations of PatFrag support, one for immediates
 ///   and one for non-immediates.
 /// * At the point we handle the predicate, the OperandMatcher hasn't been
 ///   created yet. If we were to sink the predicate to the OperandMatcher we
 ///   would also have to complicate (or duplicate) the code that descends and
 ///   creates matchers for the subtree.
 /// Overall, it's simpler to handle it in the place it was found.
 class InstructionImmPredicateMatcher : public InstructionPredicateMatcher {
 protected:
   TreePredicateFn Predicate;
 
 public:
   InstructionImmPredicateMatcher(unsigned InsnVarID,
                                  const TreePredicateFn &Predicate)
       : InstructionPredicateMatcher(IPM_ImmPredicate, InsnVarID),
         Predicate(Predicate) {}
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
            Predicate.getOrigPatFragRecord() ==
                cast<InstructionImmPredicateMatcher>(&B)
                    ->Predicate.getOrigPatFragRecord();
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_ImmPredicate;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode(getMatchOpcodeForPredicate(Predicate))
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Predicate")
           << MatchTable::NamedValue(getEnumNameForPredicate(Predicate))
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that a memory instruction has a atomic ordering
 /// MachineMemoryOperand.
 class AtomicOrderingMMOPredicateMatcher : public InstructionPredicateMatcher {
 public:
   enum AOComparator {
     AO_Exactly,
     AO_OrStronger,
     AO_WeakerThan,
   };
 
 protected:
   StringRef Order;
   AOComparator Comparator;
 
 public:
   AtomicOrderingMMOPredicateMatcher(unsigned InsnVarID, StringRef Order,
                                     AOComparator Comparator = AO_Exactly)
       : InstructionPredicateMatcher(IPM_AtomicOrderingMMO, InsnVarID),
         Order(Order), Comparator(Comparator) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_AtomicOrderingMMO;
   }
 
   bool isIdentical(const PredicateMatcher &B) const override {
     if (!InstructionPredicateMatcher::isIdentical(B))
       return false;
     const auto &R = *cast<AtomicOrderingMMOPredicateMatcher>(&B);
     return Order == R.Order && Comparator == R.Comparator;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     StringRef Opcode = "GIM_CheckAtomicOrdering";
 
     if (Comparator == AO_OrStronger)
       Opcode = "GIM_CheckAtomicOrderingOrStrongerThan";
     if (Comparator == AO_WeakerThan)
       Opcode = "GIM_CheckAtomicOrderingWeakerThan";
 
     Table << MatchTable::Opcode(Opcode) << MatchTable::Comment("MI")
           << MatchTable::IntValue(InsnVarID) << MatchTable::Comment("Order")
           << MatchTable::NamedValue(("(int64_t)AtomicOrdering::" + Order).str())
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that the size of an MMO is exactly N bytes.
 class MemorySizePredicateMatcher : public InstructionPredicateMatcher {
 protected:
   unsigned MMOIdx;
   uint64_t Size;
 
 public:
   MemorySizePredicateMatcher(unsigned InsnVarID, unsigned MMOIdx, unsigned Size)
       : InstructionPredicateMatcher(IPM_MemoryLLTSize, InsnVarID),
         MMOIdx(MMOIdx), Size(Size) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_MemoryLLTSize;
   }
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
            MMOIdx == cast<MemorySizePredicateMatcher>(&B)->MMOIdx &&
            Size == cast<MemorySizePredicateMatcher>(&B)->Size;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckMemorySizeEqualTo")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("MMO") << MatchTable::IntValue(MMOIdx)
           << MatchTable::Comment("Size") << MatchTable::IntValue(Size)
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that the size of an MMO is less-than, equal-to, or
 /// greater than a given LLT.
 class MemoryVsLLTSizePredicateMatcher : public InstructionPredicateMatcher {
 public:
   enum RelationKind {
     GreaterThan,
     EqualTo,
     LessThan,
   };
 
 protected:
   unsigned MMOIdx;
   RelationKind Relation;
   unsigned OpIdx;
 
 public:
   MemoryVsLLTSizePredicateMatcher(unsigned InsnVarID, unsigned MMOIdx,
                                   enum RelationKind Relation,
                                   unsigned OpIdx)
       : InstructionPredicateMatcher(IPM_MemoryVsLLTSize, InsnVarID),
         MMOIdx(MMOIdx), Relation(Relation), OpIdx(OpIdx) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_MemoryVsLLTSize;
   }
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
            MMOIdx == cast<MemoryVsLLTSizePredicateMatcher>(&B)->MMOIdx &&
            Relation == cast<MemoryVsLLTSizePredicateMatcher>(&B)->Relation &&
            OpIdx == cast<MemoryVsLLTSizePredicateMatcher>(&B)->OpIdx;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode(Relation == EqualTo
                                     ? "GIM_CheckMemorySizeEqualToLLT"
                                     : Relation == GreaterThan
                                           ? "GIM_CheckMemorySizeGreaterThanLLT"
                                           : "GIM_CheckMemorySizeLessThanLLT")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("MMO") << MatchTable::IntValue(MMOIdx)
           << MatchTable::Comment("OpIdx") << MatchTable::IntValue(OpIdx)
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that a set of predicates and operands match for a
 /// particular instruction.
 ///
 /// Typical predicates include:
 /// * Has a specific opcode.
 /// * Has an nsw/nuw flag or doesn't.
 class InstructionMatcher final : public PredicateListMatcher<PredicateMatcher> {
 protected:
   typedef std::vector<std::unique_ptr<OperandMatcher>> OperandVec;
 
   RuleMatcher &Rule;
 
   /// The operands to match. All rendered operands must be present even if the
   /// condition is always true.
   OperandVec Operands;
   bool NumOperandsCheck = true;
 
   std::string SymbolicName;
   unsigned InsnVarID;
 
 public:
   InstructionMatcher(RuleMatcher &Rule, StringRef SymbolicName)
       : Rule(Rule), SymbolicName(SymbolicName) {
     // We create a new instruction matcher.
     // Get a new ID for that instruction.
     InsnVarID = Rule.implicitlyDefineInsnVar(*this);
   }
 
   /// Construct a new instruction predicate and add it to the matcher.
   template <class Kind, class... Args>
   Optional<Kind *> addPredicate(Args &&... args) {
     Predicates.emplace_back(
         llvm::make_unique<Kind>(getInsnVarID(), std::forward<Args>(args)...));
     return static_cast<Kind *>(Predicates.back().get());
   }
 
   RuleMatcher &getRuleMatcher() const { return Rule; }
 
   unsigned getInsnVarID() const { return InsnVarID; }
 
   /// Add an operand to the matcher.
   OperandMatcher &addOperand(unsigned OpIdx, const std::string &SymbolicName,
                              unsigned AllocatedTemporariesBaseID) {
     Operands.emplace_back(new OperandMatcher(*this, OpIdx, SymbolicName,
                                              AllocatedTemporariesBaseID));
     if (!SymbolicName.empty())
       Rule.defineOperand(SymbolicName, *Operands.back());
 
     return *Operands.back();
   }
 
   OperandMatcher &getOperand(unsigned OpIdx) {
     auto I = std::find_if(Operands.begin(), Operands.end(),
                           [&OpIdx](const std::unique_ptr<OperandMatcher> &X) {
                             return X->getOpIdx() == OpIdx;
                           });
     if (I != Operands.end())
       return **I;
     llvm_unreachable("Failed to lookup operand");
   }
 
   StringRef getSymbolicName() const { return SymbolicName; }
   unsigned getNumOperands() const { return Operands.size(); }
   OperandVec::iterator operands_begin() { return Operands.begin(); }
   OperandVec::iterator operands_end() { return Operands.end(); }
   iterator_range<OperandVec::iterator> operands() {
     return make_range(operands_begin(), operands_end());
   }
   OperandVec::const_iterator operands_begin() const { return Operands.begin(); }
   OperandVec::const_iterator operands_end() const { return Operands.end(); }
   iterator_range<OperandVec::const_iterator> operands() const {
     return make_range(operands_begin(), operands_end());
   }
   bool operands_empty() const { return Operands.empty(); }
 
   void pop_front() { Operands.erase(Operands.begin()); }
 
   void optimize();
 
   /// Emit MatchTable opcodes that test whether the instruction named in
   /// InsnVarName matches all the predicates and all the operands.
   void emitPredicateOpcodes(MatchTable &Table, RuleMatcher &Rule) {
     if (NumOperandsCheck)
       InstructionNumOperandsMatcher(InsnVarID, getNumOperands())
           .emitPredicateOpcodes(Table, Rule);
 
     emitPredicateListOpcodes(Table, Rule);
 
     for (const auto &Operand : Operands)
       Operand->emitPredicateOpcodes(Table, Rule);
   }
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   bool isHigherPriorityThan(InstructionMatcher &B) {
     // Instruction matchers involving more operands have higher priority.
     if (Operands.size() > B.Operands.size())
       return true;
     if (Operands.size() < B.Operands.size())
       return false;
 
     for (auto &&P : zip(predicates(), B.predicates())) {
       auto L = static_cast<InstructionPredicateMatcher *>(std::get<0>(P).get());
       auto R = static_cast<InstructionPredicateMatcher *>(std::get<1>(P).get());
       if (L->isHigherPriorityThan(*R))
         return true;
       if (R->isHigherPriorityThan(*L))
         return false;
     }
 
     for (const auto &Operand : zip(Operands, B.Operands)) {
       if (std::get<0>(Operand)->isHigherPriorityThan(*std::get<1>(Operand)))
         return true;
       if (std::get<1>(Operand)->isHigherPriorityThan(*std::get<0>(Operand)))
         return false;
     }
 
     return false;
   };
 
   /// Report the maximum number of temporary operands needed by the instruction
   /// matcher.
   unsigned countRendererFns() {
     return std::accumulate(
                predicates().begin(), predicates().end(), 0,
                [](unsigned A,
                   const std::unique_ptr<PredicateMatcher> &Predicate) {
                  return A + Predicate->countRendererFns();
                }) +
            std::accumulate(
                Operands.begin(), Operands.end(), 0,
                [](unsigned A, const std::unique_ptr<OperandMatcher> &Operand) {
                  return A + Operand->countRendererFns();
                });
   }
 
   InstructionOpcodeMatcher &getOpcodeMatcher() {
     for (auto &P : predicates())
       if (auto *OpMatcher = dyn_cast<InstructionOpcodeMatcher>(P.get()))
         return *OpMatcher;
     llvm_unreachable("Didn't find an opcode matcher");
   }
 
   bool isConstantInstruction() {
     return getOpcodeMatcher().isConstantInstruction();
   }
 
   StringRef getOpcode() { return getOpcodeMatcher().getOpcode(); }
 };
 
 StringRef RuleMatcher::getOpcode() const {
   return Matchers.front()->getOpcode();
 }
 
 unsigned RuleMatcher::getNumOperands() const {
   return Matchers.front()->getNumOperands();
 }
 
-LLTCodeGen RuleMatcher::getFirstConditionAsRootType() {
-  InstructionMatcher &InsnMatcher = *Matchers.front();
-  if (!InsnMatcher.predicates_empty())
-    if (const auto *TM =
-            dyn_cast<LLTOperandMatcher>(&**InsnMatcher.predicates_begin()))
-      if (TM->getInsnVarID() == 0 && TM->getOpIdx() == 0)
-        return TM->getTy();
-  return {};
-}
-
 /// Generates code to check that the operand is a register defined by an
 /// instruction that matches the given instruction matcher.
 ///
 /// For example, the pattern:
 ///   (set $dst, (G_MUL (G_ADD $src1, $src2), $src3))
 /// would use an InstructionOperandMatcher for operand 1 of the G_MUL to match
 /// the:
 ///   (G_ADD $src1, $src2)
 /// subpattern.
 class InstructionOperandMatcher : public OperandPredicateMatcher {
 protected:
   std::unique_ptr<InstructionMatcher> InsnMatcher;
 
 public:
   InstructionOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                             RuleMatcher &Rule, StringRef SymbolicName)
       : OperandPredicateMatcher(OPM_Instruction, InsnVarID, OpIdx),
         InsnMatcher(new InstructionMatcher(Rule, SymbolicName)) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_Instruction;
   }
 
   InstructionMatcher &getInsnMatcher() const { return *InsnMatcher; }
 
   void emitCaptureOpcodes(MatchTable &Table, RuleMatcher &Rule) const {
     const unsigned NewInsnVarID = InsnMatcher->getInsnVarID();
     Table << MatchTable::Opcode("GIM_RecordInsn")
           << MatchTable::Comment("DefineMI")
           << MatchTable::IntValue(NewInsnVarID) << MatchTable::Comment("MI")
           << MatchTable::IntValue(getInsnVarID())
           << MatchTable::Comment("OpIdx") << MatchTable::IntValue(getOpIdx())
           << MatchTable::Comment("MIs[" + llvm::to_string(NewInsnVarID) + "]")
           << MatchTable::LineBreak;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     emitCaptureOpcodes(Table, Rule);
     InsnMatcher->emitPredicateOpcodes(Table, Rule);
   }
 
   bool isHigherPriorityThan(const OperandPredicateMatcher &B) const override {
     if (OperandPredicateMatcher::isHigherPriorityThan(B))
       return true;
     if (B.OperandPredicateMatcher::isHigherPriorityThan(*this))
       return false;
 
     if (const InstructionOperandMatcher *BP =
             dyn_cast<InstructionOperandMatcher>(&B))
       if (InsnMatcher->isHigherPriorityThan(*BP->InsnMatcher))
         return true;
     return false;
   }
 };
 
 void InstructionMatcher::optimize() {
   SmallVector<std::unique_ptr<PredicateMatcher>, 8> Stash;
   const auto &OpcMatcher = getOpcodeMatcher();
 
   Stash.push_back(predicates_pop_front());
   if (Stash.back().get() == &OpcMatcher) {
     if (NumOperandsCheck && OpcMatcher.getNumOperands() < getNumOperands())
       Stash.emplace_back(
           new InstructionNumOperandsMatcher(InsnVarID, getNumOperands()));
     NumOperandsCheck = false;
   }
 
   if (InsnVarID > 0) {
     assert(!Operands.empty() && "Nested instruction is expected to def a vreg");
     for (auto &OP : Operands[0]->predicates())
       OP.reset();
     Operands[0]->eraseNullPredicates();
   }
   while (!Stash.empty())
     prependPredicate(Stash.pop_back_val());
 }
 
 //===- Actions ------------------------------------------------------------===//
 class OperandRenderer {
 public:
   enum RendererKind {
     OR_Copy,
     OR_CopyOrAddZeroReg,
     OR_CopySubReg,
     OR_CopyConstantAsImm,
     OR_CopyFConstantAsFPImm,
     OR_Imm,
     OR_Register,
     OR_TempRegister,
     OR_ComplexPattern,
     OR_Custom
   };
 
 protected:
   RendererKind Kind;
 
 public:
   OperandRenderer(RendererKind Kind) : Kind(Kind) {}
   virtual ~OperandRenderer() {}
 
   RendererKind getKind() const { return Kind; }
 
   virtual void emitRenderOpcodes(MatchTable &Table,
                                  RuleMatcher &Rule) const = 0;
 };
 
 /// A CopyRenderer emits code to copy a single operand from an existing
 /// instruction to the one being built.
 class CopyRenderer : public OperandRenderer {
 protected:
   unsigned NewInsnID;
   /// The name of the operand.
   const StringRef SymbolicName;
 
 public:
   CopyRenderer(unsigned NewInsnID, StringRef SymbolicName)
       : OperandRenderer(OR_Copy), NewInsnID(NewInsnID),
         SymbolicName(SymbolicName) {
     assert(!SymbolicName.empty() && "Cannot copy from an unspecified source");
   }
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_Copy;
   }
 
   const StringRef getSymbolicName() const { return SymbolicName; }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     const OperandMatcher &Operand = Rule.getOperandMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(Operand.getInstructionMatcher());
     Table << MatchTable::Opcode("GIR_Copy") << MatchTable::Comment("NewInsnID")
           << MatchTable::IntValue(NewInsnID) << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID) << MatchTable::Comment("OpIdx")
           << MatchTable::IntValue(Operand.getOpIdx())
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// A CopyOrAddZeroRegRenderer emits code to copy a single operand from an
 /// existing instruction to the one being built. If the operand turns out to be
 /// a 'G_CONSTANT 0' then it replaces the operand with a zero register.
 class CopyOrAddZeroRegRenderer : public OperandRenderer {
 protected:
   unsigned NewInsnID;
   /// The name of the operand.
   const StringRef SymbolicName;
   const Record *ZeroRegisterDef;
 
 public:
   CopyOrAddZeroRegRenderer(unsigned NewInsnID,
                            StringRef SymbolicName, Record *ZeroRegisterDef)
       : OperandRenderer(OR_CopyOrAddZeroReg), NewInsnID(NewInsnID),
         SymbolicName(SymbolicName), ZeroRegisterDef(ZeroRegisterDef) {
     assert(!SymbolicName.empty() && "Cannot copy from an unspecified source");
   }
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_CopyOrAddZeroReg;
   }
 
   const StringRef getSymbolicName() const { return SymbolicName; }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     const OperandMatcher &Operand = Rule.getOperandMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(Operand.getInstructionMatcher());
     Table << MatchTable::Opcode("GIR_CopyOrAddZeroReg")
           << MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
           << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID) << MatchTable::Comment("OpIdx")
           << MatchTable::IntValue(Operand.getOpIdx())
           << MatchTable::NamedValue(
                  (ZeroRegisterDef->getValue("Namespace")
                       ? ZeroRegisterDef->getValueAsString("Namespace")
                       : ""),
                  ZeroRegisterDef->getName())
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// A CopyConstantAsImmRenderer emits code to render a G_CONSTANT instruction to
 /// an extended immediate operand.
 class CopyConstantAsImmRenderer : public OperandRenderer {
 protected:
   unsigned NewInsnID;
   /// The name of the operand.
   const std::string SymbolicName;
   bool Signed;
 
 public:
   CopyConstantAsImmRenderer(unsigned NewInsnID, StringRef SymbolicName)
       : OperandRenderer(OR_CopyConstantAsImm), NewInsnID(NewInsnID),
         SymbolicName(SymbolicName), Signed(true) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_CopyConstantAsImm;
   }
 
   const StringRef getSymbolicName() const { return SymbolicName; }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     InstructionMatcher &InsnMatcher = Rule.getInstructionMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(InsnMatcher);
     Table << MatchTable::Opcode(Signed ? "GIR_CopyConstantAsSImm"
                                        : "GIR_CopyConstantAsUImm")
           << MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
           << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID)
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// A CopyFConstantAsFPImmRenderer emits code to render a G_FCONSTANT
 /// instruction to an extended immediate operand.
 class CopyFConstantAsFPImmRenderer : public OperandRenderer {
 protected:
   unsigned NewInsnID;
   /// The name of the operand.
   const std::string SymbolicName;
 
 public:
   CopyFConstantAsFPImmRenderer(unsigned NewInsnID, StringRef SymbolicName)
       : OperandRenderer(OR_CopyFConstantAsFPImm), NewInsnID(NewInsnID),
         SymbolicName(SymbolicName) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_CopyFConstantAsFPImm;
   }
 
   const StringRef getSymbolicName() const { return SymbolicName; }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     InstructionMatcher &InsnMatcher = Rule.getInstructionMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(InsnMatcher);
     Table << MatchTable::Opcode("GIR_CopyFConstantAsFPImm")
           << MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
           << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID)
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// A CopySubRegRenderer emits code to copy a single register operand from an
 /// existing instruction to the one being built and indicate that only a
 /// subregister should be copied.
 class CopySubRegRenderer : public OperandRenderer {
 protected:
   unsigned NewInsnID;
   /// The name of the operand.
   const StringRef SymbolicName;
   /// The subregister to extract.
   const CodeGenSubRegIndex *SubReg;
 
 public:
   CopySubRegRenderer(unsigned NewInsnID, StringRef SymbolicName,
                      const CodeGenSubRegIndex *SubReg)
       : OperandRenderer(OR_CopySubReg), NewInsnID(NewInsnID),
         SymbolicName(SymbolicName), SubReg(SubReg) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_CopySubReg;
   }
 
   const StringRef getSymbolicName() const { return SymbolicName; }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     const OperandMatcher &Operand = Rule.getOperandMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(Operand.getInstructionMatcher());
     Table << MatchTable::Opcode("GIR_CopySubReg")
           << MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
           << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID) << MatchTable::Comment("OpIdx")
           << MatchTable::IntValue(Operand.getOpIdx())
           << MatchTable::Comment("SubRegIdx")
           << MatchTable::IntValue(SubReg->EnumValue)
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// Adds a specific physical register to the instruction being built.
 /// This is typically useful for WZR/XZR on AArch64.
 class AddRegisterRenderer : public OperandRenderer {
 protected:
   unsigned InsnID;
   const Record *RegisterDef;
 
 public:
   AddRegisterRenderer(unsigned InsnID, const Record *RegisterDef)
       : OperandRenderer(OR_Register), InsnID(InsnID), RegisterDef(RegisterDef) {
   }
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_Register;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_AddRegister")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::NamedValue(
                  (RegisterDef->getValue("Namespace")
                       ? RegisterDef->getValueAsString("Namespace")
                       : ""),
                  RegisterDef->getName())
           << MatchTable::LineBreak;
   }
 };
 
 /// Adds a specific temporary virtual register to the instruction being built.
 /// This is used to chain instructions together when emitting multiple
 /// instructions.
 class TempRegRenderer : public OperandRenderer {
 protected:
   unsigned InsnID;
   unsigned TempRegID;
   bool IsDef;
 
 public:
   TempRegRenderer(unsigned InsnID, unsigned TempRegID, bool IsDef = false)
       : OperandRenderer(OR_Register), InsnID(InsnID), TempRegID(TempRegID),
         IsDef(IsDef) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_TempRegister;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_AddTempRegister")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::Comment("TempRegID") << MatchTable::IntValue(TempRegID)
           << MatchTable::Comment("TempRegFlags");
     if (IsDef)
       Table << MatchTable::NamedValue("RegState::Define");
     else
       Table << MatchTable::IntValue(0);
     Table << MatchTable::LineBreak;
   }
 };
 
 /// Adds a specific immediate to the instruction being built.
 class ImmRenderer : public OperandRenderer {
 protected:
   unsigned InsnID;
   int64_t Imm;
 
 public:
   ImmRenderer(unsigned InsnID, int64_t Imm)
       : OperandRenderer(OR_Imm), InsnID(InsnID), Imm(Imm) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_Imm;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_AddImm") << MatchTable::Comment("InsnID")
           << MatchTable::IntValue(InsnID) << MatchTable::Comment("Imm")
           << MatchTable::IntValue(Imm) << MatchTable::LineBreak;
   }
 };
 
 /// Adds operands by calling a renderer function supplied by the ComplexPattern
 /// matcher function.
 class RenderComplexPatternOperand : public OperandRenderer {
 private:
   unsigned InsnID;
   const Record &TheDef;
   /// The name of the operand.
   const StringRef SymbolicName;
   /// The renderer number. This must be unique within a rule since it's used to
   /// identify a temporary variable to hold the renderer function.
   unsigned RendererID;
   /// When provided, this is the suboperand of the ComplexPattern operand to
   /// render. Otherwise all the suboperands will be rendered.
   Optional<unsigned> SubOperand;
 
   unsigned getNumOperands() const {
     return TheDef.getValueAsDag("Operands")->getNumArgs();
   }
 
 public:
   RenderComplexPatternOperand(unsigned InsnID, const Record &TheDef,
                               StringRef SymbolicName, unsigned RendererID,
                               Optional<unsigned> SubOperand = None)
       : OperandRenderer(OR_ComplexPattern), InsnID(InsnID), TheDef(TheDef),
         SymbolicName(SymbolicName), RendererID(RendererID),
         SubOperand(SubOperand) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_ComplexPattern;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode(SubOperand.hasValue() ? "GIR_ComplexSubOperandRenderer"
                                                       : "GIR_ComplexRenderer")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::Comment("RendererID")
           << MatchTable::IntValue(RendererID);
     if (SubOperand.hasValue())
       Table << MatchTable::Comment("SubOperand")
             << MatchTable::IntValue(SubOperand.getValue());
     Table << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 class CustomRenderer : public OperandRenderer {
 protected:
   unsigned InsnID;
   const Record &Renderer;
   /// The name of the operand.
   const std::string SymbolicName;
 
 public:
   CustomRenderer(unsigned InsnID, const Record &Renderer,
                  StringRef SymbolicName)
       : OperandRenderer(OR_Custom), InsnID(InsnID), Renderer(Renderer),
         SymbolicName(SymbolicName) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_Custom;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     InstructionMatcher &InsnMatcher = Rule.getInstructionMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(InsnMatcher);
     Table << MatchTable::Opcode("GIR_CustomRenderer")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID)
           << MatchTable::Comment("Renderer")
           << MatchTable::NamedValue(
                  "GICR_" + Renderer.getValueAsString("RendererFn").str())
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// An action taken when all Matcher predicates succeeded for a parent rule.
 ///
 /// Typical actions include:
 /// * Changing the opcode of an instruction.
 /// * Adding an operand to an instruction.
 class MatchAction {
 public:
   virtual ~MatchAction() {}
 
   /// Emit the MatchTable opcodes to implement the action.
   virtual void emitActionOpcodes(MatchTable &Table,
                                  RuleMatcher &Rule) const = 0;
 };
 
 /// Generates a comment describing the matched rule being acted upon.
 class DebugCommentAction : public MatchAction {
 private:
   std::string S;
 
 public:
   DebugCommentAction(StringRef S) : S(S) {}
 
   void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Comment(S) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to build an instruction or mutate an existing instruction
 /// into the desired instruction when this is possible.
 class BuildMIAction : public MatchAction {
 private:
   unsigned InsnID;
   const CodeGenInstruction *I;
   InstructionMatcher *Matched;
   std::vector<std::unique_ptr<OperandRenderer>> OperandRenderers;
 
   /// True if the instruction can be built solely by mutating the opcode.
   bool canMutate(RuleMatcher &Rule, const InstructionMatcher *Insn) const {
     if (!Insn)
       return false;
 
     if (OperandRenderers.size() != Insn->getNumOperands())
       return false;
 
     for (const auto &Renderer : enumerate(OperandRenderers)) {
       if (const auto *Copy = dyn_cast<CopyRenderer>(&*Renderer.value())) {
         const OperandMatcher &OM = Rule.getOperandMatcher(Copy->getSymbolicName());
         if (Insn != &OM.getInstructionMatcher() ||
             OM.getOpIdx() != Renderer.index())
           return false;
       } else
         return false;
     }
 
     return true;
   }
 
 public:
   BuildMIAction(unsigned InsnID, const CodeGenInstruction *I)
       : InsnID(InsnID), I(I), Matched(nullptr) {}
 
   unsigned getInsnID() const { return InsnID; }
   const CodeGenInstruction *getCGI() const { return I; }
 
   void chooseInsnToMutate(RuleMatcher &Rule) {
     for (auto *MutateCandidate : Rule.mutatable_insns()) {
       if (canMutate(Rule, MutateCandidate)) {
         // Take the first one we're offered that we're able to mutate.
         Rule.reserveInsnMatcherForMutation(MutateCandidate);
         Matched = MutateCandidate;
         return;
       }
     }
   }
 
   template <class Kind, class... Args>
   Kind &addRenderer(Args&&... args) {
     OperandRenderers.emplace_back(
         llvm::make_unique<Kind>(InsnID, std::forward<Args>(args)...));
     return *static_cast<Kind *>(OperandRenderers.back().get());
   }
 
   void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     if (Matched) {
       assert(canMutate(Rule, Matched) &&
              "Arranged to mutate an insn that isn't mutatable");
 
       unsigned RecycleInsnID = Rule.getInsnVarID(*Matched);
       Table << MatchTable::Opcode("GIR_MutateOpcode")
             << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
             << MatchTable::Comment("RecycleInsnID")
             << MatchTable::IntValue(RecycleInsnID)
             << MatchTable::Comment("Opcode")
             << MatchTable::NamedValue(I->Namespace, I->TheDef->getName())
             << MatchTable::LineBreak;
 
       if (!I->ImplicitDefs.empty() || !I->ImplicitUses.empty()) {
         for (auto Def : I->ImplicitDefs) {
           auto Namespace = Def->getValue("Namespace")
                                ? Def->getValueAsString("Namespace")
                                : "";
           Table << MatchTable::Opcode("GIR_AddImplicitDef")
                 << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
                 << MatchTable::NamedValue(Namespace, Def->getName())
                 << MatchTable::LineBreak;
         }
         for (auto Use : I->ImplicitUses) {
           auto Namespace = Use->getValue("Namespace")
                                ? Use->getValueAsString("Namespace")
                                : "";
           Table << MatchTable::Opcode("GIR_AddImplicitUse")
                 << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
                 << MatchTable::NamedValue(Namespace, Use->getName())
                 << MatchTable::LineBreak;
         }
       }
       return;
     }
 
     // TODO: Simple permutation looks like it could be almost as common as
     //       mutation due to commutative operations.
 
     Table << MatchTable::Opcode("GIR_BuildMI") << MatchTable::Comment("InsnID")
           << MatchTable::IntValue(InsnID) << MatchTable::Comment("Opcode")
           << MatchTable::NamedValue(I->Namespace, I->TheDef->getName())
           << MatchTable::LineBreak;
     for (const auto &Renderer : OperandRenderers)
       Renderer->emitRenderOpcodes(Table, Rule);
 
     if (I->mayLoad || I->mayStore) {
       Table << MatchTable::Opcode("GIR_MergeMemOperands")
             << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
             << MatchTable::Comment("MergeInsnID's");
       // Emit the ID's for all the instructions that are matched by this rule.
       // TODO: Limit this to matched instructions that mayLoad/mayStore or have
       //       some other means of having a memoperand. Also limit this to
       //       emitted instructions that expect to have a memoperand too. For
       //       example, (G_SEXT (G_LOAD x)) that results in separate load and
       //       sign-extend instructions shouldn't put the memoperand on the
       //       sign-extend since it has no effect there.
       std::vector<unsigned> MergeInsnIDs;
       for (const auto &IDMatcherPair : Rule.defined_insn_vars())
         MergeInsnIDs.push_back(IDMatcherPair.second);
       llvm::sort(MergeInsnIDs.begin(), MergeInsnIDs.end());
       for (const auto &MergeInsnID : MergeInsnIDs)
         Table << MatchTable::IntValue(MergeInsnID);
       Table << MatchTable::NamedValue("GIU_MergeMemOperands_EndOfList")
             << MatchTable::LineBreak;
     }
 
     // FIXME: This is a hack but it's sufficient for ISel. We'll need to do
     //        better for combines. Particularly when there are multiple match
     //        roots.
     if (InsnID == 0)
       Table << MatchTable::Opcode("GIR_EraseFromParent")
             << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
             << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to constrain the operands of an output instruction to the
 /// register classes specified by the definition of that instruction.
 class ConstrainOperandsToDefinitionAction : public MatchAction {
   unsigned InsnID;
 
 public:
   ConstrainOperandsToDefinitionAction(unsigned InsnID) : InsnID(InsnID) {}
 
   void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_ConstrainSelectedInstOperands")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to constrain the specified operand of an output instruction
 /// to the specified register class.
 class ConstrainOperandToRegClassAction : public MatchAction {
   unsigned InsnID;
   unsigned OpIdx;
   const CodeGenRegisterClass &RC;
 
 public:
   ConstrainOperandToRegClassAction(unsigned InsnID, unsigned OpIdx,
                                    const CodeGenRegisterClass &RC)
       : InsnID(InsnID), OpIdx(OpIdx), RC(RC) {}
 
   void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_ConstrainOperandRC")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::Comment("RC " + RC.getName())
           << MatchTable::IntValue(RC.EnumValue) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to create a temporary register which can be used to chain
 /// instructions together.
 class MakeTempRegisterAction : public MatchAction {
 private:
   LLTCodeGen Ty;
   unsigned TempRegID;
 
 public:
   MakeTempRegisterAction(const LLTCodeGen &Ty, unsigned TempRegID)
       : Ty(Ty), TempRegID(TempRegID) {}
 
   void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_MakeTempReg")
           << MatchTable::Comment("TempRegID") << MatchTable::IntValue(TempRegID)
           << MatchTable::Comment("TypeID")
           << MatchTable::NamedValue(Ty.getCxxEnumValue())
           << MatchTable::LineBreak;
   }
 };
 
 InstructionMatcher &RuleMatcher::addInstructionMatcher(StringRef SymbolicName) {
   Matchers.emplace_back(new InstructionMatcher(*this, SymbolicName));
   MutatableInsns.insert(Matchers.back().get());
   return *Matchers.back();
 }
 
 void RuleMatcher::addRequiredFeature(Record *Feature) {
   RequiredFeatures.push_back(Feature);
 }
 
 const std::vector<Record *> &RuleMatcher::getRequiredFeatures() const {
   return RequiredFeatures;
 }
 
 // Emplaces an action of the specified Kind at the end of the action list.
 //
 // Returns a reference to the newly created action.
 //
 // Like std::vector::emplace_back(), may invalidate all iterators if the new
 // size exceeds the capacity. Otherwise, only invalidates the past-the-end
 // iterator.
 template <class Kind, class... Args>
 Kind &RuleMatcher::addAction(Args &&... args) {
   Actions.emplace_back(llvm::make_unique<Kind>(std::forward<Args>(args)...));
   return *static_cast<Kind *>(Actions.back().get());
 }
 
 // Emplaces an action of the specified Kind before the given insertion point.
 //
 // Returns an iterator pointing at the newly created instruction.
 //
 // Like std::vector::insert(), may invalidate all iterators if the new size
 // exceeds the capacity. Otherwise, only invalidates the iterators from the
 // insertion point onwards.
 template <class Kind, class... Args>
 action_iterator RuleMatcher::insertAction(action_iterator InsertPt,
                                           Args &&... args) {
   return Actions.emplace(InsertPt,
                          llvm::make_unique<Kind>(std::forward<Args>(args)...));
 }
 
 unsigned RuleMatcher::implicitlyDefineInsnVar(InstructionMatcher &Matcher) {
   unsigned NewInsnVarID = NextInsnVarID++;
   InsnVariableIDs[&Matcher] = NewInsnVarID;
   return NewInsnVarID;
 }
 
 unsigned RuleMatcher::getInsnVarID(InstructionMatcher &InsnMatcher) const {
   const auto &I = InsnVariableIDs.find(&InsnMatcher);
   if (I != InsnVariableIDs.end())
     return I->second;
   llvm_unreachable("Matched Insn was not captured in a local variable");
 }
 
 void RuleMatcher::defineOperand(StringRef SymbolicName, OperandMatcher &OM) {
   if (DefinedOperands.find(SymbolicName) == DefinedOperands.end()) {
     DefinedOperands[SymbolicName] = &OM;
     return;
   }
 
   // If the operand is already defined, then we must ensure both references in
   // the matcher have the exact same node.
   OM.addPredicate<SameOperandMatcher>(OM.getSymbolicName());
 }
 
 InstructionMatcher &
 RuleMatcher::getInstructionMatcher(StringRef SymbolicName) const {
   for (const auto &I : InsnVariableIDs)
     if (I.first->getSymbolicName() == SymbolicName)
       return *I.first;
   llvm_unreachable(
       ("Failed to lookup instruction " + SymbolicName).str().c_str());
 }
 
 const OperandMatcher &
 RuleMatcher::getOperandMatcher(StringRef Name) const {
   const auto &I = DefinedOperands.find(Name);
 
   if (I == DefinedOperands.end())
     PrintFatalError(SrcLoc, "Operand " + Name + " was not declared in matcher");
 
   return *I->second;
 }
 
 void RuleMatcher::emit(MatchTable &Table) {
   if (Matchers.empty())
     llvm_unreachable("Unexpected empty matcher!");
 
   // The representation supports rules that require multiple roots such as:
   //    %ptr(p0) = ...
   //    %elt0(s32) = G_LOAD %ptr
   //    %1(p0) = G_ADD %ptr, 4
   //    %elt1(s32) = G_LOAD p0 %1
   // which could be usefully folded into:
   //    %ptr(p0) = ...
   //    %elt0(s32), %elt1(s32) = TGT_LOAD_PAIR %ptr
   // on some targets but we don't need to make use of that yet.
   assert(Matchers.size() == 1 && "Cannot handle multi-root matchers yet");
 
   unsigned LabelID = Table.allocateLabelID();
   Table << MatchTable::Opcode("GIM_Try", +1)
         << MatchTable::Comment("On fail goto")
         << MatchTable::JumpTarget(LabelID)
         << MatchTable::Comment(("Rule ID " + Twine(RuleID) + " //").str())
         << MatchTable::LineBreak;
 
   if (!RequiredFeatures.empty()) {
     Table << MatchTable::Opcode("GIM_CheckFeatures")
           << MatchTable::NamedValue(getNameForFeatureBitset(RequiredFeatures))
           << MatchTable::LineBreak;
   }
 
   Matchers.front()->emitPredicateOpcodes(Table, *this);
 
   // We must also check if it's safe to fold the matched instructions.
   if (InsnVariableIDs.size() >= 2) {
     // Invert the map to create stable ordering (by var names)
     SmallVector<unsigned, 2> InsnIDs;
     for (const auto &Pair : InsnVariableIDs) {
       // Skip the root node since it isn't moving anywhere. Everything else is
       // sinking to meet it.
       if (Pair.first == Matchers.front().get())
         continue;
 
       InsnIDs.push_back(Pair.second);
     }
     llvm::sort(InsnIDs.begin(), InsnIDs.end());
 
     for (const auto &InsnID : InsnIDs) {
       // Reject the difficult cases until we have a more accurate check.
       Table << MatchTable::Opcode("GIM_CheckIsSafeToFold")
             << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
             << MatchTable::LineBreak;
 
       // FIXME: Emit checks to determine it's _actually_ safe to fold and/or
       //        account for unsafe cases.
       //
       //        Example:
       //          MI1--> %0 = ...
       //                 %1 = ... %0
       //          MI0--> %2 = ... %0
       //          It's not safe to erase MI1. We currently handle this by not
       //          erasing %0 (even when it's dead).
       //
       //        Example:
       //          MI1--> %0 = load volatile @a
       //                 %1 = load volatile @a
       //          MI0--> %2 = ... %0
       //          It's not safe to sink %0's def past %1. We currently handle
       //          this by rejecting all loads.
       //
       //        Example:
       //          MI1--> %0 = load @a
       //                 %1 = store @a
       //          MI0--> %2 = ... %0
       //          It's not safe to sink %0's def past %1. We currently handle
       //          this by rejecting all loads.
       //
       //        Example:
       //                   G_CONDBR %cond, @BB1
       //                 BB0:
       //          MI1-->   %0 = load @a
       //                   G_BR @BB1
       //                 BB1:
       //          MI0-->   %2 = ... %0
       //          It's not always safe to sink %0 across control flow. In this
       //          case it may introduce a memory fault. We currentl handle this
       //          by rejecting all loads.
     }
   }
 
   for (const auto &PM : EpilogueMatchers)
     PM->emitPredicateOpcodes(Table, *this);
 
   for (const auto &MA : Actions)
     MA->emitActionOpcodes(Table, *this);
 
   if (Table.isWithCoverage())
     Table << MatchTable::Opcode("GIR_Coverage") << MatchTable::IntValue(RuleID)
           << MatchTable::LineBreak;
   else
     Table << MatchTable::Comment(("GIR_Coverage, " + Twine(RuleID) + ",").str())
           << MatchTable::LineBreak;
 
   Table << MatchTable::Opcode("GIR_Done", -1) << MatchTable::LineBreak
         << MatchTable::Label(LabelID);
   ++NumPatternEmitted;
 }
 
 bool RuleMatcher::isHigherPriorityThan(const RuleMatcher &B) const {
   // Rules involving more match roots have higher priority.
   if (Matchers.size() > B.Matchers.size())
     return true;
   if (Matchers.size() < B.Matchers.size())
     return false;
 
   for (const auto &Matcher : zip(Matchers, B.Matchers)) {
     if (std::get<0>(Matcher)->isHigherPriorityThan(*std::get<1>(Matcher)))
       return true;
     if (std::get<1>(Matcher)->isHigherPriorityThan(*std::get<0>(Matcher)))
       return false;
   }
 
   return false;
 }
 
 unsigned RuleMatcher::countRendererFns() const {
   return std::accumulate(
       Matchers.begin(), Matchers.end(), 0,
       [](unsigned A, const std::unique_ptr<InstructionMatcher> &Matcher) {
         return A + Matcher->countRendererFns();
       });
 }
 
 bool OperandPredicateMatcher::isHigherPriorityThan(
     const OperandPredicateMatcher &B) const {
   // Generally speaking, an instruction is more important than an Int or a
   // LiteralInt because it can cover more nodes but theres an exception to
   // this. G_CONSTANT's are less important than either of those two because they
   // are more permissive.
 
   const InstructionOperandMatcher *AOM =
       dyn_cast<InstructionOperandMatcher>(this);
   const InstructionOperandMatcher *BOM =
       dyn_cast<InstructionOperandMatcher>(&B);
   bool AIsConstantInsn = AOM && AOM->getInsnMatcher().isConstantInstruction();
   bool BIsConstantInsn = BOM && BOM->getInsnMatcher().isConstantInstruction();
 
   if (AOM && BOM) {
     // The relative priorities between a G_CONSTANT and any other instruction
     // don't actually matter but this code is needed to ensure a strict weak
     // ordering. This is particularly important on Windows where the rules will
     // be incorrectly sorted without it.
     if (AIsConstantInsn != BIsConstantInsn)
       return AIsConstantInsn < BIsConstantInsn;
     return false;
   }
 
   if (AOM && AIsConstantInsn && (B.Kind == OPM_Int || B.Kind == OPM_LiteralInt))
     return false;
   if (BOM && BIsConstantInsn && (Kind == OPM_Int || Kind == OPM_LiteralInt))
     return true;
 
   return Kind < B.Kind;
 }
 
 void SameOperandMatcher::emitPredicateOpcodes(MatchTable &Table,
                                               RuleMatcher &Rule) const {
   const OperandMatcher &OtherOM = Rule.getOperandMatcher(MatchingName);
   unsigned OtherInsnVarID = Rule.getInsnVarID(OtherOM.getInstructionMatcher());
   assert(OtherInsnVarID == OtherOM.getInstructionMatcher().getInsnVarID());
 
   Table << MatchTable::Opcode("GIM_CheckIsSameOperand")
         << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
         << MatchTable::Comment("OpIdx") << MatchTable::IntValue(OpIdx)
         << MatchTable::Comment("OtherMI")
         << MatchTable::IntValue(OtherInsnVarID)
         << MatchTable::Comment("OtherOpIdx")
         << MatchTable::IntValue(OtherOM.getOpIdx())
         << MatchTable::LineBreak;
 }
 
 //===- GlobalISelEmitter class --------------------------------------------===//
 
 class GlobalISelEmitter {
 public:
   explicit GlobalISelEmitter(RecordKeeper &RK);
   void run(raw_ostream &OS);
 
 private:
   const RecordKeeper &RK;
   const CodeGenDAGPatterns CGP;
   const CodeGenTarget &Target;
   CodeGenRegBank CGRegs;
 
   /// Keep track of the equivalence between SDNodes and Instruction by mapping
   /// SDNodes to the GINodeEquiv mapping. We need to map to the GINodeEquiv to
   /// check for attributes on the relation such as CheckMMOIsNonAtomic.
   /// This is defined using 'GINodeEquiv' in the target description.
   DenseMap<Record *, Record *> NodeEquivs;
 
   /// Keep track of the equivalence between ComplexPattern's and
   /// GIComplexOperandMatcher. Map entries are specified by subclassing
   /// GIComplexPatternEquiv.
   DenseMap<const Record *, const Record *> ComplexPatternEquivs;
 
   /// Keep track of the equivalence between SDNodeXForm's and
   /// GICustomOperandRenderer. Map entries are specified by subclassing
   /// GISDNodeXFormEquiv.
   DenseMap<const Record *, const Record *> SDNodeXFormEquivs;
 
   /// Keep track of Scores of PatternsToMatch similar to how the DAG does.
   /// This adds compatibility for RuleMatchers to use this for ordering rules.
   DenseMap<uint64_t, int> RuleMatcherScores;
 
   // Map of predicates to their subtarget features.
   SubtargetFeatureInfoMap SubtargetFeatures;
 
   // Rule coverage information.
   Optional<CodeGenCoverage> RuleCoverage;
 
   void gatherOpcodeValues();
   void gatherTypeIDValues();
   void gatherNodeEquivs();
   Record *findNodeEquiv(Record *N) const;
   const CodeGenInstruction *getEquivNode(Record &Equiv,
                                          const TreePatternNode *N) const;
 
   Error importRulePredicates(RuleMatcher &M, ArrayRef<Predicate> Predicates);
   Expected<InstructionMatcher &> createAndImportSelDAGMatcher(
       RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
       const TreePatternNode *Src, unsigned &TempOpIdx) const;
   Error importComplexPatternOperandMatcher(OperandMatcher &OM, Record *R,
                                            unsigned &TempOpIdx) const;
   Error importChildMatcher(RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
                            const TreePatternNode *SrcChild,
                            bool OperandIsAPointer, unsigned OpIdx,
                            unsigned &TempOpIdx) const;
 
   Expected<BuildMIAction &>
   createAndImportInstructionRenderer(RuleMatcher &M,
                                      const TreePatternNode *Dst);
   Expected<action_iterator> createAndImportSubInstructionRenderer(
       action_iterator InsertPt, RuleMatcher &M, const TreePatternNode *Dst,
       unsigned TempReg);
   Expected<action_iterator>
   createInstructionRenderer(action_iterator InsertPt, RuleMatcher &M,
                             const TreePatternNode *Dst);
   void importExplicitDefRenderers(BuildMIAction &DstMIBuilder);
   Expected<action_iterator>
   importExplicitUseRenderers(action_iterator InsertPt, RuleMatcher &M,
                              BuildMIAction &DstMIBuilder,
                              const llvm::TreePatternNode *Dst);
   Expected<action_iterator>
   importExplicitUseRenderer(action_iterator InsertPt, RuleMatcher &Rule,
                             BuildMIAction &DstMIBuilder,
                             TreePatternNode *DstChild);
   Error importDefaultOperandRenderers(BuildMIAction &DstMIBuilder,
                                       DagInit *DefaultOps) const;
   Error
   importImplicitDefRenderers(BuildMIAction &DstMIBuilder,
                              const std::vector<Record *> &ImplicitDefs) const;
 
   void emitImmPredicates(raw_ostream &OS, StringRef TypeIdentifier,
                          StringRef Type,
                          std::function<bool(const Record *R)> Filter);
 
   /// Analyze pattern \p P, returning a matcher for it if possible.
   /// Otherwise, return an Error explaining why we don't support it.
   Expected<RuleMatcher> runOnPattern(const PatternToMatch &P);
 
   void declareSubtargetFeature(Record *Predicate);
 
   MatchTable buildMatchTable(MutableArrayRef<RuleMatcher> Rules, bool Optimize,
                              bool WithCoverage);
 
 public:
   /// Takes a sequence of \p Rules and group them based on the predicates
   /// they share. \p MatcherStorage is used as a memory container
   /// for the group that are created as part of this process.
   ///
   /// What this optimization does looks like if GroupT = GroupMatcher:
   /// Output without optimization:
   /// \verbatim
   /// # R1
   ///  # predicate A
   ///  # predicate B
   ///  ...
   /// # R2
   ///  # predicate A // <-- effectively this is going to be checked twice.
   ///                //     Once in R1 and once in R2.
   ///  # predicate C
   /// \endverbatim
   /// Output with optimization:
   /// \verbatim
   /// # Group1_2
   ///  # predicate A // <-- Check is now shared.
   ///  # R1
   ///   # predicate B
   ///  # R2
   ///   # predicate C
   /// \endverbatim
   template <class GroupT>
   static std::vector<Matcher *> optimizeRules(
       ArrayRef<Matcher *> Rules,
       std::vector<std::unique_ptr<Matcher>> &MatcherStorage);
 };
 
 void GlobalISelEmitter::gatherOpcodeValues() {
   InstructionOpcodeMatcher::initOpcodeValuesMap(Target);
 }
 
 void GlobalISelEmitter::gatherTypeIDValues() {
   LLTOperandMatcher::initTypeIDValuesMap();
 }
 
 void GlobalISelEmitter::gatherNodeEquivs() {
   assert(NodeEquivs.empty());
   for (Record *Equiv : RK.getAllDerivedDefinitions("GINodeEquiv"))
     NodeEquivs[Equiv->getValueAsDef("Node")] = Equiv;
 
   assert(ComplexPatternEquivs.empty());
   for (Record *Equiv : RK.getAllDerivedDefinitions("GIComplexPatternEquiv")) {
     Record *SelDAGEquiv = Equiv->getValueAsDef("SelDAGEquivalent");
     if (!SelDAGEquiv)
       continue;
     ComplexPatternEquivs[SelDAGEquiv] = Equiv;
  }
 
  assert(SDNodeXFormEquivs.empty());
  for (Record *Equiv : RK.getAllDerivedDefinitions("GISDNodeXFormEquiv")) {
    Record *SelDAGEquiv = Equiv->getValueAsDef("SelDAGEquivalent");
    if (!SelDAGEquiv)
      continue;
    SDNodeXFormEquivs[SelDAGEquiv] = Equiv;
  }
 }
 
 Record *GlobalISelEmitter::findNodeEquiv(Record *N) const {
   return NodeEquivs.lookup(N);
 }
 
 const CodeGenInstruction *
 GlobalISelEmitter::getEquivNode(Record &Equiv, const TreePatternNode *N) const {
   for (const auto &Predicate : N->getPredicateFns()) {
     if (!Equiv.isValueUnset("IfSignExtend") && Predicate.isLoad() &&
         Predicate.isSignExtLoad())
       return &Target.getInstruction(Equiv.getValueAsDef("IfSignExtend"));
     if (!Equiv.isValueUnset("IfZeroExtend") && Predicate.isLoad() &&
         Predicate.isZeroExtLoad())
       return &Target.getInstruction(Equiv.getValueAsDef("IfZeroExtend"));
   }
   return &Target.getInstruction(Equiv.getValueAsDef("I"));
 }
 
 GlobalISelEmitter::GlobalISelEmitter(RecordKeeper &RK)
     : RK(RK), CGP(RK), Target(CGP.getTargetInfo()),
       CGRegs(RK, Target.getHwModes()) {}
 
 //===- Emitter ------------------------------------------------------------===//
 
 Error
 GlobalISelEmitter::importRulePredicates(RuleMatcher &M,
                                         ArrayRef<Predicate> Predicates) {
   for (const Predicate &P : Predicates) {
     if (!P.Def)
       continue;
     declareSubtargetFeature(P.Def);
     M.addRequiredFeature(P.Def);
   }
 
   return Error::success();
 }
 
 Expected<InstructionMatcher &> GlobalISelEmitter::createAndImportSelDAGMatcher(
     RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
     const TreePatternNode *Src, unsigned &TempOpIdx) const {
   Record *SrcGIEquivOrNull = nullptr;
   const CodeGenInstruction *SrcGIOrNull = nullptr;
 
   // Start with the defined operands (i.e., the results of the root operator).
   if (Src->getExtTypes().size() > 1)
     return failedImport("Src pattern has multiple results");
 
   if (Src->isLeaf()) {
     Init *SrcInit = Src->getLeafValue();
     if (isa<IntInit>(SrcInit)) {
       InsnMatcher.addPredicate<InstructionOpcodeMatcher>(
           &Target.getInstruction(RK.getDef("G_CONSTANT")));
     } else
       return failedImport(
           "Unable to deduce gMIR opcode to handle Src (which is a leaf)");
   } else {
     SrcGIEquivOrNull = findNodeEquiv(Src->getOperator());
     if (!SrcGIEquivOrNull)
       return failedImport("Pattern operator lacks an equivalent Instruction" +
                           explainOperator(Src->getOperator()));
     SrcGIOrNull = getEquivNode(*SrcGIEquivOrNull, Src);
 
     // The operators look good: match the opcode
     InsnMatcher.addPredicate<InstructionOpcodeMatcher>(SrcGIOrNull);
   }
 
   unsigned OpIdx = 0;
   for (const TypeSetByHwMode &VTy : Src->getExtTypes()) {
     // Results don't have a name unless they are the root node. The caller will
     // set the name if appropriate.
     OperandMatcher &OM = InsnMatcher.addOperand(OpIdx++, "", TempOpIdx);
     if (auto Error = OM.addTypeCheckPredicate(VTy, false /* OperandIsAPointer */))
       return failedImport(toString(std::move(Error)) +
                           " for result of Src pattern operator");
   }
 
   for (const auto &Predicate : Src->getPredicateFns()) {
     if (Predicate.isAlwaysTrue())
       continue;
 
     if (Predicate.isImmediatePattern()) {
       InsnMatcher.addPredicate<InstructionImmPredicateMatcher>(Predicate);
       continue;
     }
 
     // G_LOAD is used for both non-extending and any-extending loads. 
     if (Predicate.isLoad() && Predicate.isNonExtLoad()) {
       InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
           0, MemoryVsLLTSizePredicateMatcher::EqualTo, 0);
       continue;
     }
     if (Predicate.isLoad() && Predicate.isAnyExtLoad()) {
       InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
           0, MemoryVsLLTSizePredicateMatcher::LessThan, 0);
       continue;
     }
 
     // No check required. We already did it by swapping the opcode.
     if (!SrcGIEquivOrNull->isValueUnset("IfSignExtend") &&
         Predicate.isSignExtLoad())
       continue;
 
     // No check required. We already did it by swapping the opcode.
     if (!SrcGIEquivOrNull->isValueUnset("IfZeroExtend") &&
         Predicate.isZeroExtLoad())
       continue;
 
     // No check required. G_STORE by itself is a non-extending store.
     if (Predicate.isNonTruncStore())
       continue;
 
     if (Predicate.isLoad() || Predicate.isStore() || Predicate.isAtomic()) {
       if (Predicate.getMemoryVT() != nullptr) {
         Optional<LLTCodeGen> MemTyOrNone =
             MVTToLLT(getValueType(Predicate.getMemoryVT()));
 
         if (!MemTyOrNone)
           return failedImport("MemVT could not be converted to LLT");
 
         // MMO's work in bytes so we must take care of unusual types like i1
         // don't round down.
         unsigned MemSizeInBits =
             llvm::alignTo(MemTyOrNone->get().getSizeInBits(), 8);
 
         InsnMatcher.addPredicate<MemorySizePredicateMatcher>(
             0, MemSizeInBits / 8);
         continue;
       }
     }
 
     if (Predicate.isLoad() || Predicate.isStore()) {
       // No check required. A G_LOAD/G_STORE is an unindexed load.
       if (Predicate.isUnindexed())
         continue;
     }
 
     if (Predicate.isAtomic()) {
       if (Predicate.isAtomicOrderingMonotonic()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "Monotonic");
         continue;
       }
       if (Predicate.isAtomicOrderingAcquire()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>("Acquire");
         continue;
       }
       if (Predicate.isAtomicOrderingRelease()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>("Release");
         continue;
       }
       if (Predicate.isAtomicOrderingAcquireRelease()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "AcquireRelease");
         continue;
       }
       if (Predicate.isAtomicOrderingSequentiallyConsistent()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "SequentiallyConsistent");
         continue;
       }
 
       if (Predicate.isAtomicOrderingAcquireOrStronger()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "Acquire", AtomicOrderingMMOPredicateMatcher::AO_OrStronger);
         continue;
       }
       if (Predicate.isAtomicOrderingWeakerThanAcquire()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "Acquire", AtomicOrderingMMOPredicateMatcher::AO_WeakerThan);
         continue;
       }
 
       if (Predicate.isAtomicOrderingReleaseOrStronger()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "Release", AtomicOrderingMMOPredicateMatcher::AO_OrStronger);
         continue;
       }
       if (Predicate.isAtomicOrderingWeakerThanRelease()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "Release", AtomicOrderingMMOPredicateMatcher::AO_WeakerThan);
         continue;
       }
     }
 
     return failedImport("Src pattern child has predicate (" +
                         explainPredicates(Src) + ")");
   }
   if (SrcGIEquivOrNull && SrcGIEquivOrNull->getValueAsBit("CheckMMOIsNonAtomic"))
     InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>("NotAtomic");
 
   if (Src->isLeaf()) {
     Init *SrcInit = Src->getLeafValue();
     if (IntInit *SrcIntInit = dyn_cast<IntInit>(SrcInit)) {
       OperandMatcher &OM =
           InsnMatcher.addOperand(OpIdx++, Src->getName(), TempOpIdx);
       OM.addPredicate<LiteralIntOperandMatcher>(SrcIntInit->getValue());
     } else
       return failedImport(
           "Unable to deduce gMIR opcode to handle Src (which is a leaf)");
   } else {
     assert(SrcGIOrNull &&
            "Expected to have already found an equivalent Instruction");
     if (SrcGIOrNull->TheDef->getName() == "G_CONSTANT" ||
         SrcGIOrNull->TheDef->getName() == "G_FCONSTANT") {
       // imm/fpimm still have operands but we don't need to do anything with it
       // here since we don't support ImmLeaf predicates yet. However, we still
       // need to note the hidden operand to get GIM_CheckNumOperands correct.
       InsnMatcher.addOperand(OpIdx++, "", TempOpIdx);
       return InsnMatcher;
     }
 
     // Match the used operands (i.e. the children of the operator).
     for (unsigned i = 0, e = Src->getNumChildren(); i != e; ++i) {
       TreePatternNode *SrcChild = Src->getChild(i);
 
       // SelectionDAG allows pointers to be represented with iN since it doesn't
       // distinguish between pointers and integers but they are different types in GlobalISel.
       // Coerce integers to pointers to address space 0 if the context indicates a pointer.
       bool OperandIsAPointer = SrcGIOrNull->isOperandAPointer(i);
 
       // For G_INTRINSIC/G_INTRINSIC_W_SIDE_EFFECTS, the operand immediately
       // following the defs is an intrinsic ID.
       if ((SrcGIOrNull->TheDef->getName() == "G_INTRINSIC" ||
            SrcGIOrNull->TheDef->getName() == "G_INTRINSIC_W_SIDE_EFFECTS") &&
           i == 0) {
         if (const CodeGenIntrinsic *II = Src->getIntrinsicInfo(CGP)) {
           OperandMatcher &OM =
               InsnMatcher.addOperand(OpIdx++, SrcChild->getName(), TempOpIdx);
           OM.addPredicate<IntrinsicIDOperandMatcher>(II);
           continue;
         }
 
         return failedImport("Expected IntInit containing instrinsic ID)");
       }
 
       if (auto Error =
               importChildMatcher(Rule, InsnMatcher, SrcChild, OperandIsAPointer,
                                  OpIdx++, TempOpIdx))
         return std::move(Error);
     }
   }
 
   return InsnMatcher;
 }
 
 Error GlobalISelEmitter::importComplexPatternOperandMatcher(
     OperandMatcher &OM, Record *R, unsigned &TempOpIdx) const {
   const auto &ComplexPattern = ComplexPatternEquivs.find(R);
   if (ComplexPattern == ComplexPatternEquivs.end())
     return failedImport("SelectionDAG ComplexPattern (" + R->getName() +
                         ") not mapped to GlobalISel");
 
   OM.addPredicate<ComplexPatternOperandMatcher>(OM, *ComplexPattern->second);
   TempOpIdx++;
   return Error::success();
 }
 
 Error GlobalISelEmitter::importChildMatcher(RuleMatcher &Rule,
                                             InstructionMatcher &InsnMatcher,
                                             const TreePatternNode *SrcChild,
                                             bool OperandIsAPointer,
                                             unsigned OpIdx,
                                             unsigned &TempOpIdx) const {
   OperandMatcher &OM =
       InsnMatcher.addOperand(OpIdx, SrcChild->getName(), TempOpIdx);
   if (OM.isSameAsAnotherOperand())
     return Error::success();
 
   ArrayRef<TypeSetByHwMode> ChildTypes = SrcChild->getExtTypes();
   if (ChildTypes.size() != 1)
     return failedImport("Src pattern child has multiple results");
 
   // Check MBB's before the type check since they are not a known type.
   if (!SrcChild->isLeaf()) {
     if (SrcChild->getOperator()->isSubClassOf("SDNode")) {
       auto &ChildSDNI = CGP.getSDNodeInfo(SrcChild->getOperator());
       if (ChildSDNI.getSDClassName() == "BasicBlockSDNode") {
         OM.addPredicate<MBBOperandMatcher>();
         return Error::success();
       }
     }
   }
 
   if (auto Error =
           OM.addTypeCheckPredicate(ChildTypes.front(), OperandIsAPointer))
     return failedImport(toString(std::move(Error)) + " for Src operand (" +
                         to_string(*SrcChild) + ")");
 
   // Check for nested instructions.
   if (!SrcChild->isLeaf()) {
     if (SrcChild->getOperator()->isSubClassOf("ComplexPattern")) {
       // When a ComplexPattern is used as an operator, it should do the same
       // thing as when used as a leaf. However, the children of the operator
       // name the sub-operands that make up the complex operand and we must
       // prepare to reference them in the renderer too.
       unsigned RendererID = TempOpIdx;
       if (auto Error = importComplexPatternOperandMatcher(
               OM, SrcChild->getOperator(), TempOpIdx))
         return Error;
 
       for (unsigned i = 0, e = SrcChild->getNumChildren(); i != e; ++i) {
         auto *SubOperand = SrcChild->getChild(i);
         if (!SubOperand->getName().empty())
           Rule.defineComplexSubOperand(SubOperand->getName(),
                                        SrcChild->getOperator(), RendererID, i);
       }
 
       return Error::success();
     }
 
     auto MaybeInsnOperand = OM.addPredicate<InstructionOperandMatcher>(
         InsnMatcher.getRuleMatcher(), SrcChild->getName());
     if (!MaybeInsnOperand.hasValue()) {
       // This isn't strictly true. If the user were to provide exactly the same
       // matchers as the original operand then we could allow it. However, it's
       // simpler to not permit the redundant specification.
       return failedImport("Nested instruction cannot be the same as another operand");
     }
 
     // Map the node to a gMIR instruction.
     InstructionOperandMatcher &InsnOperand = **MaybeInsnOperand;
     auto InsnMatcherOrError = createAndImportSelDAGMatcher(
         Rule, InsnOperand.getInsnMatcher(), SrcChild, TempOpIdx);
     if (auto Error = InsnMatcherOrError.takeError())
       return Error;
 
     return Error::success();
   }
 
   if (SrcChild->hasAnyPredicate())
     return failedImport("Src pattern child has unsupported predicate");
 
   // Check for constant immediates.
   if (auto *ChildInt = dyn_cast<IntInit>(SrcChild->getLeafValue())) {
     OM.addPredicate<ConstantIntOperandMatcher>(ChildInt->getValue());
     return Error::success();
   }
 
   // Check for def's like register classes or ComplexPattern's.
   if (auto *ChildDefInit = dyn_cast<DefInit>(SrcChild->getLeafValue())) {
     auto *ChildRec = ChildDefInit->getDef();
 
     // Check for register classes.
     if (ChildRec->isSubClassOf("RegisterClass") ||
         ChildRec->isSubClassOf("RegisterOperand")) {
       OM.addPredicate<RegisterBankOperandMatcher>(
           Target.getRegisterClass(getInitValueAsRegClass(ChildDefInit)));
       return Error::success();
     }
 
     // Check for ValueType.
     if (ChildRec->isSubClassOf("ValueType")) {
       // We already added a type check as standard practice so this doesn't need
       // to do anything.
       return Error::success();
     }
 
     // Check for ComplexPattern's.
     if (ChildRec->isSubClassOf("ComplexPattern"))
       return importComplexPatternOperandMatcher(OM, ChildRec, TempOpIdx);
 
     if (ChildRec->isSubClassOf("ImmLeaf")) {
       return failedImport(
           "Src pattern child def is an unsupported tablegen class (ImmLeaf)");
     }
 
     return failedImport(
         "Src pattern child def is an unsupported tablegen class");
   }
 
   return failedImport("Src pattern child is an unsupported kind");
 }
 
 Expected<action_iterator> GlobalISelEmitter::importExplicitUseRenderer(
     action_iterator InsertPt, RuleMatcher &Rule, BuildMIAction &DstMIBuilder,
     TreePatternNode *DstChild) {
 
   const auto &SubOperand = Rule.getComplexSubOperand(DstChild->getName());
   if (SubOperand.hasValue()) {
     DstMIBuilder.addRenderer<RenderComplexPatternOperand>(
         *std::get<0>(*SubOperand), DstChild->getName(),
         std::get<1>(*SubOperand), std::get<2>(*SubOperand));
     return InsertPt;
   }
 
   if (!DstChild->isLeaf()) {
 
     if (DstChild->getOperator()->isSubClassOf("SDNodeXForm")) {
       auto Child = DstChild->getChild(0);
       auto I = SDNodeXFormEquivs.find(DstChild->getOperator());
       if (I != SDNodeXFormEquivs.end()) {
         DstMIBuilder.addRenderer<CustomRenderer>(*I->second, Child->getName());
         return InsertPt;
       }
       return failedImport("SDNodeXForm " + Child->getName() +
                           " has no custom renderer");
     }
 
     // We accept 'bb' here. It's an operator because BasicBlockSDNode isn't
     // inline, but in MI it's just another operand.
     if (DstChild->getOperator()->isSubClassOf("SDNode")) {
       auto &ChildSDNI = CGP.getSDNodeInfo(DstChild->getOperator());
       if (ChildSDNI.getSDClassName() == "BasicBlockSDNode") {
         DstMIBuilder.addRenderer<CopyRenderer>(DstChild->getName());
         return InsertPt;
       }
     }
 
     // Similarly, imm is an operator in TreePatternNode's view but must be
     // rendered as operands.
     // FIXME: The target should be able to choose sign-extended when appropriate
     //        (e.g. on Mips).
     if (DstChild->getOperator()->getName() == "imm") {
       DstMIBuilder.addRenderer<CopyConstantAsImmRenderer>(DstChild->getName());
       return InsertPt;
     } else if (DstChild->getOperator()->getName() == "fpimm") {
       DstMIBuilder.addRenderer<CopyFConstantAsFPImmRenderer>(
           DstChild->getName());
       return InsertPt;
     }
 
     if (DstChild->getOperator()->isSubClassOf("Instruction")) {
       ArrayRef<TypeSetByHwMode> ChildTypes = DstChild->getExtTypes();
       if (ChildTypes.size() != 1)
         return failedImport("Dst pattern child has multiple results");
 
       Optional<LLTCodeGen> OpTyOrNone = None;
       if (ChildTypes.front().isMachineValueType())
         OpTyOrNone =
             MVTToLLT(ChildTypes.front().getMachineValueType().SimpleTy);
       if (!OpTyOrNone)
         return failedImport("Dst operand has an unsupported type");
 
       unsigned TempRegID = Rule.allocateTempRegID();
       InsertPt = Rule.insertAction<MakeTempRegisterAction>(
           InsertPt, OpTyOrNone.getValue(), TempRegID);
       DstMIBuilder.addRenderer<TempRegRenderer>(TempRegID);
 
       auto InsertPtOrError = createAndImportSubInstructionRenderer(
           ++InsertPt, Rule, DstChild, TempRegID);
       if (auto Error = InsertPtOrError.takeError())
         return std::move(Error);
       return InsertPtOrError.get();
     }
 
     return failedImport("Dst pattern child isn't a leaf node or an MBB" + llvm::to_string(*DstChild));
   }
 
   // It could be a specific immediate in which case we should just check for
   // that immediate.
   if (const IntInit *ChildIntInit =
           dyn_cast<IntInit>(DstChild->getLeafValue())) {
     DstMIBuilder.addRenderer<ImmRenderer>(ChildIntInit->getValue());
     return InsertPt;
   }
 
   // Otherwise, we're looking for a bog-standard RegisterClass operand.
   if (auto *ChildDefInit = dyn_cast<DefInit>(DstChild->getLeafValue())) {
     auto *ChildRec = ChildDefInit->getDef();
 
     ArrayRef<TypeSetByHwMode> ChildTypes = DstChild->getExtTypes();
     if (ChildTypes.size() != 1)
       return failedImport("Dst pattern child has multiple results");
 
     Optional<LLTCodeGen> OpTyOrNone = None;
     if (ChildTypes.front().isMachineValueType())
       OpTyOrNone = MVTToLLT(ChildTypes.front().getMachineValueType().SimpleTy);
     if (!OpTyOrNone)
       return failedImport("Dst operand has an unsupported type");
 
     if (ChildRec->isSubClassOf("Register")) {
       DstMIBuilder.addRenderer<AddRegisterRenderer>(ChildRec);
       return InsertPt;
     }
 
     if (ChildRec->isSubClassOf("RegisterClass") ||
         ChildRec->isSubClassOf("RegisterOperand") ||
         ChildRec->isSubClassOf("ValueType")) {
       if (ChildRec->isSubClassOf("RegisterOperand") &&
           !ChildRec->isValueUnset("GIZeroRegister")) {
         DstMIBuilder.addRenderer<CopyOrAddZeroRegRenderer>(
             DstChild->getName(), ChildRec->getValueAsDef("GIZeroRegister"));
         return InsertPt;
       }
 
       DstMIBuilder.addRenderer<CopyRenderer>(DstChild->getName());
       return InsertPt;
     }
 
     if (ChildRec->isSubClassOf("ComplexPattern")) {
       const auto &ComplexPattern = ComplexPatternEquivs.find(ChildRec);
       if (ComplexPattern == ComplexPatternEquivs.end())
         return failedImport(
             "SelectionDAG ComplexPattern not mapped to GlobalISel");
 
       const OperandMatcher &OM = Rule.getOperandMatcher(DstChild->getName());
       DstMIBuilder.addRenderer<RenderComplexPatternOperand>(
           *ComplexPattern->second, DstChild->getName(),
           OM.getAllocatedTemporariesBaseID());
       return InsertPt;
     }
 
     return failedImport(
         "Dst pattern child def is an unsupported tablegen class");
   }
 
   return failedImport("Dst pattern child is an unsupported kind");
 }
 
 Expected<BuildMIAction &> GlobalISelEmitter::createAndImportInstructionRenderer(
     RuleMatcher &M, const TreePatternNode *Dst) {
   auto InsertPtOrError = createInstructionRenderer(M.actions_end(), M, Dst);
   if (auto Error = InsertPtOrError.takeError())
     return std::move(Error);
 
   action_iterator InsertPt = InsertPtOrError.get();
   BuildMIAction &DstMIBuilder = *static_cast<BuildMIAction *>(InsertPt->get());
 
   importExplicitDefRenderers(DstMIBuilder);
 
   if (auto Error = importExplicitUseRenderers(InsertPt, M, DstMIBuilder, Dst)
                        .takeError())
     return std::move(Error);
 
   return DstMIBuilder;
 }
 
 Expected<action_iterator>
 GlobalISelEmitter::createAndImportSubInstructionRenderer(
     const action_iterator InsertPt, RuleMatcher &M, const TreePatternNode *Dst,
     unsigned TempRegID) {
   auto InsertPtOrError = createInstructionRenderer(InsertPt, M, Dst);
 
   // TODO: Assert there's exactly one result.
 
   if (auto Error = InsertPtOrError.takeError())
     return std::move(Error);
 
   BuildMIAction &DstMIBuilder =
       *static_cast<BuildMIAction *>(InsertPtOrError.get()->get());
 
   // Assign the result to TempReg.
   DstMIBuilder.addRenderer<TempRegRenderer>(TempRegID, true);
 
   InsertPtOrError =
       importExplicitUseRenderers(InsertPtOrError.get(), M, DstMIBuilder, Dst);
   if (auto Error = InsertPtOrError.takeError())
     return std::move(Error);
 
   M.insertAction<ConstrainOperandsToDefinitionAction>(InsertPt,
                                                       DstMIBuilder.getInsnID());
   return InsertPtOrError.get();
 }
 
 Expected<action_iterator> GlobalISelEmitter::createInstructionRenderer(
     action_iterator InsertPt, RuleMatcher &M, const TreePatternNode *Dst) {
   Record *DstOp = Dst->getOperator();
   if (!DstOp->isSubClassOf("Instruction")) {
     if (DstOp->isSubClassOf("ValueType"))
       return failedImport(
           "Pattern operator isn't an instruction (it's a ValueType)");
     return failedImport("Pattern operator isn't an instruction");
   }
   CodeGenInstruction *DstI = &Target.getInstruction(DstOp);
 
   // COPY_TO_REGCLASS is just a copy with a ConstrainOperandToRegClassAction
   // attached. Similarly for EXTRACT_SUBREG except that's a subregister copy.
   if (DstI->TheDef->getName() == "COPY_TO_REGCLASS")
     DstI = &Target.getInstruction(RK.getDef("COPY"));
   else if (DstI->TheDef->getName() == "EXTRACT_SUBREG")
     DstI = &Target.getInstruction(RK.getDef("COPY"));
   else if (DstI->TheDef->getName() == "REG_SEQUENCE")
     return failedImport("Unable to emit REG_SEQUENCE");
 
   return M.insertAction<BuildMIAction>(InsertPt, M.allocateOutputInsnID(),
                                        DstI);
 }
 
 void GlobalISelEmitter::importExplicitDefRenderers(
     BuildMIAction &DstMIBuilder) {
   const CodeGenInstruction *DstI = DstMIBuilder.getCGI();
   for (unsigned I = 0; I < DstI->Operands.NumDefs; ++I) {
     const CGIOperandList::OperandInfo &DstIOperand = DstI->Operands[I];
     DstMIBuilder.addRenderer<CopyRenderer>(DstIOperand.Name);
   }
 }
 
 Expected<action_iterator> GlobalISelEmitter::importExplicitUseRenderers(
     action_iterator InsertPt, RuleMatcher &M, BuildMIAction &DstMIBuilder,
     const llvm::TreePatternNode *Dst) {
   const CodeGenInstruction *DstI = DstMIBuilder.getCGI();
   CodeGenInstruction *OrigDstI = &Target.getInstruction(Dst->getOperator());
 
   // EXTRACT_SUBREG needs to use a subregister COPY.
   if (OrigDstI->TheDef->getName() == "EXTRACT_SUBREG") {
     if (!Dst->getChild(0)->isLeaf())
       return failedImport("EXTRACT_SUBREG child #1 is not a leaf");
 
     if (DefInit *SubRegInit =
             dyn_cast<DefInit>(Dst->getChild(1)->getLeafValue())) {
       Record *RCDef = getInitValueAsRegClass(Dst->getChild(0)->getLeafValue());
       if (!RCDef)
         return failedImport("EXTRACT_SUBREG child #0 could not "
                             "be coerced to a register class");
 
       CodeGenRegisterClass *RC = CGRegs.getRegClass(RCDef);
       CodeGenSubRegIndex *SubIdx = CGRegs.getSubRegIdx(SubRegInit->getDef());
 
       const auto &SrcRCDstRCPair =
           RC->getMatchingSubClassWithSubRegs(CGRegs, SubIdx);
       if (SrcRCDstRCPair.hasValue()) {
         assert(SrcRCDstRCPair->second && "Couldn't find a matching subclass");
         if (SrcRCDstRCPair->first != RC)
           return failedImport("EXTRACT_SUBREG requires an additional COPY");
       }
 
       DstMIBuilder.addRenderer<CopySubRegRenderer>(Dst->getChild(0)->getName(),
                                                    SubIdx);
       return InsertPt;
     }
 
     return failedImport("EXTRACT_SUBREG child #1 is not a subreg index");
   }
 
   // Render the explicit uses.
   unsigned DstINumUses = OrigDstI->Operands.size() - OrigDstI->Operands.NumDefs;
   unsigned ExpectedDstINumUses = Dst->getNumChildren();
   if (OrigDstI->TheDef->getName() == "COPY_TO_REGCLASS") {
     DstINumUses--; // Ignore the class constraint.
     ExpectedDstINumUses--;
   }
 
   unsigned Child = 0;
   unsigned NumDefaultOps = 0;
   for (unsigned I = 0; I != DstINumUses; ++I) {
     const CGIOperandList::OperandInfo &DstIOperand =
         DstI->Operands[DstI->Operands.NumDefs + I];
 
     // If the operand has default values, introduce them now.
     // FIXME: Until we have a decent test case that dictates we should do
     // otherwise, we're going to assume that operands with default values cannot
     // be specified in the patterns. Therefore, adding them will not cause us to
     // end up with too many rendered operands.
     if (DstIOperand.Rec->isSubClassOf("OperandWithDefaultOps")) {
       DagInit *DefaultOps = DstIOperand.Rec->getValueAsDag("DefaultOps");
       if (auto Error = importDefaultOperandRenderers(DstMIBuilder, DefaultOps))
         return std::move(Error);
       ++NumDefaultOps;
       continue;
     }
 
     auto InsertPtOrError = importExplicitUseRenderer(InsertPt, M, DstMIBuilder,
                                                      Dst->getChild(Child));
     if (auto Error = InsertPtOrError.takeError())
       return std::move(Error);
     InsertPt = InsertPtOrError.get();
     ++Child;
   }
 
   if (NumDefaultOps + ExpectedDstINumUses != DstINumUses)
     return failedImport("Expected " + llvm::to_string(DstINumUses) +
                         " used operands but found " +
                         llvm::to_string(ExpectedDstINumUses) +
                         " explicit ones and " + llvm::to_string(NumDefaultOps) +
                         " default ones");
 
   return InsertPt;
 }
 
 Error GlobalISelEmitter::importDefaultOperandRenderers(
     BuildMIAction &DstMIBuilder, DagInit *DefaultOps) const {
   for (const auto *DefaultOp : DefaultOps->getArgs()) {
     // Look through ValueType operators.
     if (const DagInit *DefaultDagOp = dyn_cast<DagInit>(DefaultOp)) {
       if (const DefInit *DefaultDagOperator =
               dyn_cast<DefInit>(DefaultDagOp->getOperator())) {
         if (DefaultDagOperator->getDef()->isSubClassOf("ValueType"))
           DefaultOp = DefaultDagOp->getArg(0);
       }
     }
 
     if (const DefInit *DefaultDefOp = dyn_cast<DefInit>(DefaultOp)) {
       DstMIBuilder.addRenderer<AddRegisterRenderer>(DefaultDefOp->getDef());
       continue;
     }
 
     if (const IntInit *DefaultIntOp = dyn_cast<IntInit>(DefaultOp)) {
       DstMIBuilder.addRenderer<ImmRenderer>(DefaultIntOp->getValue());
       continue;
     }
 
     return failedImport("Could not add default op");
   }
 
   return Error::success();
 }
 
 Error GlobalISelEmitter::importImplicitDefRenderers(
     BuildMIAction &DstMIBuilder,
     const std::vector<Record *> &ImplicitDefs) const {
   if (!ImplicitDefs.empty())
     return failedImport("Pattern defines a physical register");
   return Error::success();
 }
 
 Expected<RuleMatcher> GlobalISelEmitter::runOnPattern(const PatternToMatch &P) {
   // Keep track of the matchers and actions to emit.
   int Score = P.getPatternComplexity(CGP);
   RuleMatcher M(P.getSrcRecord()->getLoc());
   RuleMatcherScores[M.getRuleID()] = Score;
   M.addAction<DebugCommentAction>(llvm::to_string(*P.getSrcPattern()) +
                                   "  =>  " +
                                   llvm::to_string(*P.getDstPattern()));
 
   if (auto Error = importRulePredicates(M, P.getPredicates()))
     return std::move(Error);
 
   // Next, analyze the pattern operators.
   TreePatternNode *Src = P.getSrcPattern();
   TreePatternNode *Dst = P.getDstPattern();
 
   // If the root of either pattern isn't a simple operator, ignore it.
   if (auto Err = isTrivialOperatorNode(Dst))
     return failedImport("Dst pattern root isn't a trivial operator (" +
                         toString(std::move(Err)) + ")");
   if (auto Err = isTrivialOperatorNode(Src))
     return failedImport("Src pattern root isn't a trivial operator (" +
                         toString(std::move(Err)) + ")");
 
   // The different predicates and matchers created during
   // addInstructionMatcher use the RuleMatcher M to set up their
   // instruction ID (InsnVarID) that are going to be used when
   // M is going to be emitted.
   // However, the code doing the emission still relies on the IDs
   // returned during that process by the RuleMatcher when issuing
   // the recordInsn opcodes.
   // Because of that:
   // 1. The order in which we created the predicates
   //    and such must be the same as the order in which we emit them,
   //    and
   // 2. We need to reset the generation of the IDs in M somewhere between
   //    addInstructionMatcher and emit
   //
   // FIXME: Long term, we don't want to have to rely on this implicit
   // naming being the same. One possible solution would be to have
   // explicit operator for operation capture and reference those.
   // The plus side is that it would expose opportunities to share
   // the capture accross rules. The downside is that it would
   // introduce a dependency between predicates (captures must happen
   // before their first use.)
   InstructionMatcher &InsnMatcherTemp = M.addInstructionMatcher(Src->getName());
   unsigned TempOpIdx = 0;
   auto InsnMatcherOrError =
       createAndImportSelDAGMatcher(M, InsnMatcherTemp, Src, TempOpIdx);
   if (auto Error = InsnMatcherOrError.takeError())
     return std::move(Error);
   InstructionMatcher &InsnMatcher = InsnMatcherOrError.get();
 
   if (Dst->isLeaf()) {
     Record *RCDef = getInitValueAsRegClass(Dst->getLeafValue());
 
     const CodeGenRegisterClass &RC = Target.getRegisterClass(RCDef);
     if (RCDef) {
       // We need to replace the def and all its uses with the specified
       // operand. However, we must also insert COPY's wherever needed.
       // For now, emit a copy and let the register allocator clean up.
       auto &DstI = Target.getInstruction(RK.getDef("COPY"));
       const auto &DstIOperand = DstI.Operands[0];
 
       OperandMatcher &OM0 = InsnMatcher.getOperand(0);
       OM0.setSymbolicName(DstIOperand.Name);
       M.defineOperand(OM0.getSymbolicName(), OM0);
       OM0.addPredicate<RegisterBankOperandMatcher>(RC);
 
       auto &DstMIBuilder =
           M.addAction<BuildMIAction>(M.allocateOutputInsnID(), &DstI);
       DstMIBuilder.addRenderer<CopyRenderer>(DstIOperand.Name);
       DstMIBuilder.addRenderer<CopyRenderer>(Dst->getName());
       M.addAction<ConstrainOperandToRegClassAction>(0, 0, RC);
 
       // We're done with this pattern!  It's eligible for GISel emission; return
       // it.
       ++NumPatternImported;
       return std::move(M);
     }
 
     return failedImport("Dst pattern root isn't a known leaf");
   }
 
   // Start with the defined operands (i.e., the results of the root operator).
   Record *DstOp = Dst->getOperator();
   if (!DstOp->isSubClassOf("Instruction"))
     return failedImport("Pattern operator isn't an instruction");
 
   auto &DstI = Target.getInstruction(DstOp);
   if (DstI.Operands.NumDefs != Src->getExtTypes().size())
     return failedImport("Src pattern results and dst MI defs are different (" +
                         to_string(Src->getExtTypes().size()) + " def(s) vs " +
                         to_string(DstI.Operands.NumDefs) + " def(s))");
 
   // The root of the match also has constraints on the register bank so that it
   // matches the result instruction.
   unsigned OpIdx = 0;
   for (const TypeSetByHwMode &VTy : Src->getExtTypes()) {
     (void)VTy;
 
     const auto &DstIOperand = DstI.Operands[OpIdx];
     Record *DstIOpRec = DstIOperand.Rec;
     if (DstI.TheDef->getName() == "COPY_TO_REGCLASS") {
       DstIOpRec = getInitValueAsRegClass(Dst->getChild(1)->getLeafValue());
 
       if (DstIOpRec == nullptr)
         return failedImport(
             "COPY_TO_REGCLASS operand #1 isn't a register class");
     } else if (DstI.TheDef->getName() == "EXTRACT_SUBREG") {
       if (!Dst->getChild(0)->isLeaf())
         return failedImport("EXTRACT_SUBREG operand #0 isn't a leaf");
 
       // We can assume that a subregister is in the same bank as it's super
       // register.
       DstIOpRec = getInitValueAsRegClass(Dst->getChild(0)->getLeafValue());
 
       if (DstIOpRec == nullptr)
         return failedImport(
             "EXTRACT_SUBREG operand #0 isn't a register class");
     } else if (DstIOpRec->isSubClassOf("RegisterOperand"))
       DstIOpRec = DstIOpRec->getValueAsDef("RegClass");
     else if (!DstIOpRec->isSubClassOf("RegisterClass"))
       return failedImport("Dst MI def isn't a register class" +
                           to_string(*Dst));
 
     OperandMatcher &OM = InsnMatcher.getOperand(OpIdx);
     OM.setSymbolicName(DstIOperand.Name);
     M.defineOperand(OM.getSymbolicName(), OM);
     OM.addPredicate<RegisterBankOperandMatcher>(
         Target.getRegisterClass(DstIOpRec));
     ++OpIdx;
   }
 
   auto DstMIBuilderOrError = createAndImportInstructionRenderer(M, Dst);
   if (auto Error = DstMIBuilderOrError.takeError())
     return std::move(Error);
   BuildMIAction &DstMIBuilder = DstMIBuilderOrError.get();
 
   // Render the implicit defs.
   // These are only added to the root of the result.
   if (auto Error = importImplicitDefRenderers(DstMIBuilder, P.getDstRegs()))
     return std::move(Error);
 
   DstMIBuilder.chooseInsnToMutate(M);
 
   // Constrain the registers to classes. This is normally derived from the
   // emitted instruction but a few instructions require special handling.
   if (DstI.TheDef->getName() == "COPY_TO_REGCLASS") {
     // COPY_TO_REGCLASS does not provide operand constraints itself but the
     // result is constrained to the class given by the second child.
     Record *DstIOpRec =
         getInitValueAsRegClass(Dst->getChild(1)->getLeafValue());
 
     if (DstIOpRec == nullptr)
       return failedImport("COPY_TO_REGCLASS operand #1 isn't a register class");
 
     M.addAction<ConstrainOperandToRegClassAction>(
         0, 0, Target.getRegisterClass(DstIOpRec));
 
     // We're done with this pattern!  It's eligible for GISel emission; return
     // it.
     ++NumPatternImported;
     return std::move(M);
   }
 
   if (DstI.TheDef->getName() == "EXTRACT_SUBREG") {
     // EXTRACT_SUBREG selects into a subregister COPY but unlike most
     // instructions, the result register class is controlled by the
     // subregisters of the operand. As a result, we must constrain the result
     // class rather than check that it's already the right one.
     if (!Dst->getChild(0)->isLeaf())
       return failedImport("EXTRACT_SUBREG child #1 is not a leaf");
 
     DefInit *SubRegInit = dyn_cast<DefInit>(Dst->getChild(1)->getLeafValue());
     if (!SubRegInit)
       return failedImport("EXTRACT_SUBREG child #1 is not a subreg index");
 
     // Constrain the result to the same register bank as the operand.
     Record *DstIOpRec =
         getInitValueAsRegClass(Dst->getChild(0)->getLeafValue());
 
     if (DstIOpRec == nullptr)
       return failedImport("EXTRACT_SUBREG operand #1 isn't a register class");
 
     CodeGenSubRegIndex *SubIdx = CGRegs.getSubRegIdx(SubRegInit->getDef());
     CodeGenRegisterClass *SrcRC = CGRegs.getRegClass(DstIOpRec);
 
     // It would be nice to leave this constraint implicit but we're required
     // to pick a register class so constrain the result to a register class
     // that can hold the correct MVT.
     //
     // FIXME: This may introduce an extra copy if the chosen class doesn't
     //        actually contain the subregisters.
     assert(Src->getExtTypes().size() == 1 &&
              "Expected Src of EXTRACT_SUBREG to have one result type");
 
     const auto &SrcRCDstRCPair =
         SrcRC->getMatchingSubClassWithSubRegs(CGRegs, SubIdx);
     assert(SrcRCDstRCPair->second && "Couldn't find a matching subclass");
     M.addAction<ConstrainOperandToRegClassAction>(0, 0, *SrcRCDstRCPair->second);
     M.addAction<ConstrainOperandToRegClassAction>(0, 1, *SrcRCDstRCPair->first);
 
     // We're done with this pattern!  It's eligible for GISel emission; return
     // it.
     ++NumPatternImported;
     return std::move(M);
   }
 
   M.addAction<ConstrainOperandsToDefinitionAction>(0);
 
   // We're done with this pattern!  It's eligible for GISel emission; return it.
   ++NumPatternImported;
   return std::move(M);
 }
 
 // Emit imm predicate table and an enum to reference them with.
 // The 'Predicate_' part of the name is redundant but eliminating it is more
 // trouble than it's worth.
 void GlobalISelEmitter::emitImmPredicates(
     raw_ostream &OS, StringRef TypeIdentifier, StringRef Type,
     std::function<bool(const Record *R)> Filter) {
   std::vector<const Record *> MatchedRecords;
   const auto &Defs = RK.getAllDerivedDefinitions("PatFrag");
   std::copy_if(Defs.begin(), Defs.end(), std::back_inserter(MatchedRecords),
                [&](Record *Record) {
                  return !Record->getValueAsString("ImmediateCode").empty() &&
                         Filter(Record);
                });
 
   if (!MatchedRecords.empty()) {
     OS << "// PatFrag predicates.\n"
        << "enum {\n";
     std::string EnumeratorSeparator =
         (" = GIPFP_" + TypeIdentifier + "_Invalid + 1,\n").str();
     for (const auto *Record : MatchedRecords) {
       OS << "  GIPFP_" << TypeIdentifier << "_Predicate_" << Record->getName()
          << EnumeratorSeparator;
       EnumeratorSeparator = ",\n";
     }
     OS << "};\n";
   }
 
   OS << "bool " << Target.getName() << "InstructionSelector::testImmPredicate_"
      << TypeIdentifier << "(unsigned PredicateID, " << Type
      << " Imm) const {\n";
   if (!MatchedRecords.empty())
     OS << "  switch (PredicateID) {\n";
   for (const auto *Record : MatchedRecords) {
     OS << "  case GIPFP_" << TypeIdentifier << "_Predicate_"
        << Record->getName() << ": {\n"
        << "    " << Record->getValueAsString("ImmediateCode") << "\n"
        << "    llvm_unreachable(\"ImmediateCode should have returned\");\n"
        << "    return false;\n"
        << "  }\n";
   }
   if (!MatchedRecords.empty())
     OS << "  }\n";
   OS << "  llvm_unreachable(\"Unknown predicate\");\n"
      << "  return false;\n"
      << "}\n";
 }
 
 template <class GroupT>
 std::vector<Matcher *> GlobalISelEmitter::optimizeRules(
     ArrayRef<Matcher *> Rules,
     std::vector<std::unique_ptr<Matcher>> &MatcherStorage) {
 
   std::vector<Matcher *> OptRules;
   std::unique_ptr<GroupT> CurrentGroup = make_unique<GroupT>();
   assert(CurrentGroup->empty() && "Newly created group isn't empty!");
   unsigned NumGroups = 0;
 
   auto ProcessCurrentGroup = [&]() {
     if (CurrentGroup->empty())
       // An empty group is good to be reused:
       return;
 
     // If the group isn't large enough to provide any benefit, move all the
     // added rules out of it and make sure to re-create the group to properly
     // re-initialize it:
     if (CurrentGroup->size() < 2)
       for (Matcher *M : CurrentGroup->matchers())
         OptRules.push_back(M);
     else {
       CurrentGroup->finalize();
       OptRules.push_back(CurrentGroup.get());
       MatcherStorage.emplace_back(std::move(CurrentGroup));
       ++NumGroups;
     }
     CurrentGroup = make_unique<GroupT>();
   };
   for (Matcher *Rule : Rules) {
     // Greedily add as many matchers as possible to the current group:
     if (CurrentGroup->addMatcher(*Rule))
       continue;
 
     ProcessCurrentGroup();
     assert(CurrentGroup->empty() && "A group wasn't properly re-initialized");
 
     // Try to add the pending matcher to a newly created empty group:
     if (!CurrentGroup->addMatcher(*Rule))
       // If we couldn't add the matcher to an empty group, that group type
       // doesn't support that kind of matchers at all, so just skip it:
       OptRules.push_back(Rule);
   }
   ProcessCurrentGroup();
 
   DEBUG(dbgs() << "NumGroups: " << NumGroups << "\n");
   assert(CurrentGroup->empty() && "The last group wasn't properly processed");
   return OptRules;
 }
 
 MatchTable
 GlobalISelEmitter::buildMatchTable(MutableArrayRef<RuleMatcher> Rules,
                                    bool Optimize, bool WithCoverage) {
   std::vector<Matcher *> InputRules;
   for (Matcher &Rule : Rules)
     InputRules.push_back(&Rule);
 
   if (!Optimize)
     return MatchTable::buildTable(InputRules, WithCoverage);
 
+  unsigned CurrentOrdering = 0;
+  StringMap<unsigned> OpcodeOrder;
+  for (RuleMatcher &Rule : Rules) {
+    const StringRef Opcode = Rule.getOpcode();
+    assert(!Opcode.empty() && "Didn't expect an undefined opcode");
+    if (OpcodeOrder.count(Opcode) == 0)
+      OpcodeOrder[Opcode] = CurrentOrdering++;
+  }
+
+  std::stable_sort(InputRules.begin(), InputRules.end(),
+                   [&OpcodeOrder](const Matcher *A, const Matcher *B) {
+                     auto *L = static_cast<const RuleMatcher *>(A);
+                     auto *R = static_cast<const RuleMatcher *>(B);
+                     return std::make_tuple(OpcodeOrder[L->getOpcode()],
+                                            L->getNumOperands()) <
+                            std::make_tuple(OpcodeOrder[R->getOpcode()],
+                                            R->getNumOperands());
+                   });
+
   for (Matcher *Rule : InputRules)
     Rule->optimize();
 
   std::vector<std::unique_ptr<Matcher>> MatcherStorage;
   std::vector<Matcher *> OptRules =
       optimizeRules<GroupMatcher>(InputRules, MatcherStorage);
 
   for (Matcher *Rule : OptRules)
     Rule->optimize();
 
   return MatchTable::buildTable(OptRules, WithCoverage);
 }
 
 void GlobalISelEmitter::run(raw_ostream &OS) {
   if (!UseCoverageFile.empty()) {
     RuleCoverage = CodeGenCoverage();
     auto RuleCoverageBufOrErr = MemoryBuffer::getFile(UseCoverageFile);
     if (!RuleCoverageBufOrErr) {
       PrintWarning(SMLoc(), "Missing rule coverage data");
       RuleCoverage = None;
     } else {
       if (!RuleCoverage->parse(*RuleCoverageBufOrErr.get(), Target.getName())) {
         PrintWarning(SMLoc(), "Ignoring invalid or missing rule coverage data");
         RuleCoverage = None;
       }
     }
   }
 
   // Track the run-time opcode values
   gatherOpcodeValues();
   // Track the run-time LLT ID values
   gatherTypeIDValues();
 
   // Track the GINodeEquiv definitions.
   gatherNodeEquivs();
 
   emitSourceFileHeader(("Global Instruction Selector for the " +
                        Target.getName() + " target").str(), OS);
   std::vector<RuleMatcher> Rules;
   // Look through the SelectionDAG patterns we found, possibly emitting some.
   for (const PatternToMatch &Pat : CGP.ptms()) {
     ++NumPatternTotal;
 
     auto MatcherOrErr = runOnPattern(Pat);
 
     // The pattern analysis can fail, indicating an unsupported pattern.
     // Report that if we've been asked to do so.
     if (auto Err = MatcherOrErr.takeError()) {
       if (WarnOnSkippedPatterns) {
         PrintWarning(Pat.getSrcRecord()->getLoc(),
                      "Skipped pattern: " + toString(std::move(Err)));
       } else {
         consumeError(std::move(Err));
       }
       ++NumPatternImportsSkipped;
       continue;
     }
 
     if (RuleCoverage) {
       if (RuleCoverage->isCovered(MatcherOrErr->getRuleID()))
         ++NumPatternsTested;
       else
         PrintWarning(Pat.getSrcRecord()->getLoc(),
                      "Pattern is not covered by a test");
     }
     Rules.push_back(std::move(MatcherOrErr.get()));
   }
 
   // Comparison function to order records by name.
   auto orderByName = [](const Record *A, const Record *B) {
     return A->getName() < B->getName();
   };
 
   std::vector<Record *> ComplexPredicates =
       RK.getAllDerivedDefinitions("GIComplexOperandMatcher");
   llvm::sort(ComplexPredicates.begin(), ComplexPredicates.end(), orderByName);
 
   std::vector<Record *> CustomRendererFns =
       RK.getAllDerivedDefinitions("GICustomOperandRenderer");
   llvm::sort(CustomRendererFns.begin(), CustomRendererFns.end(), orderByName);
 
   unsigned MaxTemporaries = 0;
   for (const auto &Rule : Rules)
     MaxTemporaries = std::max(MaxTemporaries, Rule.countRendererFns());
 
   OS << "#ifdef GET_GLOBALISEL_PREDICATE_BITSET\n"
      << "const unsigned MAX_SUBTARGET_PREDICATES = " << SubtargetFeatures.size()
      << ";\n"
      << "using PredicateBitset = "
         "llvm::PredicateBitsetImpl<MAX_SUBTARGET_PREDICATES>;\n"
      << "#endif // ifdef GET_GLOBALISEL_PREDICATE_BITSET\n\n";
 
   OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n"
      << "  mutable MatcherState State;\n"
      << "  typedef "
         "ComplexRendererFns("
      << Target.getName()
      << "InstructionSelector::*ComplexMatcherMemFn)(MachineOperand &) const;\n"
 
      << "  typedef void(" << Target.getName()
      << "InstructionSelector::*CustomRendererFn)(MachineInstrBuilder &, const "
         "MachineInstr&) "
         "const;\n"
      << "  const ISelInfoTy<PredicateBitset, ComplexMatcherMemFn, "
         "CustomRendererFn> "
         "ISelInfo;\n";
   OS << "  static " << Target.getName()
      << "InstructionSelector::ComplexMatcherMemFn ComplexPredicateFns[];\n"
      << "  static " << Target.getName()
      << "InstructionSelector::CustomRendererFn CustomRenderers[];\n"
      << "  bool testImmPredicate_I64(unsigned PredicateID, int64_t Imm) const "
         "override;\n"
      << "  bool testImmPredicate_APInt(unsigned PredicateID, const APInt &Imm) "
         "const override;\n"
      << "  bool testImmPredicate_APFloat(unsigned PredicateID, const APFloat "
         "&Imm) const override;\n"
      << "  const int64_t *getMatchTable() const override;\n"
      << "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n\n";
 
   OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n"
      << ", State(" << MaxTemporaries << "),\n"
      << "ISelInfo(TypeObjects, NumTypeObjects, FeatureBitsets"
      << ", ComplexPredicateFns, CustomRenderers)\n"
      << "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n\n";
 
   OS << "#ifdef GET_GLOBALISEL_IMPL\n";
   SubtargetFeatureInfo::emitSubtargetFeatureBitEnumeration(SubtargetFeatures,
                                                            OS);
 
   // Separate subtarget features by how often they must be recomputed.
   SubtargetFeatureInfoMap ModuleFeatures;
   std::copy_if(SubtargetFeatures.begin(), SubtargetFeatures.end(),
                std::inserter(ModuleFeatures, ModuleFeatures.end()),
                [](const SubtargetFeatureInfoMap::value_type &X) {
                  return !X.second.mustRecomputePerFunction();
                });
   SubtargetFeatureInfoMap FunctionFeatures;
   std::copy_if(SubtargetFeatures.begin(), SubtargetFeatures.end(),
                std::inserter(FunctionFeatures, FunctionFeatures.end()),
                [](const SubtargetFeatureInfoMap::value_type &X) {
                  return X.second.mustRecomputePerFunction();
                });
 
   SubtargetFeatureInfo::emitComputeAvailableFeatures(
       Target.getName(), "InstructionSelector", "computeAvailableModuleFeatures",
       ModuleFeatures, OS);
   SubtargetFeatureInfo::emitComputeAvailableFeatures(
       Target.getName(), "InstructionSelector",
       "computeAvailableFunctionFeatures", FunctionFeatures, OS,
       "const MachineFunction *MF");
 
   // Emit a table containing the LLT objects needed by the matcher and an enum
   // for the matcher to reference them with.
   std::vector<LLTCodeGen> TypeObjects;
   for (const auto &Ty : KnownTypes)
     TypeObjects.push_back(Ty);
   llvm::sort(TypeObjects.begin(), TypeObjects.end());
   OS << "// LLT Objects.\n"
      << "enum {\n";
   for (const auto &TypeObject : TypeObjects) {
     OS << "  ";
     TypeObject.emitCxxEnumValue(OS);
     OS << ",\n";
   }
   OS << "};\n";
   OS << "const static size_t NumTypeObjects = " << TypeObjects.size() << ";\n"
      << "const static LLT TypeObjects[] = {\n";
   for (const auto &TypeObject : TypeObjects) {
     OS << "  ";
     TypeObject.emitCxxConstructorCall(OS);
     OS << ",\n";
   }
   OS << "};\n\n";
 
   // Emit a table containing the PredicateBitsets objects needed by the matcher
   // and an enum for the matcher to reference them with.
   std::vector<std::vector<Record *>> FeatureBitsets;
   for (auto &Rule : Rules)
     FeatureBitsets.push_back(Rule.getRequiredFeatures());
   llvm::sort(
       FeatureBitsets.begin(), FeatureBitsets.end(),
       [&](const std::vector<Record *> &A, const std::vector<Record *> &B) {
         if (A.size() < B.size())
           return true;
         if (A.size() > B.size())
           return false;
         for (const auto &Pair : zip(A, B)) {
           if (std::get<0>(Pair)->getName() < std::get<1>(Pair)->getName())
             return true;
           if (std::get<0>(Pair)->getName() > std::get<1>(Pair)->getName())
             return false;
         }
         return false;
       });
   FeatureBitsets.erase(
       std::unique(FeatureBitsets.begin(), FeatureBitsets.end()),
       FeatureBitsets.end());
   OS << "// Feature bitsets.\n"
      << "enum {\n"
      << "  GIFBS_Invalid,\n";
   for (const auto &FeatureBitset : FeatureBitsets) {
     if (FeatureBitset.empty())
       continue;
     OS << "  " << getNameForFeatureBitset(FeatureBitset) << ",\n";
   }
   OS << "};\n"
      << "const static PredicateBitset FeatureBitsets[] {\n"
      << "  {}, // GIFBS_Invalid\n";
   for (const auto &FeatureBitset : FeatureBitsets) {
     if (FeatureBitset.empty())
       continue;
     OS << "  {";
     for (const auto &Feature : FeatureBitset) {
       const auto &I = SubtargetFeatures.find(Feature);
       assert(I != SubtargetFeatures.end() && "Didn't import predicate?");
       OS << I->second.getEnumBitName() << ", ";
     }
     OS << "},\n";
   }
   OS << "};\n\n";
 
   // Emit complex predicate table and an enum to reference them with.
   OS << "// ComplexPattern predicates.\n"
      << "enum {\n"
      << "  GICP_Invalid,\n";
   for (const auto &Record : ComplexPredicates)
     OS << "  GICP_" << Record->getName() << ",\n";
   OS << "};\n"
      << "// See constructor for table contents\n\n";
 
   emitImmPredicates(OS, "I64", "int64_t", [](const Record *R) {
     bool Unset;
     return !R->getValueAsBitOrUnset("IsAPFloat", Unset) &&
            !R->getValueAsBit("IsAPInt");
   });
   emitImmPredicates(OS, "APFloat", "const APFloat &", [](const Record *R) {
     bool Unset;
     return R->getValueAsBitOrUnset("IsAPFloat", Unset);
   });
   emitImmPredicates(OS, "APInt", "const APInt &", [](const Record *R) {
     return R->getValueAsBit("IsAPInt");
   });
   OS << "\n";
 
   OS << Target.getName() << "InstructionSelector::ComplexMatcherMemFn\n"
      << Target.getName() << "InstructionSelector::ComplexPredicateFns[] = {\n"
      << "  nullptr, // GICP_Invalid\n";
   for (const auto &Record : ComplexPredicates)
     OS << "  &" << Target.getName()
        << "InstructionSelector::" << Record->getValueAsString("MatcherFn")
        << ", // " << Record->getName() << "\n";
   OS << "};\n\n";
 
   OS << "// Custom renderers.\n"
      << "enum {\n"
      << "  GICR_Invalid,\n";
   for (const auto &Record : CustomRendererFns)
     OS << "  GICR_" << Record->getValueAsString("RendererFn") << ", \n";
   OS << "};\n";
 
   OS << Target.getName() << "InstructionSelector::CustomRendererFn\n"
      << Target.getName() << "InstructionSelector::CustomRenderers[] = {\n"
      << "  nullptr, // GICP_Invalid\n";
   for (const auto &Record : CustomRendererFns)
     OS << "  &" << Target.getName()
        << "InstructionSelector::" << Record->getValueAsString("RendererFn")
        << ", // " << Record->getName() << "\n";
   OS << "};\n\n";
 
   std::stable_sort(Rules.begin(), Rules.end(), [&](const RuleMatcher &A,
                                                    const RuleMatcher &B) {
     int ScoreA = RuleMatcherScores[A.getRuleID()];
     int ScoreB = RuleMatcherScores[B.getRuleID()];
     if (ScoreA > ScoreB)
       return true;
     if (ScoreB > ScoreA)
       return false;
     if (A.isHigherPriorityThan(B)) {
       assert(!B.isHigherPriorityThan(A) && "Cannot be more important "
                                            "and less important at "
                                            "the same time");
       return true;
     }
     return false;
   });
 
   OS << "bool " << Target.getName()
      << "InstructionSelector::selectImpl(MachineInstr &I, CodeGenCoverage "
         "&CoverageInfo) const {\n"
      << "  MachineFunction &MF = *I.getParent()->getParent();\n"
      << "  MachineRegisterInfo &MRI = MF.getRegInfo();\n"
      << "  // FIXME: This should be computed on a per-function basis rather "
         "than per-insn.\n"
      << "  AvailableFunctionFeatures = computeAvailableFunctionFeatures(&STI, "
         "&MF);\n"
      << "  const PredicateBitset AvailableFeatures = getAvailableFeatures();\n"
      << "  NewMIVector OutMIs;\n"
      << "  State.MIs.clear();\n"
      << "  State.MIs.push_back(&I);\n\n"
      << "  if (executeMatchTable(*this, OutMIs, State, ISelInfo"
      << ", getMatchTable(), TII, MRI, TRI, RBI, AvailableFeatures"
      << ", CoverageInfo)) {\n"
      << "    return true;\n"
      << "  }\n\n"
      << "  return false;\n"
      << "}\n\n";
 
   const MatchTable Table =
       buildMatchTable(Rules, OptimizeMatchTable, GenerateCoverage);
   OS << "const int64_t *" << Target.getName()
      << "InstructionSelector::getMatchTable() const {\n";
   Table.emitDeclaration(OS);
   OS << "  return ";
   Table.emitUse(OS);
   OS << ";\n}\n";
   OS << "#endif // ifdef GET_GLOBALISEL_IMPL\n";
 
   OS << "#ifdef GET_GLOBALISEL_PREDICATES_DECL\n"
      << "PredicateBitset AvailableModuleFeatures;\n"
      << "mutable PredicateBitset AvailableFunctionFeatures;\n"
      << "PredicateBitset getAvailableFeatures() const {\n"
      << "  return AvailableModuleFeatures | AvailableFunctionFeatures;\n"
      << "}\n"
      << "PredicateBitset\n"
      << "computeAvailableModuleFeatures(const " << Target.getName()
      << "Subtarget *Subtarget) const;\n"
      << "PredicateBitset\n"
      << "computeAvailableFunctionFeatures(const " << Target.getName()
      << "Subtarget *Subtarget,\n"
      << "                                 const MachineFunction *MF) const;\n"
      << "#endif // ifdef GET_GLOBALISEL_PREDICATES_DECL\n";
 
   OS << "#ifdef GET_GLOBALISEL_PREDICATES_INIT\n"
      << "AvailableModuleFeatures(computeAvailableModuleFeatures(&STI)),\n"
      << "AvailableFunctionFeatures()\n"
      << "#endif // ifdef GET_GLOBALISEL_PREDICATES_INIT\n";
 }
 
 void GlobalISelEmitter::declareSubtargetFeature(Record *Predicate) {
   if (SubtargetFeatures.count(Predicate) == 0)
     SubtargetFeatures.emplace(
         Predicate, SubtargetFeatureInfo(Predicate, SubtargetFeatures.size()));
 }
 
 void RuleMatcher::optimize() {
   for (auto &Item : InsnVariableIDs) {
     InstructionMatcher &InsnMatcher = *Item.first;
     for (auto &OM : InsnMatcher.operands()) {
       // Register Banks checks rarely fail, but often crash as targets usually
       // provide only partially defined RegisterBankInfo::getRegBankFromRegClass
       // method. Often the problem is hidden as non-optimized MatchTable checks
       // banks rather late, most notably after checking target / function /
       // module features and a few opcodes. That makes these checks a)
       // beneficial to delay until the very end (we don't want to perform a lot
       // of checks that all pass and then fail at the very end) b) not safe to
       // have as early checks.
       for (auto &OP : OM->predicates())
         if (isa<RegisterBankOperandMatcher>(OP) ||
             isa<ComplexPatternOperandMatcher>(OP))
           EpilogueMatchers.emplace_back(std::move(OP));
       OM->eraseNullPredicates();
     }
     InsnMatcher.optimize();
   }
   llvm::sort(
       EpilogueMatchers.begin(), EpilogueMatchers.end(),
       [](const std::unique_ptr<PredicateMatcher> &L,
          const std::unique_ptr<PredicateMatcher> &R) {
         return std::make_tuple(L->getKind(), L->getInsnVarID(), L->getOpIdx()) <
                std::make_tuple(R->getKind(), R->getInsnVarID(), R->getOpIdx());
       });
 }
 
 bool RuleMatcher::hasFirstCondition() const {
   if (insnmatchers_empty())
     return false;
   InstructionMatcher &Matcher = insnmatchers_front();
   if (!Matcher.predicates_empty())
     return true;
   for (auto &OM : Matcher.operands())
     for (auto &OP : OM->predicates())
       if (!isa<InstructionOperandMatcher>(OP))
         return true;
   return false;
 }
 
 const PredicateMatcher &RuleMatcher::getFirstCondition() const {
   assert(!insnmatchers_empty() &&
          "Trying to get a condition from an empty RuleMatcher");
 
   InstructionMatcher &Matcher = insnmatchers_front();
   if (!Matcher.predicates_empty())
     return **Matcher.predicates_begin();
   // If there is no more predicate on the instruction itself, look at its
   // operands.
   for (auto &OM : Matcher.operands())
     for (auto &OP : OM->predicates())
       if (!isa<InstructionOperandMatcher>(OP))
         return *OP;
 
   llvm_unreachable("Trying to get a condition from an InstructionMatcher with "
                    "no conditions");
 }
 
 std::unique_ptr<PredicateMatcher> RuleMatcher::popFirstCondition() {
   assert(!insnmatchers_empty() &&
          "Trying to pop a condition from an empty RuleMatcher");
 
   InstructionMatcher &Matcher = insnmatchers_front();
   if (!Matcher.predicates_empty())
     return Matcher.predicates_pop_front();
   // If there is no more predicate on the instruction itself, look at its
   // operands.
   for (auto &OM : Matcher.operands())
     for (auto &OP : OM->predicates())
       if (!isa<InstructionOperandMatcher>(OP)) {
         std::unique_ptr<PredicateMatcher> Result = std::move(OP);
         OM->eraseNullPredicates();
         return Result;
       }
 
   llvm_unreachable("Trying to pop a condition from an InstructionMatcher with "
                    "no conditions");
 }
 
 bool GroupMatcher::candidateConditionMatches(
     const PredicateMatcher &Predicate) const {
 
   if (empty()) {
     // Sharing predicates for nested instructions is not supported yet as we
     // currently don't hoist the GIM_RecordInsn's properly, therefore we can
     // only work on the original root instruction (InsnVarID == 0):
     if (Predicate.getInsnVarID() != 0)
       return false;
     // ... otherwise an empty group can handle any predicate with no specific
     // requirements:
     return true;
   }
 
   const Matcher &Representative = **Matchers.begin();
   const auto &RepresentativeCondition = Representative.getFirstCondition();
   // ... if not empty, the group can only accomodate matchers with the exact
   // same first condition:
   return Predicate.isIdentical(RepresentativeCondition);
 }
 
 bool GroupMatcher::addMatcher(Matcher &Candidate) {
   if (!Candidate.hasFirstCondition())
     return false;
 
   const PredicateMatcher &Predicate = Candidate.getFirstCondition();
   if (!candidateConditionMatches(Predicate))
     return false;
 
   Matchers.push_back(&Candidate);
   return true;
 }
 
 void GroupMatcher::finalize() {
   assert(Conditions.empty() && "Already finalized?");
   if (empty())
     return;
 
   Matcher &FirstRule = **Matchers.begin();
 
   Conditions.push_back(FirstRule.popFirstCondition());
   for (unsigned I = 1, E = Matchers.size(); I < E; ++I)
     Matchers[I]->popFirstCondition();
 }
 
 void GroupMatcher::emit(MatchTable &Table) {
   unsigned LabelID = ~0U;
   if (!Conditions.empty()) {
     LabelID = Table.allocateLabelID();
     Table << MatchTable::Opcode("GIM_Try", +1)
           << MatchTable::Comment("On fail goto")
           << MatchTable::JumpTarget(LabelID) << MatchTable::LineBreak;
   }
   for (auto &Condition : Conditions)
     Condition->emitPredicateOpcodes(
         Table, *static_cast<RuleMatcher *>(*Matchers.begin()));
 
   for (const auto &M : Matchers)
     M->emit(Table);
 
   // Exit the group
   if (!Conditions.empty())
     Table << MatchTable::Opcode("GIM_Reject", -1) << MatchTable::LineBreak
           << MatchTable::Label(LabelID);
 }
 
 unsigned OperandMatcher::getInsnVarID() const { return Insn.getInsnVarID(); }
 
 } // end anonymous namespace
 
 //===----------------------------------------------------------------------===//
 
 namespace llvm {
 void EmitGlobalISel(RecordKeeper &RK, raw_ostream &OS) {
   GlobalISelEmitter(RK).run(OS);
 }
 } // End llvm namespace

llvm-svn: 332999
 2018-05-22 16:51:54 +00:00
Dan Liew
d7e3a3632a [lit] Try to make shtest-timeout.py test more reliable by using a 
larger timeout value. This really isn't very good because it will
still be susceptible to machine performance.

While we are here also fix a bug in validation of
`maxIndividualTestTime` where previously it wasn't checked if the
type was an int.

rdar://problem/40221572

llvm-svn: 332987
 2018-05-22 15:06:29 +00:00
Dan Liew
bae1119a38 [lit] Don't run slow.py in shtest-timeout.py test. 
The program used to be used in `quick_then_slow.py` but that was
removed in r328702. The tests always run `slow.py` on its own but
this doesn't really test additional code so we'll just drop running
`slow.py` so the tests run faster.

rdar://problem/40221572

llvm-svn: 332986
 2018-05-22 15:06:24 +00:00
Dan Liew
b2006c8dcb [lit] Don't check output of commands used in shtest-timeout.py test. 
If the system is under heavy load 1 second might not be long enough
for it to produce output which could lead to spurious test failures.
What matters is that the right test cases reach a timeout.

rdar://problem/40221572

llvm-svn: 332985
 2018-05-22 15:06:20 +00:00
Simon Dardis
b7fb5f7be7 [FastISel] Permit instructions to be skipped for FastISel generation. 
Some ISA's such as microMIPS32(R6) have instructions which are near identical
for code generation purposes, e.g. xor and xor16. These instructions take the
same value types for operands and return values, have the same
instruction predicates and map to the same ISD opcode. (These instructions do
differ by register classes.)

In such cases, the FastISel generator rejects the instruction definition.

This patch borrows the 'FastIselShouldIgnore' bit from rL129692 and enables
applying it to an instruction definition.

Reviewers: mcrosier

Differential Revision: https://reviews.llvm.org/D46953

llvm-svn: 332983
 2018-05-22 14:36:58 +00:00
Roman Tereshin
fa30317850 [GlobalISel][InstructionSelect] Removing redundant num operands and nested def operands checks, perf patch 2 
This patch continues a series of patches that decrease time spent by
GlobalISel in its InstructionSelect pass by roughly 60% for -O0 builds
for large inputs as measured on sqlite3-amalgamation
(http://sqlite.org/download.html) targeting AArch64.

This commit specifically removes number of operands checks that are
redundant if the instruction's opcode already guarantees that number
of operands (or more), and also avoids any kind of checks on a def
operand of a nested instruction as everything about it was already
checked at its use.

The expected performance implication is about 3% off InstructionSelect
comparing to the baseline (before the series of patches)

This patch also contains a bit of NFC changes required for further
patches in the series.

Every commit planned shares the same Phabricator Review.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 332945
 2018-05-22 04:31:50 +00:00
Roman Tereshin
dfebbe6a7d Reapply r332907 "[GlobalISel] Improving InstructionSelect's performance by reducing MatchTable..." 
Apparently the compile time problem was caused by the fact that not
all compilers / STL implementations can automatically convert
std::unique_ptr<Derived> to std::unique_ptr<Base>. Fixed (hopefully)
by making sure it's std::unique_ptr<Derived>&& (rvalue ref) to
std::unique_ptr<Base> conversion instead.

llvm-svn: 332917
 2018-05-21 23:28:51 +00:00