RPCS3/llvm-mirror
1
0
Fork 0
mirror of https://github.com/RPCS3/llvm-mirror.git synced 2024-11-25 12:12:47 +01:00
Code Issues Projects Releases Wiki Activity
c32dde6058
llvm-mirror/lib/IR/AutoUpgrade.cpp

4200 lines
184 KiB
C++
Raw Normal View History

This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
//===-- AutoUpgrade.cpp - Implement auto-upgrade helper functions ---------===//
//
Update the file headers across all of the LLVM projects in the monorepo to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
2019-01-19 09:50:56 +01:00
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
//
//===----------------------------------------------------------------------===//
//
[X86, AVX] replace vinsertf128 intrinsics with generic shuffles We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is the sibling patch for the Clang half of this change: http://reviews.llvm.org/D8088 Differential Revision: http://reviews.llvm.org/D8086 llvm-svn: 231794
2015-03-10 17:08:36 +01:00
// This file implements the auto-upgrade helper functions.
// This is where deprecated IR intrinsics and other IR features are updated to
// current specifications.
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
//
//===----------------------------------------------------------------------===//
[Layering] Move AutoUpgrade.h into the IR library where its implementation already lives. llvm-svn: 202961
2014-03-05 11:34:14 +01:00
#include "llvm/IR/AutoUpgrade.h"
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 02:00:32 +01:00
#include "llvm/ADT/StringSwitch.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 12:36:10 +01:00
#include "llvm/IR/Constants.h"
[cleanup] Re-sort all the #include lines in LLVM using utils/sort_includes.py. I clearly haven't done this in a while, so more changed than usual. This even uncovered a missing include from the InstrProf library that I've added. No functionality changed here, just mechanical cleanup of the include order. llvm-svn: 225974
2015-01-14 12:23:27 +01:00
#include "llvm/IR/DIBuilder.h"
[Layering] Move DebugInfo.h into the IR library where its implementation already lives. llvm-svn: 203046
2014-03-06 01:46:21 +01:00
#include "llvm/IR/DebugInfo.h"
Report a warning when dropping outdated debug info metadata. Use DiagnosticInfo to emit the warning. llvm-svn: 199346
2014-01-16 02:51:12 +01:00
#include "llvm/IR/DiagnosticInfo.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 12:36:10 +01:00
#include "llvm/IR/Function.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/Instruction.h"
Remove alignment argument from memcpy/memmove/memset in favour of alignment attributes (Step 1) Summary: This is a resurrection of work first proposed and discussed in Aug 2015: http://lists.llvm.org/pipermail/llvm-dev/2015-August/089384.html and initially landed (but then backed out) in Nov 2015: http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20151109/312083.html The @llvm.memcpy/memmove/memset intrinsics currently have an explicit argument which is required to be a constant integer. It represents the alignment of the dest (and source), and so must be the minimum of the actual alignment of the two. This change is the first in a series that allows source and dest to each have their own alignments by using the alignment attribute on their arguments. In this change we: 1) Remove the alignment argument. 2) Add alignment attributes to the source & dest arguments. We, temporarily, require that the alignments for source & dest be equal. For example, code which used to read: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 100, i32 4, i1 false) will now read call void @llvm.memcpy.p0i8.p0i8.i32(i8* align 4 %dest, i8* align 4 %src, i32 100, i1 false) Downstream users may have to update their lit tests that check for @llvm.memcpy/memmove/memset call/declaration patterns. The following extended sed script may help with updating the majority of your tests, but it does not catch all possible patterns so some manual checking and updating will be required. s~declare void @llvm\.mem(set|cpy|move)\.p([^(]*)\((.*), i32, i1\)~declare void @llvm.mem\1.p\2(\3, i1)~g s~call void @llvm\.memset\.p([^(]*)i8\(i8([^*]*)\* (.*), i8 (.*), i8 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i8(i8\2* \3, i8 \4, i8 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i16\(i8([^*]*)\* (.*), i8 (.*), i16 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i16(i8\2* \3, i8 \4, i16 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i32\(i8([^*]*)\* (.*), i8 (.*), i32 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i32(i8\2* \3, i8 \4, i32 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i64\(i8([^*]*)\* (.*), i8 (.*), i64 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i64(i8\2* \3, i8 \4, i64 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i128\(i8([^*]*)\* (.*), i8 (.*), i128 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i128(i8\2* \3, i8 \4, i128 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i8\(i8([^*]*)\* (.*), i8 (.*), i8 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i8(i8\2* align \6 \3, i8 \4, i8 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i16\(i8([^*]*)\* (.*), i8 (.*), i16 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i16(i8\2* align \6 \3, i8 \4, i16 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i32\(i8([^*]*)\* (.*), i8 (.*), i32 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i32(i8\2* align \6 \3, i8 \4, i32 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i64\(i8([^*]*)\* (.*), i8 (.*), i64 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i64(i8\2* align \6 \3, i8 \4, i64 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i128\(i8([^*]*)\* (.*), i8 (.*), i128 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i128(i8\2* align \6 \3, i8 \4, i128 \5, i1 \7)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i8\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i8 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i8(i8\3* \4, i8\5* \6, i8 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i16\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i16 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i16(i8\3* \4, i8\5* \6, i16 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i32\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i32 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i32(i8\3* \4, i8\5* \6, i32 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i64\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i64 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i64(i8\3* \4, i8\5* \6, i64 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i128\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i128 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i128(i8\3* \4, i8\5* \6, i128 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i8\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i8 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i8(i8\3* align \8 \4, i8\5* align \8 \6, i8 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i16\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i16 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i16(i8\3* align \8 \4, i8\5* align \8 \6, i16 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i32\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i32 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i32(i8\3* align \8 \4, i8\5* align \8 \6, i32 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i64\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i64 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i64(i8\3* align \8 \4, i8\5* align \8 \6, i64 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i128\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i128 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i128(i8\3* align \8 \4, i8\5* align \8 \6, i128 \7, i1 \9)~g The remaining changes in the series will: Step 2) Expand the IRBuilder API to allow creation of memcpy/memmove with differing source and dest alignments. Step 3) Update Clang to use the new IRBuilder API. Step 4) Update Polly to use the new IRBuilder API. Step 5) Update LLVM passes that create memcpy/memmove calls to use the new IRBuilder API, and those that use use MemIntrinsicInst::[get|set]Alignment() to use getDestAlignment() and getSourceAlignment() instead. Step 6) Remove the single-alignment IRBuilder API for memcpy/memmove, and the MemIntrinsicInst::[get|set]Alignment() methods. Reviewers: pete, hfinkel, lhames, reames, bollu Reviewed By: reames Subscribers: niosHD, reames, jholewinski, qcolombet, jfb, sanjoy, arsenm, dschuff, dylanmckay, mehdi_amini, sdardis, nemanjai, david2050, nhaehnle, javed.absar, sbc100, jgravelle-google, eraman, aheejin, kbarton, JDevlieghere, asb, rbar, johnrusso, simoncook, jordy.potman.lists, apazos, sabuasal, llvm-commits Differential Revision: https://reviews.llvm.org/D41675 llvm-svn: 322965
2018-01-19 18:13:12 +01:00
#include "llvm/IR/IntrinsicInst.h"
[IR] Split out target specific intrinsic enums into separate headers This has two main effects: - Optimizes debug info size by saving 221.86 MB of obj file size in a Windows optimized+debug build of 'all'. This is 3.03% of 7,332.7MB of object file size. - Incremental step towards decoupling target intrinsics. The enums are still compact, so adding and removing a single target-specific intrinsic will trigger a rebuild of all of LLVM. Assigning distinct target id spaces is potential future work. Part of PR34259 Reviewers: efriedma, echristo, MaskRay Reviewed By: echristo, MaskRay Differential Revision: https://reviews.llvm.org/D71320
2019-12-11 16:55:26 +01:00
#include "llvm/IR/IntrinsicsAArch64.h"
#include "llvm/IR/IntrinsicsARM.h"
#include "llvm/IR/IntrinsicsX86.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 12:36:10 +01:00
#include "llvm/IR/LLVMContext.h"
#include "llvm/IR/Module.h"
Move the stripping of invalid debug info from the Verifier to AutoUpgrade. This came out of a recent discussion on llvm-dev (https://reviews.llvm.org/D38042). Currently the Verifier will strip the debug info metadata from a module if it finds the dbeug info to be malformed. This feature is very valuable since it allows us to improve the Verifier by making it stricter without breaking bcompatibility, but arguable the Verifier pass should not be modifying the IR. This patch moves the stripping of broken debug info into AutoUpgrade (UpgradeDebugInfo to be precise), which is a much better location for this since the stripping of malformed (i.e., produced by older, buggy versions of Clang) is a (harsh) form of AutoUpgrade. This change is mostly NFC in nature, the one big difference is the behavior when LLVM module passes are introducing malformed debug info. Prior to this patch, a NoAsserts build would have printed a warning and stripped the debug info, after this patch the Verifier will report a fatal error. I believe this behavior is actually more desirable anyway. Differential Revision: https://reviews.llvm.org/D38184 llvm-svn: 314699
2017-10-02 20:31:29 +02:00
#include "llvm/IR/Verifier.h"
assert(0) -> LLVM_UNREACHABLE. Make llvm_unreachable take an optional string, thus moving the cerr<< out of line. LLVM_UNREACHABLE is now a simple wrapper that makes the message go away for NDEBUG builds. llvm-svn: 75379
2009-07-11 22:10:48 +02:00
#include "llvm/Support/ErrorHandling.h"
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 12:56:37 +02:00
#include "llvm/Support/Regex.h"
Unbreak build with gcc 4.3: provide missed includes and silence most annoying warnings. llvm-svn: 47367
2008-02-20 12:08:44 +01:00
#include <cstring>
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
using namespace llvm;
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 17:51:42 +02:00
static void rename(GlobalValue *GV) { GV->setName(GV->getName() + ".old"); }
[IR][X86] Rename an AutoUpgrade helper function to more accurately match what intrinsics it handles. NFC llvm-svn: 295425
2017-02-17 08:07:21 +01:00
// Upgrade the declarations of the SSE4.1 ptest intrinsics whose arguments have
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-10 20:42:51 +02:00
// changed their type from v4f32 to v2i64.
[IR][X86] Rename an AutoUpgrade helper function to more accurately match what intrinsics it handles. NFC llvm-svn: 295425
2017-02-17 08:07:21 +01:00
static bool UpgradePTESTIntrinsic(Function* F, Intrinsic::ID IID,
Function *&NewFn) {
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-10 20:42:51 +02:00
// Check whether this is an old version of the function, which received
// v4f32 arguments.
Type *Arg0Type = F->getFunctionType()->getParamType(0);
if (Arg0Type != VectorType::get(Type::getFloatTy(F->getContext()), 4))
return false;
// Yes, it's old, replace it with new version.
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 17:51:42 +02:00
rename(F);
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-10 20:42:51 +02:00
NewFn = Intrinsic::getDeclaration(F->getParent(), IID);
return true;
}
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
[x86] Fix a pretty horrible bug and inconsistency in the x86 asm parsing (and latent bug in the instruction definitions). This is effectively a revert of r136287 which tried to address a specific and narrow case of immediate operands failing to be accepted by x86 instructions with a pretty heavy hammer: it introduced a new kind of operand that behaved differently. All of that is removed with this commit, but the test cases are both preserved and enhanced. The core problem that r136287 and this commit are trying to handle is that gas accepts both of the following instructions: insertps $192, %xmm0, %xmm1 insertps $-64, %xmm0, %xmm1 These will encode to the same byte sequence, with the immediate occupying an 8-bit entry. The first form was fixed by r136287 but that broke the prior handling of the second form! =[ Ironically, we would still emit the second form in some cases and then be unable to re-assemble the output. The reason why the first instruction failed to be handled is because prior to r136287 the operands ere marked 'i32i8imm' which forces them to be sign-extenable. Clearly, that won't work for 192 in a single byte. However, making thim zero-extended or "unsigned" doesn't really address the core issue either because it breaks negative immediates. The correct fix is to make these operands 'i8imm' reflecting that they can be either signed or unsigned but must be 8-bit immediates. This patch backs out r136287 and then changes those places as well as some others to use 'i8imm' rather than one of the extended variants. Naturally, this broke something else. The custom DAG nodes had to be updated to have a much more accurate type constraint of an i8 node, and a bunch of Pat immediates needed to be specified as i8 values. The fallout didn't end there though. We also then ceased to be able to match the instruction-specific intrinsics to the instructions so modified. Digging, this is because they too used i32 rather than i8 in their signature. So I've also switched those intrinsics to i8 arguments in line with the instructions. In order to make the intrinsic adjustments of course, I also had to add auto upgrading for the intrinsics. I suspect that the intrinsic argument types may have led everything down this rabbit hole. Pretty happy with the result. llvm-svn: 217310
2014-09-06 12:00:01 +02:00
// Upgrade the declarations of intrinsic functions whose 8-bit immediate mask
// arguments have changed their type from i32 to i8.
static bool UpgradeX86IntrinsicsWith8BitMask(Function *F, Intrinsic::ID IID,
Function *&NewFn) {
// Check that the last argument is an i32.
Type *LastArgType = F->getFunctionType()->getParamType(
F->getFunctionType()->getNumParams() - 1);
if (!LastArgType->isIntegerTy(32))
return false;
// Move this function aside and map down.
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 17:51:42 +02:00
rename(F);
[x86] Fix a pretty horrible bug and inconsistency in the x86 asm parsing (and latent bug in the instruction definitions). This is effectively a revert of r136287 which tried to address a specific and narrow case of immediate operands failing to be accepted by x86 instructions with a pretty heavy hammer: it introduced a new kind of operand that behaved differently. All of that is removed with this commit, but the test cases are both preserved and enhanced. The core problem that r136287 and this commit are trying to handle is that gas accepts both of the following instructions: insertps $192, %xmm0, %xmm1 insertps $-64, %xmm0, %xmm1 These will encode to the same byte sequence, with the immediate occupying an 8-bit entry. The first form was fixed by r136287 but that broke the prior handling of the second form! =[ Ironically, we would still emit the second form in some cases and then be unable to re-assemble the output. The reason why the first instruction failed to be handled is because prior to r136287 the operands ere marked 'i32i8imm' which forces them to be sign-extenable. Clearly, that won't work for 192 in a single byte. However, making thim zero-extended or "unsigned" doesn't really address the core issue either because it breaks negative immediates. The correct fix is to make these operands 'i8imm' reflecting that they can be either signed or unsigned but must be 8-bit immediates. This patch backs out r136287 and then changes those places as well as some others to use 'i8imm' rather than one of the extended variants. Naturally, this broke something else. The custom DAG nodes had to be updated to have a much more accurate type constraint of an i8 node, and a bunch of Pat immediates needed to be specified as i8 values. The fallout didn't end there though. We also then ceased to be able to match the instruction-specific intrinsics to the instructions so modified. Digging, this is because they too used i32 rather than i8 in their signature. So I've also switched those intrinsics to i8 arguments in line with the instructions. In order to make the intrinsic adjustments of course, I also had to add auto upgrading for the intrinsics. I suspect that the intrinsic argument types may have led everything down this rabbit hole. Pretty happy with the result. llvm-svn: 217310
2014-09-06 12:00:01 +02:00
NewFn = Intrinsic::getDeclaration(F->getParent(), IID);
return true;
}
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
static bool ShouldUpgradeX86Intrinsic(Function *F, StringRef Name) {
// All of the intrinsics matches below should be marked with which llvm
// version started autoupgrading them. At some point in the future we would
// like to use this information to remove upgrade code for some older
// intrinsics. It is currently undecided how we will determine that future
// point.
[X86] Merge addcarryx/addcarry intrinsic into a single addcarry intrinsic. Both intrinsics do the exact same thing so we really only need one. Earlier in the 8.0 cycle we changed the signature of this intrinsic without renaming it. But it looks difficult to get the autoupgrade code to allow me to merge the intrinsics and change the signature at the same time. So I've renamed the intrinsic slightly for the new merged intrinsic. I'm skipping autoupgrading from the previous new to 8.0 signature. I've also renamed the subborrow for consistency. llvm-svn: 348737
2018-12-10 07:07:50 +01:00
if (Name == "addcarryx.u32" || // Added in 8.0
Name == "addcarryx.u64" || // Added in 8.0
Name == "addcarry.u32" || // Added in 8.0
Name == "addcarry.u64" || // Added in 8.0
Name == "subborrow.u32" || // Added in 8.0
Name == "subborrow.u64" || // Added in 8.0
[X86][SSE] Auto upgrade PADDS/PSUBS intrinsics to SADD_SAT/SSUB_SAT generic intrinsics (llvm) This auto upgrades the signed SSE saturated math intrinsics to SADD_SAT/SSUB_SAT generic intrinsics. Clang counterpart: https://reviews.llvm.org/D55890 Differential Revision: https://reviews.llvm.org/D55894 llvm-svn: 349892
2018-12-21 10:04:14 +01:00
Name.startswith("sse2.padds.") || // Added in 8.0
Name.startswith("sse2.psubs.") || // Added in 8.0
[X86] Merge addcarryx/addcarry intrinsic into a single addcarry intrinsic. Both intrinsics do the exact same thing so we really only need one. Earlier in the 8.0 cycle we changed the signature of this intrinsic without renaming it. But it looks difficult to get the autoupgrade code to allow me to merge the intrinsics and change the signature at the same time. So I've renamed the intrinsic slightly for the new merged intrinsic. I'm skipping autoupgrading from the previous new to 8.0 signature. I've also renamed the subborrow for consistency. llvm-svn: 348737
2018-12-10 07:07:50 +01:00
Name.startswith("sse2.paddus.") || // Added in 8.0
[X86] Remove the unused masked 128 and 256-bit masked padds/psubs intrinsics. Still need to remove masking from the 512-bit versions. llvm-svn: 339841
2018-08-16 08:20:22 +02:00
Name.startswith("sse2.psubus.") || // Added in 8.0
[X86][SSE] Auto upgrade PADDS/PSUBS intrinsics to SADD_SAT/SSUB_SAT generic intrinsics (llvm) This auto upgrades the signed SSE saturated math intrinsics to SADD_SAT/SSUB_SAT generic intrinsics. Clang counterpart: https://reviews.llvm.org/D55890 Differential Revision: https://reviews.llvm.org/D55894 llvm-svn: 349892
2018-12-21 10:04:14 +01:00
Name.startswith("avx2.padds.") || // Added in 8.0
Name.startswith("avx2.psubs.") || // Added in 8.0
[X86] Remove the unused masked 128 and 256-bit masked padds/psubs intrinsics. Still need to remove masking from the 512-bit versions. llvm-svn: 339841
2018-08-16 08:20:22 +02:00
Name.startswith("avx2.paddus.") || // Added in 8.0
Name.startswith("avx2.psubus.") || // Added in 8.0
[X86][SSE] Auto upgrade PADDS/PSUBS intrinsics to SADD_SAT/SSUB_SAT generic intrinsics (llvm) This auto upgrades the signed SSE saturated math intrinsics to SADD_SAT/SSUB_SAT generic intrinsics. Clang counterpart: https://reviews.llvm.org/D55890 Differential Revision: https://reviews.llvm.org/D55894 llvm-svn: 349892
2018-12-21 10:04:14 +01:00
Name.startswith("avx512.padds.") || // Added in 8.0
Name.startswith("avx512.psubs.") || // Added in 8.0
Name.startswith("avx512.mask.padds.") || // Added in 8.0
Name.startswith("avx512.mask.psubs.") || // Added in 8.0
[X86] Remove the unused masked 128 and 256-bit masked padds/psubs intrinsics. Still need to remove masking from the 512-bit versions. llvm-svn: 339841
2018-08-16 08:20:22 +02:00
Name.startswith("avx512.mask.paddus.") || // Added in 8.0
Name.startswith("avx512.mask.psubus.") || // Added in 8.0
[X86] Lowering addus/subus intrinsics to native IR Summary: This revision improves previous version (rL330322) which has been reverted due to crashes. This is the patch that lowers x86 intrinsics to native IR in order to enable optimizations. The patch also includes folding of previously missing saturation patterns so that IR emits the same machine instructions as the intrinsics. Reviewers: craig.topper, spatel, RKSimon Reviewed By: craig.topper Subscribers: mike.dvoretsky, DavidKreitzer, sroland, llvm-commits Differential Revision: https://reviews.llvm.org/D46179 llvm-svn: 339650
2018-08-14 10:00:56 +02:00
Name=="ssse3.pabs.b.128" || // Added in 6.0
[X86] [PATCH] [intrinsics] Lowering X86 ABS intrinsics to IR. (llvm) This patch, together with a matching clang patch (https://reviews.llvm.org/D37694), implements the lowering of X86 ABS intrinsics to IR. differential revision: https://reviews.llvm.org/D37693. llvm-svn: 313134
2017-09-13 11:02:36 +02:00
Name=="ssse3.pabs.w.128" || // Added in 6.0
Name=="ssse3.pabs.d.128" || // Added in 6.0
[X86] Remove FMA4 scalar intrinsics. Use llvm.fma intrinsic instead. The intrinsics can be implemented with a f32/f64 llvm.fma intrinsic and an insert into a zero vector. There are a couple regressions here due to SelectionDAG not being able to pull an fneg through an extract_vector_elt. I'm not super worried about this though as InstCombine should be able to do it before we get to SelectionDAG. llvm-svn: 336416
2018-07-06 09:14:41 +02:00
Name.startswith("fma4.vfmadd.s") || // Added in 7.0
[X86] Remove X86 specific scalar FMA intrinsics and upgrade to tart independent FMA and extractelement/insertelement. llvm-svn: 336315
2018-07-05 08:52:55 +02:00
Name.startswith("fma.vfmadd.") || // Added in 7.0
[X86] Remove and autoupgrade a bunch of FMA instrinsics that are no longer used by clang. llvm-svn: 332146
2018-05-11 23:59:34 +02:00
Name.startswith("fma.vfmsub.") || // Added in 7.0
[X86] Remove the last of the 'x86.fma.' intrinsics and autoupgrade them to 'llvm.fma'. Add upgrade tests for all. Still need to remove the AVX512 masked versions. llvm-svn: 336383
2018-07-05 20:43:58 +02:00
Name.startswith("fma.vfmaddsub.") || // Added in 7.0
[X86] Remove and autoupgrade a bunch of FMA instrinsics that are no longer used by clang. llvm-svn: 332146
2018-05-11 23:59:34 +02:00
Name.startswith("fma.vfmsubadd.") || // Added in 7.0
Name.startswith("fma.vfnmadd.") || // Added in 7.0
Name.startswith("fma.vfnmsub.") || // Added in 7.0
[X86] Remove and autoupgrade the scalar fma intrinsics with masking. This converts them to what clang is now using for codegen. Unfortunately, there seem to be a few kinks to work out still. I'll try to address with follow up patches. llvm-svn: 336871
2018-07-12 02:29:56 +02:00
Name.startswith("avx512.mask.vfmadd.") || // Added in 7.0
Name.startswith("avx512.mask.vfnmadd.") || // Added in 7.0
Name.startswith("avx512.mask.vfnmsub.") || // Added in 7.0
Name.startswith("avx512.mask3.vfmadd.") || // Added in 7.0
Name.startswith("avx512.maskz.vfmadd.") || // Added in 7.0
Name.startswith("avx512.mask3.vfmsub.") || // Added in 7.0
Name.startswith("avx512.mask3.vfnmsub.") || // Added in 7.0
Name.startswith("avx512.mask.vfmaddsub.") || // Added in 7.0
Name.startswith("avx512.maskz.vfmaddsub.") || // Added in 7.0
Name.startswith("avx512.mask3.vfmaddsub.") || // Added in 7.0
Name.startswith("avx512.mask3.vfmsubadd.") || // Added in 7.0
[x86][AVX512] Lowering shuffle i/f intrinsics to LLVM IR This patch, together with a matching clang patch (https://reviews.llvm.org/D38672), implements the lowering of X86 shuffle i/f intrinsics to IR. Differential Revision: https://reviews.llvm.org/D38671 Change-Id: I1e7d359a74743e995ec356237a85214ce55d3661 llvm-svn: 318026
2017-11-13 10:16:39 +01:00
Name.startswith("avx512.mask.shuf.i") || // Added in 6.0
Name.startswith("avx512.mask.shuf.f") || // Added in 6.0
Remove trailing space sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h} llvm-svn: 338293
2018-07-30 21:41:25 +02:00
Name.startswith("avx512.kunpck") || //added in 6.0
[X86] [PATCH] [intrinsics] Lowering X86 ABS intrinsics to IR. (llvm) This patch, together with a matching clang patch (https://reviews.llvm.org/D37694), implements the lowering of X86 ABS intrinsics to IR. differential revision: https://reviews.llvm.org/D37693. llvm-svn: 313134
2017-09-13 11:02:36 +02:00
Name.startswith("avx2.pabs.") || // Added in 6.0
Name.startswith("avx512.mask.pabs.") || // Added in 6.0
[x86][AVX512] Lowering Broadcastm intrinsics to LLVM IR This patch, together with a matching clang patch (https://reviews.llvm.org/D38683), implements the lowering of X86 broadcastm intrinsics to IR. Differential Revision: https://reviews.llvm.org/D38684 Change-Id: I709ac0b34641095397e994c8ff7e15d1315b3540 llvm-svn: 317458
2017-11-06 08:09:24 +01:00
Name.startswith("avx512.broadcastm") || // Added in 6.0
[X86] Lowering sqrt intrinsics to native IR Summary: Complementary patch to lowering sqrt intrinsics in Clang. Reviewers: craig.topper, spatel, RKSimon, DavidKreitzer, uriel.k Reviewed By: craig.topper Subscribers: tkrupa, mike.dvoretsky, llvm-commits Differential Revision: https://reviews.llvm.org/D41599 llvm-svn: 334849
2018-06-15 20:05:24 +02:00
Name == "sse.sqrt.ss" || // Added in 7.0
Name == "sse2.sqrt.sd" || // Added in 7.0
[X86] Remove masking from the avx512 packed sqrt intrinsics. Use select in IR instead. While there improve the coverage of the intrinsic testing and add fast-isel tests. llvm-svn: 335944
2018-06-29 07:43:26 +02:00
Name.startswith("avx512.mask.sqrt.p") || // Added in 7.0
[X86] Lowering sqrt intrinsics to native IR Summary: Complementary patch to lowering sqrt intrinsics in Clang. Reviewers: craig.topper, spatel, RKSimon, DavidKreitzer, uriel.k Reviewed By: craig.topper Subscribers: tkrupa, mike.dvoretsky, llvm-commits Differential Revision: https://reviews.llvm.org/D41599 llvm-svn: 334849
2018-06-15 20:05:24 +02:00
Name.startswith("avx.sqrt.p") || // Added in 7.0
Name.startswith("sse2.sqrt.p") || // Added in 7.0
Name.startswith("sse.sqrt.p") || // Added in 7.0
[x86] Lowering Mask Set1 intrinsics to LLVM IR This patch, together with a matching clang patch (https://reviews.llvm.org/D37668), implements the lowering of X86 mask set1 intrinsics to IR. Differential Revision: https://reviews.llvm.org/D37669 llvm-svn: 313625
2017-09-19 13:03:06 +02:00
Name.startswith("avx512.mask.pbroadcast") || // Added in 6.0
[X86] [PATCH] [intrinsics] Lowering X86 ABS intrinsics to IR. (llvm) This patch, together with a matching clang patch (https://reviews.llvm.org/D37694), implements the lowering of X86 ABS intrinsics to IR. differential revision: https://reviews.llvm.org/D37693. llvm-svn: 313134
2017-09-13 11:02:36 +02:00
Name.startswith("sse2.pcmpeq.") || // Added in 3.1
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name.startswith("sse2.pcmpgt.") || // Added in 3.1
Name.startswith("avx2.pcmpeq.") || // Added in 3.1
Name.startswith("avx2.pcmpgt.") || // Added in 3.1
Name.startswith("avx512.mask.pcmpeq.") || // Added in 3.9
Name.startswith("avx512.mask.pcmpgt.") || // Added in 3.9
[X86] Remove VPERM2F128/VPERM2I128 intrinsics and autoupgrade to native shuffles. I've moved the test cases from the InstCombine optimizations to the backend to keep the coverage we had there. It covered every possible immediate so I've preserved the resulting shuffle mask for each of those immediates. llvm-svn: 313450
2017-09-16 09:36:14 +02:00
Name.startswith("avx.vperm2f128.") || // Added in 6.0
Name == "avx2.vperm2i128" || // Added in 6.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name == "sse.add.ss" || // Added in 4.0
Name == "sse2.add.sd" || // Added in 4.0
Name == "sse.sub.ss" || // Added in 4.0
Name == "sse2.sub.sd" || // Added in 4.0
Name == "sse.mul.ss" || // Added in 4.0
Name == "sse2.mul.sd" || // Added in 4.0
Name == "sse.div.ss" || // Added in 4.0
Name == "sse2.div.sd" || // Added in 4.0
Name == "sse41.pmaxsb" || // Added in 3.9
Name == "sse2.pmaxs.w" || // Added in 3.9
Name == "sse41.pmaxsd" || // Added in 3.9
Name == "sse2.pmaxu.b" || // Added in 3.9
Name == "sse41.pmaxuw" || // Added in 3.9
Name == "sse41.pmaxud" || // Added in 3.9
Name == "sse41.pminsb" || // Added in 3.9
Name == "sse2.pmins.w" || // Added in 3.9
Name == "sse41.pminsd" || // Added in 3.9
Name == "sse2.pminu.b" || // Added in 3.9
Name == "sse41.pminuw" || // Added in 3.9
Name == "sse41.pminud" || // Added in 3.9
[X86] Remove and autoupgrade kand/kandn/kor/kxor/kxnor/knot intrinsics. Clang already stopped using these a couple months ago. The test cases aren't great as there is nothing forcing the operations to stay in k-registers so some of them moved back to scalar ops due to the bitcasts being moved around. llvm-svn: 324177
2018-02-03 21:18:25 +01:00
Name == "avx512.kand.w" || // Added in 7.0
Name == "avx512.kandn.w" || // Added in 7.0
Name == "avx512.knot.w" || // Added in 7.0
Name == "avx512.kor.w" || // Added in 7.0
Name == "avx512.kxor.w" || // Added in 7.0
Name == "avx512.kxnor.w" || // Added in 7.0
[X86] Remove kortest intrinsics and replace with native IR. llvm-svn: 324646
2018-02-08 21:16:06 +01:00
Name == "avx512.kortestc.w" || // Added in 7.0
Name == "avx512.kortestz.w" || // Added in 7.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name.startswith("avx512.mask.pshuf.b.") || // Added in 4.0
Name.startswith("avx2.pmax") || // Added in 3.9
Name.startswith("avx2.pmin") || // Added in 3.9
Name.startswith("avx512.mask.pmax") || // Added in 4.0
Name.startswith("avx512.mask.pmin") || // Added in 4.0
Name.startswith("avx2.vbroadcast") || // Added in 3.8
Name.startswith("avx2.pbroadcast") || // Added in 3.8
Name.startswith("avx.vpermil.") || // Added in 3.1
Name.startswith("sse2.pshuf") || // Added in 3.9
Name.startswith("avx512.pbroadcast") || // Added in 3.9
Name.startswith("avx512.mask.broadcast.s") || // Added in 3.9
Name.startswith("avx512.mask.movddup") || // Added in 3.9
Name.startswith("avx512.mask.movshdup") || // Added in 3.9
Name.startswith("avx512.mask.movsldup") || // Added in 3.9
Name.startswith("avx512.mask.pshuf.d.") || // Added in 3.9
Name.startswith("avx512.mask.pshufl.w.") || // Added in 3.9
Name.startswith("avx512.mask.pshufh.w.") || // Added in 3.9
Name.startswith("avx512.mask.shuf.p") || // Added in 4.0
Name.startswith("avx512.mask.vpermil.p") || // Added in 3.9
Name.startswith("avx512.mask.perm.df.") || // Added in 3.9
Name.startswith("avx512.mask.perm.di.") || // Added in 3.9
Name.startswith("avx512.mask.punpckl") || // Added in 3.9
Name.startswith("avx512.mask.punpckh") || // Added in 3.9
Name.startswith("avx512.mask.unpckl.") || // Added in 3.9
Name.startswith("avx512.mask.unpckh.") || // Added in 3.9
Name.startswith("avx512.mask.pand.") || // Added in 3.9
Name.startswith("avx512.mask.pandn.") || // Added in 3.9
Name.startswith("avx512.mask.por.") || // Added in 3.9
Name.startswith("avx512.mask.pxor.") || // Added in 3.9
Name.startswith("avx512.mask.and.") || // Added in 3.9
Name.startswith("avx512.mask.andn.") || // Added in 3.9
Name.startswith("avx512.mask.or.") || // Added in 3.9
Name.startswith("avx512.mask.xor.") || // Added in 3.9
Name.startswith("avx512.mask.padd.") || // Added in 4.0
Name.startswith("avx512.mask.psub.") || // Added in 4.0
Name.startswith("avx512.mask.pmull.") || // Added in 4.0
Name.startswith("avx512.mask.cvtdq2pd.") || // Added in 4.0
Name.startswith("avx512.mask.cvtudq2pd.") || // Added in 4.0
[X86] Remove GCCBuiltins from 512-bit cvt(u)qqtops, cvt(u)qqtopd, and cvt(u)dqtops intrinsics. Add new variadic uitofp/sitofp with rounding mode intrinsics. Summary: See clang patch D56998 for a full description. Reviewers: RKSimon, spatel Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D56999 llvm-svn: 352266
2019-01-26 03:41:54 +01:00
Name.startswith("avx512.mask.cvtudq2ps.") || // Added in 7.0 updated 9.0
Name.startswith("avx512.mask.cvtqq2pd.") || // Added in 7.0 updated 9.0
Name.startswith("avx512.mask.cvtuqq2pd.") || // Added in 7.0 updated 9.0
Name.startswith("avx512.mask.cvtdq2ps.") || // Added in 7.0 updated 9.0
Name == "avx512.mask.cvtqq2ps.256" || // Added in 9.0
Name == "avx512.mask.cvtqq2ps.512" || // Added in 9.0
Name == "avx512.mask.cvtuqq2ps.256" || // Added in 9.0
Name == "avx512.mask.cvtuqq2ps.512" || // Added in 9.0
[X86] Remove some unused masked conversion intrinsics that can be replaced with an older intrinsic and a select. This is what clang already uses. llvm-svn: 332170
2018-05-12 04:34:28 +02:00
Name == "avx512.mask.cvtpd2dq.256" || // Added in 7.0
Name == "avx512.mask.cvtpd2ps.256" || // Added in 7.0
Name == "avx512.mask.cvttpd2dq.256" || // Added in 7.0
Name == "avx512.mask.cvttps2dq.128" || // Added in 7.0
Name == "avx512.mask.cvttps2dq.256" || // Added in 7.0
Name == "avx512.mask.cvtps2pd.128" || // Added in 7.0
Name == "avx512.mask.cvtps2pd.256" || // Added in 7.0
[X86] Remove and autoupgrade the cvtusi2sd intrinsic. Use uitofp+insertelement instead. llvm-svn: 332206
2018-05-14 02:06:49 +02:00
Name == "avx512.cvtusi2sd" || // Added in 7.0
[X86] Remove mask arguments from permvar builtins/intrinsics. Use a select in IR instead. Someday maybe we'll use selects for all intrinsics. llvm-svn: 332824
2018-05-21 01:34:04 +02:00
Name.startswith("avx512.mask.permvar.") || // Added in 7.0
[X86] Remove the pmuldq/pmuldq intrinsics and replace with native IR. This completes the work started in r329604 and r329605 when we changed clang to no longer use the intrinsics. We lost some InstCombine SimplifyDemandedBit optimizations through this change as we aren't able to fold 'and', bitcast, shuffle very well. llvm-svn: 329990
2018-04-13 08:07:18 +02:00
Name == "sse2.pmulu.dq" || // Added in 7.0
Name == "sse41.pmuldq" || // Added in 7.0
Name == "avx2.pmulu.dq" || // Added in 7.0
Name == "avx2.pmul.dq" || // Added in 7.0
Name == "avx512.pmulu.dq.512" || // Added in 7.0
Name == "avx512.pmul.dq.512" || // Added in 7.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name.startswith("avx512.mask.pmul.dq.") || // Added in 4.0
Name.startswith("avx512.mask.pmulu.dq.") || // Added in 4.0
[X86] Add 512-bit unmasked pmulhrsw/pmulhw/pmulhuw intrinsics. Remove and auto upgrade 128/256/512 bit masked pmulhrsw/pmulhw/pmulhuw intrinsics. The 128 and 256 bit versions were already not used by clang. This adds an equivalent unmasked 512 bit version. Then autoupgrades all sizes to use unmasked intrinsics plus select. llvm-svn: 325559
2018-02-20 08:28:14 +01:00
Name.startswith("avx512.mask.pmul.hr.sw.") || // Added in 7.0
Name.startswith("avx512.mask.pmulh.w.") || // Added in 7.0
Name.startswith("avx512.mask.pmulhu.w.") || // Added in 7.0
[X86] Remove 128/256-bit masked pmaddubsw and pmaddwd intrinsics. Replace 512-bit masked intrinsic with unmasked intrinsic and a select. The 128/256-bit versions were no longer used by clang. It uses the legacy SSE/AVX2 version and a select. The 512-bit was changed to the same for consistency. llvm-svn: 329774
2018-04-11 06:55:04 +02:00
Name.startswith("avx512.mask.pmaddw.d.") || // Added in 7.0
Name.startswith("avx512.mask.pmaddubs.w.") || // Added in 7.0
[IR][X86] Fix llvm version number in comments in AutoUpgrade. Forgot the next release is 5.0 not 4.1 llvm-svn: 296092
2017-02-24 06:35:07 +01:00
Name.startswith("avx512.mask.packsswb.") || // Added in 5.0
Name.startswith("avx512.mask.packssdw.") || // Added in 5.0
Name.startswith("avx512.mask.packuswb.") || // Added in 5.0
Name.startswith("avx512.mask.packusdw.") || // Added in 5.0
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-22 22:11:01 +02:00
Name.startswith("avx512.mask.cmp.b") || // Added in 5.0
Name.startswith("avx512.mask.cmp.d") || // Added in 5.0
Name.startswith("avx512.mask.cmp.q") || // Added in 5.0
Name.startswith("avx512.mask.cmp.w") || // Added in 5.0
[X86] Rename the autoupgraded of packed fp compare and fpclass intrinsics that don't take a mask as input to exclude '.mask.' from their name. I think the intrinsics named 'avx512.mask.' should refer to the previous behavior of taking a mask argument in the intrinsic instead of using a 'select' or 'and' instruction in IR to accomplish the masking. This is more consistent with the goal that eventually we will have no intrinsics that have masking builtin. When we reach that goal, we should have no intrinsics named "avx512.mask". llvm-svn: 335744
2018-06-27 17:57:53 +02:00
Name.startswith("avx512.mask.cmp.p") || // Added in 7.0
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-22 22:11:01 +02:00
Name.startswith("avx512.mask.ucmp.") || // Added in 5.0
[X86] Remove llvm.x86.avx512.cvt*2mask.* intrinsics and autoupgrade to (icmp slt X, 0) I had to drop fast-isel-abort from a test because we can't fast isel some of the mask stuff. When we used intrinsics we implicitly fell back to SelectionDAG for the intrinsic call without triggering the abort error. But with native IR that doesn't happen the same way. llvm-svn: 322050
2018-01-09 01:50:47 +01:00
Name.startswith("avx512.cvtb2mask.") || // Added in 7.0
Name.startswith("avx512.cvtw2mask.") || // Added in 7.0
Name.startswith("avx512.cvtd2mask.") || // Added in 7.0
Name.startswith("avx512.cvtq2mask.") || // Added in 7.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name.startswith("avx512.mask.vpermilvar.") || // Added in 4.0
Name.startswith("avx512.mask.psll.d") || // Added in 4.0
Name.startswith("avx512.mask.psll.q") || // Added in 4.0
Name.startswith("avx512.mask.psll.w") || // Added in 4.0
Name.startswith("avx512.mask.psra.d") || // Added in 4.0
Name.startswith("avx512.mask.psra.q") || // Added in 4.0
Name.startswith("avx512.mask.psra.w") || // Added in 4.0
Name.startswith("avx512.mask.psrl.d") || // Added in 4.0
Name.startswith("avx512.mask.psrl.q") || // Added in 4.0
Name.startswith("avx512.mask.psrl.w") || // Added in 4.0
Name.startswith("avx512.mask.pslli") || // Added in 4.0
Name.startswith("avx512.mask.psrai") || // Added in 4.0
Name.startswith("avx512.mask.psrli") || // Added in 4.0
Name.startswith("avx512.mask.psllv") || // Added in 4.0
Name.startswith("avx512.mask.psrav") || // Added in 4.0
Name.startswith("avx512.mask.psrlv") || // Added in 4.0
Name.startswith("sse41.pmovsx") || // Added in 3.8
Name.startswith("sse41.pmovzx") || // Added in 3.9
Name.startswith("avx2.pmovsx") || // Added in 3.9
Name.startswith("avx2.pmovzx") || // Added in 3.9
Name.startswith("avx512.mask.pmovsx") || // Added in 4.0
Name.startswith("avx512.mask.pmovzx") || // Added in 4.0
[AVX-512] Remove lzcnt intrinsics and autoupgrade them to generic ctlz intrinsics with select. Clang has been emitting cltz intrinsics for a while now. llvm-svn: 296091
2017-02-24 06:35:04 +01:00
Name.startswith("avx512.mask.lzcnt.") || // Added in 5.0
[X86] Remove masking from vpternlog intrinsics. Use a select in IR instead. This removes 6 intrinsics since we no longer need separate mask and maskz intrinsics. Differential Revision: https://reviews.llvm.org/D47124 llvm-svn: 332890
2018-05-21 22:58:09 +02:00
Name.startswith("avx512.mask.pternlog.") || // Added in 7.0
Name.startswith("avx512.maskz.pternlog.") || // Added in 7.0
[X86] Remove masking from avx512ifma intrinsics. Use a select instead. This allows us to avoid having mask and maskz variant. Reducing from 12 intrinsics to 6. llvm-svn: 333346
2018-05-26 20:55:19 +02:00
Name.startswith("avx512.mask.vpmadd52") || // Added in 7.0
Name.startswith("avx512.maskz.vpmadd52") || // Added in 7.0
[X86] Remove masked vpermi2var/vpermt2var intrinsics and autoupgrade. We have unmasked intrinsics now and wrap them with a select. This is a net reduction of 36 intrinsics from before the unmasked intrinsics were added. llvm-svn: 333388
2018-05-29 07:22:05 +02:00
Name.startswith("avx512.mask.vpermi2var.") || // Added in 7.0
Name.startswith("avx512.mask.vpermt2var.") || // Added in 7.0
Name.startswith("avx512.maskz.vpermt2var.") || // Added in 7.0
[X86] Remove and autoupgrade masked avx512vnni intrinsics using the unmasked intrinsics and select instructions. llvm-svn: 333857
2018-06-04 01:24:17 +02:00
Name.startswith("avx512.mask.vpdpbusd.") || // Added in 7.0
Name.startswith("avx512.maskz.vpdpbusd.") || // Added in 7.0
Name.startswith("avx512.mask.vpdpbusds.") || // Added in 7.0
Name.startswith("avx512.maskz.vpdpbusds.") || // Added in 7.0
Name.startswith("avx512.mask.vpdpwssd.") || // Added in 7.0
Name.startswith("avx512.maskz.vpdpwssd.") || // Added in 7.0
Name.startswith("avx512.mask.vpdpwssds.") || // Added in 7.0
Name.startswith("avx512.maskz.vpdpwssds.") || // Added in 7.0
[X86] Remove masking from dbpsadbw intrinsics, use select in IR instead. llvm-svn: 334384
2018-06-11 08:18:22 +02:00
Name.startswith("avx512.mask.dbpsadbw.") || // Added in 7.0
[X86] Remove masking from avx512vbmi2 concat and shift by immediate intrinsics. Use select in IR instead. llvm-svn: 334576
2018-06-13 09:19:21 +02:00
Name.startswith("avx512.mask.vpshld.") || // Added in 7.0
Name.startswith("avx512.mask.vpshrd.") || // Added in 7.0
Recommit r350554 "[X86] Remove AVX512VBMI2 concat and shift intrinsics. Replace with target independent funnel shift intrinsics." The MSVC limit we hit on AutoUpgrade.cpp has been worked around for now. llvm-svn: 350567
2019-01-07 22:00:32 +01:00
Name.startswith("avx512.mask.vpshldv.") || // Added in 8.0
Name.startswith("avx512.mask.vpshrdv.") || // Added in 8.0
Name.startswith("avx512.maskz.vpshldv.") || // Added in 8.0
Name.startswith("avx512.maskz.vpshrdv.") || // Added in 8.0
Name.startswith("avx512.vpshld.") || // Added in 8.0
Name.startswith("avx512.vpshrd.") || // Added in 8.0
[X86] Remove masking from 512-bit floating max/min intrinsics. Use select instruction instead. llvm-svn: 335199
2018-06-21 07:00:56 +02:00
Name.startswith("avx512.mask.add.p") || // Added in 7.0. 128/256 in 4.0
Name.startswith("avx512.mask.sub.p") || // Added in 7.0. 128/256 in 4.0
Name.startswith("avx512.mask.mul.p") || // Added in 7.0. 128/256 in 4.0
Name.startswith("avx512.mask.div.p") || // Added in 7.0. 128/256 in 4.0
Name.startswith("avx512.mask.max.p") || // Added in 7.0. 128/256 in 5.0
Name.startswith("avx512.mask.min.p") || // Added in 7.0. 128/256 in 5.0
[X86] Rename the autoupgraded of packed fp compare and fpclass intrinsics that don't take a mask as input to exclude '.mask.' from their name. I think the intrinsics named 'avx512.mask.' should refer to the previous behavior of taking a mask argument in the intrinsic instead of using a 'select' or 'and' instruction in IR to accomplish the masking. This is more consistent with the goal that eventually we will have no intrinsics that have masking builtin. When we reach that goal, we should have no intrinsics named "avx512.mask". llvm-svn: 335744
2018-06-27 17:57:53 +02:00
Name.startswith("avx512.mask.fpclass.p") || // Added in 7.0
[X86] Remove mask parameter from vpshufbitqmb intrinsics. Change result to a vXi1 vector. The input mask can be represented with an AND in IR. Fixes PR40258 llvm-svn: 351028
2019-01-14 01:03:50 +01:00
Name.startswith("avx512.mask.vpshufbitqmb.") || // Added in 8.0
[X86] Remove mask parameter from avx512 pmultishiftqb intrinsics. Use select in IR instead. Fixes PR40259 llvm-svn: 351035
2019-01-14 09:46:45 +01:00
Name.startswith("avx512.mask.pmultishift.qb.") || // Added in 8.0
[X86] Remove and autoupgrade vpconflict intrinsics that take a mask and passthru argument. We have unmasked versions as of r352172 llvm-svn: 352270
2019-01-26 07:27:01 +01:00
Name.startswith("avx512.mask.conflict.") || // Added in 9.0
[X86] Remove and autoupgrade vpmovqd/vpmovwb intrinsics using trunc+select. llvm-svn: 351729
2019-01-21 09:16:59 +01:00
Name == "avx512.mask.pmov.qd.256" || // Added in 9.0
Name == "avx512.mask.pmov.qd.512" || // Added in 9.0
Name == "avx512.mask.pmov.wb.256" || // Added in 9.0
Name == "avx512.mask.pmov.wb.512" || // Added in 9.0
[X86] Remove and autoupgrade cvtsi2ss/cvtsi2sd intrinsics to match what clang has used for a very long time. llvm-svn: 332186
2018-05-13 01:14:39 +02:00
Name == "sse.cvtsi2ss" || // Added in 7.0
Name == "sse.cvtsi642ss" || // Added in 7.0
Name == "sse2.cvtsi2sd" || // Added in 7.0
Name == "sse2.cvtsi642sd" || // Added in 7.0
[X86] Remove an autoupgrade legacy cvtss2sd intrinsics. llvm-svn: 332187
2018-05-13 02:29:40 +02:00
Name == "sse2.cvtss2sd" || // Added in 7.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name == "sse2.cvtdq2pd" || // Added in 3.9
[X86] Remove 128/256-bit cvtdq2ps, cvtudq2ps, cvtqq2pd, cvtuqq2pd intrinsics. These can all be implemented with sitofp/uitofp instructions. llvm-svn: 332916
2018-05-22 01:15:00 +02:00
Name == "sse2.cvtdq2ps" || // Added in 7.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name == "sse2.cvtps2pd" || // Added in 3.9
Name == "avx.cvtdq2.pd.256" || // Added in 3.9
[X86] Remove 128/256-bit cvtdq2ps, cvtudq2ps, cvtqq2pd, cvtuqq2pd intrinsics. These can all be implemented with sitofp/uitofp instructions. llvm-svn: 332916
2018-05-22 01:15:00 +02:00
Name == "avx.cvtdq2.ps.256" || // Added in 7.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name == "avx.cvt.ps2.pd.256" || // Added in 3.9
Name.startswith("avx.vinsertf128.") || // Added in 3.7
Name == "avx2.vinserti128" || // Added in 3.7
Name.startswith("avx512.mask.insert") || // Added in 4.0
Name.startswith("avx.vextractf128.") || // Added in 3.7
Name == "avx2.vextracti128" || // Added in 3.7
Name.startswith("avx512.mask.vextract") || // Added in 4.0
Name.startswith("sse4a.movnt.") || // Added in 3.9
Name.startswith("avx.movnt.") || // Added in 3.2
Name.startswith("avx512.storent.") || // Added in 3.9
[X86][SSE] Update MOVNTDQA non-temporal loads to generic implementation (LLVM) MOVNTDQA non-temporal aligned vector loads can be correctly represented using generic builtin loads, allowing us to remove the existing x86 intrinsics. Clang companion patch: D31766. Differential Revision: https://reviews.llvm.org/D31767 llvm-svn: 300325
2017-04-14 17:05:35 +02:00
Name == "sse41.movntdqa" || // Added in 5.0
Name == "avx2.movntdqa" || // Added in 5.0
Name == "avx512.movntdqa" || // Added in 5.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name == "sse2.storel.dq" || // Added in 3.9
Name.startswith("sse.storeu.") || // Added in 3.9
Name.startswith("sse2.storeu.") || // Added in 3.9
Name.startswith("avx.storeu.") || // Added in 3.9
Name.startswith("avx512.mask.storeu.") || // Added in 3.9
Name.startswith("avx512.mask.store.p") || // Added in 3.9
Name.startswith("avx512.mask.store.b.") || // Added in 3.9
Name.startswith("avx512.mask.store.w.") || // Added in 3.9
Name.startswith("avx512.mask.store.d.") || // Added in 3.9
Name.startswith("avx512.mask.store.q.") || // Added in 3.9
[X86] Remove and autoupgrade the avx512.mask.store.ss intrinsic. llvm-svn: 332079
2018-05-11 06:33:18 +02:00
Name == "avx512.mask.store.ss" || // Added in 7.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name.startswith("avx512.mask.loadu.") || // Added in 3.9
Name.startswith("avx512.mask.load.") || // Added in 3.9
[X86] Remove and autoupgrade the expandload and compressstore intrinsics. We use the target independent intrinsics now. llvm-svn: 334381
2018-06-11 03:25:22 +02:00
Name.startswith("avx512.mask.expand.load.") || // Added in 7.0
Name.startswith("avx512.mask.compress.store.") || // Added in 7.0
[X86] Add new variadic avx512 compress/expand intrinsics that use vXi1 types for the mask argument. Remove and autoupgrade the old intrinsics llvm-svn: 352343
2019-01-28 08:03:03 +01:00
Name.startswith("avx512.mask.expand.b") || // Added in 9.0
Name.startswith("avx512.mask.expand.w") || // Added in 9.0
Name.startswith("avx512.mask.expand.d") || // Added in 9.0
Name.startswith("avx512.mask.expand.q") || // Added in 9.0
Name.startswith("avx512.mask.expand.p") || // Added in 9.0
Name.startswith("avx512.mask.compress.b") || // Added in 9.0
Name.startswith("avx512.mask.compress.w") || // Added in 9.0
Name.startswith("avx512.mask.compress.d") || // Added in 9.0
Name.startswith("avx512.mask.compress.q") || // Added in 9.0
Name.startswith("avx512.mask.compress.p") || // Added in 9.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name == "sse42.crc32.64.8" || // Added in 3.4
Name.startswith("avx.vbroadcast.s") || // Added in 3.5
[X86] Remove and autoupgrade avx512.vbroadcast.ss/avx512.vbroadcast.sd intrinsics. llvm-svn: 332271
2018-05-14 20:21:22 +02:00
Name.startswith("avx512.vbroadcast.s") || // Added in 7.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name.startswith("avx512.mask.palignr.") || // Added in 3.9
Name.startswith("avx512.mask.valign.") || // Added in 4.0
Name.startswith("sse2.psll.dq") || // Added in 3.7
Name.startswith("sse2.psrl.dq") || // Added in 3.7
Name.startswith("avx2.psll.dq") || // Added in 3.7
Name.startswith("avx2.psrl.dq") || // Added in 3.7
Name.startswith("avx512.psll.dq") || // Added in 3.9
Name.startswith("avx512.psrl.dq") || // Added in 3.9
Name == "sse41.pblendw" || // Added in 3.7
Name.startswith("sse41.blendp") || // Added in 3.7
Name.startswith("avx.blend.p") || // Added in 3.7
Name == "avx2.pblendw" || // Added in 3.7
Name.startswith("avx2.pblendd.") || // Added in 3.7
Name.startswith("avx.vbroadcastf128") || // Added in 4.0
Name == "avx2.vbroadcasti128" || // Added in 3.7
[X86] Finishing broadcastf32x2 and broadcasti32x2 intrinsics lowering to IR. llvm side. Removing X86 broadcast(f/i)32x2 intrinsics from llvm. Adding autoUpgrade support. Moving matching tests from avx512dq-intrinsics.ll to avx512dq-intrinsics-upgrade.ll and from avx512dqvl-intrinsics.ll to avx512dqvl-intrinsics-upgrade.ll. Differential Revision: https://reviews.llvm.org/D38220 llvm-svn: 314195
2017-09-26 09:39:39 +02:00
Name.startswith("avx512.mask.broadcastf") || // Added in 6.0
Name.startswith("avx512.mask.broadcasti") || // Added in 6.0
[X86][IR] In AutoUpgrade, check explicitly for xop.vpcmov and xop.vpcmov.256 instead of anything starting with xop.vpcmov There were some older intrinsics that only existed for less than a month in 2012 that still exist in some out of tree test files that start with this string, but aren't able to be handled by the current upgrade code and fire an assert. Now we'll go back to treating them as not intrinsics at all and just passing them through to output. Fixes PR32041, sort of. llvm-svn: 295930
2017-02-23 04:22:14 +01:00
Name == "xop.vpcmov" || // Added in 3.8
[IR][X86] Fix llvm version number in comments in AutoUpgrade. Forgot the next release is 5.0 not 4.1 llvm-svn: 296092
2017-02-24 06:35:07 +01:00
Name == "xop.vpcmov.256" || // Added in 5.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
Name.startswith("avx512.mask.move.s") || // Added in 4.0
[X86][LLVM] Converting __mm{|256|512}_movm_epi{8|16|32|64} LLVMIR call into generic intrinsics. This patch is a part one of two reviews, one for the clang and the other for LLVM. The patch deletes the back-end intrinsics and adds support for them in the auto upgrade. Differential Revision: https://reviews.llvm.org/D31393 llvm-svn: 299432
2017-04-04 15:32:14 +02:00
Name.startswith("avx512.cvtmask2") || // Added in 5.0
[X86] Auto upgrade VPCOM/VPCOMU intrinsics to generic integer comparisons This causes a couple of changes in the upgrade tests as signed/unsigned eq/ne are equivalent and we constant fold true/false codes, these changes are the same as what we already do for avx512 cmp/ucmp. Noticed while cleaning up vector integer comparison costs for PR40376. llvm-svn: 351697
2019-01-20 20:27:40 +01:00
Name.startswith("xop.vpcom") || // Added in 3.2, Updated in 9.0
[X86] Auto upgrade XOP/AVX512 rotation intrinsics to generic funnel shift intrinsics (llvm) This emits FSHL/FSHR generic intrinsics for the XOP VPROT and AVX512 VPROL/VPROR rotation intrinsics. Clang counterpart: https://reviews.llvm.org/D55937 Differential Revision: https://reviews.llvm.org/D55938 llvm-svn: 349795
2018-12-20 20:01:07 +01:00
Name.startswith("xop.vprot") || // Added in 8.0
Name.startswith("avx512.prol") || // Added in 8.0
Name.startswith("avx512.pror") || // Added in 8.0
Name.startswith("avx512.mask.prorv.") || // Added in 8.0
Name.startswith("avx512.mask.pror.") || // Added in 8.0
Name.startswith("avx512.mask.prolv.") || // Added in 8.0
Name.startswith("avx512.mask.prol.") || // Added in 8.0
[X86] test/testn intrinsics lowering to IR. llvm part. Remove builtins from llvm and add AutoUpgrade support. Also add fast-isel tests for the TEST and TESTN instructions. Differential Revision: https://reviews.llvm.org/D38736 llvm-svn: 318036
2017-11-13 13:51:18 +01:00
Name.startswith("avx512.ptestm") || //Added in 6.0
Name.startswith("avx512.ptestnm") || //Added in 6.0
[X86] Lower _mm[256|512]_[mask[z]]_avg_epu[8|16] intrinsics to native llvm IR Differential Revision: https://reviews.llvm.org/D37560 llvm-svn: 313013
2017-09-12 09:50:35 +02:00
Name.startswith("avx512.mask.pavg")) // Added in 6.0
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
return true;
return false;
}
static bool UpgradeX86IntrinsicFunction(Function *F, StringRef Name,
Function *&NewFn) {
// Only handle intrinsics that start with "x86.".
if (!Name.startswith("x86."))
return false;
// Remove "x86." prefix.
Name = Name.substr(4);
if (ShouldUpgradeX86Intrinsic(F, Name)) {
NewFn = nullptr;
return true;
}
[X86] Add some comments about when some X86 intrinsic autoupgrade code was added. Someday we'd like to remove old autoupgrade code so it helps to annotate how long its been there so we don't have to go digging through commit history. llvm-svn: 348728
2018-12-09 19:02:40 +01:00
if (Name == "rdtscp") { // Added in 8.0
[X86] Modify the the rdtscp intrinsic to return values instead of taking a pointer argument Similar to what was recently done for addcarry/subborrow and has been done for rdrand/rdseed for a while. It's better to use two results and an explicit store in IR when the store isn't part of the semantics of the instruction. This allows store->load forwarding to happen in the middle end. Or the store to be removed if its never loaded. Differential Revision: https://reviews.llvm.org/D51803 llvm-svn: 341698
2018-09-07 21:14:15 +02:00
// If this intrinsic has 0 operands, it's the new version.
if (F->getFunctionType()->getNumParams() == 0)
return false;
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::x86_rdtscp);
return true;
}
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
// SSE4.1 ptest functions may have an old signature.
if (Name.startswith("sse41.ptest")) { // Added in 3.2
if (Name.substr(11) == "c")
[IR][X86] Rename an AutoUpgrade helper function to more accurately match what intrinsics it handles. NFC llvm-svn: 295425
2017-02-17 08:07:21 +01:00
return UpgradePTESTIntrinsic(F, Intrinsic::x86_sse41_ptestc, NewFn);
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
if (Name.substr(11) == "z")
[IR][X86] Rename an AutoUpgrade helper function to more accurately match what intrinsics it handles. NFC llvm-svn: 295425
2017-02-17 08:07:21 +01:00
return UpgradePTESTIntrinsic(F, Intrinsic::x86_sse41_ptestz, NewFn);
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
if (Name.substr(11) == "nzc")
[IR][X86] Rename an AutoUpgrade helper function to more accurately match what intrinsics it handles. NFC llvm-svn: 295425
2017-02-17 08:07:21 +01:00
return UpgradePTESTIntrinsic(F, Intrinsic::x86_sse41_ptestnzc, NewFn);
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
}
// Several blend and other instructions with masks used the wrong number of
// bits.
if (Name == "sse41.insertps") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_sse41_insertps,
NewFn);
if (Name == "sse41.dppd") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_sse41_dppd,
NewFn);
if (Name == "sse41.dpps") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_sse41_dpps,
NewFn);
if (Name == "sse41.mpsadbw") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_sse41_mpsadbw,
NewFn);
if (Name == "avx.dp.ps.256") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_avx_dp_ps_256,
NewFn);
if (Name == "avx2.mpsadbw") // Added in 3.6
return UpgradeX86IntrinsicsWith8BitMask(F, Intrinsic::x86_avx2_mpsadbw,
NewFn);
// frcz.ss/sd may need to have an argument dropped. Added in 3.2
if (Name.startswith("xop.vfrcz.ss") && F->arg_size() == 2) {
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::x86_xop_vfrcz_ss);
return true;
}
if (Name.startswith("xop.vfrcz.sd") && F->arg_size() == 2) {
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::x86_xop_vfrcz_sd);
return true;
}
// Upgrade any XOP PERMIL2 index operand still using a float/double vector.
if (Name.startswith("xop.vpermil2")) { // Added in 3.9
[IR][X86] Simplify some AutoUpgrade code slightly. NFC llvm-svn: 295426
2017-02-17 08:07:24 +01:00
auto Idx = F->getFunctionType()->getParamType(2);
if (Idx->isFPOrFPVectorTy()) {
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
rename(F);
unsigned IdxSize = Idx->getPrimitiveSizeInBits();
unsigned EltSize = Idx->getScalarSizeInBits();
Intrinsic::ID Permil2ID;
if (EltSize == 64 && IdxSize == 128)
Permil2ID = Intrinsic::x86_xop_vpermil2pd;
else if (EltSize == 32 && IdxSize == 128)
Permil2ID = Intrinsic::x86_xop_vpermil2ps;
else if (EltSize == 64 && IdxSize == 256)
Permil2ID = Intrinsic::x86_xop_vpermil2pd_256;
else
Permil2ID = Intrinsic::x86_xop_vpermil2ps_256;
NewFn = Intrinsic::getDeclaration(F->getParent(), Permil2ID);
return true;
}
}
[AutoUpgrade] Fix AutoUpgrade for x86.seh.recoverfp Summary: This fixes the bug in https://reviews.llvm.org/D56747#inline-502711. Reviewers: efriedma Reviewed By: efriedma Subscribers: javed.absar, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D57614 llvm-svn: 352945
2019-02-02 02:32:48 +01:00
if (Name == "seh.recoverfp") {
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::eh_recoverfp);
return true;
}
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
return false;
}
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-17 23:33:23 +01:00
static bool UpgradeIntrinsicFunction1(Function *F, Function *&NewFn) {
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
assert(F && "Illegal to upgrade a non-existent Function.");
// Quickly eliminate it, if it's not a candidate.
Move setName after accessing Name llvm-svn: 274862
2016-07-08 18:09:51 +02:00
StringRef Name = F->getName();
rework the remaining autoupgrade logic to use a StringRef instead of creating a temporary std::string for every function being checked. llvm-svn: 133355
2011-06-18 20:56:39 +02:00
if (Name.size() <= 8 || !Name.startswith("llvm."))
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-17 23:33:23 +01:00
return false;
rework the remaining autoupgrade logic to use a StringRef instead of creating a temporary std::string for every function being checked. llvm-svn: 133355
2011-06-18 20:56:39 +02:00
Name = Name.substr(5); // Strip off "llvm."
Eli managed to kill off llvm.membarrier in llvm 3.0 also, this means that mainline needs no autoupgrade logic for intrinsics yet, woohoo! llvm-svn: 145178
2011-11-27 09:42:07 +01:00
rework the remaining autoupgrade logic to use a StringRef instead of creating a temporary std::string for every function being checked. llvm-svn: 133355
2011-06-18 20:56:39 +02:00
switch (Name[0]) {
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
default: break;
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 01:25:25 +02:00
case 'a': {
[ARM] Remove rbit intrinsics and autoupgrade to generic bitreverse. Testing already covered by CodeGen/ARM/rbit.ll llvm-svn: 291587
2017-01-10 20:23:51 +01:00
if (Name.startswith("arm.rbit") || Name.startswith("aarch64.rbit")) {
[AArch64] Add support for lowering bitreverse to the rbit instruction. Differential Revision: https://reviews.llvm.org/D28379 llvm-svn: 291575
2017-01-10 18:20:33 +01:00
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::bitreverse,
F->arg_begin()->getType());
return true;
}
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 01:25:25 +02:00
if (Name.startswith("arm.neon.vclz")) {
Type* args[2] = {
Remove trailing whitespace, fix file path in comment llvm-svn: 186766
2013-07-20 19:46:00 +02:00
F->arg_begin()->getType(),
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 01:25:25 +02:00
Type::getInt1Ty(F->getContext())
};
// Can't use Intrinsic::getDeclaration here as it adds a ".i1" to
// the end of the name. Change name from llvm.arm.neon.vclz.* to
// llvm.ctlz.*
FunctionType* fType = FunctionType::get(F->getReturnType(), args, false);
Allow creating llvm::Function in non-zero address spaces Most users won't have to worry about this as all of the 'getOrInsertFunction' functions on Module will default to the program address space. An overload has been added to Function::Create to abstract away the details for most callers. This is based on https://reviews.llvm.org/D37054 but without the changes to make passing a Module to Function::Create() mandatory. I have also added some more tests and fixed the LLParser to accept call instructions for types in the program address space. Reviewed By: bjope Differential Revision: https://reviews.llvm.org/D47541 llvm-svn: 340519
2018-08-23 11:25:17 +02:00
NewFn = Function::Create(fType, F->getLinkage(), F->getAddressSpace(),
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 01:25:25 +02:00
"llvm.ctlz." + Name.substr(14), F->getParent());
return true;
}
More replacing of target-dependent intrinsics with target-indepdent intrinsics. The second instruction(s) to be handled are the vector versions of count set bits (ctpop). The changes here are to clang so that it generates a target independent vector ctpop when it sees an ARM dependent vector bits set count. The changes in llvm are to match the target independent vector ctpop and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector pop counts with target-independent ctpops. There are also changes to an existing test case in llvm for ARM vector count instructions and to a test for the bitcode upgrade. <rdar://problem/11892519> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160410
2012-07-18 02:02:16 +02:00
if (Name.startswith("arm.neon.vcnt")) {
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::ctpop,
F->arg_begin()->getType());
return true;
}
Regex: Make "match" and "sub" const member functions Summary: The Regex "match" and "sub" member functions were previously not "const" because they wrote to the "error" member variable. This commit removes those assignments, and instead assumes that the validity of the regex is already known after the initial compilation of the regular expression. As a result, these member functions were possible to make "const". This makes it easier to do things like pre-compile Regexes up-front, and makes "match" and "sub" thread-safe. The error status is now returned as an optional output, which also makes the API of "match" and "sub" more consistent with each other. Also, some uses of Regex that could be refactored to be const were made const. Patch by Nicolas Guillemot Reviewers: jankratochvil, thopre Subscribers: llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D67241 llvm-svn: 372764
2019-09-24 16:42:36 +02:00
static const Regex vldRegex("^arm\\.neon\\.vld([1234]|[234]lane)\\.v[a-z0-9]*$");
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 12:56:37 +02:00
if (vldRegex.match(Name)) {
auto fArgs = F->getFunctionType()->params();
SmallVector<Type *, 4> Tys(fArgs.begin(), fArgs.end());
// Can't use Intrinsic::getDeclaration here as the return types might
// then only be structurally equal.
FunctionType* fType = FunctionType::get(F->getReturnType(), Tys, false);
Allow creating llvm::Function in non-zero address spaces Most users won't have to worry about this as all of the 'getOrInsertFunction' functions on Module will default to the program address space. An overload has been added to Function::Create to abstract away the details for most callers. This is based on https://reviews.llvm.org/D37054 but without the changes to make passing a Module to Function::Create() mandatory. I have also added some more tests and fixed the LLParser to accept call instructions for types in the program address space. Reviewed By: bjope Differential Revision: https://reviews.llvm.org/D47541 llvm-svn: 340519
2018-08-23 11:25:17 +02:00
NewFn = Function::Create(fType, F->getLinkage(), F->getAddressSpace(),
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 12:56:37 +02:00
"llvm." + Name + ".p0i8", F->getParent());
return true;
}
Regex: Make "match" and "sub" const member functions Summary: The Regex "match" and "sub" member functions were previously not "const" because they wrote to the "error" member variable. This commit removes those assignments, and instead assumes that the validity of the regex is already known after the initial compilation of the regular expression. As a result, these member functions were possible to make "const". This makes it easier to do things like pre-compile Regexes up-front, and makes "match" and "sub" thread-safe. The error status is now returned as an optional output, which also makes the API of "match" and "sub" more consistent with each other. Also, some uses of Regex that could be refactored to be const were made const. Patch by Nicolas Guillemot Reviewers: jankratochvil, thopre Subscribers: llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D67241 llvm-svn: 372764
2019-09-24 16:42:36 +02:00
static const Regex vstRegex("^arm\\.neon\\.vst([1234]|[234]lane)\\.v[a-z0-9]*$");
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 12:56:37 +02:00
if (vstRegex.match(Name)) {
Make a bunch of static arrays const. llvm-svn: 250642
2015-10-18 07:15:34 +02:00
static const Intrinsic::ID StoreInts[] = {Intrinsic::arm_neon_vst1,
Intrinsic::arm_neon_vst2,
Intrinsic::arm_neon_vst3,
Intrinsic::arm_neon_vst4};
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 12:56:37 +02:00
Make a bunch of static arrays const. llvm-svn: 250642
2015-10-18 07:15:34 +02:00
static const Intrinsic::ID StoreLaneInts[] = {
Intrinsic::arm_neon_vst2lane, Intrinsic::arm_neon_vst3lane,
Intrinsic::arm_neon_vst4lane
};
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 12:56:37 +02:00
auto fArgs = F->getFunctionType()->params();
Type *Tys[] = {fArgs[0], fArgs[1]};
if (Name.find("lane") == StringRef::npos)
NewFn = Intrinsic::getDeclaration(F->getParent(),
StoreInts[fArgs.size() - 3], Tys);
else
NewFn = Intrinsic::getDeclaration(F->getParent(),
StoreLaneInts[fArgs.size() - 5], Tys);
return true;
}
[AArch64] [ARM] Make a target-independent llvm.thread.pointer intrinsic. Both AArch64 and ARM support llvm.<arch>.thread.pointer intrinsics that just return the thread pointer. I have a pending patch that does the same for SystemZ (D19054), and there are many more targets that could benefit from one. This patch merges the ARM and AArch64 intrinsics into a single target independent one that will also be used by subsequent targets. Differential Revision: http://reviews.llvm.org/D19098 llvm-svn: 266818
2016-04-19 22:51:05 +02:00
if (Name == "aarch64.thread.pointer" || Name == "arm.thread.pointer") {
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::thread_pointer);
return true;
}
[ARM] Replace arm_neon_vqadds with sadd_sat This replaces the A32 NEON vqadds, vqaddu, vqsubs and vqsubu intrinsics with the target independent sadd_sat, uadd_sat, ssub_sat and usub_sat. This helps generate vqadds from standard IR nodes, which might be produced from the vectoriser. The old variants are removed in the process. Differential Revision: https://reviews.llvm.org/D69350
2019-11-27 12:01:27 +01:00
if (Name.startswith("arm.neon.vqadds.")) {
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::sadd_sat,
F->arg_begin()->getType());
return true;
}
if (Name.startswith("arm.neon.vqaddu.")) {
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::uadd_sat,
F->arg_begin()->getType());
return true;
}
if (Name.startswith("arm.neon.vqsubs.")) {
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::ssub_sat,
F->arg_begin()->getType());
return true;
}
if (Name.startswith("arm.neon.vqsubu.")) {
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::usub_sat,
F->arg_begin()->getType());
return true;
}
[AArch64] Split the neon.addp intrinsic into integer and fp variants. This is the result of discussions on the list about how to deal with intrinsics which require codegen to disambiguate them via only the integer/fp overloads. It causes problems for GlobalISel as some of that information is lost during translation, while with other operations like IR instructions the information is encoded into the instruction opcode. This patch changes clang to emit the new faddp intrinsic if the vector operands to the builtin have FP element types. LLVM IR AutoUpgrade has been taught to upgrade existing calls to aarch64.neon.addp with fp vector arguments, and we remove the workarounds introduced for GlobalISel in r355865. This is a more permanent solution to PR40968. Differential Revision: https://reviews.llvm.org/D59655 llvm-svn: 356722
2019-03-21 23:31:37 +01:00
if (Name.startswith("aarch64.neon.addp")) {
if (F->arg_size() != 2)
break; // Invalid IR.
AArch64: fix bitcode upgrade of @llvm.neon.addp. We were upgrading it to faddp, but a version taking two type parameters instead of one. This then got upgraded a second time to the version with just one parameter, but occasionally (for reasons I don't understand) this unusual two-stage process corrupted a use-list, leading to a crash when the two faddp declarations didn't match.
2020-01-09 15:28:48 +01:00
VectorType *Ty = dyn_cast<VectorType>(F->getReturnType());
if (Ty && Ty->getElementType()->isFloatingPointTy()) {
[AArch64] Split the neon.addp intrinsic into integer and fp variants. This is the result of discussions on the list about how to deal with intrinsics which require codegen to disambiguate them via only the integer/fp overloads. It causes problems for GlobalISel as some of that information is lost during translation, while with other operations like IR instructions the information is encoded into the instruction opcode. This patch changes clang to emit the new faddp intrinsic if the vector operands to the builtin have FP element types. LLVM IR AutoUpgrade has been taught to upgrade existing calls to aarch64.neon.addp with fp vector arguments, and we remove the workarounds introduced for GlobalISel in r355865. This is a more permanent solution to PR40968. Differential Revision: https://reviews.llvm.org/D59655 llvm-svn: 356722
2019-03-21 23:31:37 +01:00
NewFn = Intrinsic::getDeclaration(F->getParent(),
AArch64: fix bitcode upgrade of @llvm.neon.addp. We were upgrading it to faddp, but a version taking two type parameters instead of one. This then got upgraded a second time to the version with just one parameter, but occasionally (for reasons I don't understand) this unusual two-stage process corrupted a use-list, leading to a crash when the two faddp declarations didn't match.
2020-01-09 15:28:48 +01:00
Intrinsic::aarch64_neon_faddp, Ty);
[AArch64] Split the neon.addp intrinsic into integer and fp variants. This is the result of discussions on the list about how to deal with intrinsics which require codegen to disambiguate them via only the integer/fp overloads. It causes problems for GlobalISel as some of that information is lost during translation, while with other operations like IR instructions the information is encoded into the instruction opcode. This patch changes clang to emit the new faddp intrinsic if the vector operands to the builtin have FP element types. LLVM IR AutoUpgrade has been taught to upgrade existing calls to aarch64.neon.addp with fp vector arguments, and we remove the workarounds introduced for GlobalISel in r355865. This is a more permanent solution to PR40968. Differential Revision: https://reviews.llvm.org/D59655 llvm-svn: 356722
2019-03-21 23:31:37 +01:00
return true;
}
}
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 01:25:25 +02:00
break;
}
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 12:56:37 +02:00
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 05:26:04 +01:00
case 'c': {
if (Name.startswith("ctlz.") && F->arg_size() == 1) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 17:51:42 +02:00
rename(F);
Don't rely in there being one argument before we've actually identified a function to upgrade. Also, simplify the code a bit at the expense of one line. llvm-svn: 146368
2011-12-12 11:57:20 +01:00
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::ctlz,
F->arg_begin()->getType());
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 05:26:04 +01:00
return true;
}
if (Name.startswith("cttz.") && F->arg_size() == 1) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 17:51:42 +02:00
rename(F);
Don't rely in there being one argument before we've actually identified a function to upgrade. Also, simplify the code a bit at the expense of one line. llvm-svn: 146368
2011-12-12 11:57:20 +01:00
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::cttz,
F->arg_begin()->getType());
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 05:26:04 +01:00
return true;
}
break;
}
Remove the obsolete offset parameter from @llvm.dbg.value There is no situation where this rarely-used argument cannot be substituted with a DIExpression and removing it allows us to simplify the DWARF backend. Note that this patch does not yet remove any of the newly dead code. rdar://problem/33580047 Differential Revision: https://reviews.llvm.org/D35951 llvm-svn: 309426
2017-07-28 22:21:02 +02:00
case 'd': {
if (Name == "dbg.value" && F->arg_size() == 4) {
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::dbg_value);
return true;
}
break;
}
Change semantics of fadd/fmul vector reductions. This patch changes how LLVM handles the accumulator/start value in the reduction, by never ignoring it regardless of the presence of fast-math flags on callsites. This change introduces the following new intrinsics to replace the existing ones: llvm.experimental.vector.reduce.fadd -> llvm.experimental.vector.reduce.v2.fadd llvm.experimental.vector.reduce.fmul -> llvm.experimental.vector.reduce.v2.fmul and adds functionality to auto-upgrade existing LLVM IR and bitcode. Reviewers: RKSimon, greened, dmgreen, nikic, simoll, aemerson Reviewed By: nikic Differential Revision: https://reviews.llvm.org/D60261 llvm-svn: 363035
2019-06-11 10:22:10 +02:00
case 'e': {
SmallVector<StringRef, 2> Groups;
Regex: Make "match" and "sub" const member functions Summary: The Regex "match" and "sub" member functions were previously not "const" because they wrote to the "error" member variable. This commit removes those assignments, and instead assumes that the validity of the regex is already known after the initial compilation of the regular expression. As a result, these member functions were possible to make "const". This makes it easier to do things like pre-compile Regexes up-front, and makes "match" and "sub" thread-safe. The error status is now returned as an optional output, which also makes the API of "match" and "sub" more consistent with each other. Also, some uses of Regex that could be refactored to be const were made const. Patch by Nicolas Guillemot Reviewers: jankratochvil, thopre Subscribers: llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D67241 llvm-svn: 372764
2019-09-24 16:42:36 +02:00
static const Regex R("^experimental.vector.reduce.([a-z]+)\\.[fi][0-9]+");
Change semantics of fadd/fmul vector reductions. This patch changes how LLVM handles the accumulator/start value in the reduction, by never ignoring it regardless of the presence of fast-math flags on callsites. This change introduces the following new intrinsics to replace the existing ones: llvm.experimental.vector.reduce.fadd -> llvm.experimental.vector.reduce.v2.fadd llvm.experimental.vector.reduce.fmul -> llvm.experimental.vector.reduce.v2.fmul and adds functionality to auto-upgrade existing LLVM IR and bitcode. Reviewers: RKSimon, greened, dmgreen, nikic, simoll, aemerson Reviewed By: nikic Differential Revision: https://reviews.llvm.org/D60261 llvm-svn: 363035
2019-06-11 10:22:10 +02:00
if (R.match(Name, &Groups)) {
Intrinsic::ID ID = Intrinsic::not_intrinsic;
if (Groups[1] == "fadd")
ID = Intrinsic::experimental_vector_reduce_v2_fadd;
if (Groups[1] == "fmul")
ID = Intrinsic::experimental_vector_reduce_v2_fmul;
if (ID != Intrinsic::not_intrinsic) {
rename(F);
auto Args = F->getFunctionType()->params();
Type *Tys[] = {F->getFunctionType()->getReturnType(), Args[1]};
NewFn = Intrinsic::getDeclaration(F->getParent(), ID, Tys);
return true;
}
}
break;
}
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-10 22:18:21 +02:00
case 'i':
case 'l': {
bool IsLifetimeStart = Name.startswith("lifetime.start");
if (IsLifetimeStart || Name.startswith("invariant.start")) {
Intrinsic::ID ID = IsLifetimeStart ?
Intrinsic::lifetime_start : Intrinsic::invariant_start;
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 01:31:24 +02:00
auto Args = F->getFunctionType()->params();
Type* ObjectPtr[1] = {Args[1]};
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-10 22:18:21 +02:00
if (F->getName() != Intrinsic::getName(ID, ObjectPtr)) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 17:51:42 +02:00
rename(F);
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-10 22:18:21 +02:00
NewFn = Intrinsic::getDeclaration(F->getParent(), ID, ObjectPtr);
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 01:31:24 +02:00
return true;
}
}
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-10 22:18:21 +02:00
bool IsLifetimeEnd = Name.startswith("lifetime.end");
if (IsLifetimeEnd || Name.startswith("invariant.end")) {
Intrinsic::ID ID = IsLifetimeEnd ?
Intrinsic::lifetime_end : Intrinsic::invariant_end;
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 01:31:24 +02:00
auto Args = F->getFunctionType()->params();
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-10 22:18:21 +02:00
Type* ObjectPtr[1] = {Args[IsLifetimeEnd ? 1 : 2]};
if (F->getName() != Intrinsic::getName(ID, ObjectPtr)) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 17:51:42 +02:00
rename(F);
Add address space mangling to lifetime intrinsics In preparation for allowing allocas to have non-0 addrspace. llvm-svn: 299876
2017-04-10 22:18:21 +02:00
NewFn = Intrinsic::getDeclaration(F->getParent(), ID, ObjectPtr);
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 01:31:24 +02:00
return true;
}
}
Rename invariant.group.barrier to launder.invariant.group Summary: This is one of the initial commit of "RFC: Devirtualization v2" proposal: https://docs.google.com/document/d/16GVtCpzK8sIHNc2qZz6RN8amICNBtvjWUod2SujZVEo/edit?usp=sharing Reviewers: rsmith, amharc, kuhar, sanjoy Subscribers: arsenm, nhaehnle, javed.absar, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D45111 llvm-svn: 331448
2018-05-03 13:03:01 +02:00
if (Name.startswith("invariant.group.barrier")) {
// Rename invariant.group.barrier to launder.invariant.group
auto Args = F->getFunctionType()->params();
Type* ObjectPtr[1] = {Args[0]};
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::launder_invariant_group, ObjectPtr);
return true;
}
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 01:31:24 +02:00
break;
}
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-28 20:27:25 +02:00
case 'm': {
if (Name.startswith("masked.load.")) {
Type *Tys[] = { F->getReturnType(), F->arg_begin()->getType() };
if (F->getName() != Intrinsic::getName(Intrinsic::masked_load, Tys)) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 17:51:42 +02:00
rename(F);
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-28 20:27:25 +02:00
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::masked_load,
Tys);
return true;
}
}
if (Name.startswith("masked.store.")) {
auto Args = F->getFunctionType()->params();
Type *Tys[] = { Args[0], Args[1] };
if (F->getName() != Intrinsic::getName(Intrinsic::masked_store, Tys)) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 17:51:42 +02:00
rename(F);
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-28 20:27:25 +02:00
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::masked_store,
Tys);
return true;
}
}
Support arbitrary address space pointers in masked gather/scatter intrinsics. Fixes PR31789 - When loop-vectorize tries to use these intrinsics for a non-default address space pointer we fail with a "Calling a function with a bad singature!" assertion. This patch solves this by adding the 'vector of pointers' argument as an overloaded type which will determine the address space. Differential revision: https://reviews.llvm.org/D31490 llvm-svn: 302018
2017-05-03 14:28:54 +02:00
// Renaming gather/scatter intrinsics with no address space overloading
// to the new overload which includes an address space
if (Name.startswith("masked.gather.")) {
Type *Tys[] = {F->getReturnType(), F->arg_begin()->getType()};
if (F->getName() != Intrinsic::getName(Intrinsic::masked_gather, Tys)) {
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::masked_gather, Tys);
return true;
}
}
if (Name.startswith("masked.scatter.")) {
auto Args = F->getFunctionType()->params();
Type *Tys[] = {Args[0], Args[1]};
if (F->getName() != Intrinsic::getName(Intrinsic::masked_scatter, Tys)) {
rename(F);
NewFn = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::masked_scatter, Tys);
return true;
}
}
Remove alignment argument from memcpy/memmove/memset in favour of alignment attributes (Step 1) Summary: This is a resurrection of work first proposed and discussed in Aug 2015: http://lists.llvm.org/pipermail/llvm-dev/2015-August/089384.html and initially landed (but then backed out) in Nov 2015: http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20151109/312083.html The @llvm.memcpy/memmove/memset intrinsics currently have an explicit argument which is required to be a constant integer. It represents the alignment of the dest (and source), and so must be the minimum of the actual alignment of the two. This change is the first in a series that allows source and dest to each have their own alignments by using the alignment attribute on their arguments. In this change we: 1) Remove the alignment argument. 2) Add alignment attributes to the source & dest arguments. We, temporarily, require that the alignments for source & dest be equal. For example, code which used to read: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 100, i32 4, i1 false) will now read call void @llvm.memcpy.p0i8.p0i8.i32(i8* align 4 %dest, i8* align 4 %src, i32 100, i1 false) Downstream users may have to update their lit tests that check for @llvm.memcpy/memmove/memset call/declaration patterns. The following extended sed script may help with updating the majority of your tests, but it does not catch all possible patterns so some manual checking and updating will be required. s~declare void @llvm\.mem(set|cpy|move)\.p([^(]*)\((.*), i32, i1\)~declare void @llvm.mem\1.p\2(\3, i1)~g s~call void @llvm\.memset\.p([^(]*)i8\(i8([^*]*)\* (.*), i8 (.*), i8 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i8(i8\2* \3, i8 \4, i8 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i16\(i8([^*]*)\* (.*), i8 (.*), i16 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i16(i8\2* \3, i8 \4, i16 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i32\(i8([^*]*)\* (.*), i8 (.*), i32 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i32(i8\2* \3, i8 \4, i32 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i64\(i8([^*]*)\* (.*), i8 (.*), i64 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i64(i8\2* \3, i8 \4, i64 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i128\(i8([^*]*)\* (.*), i8 (.*), i128 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i128(i8\2* \3, i8 \4, i128 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i8\(i8([^*]*)\* (.*), i8 (.*), i8 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i8(i8\2* align \6 \3, i8 \4, i8 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i16\(i8([^*]*)\* (.*), i8 (.*), i16 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i16(i8\2* align \6 \3, i8 \4, i16 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i32\(i8([^*]*)\* (.*), i8 (.*), i32 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i32(i8\2* align \6 \3, i8 \4, i32 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i64\(i8([^*]*)\* (.*), i8 (.*), i64 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i64(i8\2* align \6 \3, i8 \4, i64 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i128\(i8([^*]*)\* (.*), i8 (.*), i128 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i128(i8\2* align \6 \3, i8 \4, i128 \5, i1 \7)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i8\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i8 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i8(i8\3* \4, i8\5* \6, i8 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i16\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i16 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i16(i8\3* \4, i8\5* \6, i16 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i32\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i32 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i32(i8\3* \4, i8\5* \6, i32 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i64\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i64 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i64(i8\3* \4, i8\5* \6, i64 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i128\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i128 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i128(i8\3* \4, i8\5* \6, i128 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i8\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i8 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i8(i8\3* align \8 \4, i8\5* align \8 \6, i8 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i16\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i16 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i16(i8\3* align \8 \4, i8\5* align \8 \6, i16 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i32\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i32 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i32(i8\3* align \8 \4, i8\5* align \8 \6, i32 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i64\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i64 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i64(i8\3* align \8 \4, i8\5* align \8 \6, i64 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i128\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i128 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i128(i8\3* align \8 \4, i8\5* align \8 \6, i128 \7, i1 \9)~g The remaining changes in the series will: Step 2) Expand the IRBuilder API to allow creation of memcpy/memmove with differing source and dest alignments. Step 3) Update Clang to use the new IRBuilder API. Step 4) Update Polly to use the new IRBuilder API. Step 5) Update LLVM passes that create memcpy/memmove calls to use the new IRBuilder API, and those that use use MemIntrinsicInst::[get|set]Alignment() to use getDestAlignment() and getSourceAlignment() instead. Step 6) Remove the single-alignment IRBuilder API for memcpy/memmove, and the MemIntrinsicInst::[get|set]Alignment() methods. Reviewers: pete, hfinkel, lhames, reames, bollu Reviewed By: reames Subscribers: niosHD, reames, jholewinski, qcolombet, jfb, sanjoy, arsenm, dschuff, dylanmckay, mehdi_amini, sdardis, nemanjai, david2050, nhaehnle, javed.absar, sbc100, jgravelle-google, eraman, aheejin, kbarton, JDevlieghere, asb, rbar, johnrusso, simoncook, jordy.potman.lists, apazos, sabuasal, llvm-commits Differential Revision: https://reviews.llvm.org/D41675 llvm-svn: 322965
2018-01-19 18:13:12 +01:00
// Updating the memory intrinsics (memcpy/memmove/memset) that have an
// alignment parameter to embedding the alignment as an attribute of
// the pointer args.
if (Name.startswith("memcpy.") && F->arg_size() == 5) {
rename(F);
// Get the types of dest, src, and len
ArrayRef<Type *> ParamTypes = F->getFunctionType()->params().slice(0, 3);
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::memcpy,
ParamTypes);
return true;
}
if (Name.startswith("memmove.") && F->arg_size() == 5) {
rename(F);
// Get the types of dest, src, and len
ArrayRef<Type *> ParamTypes = F->getFunctionType()->params().slice(0, 3);
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::memmove,
ParamTypes);
return true;
}
if (Name.startswith("memset.") && F->arg_size() == 5) {
rename(F);
// Get the types of dest, and len
const auto *FT = F->getFunctionType();
Type *ParamTypes[2] = {
FT->getParamType(0), // Dest
FT->getParamType(2) // len
};
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::memset,
ParamTypes);
return true;
}
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-28 20:27:25 +02:00
break;
}
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 02:00:32 +01:00
case 'n': {
if (Name.startswith("nvvm.")) {
Name = Name.substr(5);
// The following nvvm intrinsics correspond exactly to an LLVM intrinsic.
Intrinsic::ID IID = StringSwitch<Intrinsic::ID>(Name)
.Cases("brev32", "brev64", Intrinsic::bitreverse)
.Case("clz.i", Intrinsic::ctlz)
.Case("popc.i", Intrinsic::ctpop)
.Default(Intrinsic::not_intrinsic);
if (IID != Intrinsic::not_intrinsic && F->arg_size() == 1) {
NewFn = Intrinsic::getDeclaration(F->getParent(), IID,
{F->getReturnType()});
return true;
}
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-28 20:27:25 +02:00
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 02:00:32 +01:00
// The following nvvm intrinsics correspond exactly to an LLVM idiom, but
// not to an intrinsic alone. We expand them in UpgradeIntrinsicCall.
//
// TODO: We could add lohi.i2d.
bool Expand = StringSwitch<bool>(Name)
.Cases("abs.i", "abs.ll", true)
.Cases("clz.ll", "popc.ll", "h2f", true)
.Cases("max.i", "max.ll", "max.ui", "max.ull", true)
.Cases("min.i", "min.ll", "min.ui", "min.ull", true)
[NVPTX] Use atomicrmw fadd instead of intrinsics AutoUpgrade the old intrinsics to atomicrmw fadd. llvm-svn: 365796
2019-07-11 19:11:25 +02:00
.StartsWith("atomic.load.add.f32.p", true)
.StartsWith("atomic.load.add.f64.p", true)
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 02:00:32 +01:00
.Default(false);
if (Expand) {
NewFn = nullptr;
return true;
}
}
Added missing break. llvm-svn: 303454
2017-05-19 22:31:51 +02:00
break;
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 02:00:32 +01:00
}
Change objectsize intrinsic to accept different address spaces. Bitcasting everything to i8* won't work. Autoupgrade the old intrinsic declarations to use the new mangling. llvm-svn: 192117
2013-10-07 20:06:48 +02:00
case 'o':
// We only need to change the name to match the mangling including the
// address space.
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-21 21:08:59 +01:00
if (Name.startswith("objectsize.")) {
Change objectsize intrinsic to accept different address spaces. Bitcasting everything to i8* won't work. Autoupgrade the old intrinsic declarations to use the new mangling. llvm-svn: 192117
2013-10-07 20:06:48 +02:00
Type *Tys[2] = { F->getReturnType(), F->arg_begin()->getType() };
Fix auto-upgrade for the new parameter to llvm.objectsize r352664 added a 'dynamic' parameter to objectsize, but the AutoUpgrade changes were incomplete. Also, fix an off-by-one error I made in the upgrade logic that is now no longer unreachable. Differential revision: https://reviews.llvm.org/D58071 llvm-svn: 353884
2019-02-12 22:55:38 +01:00
if (F->arg_size() == 2 || F->arg_size() == 3 ||
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-21 21:08:59 +01:00
F->getName() != Intrinsic::getName(Intrinsic::objectsize, Tys)) {
Don't drop the llvm. prefix when renaming. If the llvm. prefix is dropped other parts of llvm don't see this as an intrinsic. This means that the number of regular symbols depends on the context the module is loaded into, which causes LTO to abort. Fixes PR30509. llvm-svn: 283117
2016-10-03 17:51:42 +02:00
rename(F);
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-21 21:08:59 +01:00
NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::objectsize,
Tys);
Change objectsize intrinsic to accept different address spaces. Bitcasting everything to i8* won't work. Autoupgrade the old intrinsic declarations to use the new mangling. llvm-svn: 192117
2013-10-07 20:06:48 +02:00
return true;
}
}
break;
Allow prefetching from non-zero address spaces Summary: This is useful for targets which have prefetch instructions for non-default address spaces. <rdar://problem/42662136> Subscribers: nemanjai, javed.absar, hiraditya, kbarton, jkorous, dexonsmith, cfe-commits, llvm-commits, RKSimon, hfinkel, t.p.northover, craig.topper, anemet Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D65254 llvm-svn: 367032
2019-07-25 18:11:57 +02:00
case 'p':
if (Name == "prefetch") {
// Handle address space overloading.
Type *Tys[] = {F->arg_begin()->getType()};
if (F->getName() != Intrinsic::getName(Intrinsic::prefetch, Tys)) {
rename(F);
NewFn =
Intrinsic::getDeclaration(F->getParent(), Intrinsic::prefetch, Tys);
return true;
}
}
break;
[SSP] Remove llvm.stackprotectorcheck. This is a cleanup patch for SSP support in LLVM. There is no functional change. llvm.stackprotectorcheck is not needed, because SelectionDAG isn't actually lowering it in SelectBasicBlock; rather, it adds check code in FinishBasicBlock, ignoring the position where the intrinsic is inserted (See FindSplitPointForStackProtector()). llvm-svn: 265851
2016-04-08 23:26:31 +02:00
case 's':
if (Name == "stackprotectorcheck") {
NewFn = nullptr;
return true;
}
Add a missing break statement. NFC. llvm-svn: 286203
2016-11-08 05:01:50 +01:00
break;
[SSP] Remove llvm.stackprotectorcheck. This is a cleanup patch for SSP support in LLVM. There is no functional change. llvm.stackprotectorcheck is not needed, because SelectionDAG isn't actually lowering it in SelectBasicBlock; rather, it adds check code in FinishBasicBlock, ignoring the position where the intrinsic is inserted (See FindSplitPointForStackProtector()). llvm-svn: 265851
2016-04-08 23:26:31 +02:00
[IR][X86] Move X86 specific portions of UpgradeIntrinsicFunction1 to a couple helper functions. NFC This enables some early outs to avoid repeatedly using IsX86 check to qualify. I hope to continue to improve this to shorten the lengths of some of the string comparisons. llvm-svn: 295424
2017-02-17 08:07:19 +01:00
case 'x':
if (UpgradeX86IntrinsicFunction(F, Name, NewFn))
Fix intrinsics for XOP frczss/sd instructions. These instructions only take one source register and zero the upper bits of the destination rather than preserving them. llvm-svn: 158396
2012-06-13 09:18:53 +02:00
return true;
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
}
Implement intrinsic mangling for literal struct types. Fixes PR 31921 Summary: Predicateinfo requires an ugly workaround to try to avoid literal struct types due to the intrinsic mangling not being implemented. This workaround actually does not work in all cases (you can hit the assert by bootstrapping with -print-predicateinfo), and can't be made to work without DFS'ing the type (IE copying getMangledStr and using a version that detects if it would crash). Rather than do that, i just implemented the mangling. It seems simple, since they are unified structurally. Looking at the overloaded-mangling testcase we have, it actually turns out the gc intrinsics will *also* crash if you try to use a literal struct. Thus, the testcase added fails before this patch, and works after, without needing to resort to predicateinfo. Reviewers: chandlerc, davide Subscribers: llvm-commits, sanjoy Differential Revision: https://reviews.llvm.org/D29925 llvm-svn: 295253
2017-02-16 00:16:20 +01:00
// Remangle our intrinsic since we upgrade the mangling
auto Result = llvm::Intrinsic::remangleIntrinsicFunction(F);
if (Result != None) {
NewFn = Result.getValue();
return true;
}
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-10 20:42:51 +02:00
// This may not belong here. This function is effectively being overloaded
// to both detect an intrinsic which needs upgrading, and to provide the
// upgraded form of the intrinsic. We should perhaps have two separate
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
// functions for this.
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-17 23:33:23 +01:00
return false;
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
}
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-17 23:33:23 +01:00
bool llvm::UpgradeIntrinsicFunction(Function *F, Function *&NewFn) {
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 08:08:46 +02:00
NewFn = nullptr;
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-17 23:33:23 +01:00
bool Upgraded = UpgradeIntrinsicFunction1(F, NewFn);
Remove always-true comparison, NFC. Summary: Looking at r241279, I noticed that UpgradedIntrinsics only gets written to in the following code: if (UpgradeIntrinsicFunction(&F, NewFn)) UpgradedIntrinsics[&F] = NewFn; Looking through UpgradeIntrinsicFunction, we always return false OR NewFn will be set to a different function from our source. This patch pulls the F != NewFn into UpgradeIntrinsicFunction as an assert, and removes the check from callers of UpgradeIntrinsicFunction. Reviewers: rafael, chandlerc Subscribers: llvm-commits-list Differential Revision: http://reviews.llvm.org/D10915 llvm-svn: 241369
2015-07-03 22:12:01 +02:00
assert(F != NewFn && "Intrinsic function upgraded to the same function");
Rather than having special rules like "intrinsics cannot throw exceptions", just mark intrinsics with the nounwind attribute. Likewise, mark intrinsics as readnone/readonly and get rid of special aliasing logic (which didn't use anything more than this anyway). llvm-svn: 44544
2007-12-03 21:06:50 +01:00
// Upgrade intrinsic attributes. This does not change the function.
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-17 23:33:23 +01:00
if (NewFn)
F = NewFn;
Change Function::getIntrinsicID() to return an Intrinsic::ID. NFC. Now that Intrinsic::ID is a typed enum, we can forward declare it and so return it from this method. This updates all users which were either using an unsigned to store it, or had a now unnecessary cast. llvm-svn: 237810
2015-05-20 19:16:39 +02:00
if (Intrinsic::ID id = F->getIntrinsicID())
F->setAttributes(Intrinsic::getAttributes(F->getContext(), id));
Rather than having special rules like "intrinsics cannot throw exceptions", just mark intrinsics with the nounwind attribute. Likewise, mark intrinsics as readnone/readonly and get rid of special aliasing logic (which didn't use anything more than this anyway). llvm-svn: 44544
2007-12-03 21:06:50 +01:00
return Upgraded;
}
[IR] Disallow llvm.global_ctors and llvm.global_dtors of the 2-field form in textual format The 3-field form was introduced by D3499 in 2014 and the legacy 2-field form was planned to be removed in LLVM 4.0 For the textual format, this patch migrates the existing 2-field form to use the 3-field form and deletes the compatibility code. test/Verifier/global-ctors-2.ll checks we have a friendly error message. For bitcode, lib/IR/AutoUpgrade UpgradeGlobalVariables will upgrade the 2-field form (add i8* null as the third field). Reviewed By: rnk, dexonsmith Differential Revision: https://reviews.llvm.org/D61547 llvm-svn: 360742
2019-05-15 04:35:32 +02:00
GlobalVariable *llvm::UpgradeGlobalVariable(GlobalVariable *GV) {
if (!(GV->hasName() && (GV->getName() == "llvm.global_ctors" ||
GV->getName() == "llvm.global_dtors")) ||
!GV->hasInitializer())
return nullptr;
ArrayType *ATy = dyn_cast<ArrayType>(GV->getValueType());
if (!ATy)
return nullptr;
StructType *STy = dyn_cast<StructType>(ATy->getElementType());
if (!STy || STy->getNumElements() != 2)
return nullptr;
LLVMContext &C = GV->getContext();
IRBuilder<> IRB(C);
auto EltTy = StructType::get(STy->getElementType(0), STy->getElementType(1),
IRB.getInt8PtrTy());
Constant *Init = GV->getInitializer();
unsigned N = Init->getNumOperands();
std::vector<Constant *> NewCtors(N);
for (unsigned i = 0; i != N; ++i) {
auto Ctor = cast<Constant>(Init->getOperand(i));
NewCtors[i] = ConstantStruct::get(
EltTy, Ctor->getAggregateElement(0u), Ctor->getAggregateElement(1),
Constant::getNullValue(IRB.getInt8PtrTy()));
}
Constant *NewInit = ConstantArray::get(ArrayType::get(EltTy, N), NewCtors);
return new GlobalVariable(NewInit->getType(), false, GV->getLinkage(),
NewInit, GV->getName());
Auto-upgrade the magic ".llvm.eh.catch.all.value" global to "llvm.eh.catch.all.value". Only the name needs to be changed. llvm-svn: 113600
2010-09-10 20:51:56 +02:00
}
[X86][AVX512] Dropped avx512 VPSLLDQ/VPSRLDQ intrinsics Auto-upgrade to generic shuffles like sse/avx2 implementations now that we can lower to VPSLLDQ/VPSRLDQ llvm-svn: 272308
2016-06-09 23:09:03 +02:00
// Handles upgrading SSE2/AVX2/AVX512BW PSLLDQ intrinsics by converting them
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
// to byte shuffles.
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
static Value *UpgradeX86PSLLDQIntrinsics(IRBuilder<> &Builder,
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 08:37:33 +02:00
Value *Op, unsigned Shift) {
Type *ResultTy = Op->getType();
unsigned NumElts = ResultTy->getVectorNumElements() * 8;
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
// Bitcast from a 64-bit element type to a byte element type.
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
Type *VecTy = VectorType::get(Builder.getInt8Ty(), NumElts);
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 08:37:33 +02:00
Op = Builder.CreateBitCast(Op, VecTy, "cast");
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
// We'll be shuffling in zeroes.
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 08:37:33 +02:00
Value *Res = Constant::getNullValue(VecTy);
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
// If shift is less than 16, emit a shuffle to move the bytes. Otherwise,
// we'll just return the zero vector.
if (Shift < 16) {
[IR] Require ArrayRef of 'uint32_t' instead of 'int' for the mask argument for one of the signatures of CreateShuffleVector. This better emphasises that you can't use it for the -1 as undef behavior. llvm-svn: 272491
2016-06-12 02:41:19 +02:00
uint32_t Idxs[64];
[X86][AVX512] Dropped avx512 VPSLLDQ/VPSRLDQ intrinsics Auto-upgrade to generic shuffles like sse/avx2 implementations now that we can lower to VPSLLDQ/VPSRLDQ llvm-svn: 272308
2016-06-09 23:09:03 +02:00
// 256/512-bit version is split into 2/4 16-byte lanes.
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
for (unsigned l = 0; l != NumElts; l += 16)
for (unsigned i = 0; i != 16; ++i) {
unsigned Idx = NumElts + i - Shift;
if (Idx < NumElts)
Idx -= NumElts - 16; // end of lane, switch operand.
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 08:37:33 +02:00
Idxs[l + i] = Idx + l;
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
}
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 08:37:33 +02:00
Res = Builder.CreateShuffleVector(Res, Op, makeArrayRef(Idxs, NumElts));
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
}
// Bitcast back to a 64-bit element type.
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 08:37:33 +02:00
return Builder.CreateBitCast(Res, ResultTy, "cast");
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
}
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 04:36:42 +02:00
// Handles upgrading SSE2/AVX2/AVX512BW PSRLDQ intrinsics by converting them
// to byte shuffles.
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
static Value *UpgradeX86PSRLDQIntrinsics(IRBuilder<> &Builder, Value *Op,
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 04:36:42 +02:00
unsigned Shift) {
Type *ResultTy = Op->getType();
unsigned NumElts = ResultTy->getVectorNumElements() * 8;
// Bitcast from a 64-bit element type to a byte element type.
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
Type *VecTy = VectorType::get(Builder.getInt8Ty(), NumElts);
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 04:36:42 +02:00
Op = Builder.CreateBitCast(Op, VecTy, "cast");
// We'll be shuffling in zeroes.
Value *Res = Constant::getNullValue(VecTy);
// If shift is less than 16, emit a shuffle to move the bytes. Otherwise,
// we'll just return the zero vector.
if (Shift < 16) {
uint32_t Idxs[64];
// 256/512-bit version is split into 2/4 16-byte lanes.
for (unsigned l = 0; l != NumElts; l += 16)
for (unsigned i = 0; i != 16; ++i) {
unsigned Idx = i + Shift;
if (Idx >= 16)
Idx += NumElts - 16; // end of lane, switch operand.
Idxs[l + i] = Idx + l;
}
Res = Builder.CreateShuffleVector(Op, Res, makeArrayRef(Idxs, NumElts));
}
// Bitcast back to a 64-bit element type.
return Builder.CreateBitCast(Res, ResultTy, "cast");
}
static Value *getX86MaskVec(IRBuilder<> &Builder, Value *Mask,
unsigned NumElts) {
llvm::VectorType *MaskTy = llvm::VectorType::get(Builder.getInt1Ty(),
cast<IntegerType>(Mask->getType())->getBitWidth());
Mask = Builder.CreateBitCast(Mask, MaskTy);
// If we have less than 8 elements, then the starting mask was an i8 and
// we need to extract down to the right number of elements.
if (NumElts < 8) {
uint32_t Indices[4];
for (unsigned i = 0; i != NumElts; ++i)
Indices[i] = i;
Mask = Builder.CreateShuffleVector(Mask, Mask,
makeArrayRef(Indices, NumElts),
"extract");
}
return Mask;
}
static Value *EmitX86Select(IRBuilder<> &Builder, Value *Mask,
Value *Op0, Value *Op1) {
[X86] Remove and autoupgrade the scalar fma intrinsics with masking. This converts them to what clang is now using for codegen. Unfortunately, there seem to be a few kinks to work out still. I'll try to address with follow up patches. llvm-svn: 336871
2018-07-12 02:29:56 +02:00
// If the mask is all ones just emit the first operation.
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 04:36:42 +02:00
if (const auto *C = dyn_cast<Constant>(Mask))
if (C->isAllOnesValue())
return Op0;
Mask = getX86MaskVec(Builder, Mask, Op0->getType()->getVectorNumElements());
return Builder.CreateSelect(Mask, Op0, Op1);
}
[X86] Remove and autoupgrade the scalar fma intrinsics with masking. This converts them to what clang is now using for codegen. Unfortunately, there seem to be a few kinks to work out still. I'll try to address with follow up patches. llvm-svn: 336871
2018-07-12 02:29:56 +02:00
static Value *EmitX86ScalarSelect(IRBuilder<> &Builder, Value *Mask,
Value *Op0, Value *Op1) {
// If the mask is all ones just emit the first operation.
if (const auto *C = dyn_cast<Constant>(Mask))
if (C->isAllOnesValue())
return Op0;
llvm::VectorType *MaskTy =
llvm::VectorType::get(Builder.getInt1Ty(),
Mask->getType()->getIntegerBitWidth());
Mask = Builder.CreateBitCast(Mask, MaskTy);
Mask = Builder.CreateExtractElement(Mask, (uint64_t)0);
return Builder.CreateSelect(Mask, Op0, Op1);
}
[AVX-512] Remove intrinsics for valignd/q and autoupgrade them to native shuffles. llvm-svn: 287744
2016-11-23 07:54:55 +01:00
// Handle autoupgrade for masked PALIGNR and VALIGND/Q intrinsics.
// PALIGNR handles large immediates by shifting while VALIGN masks the immediate
// so we need to handle both cases. VALIGN also doesn't have 128-bit lanes.
static Value *UpgradeX86ALIGNIntrinsics(IRBuilder<> &Builder, Value *Op0,
Value *Op1, Value *Shift,
Value *Passthru, Value *Mask,
bool IsVALIGN) {
[AVX512] Remove masked palignr intrinsics and auto-upgrade them to native IR of vector shuffle and select. llvm-svn: 271872
2016-06-06 08:12:54 +02:00
unsigned ShiftVal = cast<llvm::ConstantInt>(Shift)->getZExtValue();
unsigned NumElts = Op0->getType()->getVectorNumElements();
[AVX-512] Remove intrinsics for valignd/q and autoupgrade them to native shuffles. llvm-svn: 287744
2016-11-23 07:54:55 +01:00
assert((IsVALIGN || NumElts % 16 == 0) && "Illegal NumElts for PALIGNR!");
assert((!IsVALIGN || NumElts <= 16) && "NumElts too large for VALIGN!");
assert(isPowerOf2_32(NumElts) && "NumElts not a power of 2!");
// Mask the immediate for VALIGN.
if (IsVALIGN)
ShiftVal &= (NumElts - 1);
[AVX512] Remove masked palignr intrinsics and auto-upgrade them to native IR of vector shuffle and select. llvm-svn: 271872
2016-06-06 08:12:54 +02:00
// If palignr is shifting the pair of vectors more than the size of two
// lanes, emit zero.
if (ShiftVal >= 32)
return llvm::Constant::getNullValue(Op0->getType());
// If palignr is shifting the pair of input vectors more than one lane,
// but less than two lanes, convert to shifting in zeroes.
if (ShiftVal > 16) {
ShiftVal -= 16;
Op1 = Op0;
Op0 = llvm::Constant::getNullValue(Op0->getType());
}
[IR] Require ArrayRef of 'uint32_t' instead of 'int' for the mask argument for one of the signatures of CreateShuffleVector. This better emphasises that you can't use it for the -1 as undef behavior. llvm-svn: 272491
2016-06-12 02:41:19 +02:00
uint32_t Indices[64];
[AVX512] Remove masked palignr intrinsics and auto-upgrade them to native IR of vector shuffle and select. llvm-svn: 271872
2016-06-06 08:12:54 +02:00
// 256-bit palignr operates on 128-bit lanes so we need to handle that
[AVX-512] Remove intrinsics for valignd/q and autoupgrade them to native shuffles. llvm-svn: 287744
2016-11-23 07:54:55 +01:00
for (unsigned l = 0; l < NumElts; l += 16) {
[AVX512] Remove masked palignr intrinsics and auto-upgrade them to native IR of vector shuffle and select. llvm-svn: 271872
2016-06-06 08:12:54 +02:00
for (unsigned i = 0; i != 16; ++i) {
unsigned Idx = ShiftVal + i;
[AVX-512] Remove intrinsics for valignd/q and autoupgrade them to native shuffles. llvm-svn: 287744
2016-11-23 07:54:55 +01:00
if (!IsVALIGN && Idx >= 16) // Disable wrap for VALIGN.
[AVX512] Remove masked palignr intrinsics and auto-upgrade them to native IR of vector shuffle and select. llvm-svn: 271872
2016-06-06 08:12:54 +02:00
Idx += NumElts - 16; // End of lane, switch operand.
Indices[l + i] = Idx + l;
}
}
Value *Align = Builder.CreateShuffleVector(Op1, Op0,
makeArrayRef(Indices, NumElts),
"palignr");
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 04:36:42 +02:00
return EmitX86Select(Builder, Mask, Align, Passthru);
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
}
[X86][AutoUpgrade] Make some tweaks to reduce the number of nested if/else in the intrinsic upgrade code to avoid an MSVC compiler limit. MSVC has a nesting limit of around 110-130. An if/else if/else if counts against this next level. The autoupgrade code consists a long chain of these checking matches against strings. This commit moves some code to a helper function to move out a large if/else chain that was inside of one of the blocks into a separate function. There are more of these we could move or we could change some to lookup tables. I've also merged together a few similar blocks in the outer chain. This should buy us some margin for a little bit. llvm-svn: 350564
2019-01-07 21:13:45 +01:00
static Value *UpgradeX86VPERMT2Intrinsics(IRBuilder<> &Builder, CallInst &CI,
bool ZeroMask, bool IndexForm) {
Type *Ty = CI.getType();
unsigned VecWidth = Ty->getPrimitiveSizeInBits();
unsigned EltWidth = Ty->getScalarSizeInBits();
bool IsFloat = Ty->isFPOrFPVectorTy();
Intrinsic::ID IID;
if (VecWidth == 128 && EltWidth == 32 && IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_ps_128;
else if (VecWidth == 128 && EltWidth == 32 && !IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_d_128;
else if (VecWidth == 128 && EltWidth == 64 && IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_pd_128;
else if (VecWidth == 128 && EltWidth == 64 && !IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_q_128;
else if (VecWidth == 256 && EltWidth == 32 && IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_ps_256;
else if (VecWidth == 256 && EltWidth == 32 && !IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_d_256;
else if (VecWidth == 256 && EltWidth == 64 && IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_pd_256;
else if (VecWidth == 256 && EltWidth == 64 && !IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_q_256;
else if (VecWidth == 512 && EltWidth == 32 && IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_ps_512;
else if (VecWidth == 512 && EltWidth == 32 && !IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_d_512;
else if (VecWidth == 512 && EltWidth == 64 && IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_pd_512;
else if (VecWidth == 512 && EltWidth == 64 && !IsFloat)
IID = Intrinsic::x86_avx512_vpermi2var_q_512;
else if (VecWidth == 128 && EltWidth == 16)
IID = Intrinsic::x86_avx512_vpermi2var_hi_128;
else if (VecWidth == 256 && EltWidth == 16)
IID = Intrinsic::x86_avx512_vpermi2var_hi_256;
else if (VecWidth == 512 && EltWidth == 16)
IID = Intrinsic::x86_avx512_vpermi2var_hi_512;
else if (VecWidth == 128 && EltWidth == 8)
IID = Intrinsic::x86_avx512_vpermi2var_qi_128;
else if (VecWidth == 256 && EltWidth == 8)
IID = Intrinsic::x86_avx512_vpermi2var_qi_256;
else if (VecWidth == 512 && EltWidth == 8)
IID = Intrinsic::x86_avx512_vpermi2var_qi_512;
else
llvm_unreachable("Unexpected intrinsic");
Value *Args[] = { CI.getArgOperand(0) , CI.getArgOperand(1),
CI.getArgOperand(2) };
// If this isn't index form we need to swap operand 0 and 1.
if (!IndexForm)
std::swap(Args[0], Args[1]);
Value *V = Builder.CreateCall(Intrinsic::getDeclaration(CI.getModule(), IID),
Args);
Value *PassThru = ZeroMask ? ConstantAggregateZero::get(Ty)
: Builder.CreateBitCast(CI.getArgOperand(1),
Ty);
return EmitX86Select(Builder, CI.getArgOperand(3), V, PassThru);
}
[X86] Lowering addus/subus intrinsics to native IR Summary: This revision improves previous version (rL330322) which has been reverted due to crashes. This is the patch that lowers x86 intrinsics to native IR in order to enable optimizations. The patch also includes folding of previously missing saturation patterns so that IR emits the same machine instructions as the intrinsics. Reviewers: craig.topper, spatel, RKSimon Reviewed By: craig.topper Subscribers: mike.dvoretsky, DavidKreitzer, sroland, llvm-commits Differential Revision: https://reviews.llvm.org/D46179 llvm-svn: 339650
2018-08-14 10:00:56 +02:00
static Value *UpgradeX86AddSubSatIntrinsics(IRBuilder<> &Builder, CallInst &CI,
[X86][SSE] Auto upgrade PADDS/PSUBS intrinsics to SADD_SAT/SSUB_SAT generic intrinsics (llvm) This auto upgrades the signed SSE saturated math intrinsics to SADD_SAT/SSUB_SAT generic intrinsics. Clang counterpart: https://reviews.llvm.org/D55890 Differential Revision: https://reviews.llvm.org/D55894 llvm-svn: 349892
2018-12-21 10:04:14 +01:00
bool IsSigned, bool IsAddition) {
[X86][SSE] Auto upgrade PADDUS/PSUBUS intrinsics to UADD_SAT/USUB_SAT generic intrinsics (llvm) Now that we use the generic ISD opcodes, we can use the generic intrinsics directly as well. This fixes the poor fast-isel codegen by not expanding to an easily broken IR code sequence. I'm intending to deal with the signed saturation equivalents as well. Clang counterpart: https://reviews.llvm.org/D55879 Differential Revision: https://reviews.llvm.org/D55855 llvm-svn: 349630
2018-12-19 15:43:36 +01:00
Type *Ty = CI.getType();
[X86] Lowering addus/subus intrinsics to native IR Summary: This revision improves previous version (rL330322) which has been reverted due to crashes. This is the patch that lowers x86 intrinsics to native IR in order to enable optimizations. The patch also includes folding of previously missing saturation patterns so that IR emits the same machine instructions as the intrinsics. Reviewers: craig.topper, spatel, RKSimon Reviewed By: craig.topper Subscribers: mike.dvoretsky, DavidKreitzer, sroland, llvm-commits Differential Revision: https://reviews.llvm.org/D46179 llvm-svn: 339650
2018-08-14 10:00:56 +02:00
Value *Op0 = CI.getOperand(0);
Value *Op1 = CI.getOperand(1);
[X86][SSE] Auto upgrade PADDS/PSUBS intrinsics to SADD_SAT/SSUB_SAT generic intrinsics (llvm) This auto upgrades the signed SSE saturated math intrinsics to SADD_SAT/SSUB_SAT generic intrinsics. Clang counterpart: https://reviews.llvm.org/D55890 Differential Revision: https://reviews.llvm.org/D55894 llvm-svn: 349892
2018-12-21 10:04:14 +01:00
Intrinsic::ID IID =
IsSigned ? (IsAddition ? Intrinsic::sadd_sat : Intrinsic::ssub_sat)
: (IsAddition ? Intrinsic::uadd_sat : Intrinsic::usub_sat);
[X86][SSE] Auto upgrade PADDUS/PSUBUS intrinsics to UADD_SAT/USUB_SAT generic intrinsics (llvm) Now that we use the generic ISD opcodes, we can use the generic intrinsics directly as well. This fixes the poor fast-isel codegen by not expanding to an easily broken IR code sequence. I'm intending to deal with the signed saturation equivalents as well. Clang counterpart: https://reviews.llvm.org/D55879 Differential Revision: https://reviews.llvm.org/D55855 llvm-svn: 349630
2018-12-19 15:43:36 +01:00
Function *Intrin = Intrinsic::getDeclaration(CI.getModule(), IID, Ty);
Value *Res = Builder.CreateCall(Intrin, {Op0, Op1});
[X86] Lowering addus/subus intrinsics to native IR Summary: This revision improves previous version (rL330322) which has been reverted due to crashes. This is the patch that lowers x86 intrinsics to native IR in order to enable optimizations. The patch also includes folding of previously missing saturation patterns so that IR emits the same machine instructions as the intrinsics. Reviewers: craig.topper, spatel, RKSimon Reviewed By: craig.topper Subscribers: mike.dvoretsky, DavidKreitzer, sroland, llvm-commits Differential Revision: https://reviews.llvm.org/D46179 llvm-svn: 339650
2018-08-14 10:00:56 +02:00
if (CI.getNumArgOperands() == 4) { // For masked intrinsics.
Value *VecSrc = CI.getOperand(2);
Value *Mask = CI.getOperand(3);
Res = EmitX86Select(Builder, Mask, Res, VecSrc);
}
return Res;
}
[X86] Auto upgrade XOP/AVX512 rotation intrinsics to generic funnel shift intrinsics (llvm) This emits FSHL/FSHR generic intrinsics for the XOP VPROT and AVX512 VPROL/VPROR rotation intrinsics. Clang counterpart: https://reviews.llvm.org/D55937 Differential Revision: https://reviews.llvm.org/D55938 llvm-svn: 349795
2018-12-20 20:01:07 +01:00
static Value *upgradeX86Rotate(IRBuilder<> &Builder, CallInst &CI,
bool IsRotateRight) {
Type *Ty = CI.getType();
Value *Src = CI.getArgOperand(0);
Value *Amt = CI.getArgOperand(1);
// Amount may be scalar immediate, in which case create a splat vector.
// Funnel shifts amounts are treated as modulo and types are all power-of-2 so
// we only care about the lowest log2 bits anyway.
if (Amt->getType() != Ty) {
unsigned NumElts = Ty->getVectorNumElements();
Amt = Builder.CreateIntCast(Amt, Ty->getScalarType(), false);
Amt = Builder.CreateVectorSplat(NumElts, Amt);
}
Intrinsic::ID IID = IsRotateRight ? Intrinsic::fshr : Intrinsic::fshl;
Function *Intrin = Intrinsic::getDeclaration(CI.getModule(), IID, Ty);
Value *Res = Builder.CreateCall(Intrin, {Src, Src, Amt});
if (CI.getNumArgOperands() == 4) { // For masked intrinsics.
Value *VecSrc = CI.getOperand(2);
Value *Mask = CI.getOperand(3);
Res = EmitX86Select(Builder, Mask, Res, VecSrc);
}
return Res;
}
[X86] Auto upgrade old style VPCOM/VPCOMU intrinsics to generic integer comparisons We were upgrading these to the new style VPCOM/VPCOMU intrinsics (which includes the condition code immediate), but we'll be getting rid of those shortly, so convert these to generics first. This causes a couple of changes in the upgrade tests as signed/unsigned eq/ne are equivalent and we constant fold true/false codes, these changes are the same as what we already do for avx512 cmp/ucmp. Noticed while cleaning up vector integer comparison costs for PR40376. llvm-svn: 351690
2019-01-20 18:36:22 +01:00
static Value *upgradeX86vpcom(IRBuilder<> &Builder, CallInst &CI, unsigned Imm,
bool IsSigned) {
Type *Ty = CI.getType();
Value *LHS = CI.getArgOperand(0);
Value *RHS = CI.getArgOperand(1);
CmpInst::Predicate Pred;
switch (Imm) {
case 0x0:
Pred = IsSigned ? ICmpInst::ICMP_SLT : ICmpInst::ICMP_ULT;
break;
case 0x1:
Pred = IsSigned ? ICmpInst::ICMP_SLE : ICmpInst::ICMP_ULE;
break;
case 0x2:
Pred = IsSigned ? ICmpInst::ICMP_SGT : ICmpInst::ICMP_UGT;
break;
case 0x3:
Pred = IsSigned ? ICmpInst::ICMP_SGE : ICmpInst::ICMP_UGE;
break;
case 0x4:
Pred = ICmpInst::ICMP_EQ;
break;
case 0x5:
Pred = ICmpInst::ICMP_NE;
break;
case 0x6:
return Constant::getNullValue(Ty); // FALSE
case 0x7:
return Constant::getAllOnesValue(Ty); // TRUE
default:
llvm_unreachable("Unknown XOP vpcom/vpcomu predicate");
}
Value *Cmp = Builder.CreateICmp(Pred, LHS, RHS);
Value *Ext = Builder.CreateSExt(Cmp, Ty);
return Ext;
}
Recommit r350554 "[X86] Remove AVX512VBMI2 concat and shift intrinsics. Replace with target independent funnel shift intrinsics." The MSVC limit we hit on AutoUpgrade.cpp has been worked around for now. llvm-svn: 350567
2019-01-07 22:00:32 +01:00
static Value *upgradeX86ConcatShift(IRBuilder<> &Builder, CallInst &CI,
bool IsShiftRight, bool ZeroMask) {
Type *Ty = CI.getType();
Value *Op0 = CI.getArgOperand(0);
Value *Op1 = CI.getArgOperand(1);
Value *Amt = CI.getArgOperand(2);
if (IsShiftRight)
std::swap(Op0, Op1);
// Amount may be scalar immediate, in which case create a splat vector.
// Funnel shifts amounts are treated as modulo and types are all power-of-2 so
// we only care about the lowest log2 bits anyway.
if (Amt->getType() != Ty) {
unsigned NumElts = Ty->getVectorNumElements();
Amt = Builder.CreateIntCast(Amt, Ty->getScalarType(), false);
Amt = Builder.CreateVectorSplat(NumElts, Amt);
}
Intrinsic::ID IID = IsShiftRight ? Intrinsic::fshr : Intrinsic::fshl;
Function *Intrin = Intrinsic::getDeclaration(CI.getModule(), IID, Ty);
Value *Res = Builder.CreateCall(Intrin, {Op0, Op1, Amt});
unsigned NumArgs = CI.getNumArgOperands();
if (NumArgs >= 4) { // For masked intrinsics.
Value *VecSrc = NumArgs == 5 ? CI.getArgOperand(3) :
ZeroMask ? ConstantAggregateZero::get(CI.getType()) :
CI.getArgOperand(0);
Value *Mask = CI.getOperand(NumArgs - 1);
Res = EmitX86Select(Builder, Mask, Res, VecSrc);
}
return Res;
}
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
static Value *UpgradeMaskedStore(IRBuilder<> &Builder,
[AVX512] Remove masked store intrinsics. Clang now emits generic masked store intrinsics instead. The intrinsics will be autoupgraded to the same generic masked stores. llvm-svn: 271245
2016-05-31 03:50:02 +02:00
Value *Ptr, Value *Data, Value *Mask,
bool Aligned) {
// Cast the pointer to the right type.
Ptr = Builder.CreateBitCast(Ptr,
llvm::PointerType::getUnqual(Data->getType()));
[Alignment][NFC] Use Align with CreateMaskedStore Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet Subscribers: hiraditya, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73106
2020-01-21 16:13:04 +01:00
const Align Alignment =
Aligned ? Align(cast<VectorType>(Data->getType())->getBitWidth() / 8)
[Alignment][NFC] Deprecate Align::None() Summary: This is a follow up on https://reviews.llvm.org/D71473#inline-647262. There's a caveat here that `Align(1)` relies on the compiler understanding of `Log2_64` implementation to produce good code. One could use `Align()` as a replacement but I believe it is less clear that the alignment is one in that case. Reviewers: xbolva00, courbet, bollu Subscribers: arsenm, dylanmckay, sdardis, nemanjai, jvesely, nhaehnle, hiraditya, kbarton, jrtc27, atanasyan, jsji, Jim, kerbowa, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73099
2020-01-21 15:00:04 +01:00
: Align(1);
[AVX512] Remove masked store intrinsics. Clang now emits generic masked store intrinsics instead. The intrinsics will be autoupgraded to the same generic masked stores. llvm-svn: 271245
2016-05-31 03:50:02 +02:00
// If the mask is all ones just emit a regular store.
if (const auto *C = dyn_cast<Constant>(Mask))
if (C->isAllOnesValue())
[Alignment][NFC] Use Align with CreateMaskedStore Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet Subscribers: hiraditya, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73106
2020-01-21 16:13:04 +01:00
return Builder.CreateAlignedStore(Data, Ptr, Alignment);
[AVX512] Remove masked store intrinsics. Clang now emits generic masked store intrinsics instead. The intrinsics will be autoupgraded to the same generic masked stores. llvm-svn: 271245
2016-05-31 03:50:02 +02:00
// Convert the mask from an integer type to a vector of i1.
unsigned NumElts = Data->getType()->getVectorNumElements();
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 04:36:42 +02:00
Mask = getX86MaskVec(Builder, Mask, NumElts);
[Alignment][NFC] Use Align with CreateMaskedStore Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet Subscribers: hiraditya, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73106
2020-01-21 16:13:04 +01:00
return Builder.CreateMaskedStore(Data, Ptr, Alignment, Mask);
[AVX512] Remove masked store intrinsics. Clang now emits generic masked store intrinsics instead. The intrinsics will be autoupgraded to the same generic masked stores. llvm-svn: 271245
2016-05-31 03:50:02 +02:00
}
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
static Value *UpgradeMaskedLoad(IRBuilder<> &Builder,
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 06:19:36 +02:00
Value *Ptr, Value *Passthru, Value *Mask,
bool Aligned) {
[opaque pointer types] Pass value type to LoadInst creation. This cleans up all LoadInst creation in LLVM to explicitly pass the value type rather than deriving it from the pointer's element-type. Differential Revision: https://reviews.llvm.org/D57172 llvm-svn: 352911
2019-02-01 21:44:24 +01:00
Type *ValTy = Passthru->getType();
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 06:19:36 +02:00
// Cast the pointer to the right type.
[opaque pointer types] Pass value type to LoadInst creation. This cleans up all LoadInst creation in LLVM to explicitly pass the value type rather than deriving it from the pointer's element-type. Differential Revision: https://reviews.llvm.org/D57172 llvm-svn: 352911
2019-02-01 21:44:24 +01:00
Ptr = Builder.CreateBitCast(Ptr, llvm::PointerType::getUnqual(ValTy));
[Alignment][NFC] Use Align with CreateMaskedLoad Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet Subscribers: hiraditya, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73087
2020-01-21 11:21:31 +01:00
const Align Alignment =
Aligned ? Align(cast<VectorType>(Passthru->getType())->getBitWidth() / 8)
[Alignment][NFC] Deprecate Align::None() Summary: This is a follow up on https://reviews.llvm.org/D71473#inline-647262. There's a caveat here that `Align(1)` relies on the compiler understanding of `Log2_64` implementation to produce good code. One could use `Align()` as a replacement but I believe it is less clear that the alignment is one in that case. Reviewers: xbolva00, courbet, bollu Subscribers: arsenm, dylanmckay, sdardis, nemanjai, jvesely, nhaehnle, hiraditya, kbarton, jrtc27, atanasyan, jsji, Jim, kerbowa, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73099
2020-01-21 15:00:04 +01:00
: Align(1);
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 06:19:36 +02:00
// If the mask is all ones just emit a regular store.
if (const auto *C = dyn_cast<Constant>(Mask))
if (C->isAllOnesValue())
[Alignment][NFC] Use Align with CreateMaskedLoad Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet Subscribers: hiraditya, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73087
2020-01-21 11:21:31 +01:00
return Builder.CreateAlignedLoad(ValTy, Ptr, Alignment);
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 06:19:36 +02:00
// Convert the mask from an integer type to a vector of i1.
unsigned NumElts = Passthru->getType()->getVectorNumElements();
[X86] Refactor some of the X86 autoupgrade code to split mask vector and select generation into routines that can be reused for future intrinsic upgrades. NFC llvm-svn: 272526
2016-06-13 04:36:42 +02:00
Mask = getX86MaskVec(Builder, Mask, NumElts);
[Alignment][NFC] Use Align with CreateMaskedLoad Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet Subscribers: hiraditya, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73087
2020-01-21 11:21:31 +01:00
return Builder.CreateMaskedLoad(Ptr, Alignment, Mask, Passthru);
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 06:19:36 +02:00
}
[X86] [PATCH] [intrinsics] Lowering X86 ABS intrinsics to IR. (llvm) This patch, together with a matching clang patch (https://reviews.llvm.org/D37694), implements the lowering of X86 ABS intrinsics to IR. differential revision: https://reviews.llvm.org/D37693. llvm-svn: 313134
2017-09-13 11:02:36 +02:00
static Value *upgradeAbs(IRBuilder<> &Builder, CallInst &CI) {
Value *Op0 = CI.getArgOperand(0);
llvm::Type *Ty = Op0->getType();
Value *Zero = llvm::Constant::getNullValue(Ty);
Value *Cmp = Builder.CreateICmp(ICmpInst::ICMP_SGT, Op0, Zero);
Value *Neg = Builder.CreateNeg(Op0);
Value *Res = Builder.CreateSelect(Cmp, Op0, Neg);
if (CI.getNumArgOperands() == 3)
Res = EmitX86Select(Builder,CI.getArgOperand(2), Res, CI.getArgOperand(1));
return Res;
}
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 17:48:30 +02:00
static Value *upgradeIntMinMax(IRBuilder<> &Builder, CallInst &CI,
ICmpInst::Predicate Pred) {
Value *Op0 = CI.getArgOperand(0);
Value *Op1 = CI.getArgOperand(1);
Value *Cmp = Builder.CreateICmp(Pred, Op0, Op1);
[AVX-512] Remove masked pmin/pmax intrinsics and autoupgrade to native IR. Clang patch to replace 512-bit vector and 64-bit element versions with native IR will follow. llvm-svn: 284955
2016-10-24 06:04:16 +02:00
Value *Res = Builder.CreateSelect(Cmp, Op0, Op1);
if (CI.getNumArgOperands() == 4)
Res = EmitX86Select(Builder, CI.getArgOperand(3), Res, CI.getArgOperand(2));
return Res;
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 17:48:30 +02:00
}
[X86] Remove the pmuldq/pmuldq intrinsics and replace with native IR. This completes the work started in r329604 and r329605 when we changed clang to no longer use the intrinsics. We lost some InstCombine SimplifyDemandedBit optimizations through this change as we aren't able to fold 'and', bitcast, shuffle very well. llvm-svn: 329990
2018-04-13 08:07:18 +02:00
static Value *upgradePMULDQ(IRBuilder<> &Builder, CallInst &CI, bool IsSigned) {
Type *Ty = CI.getType();
// Arguments have a vXi32 type so cast to vXi64.
Value *LHS = Builder.CreateBitCast(CI.getArgOperand(0), Ty);
Value *RHS = Builder.CreateBitCast(CI.getArgOperand(1), Ty);
if (IsSigned) {
// Shift left then arithmetic shift right.
Constant *ShiftAmt = ConstantInt::get(Ty, 32);
LHS = Builder.CreateShl(LHS, ShiftAmt);
LHS = Builder.CreateAShr(LHS, ShiftAmt);
RHS = Builder.CreateShl(RHS, ShiftAmt);
RHS = Builder.CreateAShr(RHS, ShiftAmt);
} else {
// Clear the upper bits.
Constant *Mask = ConstantInt::get(Ty, 0xffffffff);
LHS = Builder.CreateAnd(LHS, Mask);
RHS = Builder.CreateAnd(RHS, Mask);
}
Value *Res = Builder.CreateMul(LHS, RHS);
if (CI.getNumArgOperands() == 4)
Res = EmitX86Select(Builder, CI.getArgOperand(3), Res, CI.getArgOperand(2));
return Res;
}
[X86] test/testn intrinsics lowering to IR. llvm part. Remove builtins from llvm and add AutoUpgrade support. Also add fast-isel tests for the TEST and TESTN instructions. Differential Revision: https://reviews.llvm.org/D38736 llvm-svn: 318036
2017-11-13 13:51:18 +01:00
// Applying mask on vector of i1's and make sure result is at least 8 bits wide.
[X86] Rename the autoupgraded of packed fp compare and fpclass intrinsics that don't take a mask as input to exclude '.mask.' from their name. I think the intrinsics named 'avx512.mask.' should refer to the previous behavior of taking a mask argument in the intrinsic instead of using a 'select' or 'and' instruction in IR to accomplish the masking. This is more consistent with the goal that eventually we will have no intrinsics that have masking builtin. When we reach that goal, we should have no intrinsics named "avx512.mask". llvm-svn: 335744
2018-06-27 17:57:53 +02:00
static Value *ApplyX86MaskOn1BitsVec(IRBuilder<> &Builder, Value *Vec,
Value *Mask) {
unsigned NumElts = Vec->getType()->getVectorNumElements();
[X86] Remove llvm.x86.avx512.cvt*2mask.* intrinsics and autoupgrade to (icmp slt X, 0) I had to drop fast-isel-abort from a test because we can't fast isel some of the mask stuff. When we used intrinsics we implicitly fell back to SelectionDAG for the intrinsic call without triggering the abort error. But with native IR that doesn't happen the same way. llvm-svn: 322050
2018-01-09 01:50:47 +01:00
if (Mask) {
const auto *C = dyn_cast<Constant>(Mask);
if (!C || !C->isAllOnesValue())
Vec = Builder.CreateAnd(Vec, getX86MaskVec(Builder, Mask, NumElts));
}
[X86] test/testn intrinsics lowering to IR. llvm part. Remove builtins from llvm and add AutoUpgrade support. Also add fast-isel tests for the TEST and TESTN instructions. Differential Revision: https://reviews.llvm.org/D38736 llvm-svn: 318036
2017-11-13 13:51:18 +01:00
if (NumElts < 8) {
uint32_t Indices[8];
for (unsigned i = 0; i != NumElts; ++i)
Indices[i] = i;
for (unsigned i = NumElts; i != 8; ++i)
Indices[i] = NumElts + i % NumElts;
Vec = Builder.CreateShuffleVector(Vec,
Constant::getNullValue(Vec->getType()),
Indices);
}
return Builder.CreateBitCast(Vec, Builder.getIntNTy(std::max(NumElts, 8U)));
}
[AVX512] Remove the masked vpcmpeq/vcmpgt intrinsics and autoupgrade them to native icmps. llvm-svn: 273240
2016-06-21 05:53:24 +02:00
static Value *upgradeMaskedCompare(IRBuilder<> &Builder, CallInst &CI,
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-22 22:11:01 +02:00
unsigned CC, bool Signed) {
[AVX512] Remove the masked vpcmpeq/vcmpgt intrinsics and autoupgrade them to native icmps. llvm-svn: 273240
2016-06-21 05:53:24 +02:00
Value *Op0 = CI.getArgOperand(0);
unsigned NumElts = Op0->getType()->getVectorNumElements();
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-22 22:11:01 +02:00
Value *Cmp;
if (CC == 3) {
Cmp = Constant::getNullValue(llvm::VectorType::get(Builder.getInt1Ty(), NumElts));
} else if (CC == 7) {
Cmp = Constant::getAllOnesValue(llvm::VectorType::get(Builder.getInt1Ty(), NumElts));
} else {
ICmpInst::Predicate Pred;
switch (CC) {
default: llvm_unreachable("Unknown condition code");
case 0: Pred = ICmpInst::ICMP_EQ; break;
case 1: Pred = Signed ? ICmpInst::ICMP_SLT : ICmpInst::ICMP_ULT; break;
case 2: Pred = Signed ? ICmpInst::ICMP_SLE : ICmpInst::ICMP_ULE; break;
case 4: Pred = ICmpInst::ICMP_NE; break;
case 5: Pred = Signed ? ICmpInst::ICMP_SGE : ICmpInst::ICMP_UGE; break;
case 6: Pred = Signed ? ICmpInst::ICMP_SGT : ICmpInst::ICMP_UGT; break;
}
Cmp = Builder.CreateICmp(Pred, Op0, CI.getArgOperand(1));
}
Value *Mask = CI.getArgOperand(CI.getNumArgOperands() - 1);
[AVX512] Remove the masked vpcmpeq/vcmpgt intrinsics and autoupgrade them to native icmps. llvm-svn: 273240
2016-06-21 05:53:24 +02:00
[X86] Rename the autoupgraded of packed fp compare and fpclass intrinsics that don't take a mask as input to exclude '.mask.' from their name. I think the intrinsics named 'avx512.mask.' should refer to the previous behavior of taking a mask argument in the intrinsic instead of using a 'select' or 'and' instruction in IR to accomplish the masking. This is more consistent with the goal that eventually we will have no intrinsics that have masking builtin. When we reach that goal, we should have no intrinsics named "avx512.mask". llvm-svn: 335744
2018-06-27 17:57:53 +02:00
return ApplyX86MaskOn1BitsVec(Builder, Cmp, Mask);
[AVX512] Remove the masked vpcmpeq/vcmpgt intrinsics and autoupgrade them to native icmps. llvm-svn: 273240
2016-06-21 05:53:24 +02:00
}
[AVX-512] Remove intrinsics for 128/256-bit masked shift by single element in xmm. Instead upgrade them to a select and the older SSE/AVX2 intrinsic. llvm-svn: 286070
2016-11-06 17:29:08 +01:00
// Replace a masked intrinsic with an older unmasked intrinsic.
static Value *UpgradeX86MaskedShift(IRBuilder<> &Builder, CallInst &CI,
Intrinsic::ID IID) {
[X86] Merge some of the autoupgrade handling for masked intrinsics that just need to upgrade to an unmasked version plus a select. NFCI These are were previously grouped in small groups of similarish intrinsics. But all the intrinsics have the same number of arguments and the same order. So we can move them all into a larger group for handling. llvm-svn: 329549
2018-04-09 08:15:09 +02:00
Function *Intrin = Intrinsic::getDeclaration(CI.getModule(), IID);
[AVX-512] Remove intrinsics for 128/256-bit masked shift by single element in xmm. Instead upgrade them to a select and the older SSE/AVX2 intrinsic. llvm-svn: 286070
2016-11-06 17:29:08 +01:00
Value *Rep = Builder.CreateCall(Intrin,
{ CI.getArgOperand(0), CI.getArgOperand(1) });
return EmitX86Select(Builder, CI.getArgOperand(3), Rep, CI.getArgOperand(2));
}
[X86][AVX512] Removing llvm x86 intrinsics for _mm_mask_move_{ss|sd} intrinsics. Differential Revision: https://reviews.llvm.org/D26128 llvm-svn: 287087
2016-11-16 10:00:28 +01:00
static Value* upgradeMaskedMove(IRBuilder<> &Builder, CallInst &CI) {
Value* A = CI.getArgOperand(0);
Value* B = CI.getArgOperand(1);
Value* Src = CI.getArgOperand(2);
Value* Mask = CI.getArgOperand(3);
Value* AndNode = Builder.CreateAnd(Mask, APInt(8, 1));
Value* Cmp = Builder.CreateIsNotNull(AndNode);
Value* Extract1 = Builder.CreateExtractElement(B, (uint64_t)0);
Value* Extract2 = Builder.CreateExtractElement(Src, (uint64_t)0);
Value* Select = Builder.CreateSelect(Cmp, Extract1, Extract2);
return Builder.CreateInsertElement(A, Select, (uint64_t)0);
}
[AVX-512] Remove intrinsics for 128/256-bit masked shift by single element in xmm. Instead upgrade them to a select and the older SSE/AVX2 intrinsic. llvm-svn: 286070
2016-11-06 17:29:08 +01:00
[X86][LLVM] Converting __mm{|256|512}_movm_epi{8|16|32|64} LLVMIR call into generic intrinsics. This patch is a part one of two reviews, one for the clang and the other for LLVM. The patch deletes the back-end intrinsics and adds support for them in the auto upgrade. Differential Revision: https://reviews.llvm.org/D31393 llvm-svn: 299432
2017-04-04 15:32:14 +02:00
static Value* UpgradeMaskToInt(IRBuilder<> &Builder, CallInst &CI) {
Value* Op = CI.getArgOperand(0);
Type* ReturnOp = CI.getType();
unsigned NumElts = CI.getType()->getVectorNumElements();
Value *Mask = getX86MaskVec(Builder, Op, NumElts);
return Builder.CreateSExt(Mask, ReturnOp, "vpmovm2");
}
[X86] Merge some of the autoupgrade handling for masked intrinsics that just need to upgrade to an unmasked version plus a select. NFCI These are were previously grouped in small groups of similarish intrinsics. But all the intrinsics have the same number of arguments and the same order. So we can move them all into a larger group for handling. llvm-svn: 329549
2018-04-09 08:15:09 +02:00
// Replace intrinsic with unmasked version and a select.
static bool upgradeAVX512MaskToSelect(StringRef Name, IRBuilder<> &Builder,
CallInst &CI, Value *&Rep) {
Name = Name.substr(12); // Remove avx512.mask.
unsigned VecWidth = CI.getType()->getPrimitiveSizeInBits();
unsigned EltWidth = CI.getType()->getScalarSizeInBits();
Intrinsic::ID IID;
if (Name.startswith("max.p")) {
if (VecWidth == 128 && EltWidth == 32)
IID = Intrinsic::x86_sse_max_ps;
else if (VecWidth == 128 && EltWidth == 64)
IID = Intrinsic::x86_sse2_max_pd;
else if (VecWidth == 256 && EltWidth == 32)
IID = Intrinsic::x86_avx_max_ps_256;
else if (VecWidth == 256 && EltWidth == 64)
IID = Intrinsic::x86_avx_max_pd_256;
else
llvm_unreachable("Unexpected intrinsic");
} else if (Name.startswith("min.p")) {
if (VecWidth == 128 && EltWidth == 32)
IID = Intrinsic::x86_sse_min_ps;
else if (VecWidth == 128 && EltWidth == 64)
IID = Intrinsic::x86_sse2_min_pd;
else if (VecWidth == 256 && EltWidth == 32)
IID = Intrinsic::x86_avx_min_ps_256;
else if (VecWidth == 256 && EltWidth == 64)
IID = Intrinsic::x86_avx_min_pd_256;
else
llvm_unreachable("Unexpected intrinsic");
} else if (Name.startswith("pshuf.b.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_ssse3_pshuf_b_128;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx2_pshuf_b;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_pshuf_b_512;
else
llvm_unreachable("Unexpected intrinsic");
} else if (Name.startswith("pmul.hr.sw.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_ssse3_pmul_hr_sw_128;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx2_pmul_hr_sw;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_pmul_hr_sw_512;
else
llvm_unreachable("Unexpected intrinsic");
} else if (Name.startswith("pmulh.w.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_sse2_pmulh_w;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx2_pmulh_w;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_pmulh_w_512;
else
llvm_unreachable("Unexpected intrinsic");
} else if (Name.startswith("pmulhu.w.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_sse2_pmulhu_w;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx2_pmulhu_w;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_pmulhu_w_512;
else
llvm_unreachable("Unexpected intrinsic");
[X86] Remove 128/256-bit masked pmaddubsw and pmaddwd intrinsics. Replace 512-bit masked intrinsic with unmasked intrinsic and a select. The 128/256-bit versions were no longer used by clang. It uses the legacy SSE/AVX2 version and a select. The 512-bit was changed to the same for consistency. llvm-svn: 329774
2018-04-11 06:55:04 +02:00
} else if (Name.startswith("pmaddw.d.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_sse2_pmadd_wd;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx2_pmadd_wd;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_pmaddw_d_512;
else
llvm_unreachable("Unexpected intrinsic");
} else if (Name.startswith("pmaddubs.w.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_ssse3_pmadd_ub_sw_128;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx2_pmadd_ub_sw;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_pmaddubs_w_512;
else
llvm_unreachable("Unexpected intrinsic");
[X86] Merge some of the autoupgrade handling for masked intrinsics that just need to upgrade to an unmasked version plus a select. NFCI These are were previously grouped in small groups of similarish intrinsics. But all the intrinsics have the same number of arguments and the same order. So we can move them all into a larger group for handling. llvm-svn: 329549
2018-04-09 08:15:09 +02:00
} else if (Name.startswith("packsswb.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_sse2_packsswb_128;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx2_packsswb;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_packsswb_512;
else
llvm_unreachable("Unexpected intrinsic");
} else if (Name.startswith("packssdw.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_sse2_packssdw_128;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx2_packssdw;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_packssdw_512;
else
llvm_unreachable("Unexpected intrinsic");
} else if (Name.startswith("packuswb.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_sse2_packuswb_128;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx2_packuswb;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_packuswb_512;
else
llvm_unreachable("Unexpected intrinsic");
} else if (Name.startswith("packusdw.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_sse41_packusdw;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx2_packusdw;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_packusdw_512;
else
llvm_unreachable("Unexpected intrinsic");
} else if (Name.startswith("vpermilvar.")) {
if (VecWidth == 128 && EltWidth == 32)
IID = Intrinsic::x86_avx_vpermilvar_ps;
else if (VecWidth == 128 && EltWidth == 64)
IID = Intrinsic::x86_avx_vpermilvar_pd;
else if (VecWidth == 256 && EltWidth == 32)
IID = Intrinsic::x86_avx_vpermilvar_ps_256;
else if (VecWidth == 256 && EltWidth == 64)
IID = Intrinsic::x86_avx_vpermilvar_pd_256;
else if (VecWidth == 512 && EltWidth == 32)
IID = Intrinsic::x86_avx512_vpermilvar_ps_512;
else if (VecWidth == 512 && EltWidth == 64)
IID = Intrinsic::x86_avx512_vpermilvar_pd_512;
else
llvm_unreachable("Unexpected intrinsic");
[X86] Remove some unused masked conversion intrinsics that can be replaced with an older intrinsic and a select. This is what clang already uses. llvm-svn: 332170
2018-05-12 04:34:28 +02:00
} else if (Name == "cvtpd2dq.256") {
[X86] Remove and autoupgrade masked vpermd/vpermps intrinsics. llvm-svn: 332198
2018-05-13 20:03:59 +02:00
IID = Intrinsic::x86_avx_cvt_pd2dq_256;
[X86] Remove some unused masked conversion intrinsics that can be replaced with an older intrinsic and a select. This is what clang already uses. llvm-svn: 332170
2018-05-12 04:34:28 +02:00
} else if (Name == "cvtpd2ps.256") {
[X86] Remove and autoupgrade masked vpermd/vpermps intrinsics. llvm-svn: 332198
2018-05-13 20:03:59 +02:00
IID = Intrinsic::x86_avx_cvt_pd2_ps_256;
[X86] Remove some unused masked conversion intrinsics that can be replaced with an older intrinsic and a select. This is what clang already uses. llvm-svn: 332170
2018-05-12 04:34:28 +02:00
} else if (Name == "cvttpd2dq.256") {
[X86] Remove and autoupgrade masked vpermd/vpermps intrinsics. llvm-svn: 332198
2018-05-13 20:03:59 +02:00
IID = Intrinsic::x86_avx_cvtt_pd2dq_256;
[X86] Remove some unused masked conversion intrinsics that can be replaced with an older intrinsic and a select. This is what clang already uses. llvm-svn: 332170
2018-05-12 04:34:28 +02:00
} else if (Name == "cvttps2dq.128") {
[X86] Remove and autoupgrade masked vpermd/vpermps intrinsics. llvm-svn: 332198
2018-05-13 20:03:59 +02:00
IID = Intrinsic::x86_sse2_cvttps2dq;
[X86] Remove some unused masked conversion intrinsics that can be replaced with an older intrinsic and a select. This is what clang already uses. llvm-svn: 332170
2018-05-12 04:34:28 +02:00
} else if (Name == "cvttps2dq.256") {
[X86] Remove and autoupgrade masked vpermd/vpermps intrinsics. llvm-svn: 332198
2018-05-13 20:03:59 +02:00
IID = Intrinsic::x86_avx_cvtt_ps2dq_256;
[X86] Remove mask arguments from permvar builtins/intrinsics. Use a select in IR instead. Someday maybe we'll use selects for all intrinsics. llvm-svn: 332824
2018-05-21 01:34:04 +02:00
} else if (Name.startswith("permvar.")) {
bool IsFloat = CI.getType()->isFPOrFPVectorTy();
if (VecWidth == 256 && EltWidth == 32 && IsFloat)
IID = Intrinsic::x86_avx2_permps;
else if (VecWidth == 256 && EltWidth == 32 && !IsFloat)
IID = Intrinsic::x86_avx2_permd;
else if (VecWidth == 256 && EltWidth == 64 && IsFloat)
IID = Intrinsic::x86_avx512_permvar_df_256;
else if (VecWidth == 256 && EltWidth == 64 && !IsFloat)
IID = Intrinsic::x86_avx512_permvar_di_256;
else if (VecWidth == 512 && EltWidth == 32 && IsFloat)
IID = Intrinsic::x86_avx512_permvar_sf_512;
else if (VecWidth == 512 && EltWidth == 32 && !IsFloat)
IID = Intrinsic::x86_avx512_permvar_si_512;
else if (VecWidth == 512 && EltWidth == 64 && IsFloat)
IID = Intrinsic::x86_avx512_permvar_df_512;
else if (VecWidth == 512 && EltWidth == 64 && !IsFloat)
IID = Intrinsic::x86_avx512_permvar_di_512;
else if (VecWidth == 128 && EltWidth == 16)
IID = Intrinsic::x86_avx512_permvar_hi_128;
else if (VecWidth == 256 && EltWidth == 16)
IID = Intrinsic::x86_avx512_permvar_hi_256;
else if (VecWidth == 512 && EltWidth == 16)
IID = Intrinsic::x86_avx512_permvar_hi_512;
else if (VecWidth == 128 && EltWidth == 8)
IID = Intrinsic::x86_avx512_permvar_qi_128;
else if (VecWidth == 256 && EltWidth == 8)
IID = Intrinsic::x86_avx512_permvar_qi_256;
else if (VecWidth == 512 && EltWidth == 8)
IID = Intrinsic::x86_avx512_permvar_qi_512;
else
llvm_unreachable("Unexpected intrinsic");
[X86] Remove masking from dbpsadbw intrinsics, use select in IR instead. llvm-svn: 334384
2018-06-11 08:18:22 +02:00
} else if (Name.startswith("dbpsadbw.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_avx512_dbpsadbw_128;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx512_dbpsadbw_256;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_dbpsadbw_512;
else
llvm_unreachable("Unexpected intrinsic");
[X86] Remove mask parameter from avx512 pmultishiftqb intrinsics. Use select in IR instead. Fixes PR40259 llvm-svn: 351035
2019-01-14 09:46:45 +01:00
} else if (Name.startswith("pmultishift.qb.")) {
if (VecWidth == 128)
IID = Intrinsic::x86_avx512_pmultishift_qb_128;
else if (VecWidth == 256)
IID = Intrinsic::x86_avx512_pmultishift_qb_256;
else if (VecWidth == 512)
IID = Intrinsic::x86_avx512_pmultishift_qb_512;
else
llvm_unreachable("Unexpected intrinsic");
[X86] Remove and autoupgrade vpconflict intrinsics that take a mask and passthru argument. We have unmasked versions as of r352172 llvm-svn: 352270
2019-01-26 07:27:01 +01:00
} else if (Name.startswith("conflict.")) {
if (Name[9] == 'd' && VecWidth == 128)
IID = Intrinsic::x86_avx512_conflict_d_128;
else if (Name[9] == 'd' && VecWidth == 256)
IID = Intrinsic::x86_avx512_conflict_d_256;
else if (Name[9] == 'd' && VecWidth == 512)
IID = Intrinsic::x86_avx512_conflict_d_512;
else if (Name[9] == 'q' && VecWidth == 128)
IID = Intrinsic::x86_avx512_conflict_q_128;
else if (Name[9] == 'q' && VecWidth == 256)
IID = Intrinsic::x86_avx512_conflict_q_256;
else if (Name[9] == 'q' && VecWidth == 512)
IID = Intrinsic::x86_avx512_conflict_q_512;
else
llvm_unreachable("Unexpected intrinsic");
[X86] Restore the pavg intrinsics. The pattern we replaced these with may be too hard to match as demonstrated by PR41496 and PR41316. This patch restores the intrinsics and then we can start focusing on the optimizing the intrinsics. I've mostly reverted the original patch that removed them. Though I modified the avx512 intrinsics to not have masking built in. Differential Revision: https://reviews.llvm.org/D60674 llvm-svn: 358427
2019-04-15 19:17:35 +02:00
} else if (Name.startswith("pavg.")) {
if (Name[5] == 'b' && VecWidth == 128)
IID = Intrinsic::x86_sse2_pavg_b;
else if (Name[5] == 'b' && VecWidth == 256)
IID = Intrinsic::x86_avx2_pavg_b;
else if (Name[5] == 'b' && VecWidth == 512)
IID = Intrinsic::x86_avx512_pavg_b_512;
else if (Name[5] == 'w' && VecWidth == 128)
IID = Intrinsic::x86_sse2_pavg_w;
else if (Name[5] == 'w' && VecWidth == 256)
IID = Intrinsic::x86_avx2_pavg_w;
else if (Name[5] == 'w' && VecWidth == 512)
IID = Intrinsic::x86_avx512_pavg_w_512;
else
llvm_unreachable("Unexpected intrinsic");
[X86] Merge some of the autoupgrade handling for masked intrinsics that just need to upgrade to an unmasked version plus a select. NFCI These are were previously grouped in small groups of similarish intrinsics. But all the intrinsics have the same number of arguments and the same order. So we can move them all into a larger group for handling. llvm-svn: 329549
2018-04-09 08:15:09 +02:00
} else
return false;
[X86] Remove some unused masked conversion intrinsics that can be replaced with an older intrinsic and a select. This is what clang already uses. llvm-svn: 332170
2018-05-12 04:34:28 +02:00
SmallVector<Value *, 4> Args(CI.arg_operands().begin(),
CI.arg_operands().end());
Args.pop_back();
Args.pop_back();
[X86] Merge some of the autoupgrade handling for masked intrinsics that just need to upgrade to an unmasked version plus a select. NFCI These are were previously grouped in small groups of similarish intrinsics. But all the intrinsics have the same number of arguments and the same order. So we can move them all into a larger group for handling. llvm-svn: 329549
2018-04-09 08:15:09 +02:00
Rep = Builder.CreateCall(Intrinsic::getDeclaration(CI.getModule(), IID),
[X86] Remove some unused masked conversion intrinsics that can be replaced with an older intrinsic and a select. This is what clang already uses. llvm-svn: 332170
2018-05-12 04:34:28 +02:00
Args);
unsigned NumArgs = CI.getNumArgOperands();
Rep = EmitX86Select(Builder, CI.getArgOperand(NumArgs - 1), Rep,
CI.getArgOperand(NumArgs - 2));
[X86] Merge some of the autoupgrade handling for masked intrinsics that just need to upgrade to an unmasked version plus a select. NFCI These are were previously grouped in small groups of similarish intrinsics. But all the intrinsics have the same number of arguments and the same order. So we can move them all into a larger group for handling. llvm-svn: 329549
2018-04-09 08:15:09 +02:00
return true;
}
[IR] Upgrade comment token in objc retain release marker for asm call Older compiler issued '#' instead of ';' llvm-svn: 330173
2018-04-17 06:02:24 +02:00
/// Upgrade comment in call to inline asm that represents an objc retain release
/// marker.
void llvm::UpgradeInlineAsmString(std::string *AsmStr) {
Fix -Wtautological-compare warning with npos on Windows llvm-svn: 330614
2018-04-23 18:47:27 +02:00
size_t Pos;
[IR] Upgrade comment token in objc retain release marker for asm call Older compiler issued '#' instead of ';' llvm-svn: 330173
2018-04-17 06:02:24 +02:00
if (AsmStr->find("mov\tfp") == 0 &&
AsmStr->find("objc_retainAutoreleaseReturnValue") != std::string::npos &&
(Pos = AsmStr->find("# marker")) != std::string::npos) {
AsmStr->replace(Pos, 1, ";");
}
return;
}
fix comments; NFC llvm-svn: 272848
2016-06-16 00:01:28 +02:00
/// Upgrade a call to an old intrinsic. All argument and return casting must be
/// provided to seamlessly integrate with existing context.
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
void llvm::UpgradeIntrinsicCall(CallInst *CI, Function *NewFn) {
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 07:10:55 +01:00
Function *F = CI->getCalledFunction();
Fix unused value warning for value used only in assert. llvm-svn: 146440
2011-12-12 23:59:34 +01:00
LLVMContext &C = CI->getContext();
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 05:26:04 +01:00
IRBuilder<> Builder(C);
IR: Remove implicit iterator conversions from lib/IR, NFC Stop converting implicitly between iterators and pointers/references in lib/IR. For convenience, I've added a `getIterator()` accessor to `ilist_node` so that callers don't need to know how to spell the iterator class (i.e., they can use `X.getIterator()` instead of `Function::iterator(X)`). I'll eventually disallow these implicit conversions entirely, but there's a lot of code, so it doesn't make sense to do it all in one patch. One library or so at a time. Why? To root out cases of `getNextNode()` and `getPrevNode()` being used in iterator logic. The design of `ilist` makes that invalid when the current node could be at the back of the list, but it happens to "work" right now because of a bug where those functions never return `nullptr` if you're using a half-node sentinel. Before I can fix the function, I have to remove uses of it that rely on it misbehaving. (Maybe the function should just be deleted anyway? But I don't want deleting it -- potentially a huge project -- to block fixing ilist/iplist.) llvm-svn: 249782
2015-10-09 01:49:46 +02:00
Builder.SetInsertPoint(CI->getParent(), CI->getIterator());
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 05:26:04 +01:00
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 07:10:55 +01:00
assert(F && "Intrinsic call is not direct?");
if (!NewFn) {
// Get the Function's name.
StringRef Name = F->getName();
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
assert(Name.startswith("llvm.") && "Intrinsic doesn't start with 'llvm.'");
Name = Name.substr(5);
bool IsX86 = Name.startswith("x86.");
if (IsX86)
Name = Name.substr(4);
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 02:00:32 +01:00
bool IsNVVM = Name.startswith("nvvm.");
if (IsNVVM)
Name = Name.substr(5);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
[X86][IR] Move the autoupgrading of store intrinsics out of the main nested if/else chain. This should buy a little more time against the MSVC limit mentioned in PR31034. The handlers for stores all return at the end of their block so they can be picked off early. llvm-svn: 289339
2016-12-10 22:15:48 +01:00
if (IsX86 && Name.startswith("sse4a.movnt.")) {
Module *M = F->getParent();
SmallVector<Metadata *, 1> Elts;
Elts.push_back(
ConstantAsMetadata::get(ConstantInt::get(Type::getInt32Ty(C), 1)));
MDNode *Node = MDNode::get(C, Elts);
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
// Nontemporal (unaligned) store of the 0'th element of the float/double
// vector.
Type *SrcEltTy = cast<VectorType>(Arg1->getType())->getElementType();
PointerType *EltPtrTy = PointerType::getUnqual(SrcEltTy);
Value *Addr = Builder.CreateBitCast(Arg0, EltPtrTy, "cast");
Value *Extract =
Builder.CreateExtractElement(Arg1, (uint64_t)0, "extractelement");
[Alignment][NFC] Use Align with CreateAlignedStore Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet, bollu Subscribers: arsenm, jvesely, nhaehnle, hiraditya, kerbowa, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73274
2020-01-23 16:18:34 +01:00
StoreInst *SI = Builder.CreateAlignedStore(Extract, Addr, Align(1));
[X86][IR] Move the autoupgrading of store intrinsics out of the main nested if/else chain. This should buy a little more time against the MSVC limit mentioned in PR31034. The handlers for stores all return at the end of their block so they can be picked off early. llvm-svn: 289339
2016-12-10 22:15:48 +01:00
SI->setMetadata(M->getMDKindID("nontemporal"), Node);
// Remove intrinsic.
CI->eraseFromParent();
return;
}
if (IsX86 && (Name.startswith("avx.movnt.") ||
Name.startswith("avx512.storent."))) {
Module *M = F->getParent();
SmallVector<Metadata *, 1> Elts;
Elts.push_back(
ConstantAsMetadata::get(ConstantInt::get(Type::getInt32Ty(C), 1)));
MDNode *Node = MDNode::get(C, Elts);
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
// Convert the type of the pointer to a pointer to the stored type.
Value *BC = Builder.CreateBitCast(Arg0,
PointerType::getUnqual(Arg1->getType()),
"cast");
VectorType *VTy = cast<VectorType>(Arg1->getType());
[Alignment][NFC] Use Align with CreateAlignedStore Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet, bollu Subscribers: arsenm, jvesely, nhaehnle, hiraditya, kerbowa, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73274
2020-01-23 16:18:34 +01:00
StoreInst *SI =
Builder.CreateAlignedStore(Arg1, BC, Align(VTy->getBitWidth() / 8));
[X86][IR] Move the autoupgrading of store intrinsics out of the main nested if/else chain. This should buy a little more time against the MSVC limit mentioned in PR31034. The handlers for stores all return at the end of their block so they can be picked off early. llvm-svn: 289339
2016-12-10 22:15:48 +01:00
SI->setMetadata(M->getMDKindID("nontemporal"), Node);
// Remove intrinsic.
CI->eraseFromParent();
return;
}
if (IsX86 && Name == "sse2.storel.dq") {
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
Type *NewVecTy = VectorType::get(Type::getInt64Ty(C), 2);
Value *BC0 = Builder.CreateBitCast(Arg1, NewVecTy, "cast");
Value *Elt = Builder.CreateExtractElement(BC0, (uint64_t)0);
Value *BC = Builder.CreateBitCast(Arg0,
PointerType::getUnqual(Elt->getType()),
"cast");
[Alignment][NFC] Use Align with CreateAlignedStore Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet, bollu Subscribers: arsenm, jvesely, nhaehnle, hiraditya, kerbowa, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73274
2020-01-23 16:18:34 +01:00
Builder.CreateAlignedStore(Elt, BC, Align(1));
[X86][IR] Move the autoupgrading of store intrinsics out of the main nested if/else chain. This should buy a little more time against the MSVC limit mentioned in PR31034. The handlers for stores all return at the end of their block so they can be picked off early. llvm-svn: 289339
2016-12-10 22:15:48 +01:00
// Remove intrinsic.
CI->eraseFromParent();
return;
}
if (IsX86 && (Name.startswith("sse.storeu.") ||
Name.startswith("sse2.storeu.") ||
Name.startswith("avx.storeu."))) {
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
Arg0 = Builder.CreateBitCast(Arg0,
PointerType::getUnqual(Arg1->getType()),
"cast");
[Alignment][NFC] Use Align with CreateAlignedStore Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet, bollu Subscribers: arsenm, jvesely, nhaehnle, hiraditya, kerbowa, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73274
2020-01-23 16:18:34 +01:00
Builder.CreateAlignedStore(Arg1, Arg0, Align(1));
[X86][IR] Move the autoupgrading of store intrinsics out of the main nested if/else chain. This should buy a little more time against the MSVC limit mentioned in PR31034. The handlers for stores all return at the end of their block so they can be picked off early. llvm-svn: 289339
2016-12-10 22:15:48 +01:00
// Remove intrinsic.
CI->eraseFromParent();
return;
}
[X86] Remove and autoupgrade the avx512.mask.store.ss intrinsic. llvm-svn: 332079
2018-05-11 06:33:18 +02:00
if (IsX86 && Name == "avx512.mask.store.ss") {
Value *Mask = Builder.CreateAnd(CI->getArgOperand(2), Builder.getInt8(1));
UpgradeMaskedStore(Builder, CI->getArgOperand(0), CI->getArgOperand(1),
Mask, false);
// Remove intrinsic.
CI->eraseFromParent();
return;
}
[X86][IR] Merge together some very similar AutoUpgrade handling. NFC llvm-svn: 295562
2017-02-18 20:51:14 +01:00
if (IsX86 && (Name.startswith("avx512.mask.store"))) {
// "avx512.mask.storeu." or "avx512.mask.store."
bool Aligned = Name[17] != 'u'; // "avx512.mask.storeu".
[X86][IR] Move the autoupgrading of store intrinsics out of the main nested if/else chain. This should buy a little more time against the MSVC limit mentioned in PR31034. The handlers for stores all return at the end of their block so they can be picked off early. llvm-svn: 289339
2016-12-10 22:15:48 +01:00
UpgradeMaskedStore(Builder, CI->getArgOperand(0), CI->getArgOperand(1),
[X86][IR] Merge together some very similar AutoUpgrade handling. NFC llvm-svn: 295562
2017-02-18 20:51:14 +01:00
CI->getArgOperand(2), Aligned);
[X86][IR] Move the autoupgrading of store intrinsics out of the main nested if/else chain. This should buy a little more time against the MSVC limit mentioned in PR31034. The handlers for stores all return at the end of their block so they can be picked off early. llvm-svn: 289339
2016-12-10 22:15:48 +01:00
// Remove intrinsic.
CI->eraseFromParent();
return;
}
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 07:10:55 +01:00
Value *Rep;
fix comments; NFC llvm-svn: 272848
2016-06-16 00:01:28 +02:00
// Upgrade packed integer vector compare intrinsics to compare instructions.
[X86][IR] Merge together some very similar AutoUpgrade handling. NFC llvm-svn: 295562
2017-02-18 20:51:14 +01:00
if (IsX86 && (Name.startswith("sse2.pcmp") ||
Name.startswith("avx2.pcmp"))) {
// "sse2.pcpmpeq." "sse2.pcmpgt." "avx2.pcmpeq." or "avx2.pcmpgt."
bool CmpEq = Name[9] == 'e';
Rep = Builder.CreateICmp(CmpEq ? ICmpInst::ICMP_EQ : ICmpInst::ICMP_SGT,
CI->getArgOperand(0), CI->getArgOperand(1));
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 07:10:55 +01:00
Rep = Builder.CreateSExt(Rep, CI->getType(), "");
[x86][AVX512] Lowering Broadcastm intrinsics to LLVM IR This patch, together with a matching clang patch (https://reviews.llvm.org/D38683), implements the lowering of X86 broadcastm intrinsics to IR. Differential Revision: https://reviews.llvm.org/D38684 Change-Id: I709ac0b34641095397e994c8ff7e15d1315b3540 llvm-svn: 317458
2017-11-06 08:09:24 +01:00
} else if (IsX86 && (Name.startswith("avx512.broadcastm"))) {
Type *ExtTy = Type::getInt32Ty(C);
if (CI->getOperand(0)->getType()->isIntegerTy(8))
ExtTy = Type::getInt64Ty(C);
unsigned NumElts = CI->getType()->getPrimitiveSizeInBits() /
ExtTy->getPrimitiveSizeInBits();
Rep = Builder.CreateZExt(CI->getArgOperand(0), ExtTy);
Rep = Builder.CreateVectorSplat(NumElts, Rep);
[X86] Lowering sqrt intrinsics to native IR Summary: Complementary patch to lowering sqrt intrinsics in Clang. Reviewers: craig.topper, spatel, RKSimon, DavidKreitzer, uriel.k Reviewed By: craig.topper Subscribers: tkrupa, mike.dvoretsky, llvm-commits Differential Revision: https://reviews.llvm.org/D41599 llvm-svn: 334849
2018-06-15 20:05:24 +02:00
} else if (IsX86 && (Name == "sse.sqrt.ss" ||
Name == "sse2.sqrt.sd")) {
Value *Vec = CI->getArgOperand(0);
Value *Elt0 = Builder.CreateExtractElement(Vec, (uint64_t)0);
Function *Intr = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::sqrt, Elt0->getType());
Elt0 = Builder.CreateCall(Intr, Elt0);
Rep = Builder.CreateInsertElement(Vec, Elt0, (uint64_t)0);
} else if (IsX86 && (Name.startswith("avx.sqrt.p") ||
Name.startswith("sse2.sqrt.p") ||
Name.startswith("sse.sqrt.p"))) {
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(),
Intrinsic::sqrt,
CI->getType()),
{CI->getArgOperand(0)});
[X86] Remove masking from the avx512 packed sqrt intrinsics. Use select in IR instead. While there improve the coverage of the intrinsic testing and add fast-isel tests. llvm-svn: 335944
2018-06-29 07:43:26 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.sqrt.p"))) {
if (CI->getNumArgOperands() == 4 &&
(!isa<ConstantInt>(CI->getArgOperand(3)) ||
cast<ConstantInt>(CI->getArgOperand(3))->getZExtValue() != 4)) {
Intrinsic::ID IID = Name[18] == 's' ? Intrinsic::x86_avx512_sqrt_ps_512
: Intrinsic::x86_avx512_sqrt_pd_512;
Value *Args[] = { CI->getArgOperand(0), CI->getArgOperand(3) };
Rep = Builder.CreateCall(Intrinsic::getDeclaration(CI->getModule(),
IID), Args);
} else {
[X86] Lowering sqrt intrinsics to native IR Summary: Complementary patch to lowering sqrt intrinsics in Clang. Reviewers: craig.topper, spatel, RKSimon, DavidKreitzer, uriel.k Reviewed By: craig.topper Subscribers: tkrupa, mike.dvoretsky, llvm-commits Differential Revision: https://reviews.llvm.org/D41599 llvm-svn: 334849
2018-06-15 20:05:24 +02:00
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(),
Intrinsic::sqrt,
CI->getType()),
{CI->getArgOperand(0)});
[X86] Remove masking from the avx512 packed sqrt intrinsics. Use select in IR instead. While there improve the coverage of the intrinsic testing and add fast-isel tests. llvm-svn: 335944
2018-06-29 07:43:26 +02:00
}
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[X86] test/testn intrinsics lowering to IR. llvm part. Remove builtins from llvm and add AutoUpgrade support. Also add fast-isel tests for the TEST and TESTN instructions. Differential Revision: https://reviews.llvm.org/D38736 llvm-svn: 318036
2017-11-13 13:51:18 +01:00
} else if (IsX86 && (Name.startswith("avx512.ptestm") ||
Name.startswith("avx512.ptestnm"))) {
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
Value *Mask = CI->getArgOperand(2);
Rep = Builder.CreateAnd(Op0, Op1);
llvm::Type *Ty = Op0->getType();
Value *Zero = llvm::Constant::getNullValue(Ty);
ICmpInst::Predicate Pred =
Name.startswith("avx512.ptestm") ? ICmpInst::ICMP_NE : ICmpInst::ICMP_EQ;
Rep = Builder.CreateICmp(Pred, Rep, Zero);
[X86] Rename the autoupgraded of packed fp compare and fpclass intrinsics that don't take a mask as input to exclude '.mask.' from their name. I think the intrinsics named 'avx512.mask.' should refer to the previous behavior of taking a mask argument in the intrinsic instead of using a 'select' or 'and' instruction in IR to accomplish the masking. This is more consistent with the goal that eventually we will have no intrinsics that have masking builtin. When we reach that goal, we should have no intrinsics named "avx512.mask". llvm-svn: 335744
2018-06-27 17:57:53 +02:00
Rep = ApplyX86MaskOn1BitsVec(Builder, Rep, Mask);
[X86] test/testn intrinsics lowering to IR. llvm part. Remove builtins from llvm and add AutoUpgrade support. Also add fast-isel tests for the TEST and TESTN instructions. Differential Revision: https://reviews.llvm.org/D38736 llvm-svn: 318036
2017-11-13 13:51:18 +01:00
} else if (IsX86 && (Name.startswith("avx512.mask.pbroadcast"))){
[x86] Lowering Mask Set1 intrinsics to LLVM IR This patch, together with a matching clang patch (https://reviews.llvm.org/D37668), implements the lowering of X86 mask set1 intrinsics to IR. Differential Revision: https://reviews.llvm.org/D37669 llvm-svn: 313625
2017-09-19 13:03:06 +02:00
unsigned NumElts =
CI->getArgOperand(1)->getType()->getVectorNumElements();
Rep = Builder.CreateVectorSplat(NumElts, CI->getArgOperand(0));
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[x86][AVX512] Lowering kunpack intrinsics to LLVM IR This patch, together with a matching clang patch (https://reviews.llvm.org/D39719), implements the lowering of X86 kunpack intrinsics to IR. Differential Revision: https://reviews.llvm.org/D39720 Change-Id: I4088d9428478f9457f6afddc90bd3d66b3daf0a1 llvm-svn: 319778
2017-12-05 16:42:56 +01:00
} else if (IsX86 && (Name.startswith("avx512.kunpck"))) {
[X86] Autoupgrade kunpck intrinsics using vector operations instead of scalar operations Summary: This patch changes the kunpck intrinsic autoupgrade to use vXi1 shufflevector operations to perform vector extracts and concats. This more closely matches the definition of the kunpck instructions. Currently we rely on a DAG combine to turn the scalar shift/and/or code into a concat vectors operation. By doing it in the IR we get this for free. Reviewers: spatel, RKSimon, zvi, jina.nahias Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D42018 llvm-svn: 322462
2018-01-14 20:24:10 +01:00
unsigned NumElts = CI->getType()->getScalarSizeInBits();
Value *LHS = getX86MaskVec(Builder, CI->getArgOperand(0), NumElts);
Value *RHS = getX86MaskVec(Builder, CI->getArgOperand(1), NumElts);
uint32_t Indices[64];
for (unsigned i = 0; i != NumElts; ++i)
Indices[i] = i;
// First extract half of each vector. This gives better codegen than
// doing it in a single shuffle.
LHS = Builder.CreateShuffleVector(LHS, LHS,
makeArrayRef(Indices, NumElts / 2));
RHS = Builder.CreateShuffleVector(RHS, RHS,
makeArrayRef(Indices, NumElts / 2));
// Concat the vectors.
[X86] Reverse the operand order of the autoupgrade of the kunpack builtins. The second operand needs to be in the lower bits of the concatenation. This matches llvm 5.0, gcc, and icc behavior. Fixes PR36360. llvm-svn: 324953
2018-02-12 23:38:34 +01:00
// NOTE: Operands have to be swapped to match intrinsic definition.
Rep = Builder.CreateShuffleVector(RHS, LHS,
[X86] Autoupgrade kunpck intrinsics using vector operations instead of scalar operations Summary: This patch changes the kunpck intrinsic autoupgrade to use vXi1 shufflevector operations to perform vector extracts and concats. This more closely matches the definition of the kunpck instructions. Currently we rely on a DAG combine to turn the scalar shift/and/or code into a concat vectors operation. By doing it in the IR we get this for free. Reviewers: spatel, RKSimon, zvi, jina.nahias Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D42018 llvm-svn: 322462
2018-01-14 20:24:10 +01:00
makeArrayRef(Indices, NumElts));
Rep = Builder.CreateBitCast(Rep, CI->getType());
[X86] Remove and autoupgrade kand/kandn/kor/kxor/kxnor/knot intrinsics. Clang already stopped using these a couple months ago. The test cases aren't great as there is nothing forcing the operations to stay in k-registers so some of them moved back to scalar ops due to the bitcasts being moved around. llvm-svn: 324177
2018-02-03 21:18:25 +01:00
} else if (IsX86 && Name == "avx512.kand.w") {
Value *LHS = getX86MaskVec(Builder, CI->getArgOperand(0), 16);
Value *RHS = getX86MaskVec(Builder, CI->getArgOperand(1), 16);
Rep = Builder.CreateAnd(LHS, RHS);
Rep = Builder.CreateBitCast(Rep, CI->getType());
} else if (IsX86 && Name == "avx512.kandn.w") {
Value *LHS = getX86MaskVec(Builder, CI->getArgOperand(0), 16);
Value *RHS = getX86MaskVec(Builder, CI->getArgOperand(1), 16);
LHS = Builder.CreateNot(LHS);
Rep = Builder.CreateAnd(LHS, RHS);
Rep = Builder.CreateBitCast(Rep, CI->getType());
} else if (IsX86 && Name == "avx512.kor.w") {
Value *LHS = getX86MaskVec(Builder, CI->getArgOperand(0), 16);
Value *RHS = getX86MaskVec(Builder, CI->getArgOperand(1), 16);
Rep = Builder.CreateOr(LHS, RHS);
Rep = Builder.CreateBitCast(Rep, CI->getType());
} else if (IsX86 && Name == "avx512.kxor.w") {
Value *LHS = getX86MaskVec(Builder, CI->getArgOperand(0), 16);
Value *RHS = getX86MaskVec(Builder, CI->getArgOperand(1), 16);
Rep = Builder.CreateXor(LHS, RHS);
Rep = Builder.CreateBitCast(Rep, CI->getType());
} else if (IsX86 && Name == "avx512.kxnor.w") {
Value *LHS = getX86MaskVec(Builder, CI->getArgOperand(0), 16);
Value *RHS = getX86MaskVec(Builder, CI->getArgOperand(1), 16);
LHS = Builder.CreateNot(LHS);
Rep = Builder.CreateXor(LHS, RHS);
Rep = Builder.CreateBitCast(Rep, CI->getType());
} else if (IsX86 && Name == "avx512.knot.w") {
Rep = getX86MaskVec(Builder, CI->getArgOperand(0), 16);
Rep = Builder.CreateNot(Rep);
Rep = Builder.CreateBitCast(Rep, CI->getType());
[X86] Remove kortest intrinsics and replace with native IR. llvm-svn: 324646
2018-02-08 21:16:06 +01:00
} else if (IsX86 &&
(Name == "avx512.kortestz.w" || Name == "avx512.kortestc.w")) {
Value *LHS = getX86MaskVec(Builder, CI->getArgOperand(0), 16);
Value *RHS = getX86MaskVec(Builder, CI->getArgOperand(1), 16);
Rep = Builder.CreateOr(LHS, RHS);
Rep = Builder.CreateBitCast(Rep, Builder.getInt16Ty());
Value *C;
if (Name[14] == 'c')
C = ConstantInt::getAllOnesValue(Builder.getInt16Ty());
else
C = ConstantInt::getNullValue(Builder.getInt16Ty());
Rep = Builder.CreateICmpEQ(Rep, C);
Rep = Builder.CreateZExt(Rep, Builder.getInt32Ty());
Fix MSVC "compiler limit: blocks nested too deeply" error. NFCI. MSVC only accepts if-else chains up to 127 blocks long. I've had to merge a number of intrinsic cases together to get back below this limit, resulting in some duplication of string matches; this shouldn't cause any notable increase in runtime (and even then only for old IR, nothing that clang currently emits). llvm-svn: 339666
2018-08-14 12:04:14 +02:00
} else if (IsX86 && (Name == "sse.add.ss" || Name == "sse2.add.sd" ||
Name == "sse.sub.ss" || Name == "sse2.sub.sd" ||
Name == "sse.mul.ss" || Name == "sse2.mul.sd" ||
Name == "sse.div.ss" || Name == "sse2.div.sd")) {
[X86] Remove the scalar intrinsics for fadd/fsub/fdiv/fmul Summary: These intrinsics have been unused for clang for a while. This patch removes them. We auto upgrade them to extractelements, a scalar operation and then an insertelement. This matches the sequence used by clangs intrinsic file. Reviewers: zvi, delena, RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D26660 llvm-svn: 287083
2016-11-16 06:24:10 +01:00
Type *I32Ty = Type::getInt32Ty(C);
Value *Elt0 = Builder.CreateExtractElement(CI->getArgOperand(0),
ConstantInt::get(I32Ty, 0));
Value *Elt1 = Builder.CreateExtractElement(CI->getArgOperand(1),
ConstantInt::get(I32Ty, 0));
Fix MSVC "compiler limit: blocks nested too deeply" error. NFCI. MSVC only accepts if-else chains up to 127 blocks long. I've had to merge a number of intrinsic cases together to get back below this limit, resulting in some duplication of string matches; this shouldn't cause any notable increase in runtime (and even then only for old IR, nothing that clang currently emits). llvm-svn: 339666
2018-08-14 12:04:14 +02:00
Value *EltOp;
if (Name.contains(".add."))
EltOp = Builder.CreateFAdd(Elt0, Elt1);
else if (Name.contains(".sub."))
EltOp = Builder.CreateFSub(Elt0, Elt1);
else if (Name.contains(".mul."))
EltOp = Builder.CreateFMul(Elt0, Elt1);
else
EltOp = Builder.CreateFDiv(Elt0, Elt1);
Rep = Builder.CreateInsertElement(CI->getArgOperand(0), EltOp,
[X86] Remove the scalar intrinsics for fadd/fsub/fdiv/fmul Summary: These intrinsics have been unused for clang for a while. This patch removes them. We auto upgrade them to extractelements, a scalar operation and then an insertelement. This matches the sequence used by clangs intrinsic file. Reviewers: zvi, delena, RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D26660 llvm-svn: 287083
2016-11-16 06:24:10 +01:00
ConstantInt::get(I32Ty, 0));
[X86][IR] Merge together some very similar AutoUpgrade handling. NFC llvm-svn: 295562
2017-02-18 20:51:14 +01:00
} else if (IsX86 && Name.startswith("avx512.mask.pcmp")) {
// "avx512.mask.pcmpeq." or "avx512.mask.pcmpgt."
bool CmpEq = Name[16] == 'e';
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-22 22:11:01 +02:00
Rep = upgradeMaskedCompare(Builder, *CI, CmpEq ? 0 : 6, true);
[X86] Remove mask parameter from vpshufbitqmb intrinsics. Change result to a vXi1 vector. The input mask can be represented with an AND in IR. Fixes PR40258 llvm-svn: 351028
2019-01-14 01:03:50 +01:00
} else if (IsX86 && Name.startswith("avx512.mask.vpshufbitqmb.")) {
Type *OpTy = CI->getArgOperand(0)->getType();
unsigned VecWidth = OpTy->getPrimitiveSizeInBits();
Intrinsic::ID IID;
switch (VecWidth) {
default: llvm_unreachable("Unexpected intrinsic");
case 128: IID = Intrinsic::x86_avx512_vpshufbitqmb_128; break;
case 256: IID = Intrinsic::x86_avx512_vpshufbitqmb_256; break;
case 512: IID = Intrinsic::x86_avx512_vpshufbitqmb_512; break;
}
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getOperand(0), CI->getArgOperand(1) });
Rep = ApplyX86MaskOn1BitsVec(Builder, Rep, CI->getArgOperand(2));
[X86] Rename the autoupgraded of packed fp compare and fpclass intrinsics that don't take a mask as input to exclude '.mask.' from their name. I think the intrinsics named 'avx512.mask.' should refer to the previous behavior of taking a mask argument in the intrinsic instead of using a 'select' or 'and' instruction in IR to accomplish the masking. This is more consistent with the goal that eventually we will have no intrinsics that have masking builtin. When we reach that goal, we should have no intrinsics named "avx512.mask". llvm-svn: 335744
2018-06-27 17:57:53 +02:00
} else if (IsX86 && Name.startswith("avx512.mask.fpclass.p")) {
Type *OpTy = CI->getArgOperand(0)->getType();
unsigned VecWidth = OpTy->getPrimitiveSizeInBits();
unsigned EltWidth = OpTy->getScalarSizeInBits();
Intrinsic::ID IID;
if (VecWidth == 128 && EltWidth == 32)
IID = Intrinsic::x86_avx512_fpclass_ps_128;
else if (VecWidth == 256 && EltWidth == 32)
IID = Intrinsic::x86_avx512_fpclass_ps_256;
else if (VecWidth == 512 && EltWidth == 32)
IID = Intrinsic::x86_avx512_fpclass_ps_512;
else if (VecWidth == 128 && EltWidth == 64)
IID = Intrinsic::x86_avx512_fpclass_pd_128;
else if (VecWidth == 256 && EltWidth == 64)
IID = Intrinsic::x86_avx512_fpclass_pd_256;
else if (VecWidth == 512 && EltWidth == 64)
IID = Intrinsic::x86_avx512_fpclass_pd_512;
else
llvm_unreachable("Unexpected intrinsic");
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getOperand(0), CI->getArgOperand(1) });
Rep = ApplyX86MaskOn1BitsVec(Builder, Rep, CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.cmp.p")) {
Type *OpTy = CI->getArgOperand(0)->getType();
unsigned VecWidth = OpTy->getPrimitiveSizeInBits();
unsigned EltWidth = OpTy->getScalarSizeInBits();
Intrinsic::ID IID;
if (VecWidth == 128 && EltWidth == 32)
IID = Intrinsic::x86_avx512_cmp_ps_128;
else if (VecWidth == 256 && EltWidth == 32)
IID = Intrinsic::x86_avx512_cmp_ps_256;
else if (VecWidth == 512 && EltWidth == 32)
IID = Intrinsic::x86_avx512_cmp_ps_512;
else if (VecWidth == 128 && EltWidth == 64)
IID = Intrinsic::x86_avx512_cmp_pd_128;
else if (VecWidth == 256 && EltWidth == 64)
IID = Intrinsic::x86_avx512_cmp_pd_256;
else if (VecWidth == 512 && EltWidth == 64)
IID = Intrinsic::x86_avx512_cmp_pd_512;
else
llvm_unreachable("Unexpected intrinsic");
SmallVector<Value *, 4> Args;
Args.push_back(CI->getArgOperand(0));
Args.push_back(CI->getArgOperand(1));
Args.push_back(CI->getArgOperand(2));
if (CI->getNumArgOperands() == 5)
Args.push_back(CI->getArgOperand(4));
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
Args);
Rep = ApplyX86MaskOn1BitsVec(Builder, Rep, CI->getArgOperand(3));
} else if (IsX86 && Name.startswith("avx512.mask.cmp.") &&
Name[16] != 'p') {
// Integer compare intrinsics.
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-22 22:11:01 +02:00
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
Rep = upgradeMaskedCompare(Builder, *CI, Imm, true);
[X86] Rename the autoupgraded of packed fp compare and fpclass intrinsics that don't take a mask as input to exclude '.mask.' from their name. I think the intrinsics named 'avx512.mask.' should refer to the previous behavior of taking a mask argument in the intrinsic instead of using a 'select' or 'and' instruction in IR to accomplish the masking. This is more consistent with the goal that eventually we will have no intrinsics that have masking builtin. When we reach that goal, we should have no intrinsics named "avx512.mask". llvm-svn: 335744
2018-06-27 17:57:53 +02:00
} else if (IsX86 && Name.startswith("avx512.mask.ucmp.")) {
[AVX-512] Remove and autoupgrade the masked integer compare intrinsics Summary: These intrinsics aren't used by clang and haven't been for a while. There's some really terrible codegen in the 32-bit target for avx512bw due to i64 not being legal. But as I said these intrinsics aren't used by clang even before this patch so this codegen reflects our clang behavior today. Reviewers: spatel, RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D34389 llvm-svn: 306047
2017-06-22 22:11:01 +02:00
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
Rep = upgradeMaskedCompare(Builder, *CI, Imm, false);
[X86] Remove llvm.x86.avx512.cvt*2mask.* intrinsics and autoupgrade to (icmp slt X, 0) I had to drop fast-isel-abort from a test because we can't fast isel some of the mask stuff. When we used intrinsics we implicitly fell back to SelectionDAG for the intrinsic call without triggering the abort error. But with native IR that doesn't happen the same way. llvm-svn: 322050
2018-01-09 01:50:47 +01:00
} else if (IsX86 && (Name.startswith("avx512.cvtb2mask.") ||
Name.startswith("avx512.cvtw2mask.") ||
Name.startswith("avx512.cvtd2mask.") ||
Name.startswith("avx512.cvtq2mask."))) {
Value *Op = CI->getArgOperand(0);
Value *Zero = llvm::Constant::getNullValue(Op->getType());
Rep = Builder.CreateICmp(ICmpInst::ICMP_SLT, Op, Zero);
[X86] Rename the autoupgraded of packed fp compare and fpclass intrinsics that don't take a mask as input to exclude '.mask.' from their name. I think the intrinsics named 'avx512.mask.' should refer to the previous behavior of taking a mask argument in the intrinsic instead of using a 'select' or 'and' instruction in IR to accomplish the masking. This is more consistent with the goal that eventually we will have no intrinsics that have masking builtin. When we reach that goal, we should have no intrinsics named "avx512.mask". llvm-svn: 335744
2018-06-27 17:57:53 +02:00
Rep = ApplyX86MaskOn1BitsVec(Builder, Rep, nullptr);
[X86] [PATCH] [intrinsics] Lowering X86 ABS intrinsics to IR. (llvm) This patch, together with a matching clang patch (https://reviews.llvm.org/D37694), implements the lowering of X86 ABS intrinsics to IR. differential revision: https://reviews.llvm.org/D37693. llvm-svn: 313134
2017-09-13 11:02:36 +02:00
} else if(IsX86 && (Name == "ssse3.pabs.b.128" ||
Name == "ssse3.pabs.w.128" ||
Name == "ssse3.pabs.d.128" ||
Name.startswith("avx2.pabs") ||
Name.startswith("avx512.mask.pabs"))) {
Rep = upgradeAbs(Builder, *CI);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse41.pmaxsb" ||
Name == "sse2.pmaxs.w" ||
Name == "sse41.pmaxsd" ||
[AVX-512] Remove masked pmin/pmax intrinsics and autoupgrade to native IR. Clang patch to replace 512-bit vector and 64-bit element versions with native IR will follow. llvm-svn: 284955
2016-10-24 06:04:16 +02:00
Name.startswith("avx2.pmaxs") ||
Name.startswith("avx512.mask.pmaxs"))) {
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 17:48:30 +02:00
Rep = upgradeIntMinMax(Builder, *CI, ICmpInst::ICMP_SGT);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse2.pmaxu.b" ||
Name == "sse41.pmaxuw" ||
Name == "sse41.pmaxud" ||
[AVX-512] Remove masked pmin/pmax intrinsics and autoupgrade to native IR. Clang patch to replace 512-bit vector and 64-bit element versions with native IR will follow. llvm-svn: 284955
2016-10-24 06:04:16 +02:00
Name.startswith("avx2.pmaxu") ||
Name.startswith("avx512.mask.pmaxu"))) {
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 17:48:30 +02:00
Rep = upgradeIntMinMax(Builder, *CI, ICmpInst::ICMP_UGT);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse41.pminsb" ||
Name == "sse2.pmins.w" ||
Name == "sse41.pminsd" ||
[AVX-512] Remove masked pmin/pmax intrinsics and autoupgrade to native IR. Clang patch to replace 512-bit vector and 64-bit element versions with native IR will follow. llvm-svn: 284955
2016-10-24 06:04:16 +02:00
Name.startswith("avx2.pmins") ||
Name.startswith("avx512.mask.pmins"))) {
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 17:48:30 +02:00
Rep = upgradeIntMinMax(Builder, *CI, ICmpInst::ICMP_SLT);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse2.pminu.b" ||
Name == "sse41.pminuw" ||
Name == "sse41.pminud" ||
[AVX-512] Remove masked pmin/pmax intrinsics and autoupgrade to native IR. Clang patch to replace 512-bit vector and 64-bit element versions with native IR will follow. llvm-svn: 284955
2016-10-24 06:04:16 +02:00
Name.startswith("avx2.pminu") ||
Name.startswith("avx512.mask.pminu"))) {
[x86] autoupgrade and remove SSE2/SSE41 integer min/max intrinsics Follow-up to: http://reviews.llvm.org/rL272806 http://reviews.llvm.org/rL272807 llvm-svn: 272907
2016-06-16 17:48:30 +02:00
Rep = upgradeIntMinMax(Builder, *CI, ICmpInst::ICMP_ULT);
[X86] Remove the pmuldq/pmuldq intrinsics and replace with native IR. This completes the work started in r329604 and r329605 when we changed clang to no longer use the intrinsics. We lost some InstCombine SimplifyDemandedBit optimizations through this change as we aren't able to fold 'and', bitcast, shuffle very well. llvm-svn: 329990
2018-04-13 08:07:18 +02:00
} else if (IsX86 && (Name == "sse2.pmulu.dq" ||
Name == "avx2.pmulu.dq" ||
Name == "avx512.pmulu.dq.512" ||
Name.startswith("avx512.mask.pmulu.dq."))) {
Rep = upgradePMULDQ(Builder, *CI, /*Signed*/false);
} else if (IsX86 && (Name == "sse41.pmuldq" ||
Name == "avx2.pmul.dq" ||
Name == "avx512.pmul.dq.512" ||
Name.startswith("avx512.mask.pmul.dq."))) {
Rep = upgradePMULDQ(Builder, *CI, /*Signed*/true);
[X86] Remove and autoupgrade cvtsi2ss/cvtsi2sd intrinsics to match what clang has used for a very long time. llvm-svn: 332186
2018-05-13 01:14:39 +02:00
} else if (IsX86 && (Name == "sse.cvtsi2ss" ||
Name == "sse2.cvtsi2sd" ||
Name == "sse.cvtsi642ss" ||
Name == "sse2.cvtsi642sd")) {
Rep = Builder.CreateSIToFP(CI->getArgOperand(1),
CI->getType()->getVectorElementType());
[X86] Remove and autoupgrade the cvtusi2sd intrinsic. Use uitofp+insertelement instead. llvm-svn: 332206
2018-05-14 02:06:49 +02:00
Rep = Builder.CreateInsertElement(CI->getArgOperand(0), Rep, (uint64_t)0);
} else if (IsX86 && Name == "avx512.cvtusi2sd") {
Rep = Builder.CreateUIToFP(CI->getArgOperand(1),
CI->getType()->getVectorElementType());
[X86] Remove and autoupgrade cvtsi2ss/cvtsi2sd intrinsics to match what clang has used for a very long time. llvm-svn: 332186
2018-05-13 01:14:39 +02:00
Rep = Builder.CreateInsertElement(CI->getArgOperand(0), Rep, (uint64_t)0);
[X86] Remove an autoupgrade legacy cvtss2sd intrinsics. llvm-svn: 332187
2018-05-13 02:29:40 +02:00
} else if (IsX86 && Name == "sse2.cvtss2sd") {
Rep = Builder.CreateExtractElement(CI->getArgOperand(1), (uint64_t)0);
Rep = Builder.CreateFPExt(Rep, CI->getType()->getVectorElementType());
Rep = Builder.CreateInsertElement(CI->getArgOperand(0), Rep, (uint64_t)0);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse2.cvtdq2pd" ||
[X86] Remove 128/256-bit cvtdq2ps, cvtudq2ps, cvtqq2pd, cvtuqq2pd intrinsics. These can all be implemented with sitofp/uitofp instructions. llvm-svn: 332916
2018-05-22 01:15:00 +02:00
Name == "sse2.cvtdq2ps" ||
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
Name == "avx.cvtdq2.pd.256" ||
[X86] Remove 128/256-bit cvtdq2ps, cvtudq2ps, cvtqq2pd, cvtuqq2pd intrinsics. These can all be implemented with sitofp/uitofp instructions. llvm-svn: 332916
2018-05-22 01:15:00 +02:00
Name == "avx.cvtdq2.ps.256" ||
Name.startswith("avx512.mask.cvtdq2pd.") ||
Name.startswith("avx512.mask.cvtudq2pd.") ||
[X86] Remove GCCBuiltins from 512-bit cvt(u)qqtops, cvt(u)qqtopd, and cvt(u)dqtops intrinsics. Add new variadic uitofp/sitofp with rounding mode intrinsics. Summary: See clang patch D56998 for a full description. Reviewers: RKSimon, spatel Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D56999 llvm-svn: 352266
2019-01-26 03:41:54 +01:00
Name.startswith("avx512.mask.cvtdq2ps.") ||
Name.startswith("avx512.mask.cvtudq2ps.") ||
Name.startswith("avx512.mask.cvtqq2pd.") ||
Name.startswith("avx512.mask.cvtuqq2pd.") ||
Name == "avx512.mask.cvtqq2ps.256" ||
Name == "avx512.mask.cvtqq2ps.512" ||
Name == "avx512.mask.cvtuqq2ps.256" ||
Name == "avx512.mask.cvtuqq2ps.512" ||
[X86] Remove 128/256-bit cvtdq2ps, cvtudq2ps, cvtqq2pd, cvtuqq2pd intrinsics. These can all be implemented with sitofp/uitofp instructions. llvm-svn: 332916
2018-05-22 01:15:00 +02:00
Name == "sse2.cvtps2pd" ||
[X86][AVX512] Autoupgrade lossless i32/u32 to f64 conversion intrinsics with generic IR Both the (V)CVTDQ2PD (i32 to f64) and (V)CVTUDQ2PD (u32 to f64) conversion instructions are lossless and can be safely represented as generic SINT_TO_FP/UINT_TO_FP calls instead of x86 intrinsics without affecting final codegen. LLVM counterpart to D26686 Differential Revision: https://reviews.llvm.org/D26736 llvm-svn: 287108
2016-11-16 15:48:32 +01:00
Name == "avx.cvt.ps2.pd.256" ||
[X86] Remove some unused masked conversion intrinsics that can be replaced with an older intrinsic and a select. This is what clang already uses. llvm-svn: 332170
2018-05-12 04:34:28 +02:00
Name == "avx512.mask.cvtps2pd.128" ||
[X86] Remove 128/256-bit cvtdq2ps, cvtudq2ps, cvtqq2pd, cvtuqq2pd intrinsics. These can all be implemented with sitofp/uitofp instructions. llvm-svn: 332916
2018-05-22 01:15:00 +02:00
Name == "avx512.mask.cvtps2pd.256")) {
Type *DstTy = CI->getType();
[X86][SSE] Replace (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) lossless conversion intrinsics with generic IR Followup to D20528 clang patch, this removes the (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) llvm intrinsics and auto-upgrades to sitofp/fpext instead. Differential Revision: http://reviews.llvm.org/D20568 llvm-svn: 270678
2016-05-25 10:59:18 +02:00
Rep = CI->getArgOperand(0);
[X86] Remove GCCBuiltins from 512-bit cvt(u)qqtops, cvt(u)qqtopd, and cvt(u)dqtops intrinsics. Add new variadic uitofp/sitofp with rounding mode intrinsics. Summary: See clang patch D56998 for a full description. Reviewers: RKSimon, spatel Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D56999 llvm-svn: 352266
2019-01-26 03:41:54 +01:00
Type *SrcTy = Rep->getType();
[X86][SSE] Replace (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) lossless conversion intrinsics with generic IR Followup to D20528 clang patch, this removes the (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) llvm intrinsics and auto-upgrades to sitofp/fpext instead. Differential Revision: http://reviews.llvm.org/D20568 llvm-svn: 270678
2016-05-25 10:59:18 +02:00
[X86] Remove 128/256-bit cvtdq2ps, cvtudq2ps, cvtqq2pd, cvtuqq2pd intrinsics. These can all be implemented with sitofp/uitofp instructions. llvm-svn: 332916
2018-05-22 01:15:00 +02:00
unsigned NumDstElts = DstTy->getVectorNumElements();
[X86] Remove GCCBuiltins from 512-bit cvt(u)qqtops, cvt(u)qqtopd, and cvt(u)dqtops intrinsics. Add new variadic uitofp/sitofp with rounding mode intrinsics. Summary: See clang patch D56998 for a full description. Reviewers: RKSimon, spatel Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D56999 llvm-svn: 352266
2019-01-26 03:41:54 +01:00
if (NumDstElts < SrcTy->getVectorNumElements()) {
[X86][SSE] Replace (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) lossless conversion intrinsics with generic IR Followup to D20528 clang patch, this removes the (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) llvm intrinsics and auto-upgrades to sitofp/fpext instead. Differential Revision: http://reviews.llvm.org/D20568 llvm-svn: 270678
2016-05-25 10:59:18 +02:00
assert(NumDstElts == 2 && "Unexpected vector size");
[IR] Require ArrayRef of 'uint32_t' instead of 'int' for the mask argument for one of the signatures of CreateShuffleVector. This better emphasises that you can't use it for the -1 as undef behavior. llvm-svn: 272491
2016-06-12 02:41:19 +02:00
uint32_t ShuffleMask[2] = { 0, 1 };
[X86] Remove 128/256-bit cvtdq2ps, cvtudq2ps, cvtqq2pd, cvtuqq2pd intrinsics. These can all be implemented with sitofp/uitofp instructions. llvm-svn: 332916
2018-05-22 01:15:00 +02:00
Rep = Builder.CreateShuffleVector(Rep, Rep, ShuffleMask);
[X86][SSE] Replace (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) lossless conversion intrinsics with generic IR Followup to D20528 clang patch, this removes the (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) llvm intrinsics and auto-upgrades to sitofp/fpext instead. Differential Revision: http://reviews.llvm.org/D20568 llvm-svn: 270678
2016-05-25 10:59:18 +02:00
}
[X86] Remove GCCBuiltins from 512-bit cvt(u)qqtops, cvt(u)qqtopd, and cvt(u)dqtops intrinsics. Add new variadic uitofp/sitofp with rounding mode intrinsics. Summary: See clang patch D56998 for a full description. Reviewers: RKSimon, spatel Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D56999 llvm-svn: 352266
2019-01-26 03:41:54 +01:00
bool IsPS2PD = SrcTy->getVectorElementType()->isFloatTy();
[X86] Remove 128/256-bit cvtdq2ps, cvtudq2ps, cvtqq2pd, cvtuqq2pd intrinsics. These can all be implemented with sitofp/uitofp instructions. llvm-svn: 332916
2018-05-22 01:15:00 +02:00
bool IsUnsigned = (StringRef::npos != Name.find("cvtu"));
if (IsPS2PD)
[X86][SSE] Replace (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) lossless conversion intrinsics with generic IR Followup to D20528 clang patch, this removes the (V)CVTDQ2PD(Y) and (V)CVTPS2PD(Y) llvm intrinsics and auto-upgrades to sitofp/fpext instead. Differential Revision: http://reviews.llvm.org/D20568 llvm-svn: 270678
2016-05-25 10:59:18 +02:00
Rep = Builder.CreateFPExt(Rep, DstTy, "cvtps2pd");
[X86] Remove GCCBuiltins from 512-bit cvt(u)qqtops, cvt(u)qqtopd, and cvt(u)dqtops intrinsics. Add new variadic uitofp/sitofp with rounding mode intrinsics. Summary: See clang patch D56998 for a full description. Reviewers: RKSimon, spatel Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D56999 llvm-svn: 352266
2019-01-26 03:41:54 +01:00
else if (CI->getNumArgOperands() == 4 &&
(!isa<ConstantInt>(CI->getArgOperand(3)) ||
cast<ConstantInt>(CI->getArgOperand(3))->getZExtValue() != 4)) {
Intrinsic::ID IID = IsUnsigned ? Intrinsic::x86_avx512_uitofp_round
: Intrinsic::x86_avx512_sitofp_round;
Function *F = Intrinsic::getDeclaration(CI->getModule(), IID,
{ DstTy, SrcTy });
Rep = Builder.CreateCall(F, { Rep, CI->getArgOperand(3) });
} else {
Rep = IsUnsigned ? Builder.CreateUIToFP(Rep, DstTy, "cvt")
: Builder.CreateSIToFP(Rep, DstTy, "cvt");
}
[X86][AVX512] Autoupgrade lossless i32/u32 to f64 conversion intrinsics with generic IR Both the (V)CVTDQ2PD (i32 to f64) and (V)CVTUDQ2PD (u32 to f64) conversion instructions are lossless and can be safely represented as generic SINT_TO_FP/UINT_TO_FP calls instead of x86 intrinsics without affecting final codegen. LLVM counterpart to D26686 Differential Revision: https://reviews.llvm.org/D26736 llvm-svn: 287108
2016-11-16 15:48:32 +01:00
[X86] Remove GCCBuiltins from 512-bit cvt(u)qqtops, cvt(u)qqtopd, and cvt(u)dqtops intrinsics. Add new variadic uitofp/sitofp with rounding mode intrinsics. Summary: See clang patch D56998 for a full description. Reviewers: RKSimon, spatel Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D56999 llvm-svn: 352266
2019-01-26 03:41:54 +01:00
if (CI->getNumArgOperands() >= 3)
[X86][AVX512] Autoupgrade lossless i32/u32 to f64 conversion intrinsics with generic IR Both the (V)CVTDQ2PD (i32 to f64) and (V)CVTUDQ2PD (u32 to f64) conversion instructions are lossless and can be safely represented as generic SINT_TO_FP/UINT_TO_FP calls instead of x86 intrinsics without affecting final codegen. LLVM counterpart to D26686 Differential Revision: https://reviews.llvm.org/D26736 llvm-svn: 287108
2016-11-16 15:48:32 +01:00
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[X86] Combine some of the strings in autoupgrade code. llvm-svn: 280603
2016-09-04 01:55:13 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.loadu."))) {
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
Rep = UpgradeMaskedLoad(Builder, CI->getArgOperand(0),
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 06:19:36 +02:00
CI->getArgOperand(1), CI->getArgOperand(2),
/*Aligned*/false);
[X86] Combine some of the strings in autoupgrade code. llvm-svn: 280603
2016-09-04 01:55:13 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.load."))) {
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
Rep = UpgradeMaskedLoad(Builder, CI->getArgOperand(0),
[AVX512] Remove masked load intrinsics. Clang now emits generic masked load intrinsics instead. The intrinsics will be autoupgraded to the same generic masked loads. llvm-svn: 271478
2016-06-02 06:19:36 +02:00
CI->getArgOperand(1),CI->getArgOperand(2),
/*Aligned*/true);
[X86] Remove and autoupgrade the expandload and compressstore intrinsics. We use the target independent intrinsics now. llvm-svn: 334381
2018-06-11 03:25:22 +02:00
} else if (IsX86 && Name.startswith("avx512.mask.expand.load.")) {
Type *ResultTy = CI->getType();
Type *PtrTy = ResultTy->getVectorElementType();
// Cast the pointer to element type.
Value *Ptr = Builder.CreateBitCast(CI->getOperand(0),
llvm::PointerType::getUnqual(PtrTy));
Value *MaskVec = getX86MaskVec(Builder, CI->getArgOperand(2),
ResultTy->getVectorNumElements());
Function *ELd = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::masked_expandload,
ResultTy);
Rep = Builder.CreateCall(ELd, { Ptr, MaskVec, CI->getOperand(1) });
} else if (IsX86 && Name.startswith("avx512.mask.compress.store.")) {
Type *ResultTy = CI->getArgOperand(1)->getType();
Type *PtrTy = ResultTy->getVectorElementType();
// Cast the pointer to element type.
Value *Ptr = Builder.CreateBitCast(CI->getOperand(0),
llvm::PointerType::getUnqual(PtrTy));
Value *MaskVec = getX86MaskVec(Builder, CI->getArgOperand(2),
ResultTy->getVectorNumElements());
Function *CSt = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::masked_compressstore,
ResultTy);
Rep = Builder.CreateCall(CSt, { CI->getArgOperand(1), Ptr, MaskVec });
[X86] Add new variadic avx512 compress/expand intrinsics that use vXi1 types for the mask argument. Remove and autoupgrade the old intrinsics llvm-svn: 352343
2019-01-28 08:03:03 +01:00
} else if (IsX86 && (Name.startswith("avx512.mask.compress.") ||
Name.startswith("avx512.mask.expand."))) {
Type *ResultTy = CI->getType();
Value *MaskVec = getX86MaskVec(Builder, CI->getArgOperand(2),
ResultTy->getVectorNumElements());
bool IsCompress = Name[12] == 'c';
Intrinsic::ID IID = IsCompress ? Intrinsic::x86_avx512_mask_compress
: Intrinsic::x86_avx512_mask_expand;
Function *Intr = Intrinsic::getDeclaration(F->getParent(), IID, ResultTy);
Rep = Builder.CreateCall(Intr, { CI->getOperand(0), CI->getOperand(1),
MaskVec });
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && Name.startswith("xop.vpcom")) {
[X86] Auto upgrade old style VPCOM/VPCOMU intrinsics to generic integer comparisons We were upgrading these to the new style VPCOM/VPCOMU intrinsics (which includes the condition code immediate), but we'll be getting rid of those shortly, so convert these to generics first. This causes a couple of changes in the upgrade tests as signed/unsigned eq/ne are equivalent and we constant fold true/false codes, these changes are the same as what we already do for avx512 cmp/ucmp. Noticed while cleaning up vector integer comparison costs for PR40376. llvm-svn: 351690
2019-01-20 18:36:22 +01:00
bool IsSigned;
if (Name.endswith("ub") || Name.endswith("uw") || Name.endswith("ud") ||
Name.endswith("uq"))
IsSigned = false;
else if (Name.endswith("b") || Name.endswith("w") || Name.endswith("d") ||
Name.endswith("q"))
IsSigned = true;
Replace XOP vpcom intrinsics with fewer intrinsics that take the immediate as an argument. llvm-svn: 158278
2012-06-09 18:46:13 +02:00
else
llvm_unreachable("Unknown suffix");
unsigned Imm;
[X86] Auto upgrade VPCOM/VPCOMU intrinsics to generic integer comparisons This causes a couple of changes in the upgrade tests as signed/unsigned eq/ne are equivalent and we constant fold true/false codes, these changes are the same as what we already do for avx512 cmp/ucmp. Noticed while cleaning up vector integer comparison costs for PR40376. llvm-svn: 351697
2019-01-20 20:27:40 +01:00
if (CI->getNumArgOperands() == 3) {
Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
} else {
Name = Name.substr(9); // strip off "xop.vpcom"
if (Name.startswith("lt"))
Imm = 0;
else if (Name.startswith("le"))
Imm = 1;
else if (Name.startswith("gt"))
Imm = 2;
else if (Name.startswith("ge"))
Imm = 3;
else if (Name.startswith("eq"))
Imm = 4;
else if (Name.startswith("ne"))
Imm = 5;
else if (Name.startswith("false"))
Imm = 6;
else if (Name.startswith("true"))
Imm = 7;
else
llvm_unreachable("Unknown condition");
}
[X86] Auto upgrade old style VPCOM/VPCOMU intrinsics to generic integer comparisons We were upgrading these to the new style VPCOM/VPCOMU intrinsics (which includes the condition code immediate), but we'll be getting rid of those shortly, so convert these to generics first. This causes a couple of changes in the upgrade tests as signed/unsigned eq/ne are equivalent and we constant fold true/false codes, these changes are the same as what we already do for avx512 cmp/ucmp. Noticed while cleaning up vector integer comparison costs for PR40376. llvm-svn: 351690
2019-01-20 18:36:22 +01:00
Rep = upgradeX86vpcom(Builder, *CI, Imm, IsSigned);
Recommit "[X86] Remove XOP VPCMOV intrinsics and autoupgrade them to native IR." Clang has now been fixed to not use these intrinsics. llvm-svn: 295571
2017-02-18 22:50:58 +01:00
} else if (IsX86 && Name.startswith("xop.vpcmov")) {
[X86][XOP] Add support for the matching of the VPCMOV bit select instruction XOP has the VPCMOV instruction that performs the common vector bit select operation OR( AND( SRC1, SRC3 ), AND( SRC2, ~SRC3 ) ) This patch adds tablegen pattern matching for this instruction. Differential Revision: http://reviews.llvm.org/D8841 llvm-svn: 251975
2015-11-03 21:27:01 +01:00
Value *Sel = CI->getArgOperand(2);
[X86][IR] Simplify the XOP vpcmov autoupgrade code. NFC llvm-svn: 295563
2017-02-18 20:51:19 +01:00
Value *NotSel = Builder.CreateNot(Sel);
Value *Sel0 = Builder.CreateAnd(CI->getArgOperand(0), Sel);
Value *Sel1 = Builder.CreateAnd(CI->getArgOperand(1), NotSel);
[X86][XOP] Add support for the matching of the VPCMOV bit select instruction XOP has the VPCMOV instruction that performs the common vector bit select operation OR( AND( SRC1, SRC3 ), AND( SRC2, ~SRC3 ) ) This patch adds tablegen pattern matching for this instruction. Differential Revision: http://reviews.llvm.org/D8841 llvm-svn: 251975
2015-11-03 21:27:01 +01:00
Rep = Builder.CreateOr(Sel0, Sel1);
[X86] Auto upgrade XOP/AVX512 rotation intrinsics to generic funnel shift intrinsics (llvm) This emits FSHL/FSHR generic intrinsics for the XOP VPROT and AVX512 VPROL/VPROR rotation intrinsics. Clang counterpart: https://reviews.llvm.org/D55937 Differential Revision: https://reviews.llvm.org/D55938 llvm-svn: 349795
2018-12-20 20:01:07 +01:00
} else if (IsX86 && (Name.startswith("xop.vprot") ||
Name.startswith("avx512.prol") ||
Name.startswith("avx512.mask.prol"))) {
Rep = upgradeX86Rotate(Builder, *CI, false);
} else if (IsX86 && (Name.startswith("avx512.pror") ||
Name.startswith("avx512.mask.pror"))) {
Rep = upgradeX86Rotate(Builder, *CI, true);
Recommit r350554 "[X86] Remove AVX512VBMI2 concat and shift intrinsics. Replace with target independent funnel shift intrinsics." The MSVC limit we hit on AutoUpgrade.cpp has been worked around for now. llvm-svn: 350567
2019-01-07 22:00:32 +01:00
} else if (IsX86 && (Name.startswith("avx512.vpshld.") ||
Name.startswith("avx512.mask.vpshld") ||
Name.startswith("avx512.maskz.vpshld"))) {
bool ZeroMask = Name[11] == 'z';
Rep = upgradeX86ConcatShift(Builder, *CI, false, ZeroMask);
} else if (IsX86 && (Name.startswith("avx512.vpshrd.") ||
Name.startswith("avx512.mask.vpshrd") ||
Name.startswith("avx512.maskz.vpshrd"))) {
bool ZeroMask = Name[11] == 'z';
Rep = upgradeX86ConcatShift(Builder, *CI, true, ZeroMask);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && Name == "sse42.crc32.64.8") {
Remove x86_sse42_crc32_64_8 intrinsic. It has no functional difference from x86_sse42_crc32_32_8 and was not mapped to a clang builtin. I'm not even sure why this form of the instruction is even called out explicitly in the docs. Also add AutoUpgrade support to convert it into the other intrinsic with appropriate trunc and zext. llvm-svn: 192672
2013-10-15 07:20:47 +02:00
Function *CRC32 = Intrinsic::getDeclaration(F->getParent(),
Intrinsic::x86_sse42_crc32_32_8);
Value *Trunc0 = Builder.CreateTrunc(CI->getArgOperand(0), Type::getInt32Ty(C));
Simplify IRBuilder::CreateCall* by using ArrayRef+initializer_list/braced init only llvm-svn: 237624
2015-05-19 00:13:54 +02:00
Rep = Builder.CreateCall(CRC32, {Trunc0, CI->getArgOperand(1)});
Remove x86_sse42_crc32_64_8 intrinsic. It has no functional difference from x86_sse42_crc32_32_8 and was not mapped to a clang builtin. I'm not even sure why this form of the instruction is even called out explicitly in the docs. Also add AutoUpgrade support to convert it into the other intrinsic with appropriate trunc and zext. llvm-svn: 192672
2013-10-15 07:20:47 +02:00
Rep = Builder.CreateZExt(Rep, CI->getType(), "");
[X86] Remove and autoupgrade avx512.vbroadcast.ss/avx512.vbroadcast.sd intrinsics. llvm-svn: 332271
2018-05-14 20:21:22 +02:00
} else if (IsX86 && (Name.startswith("avx.vbroadcast.s") ||
Name.startswith("avx512.vbroadcast.s"))) {
[X86] Auto-upgrade AVX1 vbroadcast intrinsics They are replaced with the same IR that is generated for the vector-initializers in avxintrin.h. The test verifies that we get back the original instruction. I haven't seen this approach to be used in other auto-upgrade tests (i.e. llc + FileCheck) but I think it's the most direct way to test this case. I believe this should work because llc upgrades calls during parsing. (Other tests mostly check that assembling and disassembling yields the upgraded IR.) llvm-svn: 209863
2014-05-30 01:35:33 +02:00
// Replace broadcasts with a series of insertelements.
Type *VecTy = CI->getType();
Type *EltTy = VecTy->getVectorElementType();
unsigned EltNum = VecTy->getVectorNumElements();
Value *Cast = Builder.CreateBitCast(CI->getArgOperand(0),
EltTy->getPointerTo());
[opaque pointer type] Pass explicit type to Load instruction creation in AutoUpgrade llvm-svn: 237838
2015-05-20 23:46:30 +02:00
Value *Load = Builder.CreateLoad(EltTy, Cast);
[X86] Auto-upgrade AVX1 vbroadcast intrinsics They are replaced with the same IR that is generated for the vector-initializers in avxintrin.h. The test verifies that we get back the original instruction. I haven't seen this approach to be used in other auto-upgrade tests (i.e. llc + FileCheck) but I think it's the most direct way to test this case. I believe this should work because llc upgrades calls during parsing. (Other tests mostly check that assembling and disassembling yields the upgraded IR.) llvm-svn: 209863
2014-05-30 01:35:33 +02:00
Type *I32Ty = Type::getInt32Ty(C);
Rep = UndefValue::get(VecTy);
for (unsigned I = 0; I < EltNum; ++I)
Rep = Builder.CreateInsertElement(Rep, Load,
ConstantInt::get(I32Ty, I));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name.startswith("sse41.pmovsx") ||
Name.startswith("sse41.pmovzx") ||
Name.startswith("avx2.pmovsx") ||
[AVX-512] Remove masked pmovzx/pmovsx builtins and autoupgrade them to selects and native zext/sext. This mostly reuses earlier autoupgrade support for the sse and avx equivalents. Just needed to add the code to add the select. llvm-svn: 286092
2016-11-07 03:12:57 +01:00
Name.startswith("avx2.pmovzx") ||
Name.startswith("avx512.mask.pmovsx") ||
Name.startswith("avx512.mask.pmovzx"))) {
[X86][SSE] Replace 128-bit SSE41 PMOVSX intrinsics with native IR This patches removes the x86.sse41.pmovsx* intrinsics, provides a suitable upgrade path and updates relevant tests to sign extend a subvector instead. LLVM counterpart to D12835 Differential Revision: http://reviews.llvm.org/D13002 llvm-svn: 248368
2015-09-23 10:48:33 +02:00
VectorType *SrcTy = cast<VectorType>(CI->getArgOperand(0)->getType());
VectorType *DstTy = cast<VectorType>(CI->getType());
unsigned NumDstElts = DstTy->getNumElements();
[X86][SSE] (Reapplied) Replace (V)PMOVSX and (V)PMOVZX integer extension intrinsics with generic IR (llvm) This patch removes the llvm intrinsics VPMOVSX and (V)PMOVZX sign/zero extension intrinsics and auto-upgrades to SEXT/ZEXT calls instead. We already did this for SSE41 PMOVSX sometime ago so much of that implementation can be reused. Reapplied now that the the companion patch (D20684) removes/auto-upgrade the clang intrinsics has been committed. Differential Revision: http://reviews.llvm.org/D20686 llvm-svn: 271131
2016-05-28 20:03:41 +02:00
// Extract a subvector of the first NumDstElts lanes and sign/zero extend.
[X86] Pre-allocate some of the shuffle mask SmallVectors in the auto upgrade code instead of calling push_back in a loop. This removes the need to check if the vector needs to grow on each iteration. llvm-svn: 272501
2016-06-12 06:48:00 +02:00
SmallVector<uint32_t, 8> ShuffleMask(NumDstElts);
[IR] Require ArrayRef of 'uint32_t' instead of 'int' for the mask argument for one of the signatures of CreateShuffleVector. This better emphasises that you can't use it for the -1 as undef behavior. llvm-svn: 272491
2016-06-12 02:41:19 +02:00
for (unsigned i = 0; i != NumDstElts; ++i)
[X86] Pre-allocate some of the shuffle mask SmallVectors in the auto upgrade code instead of calling push_back in a loop. This removes the need to check if the vector needs to grow on each iteration. llvm-svn: 272501
2016-06-12 06:48:00 +02:00
ShuffleMask[i] = i;
[X86][SSE] Replace 128-bit SSE41 PMOVSX intrinsics with native IR This patches removes the x86.sse41.pmovsx* intrinsics, provides a suitable upgrade path and updates relevant tests to sign extend a subvector instead. LLVM counterpart to D12835 Differential Revision: http://reviews.llvm.org/D13002 llvm-svn: 248368
2015-09-23 10:48:33 +02:00
Value *SV = Builder.CreateShuffleVector(
CI->getArgOperand(0), UndefValue::get(SrcTy), ShuffleMask);
[X86][SSE] (Reapplied) Replace (V)PMOVSX and (V)PMOVZX integer extension intrinsics with generic IR (llvm) This patch removes the llvm intrinsics VPMOVSX and (V)PMOVZX sign/zero extension intrinsics and auto-upgrades to SEXT/ZEXT calls instead. We already did this for SSE41 PMOVSX sometime ago so much of that implementation can be reused. Reapplied now that the the companion patch (D20684) removes/auto-upgrade the clang intrinsics has been committed. Differential Revision: http://reviews.llvm.org/D20686 llvm-svn: 271131
2016-05-28 20:03:41 +02:00
bool DoSext = (StringRef::npos != Name.find("pmovsx"));
Rep = DoSext ? Builder.CreateSExt(SV, DstTy)
: Builder.CreateZExt(SV, DstTy);
[AVX-512] Remove masked pmovzx/pmovsx builtins and autoupgrade them to selects and native zext/sext. This mostly reuses earlier autoupgrade support for the sse and avx equivalents. Just needed to add the code to add the select. llvm-svn: 286092
2016-11-07 03:12:57 +01:00
// If there are 3 arguments, it's a masked intrinsic so we need a select.
if (CI->getNumArgOperands() == 3)
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[X86] Remove and autoupgrade vpmovqd/vpmovwb intrinsics using trunc+select. llvm-svn: 351729
2019-01-21 09:16:59 +01:00
} else if (Name == "avx512.mask.pmov.qd.256" ||
Name == "avx512.mask.pmov.qd.512" ||
Name == "avx512.mask.pmov.wb.256" ||
Name == "avx512.mask.pmov.wb.512") {
Type *Ty = CI->getArgOperand(1)->getType();
Rep = Builder.CreateTrunc(CI->getArgOperand(0), Ty);
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[X86][AVX] Added support for lowering to VBROADCASTF128/VBROADCASTI128 (reapplied) As reported on PR26235, we don't currently make use of the VBROADCASTF128/VBROADCASTI128 instructions (or the AVX512 equivalents) to load+splat a 128-bit vector to both lanes of a 256-bit vector. This patch enables lowering from subvector insertion/concatenation patterns and auto-upgrades the llvm.x86.avx.vbroadcastf128.pd.256 / llvm.x86.avx.vbroadcastf128.ps.256 intrinsics to match. We could possibly investigate using VBROADCASTF128/VBROADCASTI128 to load repeated constants as well (similar to how we already do for scalar broadcasts). Reapplied with fix for PR28657 - removed intrinsic definitions (clang companion patch to be be submitted shortly). Differential Revision: https://reviews.llvm.org/D22460 llvm-svn: 276416
2016-07-22 15:58:44 +02:00
} else if (IsX86 && (Name.startswith("avx.vbroadcastf128") ||
Name == "avx2.vbroadcasti128")) {
// Replace vbroadcastf128/vbroadcasti128 with a vector load+shuffle.
Type *EltTy = CI->getType()->getVectorElementType();
unsigned NumSrcElts = 128 / EltTy->getPrimitiveSizeInBits();
Type *VT = VectorType::get(EltTy, NumSrcElts);
[opaque pointer type] Pass explicit type to Load instruction creation in AutoUpgrade llvm-svn: 237838
2015-05-20 23:46:30 +02:00
Value *Op = Builder.CreatePointerCast(CI->getArgOperand(0),
PointerType::getUnqual(VT));
[Alignement][NFC] Deprecate untyped CreateAlignedLoad Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet Subscribers: arsenm, jvesely, nhaehnle, hiraditya, kerbowa, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D73260
2020-01-23 11:33:12 +01:00
Value *Load = Builder.CreateAlignedLoad(VT, Op, Align(1));
[X86][AVX] Added support for lowering to VBROADCASTF128/VBROADCASTI128 (reapplied) As reported on PR26235, we don't currently make use of the VBROADCASTF128/VBROADCASTI128 instructions (or the AVX512 equivalents) to load+splat a 128-bit vector to both lanes of a 256-bit vector. This patch enables lowering from subvector insertion/concatenation patterns and auto-upgrades the llvm.x86.avx.vbroadcastf128.pd.256 / llvm.x86.avx.vbroadcastf128.ps.256 intrinsics to match. We could possibly investigate using VBROADCASTF128/VBROADCASTI128 to load repeated constants as well (similar to how we already do for scalar broadcasts). Reapplied with fix for PR28657 - removed intrinsic definitions (clang companion patch to be be submitted shortly). Differential Revision: https://reviews.llvm.org/D22460 llvm-svn: 276416
2016-07-22 15:58:44 +02:00
if (NumSrcElts == 2)
Rep = Builder.CreateShuffleVector(Load, UndefValue::get(Load->getType()),
{ 0, 1, 0, 1 });
else
Rep = Builder.CreateShuffleVector(Load, UndefValue::get(Load->getType()),
{ 0, 1, 2, 3, 0, 1, 2, 3 });
[x86][AVX512] Lowering shuffle i/f intrinsics to LLVM IR This patch, together with a matching clang patch (https://reviews.llvm.org/D38672), implements the lowering of X86 shuffle i/f intrinsics to IR. Differential Revision: https://reviews.llvm.org/D38671 Change-Id: I1e7d359a74743e995ec356237a85214ce55d3661 llvm-svn: 318026
2017-11-13 10:16:39 +01:00
} else if (IsX86 && (Name.startswith("avx512.mask.shuf.i") ||
Name.startswith("avx512.mask.shuf.f"))) {
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
Type *VT = CI->getType();
unsigned NumLanes = VT->getPrimitiveSizeInBits() / 128;
unsigned NumElementsInLane = 128 / VT->getScalarSizeInBits();
unsigned ControlBitsMask = NumLanes - 1;
unsigned NumControlBits = NumLanes / 2;
SmallVector<uint32_t, 8> ShuffleMask(0);
for (unsigned l = 0; l != NumLanes; ++l) {
unsigned LaneMask = (Imm >> (l * NumControlBits)) & ControlBitsMask;
// We actually need the other source.
if (l >= NumLanes / 2)
LaneMask += NumLanes;
for (unsigned i = 0; i != NumElementsInLane; ++i)
ShuffleMask.push_back(LaneMask * NumElementsInLane + i);
}
Rep = Builder.CreateShuffleVector(CI->getArgOperand(0),
CI->getArgOperand(1), ShuffleMask);
Rep = EmitX86Select(Builder, CI->getArgOperand(4), Rep,
CI->getArgOperand(3));
}else if (IsX86 && (Name.startswith("avx512.mask.broadcastf") ||
[AVX512] Remove and autoupgrade many of the broadcast intrinsics Summary: This autoupgrades most of the broadcast intrinsics. They've been unused in clang for some time. This leaves the 32x2 intrinsics because they are still used in clang. Reviewers: RKSimon, zvi, igorb Reviewed By: RKSimon Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D36606 llvm-svn: 310725
2017-08-11 18:22:45 +02:00
Name.startswith("avx512.mask.broadcasti"))) {
unsigned NumSrcElts =
CI->getArgOperand(0)->getType()->getVectorNumElements();
unsigned NumDstElts = CI->getType()->getVectorNumElements();
SmallVector<uint32_t, 8> ShuffleMask(NumDstElts);
for (unsigned i = 0; i != NumDstElts; ++i)
ShuffleMask[i] = i % NumSrcElts;
Rep = Builder.CreateShuffleVector(CI->getArgOperand(0),
CI->getArgOperand(0),
ShuffleMask);
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name.startswith("avx2.pbroadcast") ||
[X86][AVX512] Autoupgrade the BROADCAST intrinsics llvm-svn: 274550
2016-07-05 15:58:47 +02:00
Name.startswith("avx2.vbroadcast") ||
Name.startswith("avx512.pbroadcast") ||
Name.startswith("avx512.mask.broadcast.s"))) {
[X86] Replace avx2 broadcast intrinsics with native IR. Since r245605, the clang headers don't use these anymore. r245165 updated some of the tests already; update the others, add an autoupgrade, remove the intrinsics, and cleanup the definitions. Differential Revision: http://reviews.llvm.org/D10555 llvm-svn: 245606
2015-08-20 22:36:19 +02:00
// Replace vp?broadcasts with a vector shuffle.
Value *Op = CI->getArgOperand(0);
unsigned NumElts = CI->getType()->getVectorNumElements();
Type *MaskTy = VectorType::get(Type::getInt32Ty(C), NumElts);
Rep = Builder.CreateShuffleVector(Op, UndefValue::get(Op->getType()),
Constant::getNullValue(MaskTy));
[x86] Revert r330322 (& r330323): Lowering x86 adds/addus/subs/subus intrinsics The LLVM commit introduces a crash in LLVM's instruction selection. I filed http://llvm.org/PR37260 with the test case. llvm-svn: 330997
2018-04-26 23:46:01 +02:00
[X86][AVX512] Autoupgrade the BROADCAST intrinsics llvm-svn: 274550
2016-07-05 15:58:47 +02:00
if (CI->getNumArgOperands() == 3)
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[X86][SSE] Auto upgrade PADDS/PSUBS intrinsics to SADD_SAT/SSUB_SAT generic intrinsics (llvm) This auto upgrades the signed SSE saturated math intrinsics to SADD_SAT/SSUB_SAT generic intrinsics. Clang counterpart: https://reviews.llvm.org/D55890 Differential Revision: https://reviews.llvm.org/D55894 llvm-svn: 349892
2018-12-21 10:04:14 +01:00
} else if (IsX86 && (Name.startswith("sse2.padds.") ||
Name.startswith("sse2.psubs.") ||
Name.startswith("avx2.padds.") ||
Name.startswith("avx2.psubs.") ||
Name.startswith("avx512.padds.") ||
Name.startswith("avx512.psubs.") ||
Name.startswith("avx512.mask.padds.") ||
Name.startswith("avx512.mask.psubs."))) {
bool IsAdd = Name.contains(".padds");
Rep = UpgradeX86AddSubSatIntrinsics(Builder, *CI, true, IsAdd);
[X86] Remove the unused masked 128 and 256-bit masked padds/psubs intrinsics. Still need to remove masking from the 512-bit versions. llvm-svn: 339841
2018-08-16 08:20:22 +02:00
} else if (IsX86 && (Name.startswith("sse2.paddus.") ||
Name.startswith("sse2.psubus.") ||
Name.startswith("avx2.paddus.") ||
Name.startswith("avx2.psubus.") ||
Name.startswith("avx512.mask.paddus.") ||
Name.startswith("avx512.mask.psubus."))) {
Fix MSVC "compiler limit: blocks nested too deeply" error. NFCI. MSVC only accepts if-else chains up to 127 blocks long. I've had to merge a number of intrinsic cases together to get back below this limit, resulting in some duplication of string matches; this shouldn't cause any notable increase in runtime (and even then only for old IR, nothing that clang currently emits). llvm-svn: 339666
2018-08-14 12:04:14 +02:00
bool IsAdd = Name.contains(".paddus");
[X86][SSE] Auto upgrade PADDS/PSUBS intrinsics to SADD_SAT/SSUB_SAT generic intrinsics (llvm) This auto upgrades the signed SSE saturated math intrinsics to SADD_SAT/SSUB_SAT generic intrinsics. Clang counterpart: https://reviews.llvm.org/D55890 Differential Revision: https://reviews.llvm.org/D55894 llvm-svn: 349892
2018-12-21 10:04:14 +01:00
Rep = UpgradeX86AddSubSatIntrinsics(Builder, *CI, false, IsAdd);
Fix MSVC "compiler limit: blocks nested too deeply" error. NFCI. MSVC only accepts if-else chains up to 127 blocks long. I've had to merge a number of intrinsic cases together to get back below this limit, resulting in some duplication of string matches; this shouldn't cause any notable increase in runtime (and even then only for old IR, nothing that clang currently emits). llvm-svn: 339666
2018-08-14 12:04:14 +02:00
} else if (IsX86 && Name.startswith("avx512.mask.palignr.")) {
[AVX-512] Remove intrinsics for valignd/q and autoupgrade them to native shuffles. llvm-svn: 287744
2016-11-23 07:54:55 +01:00
Rep = UpgradeX86ALIGNIntrinsics(Builder, CI->getArgOperand(0),
CI->getArgOperand(1),
CI->getArgOperand(2),
CI->getArgOperand(3),
CI->getArgOperand(4),
false);
} else if (IsX86 && Name.startswith("avx512.mask.valign.")) {
Rep = UpgradeX86ALIGNIntrinsics(Builder, CI->getArgOperand(0),
CI->getArgOperand(1),
CI->getArgOperand(2),
CI->getArgOperand(3),
CI->getArgOperand(4),
true);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse2.psll.dq" ||
Name == "avx2.psll.dq")) {
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 08:37:33 +02:00
// 128/256-bit shift left specified in bits.
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
unsigned Shift = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
Rep = UpgradeX86PSLLDQIntrinsics(Builder, CI->getArgOperand(0),
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
Shift / 8); // Shift is in bits.
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse2.psrl.dq" ||
Name == "avx2.psrl.dq")) {
[X86] Simplify some of the autoupgrade code. NFC llvm-svn: 271174
2016-05-29 08:37:33 +02:00
// 128/256-bit shift right specified in bits.
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
unsigned Shift = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
Rep = UpgradeX86PSRLDQIntrinsics(Builder, CI->getArgOperand(0),
[X86] Remove AVX2 and SSE2 pslldq and psrldq intrinsics. We can represent them in IR with vector shuffles now. All their uses have been removed from clang in favor of shuffles. llvm-svn: 229640
2015-02-18 07:24:44 +01:00
Shift / 8); // Shift is in bits.
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse2.psll.dq.bs" ||
Name == "avx2.psll.dq.bs" ||
Name == "avx512.psll.dq.512")) {
[X86][AVX512] Dropped avx512 VPSLLDQ/VPSRLDQ intrinsics Auto-upgrade to generic shuffles like sse/avx2 implementations now that we can lower to VPSLLDQ/VPSRLDQ llvm-svn: 272308
2016-06-09 23:09:03 +02:00
// 128/256/512-bit shift left specified in bytes.
[X86] Remove x86.avx2.psll.dq.bs and x86.avx2.psrl.dq.bs intrinsics. llvm-svn: 229430
2015-02-16 21:51:59 +01:00
unsigned Shift = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
Rep = UpgradeX86PSLLDQIntrinsics(Builder, CI->getArgOperand(0), Shift);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse2.psrl.dq.bs" ||
Name == "avx2.psrl.dq.bs" ||
Name == "avx512.psrl.dq.512")) {
[X86][AVX512] Dropped avx512 VPSLLDQ/VPSRLDQ intrinsics Auto-upgrade to generic shuffles like sse/avx2 implementations now that we can lower to VPSLLDQ/VPSRLDQ llvm-svn: 272308
2016-06-09 23:09:03 +02:00
// 128/256/512-bit shift right specified in bytes.
[X86] Remove x86.avx2.psll.dq.bs and x86.avx2.psrl.dq.bs intrinsics. llvm-svn: 229430
2015-02-16 21:51:59 +01:00
unsigned Shift = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
[X86,IR] Remove unnecessary or unused LLVMContext parameter from some of the X86 intrinsic upgrade functions. llvm-svn: 275138
2016-07-12 03:42:33 +02:00
Rep = UpgradeX86PSRLDQIntrinsics(Builder, CI->getArgOperand(0), Shift);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse41.pblendw" ||
Name.startswith("sse41.blendp") ||
Name.startswith("avx.blend.p") ||
Name == "avx2.pblendw" ||
Name.startswith("avx2.pblendd."))) {
[X86] Remove the blendpd/blendps/pblendw/pblendd intrinsics. They can represented by shuffle_vector instructions. llvm-svn: 230860
2015-02-28 20:33:17 +01:00
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
unsigned Imm = cast <ConstantInt>(CI->getArgOperand(2))->getZExtValue();
VectorType *VecTy = cast<VectorType>(CI->getType());
unsigned NumElts = VecTy->getNumElements();
[X86] Pre-allocate some of the shuffle mask SmallVectors in the auto upgrade code instead of calling push_back in a loop. This removes the need to check if the vector needs to grow on each iteration. llvm-svn: 272501
2016-06-12 06:48:00 +02:00
SmallVector<uint32_t, 16> Idxs(NumElts);
for (unsigned i = 0; i != NumElts; ++i)
Idxs[i] = ((Imm >> (i%8)) & 1) ? i + NumElts : i;
[X86] Remove the blendpd/blendps/pblendw/pblendd intrinsics. They can represented by shuffle_vector instructions. llvm-svn: 230860
2015-02-28 20:33:17 +01:00
[X86,IR] Make use of the CreateShuffleVector form that takes an ArrayRef<uint32_t> to avoid the need to manually create a bunch of Constants and a ConstantVector. NFC llvm-svn: 272493
2016-06-12 03:05:59 +02:00
Rep = Builder.CreateShuffleVector(Op0, Op1, Idxs);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name.startswith("avx.vinsertf128.") ||
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 06:45:57 +01:00
Name == "avx2.vinserti128" ||
Name.startswith("avx512.mask.insert"))) {
[X86, AVX] replace vinsertf128 intrinsics with generic shuffles We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is the sibling patch for the Clang half of this change: http://reviews.llvm.org/D8088 Differential Revision: http://reviews.llvm.org/D8086 llvm-svn: 231794
2015-03-10 17:08:36 +01:00
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 06:45:57 +01:00
unsigned DstNumElts = CI->getType()->getVectorNumElements();
unsigned SrcNumElts = Op1->getType()->getVectorNumElements();
unsigned Scale = DstNumElts / SrcNumElts;
[X86][SSE] Replace 128-bit SSE41 PMOVSX intrinsics with native IR This patches removes the x86.sse41.pmovsx* intrinsics, provides a suitable upgrade path and updates relevant tests to sign extend a subvector instead. LLVM counterpart to D12835 Differential Revision: http://reviews.llvm.org/D13002 llvm-svn: 248368
2015-09-23 10:48:33 +02:00
[X86, AVX] replace vinsertf128 intrinsics with generic shuffles We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is the sibling patch for the Clang half of this change: http://reviews.llvm.org/D8088 Differential Revision: http://reviews.llvm.org/D8086 llvm-svn: 231794
2015-03-10 17:08:36 +01:00
// Mask off the high bits of the immediate value; hardware ignores those.
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 06:45:57 +01:00
Imm = Imm % Scale;
[X86][SSE] Replace 128-bit SSE41 PMOVSX intrinsics with native IR This patches removes the x86.sse41.pmovsx* intrinsics, provides a suitable upgrade path and updates relevant tests to sign extend a subvector instead. LLVM counterpart to D12835 Differential Revision: http://reviews.llvm.org/D13002 llvm-svn: 248368
2015-09-23 10:48:33 +02:00
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 06:45:57 +01:00
// Extend the second operand into a vector the size of the destination.
[X86, AVX] replace vinsertf128 intrinsics with generic shuffles We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is the sibling patch for the Clang half of this change: http://reviews.llvm.org/D8088 Differential Revision: http://reviews.llvm.org/D8086 llvm-svn: 231794
2015-03-10 17:08:36 +01:00
Value *UndefV = UndefValue::get(Op1->getType());
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 06:45:57 +01:00
SmallVector<uint32_t, 8> Idxs(DstNumElts);
for (unsigned i = 0; i != SrcNumElts; ++i)
[X86] Pre-allocate some of the shuffle mask SmallVectors in the auto upgrade code instead of calling push_back in a loop. This removes the need to check if the vector needs to grow on each iteration. llvm-svn: 272501
2016-06-12 06:48:00 +02:00
Idxs[i] = i;
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 06:45:57 +01:00
for (unsigned i = SrcNumElts; i != DstNumElts; ++i)
Idxs[i] = SrcNumElts;
[X86,IR] Make use of the CreateShuffleVector form that takes an ArrayRef<uint32_t> to avoid the need to manually create a bunch of Constants and a ConstantVector. NFC llvm-svn: 272493
2016-06-12 03:05:59 +02:00
Rep = Builder.CreateShuffleVector(Op1, UndefV, Idxs);
[X86, AVX] replace vinsertf128 intrinsics with generic shuffles We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is the sibling patch for the Clang half of this change: http://reviews.llvm.org/D8088 Differential Revision: http://reviews.llvm.org/D8086 llvm-svn: 231794
2015-03-10 17:08:36 +01:00
// Insert the second operand into the first operand.
// Note that there is no guarantee that instruction lowering will actually
// produce a vinsertf128 instruction for the created shuffles. In
// particular, the 0 immediate case involves no lane changes, so it can
// be handled as a blend.
// Example of shuffle mask for 32-bit elements:
// Imm = 1 <i32 0, i32 1, i32 2, i32 3, i32 8, i32 9, i32 10, i32 11>
// Imm = 0 <i32 8, i32 9, i32 10, i32 11, i32 4, i32 5, i32 6, i32 7 >
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 06:45:57 +01:00
// First fill with identify mask.
for (unsigned i = 0; i != DstNumElts; ++i)
Idxs[i] = i;
// Then replace the elements where we need to insert.
for (unsigned i = 0; i != SrcNumElts; ++i)
Idxs[i + Imm * SrcNumElts] = i + DstNumElts;
[X86,IR] Make use of the CreateShuffleVector form that takes an ArrayRef<uint32_t> to avoid the need to manually create a bunch of Constants and a ConstantVector. NFC llvm-svn: 272493
2016-06-12 03:05:59 +02:00
Rep = Builder.CreateShuffleVector(Op0, Rep, Idxs);
[AVX-512] Remove vinsert intrinsics and autoupgrade to native shufflevectors. There are some codegen problems here that I'll try to fix in future commits. llvm-svn: 290864
2017-01-03 06:45:57 +01:00
// If the intrinsic has a mask operand, handle that.
if (CI->getNumArgOperands() == 5)
Rep = EmitX86Select(Builder, CI->getArgOperand(4), Rep,
CI->getArgOperand(3));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name.startswith("avx.vextractf128.") ||
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 06:45:46 +01:00
Name == "avx2.vextracti128" ||
Name.startswith("avx512.mask.vextract"))) {
[X86, AVX] replace vextractf128 intrinsics with generic shuffles Now that we've replaced the vinsertf128 intrinsics, do the same for their extract twins. This is very much like D8086 (checked in at r231794): We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is also the LLVM sibling to the cfe D8275 patch. Differential Revision: http://reviews.llvm.org/D8276 llvm-svn: 232045
2015-03-12 16:15:19 +01:00
Value *Op0 = CI->getArgOperand(0);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 06:45:46 +01:00
unsigned DstNumElts = CI->getType()->getVectorNumElements();
unsigned SrcNumElts = Op0->getType()->getVectorNumElements();
unsigned Scale = SrcNumElts / DstNumElts;
[X86][SSE] Replace 128-bit SSE41 PMOVSX intrinsics with native IR This patches removes the x86.sse41.pmovsx* intrinsics, provides a suitable upgrade path and updates relevant tests to sign extend a subvector instead. LLVM counterpart to D12835 Differential Revision: http://reviews.llvm.org/D13002 llvm-svn: 248368
2015-09-23 10:48:33 +02:00
[X86, AVX] replace vextractf128 intrinsics with generic shuffles Now that we've replaced the vinsertf128 intrinsics, do the same for their extract twins. This is very much like D8086 (checked in at r231794): We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is also the LLVM sibling to the cfe D8275 patch. Differential Revision: http://reviews.llvm.org/D8276 llvm-svn: 232045
2015-03-12 16:15:19 +01:00
// Mask off the high bits of the immediate value; hardware ignores those.
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 06:45:46 +01:00
Imm = Imm % Scale;
[X86, AVX] replace vextractf128 intrinsics with generic shuffles Now that we've replaced the vinsertf128 intrinsics, do the same for their extract twins. This is very much like D8086 (checked in at r231794): We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is also the LLVM sibling to the cfe D8275 patch. Differential Revision: http://reviews.llvm.org/D8276 llvm-svn: 232045
2015-03-12 16:15:19 +01:00
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 06:45:46 +01:00
// Get indexes for the subvector of the input vector.
SmallVector<uint32_t, 8> Idxs(DstNumElts);
for (unsigned i = 0; i != DstNumElts; ++i) {
Idxs[i] = i + (Imm * DstNumElts);
[X86, AVX] replace vextractf128 intrinsics with generic shuffles Now that we've replaced the vinsertf128 intrinsics, do the same for their extract twins. This is very much like D8086 (checked in at r231794): We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is also the LLVM sibling to the cfe D8275 patch. Differential Revision: http://reviews.llvm.org/D8276 llvm-svn: 232045
2015-03-12 16:15:19 +01:00
}
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 06:45:46 +01:00
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
[X86, AVX] replace vextractf128 intrinsics with generic shuffles Now that we've replaced the vinsertf128 intrinsics, do the same for their extract twins. This is very much like D8086 (checked in at r231794): We want to replace as much custom x86 shuffling via intrinsics as possible because pushing the code down the generic shuffle optimization path allows for better codegen and less complexity in LLVM. This is also the LLVM sibling to the cfe D8275 patch. Differential Revision: http://reviews.llvm.org/D8276 llvm-svn: 232045
2015-03-12 16:15:19 +01:00
[AVX-512] Remove vextract intrinsics and autoupgrade to native shufflevectors. This unfortunately generates some really terrible code without VLX support due to v2i1 and v4i1 not being legal. Hopefully we can improve that in future patches. llvm-svn: 290863
2017-01-03 06:45:46 +01:00
// If the intrinsic has a mask operand, handle that.
if (CI->getNumArgOperands() == 4)
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (!IsX86 && Name == "stackprotectorcheck") {
[SSP] Remove llvm.stackprotectorcheck. This is a cleanup patch for SSP support in LLVM. There is no functional change. llvm.stackprotectorcheck is not needed, because SelectionDAG isn't actually lowering it in SelectBasicBlock; rather, it adds check code in FinishBasicBlock, ignoring the position where the intrinsic is inserted (See FindSplitPointForStackProtector()). llvm-svn: 265851
2016-04-08 23:26:31 +02:00
Rep = nullptr;
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.perm.df.") ||
Name.startswith("avx512.mask.perm.di."))) {
[X86][AVX512] Autoupgrade the VPERMPD/VPERMQ intrinsics llvm-svn: 274506
2016-07-04 16:19:05 +02:00
Value *Op0 = CI->getArgOperand(0);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
VectorType *VecTy = cast<VectorType>(CI->getType());
unsigned NumElts = VecTy->getNumElements();
SmallVector<uint32_t, 8> Idxs(NumElts);
for (unsigned i = 0; i != NumElts; ++i)
Idxs[i] = (i & ~0x3) + ((Imm >> (2 * (i & 0x3))) & 3);
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
if (CI->getNumArgOperands() == 4)
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86] Remove VPERM2F128/VPERM2I128 intrinsics and autoupgrade to native shuffles. I've moved the test cases from the InstCombine optimizations to the backend to keep the coverage we had there. It covered every possible immediate so I've preserved the resulting shuffle mask for each of those immediates. llvm-svn: 313450
2017-09-16 09:36:14 +02:00
} else if (IsX86 && (Name.startswith("avx.vperm2f128.") ||
Name == "avx2.vperm2i128")) {
// The immediate permute control byte looks like this:
// [1:0] - select 128 bits from sources for low half of destination
// [2] - ignore
// [3] - zero low half of destination
// [5:4] - select 128 bits from sources for high half of destination
// [6] - ignore
// [7] - zero high half of destination
uint8_t Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
unsigned NumElts = CI->getType()->getVectorNumElements();
unsigned HalfSize = NumElts / 2;
SmallVector<uint32_t, 8> ShuffleMask(NumElts);
// Determine which operand(s) are actually in use for this instruction.
Value *V0 = (Imm & 0x02) ? CI->getArgOperand(1) : CI->getArgOperand(0);
Value *V1 = (Imm & 0x20) ? CI->getArgOperand(1) : CI->getArgOperand(0);
// If needed, replace operands based on zero mask.
V0 = (Imm & 0x08) ? ConstantAggregateZero::get(CI->getType()) : V0;
V1 = (Imm & 0x80) ? ConstantAggregateZero::get(CI->getType()) : V1;
// Permute low half of result.
unsigned StartIndex = (Imm & 0x01) ? HalfSize : 0;
for (unsigned i = 0; i < HalfSize; ++i)
ShuffleMask[i] = StartIndex + i;
// Permute high half of result.
StartIndex = (Imm & 0x10) ? HalfSize : 0;
for (unsigned i = 0; i < HalfSize; ++i)
ShuffleMask[i + HalfSize] = NumElts + StartIndex + i;
Rep = Builder.CreateShuffleVector(V0, V1, ShuffleMask);
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name.startswith("avx.vpermil.") ||
Name == "sse2.pshuf.d" ||
Name.startswith("avx512.mask.vpermil.p") ||
Name.startswith("avx512.mask.pshuf.d."))) {
[X86] Greatly simplify the llvm.x86.avx.vpermil.* auto-upgrade code. We can fully derive everything using types of the intrinsic arguments rather than writing separate loops for each intrinsic. NFC llvm-svn: 272496
2016-06-12 05:10:47 +02:00
Value *Op0 = CI->getArgOperand(0);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
VectorType *VecTy = cast<VectorType>(CI->getType());
unsigned NumElts = VecTy->getNumElements();
[X86][AVX512] Autoupgrade the VPERMILPD/VPERMILPS intrinsics llvm-svn: 274498
2016-07-04 14:40:54 +02:00
// Calculate the size of each index in the immediate.
[X86] Greatly simplify the llvm.x86.avx.vpermil.* auto-upgrade code. We can fully derive everything using types of the intrinsic arguments rather than writing separate loops for each intrinsic. NFC llvm-svn: 272496
2016-06-12 05:10:47 +02:00
unsigned IdxSize = 64 / VecTy->getScalarSizeInBits();
unsigned IdxMask = ((1 << IdxSize) - 1);
SmallVector<uint32_t, 8> Idxs(NumElts);
// Lookup the bits for this element, wrapping around the immediate every
// 8-bits. Elements are grouped into sets of 2 or 4 elements so we need
// to offset by the first index of each group.
for (unsigned i = 0; i != NumElts; ++i)
Idxs[i] = ((Imm >> ((i * IdxSize) % 8)) & IdxMask) | (i & ~IdxMask);
[X86] Remove sse2 pshufd/pshuflw/pshufhw intrinsics and upgrade them to shufflevector. llvm-svn: 272510
2016-06-12 16:11:32 +02:00
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
[AVX512] Remove maksed pshufd, pshuflw, and phufhw intrinsics and autoupgrade them to selects and shufflevector. llvm-svn: 272527
2016-06-13 04:36:48 +02:00
if (CI->getNumArgOperands() == 4)
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse2.pshufl.w" ||
Name.startswith("avx512.mask.pshufl.w."))) {
[X86] Remove sse2 pshufd/pshuflw/pshufhw intrinsics and upgrade them to shufflevector. llvm-svn: 272510
2016-06-12 16:11:32 +02:00
Value *Op0 = CI->getArgOperand(0);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
unsigned NumElts = CI->getType()->getVectorNumElements();
SmallVector<uint32_t, 16> Idxs(NumElts);
for (unsigned l = 0; l != NumElts; l += 8) {
for (unsigned i = 0; i != 4; ++i)
Idxs[i + l] = ((Imm >> (2 * i)) & 0x3) + l;
for (unsigned i = 4; i != 8; ++i)
Idxs[i + l] = i + l;
}
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
[AVX512] Remove maksed pshufd, pshuflw, and phufhw intrinsics and autoupgrade them to selects and shufflevector. llvm-svn: 272527
2016-06-13 04:36:48 +02:00
if (CI->getNumArgOperands() == 4)
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name == "sse2.pshufh.w" ||
Name.startswith("avx512.mask.pshufh.w."))) {
[X86] Remove sse2 pshufd/pshuflw/pshufhw intrinsics and upgrade them to shufflevector. llvm-svn: 272510
2016-06-12 16:11:32 +02:00
Value *Op0 = CI->getArgOperand(0);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
unsigned NumElts = CI->getType()->getVectorNumElements();
SmallVector<uint32_t, 16> Idxs(NumElts);
for (unsigned l = 0; l != NumElts; l += 8) {
for (unsigned i = 0; i != 4; ++i)
Idxs[i + l] = i + l;
for (unsigned i = 0; i != 4; ++i)
Idxs[i + l + 4] = ((Imm >> (2 * i)) & 0x3) + 4 + l;
}
[X86] Greatly simplify the llvm.x86.avx.vpermil.* auto-upgrade code. We can fully derive everything using types of the intrinsic arguments rather than writing separate loops for each intrinsic. NFC llvm-svn: 272496
2016-06-12 05:10:47 +02:00
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
[AVX512] Remove maksed pshufd, pshuflw, and phufhw intrinsics and autoupgrade them to selects and shufflevector. llvm-svn: 272527
2016-06-13 04:36:48 +02:00
if (CI->getNumArgOperands() == 4)
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86] Remove masked shufpd/shufps intrinsics and autoupgrade to native vector shuffles. They were removed from clang previously but accidentally left in the backend. llvm-svn: 281300
2016-09-13 09:40:53 +02:00
} else if (IsX86 && Name.startswith("avx512.mask.shuf.p")) {
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
unsigned Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
unsigned NumElts = CI->getType()->getVectorNumElements();
unsigned NumLaneElts = 128/CI->getType()->getScalarSizeInBits();
unsigned HalfLaneElts = NumLaneElts / 2;
SmallVector<uint32_t, 16> Idxs(NumElts);
for (unsigned i = 0; i != NumElts; ++i) {
// Base index is the starting element of the lane.
Idxs[i] = i - (i % NumLaneElts);
// If we are half way through the lane switch to the other source.
if ((i % NumLaneElts) >= HalfLaneElts)
Idxs[i] += NumElts;
// Now select the specific element. By adding HalfLaneElts bits from
// the immediate. Wrapping around the immediate every 8-bits.
Idxs[i] += (Imm >> ((i * HalfLaneElts) % 8)) & ((1 << HalfLaneElts) - 1);
}
Rep = Builder.CreateShuffleVector(Op0, Op1, Idxs);
Rep = EmitX86Select(Builder, CI->getArgOperand(4), Rep,
CI->getArgOperand(3));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.movddup") ||
Name.startswith("avx512.mask.movshdup") ||
Name.startswith("avx512.mask.movsldup"))) {
[X86][AVX512] Autoupgrade the MOVDDUP/MOVSLDUP/MOVSHDUP intrinsics llvm-svn: 274439
2016-07-02 16:42:35 +02:00
Value *Op0 = CI->getArgOperand(0);
unsigned NumElts = CI->getType()->getVectorNumElements();
unsigned NumLaneElts = 128/CI->getType()->getScalarSizeInBits();
unsigned Offset = 0;
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
if (Name.startswith("avx512.mask.movshdup."))
[X86][AVX512] Autoupgrade the MOVDDUP/MOVSLDUP/MOVSHDUP intrinsics llvm-svn: 274439
2016-07-02 16:42:35 +02:00
Offset = 1;
SmallVector<uint32_t, 16> Idxs(NumElts);
for (unsigned l = 0; l != NumElts; l += NumLaneElts)
for (unsigned i = 0; i != NumLaneElts; i += 2) {
Idxs[i + l + 0] = i + l + Offset;
Idxs[i + l + 1] = i + l + Offset;
}
Rep = Builder.CreateShuffleVector(Op0, Op0, Idxs);
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.punpckl") ||
Name.startswith("avx512.mask.unpckl."))) {
[AVX512] Remove masked unpack intrinsics and autoupgrade to vectorshuffle and selects. llvm-svn: 273543
2016-06-23 09:37:33 +02:00
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
int NumElts = CI->getType()->getVectorNumElements();
int NumLaneElts = 128/CI->getType()->getScalarSizeInBits();
SmallVector<uint32_t, 64> Idxs(NumElts);
for (int l = 0; l != NumElts; l += NumLaneElts)
for (int i = 0; i != NumLaneElts; ++i)
Idxs[i + l] = l + (i / 2) + NumElts * (i % 2);
Rep = Builder.CreateShuffleVector(Op0, Op1, Idxs);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[IR,X86] Remove some intrinsic prefixes earlier in the auto-upgrade code so we can shorten the length of the comparison strings and avoid repeatedly comparing the common prefix. No functional change intended. llvm-svn: 274522
2016-07-04 22:56:38 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.punpckh") ||
Name.startswith("avx512.mask.unpckh."))) {
[AVX512] Remove masked unpack intrinsics and autoupgrade to vectorshuffle and selects. llvm-svn: 273543
2016-06-23 09:37:33 +02:00
Value *Op0 = CI->getArgOperand(0);
Value *Op1 = CI->getArgOperand(1);
int NumElts = CI->getType()->getVectorNumElements();
int NumLaneElts = 128/CI->getType()->getScalarSizeInBits();
SmallVector<uint32_t, 64> Idxs(NumElts);
for (int l = 0; l != NumElts; l += NumLaneElts)
for (int i = 0; i != NumLaneElts; ++i)
Idxs[i + l] = (NumLaneElts / 2) + l + (i / 2) + NumElts * (i % 2);
Rep = Builder.CreateShuffleVector(Op0, Op1, Idxs);
[AVX512] Remove masked logic op intrinsics and autoupgrade them to native IR. llvm-svn: 275155
2016-07-12 07:27:53 +02:00
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86][AutoUpgrade] Make some tweaks to reduce the number of nested if/else in the intrinsic upgrade code to avoid an MSVC compiler limit. MSVC has a nesting limit of around 110-130. An if/else if/else if counts against this next level. The autoupgrade code consists a long chain of these checking matches against strings. This commit moves some code to a helper function to move out a large if/else chain that was inside of one of the blocks into a separate function. There are more of these we could move or we could change some to lookup tables. I've also merged together a few similar blocks in the outer chain. This should buy us some margin for a little bit. llvm-svn: 350564
2019-01-07 21:13:45 +01:00
} else if (IsX86 && (Name.startswith("avx512.mask.and.") ||
Name.startswith("avx512.mask.pand."))) {
[AVX-512] Remove floating point logical operation instrinsics and replace them with native IR. llvm-svn: 280466
2016-09-02 07:29:17 +02:00
VectorType *FTy = cast<VectorType>(CI->getType());
VectorType *ITy = VectorType::getInteger(FTy);
Rep = Builder.CreateAnd(Builder.CreateBitCast(CI->getArgOperand(0), ITy),
Builder.CreateBitCast(CI->getArgOperand(1), ITy));
Rep = Builder.CreateBitCast(Rep, FTy);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86][AutoUpgrade] Make some tweaks to reduce the number of nested if/else in the intrinsic upgrade code to avoid an MSVC compiler limit. MSVC has a nesting limit of around 110-130. An if/else if/else if counts against this next level. The autoupgrade code consists a long chain of these checking matches against strings. This commit moves some code to a helper function to move out a large if/else chain that was inside of one of the blocks into a separate function. There are more of these we could move or we could change some to lookup tables. I've also merged together a few similar blocks in the outer chain. This should buy us some margin for a little bit. llvm-svn: 350564
2019-01-07 21:13:45 +01:00
} else if (IsX86 && (Name.startswith("avx512.mask.andn.") ||
Name.startswith("avx512.mask.pandn."))) {
[AVX-512] Remove floating point logical operation instrinsics and replace them with native IR. llvm-svn: 280466
2016-09-02 07:29:17 +02:00
VectorType *FTy = cast<VectorType>(CI->getType());
VectorType *ITy = VectorType::getInteger(FTy);
Rep = Builder.CreateNot(Builder.CreateBitCast(CI->getArgOperand(0), ITy));
Rep = Builder.CreateAnd(Rep,
Builder.CreateBitCast(CI->getArgOperand(1), ITy));
Rep = Builder.CreateBitCast(Rep, FTy);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86][AutoUpgrade] Make some tweaks to reduce the number of nested if/else in the intrinsic upgrade code to avoid an MSVC compiler limit. MSVC has a nesting limit of around 110-130. An if/else if/else if counts against this next level. The autoupgrade code consists a long chain of these checking matches against strings. This commit moves some code to a helper function to move out a large if/else chain that was inside of one of the blocks into a separate function. There are more of these we could move or we could change some to lookup tables. I've also merged together a few similar blocks in the outer chain. This should buy us some margin for a little bit. llvm-svn: 350564
2019-01-07 21:13:45 +01:00
} else if (IsX86 && (Name.startswith("avx512.mask.or.") ||
Name.startswith("avx512.mask.por."))) {
[AVX-512] Remove floating point logical operation instrinsics and replace them with native IR. llvm-svn: 280466
2016-09-02 07:29:17 +02:00
VectorType *FTy = cast<VectorType>(CI->getType());
VectorType *ITy = VectorType::getInteger(FTy);
Rep = Builder.CreateOr(Builder.CreateBitCast(CI->getArgOperand(0), ITy),
Builder.CreateBitCast(CI->getArgOperand(1), ITy));
Rep = Builder.CreateBitCast(Rep, FTy);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86][AutoUpgrade] Make some tweaks to reduce the number of nested if/else in the intrinsic upgrade code to avoid an MSVC compiler limit. MSVC has a nesting limit of around 110-130. An if/else if/else if counts against this next level. The autoupgrade code consists a long chain of these checking matches against strings. This commit moves some code to a helper function to move out a large if/else chain that was inside of one of the blocks into a separate function. There are more of these we could move or we could change some to lookup tables. I've also merged together a few similar blocks in the outer chain. This should buy us some margin for a little bit. llvm-svn: 350564
2019-01-07 21:13:45 +01:00
} else if (IsX86 && (Name.startswith("avx512.mask.xor.") ||
Name.startswith("avx512.mask.pxor."))) {
[AVX-512] Remove floating point logical operation instrinsics and replace them with native IR. llvm-svn: 280466
2016-09-02 07:29:17 +02:00
VectorType *FTy = cast<VectorType>(CI->getType());
VectorType *ITy = VectorType::getInteger(FTy);
Rep = Builder.CreateXor(Builder.CreateBitCast(CI->getArgOperand(0), ITy),
Builder.CreateBitCast(CI->getArgOperand(1), ITy));
Rep = Builder.CreateBitCast(Rep, FTy);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[AVX-512] Remove masked integer add/sub/mull intrinsics and upgrade to native IR. llvm-svn: 280611
2016-09-04 04:09:53 +02:00
} else if (IsX86 && Name.startswith("avx512.mask.padd.")) {
Rep = Builder.CreateAdd(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.psub.")) {
Rep = Builder.CreateSub(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
} else if (IsX86 && Name.startswith("avx512.mask.pmull.")) {
Rep = Builder.CreateMul(CI->getArgOperand(0), CI->getArgOperand(1));
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86] Remove masking from the 512-bit masked floating point add/sub/mul/div intrinsics. Use a select in IR instead. llvm-svn: 334358
2018-06-10 08:01:36 +02:00
} else if (IsX86 && Name.startswith("avx512.mask.add.p")) {
if (Name.endswith(".512")) {
Intrinsic::ID IID;
if (Name[17] == 's')
IID = Intrinsic::x86_avx512_add_ps_512;
else
IID = Intrinsic::x86_avx512_add_pd_512;
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(4) });
} else {
Rep = Builder.CreateFAdd(CI->getArgOperand(0), CI->getArgOperand(1));
}
[AVX-512] Remove 128-bit and 256-bit masked floating point add/sub/mul/div intrinsics and upgrade to native IR. llvm-svn: 280633
2016-09-04 20:13:33 +02:00
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86] Use StringRef::startswith to reduce a few compares in the intrinsic autoupgrade code. llvm-svn: 286090
2016-11-07 01:13:42 +01:00
} else if (IsX86 && Name.startswith("avx512.mask.div.p")) {
[X86] Remove masking from the 512-bit masked floating point add/sub/mul/div intrinsics. Use a select in IR instead. llvm-svn: 334358
2018-06-10 08:01:36 +02:00
if (Name.endswith(".512")) {
Intrinsic::ID IID;
if (Name[17] == 's')
IID = Intrinsic::x86_avx512_div_ps_512;
else
IID = Intrinsic::x86_avx512_div_pd_512;
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(4) });
} else {
Rep = Builder.CreateFDiv(CI->getArgOperand(0), CI->getArgOperand(1));
}
[AVX-512] Remove 128-bit and 256-bit masked floating point add/sub/mul/div intrinsics and upgrade to native IR. llvm-svn: 280633
2016-09-04 20:13:33 +02:00
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86] Use StringRef::startswith to reduce a few compares in the intrinsic autoupgrade code. llvm-svn: 286090
2016-11-07 01:13:42 +01:00
} else if (IsX86 && Name.startswith("avx512.mask.mul.p")) {
[X86] Remove masking from the 512-bit masked floating point add/sub/mul/div intrinsics. Use a select in IR instead. llvm-svn: 334358
2018-06-10 08:01:36 +02:00
if (Name.endswith(".512")) {
Intrinsic::ID IID;
if (Name[17] == 's')
IID = Intrinsic::x86_avx512_mul_ps_512;
else
IID = Intrinsic::x86_avx512_mul_pd_512;
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(4) });
} else {
Rep = Builder.CreateFMul(CI->getArgOperand(0), CI->getArgOperand(1));
}
[AVX-512] Remove 128-bit and 256-bit masked floating point add/sub/mul/div intrinsics and upgrade to native IR. llvm-svn: 280633
2016-09-04 20:13:33 +02:00
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86] Use StringRef::startswith to reduce a few compares in the intrinsic autoupgrade code. llvm-svn: 286090
2016-11-07 01:13:42 +01:00
} else if (IsX86 && Name.startswith("avx512.mask.sub.p")) {
[X86] Remove masking from the 512-bit masked floating point add/sub/mul/div intrinsics. Use a select in IR instead. llvm-svn: 334358
2018-06-10 08:01:36 +02:00
if (Name.endswith(".512")) {
Intrinsic::ID IID;
if (Name[17] == 's')
IID = Intrinsic::x86_avx512_sub_ps_512;
else
IID = Intrinsic::x86_avx512_sub_pd_512;
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(4) });
} else {
Rep = Builder.CreateFSub(CI->getArgOperand(0), CI->getArgOperand(1));
}
[AVX-512] Remove 128-bit and 256-bit masked floating point add/sub/mul/div intrinsics and upgrade to native IR. llvm-svn: 280633
2016-09-04 20:13:33 +02:00
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[X86][AutoUpgrade] Make some tweaks to reduce the number of nested if/else in the intrinsic upgrade code to avoid an MSVC compiler limit. MSVC has a nesting limit of around 110-130. An if/else if/else if counts against this next level. The autoupgrade code consists a long chain of these checking matches against strings. This commit moves some code to a helper function to move out a large if/else chain that was inside of one of the blocks into a separate function. There are more of these we could move or we could change some to lookup tables. I've also merged together a few similar blocks in the outer chain. This should buy us some margin for a little bit. llvm-svn: 350564
2019-01-07 21:13:45 +01:00
} else if (IsX86 && (Name.startswith("avx512.mask.max.p") ||
Name.startswith("avx512.mask.min.p")) &&
[X86] Remove masking from 512-bit floating max/min intrinsics. Use select instruction instead. llvm-svn: 335199
2018-06-21 07:00:56 +02:00
Name.drop_front(18) == ".512") {
[X86][AutoUpgrade] Make some tweaks to reduce the number of nested if/else in the intrinsic upgrade code to avoid an MSVC compiler limit. MSVC has a nesting limit of around 110-130. An if/else if/else if counts against this next level. The autoupgrade code consists a long chain of these checking matches against strings. This commit moves some code to a helper function to move out a large if/else chain that was inside of one of the blocks into a separate function. There are more of these we could move or we could change some to lookup tables. I've also merged together a few similar blocks in the outer chain. This should buy us some margin for a little bit. llvm-svn: 350564
2019-01-07 21:13:45 +01:00
bool IsDouble = Name[17] == 'd';
bool IsMin = Name[13] == 'i';
static const Intrinsic::ID MinMaxTbl[2][2] = {
{ Intrinsic::x86_avx512_max_ps_512, Intrinsic::x86_avx512_max_pd_512 },
{ Intrinsic::x86_avx512_min_ps_512, Intrinsic::x86_avx512_min_pd_512 }
};
Intrinsic::ID IID = MinMaxTbl[IsMin][IsDouble];
[X86] Remove masking from 512-bit floating max/min intrinsics. Use select instruction instead. llvm-svn: 335199
2018-06-21 07:00:56 +02:00
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{ CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(4) });
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep,
CI->getArgOperand(2));
[AVX-512] Remove lzcnt intrinsics and autoupgrade them to generic ctlz intrinsics with select. Clang has been emitting cltz intrinsics for a while now. llvm-svn: 296091
2017-02-24 06:35:04 +01:00
} else if (IsX86 && Name.startswith("avx512.mask.lzcnt.")) {
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(),
Intrinsic::ctlz,
CI->getType()),
{ CI->getArgOperand(0), Builder.getInt1(false) });
Rep = EmitX86Select(Builder, CI->getArgOperand(2), Rep,
CI->getArgOperand(1));
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
} else if (IsX86 && Name.startswith("avx512.mask.psll")) {
bool IsImmediate = Name[16] == 'i' ||
(Name.size() > 18 && Name[18] == 'i');
bool IsVariable = Name[16] == 'v';
char Size = Name[16] == '.' ? Name[17] :
Name[17] == '.' ? Name[18] :
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 06:04:44 +01:00
Name[18] == '.' ? Name[19] :
Name[20];
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
Intrinsic::ID IID;
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 02:53:22 +01:00
if (IsVariable && Name[17] != '.') {
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
if (Size == 'd' && Name[17] == '2') // avx512.mask.psllv2.di
IID = Intrinsic::x86_avx2_psllv_q;
else if (Size == 'd' && Name[17] == '4') // avx512.mask.psllv4.di
IID = Intrinsic::x86_avx2_psllv_q_256;
else if (Size == 's' && Name[17] == '4') // avx512.mask.psllv4.si
IID = Intrinsic::x86_avx2_psllv_d;
else if (Size == 's' && Name[17] == '8') // avx512.mask.psllv8.si
IID = Intrinsic::x86_avx2_psllv_d_256;
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 06:04:44 +01:00
else if (Size == 'h' && Name[17] == '8') // avx512.mask.psllv8.hi
IID = Intrinsic::x86_avx512_psllv_w_128;
else if (Size == 'h' && Name[17] == '1') // avx512.mask.psllv16.hi
IID = Intrinsic::x86_avx512_psllv_w_256;
else if (Name[17] == '3' && Name[18] == '2') // avx512.mask.psllv32hi
IID = Intrinsic::x86_avx512_psllv_w_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".128")) {
if (Size == 'd') // avx512.mask.psll.d.128, avx512.mask.psll.di.128
IID = IsImmediate ? Intrinsic::x86_sse2_pslli_d
: Intrinsic::x86_sse2_psll_d;
else if (Size == 'q') // avx512.mask.psll.q.128, avx512.mask.psll.qi.128
IID = IsImmediate ? Intrinsic::x86_sse2_pslli_q
: Intrinsic::x86_sse2_psll_q;
else if (Size == 'w') // avx512.mask.psll.w.128, avx512.mask.psll.wi.128
IID = IsImmediate ? Intrinsic::x86_sse2_pslli_w
: Intrinsic::x86_sse2_psll_w;
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".256")) {
if (Size == 'd') // avx512.mask.psll.d.256, avx512.mask.psll.di.256
IID = IsImmediate ? Intrinsic::x86_avx2_pslli_d
: Intrinsic::x86_avx2_psll_d;
else if (Size == 'q') // avx512.mask.psll.q.256, avx512.mask.psll.qi.256
IID = IsImmediate ? Intrinsic::x86_avx2_pslli_q
: Intrinsic::x86_avx2_psll_q;
else if (Size == 'w') // avx512.mask.psll.w.256, avx512.mask.psll.wi.256
IID = IsImmediate ? Intrinsic::x86_avx2_pslli_w
: Intrinsic::x86_avx2_psll_w;
else
llvm_unreachable("Unexpected size");
} else {
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 02:53:22 +01:00
if (Size == 'd') // psll.di.512, pslli.d, psll.d, psllv.d.512
IID = IsImmediate ? Intrinsic::x86_avx512_pslli_d_512 :
IsVariable ? Intrinsic::x86_avx512_psllv_d_512 :
Intrinsic::x86_avx512_psll_d_512;
else if (Size == 'q') // psll.qi.512, pslli.q, psll.q, psllv.q.512
IID = IsImmediate ? Intrinsic::x86_avx512_pslli_q_512 :
IsVariable ? Intrinsic::x86_avx512_psllv_q_512 :
Intrinsic::x86_avx512_psll_q_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
else if (Size == 'w') // psll.wi.512, pslli.w, psll.w
IID = IsImmediate ? Intrinsic::x86_avx512_pslli_w_512
: Intrinsic::x86_avx512_psll_w_512;
else
llvm_unreachable("Unexpected size");
}
Rep = UpgradeX86MaskedShift(Builder, *CI, IID);
} else if (IsX86 && Name.startswith("avx512.mask.psrl")) {
bool IsImmediate = Name[16] == 'i' ||
(Name.size() > 18 && Name[18] == 'i');
bool IsVariable = Name[16] == 'v';
char Size = Name[16] == '.' ? Name[17] :
Name[17] == '.' ? Name[18] :
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 06:04:44 +01:00
Name[18] == '.' ? Name[19] :
Name[20];
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
Intrinsic::ID IID;
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 02:53:22 +01:00
if (IsVariable && Name[17] != '.') {
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
if (Size == 'd' && Name[17] == '2') // avx512.mask.psrlv2.di
IID = Intrinsic::x86_avx2_psrlv_q;
else if (Size == 'd' && Name[17] == '4') // avx512.mask.psrlv4.di
IID = Intrinsic::x86_avx2_psrlv_q_256;
else if (Size == 's' && Name[17] == '4') // avx512.mask.psrlv4.si
IID = Intrinsic::x86_avx2_psrlv_d;
else if (Size == 's' && Name[17] == '8') // avx512.mask.psrlv8.si
IID = Intrinsic::x86_avx2_psrlv_d_256;
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 06:04:44 +01:00
else if (Size == 'h' && Name[17] == '8') // avx512.mask.psrlv8.hi
IID = Intrinsic::x86_avx512_psrlv_w_128;
else if (Size == 'h' && Name[17] == '1') // avx512.mask.psrlv16.hi
IID = Intrinsic::x86_avx512_psrlv_w_256;
else if (Name[17] == '3' && Name[18] == '2') // avx512.mask.psrlv32hi
IID = Intrinsic::x86_avx512_psrlv_w_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".128")) {
if (Size == 'd') // avx512.mask.psrl.d.128, avx512.mask.psrl.di.128
IID = IsImmediate ? Intrinsic::x86_sse2_psrli_d
: Intrinsic::x86_sse2_psrl_d;
else if (Size == 'q') // avx512.mask.psrl.q.128, avx512.mask.psrl.qi.128
IID = IsImmediate ? Intrinsic::x86_sse2_psrli_q
: Intrinsic::x86_sse2_psrl_q;
else if (Size == 'w') // avx512.mask.psrl.w.128, avx512.mask.psrl.wi.128
IID = IsImmediate ? Intrinsic::x86_sse2_psrli_w
: Intrinsic::x86_sse2_psrl_w;
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".256")) {
if (Size == 'd') // avx512.mask.psrl.d.256, avx512.mask.psrl.di.256
IID = IsImmediate ? Intrinsic::x86_avx2_psrli_d
: Intrinsic::x86_avx2_psrl_d;
else if (Size == 'q') // avx512.mask.psrl.q.256, avx512.mask.psrl.qi.256
IID = IsImmediate ? Intrinsic::x86_avx2_psrli_q
: Intrinsic::x86_avx2_psrl_q;
else if (Size == 'w') // avx512.mask.psrl.w.256, avx512.mask.psrl.wi.256
IID = IsImmediate ? Intrinsic::x86_avx2_psrli_w
: Intrinsic::x86_avx2_psrl_w;
else
llvm_unreachable("Unexpected size");
} else {
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 02:53:22 +01:00
if (Size == 'd') // psrl.di.512, psrli.d, psrl.d, psrl.d.512
IID = IsImmediate ? Intrinsic::x86_avx512_psrli_d_512 :
IsVariable ? Intrinsic::x86_avx512_psrlv_d_512 :
Intrinsic::x86_avx512_psrl_d_512;
else if (Size == 'q') // psrl.qi.512, psrli.q, psrl.q, psrl.q.512
IID = IsImmediate ? Intrinsic::x86_avx512_psrli_q_512 :
IsVariable ? Intrinsic::x86_avx512_psrlv_q_512 :
Intrinsic::x86_avx512_psrl_q_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
else if (Size == 'w') // psrl.wi.512, psrli.w, psrl.w)
IID = IsImmediate ? Intrinsic::x86_avx512_psrli_w_512
: Intrinsic::x86_avx512_psrl_w_512;
else
llvm_unreachable("Unexpected size");
}
Rep = UpgradeX86MaskedShift(Builder, *CI, IID);
} else if (IsX86 && Name.startswith("avx512.mask.psra")) {
bool IsImmediate = Name[16] == 'i' ||
(Name.size() > 18 && Name[18] == 'i');
bool IsVariable = Name[16] == 'v';
char Size = Name[16] == '.' ? Name[17] :
Name[17] == '.' ? Name[18] :
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 06:04:44 +01:00
Name[18] == '.' ? Name[19] :
Name[20];
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
Intrinsic::ID IID;
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 02:53:22 +01:00
if (IsVariable && Name[17] != '.') {
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
if (Size == 's' && Name[17] == '4') // avx512.mask.psrav4.si
IID = Intrinsic::x86_avx2_psrav_d;
else if (Size == 's' && Name[17] == '8') // avx512.mask.psrav8.si
IID = Intrinsic::x86_avx2_psrav_d_256;
[AVX-512] Replace masked 16-bit element variable shift intrinsics with new unmasked versions and selects. The same thing was done to 32-bit and 64-bit element sizes previously. This will allow us to support these shuffls in InstCombineCalls along with the other variable shift intrinsics. llvm-svn: 287312
2016-11-18 06:04:44 +01:00
else if (Size == 'h' && Name[17] == '8') // avx512.mask.psrav8.hi
IID = Intrinsic::x86_avx512_psrav_w_128;
else if (Size == 'h' && Name[17] == '1') // avx512.mask.psrav16.hi
IID = Intrinsic::x86_avx512_psrav_w_256;
else if (Name[17] == '3' && Name[18] == '2') // avx512.mask.psrav32hi
IID = Intrinsic::x86_avx512_psrav_w_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".128")) {
if (Size == 'd') // avx512.mask.psra.d.128, avx512.mask.psra.di.128
IID = IsImmediate ? Intrinsic::x86_sse2_psrai_d
: Intrinsic::x86_sse2_psra_d;
else if (Size == 'q') // avx512.mask.psra.q.128, avx512.mask.psra.qi.128
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 02:53:22 +01:00
IID = IsImmediate ? Intrinsic::x86_avx512_psrai_q_128 :
IsVariable ? Intrinsic::x86_avx512_psrav_q_128 :
Intrinsic::x86_avx512_psra_q_128;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
else if (Size == 'w') // avx512.mask.psra.w.128, avx512.mask.psra.wi.128
IID = IsImmediate ? Intrinsic::x86_sse2_psrai_w
: Intrinsic::x86_sse2_psra_w;
else
llvm_unreachable("Unexpected size");
} else if (Name.endswith(".256")) {
if (Size == 'd') // avx512.mask.psra.d.256, avx512.mask.psra.di.256
IID = IsImmediate ? Intrinsic::x86_avx2_psrai_d
: Intrinsic::x86_avx2_psra_d;
else if (Size == 'q') // avx512.mask.psra.q.256, avx512.mask.psra.qi.256
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 02:53:22 +01:00
IID = IsImmediate ? Intrinsic::x86_avx512_psrai_q_256 :
IsVariable ? Intrinsic::x86_avx512_psrav_q_256 :
Intrinsic::x86_avx512_psra_q_256;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
else if (Size == 'w') // avx512.mask.psra.w.256, avx512.mask.psra.wi.256
IID = IsImmediate ? Intrinsic::x86_avx2_psrai_w
: Intrinsic::x86_avx2_psra_w;
else
llvm_unreachable("Unexpected size");
} else {
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 02:53:22 +01:00
if (Size == 'd') // psra.di.512, psrai.d, psra.d, psrav.d.512
IID = IsImmediate ? Intrinsic::x86_avx512_psrai_d_512 :
IsVariable ? Intrinsic::x86_avx512_psrav_d_512 :
Intrinsic::x86_avx512_psra_d_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
else if (Size == 'q') // psra.qi.512, psrai.q, psra.q
[AVX-512] Remove and autoupgrade masked dword/qword variable shift intrinsics to the new unmasked versions and selects. llvm-svn: 286786
2016-11-14 02:53:22 +01:00
IID = IsImmediate ? Intrinsic::x86_avx512_psrai_q_512 :
IsVariable ? Intrinsic::x86_avx512_psrav_q_512 :
Intrinsic::x86_avx512_psra_q_512;
[X86][IR] Reduce the number of full string comparisons in the code that autoupgrades masked shift intrinsics. llvm-svn: 286768
2016-11-13 20:09:56 +01:00
else if (Size == 'w') // psra.wi.512, psrai.w, psra.w
IID = IsImmediate ? Intrinsic::x86_avx512_psrai_w_512
: Intrinsic::x86_avx512_psra_w_512;
else
llvm_unreachable("Unexpected size");
}
Rep = UpgradeX86MaskedShift(Builder, *CI, IID);
[X86][AVX512] Removing llvm x86 intrinsics for _mm_mask_move_{ss|sd} intrinsics. Differential Revision: https://reviews.llvm.org/D26128 llvm-svn: 287087
2016-11-16 10:00:28 +01:00
} else if (IsX86 && Name.startswith("avx512.mask.move.s")) {
Rep = upgradeMaskedMove(Builder, *CI);
[X86][LLVM] Converting __mm{|256|512}_movm_epi{8|16|32|64} LLVMIR call into generic intrinsics. This patch is a part one of two reviews, one for the clang and the other for LLVM. The patch deletes the back-end intrinsics and adds support for them in the auto upgrade. Differential Revision: https://reviews.llvm.org/D31393 llvm-svn: 299432
2017-04-04 15:32:14 +02:00
} else if (IsX86 && Name.startswith("avx512.cvtmask2")) {
Rep = UpgradeMaskToInt(Builder, *CI);
[X86][SSE] Update MOVNTDQA non-temporal loads to generic implementation (LLVM) MOVNTDQA non-temporal aligned vector loads can be correctly represented using generic builtin loads, allowing us to remove the existing x86 intrinsics. Clang companion patch: D31766. Differential Revision: https://reviews.llvm.org/D31767 llvm-svn: 300325
2017-04-14 17:05:35 +02:00
} else if (IsX86 && Name.endswith(".movntdqa")) {
Module *M = F->getParent();
MDNode *Node = MDNode::get(
C, ConstantAsMetadata::get(ConstantInt::get(Type::getInt32Ty(C), 1)));
Value *Ptr = CI->getArgOperand(0);
VectorType *VTy = cast<VectorType>(CI->getType());
// Convert the type of the pointer to a pointer to the stored type.
Value *BC =
Builder.CreateBitCast(Ptr, PointerType::getUnqual(VTy), "cast");
[Alignement][NFC] Deprecate untyped CreateAlignedLoad Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet Subscribers: arsenm, jvesely, nhaehnle, hiraditya, kerbowa, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D73260
2020-01-23 11:33:12 +01:00
LoadInst *LI =
Builder.CreateAlignedLoad(VTy, BC, Align(VTy->getBitWidth() / 8));
[X86][SSE] Update MOVNTDQA non-temporal loads to generic implementation (LLVM) MOVNTDQA non-temporal aligned vector loads can be correctly represented using generic builtin loads, allowing us to remove the existing x86 intrinsics. Clang companion patch: D31766. Differential Revision: https://reviews.llvm.org/D31767 llvm-svn: 300325
2017-04-14 17:05:35 +02:00
LI->setMetadata(M->getMDKindID("nontemporal"), Node);
Rep = LI;
[X86] Remove X86 specific scalar FMA intrinsics and upgrade to tart independent FMA and extractelement/insertelement. llvm-svn: 336315
2018-07-05 08:52:55 +02:00
} else if (IsX86 && (Name.startswith("fma.vfmadd.") ||
Name.startswith("fma.vfmsub.") ||
Name.startswith("fma.vfnmadd.") ||
Name.startswith("fma.vfnmsub."))) {
[X86] Remove some of the packed FMA3 intrinsics since we no longer use them in clang. There's a regression in here due to inability to combine fneg inputs of X86ISD::FMSUB/FNMSUB/FNMADD nodes. More removals to come, but I wanted to stop and fix the regression that showed up in this first. llvm-svn: 336303
2018-07-05 04:52:54 +02:00
bool NegMul = Name[6] == 'n';
bool NegAcc = NegMul ? Name[8] == 's' : Name[7] == 's';
[X86] Remove X86 specific scalar FMA intrinsics and upgrade to tart independent FMA and extractelement/insertelement. llvm-svn: 336315
2018-07-05 08:52:55 +02:00
bool IsScalar = NegMul ? Name[12] == 's' : Name[11] == 's';
[X86] Remove some of the packed FMA3 intrinsics since we no longer use them in clang. There's a regression in here due to inability to combine fneg inputs of X86ISD::FMSUB/FNMSUB/FNMADD nodes. More removals to come, but I wanted to stop and fix the regression that showed up in this first. llvm-svn: 336303
2018-07-05 04:52:54 +02:00
Value *Ops[] = { CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(2) };
[X86] Remove X86 specific scalar FMA intrinsics and upgrade to tart independent FMA and extractelement/insertelement. llvm-svn: 336315
2018-07-05 08:52:55 +02:00
if (IsScalar) {
Ops[0] = Builder.CreateExtractElement(Ops[0], (uint64_t)0);
Ops[1] = Builder.CreateExtractElement(Ops[1], (uint64_t)0);
Ops[2] = Builder.CreateExtractElement(Ops[2], (uint64_t)0);
}
if (NegMul && !IsScalar)
[X86] Remove some of the packed FMA3 intrinsics since we no longer use them in clang. There's a regression in here due to inability to combine fneg inputs of X86ISD::FMSUB/FNMSUB/FNMADD nodes. More removals to come, but I wanted to stop and fix the regression that showed up in this first. llvm-svn: 336303
2018-07-05 04:52:54 +02:00
Ops[0] = Builder.CreateFNeg(Ops[0]);
[X86] Remove X86 specific scalar FMA intrinsics and upgrade to tart independent FMA and extractelement/insertelement. llvm-svn: 336315
2018-07-05 08:52:55 +02:00
if (NegMul && IsScalar)
Ops[1] = Builder.CreateFNeg(Ops[1]);
[X86] Remove some of the packed FMA3 intrinsics since we no longer use them in clang. There's a regression in here due to inability to combine fneg inputs of X86ISD::FMSUB/FNMSUB/FNMADD nodes. More removals to come, but I wanted to stop and fix the regression that showed up in this first. llvm-svn: 336303
2018-07-05 04:52:54 +02:00
if (NegAcc)
Ops[2] = Builder.CreateFNeg(Ops[2]);
Rep = Builder.CreateCall(Intrinsic::getDeclaration(CI->getModule(),
Intrinsic::fma,
[X86] Remove X86 specific scalar FMA intrinsics and upgrade to tart independent FMA and extractelement/insertelement. llvm-svn: 336315
2018-07-05 08:52:55 +02:00
Ops[0]->getType()),
Ops);
if (IsScalar)
Rep = Builder.CreateInsertElement(CI->getArgOperand(0), Rep,
(uint64_t)0);
[X86] Remove FMA4 scalar intrinsics. Use llvm.fma intrinsic instead. The intrinsics can be implemented with a f32/f64 llvm.fma intrinsic and an insert into a zero vector. There are a couple regressions here due to SelectionDAG not being able to pull an fneg through an extract_vector_elt. I'm not super worried about this though as InstCombine should be able to do it before we get to SelectionDAG. llvm-svn: 336416
2018-07-06 09:14:41 +02:00
} else if (IsX86 && Name.startswith("fma4.vfmadd.s")) {
Value *Ops[] = { CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(2) };
Ops[0] = Builder.CreateExtractElement(Ops[0], (uint64_t)0);
Ops[1] = Builder.CreateExtractElement(Ops[1], (uint64_t)0);
Ops[2] = Builder.CreateExtractElement(Ops[2], (uint64_t)0);
Rep = Builder.CreateCall(Intrinsic::getDeclaration(CI->getModule(),
Intrinsic::fma,
Ops[0]->getType()),
Ops);
Rep = Builder.CreateInsertElement(Constant::getNullValue(CI->getType()),
Rep, (uint64_t)0);
[X86] Remove and autoupgrade the scalar fma intrinsics with masking. This converts them to what clang is now using for codegen. Unfortunately, there seem to be a few kinks to work out still. I'll try to address with follow up patches. llvm-svn: 336871
2018-07-12 02:29:56 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.vfmadd.s") ||
Name.startswith("avx512.maskz.vfmadd.s") ||
Name.startswith("avx512.mask3.vfmadd.s") ||
Name.startswith("avx512.mask3.vfmsub.s") ||
Name.startswith("avx512.mask3.vfnmsub.s"))) {
bool IsMask3 = Name[11] == '3';
bool IsMaskZ = Name[11] == 'z';
// Drop the "avx512.mask." to make it easier.
Name = Name.drop_front(IsMask3 || IsMaskZ ? 13 : 12);
bool NegMul = Name[2] == 'n';
bool NegAcc = NegMul ? Name[4] == 's' : Name[3] == 's';
Value *A = CI->getArgOperand(0);
Value *B = CI->getArgOperand(1);
Value *C = CI->getArgOperand(2);
if (NegMul && (IsMask3 || IsMaskZ))
A = Builder.CreateFNeg(A);
if (NegMul && !(IsMask3 || IsMaskZ))
B = Builder.CreateFNeg(B);
if (NegAcc)
C = Builder.CreateFNeg(C);
A = Builder.CreateExtractElement(A, (uint64_t)0);
B = Builder.CreateExtractElement(B, (uint64_t)0);
C = Builder.CreateExtractElement(C, (uint64_t)0);
if (!isa<ConstantInt>(CI->getArgOperand(4)) ||
cast<ConstantInt>(CI->getArgOperand(4))->getZExtValue() != 4) {
Value *Ops[] = { A, B, C, CI->getArgOperand(4) };
Intrinsic::ID IID;
if (Name.back() == 'd')
IID = Intrinsic::x86_avx512_vfmadd_f64;
else
IID = Intrinsic::x86_avx512_vfmadd_f32;
Function *FMA = Intrinsic::getDeclaration(CI->getModule(), IID);
Rep = Builder.CreateCall(FMA, Ops);
} else {
Function *FMA = Intrinsic::getDeclaration(CI->getModule(),
Intrinsic::fma,
A->getType());
Rep = Builder.CreateCall(FMA, { A, B, C });
}
Value *PassThru = IsMaskZ ? Constant::getNullValue(Rep->getType()) :
IsMask3 ? C : A;
// For Mask3 with NegAcc, we need to create a new extractelement that
// avoids the negation above.
if (NegAcc && IsMask3)
PassThru = Builder.CreateExtractElement(CI->getArgOperand(2),
(uint64_t)0);
Rep = EmitX86ScalarSelect(Builder, CI->getArgOperand(3),
Rep, PassThru);
Rep = Builder.CreateInsertElement(CI->getArgOperand(IsMask3 ? 2 : 0),
Rep, (uint64_t)0);
[X86] Remove all of the avx512 masked packed fma intrinsics. Use llvm.fma or unmasked 512-bit intrinsics with rounding mode. This upgrades all of the intrinsics to use fneg instructions to convert fma into fmsub/fnmsub/fnmadd/fmsubadd. And uses a select instruction for masking. This matches how clang uses the intrinsics these days. llvm-svn: 336409
2018-07-06 05:42:09 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.vfmadd.p") ||
Name.startswith("avx512.mask.vfnmadd.p") ||
Name.startswith("avx512.mask.vfnmsub.p") ||
Name.startswith("avx512.mask3.vfmadd.p") ||
Name.startswith("avx512.mask3.vfmsub.p") ||
Name.startswith("avx512.mask3.vfnmsub.p") ||
Name.startswith("avx512.maskz.vfmadd.p"))) {
bool IsMask3 = Name[11] == '3';
bool IsMaskZ = Name[11] == 'z';
// Drop the "avx512.mask." to make it easier.
Name = Name.drop_front(IsMask3 || IsMaskZ ? 13 : 12);
bool NegMul = Name[2] == 'n';
bool NegAcc = NegMul ? Name[4] == 's' : Name[3] == 's';
[X86] Remove and autoupgrade the scalar fma intrinsics with masking. This converts them to what clang is now using for codegen. Unfortunately, there seem to be a few kinks to work out still. I'll try to address with follow up patches. llvm-svn: 336871
2018-07-12 02:29:56 +02:00
Value *A = CI->getArgOperand(0);
Value *B = CI->getArgOperand(1);
Value *C = CI->getArgOperand(2);
if (NegMul && (IsMask3 || IsMaskZ))
A = Builder.CreateFNeg(A);
if (NegMul && !(IsMask3 || IsMaskZ))
B = Builder.CreateFNeg(B);
if (NegAcc)
C = Builder.CreateFNeg(C);
[X86] Remove all of the avx512 masked packed fma intrinsics. Use llvm.fma or unmasked 512-bit intrinsics with rounding mode. This upgrades all of the intrinsics to use fneg instructions to convert fma into fmsub/fnmsub/fnmadd/fmsubadd. And uses a select instruction for masking. This matches how clang uses the intrinsics these days. llvm-svn: 336409
2018-07-06 05:42:09 +02:00
if (CI->getNumArgOperands() == 5 &&
(!isa<ConstantInt>(CI->getArgOperand(4)) ||
cast<ConstantInt>(CI->getArgOperand(4))->getZExtValue() != 4)) {
Intrinsic::ID IID;
// Check the character before ".512" in string.
if (Name[Name.size()-5] == 's')
IID = Intrinsic::x86_avx512_vfmadd_ps_512;
else
IID = Intrinsic::x86_avx512_vfmadd_pd_512;
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
[X86] Remove and autoupgrade the scalar fma intrinsics with masking. This converts them to what clang is now using for codegen. Unfortunately, there seem to be a few kinks to work out still. I'll try to address with follow up patches. llvm-svn: 336871
2018-07-12 02:29:56 +02:00
{ A, B, C, CI->getArgOperand(4) });
[X86] Remove all of the avx512 masked packed fma intrinsics. Use llvm.fma or unmasked 512-bit intrinsics with rounding mode. This upgrades all of the intrinsics to use fneg instructions to convert fma into fmsub/fnmsub/fnmadd/fmsubadd. And uses a select instruction for masking. This matches how clang uses the intrinsics these days. llvm-svn: 336409
2018-07-06 05:42:09 +02:00
} else {
Function *FMA = Intrinsic::getDeclaration(CI->getModule(),
Intrinsic::fma,
[X86] Remove and autoupgrade the scalar fma intrinsics with masking. This converts them to what clang is now using for codegen. Unfortunately, there seem to be a few kinks to work out still. I'll try to address with follow up patches. llvm-svn: 336871
2018-07-12 02:29:56 +02:00
A->getType());
Rep = Builder.CreateCall(FMA, { A, B, C });
[X86] Remove all of the avx512 masked packed fma intrinsics. Use llvm.fma or unmasked 512-bit intrinsics with rounding mode. This upgrades all of the intrinsics to use fneg instructions to convert fma into fmsub/fnmsub/fnmadd/fmsubadd. And uses a select instruction for masking. This matches how clang uses the intrinsics these days. llvm-svn: 336409
2018-07-06 05:42:09 +02:00
}
Value *PassThru = IsMaskZ ? llvm::Constant::getNullValue(CI->getType()) :
IsMask3 ? CI->getArgOperand(2) :
CI->getArgOperand(0);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep, PassThru);
[X86] Remove the last of the 'x86.fma.' intrinsics and autoupgrade them to 'llvm.fma'. Add upgrade tests for all. Still need to remove the AVX512 masked versions. llvm-svn: 336383
2018-07-05 20:43:58 +02:00
} else if (IsX86 && (Name.startswith("fma.vfmaddsub.p") ||
Name.startswith("fma.vfmsubadd.p"))) {
bool IsSubAdd = Name[7] == 's';
int NumElts = CI->getType()->getVectorNumElements();
Value *Ops[] = { CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(2) };
Function *FMA = Intrinsic::getDeclaration(CI->getModule(), Intrinsic::fma,
Ops[0]->getType());
Value *Odd = Builder.CreateCall(FMA, Ops);
Ops[2] = Builder.CreateFNeg(Ops[2]);
Value *Even = Builder.CreateCall(FMA, Ops);
if (IsSubAdd)
std::swap(Even, Odd);
SmallVector<uint32_t, 32> Idxs(NumElts);
for (int i = 0; i != NumElts; ++i)
Idxs[i] = i + (i % 2) * NumElts;
Rep = Builder.CreateShuffleVector(Even, Odd, Idxs);
[X86] Remove all of the avx512 masked packed fma intrinsics. Use llvm.fma or unmasked 512-bit intrinsics with rounding mode. This upgrades all of the intrinsics to use fneg instructions to convert fma into fmsub/fnmsub/fnmadd/fmsubadd. And uses a select instruction for masking. This matches how clang uses the intrinsics these days. llvm-svn: 336409
2018-07-06 05:42:09 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.vfmaddsub.p") ||
Name.startswith("avx512.mask3.vfmaddsub.p") ||
Name.startswith("avx512.maskz.vfmaddsub.p") ||
Name.startswith("avx512.mask3.vfmsubadd.p"))) {
bool IsMask3 = Name[11] == '3';
bool IsMaskZ = Name[11] == 'z';
// Drop the "avx512.mask." to make it easier.
Name = Name.drop_front(IsMask3 || IsMaskZ ? 13 : 12);
bool IsSubAdd = Name[3] == 's';
if (CI->getNumArgOperands() == 5 &&
(!isa<ConstantInt>(CI->getArgOperand(4)) ||
cast<ConstantInt>(CI->getArgOperand(4))->getZExtValue() != 4)) {
Intrinsic::ID IID;
// Check the character before ".512" in string.
if (Name[Name.size()-5] == 's')
IID = Intrinsic::x86_avx512_vfmaddsub_ps_512;
else
IID = Intrinsic::x86_avx512_vfmaddsub_pd_512;
Value *Ops[] = { CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(2), CI->getArgOperand(4) };
if (IsSubAdd)
Ops[2] = Builder.CreateFNeg(Ops[2]);
Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
{CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(2), CI->getArgOperand(4)});
} else {
int NumElts = CI->getType()->getVectorNumElements();
Value *Ops[] = { CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(2) };
Function *FMA = Intrinsic::getDeclaration(CI->getModule(), Intrinsic::fma,
Ops[0]->getType());
Value *Odd = Builder.CreateCall(FMA, Ops);
Ops[2] = Builder.CreateFNeg(Ops[2]);
Value *Even = Builder.CreateCall(FMA, Ops);
if (IsSubAdd)
std::swap(Even, Odd);
SmallVector<uint32_t, 32> Idxs(NumElts);
for (int i = 0; i != NumElts; ++i)
Idxs[i] = i + (i % 2) * NumElts;
Rep = Builder.CreateShuffleVector(Even, Odd, Idxs);
}
Value *PassThru = IsMaskZ ? llvm::Constant::getNullValue(CI->getType()) :
IsMask3 ? CI->getArgOperand(2) :
CI->getArgOperand(0);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep, PassThru);
[X86] Remove masking from vpternlog intrinsics. Use a select in IR instead. This removes 6 intrinsics since we no longer need separate mask and maskz intrinsics. Differential Revision: https://reviews.llvm.org/D47124 llvm-svn: 332890
2018-05-21 22:58:09 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.pternlog.") ||
Name.startswith("avx512.maskz.pternlog."))) {
bool ZeroMask = Name[11] == 'z';
unsigned VecWidth = CI->getType()->getPrimitiveSizeInBits();
unsigned EltWidth = CI->getType()->getScalarSizeInBits();
Intrinsic::ID IID;
if (VecWidth == 128 && EltWidth == 32)
IID = Intrinsic::x86_avx512_pternlog_d_128;
else if (VecWidth == 256 && EltWidth == 32)
IID = Intrinsic::x86_avx512_pternlog_d_256;
else if (VecWidth == 512 && EltWidth == 32)
IID = Intrinsic::x86_avx512_pternlog_d_512;
else if (VecWidth == 128 && EltWidth == 64)
IID = Intrinsic::x86_avx512_pternlog_q_128;
else if (VecWidth == 256 && EltWidth == 64)
IID = Intrinsic::x86_avx512_pternlog_q_256;
else if (VecWidth == 512 && EltWidth == 64)
IID = Intrinsic::x86_avx512_pternlog_q_512;
else
llvm_unreachable("Unexpected intrinsic");
Value *Args[] = { CI->getArgOperand(0) , CI->getArgOperand(1),
CI->getArgOperand(2), CI->getArgOperand(3) };
Rep = Builder.CreateCall(Intrinsic::getDeclaration(CI->getModule(), IID),
Args);
Value *PassThru = ZeroMask ? ConstantAggregateZero::get(CI->getType())
: CI->getArgOperand(0);
Rep = EmitX86Select(Builder, CI->getArgOperand(4), Rep, PassThru);
[X86] Remove masking from avx512ifma intrinsics. Use a select instead. This allows us to avoid having mask and maskz variant. Reducing from 12 intrinsics to 6. llvm-svn: 333346
2018-05-26 20:55:19 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.vpmadd52") ||
Name.startswith("avx512.maskz.vpmadd52"))) {
bool ZeroMask = Name[11] == 'z';
bool High = Name[20] == 'h' || Name[21] == 'h';
unsigned VecWidth = CI->getType()->getPrimitiveSizeInBits();
Intrinsic::ID IID;
if (VecWidth == 128 && !High)
IID = Intrinsic::x86_avx512_vpmadd52l_uq_128;
else if (VecWidth == 256 && !High)
IID = Intrinsic::x86_avx512_vpmadd52l_uq_256;
else if (VecWidth == 512 && !High)
IID = Intrinsic::x86_avx512_vpmadd52l_uq_512;
else if (VecWidth == 128 && High)
IID = Intrinsic::x86_avx512_vpmadd52h_uq_128;
else if (VecWidth == 256 && High)
IID = Intrinsic::x86_avx512_vpmadd52h_uq_256;
else if (VecWidth == 512 && High)
IID = Intrinsic::x86_avx512_vpmadd52h_uq_512;
else
llvm_unreachable("Unexpected intrinsic");
Value *Args[] = { CI->getArgOperand(0) , CI->getArgOperand(1),
CI->getArgOperand(2) };
Rep = Builder.CreateCall(Intrinsic::getDeclaration(CI->getModule(), IID),
Args);
Value *PassThru = ZeroMask ? ConstantAggregateZero::get(CI->getType())
: CI->getArgOperand(0);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep, PassThru);
[X86] Remove masked vpermi2var/vpermt2var intrinsics and autoupgrade. We have unmasked intrinsics now and wrap them with a select. This is a net reduction of 36 intrinsics from before the unmasked intrinsics were added. llvm-svn: 333388
2018-05-29 07:22:05 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.vpermi2var.") ||
Name.startswith("avx512.mask.vpermt2var.") ||
Name.startswith("avx512.maskz.vpermt2var."))) {
bool ZeroMask = Name[11] == 'z';
bool IndexForm = Name[17] == 'i';
[X86][AutoUpgrade] Make some tweaks to reduce the number of nested if/else in the intrinsic upgrade code to avoid an MSVC compiler limit. MSVC has a nesting limit of around 110-130. An if/else if/else if counts against this next level. The autoupgrade code consists a long chain of these checking matches against strings. This commit moves some code to a helper function to move out a large if/else chain that was inside of one of the blocks into a separate function. There are more of these we could move or we could change some to lookup tables. I've also merged together a few similar blocks in the outer chain. This should buy us some margin for a little bit. llvm-svn: 350564
2019-01-07 21:13:45 +01:00
Rep = UpgradeX86VPERMT2Intrinsics(Builder, *CI, ZeroMask, IndexForm);
[X86] Remove and autoupgrade masked avx512vnni intrinsics using the unmasked intrinsics and select instructions. llvm-svn: 333857
2018-06-04 01:24:17 +02:00
} else if (IsX86 && (Name.startswith("avx512.mask.vpdpbusd.") ||
Name.startswith("avx512.maskz.vpdpbusd.") ||
Name.startswith("avx512.mask.vpdpbusds.") ||
Name.startswith("avx512.maskz.vpdpbusds."))) {
bool ZeroMask = Name[11] == 'z';
bool IsSaturating = Name[ZeroMask ? 21 : 20] == 's';
unsigned VecWidth = CI->getType()->getPrimitiveSizeInBits();
Intrinsic::ID IID;
if (VecWidth == 128 && !IsSaturating)
IID = Intrinsic::x86_avx512_vpdpbusd_128;
else if (VecWidth == 256 && !IsSaturating)
IID = Intrinsic::x86_avx512_vpdpbusd_256;
else if (VecWidth == 512 && !IsSaturating)
IID = Intrinsic::x86_avx512_vpdpbusd_512;
else if (VecWidth == 128 && IsSaturating)
IID = Intrinsic::x86_avx512_vpdpbusds_128;
else if (VecWidth == 256 && IsSaturating)
IID = Intrinsic::x86_avx512_vpdpbusds_256;
else if (VecWidth == 512 && IsSaturating)
IID = Intrinsic::x86_avx512_vpdpbusds_512;
else
llvm_unreachable("Unexpected intrinsic");
Value *Args[] = { CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(2) };
Rep = Builder.CreateCall(Intrinsic::getDeclaration(CI->getModule(), IID),
Args);
Value *PassThru = ZeroMask ? ConstantAggregateZero::get(CI->getType())
: CI->getArgOperand(0);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep, PassThru);
} else if (IsX86 && (Name.startswith("avx512.mask.vpdpwssd.") ||
Name.startswith("avx512.maskz.vpdpwssd.") ||
Name.startswith("avx512.mask.vpdpwssds.") ||
Name.startswith("avx512.maskz.vpdpwssds."))) {
bool ZeroMask = Name[11] == 'z';
bool IsSaturating = Name[ZeroMask ? 21 : 20] == 's';
unsigned VecWidth = CI->getType()->getPrimitiveSizeInBits();
Intrinsic::ID IID;
if (VecWidth == 128 && !IsSaturating)
IID = Intrinsic::x86_avx512_vpdpwssd_128;
else if (VecWidth == 256 && !IsSaturating)
IID = Intrinsic::x86_avx512_vpdpwssd_256;
else if (VecWidth == 512 && !IsSaturating)
IID = Intrinsic::x86_avx512_vpdpwssd_512;
else if (VecWidth == 128 && IsSaturating)
IID = Intrinsic::x86_avx512_vpdpwssds_128;
else if (VecWidth == 256 && IsSaturating)
IID = Intrinsic::x86_avx512_vpdpwssds_256;
else if (VecWidth == 512 && IsSaturating)
IID = Intrinsic::x86_avx512_vpdpwssds_512;
else
llvm_unreachable("Unexpected intrinsic");
Value *Args[] = { CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(2) };
Rep = Builder.CreateCall(Intrinsic::getDeclaration(CI->getModule(), IID),
Args);
Value *PassThru = ZeroMask ? ConstantAggregateZero::get(CI->getType())
: CI->getArgOperand(0);
Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep, PassThru);
[X86] Merge addcarryx/addcarry intrinsic into a single addcarry intrinsic. Both intrinsics do the exact same thing so we really only need one. Earlier in the 8.0 cycle we changed the signature of this intrinsic without renaming it. But it looks difficult to get the autoupgrade code to allow me to merge the intrinsics and change the signature at the same time. So I've renamed the intrinsic slightly for the new merged intrinsic. I'm skipping autoupgrading from the previous new to 8.0 signature. I've also renamed the subborrow for consistency. llvm-svn: 348737
2018-12-10 07:07:50 +01:00
} else if (IsX86 && (Name == "addcarryx.u32" || Name == "addcarryx.u64" ||
Name == "addcarry.u32" || Name == "addcarry.u64" ||
Name == "subborrow.u32" || Name == "subborrow.u64")) {
Intrinsic::ID IID;
if (Name[0] == 'a' && Name.back() == '2')
IID = Intrinsic::x86_addcarry_32;
else if (Name[0] == 'a' && Name.back() == '4')
IID = Intrinsic::x86_addcarry_64;
else if (Name[0] == 's' && Name.back() == '2')
IID = Intrinsic::x86_subborrow_32;
else if (Name[0] == 's' && Name.back() == '4')
IID = Intrinsic::x86_subborrow_64;
else
llvm_unreachable("Unexpected intrinsic");
// Make a call with 3 operands.
Value *Args[] = { CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(2)};
Value *NewCall = Builder.CreateCall(
Intrinsic::getDeclaration(CI->getModule(), IID),
Args);
// Extract the second result and store it.
Value *Data = Builder.CreateExtractValue(NewCall, 1);
// Cast the pointer to the right type.
Value *Ptr = Builder.CreateBitCast(CI->getArgOperand(3),
llvm::PointerType::getUnqual(Data->getType()));
[Alignment][NFC] Use Align with CreateAlignedStore Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet, bollu Subscribers: arsenm, jvesely, nhaehnle, hiraditya, kerbowa, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73274
2020-01-23 16:18:34 +01:00
Builder.CreateAlignedStore(Data, Ptr, Align(1));
[X86] Merge addcarryx/addcarry intrinsic into a single addcarry intrinsic. Both intrinsics do the exact same thing so we really only need one. Earlier in the 8.0 cycle we changed the signature of this intrinsic without renaming it. But it looks difficult to get the autoupgrade code to allow me to merge the intrinsics and change the signature at the same time. So I've renamed the intrinsic slightly for the new merged intrinsic. I'm skipping autoupgrading from the previous new to 8.0 signature. I've also renamed the subborrow for consistency. llvm-svn: 348737
2018-12-10 07:07:50 +01:00
// Replace the original call result with the first result of the new call.
Value *CF = Builder.CreateExtractValue(NewCall, 0);
CI->replaceAllUsesWith(CF);
Rep = nullptr;
[X86] Merge some of the autoupgrade handling for masked intrinsics that just need to upgrade to an unmasked version plus a select. NFCI These are were previously grouped in small groups of similarish intrinsics. But all the intrinsics have the same number of arguments and the same order. So we can move them all into a larger group for handling. llvm-svn: 329549
2018-04-09 08:15:09 +02:00
} else if (IsX86 && Name.startswith("avx512.mask.") &&
upgradeAVX512MaskToSelect(Name, Builder, *CI, Rep)) {
// Rep will be updated by the call in the condition.
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 02:00:32 +01:00
} else if (IsNVVM && (Name == "abs.i" || Name == "abs.ll")) {
Value *Arg = CI->getArgOperand(0);
Value *Neg = Builder.CreateNeg(Arg, "neg");
Value *Cmp = Builder.CreateICmpSGE(
Arg, llvm::Constant::getNullValue(Arg->getType()), "abs.cond");
Rep = Builder.CreateSelect(Cmp, Arg, Neg, "abs");
[NVPTX] Use atomicrmw fadd instead of intrinsics AutoUpgrade the old intrinsics to atomicrmw fadd. llvm-svn: 365796
2019-07-11 19:11:25 +02:00
} else if (IsNVVM && (Name.startswith("atomic.load.add.f32.p") ||
Name.startswith("atomic.load.add.f64.p"))) {
Value *Ptr = CI->getArgOperand(0);
Value *Val = CI->getArgOperand(1);
Rep = Builder.CreateAtomicRMW(AtomicRMWInst::FAdd, Ptr, Val,
AtomicOrdering::SequentiallyConsistent);
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 02:00:32 +01:00
} else if (IsNVVM && (Name == "max.i" || Name == "max.ll" ||
Name == "max.ui" || Name == "max.ull")) {
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
Value *Cmp = Name.endswith(".ui") || Name.endswith(".ull")
? Builder.CreateICmpUGE(Arg0, Arg1, "max.cond")
: Builder.CreateICmpSGE(Arg0, Arg1, "max.cond");
Rep = Builder.CreateSelect(Cmp, Arg0, Arg1, "max");
} else if (IsNVVM && (Name == "min.i" || Name == "min.ll" ||
Name == "min.ui" || Name == "min.ull")) {
Value *Arg0 = CI->getArgOperand(0);
Value *Arg1 = CI->getArgOperand(1);
Value *Cmp = Name.endswith(".ui") || Name.endswith(".ull")
? Builder.CreateICmpULE(Arg0, Arg1, "min.cond")
: Builder.CreateICmpSLE(Arg0, Arg1, "min.cond");
Rep = Builder.CreateSelect(Cmp, Arg0, Arg1, "min");
} else if (IsNVVM && Name == "clz.ll") {
// llvm.nvvm.clz.ll returns an i32, but llvm.ctlz.i64 and returns an i64.
Value *Arg = CI->getArgOperand(0);
Value *Ctlz = Builder.CreateCall(
Intrinsic::getDeclaration(F->getParent(), Intrinsic::ctlz,
{Arg->getType()}),
{Arg, Builder.getFalse()}, "ctlz");
Rep = Builder.CreateTrunc(Ctlz, Builder.getInt32Ty(), "ctlz.trunc");
} else if (IsNVVM && Name == "popc.ll") {
// llvm.nvvm.popc.ll returns an i32, but llvm.ctpop.i64 and returns an
// i64.
Value *Arg = CI->getArgOperand(0);
Value *Popc = Builder.CreateCall(
Intrinsic::getDeclaration(F->getParent(), Intrinsic::ctpop,
{Arg->getType()}),
Arg, "ctpop");
Rep = Builder.CreateTrunc(Popc, Builder.getInt32Ty(), "ctpop.trunc");
} else if (IsNVVM && Name == "h2f") {
Rep = Builder.CreateCall(Intrinsic::getDeclaration(
F->getParent(), Intrinsic::convert_from_fp16,
{Builder.getFloatTy()}),
CI->getArgOperand(0), "h2f");
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 07:10:55 +01:00
} else {
[X86] Greatly simplify the llvm.x86.avx.vpermil.* auto-upgrade code. We can fully derive everything using types of the intrinsic arguments rather than writing separate loops for each intrinsic. NFC llvm-svn: 272496
2016-06-12 05:10:47 +02:00
llvm_unreachable("Unknown function for CallInst upgrade.");
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 07:10:55 +01:00
}
[SSP] Remove llvm.stackprotectorcheck. This is a cleanup patch for SSP support in LLVM. There is no functional change. llvm.stackprotectorcheck is not needed, because SelectionDAG isn't actually lowering it in SelectBasicBlock; rather, it adds check code in FinishBasicBlock, ignoring the position where the intrinsic is inserted (See FindSplitPointForStackProtector()). llvm-svn: 265851
2016-04-08 23:26:31 +02:00
if (Rep)
CI->replaceAllUsesWith(Rep);
Add auto upgrade support for x86 pcmpgt/pcmpeq intrinics removed in r149367. llvm-svn: 149678
2012-02-03 07:10:55 +01:00
CI->eraseFromParent();
return;
}
Remove alignment argument from memcpy/memmove/memset in favour of alignment attributes (Step 1) Summary: This is a resurrection of work first proposed and discussed in Aug 2015: http://lists.llvm.org/pipermail/llvm-dev/2015-August/089384.html and initially landed (but then backed out) in Nov 2015: http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20151109/312083.html The @llvm.memcpy/memmove/memset intrinsics currently have an explicit argument which is required to be a constant integer. It represents the alignment of the dest (and source), and so must be the minimum of the actual alignment of the two. This change is the first in a series that allows source and dest to each have their own alignments by using the alignment attribute on their arguments. In this change we: 1) Remove the alignment argument. 2) Add alignment attributes to the source & dest arguments. We, temporarily, require that the alignments for source & dest be equal. For example, code which used to read: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 100, i32 4, i1 false) will now read call void @llvm.memcpy.p0i8.p0i8.i32(i8* align 4 %dest, i8* align 4 %src, i32 100, i1 false) Downstream users may have to update their lit tests that check for @llvm.memcpy/memmove/memset call/declaration patterns. The following extended sed script may help with updating the majority of your tests, but it does not catch all possible patterns so some manual checking and updating will be required. s~declare void @llvm\.mem(set|cpy|move)\.p([^(]*)\((.*), i32, i1\)~declare void @llvm.mem\1.p\2(\3, i1)~g s~call void @llvm\.memset\.p([^(]*)i8\(i8([^*]*)\* (.*), i8 (.*), i8 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i8(i8\2* \3, i8 \4, i8 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i16\(i8([^*]*)\* (.*), i8 (.*), i16 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i16(i8\2* \3, i8 \4, i16 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i32\(i8([^*]*)\* (.*), i8 (.*), i32 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i32(i8\2* \3, i8 \4, i32 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i64\(i8([^*]*)\* (.*), i8 (.*), i64 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i64(i8\2* \3, i8 \4, i64 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i128\(i8([^*]*)\* (.*), i8 (.*), i128 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i128(i8\2* \3, i8 \4, i128 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i8\(i8([^*]*)\* (.*), i8 (.*), i8 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i8(i8\2* align \6 \3, i8 \4, i8 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i16\(i8([^*]*)\* (.*), i8 (.*), i16 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i16(i8\2* align \6 \3, i8 \4, i16 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i32\(i8([^*]*)\* (.*), i8 (.*), i32 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i32(i8\2* align \6 \3, i8 \4, i32 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i64\(i8([^*]*)\* (.*), i8 (.*), i64 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i64(i8\2* align \6 \3, i8 \4, i64 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i128\(i8([^*]*)\* (.*), i8 (.*), i128 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i128(i8\2* align \6 \3, i8 \4, i128 \5, i1 \7)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i8\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i8 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i8(i8\3* \4, i8\5* \6, i8 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i16\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i16 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i16(i8\3* \4, i8\5* \6, i16 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i32\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i32 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i32(i8\3* \4, i8\5* \6, i32 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i64\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i64 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i64(i8\3* \4, i8\5* \6, i64 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i128\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i128 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i128(i8\3* \4, i8\5* \6, i128 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i8\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i8 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i8(i8\3* align \8 \4, i8\5* align \8 \6, i8 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i16\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i16 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i16(i8\3* align \8 \4, i8\5* align \8 \6, i16 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i32\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i32 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i32(i8\3* align \8 \4, i8\5* align \8 \6, i32 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i64\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i64 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i64(i8\3* align \8 \4, i8\5* align \8 \6, i64 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i128\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i128 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i128(i8\3* align \8 \4, i8\5* align \8 \6, i128 \7, i1 \9)~g The remaining changes in the series will: Step 2) Expand the IRBuilder API to allow creation of memcpy/memmove with differing source and dest alignments. Step 3) Update Clang to use the new IRBuilder API. Step 4) Update Polly to use the new IRBuilder API. Step 5) Update LLVM passes that create memcpy/memmove calls to use the new IRBuilder API, and those that use use MemIntrinsicInst::[get|set]Alignment() to use getDestAlignment() and getSourceAlignment() instead. Step 6) Remove the single-alignment IRBuilder API for memcpy/memmove, and the MemIntrinsicInst::[get|set]Alignment() methods. Reviewers: pete, hfinkel, lhames, reames, bollu Reviewed By: reames Subscribers: niosHD, reames, jholewinski, qcolombet, jfb, sanjoy, arsenm, dschuff, dylanmckay, mehdi_amini, sdardis, nemanjai, david2050, nhaehnle, javed.absar, sbc100, jgravelle-google, eraman, aheejin, kbarton, JDevlieghere, asb, rbar, johnrusso, simoncook, jordy.potman.lists, apazos, sabuasal, llvm-commits Differential Revision: https://reviews.llvm.org/D41675 llvm-svn: 322965
2018-01-19 18:13:12 +01:00
const auto &DefaultCase = [&NewFn, &CI]() -> void {
Implement intrinsic mangling for literal struct types. Fixes PR 31921 Summary: Predicateinfo requires an ugly workaround to try to avoid literal struct types due to the intrinsic mangling not being implemented. This workaround actually does not work in all cases (you can hit the assert by bootstrapping with -print-predicateinfo), and can't be made to work without DFS'ing the type (IE copying getMangledStr and using a version that detects if it would crash). Rather than do that, i just implemented the mangling. It seems simple, since they are unified structurally. Looking at the overloaded-mangling testcase we have, it actually turns out the gc intrinsics will *also* crash if you try to use a literal struct. Thus, the testcase added fails before this patch, and works after, without needing to resort to predicateinfo. Reviewers: chandlerc, davide Subscribers: llvm-commits, sanjoy Differential Revision: https://reviews.llvm.org/D29925 llvm-svn: 295253
2017-02-16 00:16:20 +01:00
// Handle generic mangling change, but nothing else
assert(
(CI->getCalledFunction()->getName() != NewFn->getName()) &&
"Unknown function for CallInst upgrade and isn't just a name change");
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
CI->setCalledFunction(NewFn);
Remove alignment argument from memcpy/memmove/memset in favour of alignment attributes (Step 1) Summary: This is a resurrection of work first proposed and discussed in Aug 2015: http://lists.llvm.org/pipermail/llvm-dev/2015-August/089384.html and initially landed (but then backed out) in Nov 2015: http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20151109/312083.html The @llvm.memcpy/memmove/memset intrinsics currently have an explicit argument which is required to be a constant integer. It represents the alignment of the dest (and source), and so must be the minimum of the actual alignment of the two. This change is the first in a series that allows source and dest to each have their own alignments by using the alignment attribute on their arguments. In this change we: 1) Remove the alignment argument. 2) Add alignment attributes to the source & dest arguments. We, temporarily, require that the alignments for source & dest be equal. For example, code which used to read: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 100, i32 4, i1 false) will now read call void @llvm.memcpy.p0i8.p0i8.i32(i8* align 4 %dest, i8* align 4 %src, i32 100, i1 false) Downstream users may have to update their lit tests that check for @llvm.memcpy/memmove/memset call/declaration patterns. The following extended sed script may help with updating the majority of your tests, but it does not catch all possible patterns so some manual checking and updating will be required. s~declare void @llvm\.mem(set|cpy|move)\.p([^(]*)\((.*), i32, i1\)~declare void @llvm.mem\1.p\2(\3, i1)~g s~call void @llvm\.memset\.p([^(]*)i8\(i8([^*]*)\* (.*), i8 (.*), i8 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i8(i8\2* \3, i8 \4, i8 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i16\(i8([^*]*)\* (.*), i8 (.*), i16 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i16(i8\2* \3, i8 \4, i16 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i32\(i8([^*]*)\* (.*), i8 (.*), i32 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i32(i8\2* \3, i8 \4, i32 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i64\(i8([^*]*)\* (.*), i8 (.*), i64 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i64(i8\2* \3, i8 \4, i64 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i128\(i8([^*]*)\* (.*), i8 (.*), i128 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i128(i8\2* \3, i8 \4, i128 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i8\(i8([^*]*)\* (.*), i8 (.*), i8 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i8(i8\2* align \6 \3, i8 \4, i8 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i16\(i8([^*]*)\* (.*), i8 (.*), i16 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i16(i8\2* align \6 \3, i8 \4, i16 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i32\(i8([^*]*)\* (.*), i8 (.*), i32 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i32(i8\2* align \6 \3, i8 \4, i32 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i64\(i8([^*]*)\* (.*), i8 (.*), i64 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i64(i8\2* align \6 \3, i8 \4, i64 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i128\(i8([^*]*)\* (.*), i8 (.*), i128 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i128(i8\2* align \6 \3, i8 \4, i128 \5, i1 \7)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i8\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i8 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i8(i8\3* \4, i8\5* \6, i8 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i16\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i16 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i16(i8\3* \4, i8\5* \6, i16 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i32\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i32 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i32(i8\3* \4, i8\5* \6, i32 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i64\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i64 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i64(i8\3* \4, i8\5* \6, i64 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i128\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i128 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i128(i8\3* \4, i8\5* \6, i128 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i8\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i8 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i8(i8\3* align \8 \4, i8\5* align \8 \6, i8 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i16\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i16 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i16(i8\3* align \8 \4, i8\5* align \8 \6, i16 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i32\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i32 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i32(i8\3* align \8 \4, i8\5* align \8 \6, i32 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i64\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i64 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i64(i8\3* align \8 \4, i8\5* align \8 \6, i64 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i128\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i128 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i128(i8\3* align \8 \4, i8\5* align \8 \6, i128 \7, i1 \9)~g The remaining changes in the series will: Step 2) Expand the IRBuilder API to allow creation of memcpy/memmove with differing source and dest alignments. Step 3) Update Clang to use the new IRBuilder API. Step 4) Update Polly to use the new IRBuilder API. Step 5) Update LLVM passes that create memcpy/memmove calls to use the new IRBuilder API, and those that use use MemIntrinsicInst::[get|set]Alignment() to use getDestAlignment() and getSourceAlignment() instead. Step 6) Remove the single-alignment IRBuilder API for memcpy/memmove, and the MemIntrinsicInst::[get|set]Alignment() methods. Reviewers: pete, hfinkel, lhames, reames, bollu Reviewed By: reames Subscribers: niosHD, reames, jholewinski, qcolombet, jfb, sanjoy, arsenm, dschuff, dylanmckay, mehdi_amini, sdardis, nemanjai, david2050, nhaehnle, javed.absar, sbc100, jgravelle-google, eraman, aheejin, kbarton, JDevlieghere, asb, rbar, johnrusso, simoncook, jordy.potman.lists, apazos, sabuasal, llvm-commits Differential Revision: https://reviews.llvm.org/D41675 llvm-svn: 322965
2018-01-19 18:13:12 +01:00
};
CallInst *NewCall = nullptr;
switch (NewFn->getIntrinsicID()) {
default: {
DefaultCase();
Implement intrinsic mangling for literal struct types. Fixes PR 31921 Summary: Predicateinfo requires an ugly workaround to try to avoid literal struct types due to the intrinsic mangling not being implemented. This workaround actually does not work in all cases (you can hit the assert by bootstrapping with -print-predicateinfo), and can't be made to work without DFS'ing the type (IE copying getMangledStr and using a version that detects if it would crash). Rather than do that, i just implemented the mangling. It seems simple, since they are unified structurally. Looking at the overloaded-mangling testcase we have, it actually turns out the gc intrinsics will *also* crash if you try to use a literal struct. Thus, the testcase added fails before this patch, and works after, without needing to resort to predicateinfo. Reviewers: chandlerc, davide Subscribers: llvm-commits, sanjoy Differential Revision: https://reviews.llvm.org/D29925 llvm-svn: 295253
2017-02-16 00:16:20 +01:00
return;
}
Change semantics of fadd/fmul vector reductions. This patch changes how LLVM handles the accumulator/start value in the reduction, by never ignoring it regardless of the presence of fast-math flags on callsites. This change introduces the following new intrinsics to replace the existing ones: llvm.experimental.vector.reduce.fadd -> llvm.experimental.vector.reduce.v2.fadd llvm.experimental.vector.reduce.fmul -> llvm.experimental.vector.reduce.v2.fmul and adds functionality to auto-upgrade existing LLVM IR and bitcode. Reviewers: RKSimon, greened, dmgreen, nikic, simoll, aemerson Reviewed By: nikic Differential Revision: https://reviews.llvm.org/D60261 llvm-svn: 363035
2019-06-11 10:22:10 +02:00
case Intrinsic::experimental_vector_reduce_v2_fmul: {
SmallVector<Value *, 2> Args;
if (CI->isFast())
Args.push_back(ConstantFP::get(CI->getOperand(0)->getType(), 1.0));
else
Args.push_back(CI->getOperand(0));
Args.push_back(CI->getOperand(1));
NewCall = Builder.CreateCall(NewFn, Args);
cast<Instruction>(NewCall)->copyFastMathFlags(CI);
break;
}
case Intrinsic::experimental_vector_reduce_v2_fadd: {
SmallVector<Value *, 2> Args;
if (CI->isFast())
Args.push_back(Constant::getNullValue(CI->getOperand(0)->getType()));
else
Args.push_back(CI->getOperand(0));
Args.push_back(CI->getOperand(1));
NewCall = Builder.CreateCall(NewFn, Args);
cast<Instruction>(NewCall)->copyFastMathFlags(CI);
break;
}
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 12:56:37 +02:00
case Intrinsic::arm_neon_vld1:
case Intrinsic::arm_neon_vld2:
case Intrinsic::arm_neon_vld3:
case Intrinsic::arm_neon_vld4:
case Intrinsic::arm_neon_vld2lane:
case Intrinsic::arm_neon_vld3lane:
case Intrinsic::arm_neon_vld4lane:
case Intrinsic::arm_neon_vst1:
case Intrinsic::arm_neon_vst2:
case Intrinsic::arm_neon_vst3:
case Intrinsic::arm_neon_vst4:
case Intrinsic::arm_neon_vst2lane:
case Intrinsic::arm_neon_vst3lane:
case Intrinsic::arm_neon_vst4lane: {
SmallVector<Value *, 4> Args(CI->arg_operands().begin(),
CI->arg_operands().end());
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall = Builder.CreateCall(NewFn, Args);
break;
[ARM][NEON] Use address space in vld([1234]|[234]lane) and vst([1234]|[234]lane) instructions This commit changes the interface of the vld[1234], vld[234]lane, and vst[1234], vst[234]lane ARM neon intrinsics and associates an address space with the pointer that these intrinsics take. This changes, e.g., <2 x i32> @llvm.arm.neon.vld1.v2i32(i8*, i32) to <2 x i32> @llvm.arm.neon.vld1.v2i32.p0i8(i8*, i32) This change ensures that address spaces are fully taken into account in the ARM target during lowering of interleaved loads and stores. Differential Revision: http://reviews.llvm.org/D12985 llvm-svn: 248887
2015-09-30 12:56:37 +02:00
}
[AArch64] Add support for lowering bitreverse to the rbit instruction. Differential Revision: https://reviews.llvm.org/D28379 llvm-svn: 291575
2017-01-10 18:20:33 +01:00
case Intrinsic::bitreverse:
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall = Builder.CreateCall(NewFn, {CI->getArgOperand(0)});
break;
[AArch64] Add support for lowering bitreverse to the rbit instruction. Differential Revision: https://reviews.llvm.org/D28379 llvm-svn: 291575
2017-01-10 18:20:33 +01:00
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 05:26:04 +01:00
case Intrinsic::ctlz:
revert my previous patches that introduced an additional parameter to the objectsize intrinsic. After a lot of discussion, we realized it's not the best option for run-time bounds checking llvm-svn: 157255
2012-05-22 17:25:31 +02:00
case Intrinsic::cttz:
Switch llvm.cttz and llvm.ctlz to accept a second i1 parameter which indicates whether the intrinsic has a defined result for a first argument equal to zero. This will eventually allow these intrinsics to accurately model the semantics of GCC's __builtin_ctz and __builtin_clz and the X86 instructions (prior to AVX) which implement them. This patch merely sets the stage by extending the signature of these intrinsics and establishing auto-upgrade logic so that the old spelling still works both in IR and in bitcode. The upgrade logic preserves the existing (inefficient) semantics. This patch should not change any behavior. CodeGen isn't updated because it can use the existing semantics regardless of the flag's value. Note that this will be followed by API updates to Clang and DragonEgg. Reviewed by Nick Lewycky! llvm-svn: 146357
2011-12-12 05:26:04 +01:00
assert(CI->getNumArgOperands() == 1 &&
"Mismatch between function args and call args");
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall =
Builder.CreateCall(NewFn, {CI->getArgOperand(0), Builder.getFalse()});
break;
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-10 20:42:51 +02:00
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-21 21:08:59 +01:00
case Intrinsic::objectsize: {
Value *NullIsUnknownSize = CI->getNumArgOperands() == 2
? Builder.getFalse()
: CI->getArgOperand(2);
Add a 'dynamic' parameter to the objectsize intrinsic This is meant to be used with clang's __builtin_dynamic_object_size. When 'true' is passed to this parameter, the intrinsic has the potential to be folded into instructions that will be evaluated at run time. When 'false', the objectsize intrinsic behaviour is unchanged. rdar://32212419 Differential revision: https://reviews.llvm.org/D56761 llvm-svn: 352664
2019-01-30 21:34:35 +01:00
Value *Dynamic =
Fix auto-upgrade for the new parameter to llvm.objectsize r352664 added a 'dynamic' parameter to objectsize, but the AutoUpgrade changes were incomplete. Also, fix an off-by-one error I made in the upgrade logic that is now no longer unreachable. Differential revision: https://reviews.llvm.org/D58071 llvm-svn: 353884
2019-02-12 22:55:38 +01:00
CI->getNumArgOperands() < 4 ? Builder.getFalse() : CI->getArgOperand(3);
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-21 21:08:59 +01:00
NewCall = Builder.CreateCall(
Add a 'dynamic' parameter to the objectsize intrinsic This is meant to be used with clang's __builtin_dynamic_object_size. When 'true' is passed to this parameter, the intrinsic has the potential to be folded into instructions that will be evaluated at run time. When 'false', the objectsize intrinsic behaviour is unchanged. rdar://32212419 Differential revision: https://reviews.llvm.org/D56761 llvm-svn: 352664
2019-01-30 21:34:35 +01:00
NewFn, {CI->getArgOperand(0), CI->getArgOperand(1), NullIsUnknownSize, Dynamic});
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
break;
Let llvm.objectsize be conservative with null pointers This adds a parameter to @llvm.objectsize that makes it return conservative values if it's given null. This fixes PR23277. Differential Revision: https://reviews.llvm.org/D28494 llvm-svn: 298430
2017-03-21 21:08:59 +01:00
}
Change objectsize intrinsic to accept different address spaces. Bitcasting everything to i8* won't work. Autoupgrade the old intrinsic declarations to use the new mangling. llvm-svn: 192117
2013-10-07 20:06:48 +02:00
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 02:00:32 +01:00
case Intrinsic::ctpop:
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall = Builder.CreateCall(NewFn, {CI->getArgOperand(0)});
break;
[NVPTX] Auto-upgrade some NVPTX intrinsics to LLVM target-generic code. Summary: Specifically, we upgrade llvm.nvvm.: * brev{32,64} * clz.{i,ll} * popc.{i,ll} * abs.{i,ll} * {min,max}.{i,ll,u,ull} * h2f These either map directly to an existing LLVM target-generic intrinsic or map to a simple LLVM target-generic idiom. In all cases, we check that the code we generate is lowered to PTX as we expect. These builtins don't need to be backfilled in clang: They're not accessible to user code from nvcc. Reviewers: tra Subscribers: majnemer, cfe-commits, llvm-commits, jholewinski Differential Revision: https://reviews.llvm.org/D28793 llvm-svn: 292694
2017-01-21 02:00:32 +01:00
case Intrinsic::convert_from_fp16:
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall = Builder.CreateCall(NewFn, {CI->getArgOperand(0)});
break;
This is one of the first steps at moving to replace target-dependent intrinsics with target-indepdent intrinsics. The first instruction(s) to be handled are the vector versions of count leading zeros (ctlz). The changes here are to clang so that it generates a target independent vector ctlz when it sees an ARM dependent vector ctlz. The changes in llvm are to match the target independent vector ctlz and in VMCore/AutoUpgrade.cpp to update any existing bc files containing ARM dependent vector ctlzs with target-independent ctlzs. There are also changes to an existing test case in llvm for ARM vector count instructions and a new test for the bitcode upgrade. <rdar://problem/11831778> There is deliberately no test for the change to clang, as so far as I know, no consensus has been reached regarding how to test neon instructions in clang; q.v. <rdar://problem/8762292> llvm-svn: 160200
2012-07-14 01:25:25 +02:00
Remove the obsolete offset parameter from @llvm.dbg.value There is no situation where this rarely-used argument cannot be substituted with a DIExpression and removing it allows us to simplify the DWARF backend. Note that this patch does not yet remove any of the newly dead code. rdar://problem/33580047 Differential Revision: https://reviews.llvm.org/D35951 llvm-svn: 309426
2017-07-28 22:21:02 +02:00
case Intrinsic::dbg_value:
// Upgrade from the old version that had an extra offset argument.
assert(CI->getNumArgOperands() == 4);
// Drop nonzero offsets instead of attempting to upgrade them.
if (auto *Offset = dyn_cast_or_null<Constant>(CI->getArgOperand(1)))
if (Offset->isZeroValue()) {
NewCall = Builder.CreateCall(
NewFn,
{CI->getArgOperand(0), CI->getArgOperand(2), CI->getArgOperand(3)});
break;
}
CI->eraseFromParent();
return;
Fix intrinsics for XOP frczss/sd instructions. These instructions only take one source register and zero the upper bits of the destination rather than preserving them. llvm-svn: 158396
2012-06-13 09:18:53 +02:00
case Intrinsic::x86_xop_vfrcz_ss:
case Intrinsic::x86_xop_vfrcz_sd:
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall = Builder.CreateCall(NewFn, {CI->getArgOperand(1)});
break;
Fix intrinsics for XOP frczss/sd instructions. These instructions only take one source register and zero the upper bits of the destination rather than preserving them. llvm-svn: 158396
2012-06-13 09:18:53 +02:00
[X86][XOP] Support for VPERMIL2PD/VPERMIL2PS 2-input shuffle instructions This patch begins adding support for lowering to the XOP VPERMIL2PD/VPERMIL2PS shuffle instructions - adding the X86ISD::VPERMIL2 opcode and cleaning up the usage. The internal llvm intrinsics were assuming the shuffle mask operand was the same type as the float/double input operands (I guess to simplify the intrinsic definitions in X86InstrXOP.td to a single value type). These needed changing to integer types (matching the clang builtin and the AMD intrinsics definitions), an auto upgrade path is added to convert old calls. Mask decoding/target shuffle support will be added in future patches. Differential Revision: http://reviews.llvm.org/D20049 llvm-svn: 271633
2016-06-03 10:06:03 +02:00
case Intrinsic::x86_xop_vpermil2pd:
case Intrinsic::x86_xop_vpermil2ps:
case Intrinsic::x86_xop_vpermil2pd_256:
case Intrinsic::x86_xop_vpermil2ps_256: {
SmallVector<Value *, 4> Args(CI->arg_operands().begin(),
CI->arg_operands().end());
VectorType *FltIdxTy = cast<VectorType>(Args[2]->getType());
VectorType *IntIdxTy = VectorType::getInteger(FltIdxTy);
Args[2] = Builder.CreateBitCast(Args[2], IntIdxTy);
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall = Builder.CreateCall(NewFn, Args);
break;
[X86][XOP] Support for VPERMIL2PD/VPERMIL2PS 2-input shuffle instructions This patch begins adding support for lowering to the XOP VPERMIL2PD/VPERMIL2PS shuffle instructions - adding the X86ISD::VPERMIL2 opcode and cleaning up the usage. The internal llvm intrinsics were assuming the shuffle mask operand was the same type as the float/double input operands (I guess to simplify the intrinsic definitions in X86InstrXOP.td to a single value type). These needed changing to integer types (matching the clang builtin and the AMD intrinsics definitions), an auto upgrade path is added to convert old calls. Mask decoding/target shuffle support will be added in future patches. Differential Revision: http://reviews.llvm.org/D20049 llvm-svn: 271633
2016-06-03 10:06:03 +02:00
}
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-10 20:42:51 +02:00
case Intrinsic::x86_sse41_ptestc:
case Intrinsic::x86_sse41_ptestz:
Fix intrinsics for XOP frczss/sd instructions. These instructions only take one source register and zero the upper bits of the destination rather than preserving them. llvm-svn: 158396
2012-06-13 09:18:53 +02:00
case Intrinsic::x86_sse41_ptestnzc: {
Add AutoUpgrade support for the SSE4 ptest intrinsics. Patch by Michael Kuperstein. llvm-svn: 158295
2012-06-10 20:42:51 +02:00
// The arguments for these intrinsics used to be v4f32, and changed
// to v2i64. This is purely a nop, since those are bitwise intrinsics.
// So, the only thing required is a bitcast for both arguments.
// First, check the arguments have the old type.
Value *Arg0 = CI->getArgOperand(0);
if (Arg0->getType() != VectorType::get(Type::getFloatTy(C), 4))
return;
// Old intrinsic, add bitcasts
Value *Arg1 = CI->getArgOperand(1);
[opaque pointer type] Verifier/AutoUpgrade: Remove a few uses of PointerType::getElementType llvm-svn: 235777
2015-04-24 23:16:07 +02:00
Type *NewVecTy = VectorType::get(Type::getInt64Ty(C), 2);
Value *BC0 = Builder.CreateBitCast(Arg0, NewVecTy, "cast");
Value *BC1 = Builder.CreateBitCast(Arg1, NewVecTy, "cast");
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall = Builder.CreateCall(NewFn, {BC0, BC1});
break;
[x86] Fix a pretty horrible bug and inconsistency in the x86 asm parsing (and latent bug in the instruction definitions). This is effectively a revert of r136287 which tried to address a specific and narrow case of immediate operands failing to be accepted by x86 instructions with a pretty heavy hammer: it introduced a new kind of operand that behaved differently. All of that is removed with this commit, but the test cases are both preserved and enhanced. The core problem that r136287 and this commit are trying to handle is that gas accepts both of the following instructions: insertps $192, %xmm0, %xmm1 insertps $-64, %xmm0, %xmm1 These will encode to the same byte sequence, with the immediate occupying an 8-bit entry. The first form was fixed by r136287 but that broke the prior handling of the second form! =[ Ironically, we would still emit the second form in some cases and then be unable to re-assemble the output. The reason why the first instruction failed to be handled is because prior to r136287 the operands ere marked 'i32i8imm' which forces them to be sign-extenable. Clearly, that won't work for 192 in a single byte. However, making thim zero-extended or "unsigned" doesn't really address the core issue either because it breaks negative immediates. The correct fix is to make these operands 'i8imm' reflecting that they can be either signed or unsigned but must be 8-bit immediates. This patch backs out r136287 and then changes those places as well as some others to use 'i8imm' rather than one of the extended variants. Naturally, this broke something else. The custom DAG nodes had to be updated to have a much more accurate type constraint of an i8 node, and a bunch of Pat immediates needed to be specified as i8 values. The fallout didn't end there though. We also then ceased to be able to match the instruction-specific intrinsics to the instructions so modified. Digging, this is because they too used i32 rather than i8 in their signature. So I've also switched those intrinsics to i8 arguments in line with the instructions. In order to make the intrinsic adjustments of course, I also had to add auto upgrading for the intrinsics. I suspect that the intrinsic argument types may have led everything down this rabbit hole. Pretty happy with the result. llvm-svn: 217310
2014-09-06 12:00:01 +02:00
}
[X86] Modify the the rdtscp intrinsic to return values instead of taking a pointer argument Similar to what was recently done for addcarry/subborrow and has been done for rdrand/rdseed for a while. It's better to use two results and an explicit store in IR when the store isn't part of the semantics of the instruction. This allows store->load forwarding to happen in the middle end. Or the store to be removed if its never loaded. Differential Revision: https://reviews.llvm.org/D51803 llvm-svn: 341698
2018-09-07 21:14:15 +02:00
case Intrinsic::x86_rdtscp: {
// This used to take 1 arguments. If we have no arguments, it is already
// upgraded.
if (CI->getNumOperands() == 0)
return;
NewCall = Builder.CreateCall(NewFn);
// Extract the second result and store it.
Value *Data = Builder.CreateExtractValue(NewCall, 1);
// Cast the pointer to the right type.
Value *Ptr = Builder.CreateBitCast(CI->getArgOperand(0),
llvm::PointerType::getUnqual(Data->getType()));
[Alignment][NFC] Use Align with CreateAlignedStore Summary: This is patch is part of a series to introduce an Alignment type. See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html See this patch for the introduction of the type: https://reviews.llvm.org/D64790 Reviewers: courbet, bollu Subscribers: arsenm, jvesely, nhaehnle, hiraditya, kerbowa, cfe-commits, llvm-commits Tags: #clang, #llvm Differential Revision: https://reviews.llvm.org/D73274
2020-01-23 16:18:34 +01:00
Builder.CreateAlignedStore(Data, Ptr, Align(1));
[X86] Modify the the rdtscp intrinsic to return values instead of taking a pointer argument Similar to what was recently done for addcarry/subborrow and has been done for rdrand/rdseed for a while. It's better to use two results and an explicit store in IR when the store isn't part of the semantics of the instruction. This allows store->load forwarding to happen in the middle end. Or the store to be removed if its never loaded. Differential Revision: https://reviews.llvm.org/D51803 llvm-svn: 341698
2018-09-07 21:14:15 +02:00
// Replace the original call result with the first result of the new call.
Value *TSC = Builder.CreateExtractValue(NewCall, 0);
std::string Name = CI->getName();
if (!Name.empty()) {
CI->setName(Name + ".old");
NewCall->setName(Name);
}
CI->replaceAllUsesWith(TSC);
CI->eraseFromParent();
return;
}
[x86] Fix a pretty horrible bug and inconsistency in the x86 asm parsing (and latent bug in the instruction definitions). This is effectively a revert of r136287 which tried to address a specific and narrow case of immediate operands failing to be accepted by x86 instructions with a pretty heavy hammer: it introduced a new kind of operand that behaved differently. All of that is removed with this commit, but the test cases are both preserved and enhanced. The core problem that r136287 and this commit are trying to handle is that gas accepts both of the following instructions: insertps $192, %xmm0, %xmm1 insertps $-64, %xmm0, %xmm1 These will encode to the same byte sequence, with the immediate occupying an 8-bit entry. The first form was fixed by r136287 but that broke the prior handling of the second form! =[ Ironically, we would still emit the second form in some cases and then be unable to re-assemble the output. The reason why the first instruction failed to be handled is because prior to r136287 the operands ere marked 'i32i8imm' which forces them to be sign-extenable. Clearly, that won't work for 192 in a single byte. However, making thim zero-extended or "unsigned" doesn't really address the core issue either because it breaks negative immediates. The correct fix is to make these operands 'i8imm' reflecting that they can be either signed or unsigned but must be 8-bit immediates. This patch backs out r136287 and then changes those places as well as some others to use 'i8imm' rather than one of the extended variants. Naturally, this broke something else. The custom DAG nodes had to be updated to have a much more accurate type constraint of an i8 node, and a bunch of Pat immediates needed to be specified as i8 values. The fallout didn't end there though. We also then ceased to be able to match the instruction-specific intrinsics to the instructions so modified. Digging, this is because they too used i32 rather than i8 in their signature. So I've also switched those intrinsics to i8 arguments in line with the instructions. In order to make the intrinsic adjustments of course, I also had to add auto upgrading for the intrinsics. I suspect that the intrinsic argument types may have led everything down this rabbit hole. Pretty happy with the result. llvm-svn: 217310
2014-09-06 12:00:01 +02:00
case Intrinsic::x86_sse41_insertps:
case Intrinsic::x86_sse41_dppd:
case Intrinsic::x86_sse41_dpps:
case Intrinsic::x86_sse41_mpsadbw:
case Intrinsic::x86_avx_dp_ps_256:
case Intrinsic::x86_avx2_mpsadbw: {
// Need to truncate the last argument from i32 to i8 -- this argument models
// an inherently 8-bit immediate operand to these x86 instructions.
SmallVector<Value *, 4> Args(CI->arg_operands().begin(),
CI->arg_operands().end());
// Replace the last argument with a trunc.
Args.back() = Builder.CreateTrunc(Args.back(), Type::getInt8Ty(C), "trunc");
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall = Builder.CreateCall(NewFn, Args);
break;
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-17 23:33:23 +01:00
}
[AArch64] [ARM] Make a target-independent llvm.thread.pointer intrinsic. Both AArch64 and ARM support llvm.<arch>.thread.pointer intrinsics that just return the thread pointer. I have a pending patch that does the same for SystemZ (D19054), and there are many more targets that could benefit from one. This patch merges the ARM and AArch64 intrinsics into a single target independent one that will also be used by subsequent targets. Differential Revision: http://reviews.llvm.org/D19098 llvm-svn: 266818
2016-04-19 22:51:05 +02:00
case Intrinsic::thread_pointer: {
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall = Builder.CreateCall(NewFn, {});
break;
[AArch64] [ARM] Make a target-independent llvm.thread.pointer intrinsic. Both AArch64 and ARM support llvm.<arch>.thread.pointer intrinsics that just return the thread pointer. I have a pending patch that does the same for SystemZ (D19054), and there are many more targets that could benefit from one. This patch merges the ARM and AArch64 intrinsics into a single target independent one that will also be used by subsequent targets. Differential Revision: http://reviews.llvm.org/D19098 llvm-svn: 266818
2016-04-19 22:51:05 +02:00
}
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-28 20:27:25 +02:00
Revert "Revert "Invariant start/end intrinsics overloaded for address space"" This reverts commit 32fc6488e48eafc0ca1bac1bd9cbf0008224d530. llvm-svn: 278609
2016-08-14 01:31:24 +02:00
case Intrinsic::invariant_start:
case Intrinsic::invariant_end:
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-28 20:27:25 +02:00
case Intrinsic::masked_load:
Support arbitrary address space pointers in masked gather/scatter intrinsics. Fixes PR31789 - When loop-vectorize tries to use these intrinsics for a non-default address space pointer we fail with a "Calling a function with a bad singature!" assertion. This patch solves this by adding the 'vector of pointers' argument as an overloaded type which will determine the address space. Differential revision: https://reviews.llvm.org/D31490 llvm-svn: 302018
2017-05-03 14:28:54 +02:00
case Intrinsic::masked_store:
case Intrinsic::masked_gather:
case Intrinsic::masked_scatter: {
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-28 20:27:25 +02:00
SmallVector<Value *, 4> Args(CI->arg_operands().begin(),
CI->arg_operands().end());
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
NewCall = Builder.CreateCall(NewFn, Args);
break;
}
Remove alignment argument from memcpy/memmove/memset in favour of alignment attributes (Step 1) Summary: This is a resurrection of work first proposed and discussed in Aug 2015: http://lists.llvm.org/pipermail/llvm-dev/2015-August/089384.html and initially landed (but then backed out) in Nov 2015: http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20151109/312083.html The @llvm.memcpy/memmove/memset intrinsics currently have an explicit argument which is required to be a constant integer. It represents the alignment of the dest (and source), and so must be the minimum of the actual alignment of the two. This change is the first in a series that allows source and dest to each have their own alignments by using the alignment attribute on their arguments. In this change we: 1) Remove the alignment argument. 2) Add alignment attributes to the source & dest arguments. We, temporarily, require that the alignments for source & dest be equal. For example, code which used to read: call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 100, i32 4, i1 false) will now read call void @llvm.memcpy.p0i8.p0i8.i32(i8* align 4 %dest, i8* align 4 %src, i32 100, i1 false) Downstream users may have to update their lit tests that check for @llvm.memcpy/memmove/memset call/declaration patterns. The following extended sed script may help with updating the majority of your tests, but it does not catch all possible patterns so some manual checking and updating will be required. s~declare void @llvm\.mem(set|cpy|move)\.p([^(]*)\((.*), i32, i1\)~declare void @llvm.mem\1.p\2(\3, i1)~g s~call void @llvm\.memset\.p([^(]*)i8\(i8([^*]*)\* (.*), i8 (.*), i8 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i8(i8\2* \3, i8 \4, i8 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i16\(i8([^*]*)\* (.*), i8 (.*), i16 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i16(i8\2* \3, i8 \4, i16 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i32\(i8([^*]*)\* (.*), i8 (.*), i32 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i32(i8\2* \3, i8 \4, i32 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i64\(i8([^*]*)\* (.*), i8 (.*), i64 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i64(i8\2* \3, i8 \4, i64 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i128\(i8([^*]*)\* (.*), i8 (.*), i128 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.memset.p\1i128(i8\2* \3, i8 \4, i128 \5, i1 \6)~g s~call void @llvm\.memset\.p([^(]*)i8\(i8([^*]*)\* (.*), i8 (.*), i8 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i8(i8\2* align \6 \3, i8 \4, i8 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i16\(i8([^*]*)\* (.*), i8 (.*), i16 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i16(i8\2* align \6 \3, i8 \4, i16 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i32\(i8([^*]*)\* (.*), i8 (.*), i32 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i32(i8\2* align \6 \3, i8 \4, i32 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i64\(i8([^*]*)\* (.*), i8 (.*), i64 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i64(i8\2* align \6 \3, i8 \4, i64 \5, i1 \7)~g s~call void @llvm\.memset\.p([^(]*)i128\(i8([^*]*)\* (.*), i8 (.*), i128 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.memset.p\1i128(i8\2* align \6 \3, i8 \4, i128 \5, i1 \7)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i8\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i8 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i8(i8\3* \4, i8\5* \6, i8 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i16\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i16 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i16(i8\3* \4, i8\5* \6, i16 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i32\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i32 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i32(i8\3* \4, i8\5* \6, i32 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i64\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i64 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i64(i8\3* \4, i8\5* \6, i64 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i128\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i128 (.*), i32 [01], i1 ([^)]*)\)~call void @llvm.mem\1.p\2i128(i8\3* \4, i8\5* \6, i128 \7, i1 \8)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i8\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i8 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i8(i8\3* align \8 \4, i8\5* align \8 \6, i8 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i16\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i16 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i16(i8\3* align \8 \4, i8\5* align \8 \6, i16 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i32\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i32 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i32(i8\3* align \8 \4, i8\5* align \8 \6, i32 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i64\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i64 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i64(i8\3* align \8 \4, i8\5* align \8 \6, i64 \7, i1 \9)~g s~call void @llvm\.mem(cpy|move)\.p([^(]*)i128\(i8([^*]*)\* (.*), i8([^*]*)\* (.*), i128 (.*), i32 ([0-9]*), i1 ([^)]*)\)~call void @llvm.mem\1.p\2i128(i8\3* align \8 \4, i8\5* align \8 \6, i128 \7, i1 \9)~g The remaining changes in the series will: Step 2) Expand the IRBuilder API to allow creation of memcpy/memmove with differing source and dest alignments. Step 3) Update Clang to use the new IRBuilder API. Step 4) Update Polly to use the new IRBuilder API. Step 5) Update LLVM passes that create memcpy/memmove calls to use the new IRBuilder API, and those that use use MemIntrinsicInst::[get|set]Alignment() to use getDestAlignment() and getSourceAlignment() instead. Step 6) Remove the single-alignment IRBuilder API for memcpy/memmove, and the MemIntrinsicInst::[get|set]Alignment() methods. Reviewers: pete, hfinkel, lhames, reames, bollu Reviewed By: reames Subscribers: niosHD, reames, jholewinski, qcolombet, jfb, sanjoy, arsenm, dschuff, dylanmckay, mehdi_amini, sdardis, nemanjai, david2050, nhaehnle, javed.absar, sbc100, jgravelle-google, eraman, aheejin, kbarton, JDevlieghere, asb, rbar, johnrusso, simoncook, jordy.potman.lists, apazos, sabuasal, llvm-commits Differential Revision: https://reviews.llvm.org/D41675 llvm-svn: 322965
2018-01-19 18:13:12 +01:00
case Intrinsic::memcpy:
case Intrinsic::memmove:
case Intrinsic::memset: {
// We have to make sure that the call signature is what we're expecting.
// We only want to change the old signatures by removing the alignment arg:
// @llvm.mem[cpy|move]...(i8*, i8*, i[32|i64], i32, i1)
// -> @llvm.mem[cpy|move]...(i8*, i8*, i[32|i64], i1)
// @llvm.memset...(i8*, i8, i[32|64], i32, i1)
// -> @llvm.memset...(i8*, i8, i[32|64], i1)
// Note: i8*'s in the above can be any pointer type
if (CI->getNumArgOperands() != 5) {
DefaultCase();
return;
}
// Remove alignment argument (3), and add alignment attributes to the
// dest/src pointers.
Value *Args[4] = {CI->getArgOperand(0), CI->getArgOperand(1),
CI->getArgOperand(2), CI->getArgOperand(4)};
NewCall = Builder.CreateCall(NewFn, Args);
auto *MemCI = cast<MemIntrinsic>(NewCall);
// All mem intrinsics support dest alignment.
const ConstantInt *Align = cast<ConstantInt>(CI->getArgOperand(3));
MemCI->setDestAlignment(Align->getZExtValue());
// Memcpy/Memmove also support source alignment.
if (auto *MTI = dyn_cast<MemTransferInst>(MemCI))
MTI->setSourceAlignment(Align->getZExtValue());
break;
}
Support arbitrary addrspace pointers in masked load/store intrinsics This is a resubmittion of 263158 change after fixing the existing problem with intrinsics mangling (see LTO and intrinsics mangling llvm-dev thread for details). This patch fixes the problem which occurs when loop-vectorize tries to use @llvm.masked.load/store intrinsic for a non-default addrspace pointer. It fails with "Calling a function with a bad signature!" assertion in CallInst constructor because it tries to pass a non-default addrspace pointer to the pointer argument which has default addrspace. The fix is to add pointer type as another overloaded type to @llvm.masked.load/store intrinsics. Reviewed By: reames Differential Revision: http://reviews.llvm.org/D17270 llvm-svn: 274043
2016-06-28 20:27:25 +02:00
}
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
assert(NewCall && "Should have either set this variable or returned through "
"the default case");
std::string Name = CI->getName();
if (!Name.empty()) {
CI->setName(Name + ".old");
NewCall->setName(Name);
Fix intrinsics for XOP frczss/sd instructions. These instructions only take one source register and zero the upper bits of the destination rather than preserving them. llvm-svn: 158396
2012-06-13 09:18:53 +02:00
}
Keep attributes, calling convention, etc, when remangling intrinsic Summary: Fix issue reported where intrinsic calling convention is dropped after r295253. Reviewers: sanjoy Subscribers: materi, llvm-commits Differential Revision: https://reviews.llvm.org/D30422 llvm-svn: 296563
2017-03-01 02:49:13 +01:00
CI->replaceAllUsesWith(NewCall);
CI->eraseFromParent();
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
}
try to make comments more meaningful; NFC Retry r266541 without the range-based-for-loop-change that was wrong. llvm-svn: 266658
2016-04-18 21:11:57 +02:00
void llvm::UpgradeCallsToIntrinsic(Function *F) {
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
assert(F && "Illegal attempt to upgrade a non-existent intrinsic.");
try to make comments more meaningful; NFC Retry r266541 without the range-based-for-loop-change that was wrong. llvm-svn: 266658
2016-04-18 21:11:57 +02:00
// Check if this function should be upgraded and get the replacement function
// if there is one.
rip out a ton of intrinsic modernization logic from AutoUpgrade.cpp, which is for pre-2.9 bitcode files. We keep x86 unaligned loads, movnt, crc32, and the target indep prefetch change. As usual, updating the testsuite is a PITA. llvm-svn: 133337
2011-06-18 08:05:24 +02:00
Function *NewFn;
Bring back int_x86_sse2_movl_dq intrinsic for backward compatibility. Make sure it's auto-upgraded to a shufflevector instruction. llvm-svn: 45131
2007-12-17 23:33:23 +01:00
if (UpgradeIntrinsicFunction(F, NewFn)) {
try to make comments more meaningful; NFC Retry r266541 without the range-based-for-loop-change that was wrong. llvm-svn: 266658
2016-04-18 21:11:57 +02:00
// Replace all users of the old function with the new function or new
// instructions. This is not a range loop because the call is deleted.
for (auto UI = F->user_begin(), UE = F->user_end(); UI != UE; )
Revert "use range loop, try to make comments more meaningful; NFCI" This reverts commit r266541 since it introduces a use-after-free: http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/11471 llvm-svn: 266550
2016-04-17 05:59:37 +02:00
if (CallInst *CI = dyn_cast<CallInst>(*UI++))
Remove always-true comparison, NFC. Summary: Looking at r241279, I noticed that UpgradedIntrinsics only gets written to in the following code: if (UpgradeIntrinsicFunction(&F, NewFn)) UpgradedIntrinsics[&F] = NewFn; Looking through UpgradeIntrinsicFunction, we always return false OR NewFn will be set to a different function from our source. This patch pulls the F != NewFn into UpgradeIntrinsicFunction as an assert, and removes the check from callers of UpgradeIntrinsicFunction. Reviewers: rafael, chandlerc Subscribers: llvm-commits-list Differential Revision: http://reviews.llvm.org/D10915 llvm-svn: 241369
2015-07-03 22:12:01 +02:00
UpgradeIntrinsicCall(CI, NewFn);
try to make comments more meaningful; NFC Retry r266541 without the range-based-for-loop-change that was wrong. llvm-svn: 266658
2016-04-18 21:11:57 +02:00
Remove always-true comparison, NFC. Summary: Looking at r241279, I noticed that UpgradedIntrinsics only gets written to in the following code: if (UpgradeIntrinsicFunction(&F, NewFn)) UpgradedIntrinsics[&F] = NewFn; Looking through UpgradeIntrinsicFunction, we always return false OR NewFn will be set to a different function from our source. This patch pulls the F != NewFn into UpgradeIntrinsicFunction as an assert, and removes the check from callers of UpgradeIntrinsicFunction. Reviewers: rafael, chandlerc Subscribers: llvm-commits-list Differential Revision: http://reviews.llvm.org/D10915 llvm-svn: 241369
2015-07-03 22:12:01 +02:00
// Remove old function, no longer used, from the module.
F->eraseFromParent();
This is the patch to provide clean intrinsic function overloading support in LLVM. It cleans up the intrinsic definitions and generally smooths the process for more complicated intrinsic writing. It will be used by the upcoming atomic intrinsics as well as vector and float intrinsics in the future. This also changes the syntax for llvm.bswap, llvm.part.set, llvm.part.select, and llvm.ct* intrinsics. They are automatically upgraded by both the LLVM ASM reader and the bitcode reader. The test cases have been updated, with special tests added to ensure the automatic upgrading is supported. llvm-svn: 40807
2007-08-04 03:51:18 +02:00
}
}
Reapply 79977. Use MDNodes to encode debug info in llvm IR. llvm-svn: 80406
2009-08-29 01:24:31 +02:00
Fix auto-upgrade of TBAA tags in Bitcode Reader If TBAA is on an intrinsic and it gets upgraded, it'll delete the call instruction that we collected in a vector. Even if we were to use WeakVH, it'll drop the TBAA and we'll hit the assert on the upgrade path. r263673 gave a shot to make sure the TBAA upgrade happens before intrinsics upgrade, but failed to account for all cases. Instead of collecting instructions in a vector, this patch makes it just upgrade the TBAA on the fly, because metadata are always already loaded at this point. Differential Revision: https://reviews.llvm.org/D24533 llvm-svn: 281549
2016-09-15 00:29:59 +02:00
MDNode *llvm::UpgradeTBAANode(MDNode &MD) {
AutoUpgrade: upgrade from scalar TBAA format to struct-path aware TBAA format. We treat TBAA tags as struct-path aware TBAA format when the first operand is a MDNode and the tag has 3 or more operands. llvm-svn: 191593
2013-09-28 02:22:27 +02:00
// Check if the tag uses struct-path aware TBAA format.
Fix auto-upgrade of TBAA tags in Bitcode Reader If TBAA is on an intrinsic and it gets upgraded, it'll delete the call instruction that we collected in a vector. Even if we were to use WeakVH, it'll drop the TBAA and we'll hit the assert on the upgrade path. r263673 gave a shot to make sure the TBAA upgrade happens before intrinsics upgrade, but failed to account for all cases. Instead of collecting instructions in a vector, this patch makes it just upgrade the TBAA on the fly, because metadata are always already loaded at this point. Differential Revision: https://reviews.llvm.org/D24533 llvm-svn: 281549
2016-09-15 00:29:59 +02:00
if (isa<MDNode>(MD.getOperand(0)) && MD.getNumOperands() >= 3)
return &MD;
AutoUpgrade: upgrade from scalar TBAA format to struct-path aware TBAA format. We treat TBAA tags as struct-path aware TBAA format when the first operand is a MDNode and the tag has 3 or more operands. llvm-svn: 191593
2013-09-28 02:22:27 +02:00
Fix auto-upgrade of TBAA tags in Bitcode Reader If TBAA is on an intrinsic and it gets upgraded, it'll delete the call instruction that we collected in a vector. Even if we were to use WeakVH, it'll drop the TBAA and we'll hit the assert on the upgrade path. r263673 gave a shot to make sure the TBAA upgrade happens before intrinsics upgrade, but failed to account for all cases. Instead of collecting instructions in a vector, this patch makes it just upgrade the TBAA on the fly, because metadata are always already loaded at this point. Differential Revision: https://reviews.llvm.org/D24533 llvm-svn: 281549
2016-09-15 00:29:59 +02:00
auto &Context = MD.getContext();
if (MD.getNumOperands() == 3) {
Metadata *Elts[] = {MD.getOperand(0), MD.getOperand(1)};
MDNode *ScalarType = MDNode::get(Context, Elts);
AutoUpgrade: upgrade from scalar TBAA format to struct-path aware TBAA format. We treat TBAA tags as struct-path aware TBAA format when the first operand is a MDNode and the tag has 3 or more operands. llvm-svn: 191593
2013-09-28 02:22:27 +02:00
// Create a MDNode <ScalarType, ScalarType, offset 0, const>
IR: Split Metadata from Value Split `Metadata` away from the `Value` class hierarchy, as part of PR21532. Assembly and bitcode changes are in the wings, but this is the bulk of the change for the IR C++ API. I have a follow-up patch prepared for `clang`. If this breaks other sub-projects, I apologize in advance :(. Help me compile it on Darwin I'll try to fix it. FWIW, the errors should be easy to fix, so it may be simpler to just fix it yourself. This breaks the build for all metadata-related code that's out-of-tree. Rest assured the transition is mechanical and the compiler should catch almost all of the problems. Here's a quick guide for updating your code: - `Metadata` is the root of a class hierarchy with three main classes: `MDNode`, `MDString`, and `ValueAsMetadata`. It is distinct from the `Value` class hierarchy. It is typeless -- i.e., instances do *not* have a `Type`. - `MDNode`'s operands are all `Metadata *` (instead of `Value *`). - `TrackingVH<MDNode>` and `WeakVH` referring to metadata can be replaced with `TrackingMDNodeRef` and `TrackingMDRef`, respectively. If you're referring solely to resolved `MDNode`s -- post graph construction -- just use `MDNode*`. - `MDNode` (and the rest of `Metadata`) have only limited support for `replaceAllUsesWith()`. As long as an `MDNode` is pointing at a forward declaration -- the result of `MDNode::getTemporary()` -- it maintains a side map of its uses and can RAUW itself. Once the forward declarations are fully resolved RAUW support is dropped on the ground. This means that uniquing collisions on changing operands cause nodes to become "distinct". (This already happened fairly commonly, whenever an operand went to null.) If you're constructing complex (non self-reference) `MDNode` cycles, you need to call `MDNode::resolveCycles()` on each node (or on a top-level node that somehow references all of the nodes). Also, don't do that. Metadata cycles (and the RAUW machinery needed to construct them) are expensive. - An `MDNode` can only refer to a `Constant` through a bridge called `ConstantAsMetadata` (one of the subclasses of `ValueAsMetadata`). As a side effect, accessing an operand of an `MDNode` that is known to be, e.g., `ConstantInt`, takes three steps: first, cast from `Metadata` to `ConstantAsMetadata`; second, extract the `Constant`; third, cast down to `ConstantInt`. The eventual goal is to introduce `MDInt`/`MDFloat`/etc. and have metadata schema owners transition away from using `Constant`s when the type isn't important (and they don't care about referring to `GlobalValue`s). In the meantime, I've added transitional API to the `mdconst` namespace that matches semantics with the old code, in order to avoid adding the error-prone three-step equivalent to every call site. If your old code was: MDNode *N = foo(); bar(isa <ConstantInt>(N->getOperand(0))); baz(cast <ConstantInt>(N->getOperand(1))); bak(cast_or_null <ConstantInt>(N->getOperand(2))); bat(dyn_cast <ConstantInt>(N->getOperand(3))); bay(dyn_cast_or_null<ConstantInt>(N->getOperand(4))); you can trivially match its semantics with: MDNode *N = foo(); bar(mdconst::hasa <ConstantInt>(N->getOperand(0))); baz(mdconst::extract <ConstantInt>(N->getOperand(1))); bak(mdconst::extract_or_null <ConstantInt>(N->getOperand(2))); bat(mdconst::dyn_extract <ConstantInt>(N->getOperand(3))); bay(mdconst::dyn_extract_or_null<ConstantInt>(N->getOperand(4))); and when you transition your metadata schema to `MDInt`: MDNode *N = foo(); bar(isa <MDInt>(N->getOperand(0))); baz(cast <MDInt>(N->getOperand(1))); bak(cast_or_null <MDInt>(N->getOperand(2))); bat(dyn_cast <MDInt>(N->getOperand(3))); bay(dyn_cast_or_null<MDInt>(N->getOperand(4))); - A `CallInst` -- specifically, intrinsic instructions -- can refer to metadata through a bridge called `MetadataAsValue`. This is a subclass of `Value` where `getType()->isMetadataTy()`. `MetadataAsValue` is the *only* class that can legally refer to a `LocalAsMetadata`, which is a bridged form of non-`Constant` values like `Argument` and `Instruction`. It can also refer to any other `Metadata` subclass. (I'll break all your testcases in a follow-up commit, when I propagate this change to assembly.) llvm-svn: 223802
2014-12-09 19:38:53 +01:00
Metadata *Elts2[] = {ScalarType, ScalarType,
Fix auto-upgrade of TBAA tags in Bitcode Reader If TBAA is on an intrinsic and it gets upgraded, it'll delete the call instruction that we collected in a vector. Even if we were to use WeakVH, it'll drop the TBAA and we'll hit the assert on the upgrade path. r263673 gave a shot to make sure the TBAA upgrade happens before intrinsics upgrade, but failed to account for all cases. Instead of collecting instructions in a vector, this patch makes it just upgrade the TBAA on the fly, because metadata are always already loaded at this point. Differential Revision: https://reviews.llvm.org/D24533 llvm-svn: 281549
2016-09-15 00:29:59 +02:00
ConstantAsMetadata::get(
Constant::getNullValue(Type::getInt64Ty(Context))),
MD.getOperand(2)};
return MDNode::get(Context, Elts2);
AutoUpgrade: upgrade from scalar TBAA format to struct-path aware TBAA format. We treat TBAA tags as struct-path aware TBAA format when the first operand is a MDNode and the tag has 3 or more operands. llvm-svn: 191593
2013-09-28 02:22:27 +02:00
}
Fix auto-upgrade of TBAA tags in Bitcode Reader If TBAA is on an intrinsic and it gets upgraded, it'll delete the call instruction that we collected in a vector. Even if we were to use WeakVH, it'll drop the TBAA and we'll hit the assert on the upgrade path. r263673 gave a shot to make sure the TBAA upgrade happens before intrinsics upgrade, but failed to account for all cases. Instead of collecting instructions in a vector, this patch makes it just upgrade the TBAA on the fly, because metadata are always already loaded at this point. Differential Revision: https://reviews.llvm.org/D24533 llvm-svn: 281549
2016-09-15 00:29:59 +02:00
// Create a MDNode <MD, MD, offset 0>
Metadata *Elts[] = {&MD, &MD, ConstantAsMetadata::get(Constant::getNullValue(
Type::getInt64Ty(Context)))};
return MDNode::get(Context, Elts);
AutoUpgrade: upgrade from scalar TBAA format to struct-path aware TBAA format. We treat TBAA tags as struct-path aware TBAA format when the first operand is a MDNode and the tag has 3 or more operands. llvm-svn: 191593
2013-09-28 02:22:27 +02:00
}
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 02:34:59 +01:00
Instruction *llvm::UpgradeBitCastInst(unsigned Opc, Value *V, Type *DestTy,
Instruction *&Temp) {
if (Opc != Instruction::BitCast)
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 08:08:46 +02:00
return nullptr;
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 02:34:59 +01:00
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 08:08:46 +02:00
Temp = nullptr;
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 02:34:59 +01:00
Type *SrcTy = V->getType();
if (SrcTy->isPtrOrPtrVectorTy() && DestTy->isPtrOrPtrVectorTy() &&
SrcTy->getPointerAddressSpace() != DestTy->getPointerAddressSpace()) {
LLVMContext &Context = V->getContext();
// We have no information about target data layout, so we assume that
// the maximum pointer size is 64bit.
Type *MidTy = Type::getInt64Ty(Context);
Temp = CastInst::Create(Instruction::PtrToInt, V, MidTy);
return CastInst::Create(Instruction::IntToPtr, Temp, DestTy);
}
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 08:08:46 +02:00
return nullptr;
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 02:34:59 +01:00
}
Value *llvm::UpgradeBitCastExpr(unsigned Opc, Constant *C, Type *DestTy) {
if (Opc != Instruction::BitCast)
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 08:08:46 +02:00
return nullptr;
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 02:34:59 +01:00
Type *SrcTy = C->getType();
if (SrcTy->isPtrOrPtrVectorTy() && DestTy->isPtrOrPtrVectorTy() &&
SrcTy->getPointerAddressSpace() != DestTy->getPointerAddressSpace()) {
LLVMContext &Context = C->getContext();
// We have no information about target data layout, so we assume that
// the maximum pointer size is 64bit.
Type *MidTy = Type::getInt64Ty(Context);
return ConstantExpr::getIntToPtr(ConstantExpr::getPtrToInt(C, MidTy),
DestTy);
}
[C++11] More 'nullptr' conversion or in some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 205831
2014-04-09 08:08:46 +02:00
return nullptr;
Add addrspacecast instruction. Patch by Michele Scandale! llvm-svn: 194760
2013-11-15 02:34:59 +01:00
}
Debug Info: drop debug info via upgrading path if version number does not match. Add a helper function getDebugInfoVersionFromModule to return the debug info version number for a module. "Verifier/module-flags-1.ll" checks for verification errors. It will seg fault when calling getDebugInfoVersionFromModule because of the incorrect format for module flags in the testing case. We make getModuleFlagsMetadata more robust by checking for error conditions. PR17982 llvm-svn: 196158
2013-12-02 22:29:56 +01:00
/// Check the debug info version number, if it is out-dated, drop the debug
/// info. Return true if module is modified.
bool llvm::UpgradeDebugInfo(Module &M) {
Report a warning when dropping outdated debug info metadata. Use DiagnosticInfo to emit the warning. llvm-svn: 199346
2014-01-16 02:51:12 +01:00
unsigned Version = getDebugMetadataVersionFromModule(M);
Move the stripping of invalid debug info from the Verifier to AutoUpgrade. This came out of a recent discussion on llvm-dev (https://reviews.llvm.org/D38042). Currently the Verifier will strip the debug info metadata from a module if it finds the dbeug info to be malformed. This feature is very valuable since it allows us to improve the Verifier by making it stricter without breaking bcompatibility, but arguable the Verifier pass should not be modifying the IR. This patch moves the stripping of broken debug info into AutoUpgrade (UpgradeDebugInfo to be precise), which is a much better location for this since the stripping of malformed (i.e., produced by older, buggy versions of Clang) is a (harsh) form of AutoUpgrade. This change is mostly NFC in nature, the one big difference is the behavior when LLVM module passes are introducing malformed debug info. Prior to this patch, a NoAsserts build would have printed a warning and stripped the debug info, after this patch the Verifier will report a fatal error. I believe this behavior is actually more desirable anyway. Differential Revision: https://reviews.llvm.org/D38184 llvm-svn: 314699
2017-10-02 20:31:29 +02:00
if (Version == DEBUG_METADATA_VERSION) {
bool BrokenDebugInfo = false;
if (verifyModule(M, &llvm::errs(), &BrokenDebugInfo))
report_fatal_error("Broken module found, compilation aborted!");
if (!BrokenDebugInfo)
// Everything is ok.
return false;
else {
// Diagnose malformed debug info.
DiagnosticInfoIgnoringInvalidDebugMetadata Diag(M);
M.getContext().diagnose(Diag);
}
}
bool Modified = StripDebugInfo(M);
if (Modified && Version != DEBUG_METADATA_VERSION) {
// Diagnose a version mismatch.
Report a warning when dropping outdated debug info metadata. Use DiagnosticInfo to emit the warning. llvm-svn: 199346
2014-01-16 02:51:12 +01:00
DiagnosticInfoDebugMetadataVersion DiagVersion(M, Version);
M.getContext().diagnose(DiagVersion);
}
Move the stripping of invalid debug info from the Verifier to AutoUpgrade. This came out of a recent discussion on llvm-dev (https://reviews.llvm.org/D38042). Currently the Verifier will strip the debug info metadata from a module if it finds the dbeug info to be malformed. This feature is very valuable since it allows us to improve the Verifier by making it stricter without breaking bcompatibility, but arguable the Verifier pass should not be modifying the IR. This patch moves the stripping of broken debug info into AutoUpgrade (UpgradeDebugInfo to be precise), which is a much better location for this since the stripping of malformed (i.e., produced by older, buggy versions of Clang) is a (harsh) form of AutoUpgrade. This change is mostly NFC in nature, the one big difference is the behavior when LLVM module passes are introducing malformed debug info. Prior to this patch, a NoAsserts build would have printed a warning and stripped the debug info, after this patch the Verifier will report a fatal error. I believe this behavior is actually more desirable anyway. Differential Revision: https://reviews.llvm.org/D38184 llvm-svn: 314699
2017-10-02 20:31:29 +02:00
return Modified;
Debug Info: drop debug info via upgrading path if version number does not match. Add a helper function getDebugInfoVersionFromModule to return the debug info version number for a module. "Verifier/module-flags-1.ll" checks for verification errors. It will seg fault when calling getDebugInfoVersionFromModule because of the incorrect format for module flags in the testing case. We make getModuleFlagsMetadata more robust by checking for error conditions. PR17982 llvm-svn: 196158
2013-12-02 22:29:56 +01:00
}
Rename loop unrolling and loop vectorizer metadata to have a common prefix. [LLVM part] These patches rename the loop unrolling and loop vectorizer metadata such that they have a common 'llvm.loop.' prefix. Metadata name changes: llvm.vectorizer.* => llvm.loop.vectorizer.* llvm.loopunroll.* => llvm.loop.unroll.* This was a suggestion from an earlier review (http://reviews.llvm.org/D4090) which added the loop unrolling metadata. Patch by Mark Heffernan. llvm-svn: 211710
2014-06-25 17:41:00 +02:00
[AutoUpgrader] Make ArcRuntime Autoupgrader more conservative Summary: This is a tweak to r368311 and r368646 which auto upgrades the calls to objc runtime functions to objc runtime intrinsics, in order to make sure that the auto upgrader does not trigger with up-to-date bitcode. It is possible for bitcode that is up-to-date to contain direct calls to objc runtime function and those are not inserted by compiler as part of ARC and they should not be upgraded. Now auto upgrader only triggers as when the old style of ARC marker is used so it is guaranteed that it won't trigger on update-to-date bitcode. This also means it won't do this upgrade for bitcode from llvm-8 and llvm-9, which preserves the behavior of those releases. Ideally they should be upgraded as well but it is more important to make sure AutoUpgrader will not trigger on up-to-date bitcode. Reviewers: ahatanak, rjmccall, dexonsmith, pete Reviewed By: dexonsmith Subscribers: hiraditya, jkorous, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D66153 llvm-svn: 368730
2019-08-13 19:52:21 +02:00
/// This checks for objc retain release marker which should be upgraded. It
/// returns true if module is modified.
static bool UpgradeRetainReleaseMarker(Module &M) {
[IR] Upgrade comment token in objc retain release marker Older compiler issued '#' instead of ';' llvm-svn: 329248
2018-04-05 04:44:46 +02:00
bool Changed = false;
[ObjC][ARC] Convert the retainRV marker that is passed as a named metadata into a module flag in the auto-upgrader and make the ARC contract pass read the marker as a module flag. This is needed to fix a bug where ARC contract wasn't inserting the retainRV marker when LTO was enabled, which caused objects returned from a function to be auto-released. rdar://problem/49464214 Differential Revision: https://reviews.llvm.org/D60303 llvm-svn: 358047
2019-04-10 08:20:20 +02:00
const char *MarkerKey = "clang.arc.retainAutoreleasedReturnValueMarker";
NamedMDNode *ModRetainReleaseMarker = M.getNamedMetadata(MarkerKey);
[IR] Upgrade comment token in objc retain release marker Older compiler issued '#' instead of ';' llvm-svn: 329248
2018-04-05 04:44:46 +02:00
if (ModRetainReleaseMarker) {
MDNode *Op = ModRetainReleaseMarker->getOperand(0);
if (Op) {
MDString *ID = dyn_cast_or_null<MDString>(Op->getOperand(0));
if (ID) {
SmallVector<StringRef, 4> ValueComp;
ID->getString().split(ValueComp, "#");
if (ValueComp.size() == 2) {
std::string NewValue = ValueComp[0].str() + ";" + ValueComp[1].str();
[ObjC][ARC] Convert the retainRV marker that is passed as a named metadata into a module flag in the auto-upgrader and make the ARC contract pass read the marker as a module flag. This is needed to fix a bug where ARC contract wasn't inserting the retainRV marker when LTO was enabled, which caused objects returned from a function to be auto-released. rdar://problem/49464214 Differential Revision: https://reviews.llvm.org/D60303 llvm-svn: 358047
2019-04-10 08:20:20 +02:00
ID = MDString::get(M.getContext(), NewValue);
[IR] Upgrade comment token in objc retain release marker Older compiler issued '#' instead of ';' llvm-svn: 329248
2018-04-05 04:44:46 +02:00
}
[ObjC][ARC] Convert the retainRV marker that is passed as a named metadata into a module flag in the auto-upgrader and make the ARC contract pass read the marker as a module flag. This is needed to fix a bug where ARC contract wasn't inserting the retainRV marker when LTO was enabled, which caused objects returned from a function to be auto-released. rdar://problem/49464214 Differential Revision: https://reviews.llvm.org/D60303 llvm-svn: 358047
2019-04-10 08:20:20 +02:00
M.addModuleFlag(Module::Error, MarkerKey, ID);
M.eraseNamedMetadata(ModRetainReleaseMarker);
Changed = true;
[IR] Upgrade comment token in objc retain release marker Older compiler issued '#' instead of ';' llvm-svn: 329248
2018-04-05 04:44:46 +02:00
}
}
}
return Changed;
}
[AutoUpgrader] Make ArcRuntime Autoupgrader more conservative Summary: This is a tweak to r368311 and r368646 which auto upgrades the calls to objc runtime functions to objc runtime intrinsics, in order to make sure that the auto upgrader does not trigger with up-to-date bitcode. It is possible for bitcode that is up-to-date to contain direct calls to objc runtime function and those are not inserted by compiler as part of ARC and they should not be upgraded. Now auto upgrader only triggers as when the old style of ARC marker is used so it is guaranteed that it won't trigger on update-to-date bitcode. This also means it won't do this upgrade for bitcode from llvm-8 and llvm-9, which preserves the behavior of those releases. Ideally they should be upgraded as well but it is more important to make sure AutoUpgrader will not trigger on up-to-date bitcode. Reviewers: ahatanak, rjmccall, dexonsmith, pete Reviewed By: dexonsmith Subscribers: hiraditya, jkorous, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D66153 llvm-svn: 368730
2019-08-13 19:52:21 +02:00
void llvm::UpgradeARCRuntime(Module &M) {
Do not call replaceAllUsesWith to upgrade calls to ARC runtime functions to intrinsic calls This fixes a bug in r368311. It turns out that the ARC runtime functions in the IR can have pointer parameter types that are not i8* or i8**. Instead of RAUWing normal functions with intrinsics, manually bitcast the arguments before passing them to the intrinsic functions and bitcast the return value back to the type of the original call instruction. This recommits r368634, which was reverted in r368637. The loop in the patch was iterating over uses of a function and deleting function calls inside it, which caused bots to crash. rdar://problem/54125406 Differential Revision: https://reviews.llvm.org/D66047 llvm-svn: 368646
2019-08-13 03:23:06 +02:00
// This lambda converts normal function calls to ARC runtime functions to
// intrinsic calls.
[ObjC][ARC] Upgrade calls to ARC runtime functions to intrinsic calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker The ARC middle-end passes stopped optimizing or transforming bitcode that has been compiled with old compilers after we started emitting calls to ARC runtime functions as intrinsic calls instead of normal function calls in the front-end and made changes to teach the ARC middle-end passes about those intrinsics (see r349534). This patch converts calls to ARC runtime functions that are not intrinsic functions to intrinsic function calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker. Checking for the presence of the marker is necessary to make sure we aren't changing ARC function calls that were originally MRR message sends (see r349952). rdar://problem/53280660 Differential Revision: https://reviews.llvm.org/D65902 llvm-svn: 368311
2019-08-08 18:59:31 +02:00
auto UpgradeToIntrinsic = [&](const char *OldFunc,
llvm::Intrinsic::ID IntrinsicFunc) {
Function *Fn = M.getFunction(OldFunc);
if (!Fn)
Change the return type of UpgradeARCRuntimeCalls to void Nothing is using the function return. llvm-svn: 368367
2019-08-09 01:33:17 +02:00
return;
[ObjC][ARC] Upgrade calls to ARC runtime functions to intrinsic calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker The ARC middle-end passes stopped optimizing or transforming bitcode that has been compiled with old compilers after we started emitting calls to ARC runtime functions as intrinsic calls instead of normal function calls in the front-end and made changes to teach the ARC middle-end passes about those intrinsics (see r349534). This patch converts calls to ARC runtime functions that are not intrinsic functions to intrinsic function calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker. Checking for the presence of the marker is necessary to make sure we aren't changing ARC function calls that were originally MRR message sends (see r349952). rdar://problem/53280660 Differential Revision: https://reviews.llvm.org/D65902 llvm-svn: 368311
2019-08-08 18:59:31 +02:00
Function *NewFn = llvm::Intrinsic::getDeclaration(&M, IntrinsicFunc);
Do not call replaceAllUsesWith to upgrade calls to ARC runtime functions to intrinsic calls This fixes a bug in r368311. It turns out that the ARC runtime functions in the IR can have pointer parameter types that are not i8* or i8**. Instead of RAUWing normal functions with intrinsics, manually bitcast the arguments before passing them to the intrinsic functions and bitcast the return value back to the type of the original call instruction. This recommits r368634, which was reverted in r368637. The loop in the patch was iterating over uses of a function and deleting function calls inside it, which caused bots to crash. rdar://problem/54125406 Differential Revision: https://reviews.llvm.org/D66047 llvm-svn: 368646
2019-08-13 03:23:06 +02:00
for (auto I = Fn->user_begin(), E = Fn->user_end(); I != E;) {
CallInst *CI = dyn_cast<CallInst>(*I++);
if (!CI || CI->getCalledFunction() != Fn)
continue;
IRBuilder<> Builder(CI->getParent(), CI->getIterator());
FunctionType *NewFuncTy = NewFn->getFunctionType();
SmallVector<Value *, 2> Args;
[ObjC][ARC] Check whether the return and parameter types of the old and new functions are compatible before upgrading a function call to an intrinsic call. Sometimes users insert calls to ARC runtime functions that are not compatible with the corresponding intrinsic functions (for example, 'i8* @objc_storeStrong' instead of 'void @objc_storeStrong'). Don't upgrade those calls. rdar://problem/56447127
2019-10-23 16:38:52 +02:00
// Don't upgrade the intrinsic if it's not valid to bitcast the return
// value to the return type of the old function.
if (NewFuncTy->getReturnType() != CI->getType() &&
!CastInst::castIsValid(Instruction::BitCast, CI,
NewFuncTy->getReturnType()))
continue;
bool InvalidCast = false;
Do not call replaceAllUsesWith to upgrade calls to ARC runtime functions to intrinsic calls This fixes a bug in r368311. It turns out that the ARC runtime functions in the IR can have pointer parameter types that are not i8* or i8**. Instead of RAUWing normal functions with intrinsics, manually bitcast the arguments before passing them to the intrinsic functions and bitcast the return value back to the type of the original call instruction. This recommits r368634, which was reverted in r368637. The loop in the patch was iterating over uses of a function and deleting function calls inside it, which caused bots to crash. rdar://problem/54125406 Differential Revision: https://reviews.llvm.org/D66047 llvm-svn: 368646
2019-08-13 03:23:06 +02:00
for (unsigned I = 0, E = CI->getNumArgOperands(); I != E; ++I) {
Value *Arg = CI->getArgOperand(I);
[ObjC][ARC] Check whether the return and parameter types of the old and new functions are compatible before upgrading a function call to an intrinsic call. Sometimes users insert calls to ARC runtime functions that are not compatible with the corresponding intrinsic functions (for example, 'i8* @objc_storeStrong' instead of 'void @objc_storeStrong'). Don't upgrade those calls. rdar://problem/56447127
2019-10-23 16:38:52 +02:00
Do not call replaceAllUsesWith to upgrade calls to ARC runtime functions to intrinsic calls This fixes a bug in r368311. It turns out that the ARC runtime functions in the IR can have pointer parameter types that are not i8* or i8**. Instead of RAUWing normal functions with intrinsics, manually bitcast the arguments before passing them to the intrinsic functions and bitcast the return value back to the type of the original call instruction. This recommits r368634, which was reverted in r368637. The loop in the patch was iterating over uses of a function and deleting function calls inside it, which caused bots to crash. rdar://problem/54125406 Differential Revision: https://reviews.llvm.org/D66047 llvm-svn: 368646
2019-08-13 03:23:06 +02:00
// Bitcast argument to the parameter type of the new function if it's
// not a variadic argument.
[ObjC][ARC] Check whether the return and parameter types of the old and new functions are compatible before upgrading a function call to an intrinsic call. Sometimes users insert calls to ARC runtime functions that are not compatible with the corresponding intrinsic functions (for example, 'i8* @objc_storeStrong' instead of 'void @objc_storeStrong'). Don't upgrade those calls. rdar://problem/56447127
2019-10-23 16:38:52 +02:00
if (I < NewFuncTy->getNumParams()) {
// Don't upgrade the intrinsic if it's not valid to bitcast the argument
// to the parameter type of the new function.
if (!CastInst::castIsValid(Instruction::BitCast, Arg,
NewFuncTy->getParamType(I))) {
InvalidCast = true;
break;
}
Do not call replaceAllUsesWith to upgrade calls to ARC runtime functions to intrinsic calls This fixes a bug in r368311. It turns out that the ARC runtime functions in the IR can have pointer parameter types that are not i8* or i8**. Instead of RAUWing normal functions with intrinsics, manually bitcast the arguments before passing them to the intrinsic functions and bitcast the return value back to the type of the original call instruction. This recommits r368634, which was reverted in r368637. The loop in the patch was iterating over uses of a function and deleting function calls inside it, which caused bots to crash. rdar://problem/54125406 Differential Revision: https://reviews.llvm.org/D66047 llvm-svn: 368646
2019-08-13 03:23:06 +02:00
Arg = Builder.CreateBitCast(Arg, NewFuncTy->getParamType(I));
[ObjC][ARC] Check whether the return and parameter types of the old and new functions are compatible before upgrading a function call to an intrinsic call. Sometimes users insert calls to ARC runtime functions that are not compatible with the corresponding intrinsic functions (for example, 'i8* @objc_storeStrong' instead of 'void @objc_storeStrong'). Don't upgrade those calls. rdar://problem/56447127
2019-10-23 16:38:52 +02:00
}
Do not call replaceAllUsesWith to upgrade calls to ARC runtime functions to intrinsic calls This fixes a bug in r368311. It turns out that the ARC runtime functions in the IR can have pointer parameter types that are not i8* or i8**. Instead of RAUWing normal functions with intrinsics, manually bitcast the arguments before passing them to the intrinsic functions and bitcast the return value back to the type of the original call instruction. This recommits r368634, which was reverted in r368637. The loop in the patch was iterating over uses of a function and deleting function calls inside it, which caused bots to crash. rdar://problem/54125406 Differential Revision: https://reviews.llvm.org/D66047 llvm-svn: 368646
2019-08-13 03:23:06 +02:00
Args.push_back(Arg);
}
[ObjC][ARC] Check whether the return and parameter types of the old and new functions are compatible before upgrading a function call to an intrinsic call. Sometimes users insert calls to ARC runtime functions that are not compatible with the corresponding intrinsic functions (for example, 'i8* @objc_storeStrong' instead of 'void @objc_storeStrong'). Don't upgrade those calls. rdar://problem/56447127
2019-10-23 16:38:52 +02:00
if (InvalidCast)
continue;
Do not call replaceAllUsesWith to upgrade calls to ARC runtime functions to intrinsic calls This fixes a bug in r368311. It turns out that the ARC runtime functions in the IR can have pointer parameter types that are not i8* or i8**. Instead of RAUWing normal functions with intrinsics, manually bitcast the arguments before passing them to the intrinsic functions and bitcast the return value back to the type of the original call instruction. This recommits r368634, which was reverted in r368637. The loop in the patch was iterating over uses of a function and deleting function calls inside it, which caused bots to crash. rdar://problem/54125406 Differential Revision: https://reviews.llvm.org/D66047 llvm-svn: 368646
2019-08-13 03:23:06 +02:00
// Create a call instruction that calls the new function.
CallInst *NewCall = Builder.CreateCall(NewFuncTy, NewFn, Args);
NewCall->setTailCallKind(cast<CallInst>(CI)->getTailCallKind());
NewCall->setName(CI->getName());
// Bitcast the return value back to the type of the old call.
Value *NewRetVal = Builder.CreateBitCast(NewCall, CI->getType());
if (!CI->use_empty())
CI->replaceAllUsesWith(NewRetVal);
CI->eraseFromParent();
}
if (Fn->use_empty())
Fn->eraseFromParent();
[ObjC][ARC] Upgrade calls to ARC runtime functions to intrinsic calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker The ARC middle-end passes stopped optimizing or transforming bitcode that has been compiled with old compilers after we started emitting calls to ARC runtime functions as intrinsic calls instead of normal function calls in the front-end and made changes to teach the ARC middle-end passes about those intrinsics (see r349534). This patch converts calls to ARC runtime functions that are not intrinsic functions to intrinsic function calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker. Checking for the presence of the marker is necessary to make sure we aren't changing ARC function calls that were originally MRR message sends (see r349952). rdar://problem/53280660 Differential Revision: https://reviews.llvm.org/D65902 llvm-svn: 368311
2019-08-08 18:59:31 +02:00
};
Do not call replaceAllUsesWith to upgrade calls to ARC runtime functions to intrinsic calls This fixes a bug in r368311. It turns out that the ARC runtime functions in the IR can have pointer parameter types that are not i8* or i8**. Instead of RAUWing normal functions with intrinsics, manually bitcast the arguments before passing them to the intrinsic functions and bitcast the return value back to the type of the original call instruction. This recommits r368634, which was reverted in r368637. The loop in the patch was iterating over uses of a function and deleting function calls inside it, which caused bots to crash. rdar://problem/54125406 Differential Revision: https://reviews.llvm.org/D66047 llvm-svn: 368646
2019-08-13 03:23:06 +02:00
// Unconditionally convert a call to "clang.arc.use" to a call to
// "llvm.objc.clang.arc.use".
Change the return type of UpgradeARCRuntimeCalls to void Nothing is using the function return. llvm-svn: 368367
2019-08-09 01:33:17 +02:00
UpgradeToIntrinsic("clang.arc.use", llvm::Intrinsic::objc_clang_arc_use);
[ObjC][ARC] Upgrade calls to ARC runtime functions to intrinsic calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker The ARC middle-end passes stopped optimizing or transforming bitcode that has been compiled with old compilers after we started emitting calls to ARC runtime functions as intrinsic calls instead of normal function calls in the front-end and made changes to teach the ARC middle-end passes about those intrinsics (see r349534). This patch converts calls to ARC runtime functions that are not intrinsic functions to intrinsic function calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker. Checking for the presence of the marker is necessary to make sure we aren't changing ARC function calls that were originally MRR message sends (see r349952). rdar://problem/53280660 Differential Revision: https://reviews.llvm.org/D65902 llvm-svn: 368311
2019-08-08 18:59:31 +02:00
[AutoUpgrader] Make ArcRuntime Autoupgrader more conservative Summary: This is a tweak to r368311 and r368646 which auto upgrades the calls to objc runtime functions to objc runtime intrinsics, in order to make sure that the auto upgrader does not trigger with up-to-date bitcode. It is possible for bitcode that is up-to-date to contain direct calls to objc runtime function and those are not inserted by compiler as part of ARC and they should not be upgraded. Now auto upgrader only triggers as when the old style of ARC marker is used so it is guaranteed that it won't trigger on update-to-date bitcode. This also means it won't do this upgrade for bitcode from llvm-8 and llvm-9, which preserves the behavior of those releases. Ideally they should be upgraded as well but it is more important to make sure AutoUpgrader will not trigger on up-to-date bitcode. Reviewers: ahatanak, rjmccall, dexonsmith, pete Reviewed By: dexonsmith Subscribers: hiraditya, jkorous, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D66153 llvm-svn: 368730
2019-08-13 19:52:21 +02:00
// Upgrade the retain release marker. If there is no need to upgrade
// the marker, that means either the module is already new enough to contain
// new intrinsics or it is not ARC. There is no need to upgrade runtime call.
if (!UpgradeRetainReleaseMarker(M))
Change the return type of UpgradeARCRuntimeCalls to void Nothing is using the function return. llvm-svn: 368367
2019-08-09 01:33:17 +02:00
return;
[ObjC][ARC] Upgrade calls to ARC runtime functions to intrinsic calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker The ARC middle-end passes stopped optimizing or transforming bitcode that has been compiled with old compilers after we started emitting calls to ARC runtime functions as intrinsic calls instead of normal function calls in the front-end and made changes to teach the ARC middle-end passes about those intrinsics (see r349534). This patch converts calls to ARC runtime functions that are not intrinsic functions to intrinsic function calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker. Checking for the presence of the marker is necessary to make sure we aren't changing ARC function calls that were originally MRR message sends (see r349952). rdar://problem/53280660 Differential Revision: https://reviews.llvm.org/D65902 llvm-svn: 368311
2019-08-08 18:59:31 +02:00
std::pair<const char *, llvm::Intrinsic::ID> RuntimeFuncs[] = {
{"objc_autorelease", llvm::Intrinsic::objc_autorelease},
{"objc_autoreleasePoolPop", llvm::Intrinsic::objc_autoreleasePoolPop},
{"objc_autoreleasePoolPush", llvm::Intrinsic::objc_autoreleasePoolPush},
{"objc_autoreleaseReturnValue",
llvm::Intrinsic::objc_autoreleaseReturnValue},
{"objc_copyWeak", llvm::Intrinsic::objc_copyWeak},
{"objc_destroyWeak", llvm::Intrinsic::objc_destroyWeak},
{"objc_initWeak", llvm::Intrinsic::objc_initWeak},
{"objc_loadWeak", llvm::Intrinsic::objc_loadWeak},
{"objc_loadWeakRetained", llvm::Intrinsic::objc_loadWeakRetained},
{"objc_moveWeak", llvm::Intrinsic::objc_moveWeak},
{"objc_release", llvm::Intrinsic::objc_release},
{"objc_retain", llvm::Intrinsic::objc_retain},
{"objc_retainAutorelease", llvm::Intrinsic::objc_retainAutorelease},
{"objc_retainAutoreleaseReturnValue",
llvm::Intrinsic::objc_retainAutoreleaseReturnValue},
{"objc_retainAutoreleasedReturnValue",
llvm::Intrinsic::objc_retainAutoreleasedReturnValue},
{"objc_retainBlock", llvm::Intrinsic::objc_retainBlock},
{"objc_storeStrong", llvm::Intrinsic::objc_storeStrong},
{"objc_storeWeak", llvm::Intrinsic::objc_storeWeak},
{"objc_unsafeClaimAutoreleasedReturnValue",
llvm::Intrinsic::objc_unsafeClaimAutoreleasedReturnValue},
{"objc_retainedObject", llvm::Intrinsic::objc_retainedObject},
{"objc_unretainedObject", llvm::Intrinsic::objc_unretainedObject},
{"objc_unretainedPointer", llvm::Intrinsic::objc_unretainedPointer},
{"objc_retain_autorelease", llvm::Intrinsic::objc_retain_autorelease},
{"objc_sync_enter", llvm::Intrinsic::objc_sync_enter},
{"objc_sync_exit", llvm::Intrinsic::objc_sync_exit},
{"objc_arc_annotation_topdown_bbstart",
llvm::Intrinsic::objc_arc_annotation_topdown_bbstart},
{"objc_arc_annotation_topdown_bbend",
llvm::Intrinsic::objc_arc_annotation_topdown_bbend},
{"objc_arc_annotation_bottomup_bbstart",
llvm::Intrinsic::objc_arc_annotation_bottomup_bbstart},
{"objc_arc_annotation_bottomup_bbend",
llvm::Intrinsic::objc_arc_annotation_bottomup_bbend}};
for (auto &I : RuntimeFuncs)
Change the return type of UpgradeARCRuntimeCalls to void Nothing is using the function return. llvm-svn: 368367
2019-08-09 01:33:17 +02:00
UpgradeToIntrinsic(I.first, I.second);
[ObjC][ARC] Upgrade calls to ARC runtime functions to intrinsic calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker The ARC middle-end passes stopped optimizing or transforming bitcode that has been compiled with old compilers after we started emitting calls to ARC runtime functions as intrinsic calls instead of normal function calls in the front-end and made changes to teach the ARC middle-end passes about those intrinsics (see r349534). This patch converts calls to ARC runtime functions that are not intrinsic functions to intrinsic function calls if the bitcode has the arm64 retainAutoreleasedReturnValue marker. Checking for the presence of the marker is necessary to make sure we aren't changing ARC function calls that were originally MRR message sends (see r349952). rdar://problem/53280660 Differential Revision: https://reviews.llvm.org/D65902 llvm-svn: 368311
2019-08-08 18:59:31 +02:00
}
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 01:14:48 +02:00
bool llvm::UpgradeModuleFlags(Module &M) {
[IR] AutoUpgrade ModuleFlagBehavior for PIC and PIE level Summary: From r303590, ModuleFlagBehavior for PIC and PIE level is changed from Error to Max. This will cause bitcode compatibility issue when linking against a bitcode static archive built with old compiler. Add an auto-ugprade path to upgrade the the ModuleFlagBehavior in the old bitcode to match the new one so IRLinker can link them. Reviewers: tejohnson, mehdi_amini, dexonsmith Reviewed By: dexonsmith Subscribers: hans, llvm-commits Differential Revision: https://reviews.llvm.org/D36556 llvm-svn: 311387
2017-08-21 23:49:13 +02:00
NamedMDNode *ModFlags = M.getModuleFlagsMetadata();
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 01:14:48 +02:00
if (!ModFlags)
return false;
[IR] AutoUpgrade ModuleFlagBehavior for PIC and PIE level Summary: From r303590, ModuleFlagBehavior for PIC and PIE level is changed from Error to Max. This will cause bitcode compatibility issue when linking against a bitcode static archive built with old compiler. Add an auto-ugprade path to upgrade the the ModuleFlagBehavior in the old bitcode to match the new one so IRLinker can link them. Reviewers: tejohnson, mehdi_amini, dexonsmith Reviewed By: dexonsmith Subscribers: hans, llvm-commits Differential Revision: https://reviews.llvm.org/D36556 llvm-svn: 311387
2017-08-21 23:49:13 +02:00
bool HasObjCFlag = false, HasClassProperties = false, Changed = false;
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 01:14:48 +02:00
for (unsigned I = 0, E = ModFlags->getNumOperands(); I != E; ++I) {
MDNode *Op = ModFlags->getOperand(I);
[IR] AutoUpgrade ModuleFlagBehavior for PIC and PIE level Summary: From r303590, ModuleFlagBehavior for PIC and PIE level is changed from Error to Max. This will cause bitcode compatibility issue when linking against a bitcode static archive built with old compiler. Add an auto-ugprade path to upgrade the the ModuleFlagBehavior in the old bitcode to match the new one so IRLinker can link them. Reviewers: tejohnson, mehdi_amini, dexonsmith Reviewed By: dexonsmith Subscribers: hans, llvm-commits Differential Revision: https://reviews.llvm.org/D36556 llvm-svn: 311387
2017-08-21 23:49:13 +02:00
if (Op->getNumOperands() != 3)
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 01:14:48 +02:00
continue;
MDString *ID = dyn_cast_or_null<MDString>(Op->getOperand(1));
if (!ID)
continue;
if (ID->getString() == "Objective-C Image Info Version")
HasObjCFlag = true;
if (ID->getString() == "Objective-C Class Properties")
HasClassProperties = true;
[IR] AutoUpgrade ModuleFlagBehavior for PIC and PIE level Summary: From r303590, ModuleFlagBehavior for PIC and PIE level is changed from Error to Max. This will cause bitcode compatibility issue when linking against a bitcode static archive built with old compiler. Add an auto-ugprade path to upgrade the the ModuleFlagBehavior in the old bitcode to match the new one so IRLinker can link them. Reviewers: tejohnson, mehdi_amini, dexonsmith Reviewed By: dexonsmith Subscribers: hans, llvm-commits Differential Revision: https://reviews.llvm.org/D36556 llvm-svn: 311387
2017-08-21 23:49:13 +02:00
// Upgrade PIC/PIE Module Flags. The module flag behavior for these two
// field was Error and now they are Max.
if (ID->getString() == "PIC Level" || ID->getString() == "PIE Level") {
if (auto *Behavior =
mdconst::dyn_extract_or_null<ConstantInt>(Op->getOperand(0))) {
if (Behavior->getLimitedValue() == Module::Error) {
Type *Int32Ty = Type::getInt32Ty(M.getContext());
Metadata *Ops[3] = {
ConstantAsMetadata::get(ConstantInt::get(Int32Ty, Module::Max)),
MDString::get(M.getContext(), ID->getString()),
Op->getOperand(2)};
ModFlags->setOperand(I, MDNode::get(M.getContext(), Ops));
Changed = true;
}
}
}
[AutoUpgrade] Fix a compatibility issue with module flag Summary: After r304661, module flag to record objective-c image info section is encoded without whitespaces after comma. The new name is equivalent to the old one, except that when LTO a module built by old compiler and a module built by a new compiler, it will fail with conflicting values. Fix the issue by removing whitespaces in bitcode upgrade path. rdar://problem/34416934 Reviewers: compnerd Reviewed By: compnerd Subscribers: mehdi_amini, hans, llvm-commits Differential Revision: https://reviews.llvm.org/D37909 llvm-svn: 313398
2017-09-15 23:12:14 +02:00
// Upgrade Objective-C Image Info Section. Removed the whitespce in the
// section name so that llvm-lto will not complain about mismatching
// module flags that is functionally the same.
if (ID->getString() == "Objective-C Image Info Section") {
if (auto *Value = dyn_cast_or_null<MDString>(Op->getOperand(2))) {
SmallVector<StringRef, 4> ValueComp;
Value->getString().split(ValueComp, " ");
if (ValueComp.size() != 1) {
std::string NewValue;
for (auto &S : ValueComp)
NewValue += S.str();
Metadata *Ops[3] = {Op->getOperand(0), Op->getOperand(1),
MDString::get(M.getContext(), NewValue)};
ModFlags->setOperand(I, MDNode::get(M.getContext(), Ops));
Changed = true;
}
}
}
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 01:14:48 +02:00
}
[IR] AutoUpgrade ModuleFlagBehavior for PIC and PIE level Summary: From r303590, ModuleFlagBehavior for PIC and PIE level is changed from Error to Max. This will cause bitcode compatibility issue when linking against a bitcode static archive built with old compiler. Add an auto-ugprade path to upgrade the the ModuleFlagBehavior in the old bitcode to match the new one so IRLinker can link them. Reviewers: tejohnson, mehdi_amini, dexonsmith Reviewed By: dexonsmith Subscribers: hans, llvm-commits Differential Revision: https://reviews.llvm.org/D36556 llvm-svn: 311387
2017-08-21 23:49:13 +02:00
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 01:14:48 +02:00
// "Objective-C Class Properties" is recently added for Objective-C. We
// upgrade ObjC bitcodes to contain a "Objective-C Class Properties" module
Fix autoupgrade logic for Objective-C class properties module flag Previous we were issuing an error when linking a module containing the new Objective-C metadata structure for class properties with an "old" one. Now instead we downgrade the module flag so that the Objective-C runtime does not expect the new metadata structure. This is consistent with what ld64 is doing on binary files. Differential Revision: https://reviews.llvm.org/D24620 llvm-svn: 281685
2016-09-16 02:38:18 +02:00
// flag of value 0, so we can correclty downgrade this flag when trying to
// link an ObjC bitcode without this module flag with an ObjC bitcode with
// this module flag.
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 01:14:48 +02:00
if (HasObjCFlag && !HasClassProperties) {
Fix autoupgrade logic for Objective-C class properties module flag Previous we were issuing an error when linking a module containing the new Objective-C metadata structure for class properties with an "old" one. Now instead we downgrade the module flag so that the Objective-C runtime does not expect the new metadata structure. This is consistent with what ld64 is doing on binary files. Differential Revision: https://reviews.llvm.org/D24620 llvm-svn: 281685
2016-09-16 02:38:18 +02:00
M.addModuleFlag(llvm::Module::Override, "Objective-C Class Properties",
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 01:14:48 +02:00
(uint32_t)0);
[IR] AutoUpgrade ModuleFlagBehavior for PIC and PIE level Summary: From r303590, ModuleFlagBehavior for PIC and PIE level is changed from Error to Max. This will cause bitcode compatibility issue when linking against a bitcode static archive built with old compiler. Add an auto-ugprade path to upgrade the the ModuleFlagBehavior in the old bitcode to match the new one so IRLinker can link them. Reviewers: tejohnson, mehdi_amini, dexonsmith Reviewed By: dexonsmith Subscribers: hans, llvm-commits Differential Revision: https://reviews.llvm.org/D36556 llvm-svn: 311387
2017-08-21 23:49:13 +02:00
Changed = true;
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 01:14:48 +02:00
}
[IR] AutoUpgrade ModuleFlagBehavior for PIC and PIE level Summary: From r303590, ModuleFlagBehavior for PIC and PIE level is changed from Error to Max. This will cause bitcode compatibility issue when linking against a bitcode static archive built with old compiler. Add an auto-ugprade path to upgrade the the ModuleFlagBehavior in the old bitcode to match the new one so IRLinker can link them. Reviewers: tejohnson, mehdi_amini, dexonsmith Reviewed By: dexonsmith Subscribers: hans, llvm-commits Differential Revision: https://reviews.llvm.org/D36556 llvm-svn: 311387
2017-08-21 23:49:13 +02:00
return Changed;
Objective-C Class Properties: Autoupgrade "Class Properties" module flag. When we have "Image Info Version" module flag but don't have "Class Properties" module flag, set "Class Properties" module flag to 0, so we can correctly emit errors when one module has the flag set and another module does not. rdar://26469641 llvm-svn: 270791
2016-05-26 01:14:48 +02:00
}
Bitcode: add an auto-upgrade for LTO section name The bitcode reader looks specifically for `__DATA, __objc_catlist` as a section name. However, SVN r304661 removed the spaces (the two names are functionally equivalent but do not compare equally lexicographically). This causes compatibility issues. Add an auto-upgrade path for removing the spaces as well as use the new name in the LTO plugin. llvm-svn: 315086
2017-10-06 20:06:59 +02:00
void llvm::UpgradeSectionAttributes(Module &M) {
auto TrimSpaces = [](StringRef Section) -> std::string {
SmallVector<StringRef, 5> Components;
Section.split(Components, ',');
SmallString<32> Buffer;
raw_svector_ostream OS(Buffer);
for (auto Component : Components)
OS << ',' << Component.trim();
return OS.str().substr(1);
};
for (auto &GV : M.globals()) {
if (!GV.hasSection())
continue;
StringRef Section = GV.getSection();
if (!Section.startswith("__DATA, __objc_catlist"))
continue;
// __DATA, __objc_catlist, regular, no_dead_strip
// __DATA,__objc_catlist,regular,no_dead_strip
GV.setSection(TrimSpaces(Section));
}
}
IR: Stop upgrading !llvm.loop attachments via MDString Remove logic to upgrade !llvm.loop by changing the MDString tag directly. This old logic would check (and change) arbitrary strings that had nothing to do with loop metadata. Instead, check !llvm.loop attachments directly, and change which strings get attached. Rather than updating the assembly-based upgrade, drop it entirely. It has been quite a while since we supported upgrading textual IR. llvm-svn: 264373
2016-03-25 01:56:13 +01:00
static bool isOldLoopArgument(Metadata *MD) {
auto *T = dyn_cast_or_null<MDTuple>(MD);
if (!T)
return false;
if (T->getNumOperands() < 1)
return false;
auto *S = dyn_cast_or_null<MDString>(T->getOperand(0));
if (!S)
return false;
return S->getString().startswith("llvm.vectorizer.");
}
static MDString *upgradeLoopTag(LLVMContext &C, StringRef OldTag) {
StringRef OldPrefix = "llvm.vectorizer.";
assert(OldTag.startswith(OldPrefix) && "Expected old prefix");
if (OldTag == "llvm.vectorizer.unroll")
return MDString::get(C, "llvm.loop.interleave.count");
return MDString::get(
C, (Twine("llvm.loop.vectorize.") + OldTag.drop_front(OldPrefix.size()))
.str());
}
static Metadata *upgradeLoopArgument(Metadata *MD) {
auto *T = dyn_cast_or_null<MDTuple>(MD);
if (!T)
return MD;
if (T->getNumOperands() < 1)
return MD;
auto *OldTag = dyn_cast_or_null<MDString>(T->getOperand(0));
if (!OldTag)
return MD;
if (!OldTag->getString().startswith("llvm.vectorizer."))
return MD;
// This has an old tag. Upgrade it.
SmallVector<Metadata *, 8> Ops;
Ops.reserve(T->getNumOperands());
Ops.push_back(upgradeLoopTag(T->getContext(), OldTag->getString()));
for (unsigned I = 1, E = T->getNumOperands(); I != E; ++I)
Ops.push_back(T->getOperand(I));
return MDTuple::get(T->getContext(), Ops);
}
MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
auto *T = dyn_cast<MDTuple>(&N);
if (!T)
return &N;
Use range algorithms instead of unpacking begin/end No functionality change is intended. llvm-svn: 278417
2016-08-11 23:15:00 +02:00
if (none_of(T->operands(), isOldLoopArgument))
IR: Stop upgrading !llvm.loop attachments via MDString Remove logic to upgrade !llvm.loop by changing the MDString tag directly. This old logic would check (and change) arbitrary strings that had nothing to do with loop metadata. Instead, check !llvm.loop attachments directly, and change which strings get attached. Rather than updating the assembly-based upgrade, drop it entirely. It has been quite a while since we supported upgrading textual IR. llvm-svn: 264373
2016-03-25 01:56:13 +01:00
return &N;
SmallVector<Metadata *, 8> Ops;
Ops.reserve(T->getNumOperands());
for (Metadata *MD : T->operands())
Ops.push_back(upgradeLoopArgument(MD));
return MDTuple::get(T->getContext(), Ops);
Rename loop unrolling and loop vectorizer metadata to have a common prefix. [LLVM part] These patches rename the loop unrolling and loop vectorizer metadata such that they have a common 'llvm.loop.' prefix. Metadata name changes: llvm.vectorizer.* => llvm.loop.vectorizer.* llvm.loopunroll.* => llvm.loop.unroll.* This was a suggestion from an earlier review (http://reviews.llvm.org/D4090) which added the loop unrolling metadata. Patch by Mark Heffernan. llvm-svn: 211710
2014-06-25 17:41:00 +02:00
}
Add AutoUpgrade function to add new address space datalayout string to existing datalayouts. Summary: Add function to AutoUpgrade to change the datalayout of old X86 datalayout strings. This adds "-p270:32:32-p271:32:32-p272:64:64" to X86 datalayouts that are otherwise valid and don't already contain it. This also removes the compatibility changes in https://reviews.llvm.org/D66843. Datalayout change in https://reviews.llvm.org/D64931. Reviewers: rnk, echristo Subscribers: hiraditya, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D67631 llvm-svn: 372267
2019-09-19 00:15:58 +02:00
std::string llvm::UpgradeDataLayoutString(StringRef DL, StringRef TT) {
std::string AddrSpaces = "-p270:32:32-p271:32:32-p272:64:64";
// If X86, and the datalayout matches the expected format, add pointer size
// address spaces to the datalayout.
Add Triple::isX86() Reviewed By: craig.topper, skan Differential Revision: https://reviews.llvm.org/D72247
2020-01-06 19:16:28 +01:00
if (!Triple(TT).isX86() || DL.contains(AddrSpaces))
Add AutoUpgrade function to add new address space datalayout string to existing datalayouts. Summary: Add function to AutoUpgrade to change the datalayout of old X86 datalayout strings. This adds "-p270:32:32-p271:32:32-p272:64:64" to X86 datalayouts that are otherwise valid and don't already contain it. This also removes the compatibility changes in https://reviews.llvm.org/D66843. Datalayout change in https://reviews.llvm.org/D64931. Reviewers: rnk, echristo Subscribers: hiraditya, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D67631 llvm-svn: 372267
2019-09-19 00:15:58 +02:00
return DL;
SmallVector<StringRef, 4> Groups;
Regex R("(e-m:[a-z](-p:32:32)?)(-[if]64:.*$)");
if (!R.match(DL, &Groups))
return DL;
SmallString<1024> Buf;
std::string Res = (Groups[1] + AddrSpaces + Groups[3]).toStringRef(Buf).str();
return Res;
}
Ignore "no-frame-pointer-elim" and "no-frame-pointer-elim-non-leaf" in favor of "frame-pointer" D56351 (included in LLVM 8.0.0) introduced "frame-pointer". All tests which use "no-frame-pointer-elim" or "no-frame-pointer-elim-non-leaf" have been migrated to use "frame-pointer". Implement UpgradeFramePointerAttributes to upgrade the two obsoleted function attributes for bitcode. Their semantics are ignored. Differential Revision: https://reviews.llvm.org/D71863
2019-12-25 03:12:15 +01:00
void llvm::UpgradeFramePointerAttributes(AttrBuilder &B) {
StringRef FramePointer;
if (B.contains("no-frame-pointer-elim")) {
// The value can be "true" or "false".
for (const auto &I : B.td_attrs())
if (I.first == "no-frame-pointer-elim")
FramePointer = I.second == "true" ? "all" : "none";
B.removeAttribute("no-frame-pointer-elim");
}
if (B.contains("no-frame-pointer-elim-non-leaf")) {
// The value is ignored. "no-frame-pointer-elim"="true" takes priority.
if (FramePointer != "all")
FramePointer = "non-leaf";
B.removeAttribute("no-frame-pointer-elim-non-leaf");
}
if (!FramePointer.empty())
B.addAttribute("frame-pointer", FramePointer);
}
Reference in New Issue Copy Permalink