RPCS3/llvm-mirror
1
0
Fork 0
mirror of https://github.com/RPCS3/llvm-mirror.git synced 2024-10-18 18:42:46 +02:00
Code Issues Projects Releases Wiki Activity

Extend coroutines to support a "returned continuation" lowering.

Browse Source 
A quick contrast of this ABI with the currently-implemented ABI:

- Allocation is implicitly managed by the lowering passes, which is fine
  for frontends that are fine with assuming that allocation cannot fail.
  This assumption is necessary to implement dynamic allocas anyway.

- The lowering attempts to fit the coroutine frame into an opaque,
  statically-sized buffer before falling back on allocation; the same
  buffer must be provided to every resume point.  A buffer must be at
  least pointer-sized.

- The resume and destroy functions have been combined; the continuation
  function takes a parameter indicating whether it has succeeded.

- Conversely, every suspend point begins its own continuation function.

- The continuation function pointer is directly returned to the caller
  instead of being stored in the frame.  The continuation can therefore
  directly destroy the frame when exiting the coroutine instead of having
  to leave it in a defunct state.

- Other values can be returned directly to the caller instead of going
  through a promise allocation.  The frontend provides a "prototype"
  function declaration from which the type, calling convention, and
  attributes of the continuation functions are taken.

- On the caller side, the frontend can generate natural IR that directly
  uses the continuation functions as long as it prevents IPO with the
  coroutine until lowering has happened.  In combination with the point
  above, the frontend is almost totally in charge of the ABI of the
  coroutine.

- Unique-yield coroutines are given some special treatment.

llvm-svn: 368788
This commit is contained in:
John McCall 2019-08-14 03:53:17 +00:00
parent 5944e43c94
commit 3f8b48091d
14 changed files with 2101 additions and 303 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

303
docs/Coroutines.rst
View File

@ -41,32 +41,144 @@ then destroy it:
In addition to the function stack frame which exists when a coroutine is
executing, there is an additional region of storage that contains objects that
keep the coroutine state when a coroutine is suspended. This region of storage
is called **coroutine frame**. It is created when a coroutine is called and
destroyed when a coroutine runs to completion or destroyed by a call to
the `coro.destroy`_ intrinsic.
is called the **coroutine frame**. It is created when a coroutine is called
and destroyed when a coroutine either runs to completion or is destroyed
while suspended.
An LLVM coroutine is represented as an LLVM function that has calls to
`coroutine intrinsics`_ defining the structure of the coroutine.
After lowering, a coroutine is split into several
functions that represent three different ways of how control can enter the
coroutine:
LLVM currently supports two styles of coroutine lowering. These styles
support substantially different sets of features, have substantially
different ABIs, and expect substantially different patterns of frontend
code generation. However, the styles also have a great deal in common.
1. a ramp function, which represents an initial invocation of the coroutine that
creates the coroutine frame and executes the coroutine code until it
encounters a suspend point or reaches the end of the function;
In all cases, an LLVM coroutine is initially represented as an ordinary LLVM
function that has calls to `coroutine intrinsics`_ defining the structure of
the coroutine. The coroutine function is then, in the most general case,
rewritten by the coroutine lowering passes to become the "ramp function",
the initial entrypoint of the coroutine, which executes until a suspend point
is first reached. The remainder of the original coroutine function is split
out into some number of "resume functions". Any state which must persist
across suspensions is stored in the coroutine frame. The resume functions
must somehow be able to handle either a "normal" resumption, which continues
the normal execution of the coroutine, or an "abnormal" resumption, which
must unwind the coroutine without attempting to suspend it.
2. a coroutine resume function that is invoked when the coroutine is resumed;
Switched-Resume Lowering
------------------------
3. a coroutine destroy function that is invoked when the coroutine is destroyed.
In LLVM's standard switched-resume lowering, signaled by the use of
`llvm.coro.id`, the coroutine frame is stored as part of a "coroutine
object" which represents a handle to a particular invocation of the
coroutine. All coroutine objects support a common ABI allowing certain
features to be used without knowing anything about the coroutine's
implementation:
.. note:: Splitting out resume and destroy functions are just one of the
possible ways of lowering the coroutine. We chose it for initial
implementation as it matches closely the mental model and results in
reasonably nice code.
- A coroutine object can be queried to see if it has reached completion
with `llvm.coro.done`.
- A coroutine object can be resumed normally if it has not already reached
completion with `llvm.coro.resume`.
- A coroutine object can be destroyed, invalidating the coroutine object,
with `llvm.coro.destroy`. This must be done separately even if the
coroutine has reached completion normally.
- "Promise" storage, which is known to have a certain size and alignment,
can be projected out of the coroutine object with `llvm.coro.promise`.
The coroutine implementation must have been compiled to define a promise
of the same size and alignment.
In general, interacting with a coroutine object in any of these ways while
it is running has undefined behavior.
The coroutine function is split into three functions, representing three
different ways that control can enter the coroutine:
1. the ramp function that is initially invoked, which takes arbitrary
arguments and returns a pointer to the coroutine object;
2. a coroutine resume function that is invoked when the coroutine is resumed,
which takes a pointer to the coroutine object and returns `void`;
3. a coroutine destroy function that is invoked when the coroutine is
destroyed, which takes a pointer to the coroutine object and returns
`void`.
Because the resume and destroy functions are shared across all suspend
points, suspend points must store the index of the active suspend in
the coroutine object, and the resume/destroy functions must switch over
that index to get back to the correct point. Hence the name of this
lowering.
Pointers to the resume and destroy functions are stored in the coroutine
object at known offsets which are fixed for all coroutines. A completed
coroutine is represented with a null resume function.
There is a somewhat complex protocol of intrinsics for allocating and
deallocating the coroutine object. It is complex in order to allow the
allocation to be elided due to inlining. This protocol is discussed
in further detail below.
The frontend may generate code to call the coroutine function directly;
this will become a call to the ramp function and will return a pointer
to the coroutine object. The frontend should always resume or destroy
the coroutine using the corresping intrinsics.
Returned-Continuation Lowering
------------------------------
In returned-continuation lowering, signaled by the use of
`llvm.coro.id.retcon` or `llvm.coro.id.retcon.once`, some aspects of
the ABI must be handled more explicitly by the frontend.
In this lowering, every suspend point takes a list of "yielded values"
which are returned back to the caller along with a function pointer,
called the continuation function. The coroutine is resumed by simply
calling this continuation function pointer. The original coroutine
is divided into the ramp function and then an arbitrary number of
these continuation functions, one for each suspend point.
LLVM actually supports two closely-related returned-continuation
lowerings:
- In normal returned-continuation lowering, the coroutine may suspend
itself multiple times. This means that a continuation function
itself returns another continuation pointer, as well as a list of
yielded values.
The coroutine indicates that it has run to completion by returning
a null continuation pointer. Any yielded values will be `undef`
should be ignored.
- In yield-once returned-continuation lowering, the coroutine must
suspend itself exactly once (or throw an exception). The ramp
function returns a continuation function pointer and yielded
values, but the continuation function simply returns `void`
when the coroutine has run to completion.
The coroutine frame is maintained in a fixed-size buffer that is
passed to the `coro.id` intrinsic, which guarantees a certain size
and alignment statically. The same buffer must be passed to the
continuation function(s). The coroutine will allocate memory if the
buffer is insufficient, in which case it will need to store at
least that pointer in the buffer; therefore the buffer must always
be at least pointer-sized. How the coroutine uses the buffer may
vary between suspend points.
In addition to the buffer pointer, continuation functions take an
argument indicating whether the coroutine is being resumed normally
(zero) or abnormally (non-zero).
LLVM is currently ineffective at statically eliminating allocations
after fully inlining returned-continuation coroutines into a caller.
This may be acceptable if LLVM's coroutine support is primarily being
used for low-level lowering and inlining is expected to be applied
earlier in the pipeline.
Coroutines by Example
=====================
The examples below are all of switched-resume coroutines.
Coroutine Representation
------------------------
@ -554,6 +666,7 @@ and python iterator `__next__` would look like:
return *(int*)coro.promise(hdl, 4, false);
}
Intrinsics
==========
@ -580,7 +693,7 @@ Overview:
"""""""""
The '``llvm.coro.destroy``' intrinsic destroys a suspended
coroutine.
switched-resume coroutine.
Arguments:
""""""""""
@ -607,7 +720,7 @@ frame. Destroying a coroutine that is not suspended leads to undefined behavior.
Overview:
"""""""""
The '``llvm.coro.resume``' intrinsic resumes a suspended coroutine.
The '``llvm.coro.resume``' intrinsic resumes a suspended switched-resume coroutine.
Arguments:
""""""""""
@ -634,8 +747,8 @@ Resuming a coroutine that is not suspended leads to undefined behavior.
Overview:
"""""""""
The '``llvm.coro.done``' intrinsic checks whether a suspended coroutine is at
the final suspend point or not.
The '``llvm.coro.done``' intrinsic checks whether a suspended
switched-resume coroutine is at the final suspend point or not.
Arguments:
""""""""""
@ -661,7 +774,7 @@ Overview:
"""""""""
The '``llvm.coro.promise``' intrinsic obtains a pointer to a
`coroutine promise`_ given a coroutine handle and vice versa.
`coroutine promise`_ given a switched-resume coroutine handle and vice versa.
Arguments:
""""""""""
@ -739,7 +852,8 @@ Overview:
"""""""""
The '``llvm.coro.size``' intrinsic returns the number of bytes
required to store a `coroutine frame`_.
required to store a `coroutine frame`_. This is only supported for
switched-resume coroutines.
Arguments:
""""""""""
@ -772,7 +886,8 @@ The first argument is a token returned by a call to '``llvm.coro.id``'
identifying the coroutine.
The second argument is a pointer to a block of memory where coroutine frame
will be stored if it is allocated dynamically.
will be stored if it is allocated dynamically. This pointer is ignored
for returned-continuation coroutines.
Semantics:
""""""""""
@ -798,7 +913,8 @@ Overview:
The '``llvm.coro.free``' intrinsic returns a pointer to a block of memory where
coroutine frame is stored or `null` if this instance of a coroutine did not use
dynamically allocated memory for its coroutine frame.
dynamically allocated memory for its coroutine frame. This intrinsic is not
supported for returned-continuation coroutines.
Arguments:
""""""""""
@ -847,6 +963,7 @@ Overview:
The '``llvm.coro.alloc``' intrinsic returns `true` if dynamic allocation is
required to obtain a memory for the coroutine frame and `false` otherwise.
This is not supported for returned-continuation coroutines.
Arguments:
""""""""""
@ -944,7 +1061,8 @@ coroutine frame.
Overview:
"""""""""
The '``llvm.coro.id``' intrinsic returns a token identifying a coroutine.
The '``llvm.coro.id``' intrinsic returns a token identifying a
switched-resume coroutine.
Arguments:
""""""""""
@ -975,6 +1093,79 @@ duplicated.
A frontend should emit exactly one `coro.id` intrinsic per coroutine.
.. _coro.id.retcon:
'llvm.coro.id.retcon' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
declare token @llvm.coro.id.retcon(i32 <size>, i32 <align>, i8* <buffer>,
i8* <continuation prototype>,
i8* <alloc>, i8* <dealloc>)
Overview:
"""""""""
The '``llvm.coro.id.retcon``' intrinsic returns a token identifying a
multiple-suspend returned-continuation coroutine.
The 'result-type sequence' of the coroutine is defined as follows:
- if the return type of the coroutine function is ``void``, it is the
empty sequence;
- if the return type of the coroutine function is a ``struct``, it is the
element types of that ``struct`` in order;
- otherwise, it is just the return type of the coroutine function.
The first element of the result-type sequence must be a pointer type;
continuation functions will be coerced to this type. The rest of
the sequence are the 'yield types', and any suspends in the coroutine
must take arguments of these types.
Arguments:
""""""""""
The first and second arguments are the expected size and alignment of
the buffer provided as the third argument. They must be constant.
The fourth argument must be a reference to a global function, called
the 'continuation prototype function'. The type, calling convention,
and attributes of any continuation functions will be taken from this
declaration. The return type of the prototype function must match the
return type of the current function. The first parameter type must be
a pointer type. The second parameter type must be an integer type;
it will be used only as a boolean flag.
The fifth argument must be a reference to a global function that will
be used to allocate memory. It may not fail, either by returning null
or throwing an exception. It must take an integer and return a pointer.
The sixth argument must be a reference to a global function that will
be used to deallocate memory. It must take a pointer and return ``void``.
'llvm.coro.id.retcon.once' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
declare token @llvm.coro.id.retcon.once(i32 <size>, i32 <align>, i8* <buffer>,
i8* <prototype>,
i8* <alloc>, i8* <dealloc>)
Overview:
"""""""""
The '``llvm.coro.id.retcon.once``' intrinsic returns a token identifying a
unique-suspend returned-continuation coroutine.
Arguments:
""""""""""
As for ``llvm.core.id.retcon``, except that the return type of the
continuation prototype must be `void` instead of matching the
coroutine's return type.
.. _coro.end:
'llvm.coro.end' Intrinsic
@ -1008,6 +1199,17 @@ The purpose of this intrinsic is to allow frontends to mark the cleanup and
other code that is only relevant during the initial invocation of the coroutine
and should not be present in resume and destroy parts.
In returned-continuation lowering, ``llvm.coro.end`` fully destroys the
coroutine frame. If the second argument is `false`, it also returns from
the coroutine with a null continuation pointer, and the next instruction
will be unreachable. If the second argument is `true`, it falls through
so that the following logic can resume unwinding. In a yield-once
coroutine, reaching a non-unwind ``llvm.coro.end`` without having first
reached a ``llvm.coro.suspend.retcon`` has undefined behavior.
The remainder of this section describes the behavior under switched-resume
lowering.
This intrinsic is lowered when a coroutine is split into
the start, resume and destroy parts. In the start part, it is a no-op,
in resume and destroy parts, it is replaced with `ret void` instruction and
@ -1078,11 +1280,11 @@ The following table summarizes the handling of `coro.end`_ intrinsic.
Overview:
"""""""""
The '``llvm.coro.suspend``' marks the point where execution of the coroutine
need to get suspended and control returned back to the caller.
Conditional branches consuming the result of this intrinsic lead to basic blocks
where coroutine should proceed when suspended (-1), resumed (0) or destroyed
(1).
The '``llvm.coro.suspend``' marks the point where execution of a
switched-resume coroutine is suspended and control is returned back
to the caller. Conditional branches consuming the result of this
intrinsic lead to basic blocks where coroutine should proceed when
suspended (-1), resumed (0) or destroyed (1).
Arguments:
""""""""""
@ -1178,6 +1380,45 @@ to the coroutine:
switch i8 %suspend1, label %suspend [i8 0, label %resume1
i8 1, label %cleanup]
.. _coro.suspend.retcon:
'llvm.coro.suspend.retcon' Intrinsic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
declare i1 @llvm.coro.suspend.retcon(...)
Overview:
"""""""""
The '``llvm.coro.suspend.retcon``' intrinsic marks the point where
execution of a returned-continuation coroutine is suspended and control
is returned back to the caller.
`llvm.coro.suspend.retcon`` does not support separate save points;
they are not useful when the continuation function is not locally
accessible. That would be a more appropriate feature for a ``passcon``
lowering that is not yet implemented.
Arguments:
""""""""""
The types of the arguments must exactly match the yielded-types sequence
of the coroutine. They will be turned into return values from the ramp
and continuation functions, along with the next continuation function.
Semantics:
""""""""""
The result of the intrinsic indicates whether the coroutine should resume
abnormally (non-zero).
In a normal coroutine, it is undefined behavior if the coroutine executes
a call to ``llvm.coro.suspend.retcon`` after resuming abnormally.
In a yield-once coroutine, it is undefined behavior if the coroutine
executes a call to ``llvm.coro.suspend.retcon`` after resuming in any way.
.. _coro.param:
'llvm.coro.param' Intrinsic

11
include/llvm/IR/Intrinsics.td
View File

@ -959,6 +959,14 @@ def int_coro_id : Intrinsic<[llvm_token_ty], [llvm_i32_ty, llvm_ptr_ty,
llvm_ptr_ty, llvm_ptr_ty],
[IntrArgMemOnly, IntrReadMem,
ReadNone<1>, ReadOnly<2>, NoCapture<2>]>;
def int_coro_id_retcon : Intrinsic<[llvm_token_ty],
[llvm_i32_ty, llvm_i32_ty, llvm_ptr_ty,
llvm_ptr_ty, llvm_ptr_ty, llvm_ptr_ty],
[]>;
def int_coro_id_retcon_once : Intrinsic<[llvm_token_ty],
[llvm_i32_ty, llvm_i32_ty, llvm_ptr_ty,
llvm_ptr_ty, llvm_ptr_ty, llvm_ptr_ty],
[]>;
def int_coro_alloc : Intrinsic<[llvm_i1_ty], [llvm_token_ty], []>;
def int_coro_begin : Intrinsic<[llvm_ptr_ty], [llvm_token_ty, llvm_ptr_ty],
[WriteOnly<1>]>;
@ -974,6 +982,9 @@ def int_coro_size : Intrinsic<[llvm_anyint_ty], [], [IntrNoMem]>;
def int_coro_save : Intrinsic<[llvm_token_ty], [llvm_ptr_ty], []>;
def int_coro_suspend : Intrinsic<[llvm_i8_ty], [llvm_token_ty, llvm_i1_ty], []>;
def int_coro_suspend_retcon : Intrinsic<[llvm_i1_ty], [llvm_vararg_ty], []>;
def int_coro_prepare_retcon : Intrinsic<[llvm_ptr_ty], [llvm_ptr_ty],
[IntrNoMem]>;
def int_coro_param : Intrinsic<[llvm_i1_ty], [llvm_ptr_ty, llvm_ptr_ty],
[IntrNoMem, ReadNone<0>, ReadNone<1>]>;

5
lib/Transforms/Coroutines/CoroCleanup.cpp
View File

@ -73,6 +73,8 @@ bool Lowerer::lowerRemainingCoroIntrinsics(Function &F) {
II->replaceAllUsesWith(ConstantInt::getTrue(Context));
break;
case Intrinsic::coro_id:
case Intrinsic::coro_id_retcon:
case Intrinsic::coro_id_retcon_once:
II->replaceAllUsesWith(ConstantTokenNone::get(Context));
break;
case Intrinsic::coro_subfn_addr:
@ -111,7 +113,8 @@ struct CoroCleanup : FunctionPass {
bool doInitialization(Module &M) override {
if (coro::declaresIntrinsics(M, {"llvm.coro.alloc", "llvm.coro.begin",
"llvm.coro.subfn.addr", "llvm.coro.free",
"llvm.coro.id"}))
"llvm.coro.id", "llvm.coro.id.retcon",
"llvm.coro.id.retcon.once"}))
L = llvm::make_unique<Lowerer>(M);
return false;
}

24
lib/Transforms/Coroutines/CoroEarly.cpp
View File

@ -91,13 +91,14 @@ void Lowerer::lowerCoroDone(IntrinsicInst *II) {
Value *Operand = II->getArgOperand(0);
// ResumeFnAddr is the first pointer sized element of the coroutine frame.
static_assert(coro::Shape::SwitchFieldIndex::Resume == 0,
"resume function not at offset zero");
auto *FrameTy = Int8Ptr;
PointerType *FramePtrTy = FrameTy->getPointerTo();
Builder.SetInsertPoint(II);
auto *BCI = Builder.CreateBitCast(Operand, FramePtrTy);
auto *Gep = Builder.CreateConstInBoundsGEP1_32(FrameTy, BCI, 0);
auto *Load = Builder.CreateLoad(FrameTy, Gep);
auto *Load = Builder.CreateLoad(BCI);
auto *Cond = Builder.CreateICmpEQ(Load, NullPtr);
II->replaceAllUsesWith(Cond);
@ -189,6 +190,10 @@ bool Lowerer::lowerEarlyIntrinsics(Function &F) {
}
}
break;
case Intrinsic::coro_id_retcon:
case Intrinsic::coro_id_retcon_once:
F.addFnAttr(CORO_PRESPLIT_ATTR, PREPARED_FOR_SPLIT);
break;
case Intrinsic::coro_resume:
lowerResumeOrDestroy(CS, CoroSubFnInst::ResumeIndex);
break;
@ -231,10 +236,17 @@ struct CoroEarly : public FunctionPass {
// This pass has work to do only if we find intrinsics we are going to lower
// in the module.
bool doInitialization(Module &M) override {
if (coro::declaresIntrinsics(
M, {"llvm.coro.id", "llvm.coro.destroy", "llvm.coro.done",
"llvm.coro.end", "llvm.coro.noop", "llvm.coro.free",
"llvm.coro.promise", "llvm.coro.resume", "llvm.coro.suspend"}))
if (coro::declaresIntrinsics(M, {"llvm.coro.id",
"llvm.coro.id.retcon",
"llvm.coro.id.retcon.once",
"llvm.coro.destroy",
"llvm.coro.done",
"llvm.coro.end",
"llvm.coro.noop",
"llvm.coro.free",
"llvm.coro.promise",
"llvm.coro.resume",
"llvm.coro.suspend"}))
L = llvm::make_unique<Lowerer>(M);
return false;
}

141
lib/Transforms/Coroutines/CoroFrame.cpp
View File

@ -120,6 +120,15 @@ struct SuspendCrossingInfo {
return false;
BasicBlock *UseBB = I->getParent();
// As a special case, treat uses by an llvm.coro.suspend.retcon
// as if they were uses in the suspend's single predecessor: the
// uses conceptually occur before the suspend.
if (isa<CoroSuspendRetconInst>(I)) {
UseBB = UseBB->getSinglePredecessor();
assert(UseBB && "should have split coro.suspend into its own block");
}
return hasPathCrossingSuspendPoint(DefBB, UseBB);
}
@ -128,7 +137,17 @@ struct SuspendCrossingInfo {
}
bool isDefinitionAcrossSuspend(Instruction &I, User *U) const {
return isDefinitionAcrossSuspend(I.getParent(), U);
auto *DefBB = I.getParent();
// As a special case, treat values produced by an llvm.coro.suspend.*
// as if they were defined in the single successor: the uses
// conceptually occur after the suspend.
if (isa<AnyCoroSuspendInst>(I)) {
DefBB = DefBB->getSingleSuccessor();
assert(DefBB && "should have split coro.suspend into its own block");
}
return isDefinitionAcrossSuspend(DefBB, U);
}
};
} // end anonymous namespace
@ -183,9 +202,10 @@ SuspendCrossingInfo::SuspendCrossingInfo(Function &F, coro::Shape &Shape)
B.Suspend = true;
B.Kills |= B.Consumes;
};
for (CoroSuspendInst *CSI : Shape.CoroSuspends) {
for (auto *CSI : Shape.CoroSuspends) {
markSuspendBlock(CSI);
markSuspendBlock(CSI->getCoroSave());
if (auto *Save = CSI->getCoroSave())
markSuspendBlock(Save);
}
// Iterate propagating consumes and kills until they stop changing.
@ -261,11 +281,13 @@ SuspendCrossingInfo::SuspendCrossingInfo(Function &F, coro::Shape &Shape)
// We build up the list of spills for every case where a use is separated
// from the definition by a suspend point.
static const unsigned InvalidFieldIndex = ~0U;
namespace {
class Spill {
Value *Def = nullptr;
Instruction *User = nullptr;
unsigned FieldNo = 0;
unsigned FieldNo = InvalidFieldIndex;
public:
Spill(Value *Def, llvm::User *U) : Def(Def), User(cast<Instruction>(U)) {}
@ -280,11 +302,11 @@ public:
// the definition the first time they encounter it. Consider refactoring
// SpillInfo into two arrays to normalize the spill representation.
unsigned fieldIndex() const {
assert(FieldNo && "Accessing unassigned field");
assert(FieldNo != InvalidFieldIndex && "Accessing unassigned field");
return FieldNo;
}
void setFieldIndex(unsigned FieldNumber) {
assert(!FieldNo && "Reassigning field number");
assert(FieldNo == InvalidFieldIndex && "Reassigning field number");
FieldNo = FieldNumber;
}
};
@ -376,18 +398,30 @@ static StructType *buildFrameType(Function &F, coro::Shape &Shape,
SmallString<32> Name(F.getName());
Name.append(".Frame");
StructType *FrameTy = StructType::create(C, Name);
auto *FramePtrTy = FrameTy->getPointerTo();
auto *FnTy = FunctionType::get(Type::getVoidTy(C), FramePtrTy,
/*isVarArg=*/false);
auto *FnPtrTy = FnTy->getPointerTo();
SmallVector<Type *, 8> Types;
AllocaInst *PromiseAlloca = Shape.getPromiseAlloca();
if (Shape.ABI == coro::ABI::Switch) {
auto *FramePtrTy = FrameTy->getPointerTo();
auto *FnTy = FunctionType::get(Type::getVoidTy(C), FramePtrTy,
/*IsVarArg=*/false);
auto *FnPtrTy = FnTy->getPointerTo();
// Figure out how wide should be an integer type storing the suspend index.
unsigned IndexBits = std::max(1U, Log2_64_Ceil(Shape.CoroSuspends.size()));
Type *PromiseType = PromiseAlloca
? PromiseAlloca->getType()->getElementType()
: Type::getInt1Ty(C);
Type *IndexType = Type::getIntNTy(C, IndexBits);
Types.push_back(FnPtrTy);
Types.push_back(FnPtrTy);
Types.push_back(PromiseType);
Types.push_back(IndexType);
} else {
assert(PromiseAlloca == nullptr && "lowering doesn't support promises");
}
// Figure out how wide should be an integer type storing the suspend index.
unsigned IndexBits = std::max(1U, Log2_64_Ceil(Shape.CoroSuspends.size()));
Type *PromiseType = Shape.PromiseAlloca
? Shape.PromiseAlloca->getType()->getElementType()
: Type::getInt1Ty(C);
SmallVector<Type *, 8> Types{FnPtrTy, FnPtrTy, PromiseType,
Type::getIntNTy(C, IndexBits)};
Value *CurrentDef = nullptr;
Padder.addTypes(Types);
@ -399,7 +433,7 @@ static StructType *buildFrameType(Function &F, coro::Shape &Shape,
CurrentDef = S.def();
// PromiseAlloca was already added to Types array earlier.
if (CurrentDef == Shape.PromiseAlloca)
if (CurrentDef == PromiseAlloca)
continue;
uint64_t Count = 1;
@ -430,6 +464,22 @@ static StructType *buildFrameType(Function &F, coro::Shape &Shape,
}
FrameTy->setBody(Types);
switch (Shape.ABI) {
case coro::ABI::Switch:
break;
// Remember whether the frame is inline in the storage.
case coro::ABI::Retcon:
case coro::ABI::RetconOnce: {
auto &Layout = F.getParent()->getDataLayout();
auto Id = Shape.getRetconCoroId();
Shape.RetconLowering.IsFrameInlineInStorage
= (Layout.getTypeAllocSize(FrameTy) <= Id->getStorageSize() &&
Layout.getABITypeAlignment(FrameTy) <= Id->getStorageAlignment());
break;
}
}
return FrameTy;
}
@ -476,7 +526,7 @@ static Instruction *splitBeforeCatchSwitch(CatchSwitchInst *CatchSwitch) {
// whatever
//
//
static Instruction *insertSpills(SpillInfo &Spills, coro::Shape &Shape) {
static Instruction *insertSpills(const SpillInfo &Spills, coro::Shape &Shape) {
auto *CB = Shape.CoroBegin;
LLVMContext &C = CB->getContext();
IRBuilder<> Builder(CB->getNextNode());
@ -488,7 +538,9 @@ static Instruction *insertSpills(SpillInfo &Spills, coro::Shape &Shape) {
Value *CurrentValue = nullptr;
BasicBlock *CurrentBlock = nullptr;
Value *CurrentReload = nullptr;
unsigned Index = 0; // Proper field number will be read from field definition.
// Proper field number will be read from field definition.
unsigned Index = InvalidFieldIndex;
// We need to keep track of any allocas that need "spilling"
// since they will live in the coroutine frame now, all access to them
@ -496,9 +548,11 @@ static Instruction *insertSpills(SpillInfo &Spills, coro::Shape &Shape) {
// we remember allocas and their indices to be handled once we processed
// all the spills.
SmallVector<std::pair<AllocaInst *, unsigned>, 4> Allocas;
// Promise alloca (if present) has a fixed field number (Shape::PromiseField)
if (Shape.PromiseAlloca)
Allocas.emplace_back(Shape.PromiseAlloca, coro::Shape::PromiseField);
// Promise alloca (if present) has a fixed field number.
if (auto *PromiseAlloca = Shape.getPromiseAlloca()) {
assert(Shape.ABI == coro::ABI::Switch);
Allocas.emplace_back(PromiseAlloca, coro::Shape::SwitchFieldIndex::Promise);
}
// Create a GEP with the given index into the coroutine frame for the original
// value Orig. Appends an extra 0 index for array-allocas, preserving the
@ -526,7 +580,7 @@ static Instruction *insertSpills(SpillInfo &Spills, coro::Shape &Shape) {
// Create a load instruction to reload the spilled value from the coroutine
// frame.
auto CreateReload = [&](Instruction *InsertBefore) {
assert(Index && "accessing unassigned field number");
assert(Index != InvalidFieldIndex && "accessing unassigned field number");
Builder.SetInsertPoint(InsertBefore);
auto *G = GetFramePointer(Index, CurrentValue);
@ -558,23 +612,33 @@ static Instruction *insertSpills(SpillInfo &Spills, coro::Shape &Shape) {
// coroutine frame.
Instruction *InsertPt = nullptr;
if (isa<Argument>(CurrentValue)) {
if (auto Arg = dyn_cast<Argument>(CurrentValue)) {
// For arguments, we will place the store instruction right after
// the coroutine frame pointer instruction, i.e. bitcast of
// coro.begin from i8* to %f.frame*.
InsertPt = FramePtr->getNextNode();
// If we're spilling an Argument, make sure we clear 'nocapture'
// from the coroutine function.
Arg->getParent()->removeParamAttr(Arg->getArgNo(),
Attribute::NoCapture);
} else if (auto *II = dyn_cast<InvokeInst>(CurrentValue)) {
// If we are spilling the result of the invoke instruction, split the
// normal edge and insert the spill in the new block.
auto NewBB = SplitEdge(II->getParent(), II->getNormalDest());
InsertPt = NewBB->getTerminator();
} else if (dyn_cast<PHINode>(CurrentValue)) {
} else if (isa<PHINode>(CurrentValue)) {
// Skip the PHINodes and EH pads instructions.
BasicBlock *DefBlock = cast<Instruction>(E.def())->getParent();
if (auto *CSI = dyn_cast<CatchSwitchInst>(DefBlock->getTerminator()))
InsertPt = splitBeforeCatchSwitch(CSI);
else
InsertPt = &*DefBlock->getFirstInsertionPt();
} else if (auto CSI = dyn_cast<AnyCoroSuspendInst>(CurrentValue)) {
// Don't spill immediately after a suspend; splitting assumes
// that the suspend will be followed by a branch.
InsertPt = CSI->getParent()->getSingleSuccessor()->getFirstNonPHI();
} else {
// For all other values, the spill is placed immediately after
// the definition.
@ -613,13 +677,14 @@ static Instruction *insertSpills(SpillInfo &Spills, coro::Shape &Shape) {
}
BasicBlock *FramePtrBB = FramePtr->getParent();
Shape.AllocaSpillBlock =
FramePtrBB->splitBasicBlock(FramePtr->getNextNode(), "AllocaSpillBB");
Shape.AllocaSpillBlock->splitBasicBlock(&Shape.AllocaSpillBlock->front(),
"PostSpill");
Builder.SetInsertPoint(&Shape.AllocaSpillBlock->front());
auto SpillBlock =
FramePtrBB->splitBasicBlock(FramePtr->getNextNode(), "AllocaSpillBB");
SpillBlock->splitBasicBlock(&SpillBlock->front(), "PostSpill");
Shape.AllocaSpillBlock = SpillBlock;
// If we found any allocas, replace all of their remaining uses with Geps.
Builder.SetInsertPoint(&SpillBlock->front());
for (auto &P : Allocas) {
auto *G = GetFramePointer(P.second, P.first);
@ -900,16 +965,17 @@ void coro::buildCoroutineFrame(Function &F, Shape &Shape) {
// access to local variables.
LowerDbgDeclare(F);
Shape.PromiseAlloca = Shape.CoroBegin->getId()->getPromise();
if (Shape.PromiseAlloca) {
Shape.CoroBegin->getId()->clearPromise();
if (Shape.ABI == coro::ABI::Switch &&
Shape.SwitchLowering.PromiseAlloca) {
Shape.getSwitchCoroId()->clearPromise();
}
// Make sure that all coro.save, coro.suspend and the fallthrough coro.end
// intrinsics are in their own blocks to simplify the logic of building up
// SuspendCrossing data.
for (CoroSuspendInst *CSI : Shape.CoroSuspends) {
splitAround(CSI->getCoroSave(), "CoroSave");
for (auto *CSI : Shape.CoroSuspends) {
if (auto *Save = CSI->getCoroSave())
splitAround(Save, "CoroSave");
splitAround(CSI, "CoroSuspend");
}
@ -957,7 +1023,8 @@ void coro::buildCoroutineFrame(Function &F, Shape &Shape) {
continue;
// The Coroutine Promise always included into coroutine frame, no need to
// check for suspend crossing.
if (Shape.PromiseAlloca == &I)
if (Shape.ABI == coro::ABI::Switch &&
Shape.SwitchLowering.PromiseAlloca == &I)
continue;
for (User *U : I.users())

151
lib/Transforms/Coroutines/CoroInstr.h
View File

@ -27,6 +27,7 @@
#include "llvm/IR/GlobalVariable.h"
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/Support/raw_ostream.h"
namespace llvm {
@ -77,10 +78,8 @@ public:
}
};
/// This represents the llvm.coro.alloc instruction.
class LLVM_LIBRARY_VISIBILITY CoroIdInst : public IntrinsicInst {
enum { AlignArg, PromiseArg, CoroutineArg, InfoArg };
/// This represents a common base class for llvm.coro.id instructions.
class LLVM_LIBRARY_VISIBILITY AnyCoroIdInst : public IntrinsicInst {
public:
CoroAllocInst *getCoroAlloc() {
for (User *U : users())
@ -97,6 +96,24 @@ public:
llvm_unreachable("no coro.begin associated with coro.id");
}
// Methods to support type inquiry through isa, cast, and dyn_cast:
static bool classof(const IntrinsicInst *I) {
auto ID = I->getIntrinsicID();
return ID == Intrinsic::coro_id ||
ID == Intrinsic::coro_id_retcon ||
ID == Intrinsic::coro_id_retcon_once;
}
static bool classof(const Value *V) {
return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
}
};
/// This represents the llvm.coro.id instruction.
class LLVM_LIBRARY_VISIBILITY CoroIdInst : public AnyCoroIdInst {
enum { AlignArg, PromiseArg, CoroutineArg, InfoArg };
public:
AllocaInst *getPromise() const {
Value *Arg = getArgOperand(PromiseArg);
return isa<ConstantPointerNull>(Arg)
@ -182,6 +199,80 @@ public:
}
};
/// This represents either the llvm.coro.id.retcon or
/// llvm.coro.id.retcon.once instruction.
class LLVM_LIBRARY_VISIBILITY AnyCoroIdRetconInst : public AnyCoroIdInst {
enum { SizeArg, AlignArg, StorageArg, PrototypeArg, AllocArg, DeallocArg };
public:
void checkWellFormed() const;
uint64_t getStorageSize() const {
return cast<ConstantInt>(getArgOperand(SizeArg))->getZExtValue();
}
uint64_t getStorageAlignment() const {
return cast<ConstantInt>(getArgOperand(AlignArg))->getZExtValue();
}
Value *getStorage() const {
return getArgOperand(StorageArg);
}
/// Return the prototype for the continuation function. The type,
/// attributes, and calling convention of the continuation function(s)
/// are taken from this declaration.
Function *getPrototype() const {
return cast<Function>(getArgOperand(PrototypeArg)->stripPointerCasts());
}
/// Return the function to use for allocating memory.
Function *getAllocFunction() const {
return cast<Function>(getArgOperand(AllocArg)->stripPointerCasts());
}
/// Return the function to use for deallocating memory.
Function *getDeallocFunction() const {
return cast<Function>(getArgOperand(DeallocArg)->stripPointerCasts());
}
// Methods to support type inquiry through isa, cast, and dyn_cast:
static bool classof(const IntrinsicInst *I) {
auto ID = I->getIntrinsicID();
return ID == Intrinsic::coro_id_retcon
|| ID == Intrinsic::coro_id_retcon_once;
}
static bool classof(const Value *V) {
return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
}
};
/// This represents the llvm.coro.id.retcon instruction.
class LLVM_LIBRARY_VISIBILITY CoroIdRetconInst
: public AnyCoroIdRetconInst {
public:
// Methods to support type inquiry through isa, cast, and dyn_cast:
static bool classof(const IntrinsicInst *I) {
return I->getIntrinsicID() == Intrinsic::coro_id_retcon;
}
static bool classof(const Value *V) {
return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
}
};
/// This represents the llvm.coro.id.retcon.once instruction.
class LLVM_LIBRARY_VISIBILITY CoroIdRetconOnceInst
: public AnyCoroIdRetconInst {
public:
// Methods to support type inquiry through isa, cast, and dyn_cast:
static bool classof(const IntrinsicInst *I) {
return I->getIntrinsicID() == Intrinsic::coro_id_retcon_once;
}
static bool classof(const Value *V) {
return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
}
};
/// This represents the llvm.coro.frame instruction.
class LLVM_LIBRARY_VISIBILITY CoroFrameInst : public IntrinsicInst {
public:
@ -215,7 +306,9 @@ class LLVM_LIBRARY_VISIBILITY CoroBeginInst : public IntrinsicInst {
enum { IdArg, MemArg };
public:
CoroIdInst *getId() const { return cast<CoroIdInst>(getArgOperand(IdArg)); }
AnyCoroIdInst *getId() const {
return cast<AnyCoroIdInst>(getArgOperand(IdArg));
}
Value *getMem() const { return getArgOperand(MemArg); }
@ -261,8 +354,22 @@ public:
}
};
class LLVM_LIBRARY_VISIBILITY AnyCoroSuspendInst : public IntrinsicInst {
public:
CoroSaveInst *getCoroSave() const;
// Methods to support type inquiry through isa, cast, and dyn_cast:
static bool classof(const IntrinsicInst *I) {
return I->getIntrinsicID() == Intrinsic::coro_suspend ||
I->getIntrinsicID() == Intrinsic::coro_suspend_retcon;
}
static bool classof(const Value *V) {
return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
}
};
/// This represents the llvm.coro.suspend instruction.
class LLVM_LIBRARY_VISIBILITY CoroSuspendInst : public IntrinsicInst {
class LLVM_LIBRARY_VISIBILITY CoroSuspendInst : public AnyCoroSuspendInst {
enum { SaveArg, FinalArg };
public:
@ -273,6 +380,7 @@ public:
assert(isa<ConstantTokenNone>(Arg));
return nullptr;
}
bool isFinal() const {
return cast<Constant>(getArgOperand(FinalArg))->isOneValue();
}
@ -286,6 +394,37 @@ public:
}
};
inline CoroSaveInst *AnyCoroSuspendInst::getCoroSave() const {
if (auto Suspend = dyn_cast<CoroSuspendInst>(this))
return Suspend->getCoroSave();
return nullptr;
}
/// This represents the llvm.coro.suspend.retcon instruction.
class LLVM_LIBRARY_VISIBILITY CoroSuspendRetconInst : public AnyCoroSuspendInst {
public:
op_iterator value_begin() { return arg_begin(); }
const_op_iterator value_begin() const { return arg_begin(); }
op_iterator value_end() { return arg_end(); }
const_op_iterator value_end() const { return arg_end(); }
iterator_range<op_iterator> value_operands() {
return make_range(value_begin(), value_end());
}
iterator_range<const_op_iterator> value_operands() const {
return make_range(value_begin(), value_end());
}
// Methods to support type inquiry through isa, cast, and dyn_cast:
static bool classof(const IntrinsicInst *I) {
return I->getIntrinsicID() == Intrinsic::coro_suspend_retcon;
}
static bool classof(const Value *V) {
return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
}
};
/// This represents the llvm.coro.size instruction.
class LLVM_LIBRARY_VISIBILITY CoroSizeInst : public IntrinsicInst {
public:

147
lib/Transforms/Coroutines/CoroInternal.h
View File

@ -12,6 +12,7 @@
#define LLVM_LIB_TRANSFORMS_COROUTINES_COROINTERNAL_H
#include "CoroInstr.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/Transforms/Coroutines.h"
namespace llvm {
@ -61,37 +62,161 @@ struct LowererBase {
Value *makeSubFnCall(Value *Arg, int Index, Instruction *InsertPt);
};
enum class ABI {
/// The "resume-switch" lowering, where there are separate resume and
/// destroy functions that are shared between all suspend points. The
/// coroutine frame implicitly stores the resume and destroy functions,
/// the current index, and any promise value.
Switch,
/// The "returned-continuation" lowering, where each suspend point creates a
/// single continuation function that is used for both resuming and
/// destroying. Does not support promises.
Retcon,
/// The "unique returned-continuation" lowering, where each suspend point
/// creates a single continuation function that is used for both resuming
/// and destroying. Does not support promises. The function is known to
/// suspend at most once during its execution, and the return value of
/// the continuation is void.
RetconOnce,
};
// Holds structural Coroutine Intrinsics for a particular function and other
// values used during CoroSplit pass.
struct LLVM_LIBRARY_VISIBILITY Shape {
CoroBeginInst *CoroBegin;
SmallVector<CoroEndInst *, 4> CoroEnds;
SmallVector<CoroSizeInst *, 2> CoroSizes;
SmallVector<CoroSuspendInst *, 4> CoroSuspends;
SmallVector<AnyCoroSuspendInst *, 4> CoroSuspends;
// Field Indexes for known coroutine frame fields.
enum {
ResumeField,
DestroyField,
PromiseField,
IndexField,
// Field indexes for special fields in the switch lowering.
struct SwitchFieldIndex {
enum {
Resume,
Destroy,
Promise,
Index,
/// The index of the first spill field.
FirstSpill
};
};
coro::ABI ABI;
StructType *FrameTy;
Instruction *FramePtr;
BasicBlock *AllocaSpillBlock;
SwitchInst *ResumeSwitch;
AllocaInst *PromiseAlloca;
bool HasFinalSuspend;
struct SwitchLoweringStorage {
SwitchInst *ResumeSwitch;
AllocaInst *PromiseAlloca;
BasicBlock *ResumeEntryBlock;
bool HasFinalSuspend;
};
struct RetconLoweringStorage {
Function *ResumePrototype;
Function *Alloc;
Function *Dealloc;
BasicBlock *ReturnBlock;
bool IsFrameInlineInStorage;
};
union {
SwitchLoweringStorage SwitchLowering;
RetconLoweringStorage RetconLowering;
};
CoroIdInst *getSwitchCoroId() const {
assert(ABI == coro::ABI::Switch);
return cast<CoroIdInst>(CoroBegin->getId());
}
AnyCoroIdRetconInst *getRetconCoroId() const {
assert(ABI == coro::ABI::Retcon ||
ABI == coro::ABI::RetconOnce);
return cast<AnyCoroIdRetconInst>(CoroBegin->getId());
}
IntegerType *getIndexType() const {
assert(ABI == coro::ABI::Switch);
assert(FrameTy && "frame type not assigned");
return cast<IntegerType>(FrameTy->getElementType(IndexField));
return cast<IntegerType>(FrameTy->getElementType(SwitchFieldIndex::Index));
}
ConstantInt *getIndex(uint64_t Value) const {
return ConstantInt::get(getIndexType(), Value);
}
PointerType *getSwitchResumePointerType() const {
assert(ABI == coro::ABI::Switch);
assert(FrameTy && "frame type not assigned");
return cast<PointerType>(FrameTy->getElementType(SwitchFieldIndex::Resume));
}
FunctionType *getResumeFunctionType() const {
switch (ABI) {
case coro::ABI::Switch: {
auto *FnPtrTy = getSwitchResumePointerType();
return cast<FunctionType>(FnPtrTy->getPointerElementType());
}
case coro::ABI::Retcon:
case coro::ABI::RetconOnce:
return RetconLowering.ResumePrototype->getFunctionType();
}
}
ArrayRef<Type*> getRetconResultTypes() const {
assert(ABI == coro::ABI::Retcon ||
ABI == coro::ABI::RetconOnce);
auto FTy = CoroBegin->getFunction()->getFunctionType();
// This is checked by AnyCoroIdRetconInst::isWellFormed().
if (auto STy = dyn_cast<StructType>(FTy->getReturnType())) {
return STy->elements().slice(1);
} else {
return ArrayRef<Type*>();
}
}
CallingConv::ID getResumeFunctionCC() const {
switch (ABI) {
case coro::ABI::Switch:
return CallingConv::Fast;
case coro::ABI::Retcon:
case coro::ABI::RetconOnce:
return RetconLowering.ResumePrototype->getCallingConv();
}
}
unsigned getFirstSpillFieldIndex() const {
switch (ABI) {
case coro::ABI::Switch:
return SwitchFieldIndex::FirstSpill;
case coro::ABI::Retcon:
case coro::ABI::RetconOnce:
return 0;
}
}
AllocaInst *getPromiseAlloca() const {
if (ABI == coro::ABI::Switch)
return SwitchLowering.PromiseAlloca;
return nullptr;
}
/// Allocate memory according to the rules of the active lowering.
///
/// \param CG - if non-null, will be updated for the new call
Value *emitAlloc(IRBuilder<> &Builder, Value *Size, CallGraph *CG) const;
/// Deallocate memory according to the rules of the active lowering.
///
/// \param CG - if non-null, will be updated for the new call
void emitDealloc(IRBuilder<> &Builder, Value *Ptr, CallGraph *CG) const;
Shape() = default;
explicit Shape(Function &F) { buildFrom(F); }
void buildFrom(Function &F);

939
lib/Transforms/Coroutines/CoroSplit.cpp
View File

File diff suppressed because it is too large Load Diff

292
lib/Transforms/Coroutines/Coroutines.cpp
View File

@ -123,12 +123,26 @@ Value *coro::LowererBase::makeSubFnCall(Value *Arg, int Index,
static bool isCoroutineIntrinsicName(StringRef Name) {
// NOTE: Must be sorted!
static const char *const CoroIntrinsics[] = {
"llvm.coro.alloc", "llvm.coro.begin", "llvm.coro.destroy",
"llvm.coro.done", "llvm.coro.end", "llvm.coro.frame",
"llvm.coro.free", "llvm.coro.id", "llvm.coro.noop",
"llvm.coro.param", "llvm.coro.promise", "llvm.coro.resume",
"llvm.coro.save", "llvm.coro.size", "llvm.coro.subfn.addr",
"llvm.coro.alloc",
"llvm.coro.begin",
"llvm.coro.destroy",
"llvm.coro.done",
"llvm.coro.end",
"llvm.coro.frame",
"llvm.coro.free",
"llvm.coro.id",
"llvm.coro.id.retcon",
"llvm.coro.id.retcon.once",
"llvm.coro.noop",
"llvm.coro.param",
"llvm.coro.prepare.retcon",
"llvm.coro.promise",
"llvm.coro.resume",
"llvm.coro.save",
"llvm.coro.size",
"llvm.coro.subfn.addr",
"llvm.coro.suspend",
"llvm.coro.suspend.retcon",
};
return Intrinsic::lookupLLVMIntrinsicByName(CoroIntrinsics, Name) != -1;
}
@ -217,9 +231,6 @@ static void clear(coro::Shape &Shape) {
Shape.FrameTy = nullptr;
Shape.FramePtr = nullptr;
Shape.AllocaSpillBlock = nullptr;
Shape.ResumeSwitch = nullptr;
Shape.PromiseAlloca = nullptr;
Shape.HasFinalSuspend = false;
}
static CoroSaveInst *createCoroSave(CoroBeginInst *CoroBegin,
@ -235,6 +246,7 @@ static CoroSaveInst *createCoroSave(CoroBeginInst *CoroBegin,
// Collect "interesting" coroutine intrinsics.
void coro::Shape::buildFrom(Function &F) {
bool HasFinalSuspend = false;
size_t FinalSuspendIndex = 0;
clear(*this);
SmallVector<CoroFrameInst *, 8> CoroFrames;
@ -257,9 +269,15 @@ void coro::Shape::buildFrom(Function &F) {
if (II->use_empty())
UnusedCoroSaves.push_back(cast<CoroSaveInst>(II));
break;
case Intrinsic::coro_suspend:
CoroSuspends.push_back(cast<CoroSuspendInst>(II));
if (CoroSuspends.back()->isFinal()) {
case Intrinsic::coro_suspend_retcon: {
auto Suspend = cast<CoroSuspendRetconInst>(II);
CoroSuspends.push_back(Suspend);
break;
}
case Intrinsic::coro_suspend: {
auto Suspend = cast<CoroSuspendInst>(II);
CoroSuspends.push_back(Suspend);
if (Suspend->isFinal()) {
if (HasFinalSuspend)
report_fatal_error(
"Only one suspend point can be marked as final");
@ -267,18 +285,23 @@ void coro::Shape::buildFrom(Function &F) {
FinalSuspendIndex = CoroSuspends.size() - 1;
}
break;
}
case Intrinsic::coro_begin: {
auto CB = cast<CoroBeginInst>(II);
if (CB->getId()->getInfo().isPreSplit()) {
if (CoroBegin)
report_fatal_error(
// Ignore coro id's that aren't pre-split.
auto Id = dyn_cast<CoroIdInst>(CB->getId());
if (Id && !Id->getInfo().isPreSplit())
break;
if (CoroBegin)
report_fatal_error(
"coroutine should have exactly one defining @llvm.coro.begin");
CB->addAttribute(AttributeList::ReturnIndex, Attribute::NonNull);
CB->addAttribute(AttributeList::ReturnIndex, Attribute::NoAlias);
CB->removeAttribute(AttributeList::FunctionIndex,
Attribute::NoDuplicate);
CoroBegin = CB;
}
CB->addAttribute(AttributeList::ReturnIndex, Attribute::NonNull);
CB->addAttribute(AttributeList::ReturnIndex, Attribute::NoAlias);
CB->removeAttribute(AttributeList::FunctionIndex,
Attribute::NoDuplicate);
CoroBegin = CB;
break;
}
case Intrinsic::coro_end:
@ -310,7 +333,7 @@ void coro::Shape::buildFrom(Function &F) {
// Replace all coro.suspend with undef and remove related coro.saves if
// present.
for (CoroSuspendInst *CS : CoroSuspends) {
for (AnyCoroSuspendInst *CS : CoroSuspends) {
CS->replaceAllUsesWith(UndefValue::get(CS->getType()));
CS->eraseFromParent();
if (auto *CoroSave = CS->getCoroSave())
@ -324,19 +347,87 @@ void coro::Shape::buildFrom(Function &F) {
return;
}
auto Id = CoroBegin->getId();
switch (auto IdIntrinsic = Id->getIntrinsicID()) {
case Intrinsic::coro_id: {
auto SwitchId = cast<CoroIdInst>(Id);
this->ABI = coro::ABI::Switch;
this->SwitchLowering.HasFinalSuspend = HasFinalSuspend;
this->SwitchLowering.ResumeSwitch = nullptr;
this->SwitchLowering.PromiseAlloca = SwitchId->getPromise();
this->SwitchLowering.ResumeEntryBlock = nullptr;
for (auto AnySuspend : CoroSuspends) {
auto Suspend = dyn_cast<CoroSuspendInst>(AnySuspend);
if (!Suspend) {
AnySuspend->dump();
report_fatal_error("coro.id must be paired with coro.suspend");
}
if (!Suspend->getCoroSave())
createCoroSave(CoroBegin, Suspend);
}
break;
}
case Intrinsic::coro_id_retcon:
case Intrinsic::coro_id_retcon_once: {
auto ContinuationId = cast<AnyCoroIdRetconInst>(Id);
ContinuationId->checkWellFormed();
this->ABI = (IdIntrinsic == Intrinsic::coro_id_retcon
? coro::ABI::Retcon
: coro::ABI::RetconOnce);
auto Prototype = ContinuationId->getPrototype();
this->RetconLowering.ResumePrototype = Prototype;
this->RetconLowering.Alloc = ContinuationId->getAllocFunction();
this->RetconLowering.Dealloc = ContinuationId->getDeallocFunction();
this->RetconLowering.ReturnBlock = nullptr;
this->RetconLowering.IsFrameInlineInStorage = false;
// Determine the result value types, and make sure they match up with
// the values passed to the suspends.
auto ResultTys = getRetconResultTypes();
for (auto AnySuspend : CoroSuspends) {
auto Suspend = dyn_cast<CoroSuspendRetconInst>(AnySuspend);
if (!Suspend) {
AnySuspend->dump();
report_fatal_error("coro.id.retcon.* must be paired with "
"coro.suspend.retcon");
}
auto SI = Suspend->value_begin(), SE = Suspend->value_end();
auto RI = ResultTys.begin(), RE = ResultTys.end();
for (; SI != SE && RI != RE; ++SI, ++RI) {
if ((*SI)->getType() != *RI) {
Suspend->dump();
Prototype->getFunctionType()->dump();
report_fatal_error("argument to coro.suspend.retcon does not "
"match corresponding prototype function result");
}
}
if (SI != SE || RI != RE) {
Suspend->dump();
Prototype->getFunctionType()->dump();
report_fatal_error("wrong number of arguments to coro.suspend.retcon");
}
}
break;
}
default:
llvm_unreachable("coro.begin is not dependent on a coro.id call");
}
// The coro.free intrinsic is always lowered to the result of coro.begin.
for (CoroFrameInst *CF : CoroFrames) {
CF->replaceAllUsesWith(CoroBegin);
CF->eraseFromParent();
}
// Canonicalize coro.suspend by inserting a coro.save if needed.
for (CoroSuspendInst *CS : CoroSuspends)
if (!CS->getCoroSave())
createCoroSave(CoroBegin, CS);
// Move final suspend to be the last element in the CoroSuspends vector.
if (HasFinalSuspend &&
if (ABI == coro::ABI::Switch &&
SwitchLowering.HasFinalSuspend &&
FinalSuspendIndex != CoroSuspends.size() - 1)
std::swap(CoroSuspends[FinalSuspendIndex], CoroSuspends.back());
@ -345,6 +436,153 @@ void coro::Shape::buildFrom(Function &F) {
CoroSave->eraseFromParent();
}
static void propagateCallAttrsFromCallee(CallInst *Call, Function *Callee) {
Call->setCallingConv(Callee->getCallingConv());
// TODO: attributes?
}
static void addCallToCallGraph(CallGraph *CG, CallInst *Call, Function *Callee){
if (CG)
(*CG)[Call->getFunction()]->addCalledFunction(Call, (*CG)[Callee]);
}
Value *coro::Shape::emitAlloc(IRBuilder<> &Builder, Value *Size,
CallGraph *CG) const {
switch (ABI) {
case coro::ABI::Switch:
llvm_unreachable("can't allocate memory in coro switch-lowering");
case coro::ABI::Retcon:
case coro::ABI::RetconOnce: {
auto Alloc = RetconLowering.Alloc;
Size = Builder.CreateIntCast(Size,
Alloc->getFunctionType()->getParamType(0),
/*is signed*/ false);
auto *Call = Builder.CreateCall(Alloc, Size);
propagateCallAttrsFromCallee(Call, Alloc);
addCallToCallGraph(CG, Call, Alloc);
return Call;
}
}
}
void coro::Shape::emitDealloc(IRBuilder<> &Builder, Value *Ptr,
CallGraph *CG) const {
switch (ABI) {
case coro::ABI::Switch:
llvm_unreachable("can't allocate memory in coro switch-lowering");
case coro::ABI::Retcon:
case coro::ABI::RetconOnce: {
auto Dealloc = RetconLowering.Dealloc;
Ptr = Builder.CreateBitCast(Ptr,
Dealloc->getFunctionType()->getParamType(0));
auto *Call = Builder.CreateCall(Dealloc, Ptr);
propagateCallAttrsFromCallee(Call, Dealloc);
addCallToCallGraph(CG, Call, Dealloc);
return;
}
}
}
LLVM_ATTRIBUTE_NORETURN
static void fail(const Instruction *I, const char *Reason, Value *V) {
I->dump();
if (V) {
errs() << " Value: ";
V->printAsOperand(llvm::errs());
errs() << '\n';
}
report_fatal_error(Reason);
}
/// Check that the given value is a well-formed prototype for the
/// llvm.coro.id.retcon.* intrinsics.
static void checkWFRetconPrototype(const AnyCoroIdRetconInst *I, Value *V) {
auto F = dyn_cast<Function>(V->stripPointerCasts());
if (!F)
fail(I, "llvm.coro.retcon.* prototype not a Function", V);
auto FT = F->getFunctionType();
if (isa<CoroIdRetconInst>(I)) {
bool ResultOkay;
if (FT->getReturnType()->isPointerTy()) {
ResultOkay = true;
} else if (auto SRetTy = dyn_cast<StructType>(FT->getReturnType())) {
ResultOkay = (!SRetTy->isOpaque() &&
SRetTy->getNumElements() > 0 &&
SRetTy->getElementType(0)->isPointerTy());
} else {
ResultOkay = false;
}
if (!ResultOkay)
fail(I, "llvm.coro.retcon prototype must return pointer as first result",
F);
if (FT->getReturnType() !=
I->getFunction()->getFunctionType()->getReturnType())
fail(I, "llvm.coro.retcon.* prototype return type must be same as"
"current function return type", F);
} else {
// No meaningful validation to do here for llvm.coro.id.unique.once.
}
if (FT->getNumParams() != 2)
fail(I, "llvm.coro.retcon.* prototype must take exactly two parameters", F);
if (!FT->getParamType(0)->isPointerTy())
fail(I, "llvm.coro.retcon.* prototype must take pointer as 1st param", F);
if (!FT->getParamType(1)->isIntegerTy()) // an i1, but not for abi purposes
fail(I, "llvm.coro.retcon.* prototype must take integer as 2nd param", F);
}
/// Check that the given value is a well-formed allocator.
static void checkWFAlloc(const Instruction *I, Value *V) {
auto F = dyn_cast<Function>(V->stripPointerCasts());
if (!F)
fail(I, "llvm.coro.* allocator not a Function", V);
auto FT = F->getFunctionType();
if (!FT->getReturnType()->isPointerTy())
fail(I, "llvm.coro.* allocator must return a pointer", F);
if (FT->getNumParams() != 1 ||
!FT->getParamType(0)->isIntegerTy())
fail(I, "llvm.coro.* allocator must take integer as only param", F);
}
/// Check that the given value is a well-formed deallocator.
static void checkWFDealloc(const Instruction *I, Value *V) {
auto F = dyn_cast<Function>(V->stripPointerCasts());
if (!F)
fail(I, "llvm.coro.* deallocator not a Function", V);
auto FT = F->getFunctionType();
if (!FT->getReturnType()->isVoidTy())
fail(I, "llvm.coro.* deallocator must return void", F);
if (FT->getNumParams() != 1 ||
!FT->getParamType(0)->isPointerTy())
fail(I, "llvm.coro.* deallocator must take pointer as only param", F);
}
static void checkConstantInt(const Instruction *I, Value *V,
const char *Reason) {
if (!isa<ConstantInt>(V)) {
fail(I, Reason, V);
}
}
void AnyCoroIdRetconInst::checkWellFormed() const {
checkConstantInt(this, getArgOperand(SizeArg),
"size argument to coro.id.retcon.* must be constant");
checkConstantInt(this, getArgOperand(AlignArg),
"alignment argument to coro.id.retcon.* must be constant");
checkWFRetconPrototype(this, getArgOperand(PrototypeArg));
checkWFAlloc(this, getArgOperand(AllocArg));
checkWFDealloc(this, getArgOperand(DeallocArg));
}
void LLVMAddCoroEarlyPass(LLVMPassManagerRef PM) {
unwrap(PM)->add(createCoroEarlyPass());
}

6
test/Transforms/Coroutines/coro-debug.ll
View File

@ -127,9 +127,9 @@ attributes #7 = { noduplicate }
!24 = !DILocation(line: 62, column: 3, scope: !6)
; CHECK: define i8* @f(i32 %x) #0 !dbg ![[ORIG:[0-9]+]]
; CHECK: define internal fastcc void @f.resume(%f.Frame* %FramePtr) #0 !dbg ![[RESUME:[0-9]+]]
; CHECK: define internal fastcc void @f.destroy(%f.Frame* %FramePtr) #0 !dbg ![[DESTROY:[0-9]+]]
; CHECK: define internal fastcc void @f.cleanup(%f.Frame* %FramePtr) #0 !dbg ![[CLEANUP:[0-9]+]]
; CHECK: define internal fastcc void @f.resume(%f.Frame* noalias nonnull %FramePtr) #0 !dbg ![[RESUME:[0-9]+]]
; CHECK: define internal fastcc void @f.destroy(%f.Frame* noalias nonnull %FramePtr) #0 !dbg ![[DESTROY:[0-9]+]]
; CHECK: define internal fastcc void @f.cleanup(%f.Frame* noalias nonnull %FramePtr) #0 !dbg ![[CLEANUP:[0-9]+]]
; CHECK: ![[ORIG]] = distinct !DISubprogram(name: "f", linkageName: "flink"
; CHECK: !DILocalVariable(name: "x", arg: 1, scope: ![[ORIG]]

116
test/Transforms/Coroutines/coro-retcon-once-value.ll Normal file
View File

@ -0,0 +1,116 @@
; RUN: opt < %s -enable-coroutines -O2 -S | FileCheck %s
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-macosx10.12.0"
define {i8*, i32} @f(i8* %buffer, i32* %array) {
entry:
%id = call token @llvm.coro.id.retcon.once(i32 8, i32 8, i8* %buffer, i8* bitcast (void (i8*, i8)* @prototype to i8*), i8* bitcast (i8* (i32)* @allocate to i8*), i8* bitcast (void (i8*)* @deallocate to i8*))
%hdl = call i8* @llvm.coro.begin(token %id, i8* null)
%load = load i32, i32* %array
%load.pos = icmp sgt i32 %load, 0
br i1 %load.pos, label %pos, label %neg
pos:
%unwind0 = call i1 (...) @llvm.coro.suspend.retcon(i32 %load)
br i1 %unwind0, label %cleanup, label %pos.cont
pos.cont:
store i32 0, i32* %array, align 4
br label %cleanup
neg:
%unwind1 = call i1 (...) @llvm.coro.suspend.retcon(i32 0)
br i1 %unwind1, label %cleanup, label %neg.cont
neg.cont:
store i32 10, i32* %array, align 4
br label %cleanup
cleanup:
call i1 @llvm.coro.end(i8* %hdl, i1 0)
unreachable
}
; CHECK-LABEL: define { i8*, i32 } @f(i8* %buffer, i32* %array)
; CHECK-NEXT: entry:
; CHECK-NEXT: [[T0:%.*]] = bitcast i8* %buffer to i32**
; CHECK-NEXT: store i32* %array, i32** [[T0]], align 8
; CHECK-NEXT: %load = load i32, i32* %array, align 4
; CHECK-NEXT: %load.pos = icmp sgt i32 %load, 0
; CHECK-NEXT: [[CONT:%.*]] = select i1 %load.pos, void (i8*, i8)* @f.resume.0, void (i8*, i8)* @f.resume.1
; CHECK-NEXT: [[VAL:%.*]] = select i1 %load.pos, i32 %load, i32 0
; CHECK-NEXT: [[CONT_CAST:%.*]] = bitcast void (i8*, i8)* [[CONT]] to i8*
; CHECK-NEXT: [[T0:%.*]] = insertvalue { i8*, i32 } undef, i8* [[CONT_CAST]], 0
; CHECK-NEXT: [[T1:%.*]] = insertvalue { i8*, i32 } [[T0]], i32 [[VAL]], 1
; CHECK-NEXT: ret { i8*, i32 } [[T1]]
; CHECK-NEXT: }
; CHECK-LABEL: define internal void @f.resume.0(i8* noalias nonnull %0, i1 zeroext %1)
; CHECK-NEXT: :
; CHECK-NEXT: [[T0:%.*]] = icmp eq i8 %1, 0
; CHECK-NEXT: br i1 [[T0]],
; CHECK: :
; CHECK-NEXT: [[T0:%.*]] = bitcast i8* %0 to i32**
; CHECK-NEXT: [[RELOAD:%.*]] = load i32*, i32** [[T0]], align 8
; CHECK-NEXT: store i32 0, i32* [[RELOAD]], align 4
; CHECK-NEXT: br label
; CHECK: :
; CHECK-NEXT: ret void
; CHECK-NEXT: }
; CHECK-LABEL: define internal void @f.resume.1(i8* noalias nonnull %0, i1 zeroext %1)
; CHECK-NEXT: :
; CHECK-NEXT: [[T0:%.*]] = icmp eq i8 %1, 0
; CHECK-NEXT: br i1 [[T0]],
; CHECK: :
; CHECK-NEXT: [[T0:%.*]] = bitcast i8* %0 to i32**
; CHECK-NEXT: [[RELOAD:%.*]] = load i32*, i32** [[T0]], align 8
; CHECK-NEXT: store i32 10, i32* [[RELOAD]], align 4
; CHECK-NEXT: br label
; CHECK: :
; CHECK-NEXT: ret void
; CHECK-NEXT: }
define void @test(i32* %array) {
entry:
%0 = alloca [8 x i8], align 8
%buffer = bitcast [8 x i8]* %0 to i8*
%prepare = call i8* @llvm.coro.prepare.retcon(i8* bitcast ({i8*, i32} (i8*, i32*)* @f to i8*))
%f = bitcast i8* %prepare to {i8*, i32} (i8*, i32*)*
%result = call {i8*, i32} %f(i8* %buffer, i32* %array)
%value = extractvalue {i8*, i32} %result, 1
call void @print(i32 %value)
%cont = extractvalue {i8*, i32} %result, 0
%cont.cast = bitcast i8* %cont to void (i8*, i8)*
call void %cont.cast(i8* %buffer, i8 zeroext 0)
ret void
}
; Unfortunately, we don't seem to fully optimize this right now due
; to some sort of phase-ordering thing.
; CHECK-LABEL: define void @test(i32* %array)
; CHECK-NEXT: entry:
; CHECK-NEXT: [[BUFFER:%.*]] = alloca i32*, align 8
; CHECK-NEXT: [[BUFFER_CAST:%.*]] = bitcast i32** [[BUFFER]] to i8*
; CHECK-NEXT: store i32* %array, i32** [[BUFFER]], align 8
; CHECK-NEXT: [[LOAD:%.*]] = load i32, i32* %array, align 4
; CHECK-NEXT: [[LOAD_POS:%.*]] = icmp sgt i32 [[LOAD]], 0
; CHECK-NEXT: [[CONT:%.*]] = select i1 [[LOAD_POS]], void (i8*, i8)* @f.resume.0, void (i8*, i8)* @f.resume.1
; CHECK-NEXT: [[VAL:%.*]] = select i1 [[LOAD_POS]], i32 [[LOAD]], i32 0
; CHECK-NEXT: call void @print(i32 [[VAL]])
; CHECK-NEXT: call void [[CONT]](i8* nonnull [[BUFFER_CAST]], i8 zeroext 0)
; CHECK-NEXT: ret void
declare token @llvm.coro.id.retcon.once(i32, i32, i8*, i8*, i8*, i8*)
declare i8* @llvm.coro.begin(token, i8*)
declare i1 @llvm.coro.suspend.retcon(...)
declare i1 @llvm.coro.end(i8*, i1)
declare i8* @llvm.coro.prepare.retcon(i8*)
declare void @prototype(i8*, i8 zeroext)
declare noalias i8* @allocate(i32 %size)
declare void @deallocate(i8* %ptr)
declare void @print(i32)

73
test/Transforms/Coroutines/coro-retcon-once-value2.ll Normal file
View File

@ -0,0 +1,73 @@
; RUN: opt < %s -coro-split -coro-cleanup -S | FileCheck %s
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-macosx10.12.0"
define {i8*, i32*} @f(i8* %buffer, i32* %ptr) "coroutine.presplit"="1" {
entry:
%temp = alloca i32, align 4
%id = call token @llvm.coro.id.retcon.once(i32 8, i32 8, i8* %buffer, i8* bitcast (void (i8*, i8)* @prototype to i8*), i8* bitcast (i8* (i32)* @allocate to i8*), i8* bitcast (void (i8*)* @deallocate to i8*))
%hdl = call i8* @llvm.coro.begin(token %id, i8* null)
%oldvalue = load i32, i32* %ptr
store i32 %oldvalue, i32* %temp
%unwind = call i1 (...) @llvm.coro.suspend.retcon(i32* %temp)
br i1 %unwind, label %cleanup, label %cont
cont:
%newvalue = load i32, i32* %temp
store i32 %newvalue, i32* %ptr
br label %cleanup
cleanup:
call i1 @llvm.coro.end(i8* %hdl, i1 0)
unreachable
}
; CHECK-LABEL: define { i8*, i32* } @f(i8* %buffer, i32* %ptr)
; CHECK-NEXT: entry:
; CHECK-NEXT: [[ALLOC:%.*]] = call i8* @allocate(i32 16)
; CHECK-NEXT: [[T0:%.*]] = bitcast i8* %buffer to i8**
; CHECK-NEXT: store i8* [[ALLOC]], i8** [[T0]]
; CHECK-NEXT: [[FRAME:%.*]] = bitcast i8* [[ALLOC]] to [[FRAME_T:%.*]]*
; CHECK-NEXT: %temp = getelementptr inbounds [[FRAME_T]], [[FRAME_T]]* [[FRAME]], i32 0, i32 1
; CHECK-NEXT: [[SPILL:%.*]] = getelementptr inbounds [[FRAME_T]], [[FRAME_T]]* [[FRAME]], i32 0, i32 0
; CHECK-NEXT: store i32* %ptr, i32** [[SPILL]]
; CHECK-NEXT: %oldvalue = load i32, i32* %ptr
; CHECK-NEXT: store i32 %oldvalue, i32* %temp
; CHECK-NEXT: [[T0:%.*]] = insertvalue { i8*, i32* } { i8* bitcast (void (i8*, i8)* @f.resume.0 to i8*), i32* undef }, i32* %temp, 1
; CHECK-NEXT: ret { i8*, i32* } [[T0]]
; CHECK-NEXT: }
; CHECK-LABEL: define internal void @f.resume.0(i8* noalias nonnull %0, i1 zeroext %1)
; CHECK-NEXT: :
; CHECK-NEXT: [[T0:%.*]] = bitcast i8* %0 to [[FRAME_T:%.*]]**
; CHECK-NEXT: [[FRAME:%.*]] = load [[FRAME_T]]*, [[FRAME_T]]** [[T0]]
; CHECK-NEXT: bitcast [[FRAME_T]]* [[FRAME]] to i8*
; CHECK-NEXT: [[T0:%.*]] = icmp ne i8 %1, 0
; CHECK-NEXT: %temp = getelementptr inbounds [[FRAME_T]], [[FRAME_T]]* [[FRAME]], i32 0, i32 1
; CHECK-NEXT: br i1 [[T0]],
; CHECK: :
; CHECK-NEXT: [[TEMP_SLOT:%.*]] = getelementptr inbounds [[FRAME_T]], [[FRAME_T]]* [[FRAME]], i32 0, i32 1
; CHECK-NEXT: [[PTR_SLOT:%.*]] = getelementptr inbounds [[FRAME_T]], [[FRAME_T]]* [[FRAME]], i32 0, i32 0
; CHECK-NEXT: [[PTR_RELOAD:%.*]] = load i32*, i32** [[PTR_SLOT]]
; CHECK-NEXT: %newvalue = load i32, i32* [[TEMP_SLOT]]
; CHECK-NEXT: store i32 %newvalue, i32* [[PTR_RELOAD]]
; CHECK-NEXT: br label
; CHECK: :
; CHECK-NEXT: [[T0:%.*]] = bitcast [[FRAME_T]]* [[FRAME]] to i8*
; CHECK-NEXT: call fastcc void @deallocate(i8* [[T0]])
; CHECK-NEXT: ret void
; CHECK-NEXT: }
declare token @llvm.coro.id.retcon.once(i32, i32, i8*, i8*, i8*, i8*)
declare i8* @llvm.coro.begin(token, i8*)
declare i1 @llvm.coro.suspend.retcon(...)
declare i1 @llvm.coro.end(i8*, i1)
declare i8* @llvm.coro.prepare.retcon(i8*)
declare void @prototype(i8*, i8 zeroext)
declare noalias i8* @allocate(i32 %size)
declare fastcc void @deallocate(i8* %ptr)
declare void @print(i32)

102
test/Transforms/Coroutines/coro-retcon-value.ll Normal file
View File

@ -0,0 +1,102 @@
; First example from Doc/Coroutines.rst (two block loop) converted to retcon
; RUN: opt < %s -enable-coroutines -O2 -S | FileCheck %s
define {i8*, i32} @f(i8* %buffer, i32 %n) {
entry:
%id = call token @llvm.coro.id.retcon(i32 8, i32 4, i8* %buffer, i8* bitcast ({i8*, i32} (i8*, i8)* @prototype to i8*), i8* bitcast (i8* (i32)* @allocate to i8*), i8* bitcast (void (i8*)* @deallocate to i8*))
%hdl = call i8* @llvm.coro.begin(token %id, i8* null)
br label %loop
loop:
%n.val = phi i32 [ %n, %entry ], [ %inc, %resume ]
%unwind0 = call i1 (...) @llvm.coro.suspend.retcon(i32 %n.val)
br i1 %unwind0, label %cleanup, label %resume
resume:
%inc = add i32 %n.val, 1
br label %loop
cleanup:
call i1 @llvm.coro.end(i8* %hdl, i1 0)
unreachable
}
; CHECK-LABEL: define { i8*, i32 } @f(i8* %buffer, i32 %n)
; CHECK-NEXT: entry:
; CHECK-NEXT: [[T0:%.*]] = bitcast i8* %buffer to i32*
; CHECK-NEXT: store i32 %n, i32* [[T0]], align 4
; CHECK-NEXT: [[RET:%.*]] = insertvalue { i8*, i32 } { i8* bitcast ({ i8*, i32 } (i8*, i8)* @f.resume.0 to i8*), i32 undef }, i32 %n, 1
; CHECK-NEXT: ret { i8*, i32 } [[RET]]
; CHECK-NEXT: }
; CHECK-LABEL: define internal { i8*, i32 } @f.resume.0(i8* noalias nonnull %0, i8 zeroext %1)
; CHECK-NEXT: :
; CHECK-NEXT: [[T0:%.*]] = icmp eq i8 %1, 0
; CHECK-NEXT: br i1 [[T0]],
; CHECK: :
; CHECK-NEXT: [[T0:%.*]] = bitcast i8* %0 to i32*
; CHECK-NEXT: [[T1:%.*]] = load i32, i32* [[T0]], align 4
; CHECK-NEXT: %inc = add i32 [[T1]], 1
; CHECK-NEXT: store i32 %inc, i32* [[T0]], align 4
; CHECK-NEXT: [[RET:%.*]] = insertvalue { i8*, i32 } { i8* bitcast ({ i8*, i32 } (i8*, i8)* @f.resume.0 to i8*), i32 undef }, i32 %inc, 1
; CHECK-NEXT: ret { i8*, i32 } [[RET]]
; CHECK: :
; CHECK-NEXT: ret { i8*, i32 } { i8* null, i32 undef }
; CHECK-NEXT: }
define i32 @main() {
entry:
%0 = alloca [8 x i8], align 4
%buffer = bitcast [8 x i8]* %0 to i8*
%prepare = call i8* @llvm.coro.prepare.retcon(i8* bitcast ({i8*, i32} (i8*, i32)* @f to i8*))
%f = bitcast i8* %prepare to {i8*, i32} (i8*, i32)*
%result0 = call {i8*, i32} %f(i8* %buffer, i32 4)
%value0 = extractvalue {i8*, i32} %result0, 1
call void @print(i32 %value0)
%cont0 = extractvalue {i8*, i32} %result0, 0
%cont0.cast = bitcast i8* %cont0 to {i8*, i32} (i8*, i8)*
%result1 = call {i8*, i32} %cont0.cast(i8* %buffer, i8 zeroext 0)
%value1 = extractvalue {i8*, i32} %result1, 1
call void @print(i32 %value1)
%cont1 = extractvalue {i8*, i32} %result1, 0
%cont1.cast = bitcast i8* %cont1 to {i8*, i32} (i8*, i8)*
%result2 = call {i8*, i32} %cont1.cast(i8* %buffer, i8 zeroext 0)
%value2 = extractvalue {i8*, i32} %result2, 1
call void @print(i32 %value2)
%cont2 = extractvalue {i8*, i32} %result2, 0
%cont2.cast = bitcast i8* %cont2 to {i8*, i32} (i8*, i8)*
call {i8*, i32} %cont2.cast(i8* %buffer, i8 zeroext 1)
ret i32 0
}
; Unfortunately, we don't seem to fully optimize this right now due
; to some sort of phase-ordering thing.
; CHECK-LABEL: define i32 @main
; CHECK-NEXT: entry:
; CHECK: [[BUFFER:%.*]] = alloca [8 x i8], align 4
; CHECK: [[SLOT:%.*]] = bitcast [8 x i8]* [[BUFFER]] to i32*
; CHECK-NEXT: store i32 4, i32* [[SLOT]], align 4
; CHECK-NEXT: call void @print(i32 4)
; CHECK-NEXT: [[LOAD:%.*]] = load i32, i32* [[SLOT]], align 4
; CHECK-NEXT: [[INC:%.*]] = add i32 [[LOAD]], 1
; CHECK-NEXT: store i32 [[INC]], i32* [[SLOT]], align 4
; CHECK-NEXT: call void @print(i32 [[INC]])
; CHECK-NEXT: [[LOAD:%.*]] = load i32, i32* [[SLOT]], align 4
; CHECK-NEXT: [[INC:%.*]] = add i32 [[LOAD]], 1
; CHECK-NEXT: store i32 [[INC]], i32* [[SLOT]], align 4
; CHECK-NEXT: call void @print(i32 [[INC]])
; CHECK-NEXT: ret i32 0
declare token @llvm.coro.id.retcon(i32, i32, i8*, i8*, i8*, i8*)
declare i8* @llvm.coro.begin(token, i8*)
declare i1 @llvm.coro.suspend.retcon(...)
declare i1 @llvm.coro.end(i8*, i1)
declare i8* @llvm.coro.prepare.retcon(i8*)
declare {i8*, i32} @prototype(i8*, i8 zeroext)
declare noalias i8* @allocate(i32 %size)
declare void @deallocate(i8* %ptr)
declare void @print(i32)

94
test/Transforms/Coroutines/coro-retcon.ll Normal file
View File

@ -0,0 +1,94 @@
; First example from Doc/Coroutines.rst (two block loop) converted to retcon
; RUN: opt < %s -enable-coroutines -O2 -S | FileCheck %s
define i8* @f(i8* %buffer, i32 %n) {
entry:
%id = call token @llvm.coro.id.retcon(i32 8, i32 4, i8* %buffer, i8* bitcast (i8* (i8*, i8)* @prototype to i8*), i8* bitcast (i8* (i32)* @allocate to i8*), i8* bitcast (void (i8*)* @deallocate to i8*))
%hdl = call i8* @llvm.coro.begin(token %id, i8* null)
br label %loop
loop:
%n.val = phi i32 [ %n, %entry ], [ %inc, %resume ]
call void @print(i32 %n.val)
%unwind0 = call i1 (...) @llvm.coro.suspend.retcon()
br i1 %unwind0, label %cleanup, label %resume
resume:
%inc = add i32 %n.val, 1
br label %loop
cleanup:
call i1 @llvm.coro.end(i8* %hdl, i1 0)
unreachable
}
; CHECK-LABEL: define i8* @f(i8* %buffer, i32 %n)
; CHECK-NEXT: entry:
; CHECK-NEXT: [[T0:%.*]] = bitcast i8* %buffer to i32*
; CHECK-NEXT: store i32 %n, i32* [[T0]], align 4
; CHECK-NEXT: call void @print(i32 %n)
; CHECK-NEXT: ret i8* bitcast (i8* (i8*, i8)* @f.resume.0 to i8*)
; CHECK-NEXT: }
; CHECK-LABEL: define internal i8* @f.resume.0(i8* noalias nonnull %0, i1 zeroext %1)
; CHECK-NEXT: :
; CHECK-NEXT: [[T0:%.*]] = icmp eq i8 %1, 0
; CHECK-NEXT: br i1 [[T0]],
; CHECK: :
; CHECK-NEXT: [[T0:%.*]] = bitcast i8* %0 to i32*
; CHECK-NEXT: [[T1:%.*]] = load i32, i32* [[T0]], align 4
; CHECK-NEXT: %inc = add i32 [[T1]], 1
; CHECK-NEXT: store i32 %inc, i32* [[T0]], align 4
; CHECK-NEXT: call void @print(i32 %inc)
; CHECK-NEXT: ret i8* bitcast (i8* (i8*, i8)* @f.resume.0 to i8*)
; CHECK: :
; CHECK-NEXT: ret i8* null
; CHECK-NEXT: }
define i32 @main() {
entry:
%0 = alloca [8 x i8], align 4
%buffer = bitcast [8 x i8]* %0 to i8*
%prepare = call i8* @llvm.coro.prepare.retcon(i8* bitcast (i8* (i8*, i32)* @f to i8*))
%f = bitcast i8* %prepare to i8* (i8*, i32)*
%cont0 = call i8* %f(i8* %buffer, i32 4)
%cont0.cast = bitcast i8* %cont0 to i8* (i8*, i8)*
%cont1 = call i8* %cont0.cast(i8* %buffer, i8 zeroext 0)
%cont1.cast = bitcast i8* %cont1 to i8* (i8*, i8)*
%cont2 = call i8* %cont1.cast(i8* %buffer, i8 zeroext 0)
%cont2.cast = bitcast i8* %cont2 to i8* (i8*, i8)*
call i8* %cont2.cast(i8* %buffer, i8 zeroext 1)
ret i32 0
}
; Unfortunately, we don't seem to fully optimize this right now due
; to some sort of phase-ordering thing.
; CHECK-LABEL: define i32 @main
; CHECK-NEXT: entry:
; CHECK: [[BUFFER:%.*]] = alloca [8 x i8], align 4
; CHECK: [[SLOT:%.*]] = bitcast [8 x i8]* [[BUFFER]] to i32*
; CHECK-NEXT: store i32 4, i32* [[SLOT]], align 4
; CHECK-NEXT: call void @print(i32 4)
; CHECK-NEXT: [[LOAD:%.*]] = load i32, i32* [[SLOT]], align 4
; CHECK-NEXT: [[INC:%.*]] = add i32 [[LOAD]], 1
; CHECK-NEXT: store i32 [[INC]], i32* [[SLOT]], align 4
; CHECK-NEXT: call void @print(i32 [[INC]])
; CHECK-NEXT: [[LOAD:%.*]] = load i32, i32* [[SLOT]], align 4
; CHECK-NEXT: [[INC:%.*]] = add i32 [[LOAD]], 1
; CHECK-NEXT: store i32 [[INC]], i32* [[SLOT]], align 4
; CHECK-NEXT: call void @print(i32 [[INC]])
; CHECK-NEXT: ret i32 0
declare token @llvm.coro.id.retcon(i32, i32, i8*, i8*, i8*, i8*)
declare i8* @llvm.coro.begin(token, i8*)
declare i1 @llvm.coro.suspend.retcon(...)
declare i1 @llvm.coro.end(i8*, i1)
declare i8* @llvm.coro.prepare.retcon(i8*)
declare i8* @prototype(i8*, i8 zeroext)
declare noalias i8* @allocate(i32 %size)
declare void @deallocate(i8* %ptr)
declare void @print(i32)