RPCS3/llvm-mirror
1
0
Fork 0
mirror of https://github.com/RPCS3/llvm-mirror.git synced 2024-10-18 10:32:48 +02:00
Code Issues Projects Releases Wiki Activity

Added Dockerfiles to build clang from sources.

Browse Source 
Reviewers: klimek, chandlerc, mehdi_amini

Reviewed By: klimek, mehdi_amini

Subscribers: mehdi_amini, jlebar, llvm-commits

Differential Revision: https://reviews.llvm.org/D34197

llvm-svn: 306810
This commit is contained in:
Ilya Biryukov 2017-06-30 09:46:45 +00:00
parent 2d3024ee86
commit dd70425800
11 changed files with 654 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

205
docs/Docker.rst Normal file
View File

@ -0,0 +1,205 @@
=========================================
A guide to Dockerfiles for building LLVM
=========================================
Introduction
============
You can find a number of sources to build docker images with LLVM components in
``llvm/utils/docker``. They can be used by anyone who wants to build the docker
images for their own use, or as a starting point for someone who wants to write
their own Dockerfiles.
We currently provide Dockerfiles with ``debian8`` and ``nvidia-cuda`` base images.
We also provide an ``example`` image, which contains placeholders that one would need
to fill out in order to produce Dockerfiles for a new docker image.
Why?
----
Docker images provide a way to produce binary distributions of
software inside a controlled environment. Having Dockerfiles to builds docker images
inside LLVM repo makes them much more discoverable than putting them into any other
place.
Docker basics
-------------
If you've never heard about Docker before, you might find this section helpful
to get a very basic explanation of it.
`Docker <https://www.docker.com/>`_ is a popular solution for running programs in
an isolated and reproducible environment, especially to maintain releases for
software deployed to large distributed fleets.
It uses linux kernel namespaces and cgroups to provide a lightweight isolation
inside currently running linux kernel.
A single active instance of dockerized environment is called a *docker
container*.
A snapshot of a docker container filesystem is called a *docker image*.
One can start a container from a prebuilt docker image.
Docker images are built from a so-called *Dockerfile*, a source file written in
a specialized language that defines instructions to be used when build
the docker image (see `official
documentation <https://docs.docker.com/engine/reference/builder/>`_ for more
details). A minimal Dockerfile typically contains a base image and a number
of RUN commands that have to be executed to build the image. When building a new
image, docker will first download your base image, mount its filesystem as
read-only and then add a writable overlay on top of it to keep track of all
filesystem modifications, performed while building your image. When the build
process is finished, a diff between your image's final filesystem state and the
base image's filesystem is stored in the resulting image.
Overview
========
The ``llvm/utils/docker`` folder contains Dockerfiles and simple bash scripts to
serve as a basis for anyone who wants to create their own Docker image with
LLVM components, compiled from sources. The sources are checked out from the
upstream svn repository when building the image.
Inside each subfolder we host Dockerfiles for two images:
- ``build/`` image is used to compile LLVM, it installs a system compiler and all
build dependencies of LLVM. After the build process is finished, the build
image will have an archive with compiled components at ``/tmp/clang.tar.gz``.
- ``release/`` image usually only contains LLVM components, compiled by the
``build/`` image, and also libstdc++ and binutils to make image minimally
useful for C++ development. The assumption is that you usually want clang to
be one of the provided components.
To build both of those images, use ``build_docker_image.sh`` script.
It will checkout LLVM sources and build clang in the ``build`` container, copy results
of the build to the local filesystem and then build the ``release`` container using
those. The ``build_docker_image.sh`` accepts a list of LLVM repositories to
checkout, and arguments for CMake invocation.
If you want to write your own docker image, start with an ``example/`` subfolder.
It provides incomplete Dockerfiles with (very few) FIXMEs explaining the steps
you need to take in order to make your Dockerfiles functional.
Usage
=====
The ``llvm/utils/build_docker_image.sh`` script provides a rather high degree of
control on how to run the build. It allows you to specify the projects to
checkout from svn and provide a list of CMake arguments to use during when
building LLVM inside docker container.
Here's a very simple example of getting a docker image with clang binary,
compiled by the system compiler in the debian8 image:
.. code-block:: bash
./llvm/utils/docker/build_docker_image.sh \
--source debian8 \
--docker-repository clang-debian8 --docker-tag "staging" \
-- \
-p clang -i install-clang -i install-clang-headers \
-- \
-DCMAKE_BUILD_TYPE=Release
Note there are two levels of ``--`` indirection. First one separates
``build_docker_image.sh`` arguments from ``llvm/utils/build_install_llvm.sh``
arguments. Second one separates CMake arguments from ``build_install_llvm.sh``
arguments. Note that build like that doesn't use a 2-stage build process that
you probably want for clang. Running a 2-stage build is a little more intricate,
this command will do that:
.. code-block:: bash
# Run a 2-stage build.
# LLVM_TARGETS_TO_BUILD=Native is to reduce stage1 compile time.
# Options, starting with BOOTSTRAP_* are passed to stage2 cmake invocation.
./build_docker_image.sh \
--source debian8 \
--docker-repository clang-debian8 --docker-tag "staging" \
-- \
-p clang -i stage2-install-clang -i stage2-install-clang-headers \
-- \
-DLLVM_TARGETS_TO_BUILD=Native -DCMAKE_BUILD_TYPE=Release \
-DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \
-DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-headers"
This will produce two images, a release image ``clang-debian8:staging`` and a
build image ``clang-debian8-build:staging`` from the latest upstream revision.
After the image is built you can run bash inside a container based on your
image like this:
.. code-block:: bash
docker run -ti clang-debian8:staging bash
Now you can run bash commands as you normally would:
.. code-block:: bash
root@80f351b51825:/# clang -v
clang version 5.0.0 (trunk 305064)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.2
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
Candidate multilib: .;@m64
Selected multilib: .;@m64
Which image should I choose?
============================
We currently provide two images: debian8-based and nvidia-cuda-based. They
differ in the base image that they use, i.e. they have a different set of
preinstalled binaries. Debian8 is very minimal, nvidia-cuda is larger, but has
preinstalled CUDA libraries and allows to access a GPU, installed on your
machine.
If you need a minimal linux distribution with only clang and libstdc++ included,
you should try debian8-based image.
If you want to use CUDA libraries and have access to a GPU on your machine,
you should choose nvidia-cuda-based image and use `nvidia-docker
<https://github.com/NVIDIA/nvidia-docker>`_ to run your docker containers. Note
that you don't need nvidia-docker to build the images, but you need it in order
to have an access to GPU from a docker container that is running the built
image.
If you have a different use-case, you could create your own image based on
``example/`` folder.
Any docker image can be built and run using only the docker binary, i.e. you can
run debian8 build on Fedora or any other Linux distribution. You don't need to
install CMake, compilers or any other clang dependencies. It is all handled
during the build process inside Docker's isolated environment.
Stable build
============
If you want a somewhat recent and somewhat stable build, use the
``branches/google/stable`` branch, i.e. the following command will produce a
debian8-based image using the latest ``google/stable`` sources for you:
.. code-block:: bash
./llvm/utils/docker/build_docker_image.sh \
-s debian8 --d clang-debian8 -t "staging" \
-- \
--branch branches/google/stable \
-p clang -i install-clang -i install-clang-headers \
-- \
-DCMAKE_BUILD_TYPE=Release
Minimizing docker image size
============================
Due to Docker restrictions we use two images (i.e., build and release folders)
for the release image to be as small as possible. It's much easier to achieve
that using two images, because Docker would store a filesystem layer for each
command in the Dockerfile, i.e. if you install some packages in one command,
then remove those in a separate command, the size of the resulting image will
still be proportinal to the size of an image with installed packages.
Therefore, we strive to provide a very simple release image which only copies
compiled clang and does not do anything else.
Docker 1.13 added a ``--squash`` flag that allows to flatten the layers of the
image, i.e. remove the parts that were actually deleted. That is an easier way
to produce the smallest images possible by using just a single image. We do not
use it because as of today the flag is in experimental stage and not everyone
may have the latest docker version available. When the flag is out of
experimental stage, we should investigate replacing two images approach with
just a single image, built using ``--squash`` flag.

4
docs/index.rst
View File

@ -91,6 +91,7 @@ representation.
CompileCudaWithLLVM
ReportingGuide
Benchmarking
Docker
:doc:`GettingStarted`
Discusses how to get up and running quickly with the LLVM infrastructure.
@ -161,6 +162,9 @@ representation.
A collection of tips for frontend authors on how to generate IR
which LLVM is able to effectively optimize.
:doc:`Docker`
A reference for using Dockerfiles provided with LLVM.
Programming Documentation
=========================

1
utils/docker/README Normal file
View File

@ -0,0 +1 @@
See llvm/docs/Docker.rst for details

121
utils/docker/build_docker_image.sh Executable file
View File

@ -0,0 +1,121 @@
#!/bin/bash
#===- llvm/utils/docker/build_docker_image.sh ----------------------------===//
#
# The LLVM Compiler Infrastructure
#
# This file is distributed under the University of Illinois Open Source
# License. See LICENSE.TXT for details.
#
#===----------------------------------------------------------------------===//
set -e
IMAGE_SOURCE=""
DOCKER_REPOSITORY=""
DOCKER_TAG=""
BUILDSCRIPT_ARGS=""
function show_usage() {
usage=$(cat << EOF
Usage: build_docker_image.sh [options] [-- [buildscript_args]...]
Available options:
-s|--source image source dir (i.e. debian8, nvidia-cuda, etc)
-d|--docker-repository docker repository for the image
-t|--docker-tag docker tag for the image
Required options: --source and --docker-repository.
All options after '--' are passed to buildscript (see
scripts/build_install_llvm.sh).
For example, running:
$ build_docker_image.sh -s debian8 -d mydocker/debian8-clang -t latest \
-- -p clang -i install-clang -i install-clang-headers
will produce two docker images:
mydocker/debian8-clang-build:latest - an intermediate image used to compile
clang.
mydocker/clang-debian8:latest - a small image with preinstalled clang.
Please note that this example produces a not very useful installation, since it
doesn't override CMake defaults, which produces a Debug and non-boostrapped
version of clang.
For an example of a somewhat more useful build, see build_clang_image.sh.
EOF
)
echo "$usage"
}
while [[ $# -gt 0 ]]; do
case "$1" in
-h|--help)
show_usage
exit 0
;;
-s|--source)
shift
IMAGE_SOURCE="$1"
shift
;;
-d|--docker-repository)
shift
DOCKER_REPOSITORY="$1"
shift
;;
-t|--docker-tag)
shift
DOCKER_TAG="$1"
shift
;;
--)
shift
BUILDSCRIPT_ARGS="$*"
shift $#
;;
*)
echo "Unknown argument $1"
exit 1
;;
esac
done
command -v docker >/dev/null ||
{
echo "Docker binary cannot be found. Please install Docker to use this script."
exit 1
}
if [ "$IMAGE_SOURCE" == "" ]; then
echo "Required argument missing: --source"
exit 1
fi
if [ "$DOCKER_REPOSITORY" == "" ]; then
echo "Required argument missing: --docker-repository"
exit 1
fi
cd $(dirname $0)
if [ ! -d $IMAGE_SOURCE ]; then
echo "No sources for '$IMAGE_SOURCE' were found in $PWD"
exit 1
fi
echo "Building from $IMAGE_SOURCE"
if [ "$DOCKER_TAG" != "" ]; then
DOCKER_TAG=":$DOCKER_TAG"
fi
echo "Building $DOCKER_REPOSITORY-build$DOCKER_TAG"
docker build -t "$DOCKER_REPOSITORY-build$DOCKER_TAG" \
--build-arg "buildscript_args=$BUILDSCRIPT_ARGS" \
-f "$IMAGE_SOURCE/build/Dockerfile" .
echo "Copying clang installation to release image sources"
docker run -v "$PWD/$IMAGE_SOURCE:/workspace" "$DOCKER_REPOSITORY-build$DOCKER_TAG" \
cp /tmp/clang.tar.gz /workspace/release
trap "rm -f $PWD/$IMAGE_SOURCE/release/clang.tar.gz" EXIT
echo "Building release image"
docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \
"$IMAGE_SOURCE/release"
echo "Done"

35
utils/docker/debian8/build/Dockerfile Normal file
View File

@ -0,0 +1,35 @@
#===- llvm/utils/docker/debian8/build/Dockerfile -------------------------===//
#
# The LLVM Compiler Infrastructure
#
# This file is distributed under the University of Illinois Open Source
# License. See LICENSE.TXT for details.
#
#===----------------------------------------------------------------------===//
# Produces an image that compiles and archives clang, based on debian8.
FROM launcher.gcr.io/google/debian8:latest
LABEL maintainer "LLVM Developers"
# Install build dependencies of llvm.
# First, Update the apt's source list and include the sources of the packages.
RUN grep deb /etc/apt/sources.list | \
sed 's/^deb/deb-src /g' >> /etc/apt/sources.list
# Install compiler, python and subversion.
RUN apt-get update && \
apt-get install -y --no-install-recommends build-essential python2.7 wget \
subversion ninja-build && \
rm -rf /var/lib/apt/lists/*
# Install cmake version that can compile clang into /usr/local.
# (Version in debian8 repos is is too old)
RUN wget -O - "https://cmake.org/files/v3.7/cmake-3.7.2-Linux-x86_64.tar.gz" | \
tar xzf - -C /usr/local --strip-components=1
# Arguments passed to build_install_clang.sh.
ARG buildscript_args
# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
ADD scripts/build_install_llvm.sh /tmp
RUN /tmp/build_install_llvm.sh ${buildscript_args}

21
utils/docker/debian8/release/Dockerfile Normal file
View File

@ -0,0 +1,21 @@
#===- llvm/utils/docker/debian8/release/Dockerfile -----------------------===//
#
# The LLVM Compiler Infrastructure
#
# This file is distributed under the University of Illinois Open Source
# License. See LICENSE.TXT for details.
#
#===----------------------------------------------------------------------===//
# A release image, containing clang installation, produced by the 'build/' image
# and adding libstdc++ and binutils.
FROM launcher.gcr.io/google/debian8:latest
LABEL maintainer "LLVM Developers"
# Install packages for minimal usefull image.
RUN apt-get update && \
apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
rm -rf /var/lib/apt/lists/*
# Unpack clang installation into this image.
ADD clang.tar.gz /

26
utils/docker/example/build/Dockerfile Normal file
View File

@ -0,0 +1,26 @@
#===- llvm/utils/docker/example/build/Dockerfile -------------------------===//
#
# The LLVM Compiler Infrastructure
#
# This file is distributed under the University of Illinois Open Source
# License. See LICENSE.TXT for details.
#
#===----------------------------------------------------------------------===//
# This is an example Dockerfile to build an image that compiles clang.
# Replace FIXMEs to prepare your own image.
# FIXME: Replace 'ubuntu' with your base image
FROM ubuntu
# FIXME: Change maintainer name
LABEL maintainer "Maintainer <maintainer@email>"
# FIXME: Install llvm/clang build dependencies. Including compiler to
# build stage1, cmake, subversion, ninja, etc.
# Arguments to pass to build_install_clang.sh.
ARG buildscript_args
# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
ADD scripts/build_install_llvm.sh /tmp
RUN /tmp/build_install_llvm.sh ${buildscript_args}

24
utils/docker/example/release/Dockerfile Normal file
View File

@ -0,0 +1,24 @@
#===- llvm/utils/docker/example/release/Dockerfile -----------------------===//
#
# The LLVM Compiler Infrastructure
#
# This file is distributed under the University of Illinois Open Source
# License. See LICENSE.TXT for details.
#
#===----------------------------------------------------------------------===//
# An image that unpacks a clang installation, compiled by the 'build/'
# container.
# Replace FIXMEs to prepare your own image.
# FIXME: Replace 'ubuntu' with your base image.
FROM ubuntu
# FIXME: Change maintainer name.
LABEL maintainer "Maintainer <maintainer@email>"
# FIXME: Install all packages you want to have in your release container.
# A minimal usefull installation must include libstdc++ and binutils.
# Unpack clang installation into this container.
# It is copied to this directory by build_docker_image.sh script.
ADD clang.tar.gz /

25
utils/docker/nvidia-cuda/build/Dockerfile Normal file
View File

@ -0,0 +1,25 @@
#===- llvm/utils/docker/nvidia-cuda/build/Dockerfile ---------------------===//
#
# The LLVM Compiler Infrastructure
#
# This file is distributed under the University of Illinois Open Source
# License. See LICENSE.TXT for details.
#
#===----------------------------------------------------------------------===//
# Produces an image that compiles and archives clang, based on nvidia/cuda
# image.
FROM nvidia/cuda:8.0-devel
LABEL maintainer "LLVM Developers"
# Arguments to pass to build_install_clang.sh.
ARG buildscript_args
# Install llvm build dependencies.
RUN apt-get update && \
apt-get install -y --no-install-recommends cmake python2.7 subversion ninja-build && \
rm -rf /var/lib/apt/lists/*
# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
ADD scripts/build_install_llvm.sh /tmp
RUN /tmp/build_install_llvm.sh ${buildscript_args}

23
utils/docker/nvidia-cuda/release/Dockerfile Normal file
View File

@ -0,0 +1,23 @@
#===- llvm/utils/docker/nvidia-cuda/release/Dockerfile -------------------===//
#
# The LLVM Compiler Infrastructure
#
# This file is distributed under the University of Illinois Open Source
# License. See LICENSE.TXT for details.
#
#===----------------------------------------------------------------------===//
# This is an example Dockerfile that copies a clang installation, compiled
# by the 'build/' container into a fresh docker image to get a container of
# minimal size.
# Replace FIXMEs to prepare a new Dockerfile.
# FIXME: Replace 'ubuntu' with your base image.
FROM nvidia/cuda:8.0-devel
# FIXME: Change maintainer name.
LABEL maintainer "LLVM Developers"
# Unpack clang installation into this container.
ADD clang.tar.gz /
# C++ standard library and binutils are already included in the base package.

169
utils/docker/scripts/build_install_llvm.sh Executable file
View File

@ -0,0 +1,169 @@
#!/usr/bin/env bash
#===- llvm/utils/docker/scripts/build_install_llvm.sh ---------------------===//
#
# The LLVM Compiler Infrastructure
#
# This file is distributed under the University of Illinois Open Source
# License. See LICENSE.TXT for details.
#
#===-----------------------------------------------------------------------===//
set -e
function show_usage() {
usage=$(cat << EOF
Usage: build_install_llvm.sh [options] -- [cmake-args]
Checkout svn sources and run cmake with the specified arguments. Used
inside docker container.
Passes additional -DCMAKE_INSTALL_PREFIX and archives the contents of
the directory to /tmp/clang.tar.gz.
Available options:
-h|--help show this help message
-b|--branch svn branch to checkout, i.e. 'trunk',
'branches/release_40'
(default: 'trunk')
-r|--revision svn revision to checkout
-p|--llvm-project name of an svn project to checkout. Will also add the
project to a list LLVM_ENABLE_PROJECTS, passed to CMake.
For clang, please use 'clang', not 'cfe'.
Project 'llvm' is always included and ignored, if
specified.
Can be specified multiple times.
-i|--install-target name of a cmake install target to build and include in
the resulting archive. Can be specified multiple times.
Required options: At least one --install-target.
All options after '--' are passed to CMake invocation.
EOF
)
echo "$usage"
}
LLVM_SVN_REV=""
LLVM_BRANCH=""
CMAKE_ARGS=""
CMAKE_INSTALL_TARGETS=""
# We always checkout llvm
LLVM_PROJECTS="llvm"
CMAKE_LLVM_ENABLE_PROJECTS=""
function contains_project() {
local TARGET_PROJ="$1"
local PROJ
for PROJ in $LLVM_PROJECTS; do
if [ "$PROJ" == "$TARGET_PROJ" ]; then
return 0
fi
done
return 1
}
while [[ $# -gt 0 ]]; do
case "$1" in
-r|--revision)
shift
LLVM_SVN_REV="$1"
;;
-b|--branch)
shift
LLVM_BRANCH="$1"
shift
;;
-p|--llvm-project)
shift
PROJ="$1"
if [ "$PROJ" == "cfe" ]; then
PROJ="clang"
fi
if ! contains_project "$PROJ" ; then
LLVM_PROJECTS="$LLVM_PROJECTS $PROJ"
CMAKE_LLVM_ENABLE_PROJECTS="$CMAKE_LLVM_ENABLED_PROJECTS;$PROJ"
else
echo "Project '$PROJ' is already enabled, ignoring extra occurences."
fi
shift
;;
-i|--install-target)
shift
CMAKE_INSTALL_TARGETS="$CMAKE_INSTALL_TARGETS $1"
shift
;;
--)
shift
CMAKE_ARGS="$*"
shift $#
;;
-h|--help)
show_usage
exit 0
;;
*)
echo "Unknown option: $1"
exit 1
esac
done
if [ "$CMAKE_INSTALL_TARGETS" == "" ]; then
echo "No install targets. Please pass one or more --install-target."
exit 1
fi
if [ "$LLVM_BRANCH" == "" ]; then
LLVM_BRANCH="trunk"
fi
if [ "$LLVM_SVN_REVISION" != "" ]; then
SVN_REV_ARG="-r$LLVM_SVN_REVISION"
else
SVN_REV_ARG=""
fi
CLANG_BUILD_DIR=/tmp/clang-build
CLANG_INSTALL_DIR=/tmp/clang-install
mkdir "$CLANG_BUILD_DIR"
# Get the sources from svn.
echo "Checking out sources from svn"
mkdir "$CLANG_BUILD_DIR/src"
for LLVM_PROJECT in $LLVM_PROJECTS; do
if [ "$LLVM_PROJECT" == "clang" ]; then
SVN_PROJECT="cfe"
else
SVN_PROJECT="$LLVM_PROJECT"
fi
echo "Checking out http://llvm.org/svn/llvm-project/$SVN_PROJECT to $CLANG_BUILD_DIR/src/$LLVM_PROJECT"
# FIXME: --trust-server-cert is required to workaround 'SSL issuer is not
# trusted' error. Using https seems preferable to http either way,
# albeit this is not secure.
svn co -q $SVN_REV_ARG --trust-server-cert \
"https://llvm.org/svn/llvm-project/$SVN_PROJECT/$LLVM_BRANCH" \
"$CLANG_BUILD_DIR/src/$LLVM_PROJECT"
done
pushd "$CLANG_BUILD_DIR"
# Run the build as specified in the build arguments.
echo "Running build"
mkdir "$CLANG_BUILD_DIR/build"
cmake -GNinja \
-DCMAKE_INSTALL_PREFIX="$CLANG_INSTALL_DIR" \
-DLLVM_ENABLE_PROJECTS="$CMAKE_LLVM_ENABLE_PROJECTS" \
$CMAKE_ARGS \
"$CLANG_BUILD_DIR/src/llvm"
ninja $CMAKE_INSTALL_TARGETS
popd
# Pack the installed clang into an archive.
echo "Archiving clang installation to /tmp/clang.tar.gz"
cd "$CLANG_INSTALL_DIR"
tar -czf /tmp/clang.tar.gz *
# Cleanup.
rm -rf "$CLANG_BUILD_DIR" "$CLANG_INSTALL_DIR"
echo "Done"