mirror of
https://github.com/RPCS3/llvm-mirror.git
synced 2024-11-22 02:33:06 +01:00
Migrate dockerfiles to use multi-stage builds.
Summary: We previously emulated multi-staged builds using two dockerfiles, native support from Docker allows us to merge them into one, simplifying our scripts. For more details about multi-stage builds, see: https://docs.docker.com/develop/develop-images/multistage-build/ Reviewers: mehdi_amini, klimek, sammccall Reviewed By: sammccall Subscribers: llvm-commits, ioeric, cfe-commits Differential Revision: https://reviews.llvm.org/D44787 llvm-svn: 328503
This commit is contained in:
Ilya Biryukov
parent
242350214f
commit
0d541f9030
@ -53,24 +53,15 @@ serve as a basis for anyone who wants to create their own Docker image with
|
||||
LLVM components, compiled from sources. The sources are checked out from the
|
||||
upstream svn repository when building the image.
|
||||
|
||||
Inside each subfolder we host Dockerfiles for two images:
|
||||
The resulting image contains only the requested LLVM components and a few extra
|
||||
packages to make the image minimally useful for C++ development, e.g. libstdc++
|
||||
and binutils.
|
||||
|
||||
- ``build/`` image is used to compile LLVM, it installs a system compiler and all
|
||||
build dependencies of LLVM. After the build process is finished, the build
|
||||
image will have an archive with compiled components at ``/tmp/clang.tar.gz``.
|
||||
- ``release/`` image usually only contains LLVM components, compiled by the
|
||||
``build/`` image, and also libstdc++ and binutils to make image minimally
|
||||
useful for C++ development. The assumption is that you usually want clang to
|
||||
be one of the provided components.
|
||||
|
||||
To build both of those images, use ``build_docker_image.sh`` script.
|
||||
It will checkout LLVM sources and build clang in the ``build`` container, copy results
|
||||
of the build to the local filesystem and then build the ``release`` container using
|
||||
those. The ``build_docker_image.sh`` accepts a list of LLVM repositories to
|
||||
checkout, and arguments for CMake invocation.
|
||||
The interface to run the build is ``build_docker_image.sh`` script. It accepts a
|
||||
list of LLVM repositories to checkout and arguments for CMake invocation.
|
||||
|
||||
If you want to write your own docker image, start with an ``example/`` subfolder.
|
||||
It provides incomplete Dockerfiles with (very few) FIXMEs explaining the steps
|
||||
It provides an incomplete Dockerfile with (very few) FIXMEs explaining the steps
|
||||
you need to take in order to make your Dockerfiles functional.
|
||||
|
||||
Usage
|
||||
@ -110,10 +101,10 @@ this command will do that:
|
||||
-DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \
|
||||
-DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-headers"
|
||||
|
||||
This will produce two images, a release image ``clang-debian8:staging`` and a
|
||||
build image ``clang-debian8-build:staging`` from the latest upstream revision.
|
||||
After the image is built you can run bash inside a container based on your
|
||||
image like this:
|
||||
This will produce a new image ``clang-debian8:staging`` from the latest
|
||||
upstream revision.
|
||||
After the image is built you can run bash inside a container based on your image
|
||||
like this:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
@ -181,19 +172,14 @@ debian8-based image using the latest ``google/stable`` sources for you:
|
||||
|
||||
Minimizing docker image size
|
||||
============================
|
||||
Due to Docker restrictions we use two images (i.e., build and release folders)
|
||||
for the release image to be as small as possible. It's much easier to achieve
|
||||
that using two images, because Docker would store a filesystem layer for each
|
||||
command in the Dockerfile, i.e. if you install some packages in one command,
|
||||
then remove those in a separate command, the size of the resulting image will
|
||||
still be proportinal to the size of an image with installed packages.
|
||||
Therefore, we strive to provide a very simple release image which only copies
|
||||
compiled clang and does not do anything else.
|
||||
|
||||
Docker 1.13 added a ``--squash`` flag that allows to flatten the layers of the
|
||||
image, i.e. remove the parts that were actually deleted. That is an easier way
|
||||
to produce the smallest images possible by using just a single image. We do not
|
||||
use it because as of today the flag is in experimental stage and not everyone
|
||||
may have the latest docker version available. When the flag is out of
|
||||
experimental stage, we should investigate replacing two images approach with
|
||||
just a single image, built using ``--squash`` flag.
|
||||
Due to how Docker's filesystem works, all intermediate writes are persisted in
|
||||
the resulting image, even if they are removed in the following commands.
|
||||
To minimize the resulting image size we use `multi-stage Docker builds
|
||||
<https://docs.docker.com/develop/develop-images/multistage-build/>`_.
|
||||
Internally Docker builds two images. The first image does all the work: installs
|
||||
build dependencies, checks out LLVM source code, compiles LLVM, etc.
|
||||
The first image is only used during build and does not have a descriptive name,
|
||||
i.e. it is only accessible via the hash value after the build is finished.
|
||||
The second image is our resulting image. It contains only the built binaries
|
||||
and not any build dependencies. It is also accessible via a descriptive name
|
||||
(specified by -d and -t flags).
|
||||
|
||||
@ -163,19 +163,9 @@ if [ "$DOCKER_TAG" != "" ]; then
|
||||
DOCKER_TAG=":$DOCKER_TAG"
|
||||
fi
|
||||
|
||||
echo "Building from $IMAGE_SOURCE"
|
||||
echo "Building $DOCKER_REPOSITORY-build$DOCKER_TAG"
|
||||
docker build -t "$DOCKER_REPOSITORY-build$DOCKER_TAG" \
|
||||
--build-arg "buildscript_args=$BUILDSCRIPT_ARGS" \
|
||||
-f "$BUILD_DIR/$IMAGE_SOURCE/build/Dockerfile" \
|
||||
"$BUILD_DIR"
|
||||
|
||||
echo "Copying clang installation to release image sources"
|
||||
docker run -v "$BUILD_DIR/$IMAGE_SOURCE:/workspace" "$DOCKER_REPOSITORY-build$DOCKER_TAG" \
|
||||
cp /tmp/clang.tar.gz /workspace/release
|
||||
|
||||
echo "Building release image"
|
||||
echo "Building ${DOCKER_REPOSITORY}${DOCKER_TAG} from $IMAGE_SOURCE"
|
||||
docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \
|
||||
"$BUILD_DIR/$IMAGE_SOURCE/release"
|
||||
|
||||
--build-arg "buildscript_args=$BUILDSCRIPT_ARGS" \
|
||||
-f "$BUILD_DIR/$IMAGE_SOURCE/Dockerfile" \
|
||||
"$BUILD_DIR"
|
||||
echo "Done"
|
||||
|
||||
@ -6,22 +6,18 @@
|
||||
# License. See LICENSE.TXT for details.
|
||||
#
|
||||
#===----------------------------------------------------------------------===//
|
||||
# Produces an image that compiles and archives clang, based on debian8.
|
||||
FROM launcher.gcr.io/google/debian8:latest
|
||||
|
||||
# Stage 1. Check out LLVM source code and run the build.
|
||||
FROM launcher.gcr.io/google/debian8:latest as builder
|
||||
LABEL maintainer "LLVM Developers"
|
||||
|
||||
# Install build dependencies of llvm.
|
||||
# First, Update the apt's source list and include the sources of the packages.
|
||||
RUN grep deb /etc/apt/sources.list | \
|
||||
sed 's/^deb/deb-src /g' >> /etc/apt/sources.list
|
||||
|
||||
# Install compiler, python and subversion.
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends ca-certificates gnupg \
|
||||
build-essential python wget subversion unzip && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install a newer ninja release. It seems the older version in the debian repos
|
||||
# randomly crashes when compiling llvm.
|
||||
RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \
|
||||
@ -29,10 +25,8 @@ RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-li
|
||||
| sha256sum -c && \
|
||||
unzip ninja-linux.zip -d /usr/local/bin && \
|
||||
rm ninja-linux.zip
|
||||
|
||||
# Import public key required for verifying signature of cmake download.
|
||||
RUN gpg --keyserver hkp://pgp.mit.edu --recv 0x2D2CEF1034921684
|
||||
|
||||
# Download, verify and install cmake version that can compile clang into /usr/local.
|
||||
# (Version in debian8 repos is is too old)
|
||||
RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \
|
||||
@ -47,9 +41,18 @@ RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \
|
||||
|
||||
ADD checksums /tmp/checksums
|
||||
ADD scripts /tmp/scripts
|
||||
|
||||
# Arguments passed to build_install_clang.sh.
|
||||
ARG buildscript_args
|
||||
|
||||
# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
|
||||
# Run the build. Results of the build will be available at /tmp/clang-install/.
|
||||
RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
|
||||
|
||||
|
||||
# Stage 2. Produce a minimal release image with build results.
|
||||
FROM launcher.gcr.io/google/debian8:latest
|
||||
LABEL maintainer "LLVM Developers"
|
||||
# Install packages for minimal useful image.
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
# Copy build results of stage 1 to /usr/local.
|
||||
COPY --from=builder /tmp/clang-install/ /usr/local/
|
||||
@ -1,21 +0,0 @@
|
||||
#===- llvm/utils/docker/debian8/release/Dockerfile -----------------------===//
|
||||
#
|
||||
# The LLVM Compiler Infrastructure
|
||||
#
|
||||
# This file is distributed under the University of Illinois Open Source
|
||||
# License. See LICENSE.TXT for details.
|
||||
#
|
||||
#===----------------------------------------------------------------------===//
|
||||
# A release image, containing clang installation, produced by the 'build/' image
|
||||
# and adding libstdc++ and binutils.
|
||||
FROM launcher.gcr.io/google/debian8:latest
|
||||
|
||||
LABEL maintainer "LLVM Developers"
|
||||
|
||||
# Install packages for minimal useful image.
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Unpack clang installation into this image.
|
||||
ADD clang.tar.gz /usr/local/
|
||||
@ -9,20 +9,29 @@
|
||||
# This is an example Dockerfile to build an image that compiles clang.
|
||||
# Replace FIXMEs to prepare your own image.
|
||||
|
||||
# Stage 1. Check out LLVM source code and run the build.
|
||||
# FIXME: Replace 'ubuntu' with your base image
|
||||
FROM ubuntu
|
||||
|
||||
FROM ubuntu as builder
|
||||
# FIXME: Change maintainer name
|
||||
LABEL maintainer "Maintainer <maintainer@email>"
|
||||
|
||||
# FIXME: Install llvm/clang build dependencies. Including compiler to
|
||||
# FIXME: Install llvm/clang build dependencies here. Including compiler to
|
||||
# build stage1, cmake, subversion, ninja, etc.
|
||||
|
||||
ADD checksums /tmp/checksums
|
||||
ADD scripts /tmp/scripts
|
||||
|
||||
# Arguments passed to build_install_clang.sh.
|
||||
ARG buildscript_args
|
||||
|
||||
# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
|
||||
# Run the build. Results of the build will be available as /tmp/clang-install.
|
||||
RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
|
||||
|
||||
|
||||
# Stage 2. Produce a minimal release image with build results.
|
||||
# FIXME: Replace 'ubuntu' with your base image.
|
||||
FROM ubuntu
|
||||
# FIXME: Change maintainer name.
|
||||
LABEL maintainer "Maintainer <maintainer@email>"
|
||||
# FIXME: Install all packages you want to have in your release container.
|
||||
# A minimal useful installation should include at least libstdc++ and binutils.
|
||||
|
||||
# Copy build results of stage 1 to /usr/local.
|
||||
COPY --from=builder /tmp/clang-install/ /usr/local/
|
||||
@ -1,24 +0,0 @@
|
||||
#===- llvm/utils/docker/example/release/Dockerfile -----------------------===//
|
||||
#
|
||||
# The LLVM Compiler Infrastructure
|
||||
#
|
||||
# This file is distributed under the University of Illinois Open Source
|
||||
# License. See LICENSE.TXT for details.
|
||||
#
|
||||
#===----------------------------------------------------------------------===//
|
||||
# An image that unpacks a clang installation, compiled by the 'build/'
|
||||
# container.
|
||||
# Replace FIXMEs to prepare your own image.
|
||||
|
||||
# FIXME: Replace 'ubuntu' with your base image.
|
||||
FROM ubuntu
|
||||
|
||||
# FIXME: Change maintainer name.
|
||||
LABEL maintainer "Maintainer <maintainer@email>"
|
||||
|
||||
# FIXME: Install all packages you want to have in your release container.
|
||||
# A minimal useful installation must include libstdc++ and binutils.
|
||||
|
||||
# Unpack clang installation into this container.
|
||||
# It is copied to this directory by build_docker_image.sh script.
|
||||
ADD clang.tar.gz /usr/local/
|
||||
@ -6,26 +6,26 @@
|
||||
# License. See LICENSE.TXT for details.
|
||||
#
|
||||
#===----------------------------------------------------------------------===//
|
||||
# Produces an image that compiles and archives clang, based on nvidia/cuda
|
||||
# image.
|
||||
FROM nvidia/cuda:8.0-devel
|
||||
|
||||
# Stage 1. Check out LLVM source code and run the build.
|
||||
FROM nvidia/cuda:8.0-devel as builder
|
||||
LABEL maintainer "LLVM Developers"
|
||||
|
||||
# Arguments to pass to build_install_clang.sh.
|
||||
ARG buildscript_args
|
||||
|
||||
# Install llvm build dependencies.
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends ca-certificates cmake python \
|
||||
subversion ninja-build && \
|
||||
subversion ninja-build && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
ADD checksums /tmp/checksums
|
||||
ADD scripts /tmp/scripts
|
||||
|
||||
# Arguments passed to build_install_clang.sh.
|
||||
ARG buildscript_args
|
||||
|
||||
# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
|
||||
# Run the build. Results of the build will be available at /tmp/clang-install/.
|
||||
RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
|
||||
|
||||
|
||||
# Stage 2. Produce a minimal release image with build results.
|
||||
FROM nvidia/cuda:8.0-devel
|
||||
LABEL maintainer "LLVM Developers"
|
||||
# Copy clang installation into this container.
|
||||
COPY --from=builder /tmp/clang-install/ /usr/local/
|
||||
# C++ standard library and binutils are already included in the base package.
|
||||
@ -1,23 +0,0 @@
|
||||
#===- llvm/utils/docker/nvidia-cuda/release/Dockerfile -------------------===//
|
||||
#
|
||||
# The LLVM Compiler Infrastructure
|
||||
#
|
||||
# This file is distributed under the University of Illinois Open Source
|
||||
# License. See LICENSE.TXT for details.
|
||||
#
|
||||
#===----------------------------------------------------------------------===//
|
||||
# This is an example Dockerfile that copies a clang installation, compiled
|
||||
# by the 'build/' container into a fresh docker image to get a container of
|
||||
# minimal size.
|
||||
# Replace FIXMEs to prepare a new Dockerfile.
|
||||
|
||||
# FIXME: Replace 'ubuntu' with your base image.
|
||||
FROM nvidia/cuda:8.0-devel
|
||||
|
||||
# FIXME: Change maintainer name.
|
||||
LABEL maintainer "LLVM Developers"
|
||||
|
||||
# Unpack clang installation into this container.
|
||||
ADD clang.tar.gz /usr/local/
|
||||
|
||||
# C++ standard library and binutils are already included in the base package.
|
||||
@ -16,8 +16,8 @@ Usage: build_install_llvm.sh [options] -- [cmake-args]
|
||||
|
||||
Checkout svn sources and run cmake with the specified arguments. Used
|
||||
inside docker container.
|
||||
Passes additional -DCMAKE_INSTALL_PREFIX and archives the contents of
|
||||
the directory to /tmp/clang.tar.gz.
|
||||
Passes additional -DCMAKE_INSTALL_PREFIX and puts the build results into
|
||||
/tmp/clang-install/ directory.
|
||||
|
||||
Available options:
|
||||
-h|--help show this help message
|
||||
@ -244,12 +244,7 @@ ninja $CMAKE_INSTALL_TARGETS
|
||||
|
||||
popd
|
||||
|
||||
# Pack the installed clang into an archive.
|
||||
echo "Archiving clang installation to /tmp/clang.tar.gz"
|
||||
cd "$CLANG_INSTALL_DIR"
|
||||
tar -czf /tmp/clang.tar.gz *
|
||||
|
||||
# Cleanup.
|
||||
rm -rf "$CLANG_BUILD_DIR" "$CLANG_INSTALL_DIR"
|
||||
rm -rf "$CLANG_BUILD_DIR"
|
||||
|
||||
echo "Done"
|
||||
|
||||
Loading…
Reference in New Issue
Block a user