Perestroika build system

Perestroika build system

Introduction

Fuel 7.0 introduces a new build system called Perestroika. It uses standard upstream Linux distribution tools in order to:

  • build packages (SBuild/Mock)
  • publish packages to repositories
  • manage package repositories (using reprepro/createrepo tools).

Every package is built in a clean and up-to-date buildroot. Packages, their dependencies, and build dependencies are fully self-contained for each OpenStack release. Any package included in any release can be rebuilt at any point in time using only the packages from that release.

The package build CI is reproducible and can be recreated from scratch in a repeatable way.

Perestroika is based on Docker which provides an easy distribution. Each supported Linux distribution contains proper Docker images with necessary tools and scripts.

For the advantages of Perestroika over OBS build system, see Replace OBS with another build system.

Perestroika structure

Scheduler

This part is based on Jenkins CI tool. All jobs are configured using jenkins-job-builder. Jenkins has a separate set of jobs for each [customer name]+[fuel version] case. Gerrit-trigger is configured to track events from the {version} branch of all the [customer-name] Gerrit projects.

Each set of jobs contains:

  • jobs for OpenStack packages for a cluster (.rpm and .deb)
  • jobs for Linux packages for a cluster (.rpm and .deb)
  • jobs for OpenStack packages for Fuel Master node (.rpm). In case of using cluster packages, they are optional.
  • jobs for non-OpenStack Fuel Master node packages (.rpm)
  • jobs for Fuel packages (.rpm and .deb)
  • a job for package publishing

Build workers

These are hardware nodes with preconfigured build tools for all the supported distributions. They are configured as Jenkins slaves.

Each worker contains:

  • preconfigured Docker images with native build tools for each distribution type:
    • mockbuild builds packages using Mock (CentOS 6 and 7 target distributions are supported).
    • sbuild builds packages using SBuild tool (Ubuntu Trusty Tahr target distribution is supported only).
  • prepared minimal build chroots for all the supported distributions:
    • These chroots are updated on a daily basis in order to be up-to-date against the upstream state.
    • Chroot updating is performed by a separate Jenkins job.
    • No build jobs can run on a build host while the updating Jenkins job is running on it.

Building stage flow:

  1. Checking out sources from Gerrit.
  2. Preparing sources to build (creating tarball files, updating changelogs).
  3. Building sources (performed in an isolated environment inside a Docker container).
  4. Getting the build stage exit status, build logs, and built packages.
  5. Parsing and archiving build logs for Jenkins artifacts.

Packaging CI uses short-lived Docker containers to perform package building. Docker images contain preconfigured build tools only. There are no chroots inside images. Build chroots are mounted to a Docker container at start in a read-only mode. Additionally tmpfs partition is mounted over a read-only chroot folder with AUFS overlays inside a Docker container. The container is destroyed once the build stage is completed.

Goals of this scheme:

  • It can run a number of containers with the only chroot simultaneously on the same build host.
  • There is no need to perform clean-up operations after the build (all changes matter inside the container only and will be purged after the container is destroyed).
  • tmpfs works much faster than disk FS/LVM snapshots.

Publisher node

If the build stage finishes successfully, Jenkins runs a publishing job. The Publisher node contains all repositories for all customer projects. It is configured as a Jenkins slave. The repositories are maintained by native tools of their respective distribution (reprepro or createrepo).

The Publisher slave is fully private and available from Jenkins Master node only because of containing a GPG key. All the packages and repositories are signed in terms of their respective distribution by GPG keys that are stored on the Publisher node.

Publishing stage:

  1. Getting built packages from the build host (over scp).
  2. Checking if packages can be added to a repository (version checking against existing packages in order to prevent downgrading).
  3. Signing new packages (all .rpm and source .deb) with GPG keys.
  4. Removing existing and adding new packages to a repository.
  5. Resigning the repository metadata.
  6. Syncing new repository state to a Mirror host (over rsync).

Mirror node

All repositories are available through http or rsync protocols and are synced by a Publisher to a Mirror host.