Customize the prebuilt mirrors¶

You can easily customize mirrored Aptly, Docker, and Git repositories by configuring contents of the mirror VM defined in the infra/mirror/init.yml file of the Reclass model.

To customize the debmirror repositories mirrors

You can either customize the already existing debmirrors content or specify any custom mirror required by your MCP deployment.

1. Customize the debmirror content as required. Example of customization:

   Note

   Starting from the MCP Build ID 2019.2.0, the default list of repositories per release is defined in reclass-system/debmirror/mirror_mirantis_com/init.yml. For earlier MCP releases, the repositories are included directly from the corresponding classes.

   debmirror:
     client:
       enabled: true
       mirrors:
         mirror_mirantis_com_ceph_luminous_xenial:
           arch:
           - amd64
           dist:
           - xenial
           extra_flags:
           - --verbose
           - --progress
           - --nosource
           - --no-check-gpg
           - --rsync-extra=none
           filter:
             '1': --exclude='(-dbg_|-dbg-)'
           force: false
           lock_target: true
           log_file: /var/log/debmirror/mirror_mirantis_com_ceph_luminous_xenial.log
           method: rsync
           mirror_host: mirror.mirantis.com
           mirror_root: :mirror/proposed/ceph-luminous/xenial/
           section:
           - main
           target_dir: /srv/volumes/aptly/public/proposed//ceph-luminous/xenial/
   

2. Include the debmirror content class to infra/mirror/init.yml. For example, to include all repositories by default for future MCP cluster update, add the following class:

   - system.debmirror.mirror_mirantis_com
   

3. Apply the debmirror state:

   salt '<offline_node_name>' state.apply debmirror
   

Example: Deliver the OpenStack Pike update repository to an offline deployment

For a fully isolated MCP cluster with no access to the Mirantis mirrors even from the apt01 node, you can enable generation of a copy of a mirrored repository directly on a host node. You can then move this copy to the apt01 node using scp or rsync, for example.

This is the exemplary procedure of the debmirror repository customization that delivers the OpenStack Pike update repository. Such customization enables you to obtain the MCP maintenance updates.

1. In reclass-system/debmirror/mirror_mirantis_com/init.yml described above, identify the repository classes available for an MCP release version deployed on your cluster and select the one that you need to receive maintenance updates for. For example, for OpenStack Pike:

   cat debmirror/mirror_mirantis_com/init.yml |grep openstack-pike
     ...
     - system.debmirror.mirror_mirantis_com.openstack-pike.xenial
     ...
   

2. Obtain the required data for the selected class and convert it to a debmirror utility.

   Note

   For human readability, the debmirror formula has the native debmirror syntax that allows you to convert a class data into a local cmdline.

   For example:

   1. Display contents of the OpenStack xenial.yml file:

      cat debmirror/mirror_mirantis_com/openstack-pike/xenial.yml
      ...
       parameters:
         debmirror:
           client:
             enabled: true
             mirrors:
               mirror_mirantis_com_openstack_pike_xenial:
                 force: ${_param:mirror_mirantis_com_openstack_pike_xenial_force}
                 lock_target: True
                 extra_flags: [ '--verbose', '--progress', '--nosource', '--no-check-gpg', '--rsync-extra=none' ]
                 method: "rsync"
                 arch: [ 'amd64' ]
                 mirror_host: "mirror.mirantis.com"
                 mirror_root: ":mirror/${_param:mcp_version}/openstack-pike/xenial/"
                 target_dir: "${_param:debmirror_mirrors_base_target_dir}/openstack-pike/xenial/"
                 log_file: "/var/log/debmirror/mirror_mirantis_com_openstack_pike_xenial.log"
                 dist: [ xenial ]
                 section: [ main ]
                 filter:
                   001: --exclude='(-|_)dbg(_|-)'
      

   2. Convert the contents obtained in the previous step into the debmirror cmdline:

      debmirror --verbose --progress --nosource --no-check-gpg --rsync-extra=none --dist=xenial --section=main \
      --method=rsync --host="mirror.mirantis.com" --root=":mirror/update/2019.2.0/openstack-pike/xenial/" \
      --arch=amd64 --exclude='(-|_)dbg(_|-)' /debmirror_example/update/2019.2.0/update/openstack-pike/xenial/
      

      In the example cmdline above, the path to mirror_root is extended with the /update/ subdirectory to fetch the update repository.

3. Create a directory for the update repository. For example:

   mkdir debmirror_example/2019.2.0/update/openstack-pike/xenial/
   

4. Run a non-customized Docker container that runs Ubuntu 16.04. For example:

   docker run -v $(pwd)/debmirror_example:/debmirror_example --hostname=docker-16  \
   --cpus=4 -ti ubuntu:xenial /bin/bash
   

5. Install debmirror in this container:

   root@docker-16:/# apt-get update && apt-get install -y xz-utils debmirror rsync apt-transport-https curl
   root@docker-16:/# curl -fsSL https://mirror.mirantis.com/update/2019.2.0/openstack-pike/xenial/archive-pike.key | apt-key add -
   

6. Run cmdline prepared in the step 2.2:

   root@docker-16:/# debmirror --verbose --progress --keyring=/etc/apt/trusted.gpg --nosource \
   --rsync-extra=none --dist=xenial --section=main --method=rsync \
   --host="mirror.mirantis.com" --root=":mirror/update/2019.2.0/openstack-pike/xenial/" \
   --arch=amd64 --exclude='(-|_)dbg(_|-)' /debmirror_example/update/2019.2.0/openstack-pike/xenial/
   

7. Exit from the Docker container.

8. Inspect the update mirror that is now locally available in:

   tree -L 5 debmirror_example/
   debmirror_example/
   └── update
       └── 2019.2.0
           └── openstack-pike
               └── xenial
                   ├── dists
                   └── pool
   

9. Move the structure of the downloaded repository to the apt01 node. By default, the update mirror structure is located on the apt01 node in /srv/volumes/aptly/public/update/2019.2.0/openstack-pike.

   Warning

   While modifying /srv/volumes/aptly/public/update/, make sure that you remove the symlinks only for those repositories that you are going to update. In this example, this is only openstack-pike. Otherwise, the main release binaries for the components that are not being updated will be lost.

   1. In /srv/volumes/aptly/public/update/, remove the default symlink that refers to the MCP release version deployed on a cluster. For example:

      rm -v /srv/volumes/aptly/public/update/2019.2.0
      

      Note

      The symlink is created in the offline mirror for backward compatibility purposes.

   2. Create the same links for the underlay repositories. Use the following script as example:

      apt01:# export release='2019.2.0'; pushd '/srv/volumes/aptly/public/update/' \
              if [[ -d ${release} && ! -h ${release} ]]; then echo 'Its already dir, nothing todo' ;else \
               rm -v ${release}; \
               mkdir -p ${release}; \
               cd ${release}; \
               for repo in $(ls ../../${release}/) ; do ln -sv ../../${release}/$repo . ; done ; \
              fi
      

   3. Remove only required symlink, for example, openstack-pike, and move the newly generated data to the new structure.

      The final example structure is as follows:

      tree -L 4 /srv/volumes/aptly/public/update/
      update/
      └── 2019.2.0
          ├── ceph-luminous -> ../../2019.2.0/ceph-luminous
          ...
          ├── maas -> ../../2019.2.0/maas
          ├── openstack-pike
          │   └── xenial
          │       ├── dists
          │       ├── pool
          ...
          ├── saltstack-2017.7 -> ../../2019.2.0/saltstack-2017.7
          └── td-agent -> ../../2019.2.0/td-agent
      

To customize the Docker images mirrors

The Docker repositories are defined as an image list that includes a registry and name for each Docker image.

1. Customize the list depending on the needs of your MCP deployment:

   • Specify a different Docker registry for the existing image to be pulled from
   • Add a new Docker image

   Сustomization example in infra/mirror/init.yml:

   Note

   Starting from the MCP Build ID 2019.2.0, the default list of repositories per release is defined in default_local_mirrror_content:docker_client_registry_image.

   docker:
     client:
       registry:
         target_registry: apt:5000
         image:
           - name: openldap:1.2.2
             registry: docker-prod-local.artifactory.mirantis.com/mirantis/external/osixia
           - name: jenkins:proposed
             registry: docker-prod-local.artifactory.mirantis.com/mirantis/cicd
             target_registry: apt:5000/mirantis/cicd
   

   Note

   The target_registry parameter specifies which registry the images will be pushed into.

2. Synchronize the Docker registry:

   salt '<offline_node_name>' state.sls docker.client.registry
   

To customize the Git repositories mirrors

The Git repositories are defined as a repository list that includes a name and URL for each Git repository.

1. Customize the Git repositories list depending on the needs of your MCP deployment.

   Note

   Starting from the MCP Build ID 2019.2.0, the default list of repositories per release is defined in default_local_mirrror_content:git_server_repos.

   Customization example in infra/mirror/init.yml:

   git:
     server:
       directory: /srv/git/
       repos:
         - name: mk-pipelines
           url: https://github.com/Mirantis/mk-pipelines.git
         - name: pipeline-library
           url: https://github.com/Mirantis/pipeline-library.git
   

2. Synchronize the Git repositories:

   salt '<offline_node_name>' state.sls git.server
   

To customize the MAAS mirrors

The MAAS mirrors are defined as image sections that include bootloaders and packages. Usually, they should not be customized since they mirror the upstream MAAS repositories directly.

1. Inspect the default MAAS pillar structure in defaults/maas.yml on the system level of the Reclass model:

   parameters:
     _param:
       maas_postgresql_server: ${_param:postgresql_server}
       default_local_mirrror_content:
         maas_mirror_image_sections:
           bootloaders:
             keyring: /usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
             upstream: ${_param:linux_system_repo_update_url}/maas-ephemeral-v3/
             local_dir: /srv/http/${_param:mcp_version}/maas-ephemeral-v3/
             count: 1
             # i386 need for pxe
             filters: ['arch~(i386|amd64)', 'os~(grub*|pxelinux*)']
           xenial:
             keyring: /usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
             upstream: ${_param:linux_system_repo_update_url}/maas-ephemeral-v3/
             local_dir: /srv/http/${_param:mcp_version}/maas-ephemeral-v3/
             count: 1
             filters: ['release~(xenial)', 'arch~(amd64)', 'subarch~(generic|hwe-16*|ga-16*)']
   

2. In infra/mirror/init.yml, add the customizations under the maas:mirror:image:sections pillar.

   Also, use this pillar to update the MAAS mirrors. For example:

   maas:
    mirror:
      enabled: true
      image:
        sections: ${_param:default_local_mirrror_content:maas_mirror_image_sections}
   

3. Synchronize the MAAS repositories:

   salt '<offline_node_name>' state.sls maas.mirror
   

To customize static binaries or images

Depending on the needs of your MCP deployment, you can customize the storage of the offline image static binaries, images, and other files.

1. In infra/mirror/init.yml, add the required customizations under the linux:system:directory and linux:system:file pillars:

   linux:
     system:
       directory:
         /srv/http/custom-binaries:
           user: www-data
           group: www-data
           mode: 755
           makedirs: true
       file:
         new_image_file:
           name: /srv/http/custom-binaries/custom-binary_1
           source: <some_source_binary_source>/custom-binary_1
           hash: <some_source_binary_source>/custom-binary_1.md5
   

2. Synchronize the customized files locally:

   salt '<offline_node_name>' state.sls linux.system.file
   

To customize the Aptly repositories mirrors

You can either customize the already existing mirrors content or specify any custom mirror required by your MCP deployment.

• To customize existing mirror sources:

  The sources for existing mirrors can be configured to use different upstream.

  Each Aptly mirror specification includes parameters that define their source on the system level of the Reclass model as well distribution, components, key URL, and GPG keys. To customize a mirror content, redefine these parameters as required.

  An example of the apt.mirantis.com mirror specification:

  _param:
    mcp_version: stable
    mirror_mirantis_openstack_xenial_extra_source: http://apt.mirantis.com/xenial/
    mirror_mirantis_openstack_xenial_extra_distribution: ${_param:mcp_version}
    mirror_mirantis_openstack_xenial_extra_components: extra
    mirror_mirantis_openstack_xenial_extra_key_url: "http://apt.mirantis.com/public.gpg"
    mirror_mirantis_openstack_xenial_extra_gpgkeys:
      - A76882D3
  aptly:
    server:
      mirror:
        mirantis_openstack_xenial_extra:
          source: ${_param:mirror_mirantis_openstack_xenial_extra_source}
          distribution: ${_param:mirror_mirantis_openstack_xenial_extra_distribution}
          components: ${_param:mirror_mirantis_openstack_xenial_extra_components}
          architectures: amd64
          key_url: ${_param:mirror_mirantis_openstack_xenial_extra_key_url}
          gpgkeys: ${_param:mirror_mirantis_openstack_xenial_extra_gpgkeys}
          publisher:
            component: extra
            distributions:
              - ubuntu-xenial/${_param:mcp_version}
  

  Note

  You can find all mirrors and their parameters that can be overridden in the aptly/server/mirror section of the Reclass System Model.

• To add new mirrors, extend the aptly:server:mirror pillar of the model using the structure defined in the example above.

  Note

  The aptly:server:mirror:<REPO_NAME>:publisher parameter specifies how the custom repository will be published.

  Example of a custom mirror specification:

  aptly:
    server:
      mirror:
        my_custom_repo_main:
          source: http://my-custom-repo.com
          distribution: custom-dist
          components: main
          architectures: amd64
          key_url: http://my-custom-repo.com/public.gpg
          gpgkeys:
            - AAAA0000
          publisher:
            component: custom-component
            distributions:
              - custom-dist/stable