1. Home
  2. MCP Operations Guide
  3. Cloud verification
  4. Perform the OpenStack high availability testing
  5. Execute the CVP - HA tests pipeline

Execute the CVP - HA tests pipeline

Execute the CVP - HA tests pipeline

This section instructs you on how to perform non-functional failover testing of OpenStack nodes that include but are not limited to the ctl, ntw, and dbs virtual machines using the CVP - HA tests Jenkins pipeline.

Note

Clone the cvp-configuration and tempest repositories to your local Gerrit and use them locally to add new or adjust the existing tests and fine-tune the Tempest configuration.

To perform the non-functional failover testing of your OpenStack deployment:

  1. In your local cvp-configuration repository, inspect and modify the following items as required:

    • The Tempest configuration

    • The skip list

    • The configure.sh setup script

      Note

      We recommend running CVP - HA tests Jenkins pipeline after the functional/integration testing is completed and all issues are resolved.

  2. In a web browser, open http://<ip_address>:8081 to access the Jenkins web UI.

    Note

    The IP address is defined in the classes/cluster/<cluster_name>/cicd/init.yml file of the Reclass model under the cicd_control_address parameter variable.

  3. Log in to the Jenkins web UI as admin.

    Note

    To obtain the password for the admin user, run the salt "cid*" pillar.data _param:jenkins_admin_password command from the Salt Master node.

  4. In the global view, find the CVP - HA tests pipeline.

  5. Select the Build with Parameters option from the drop-down menu of the pipeline.

  6. Configure the following parameters as required:

    CVP - HA tests parameters
    Parameter Description
    DEBUG_MODE If checked, keeps the container after test is performed for the debugging purposes.
    MANUAL_CONFIRMATION If checked, you will be asked for a confirmation before any destructive actions such as a node reboot or shutdown.
    PROXY If an environment uses HTTP or HTTPS proxy, verify that you specify it in this field as this proxy address will be used to clone the required repositories and install the Python requirements. For the offline mode, specify offline.
    RETRY_CHECK_STATUS Specifies the number of retries to check the node status. If you have any issues with timeouts, increase the default 200 value.
    SALT_MASTER_CREDENTIALS Specifies the credentials to Salt API stored in Jenkins, included by default. See View credentials details used in Jenkins pipelines.
    SALT_MASTER_URL

    Specifies the reachable IP address of the Salt Master node and port on which Salt API listens. For example, http://172.18.170.28:6969

    To determine on which port Salt API listens:

    1. Log in to the Salt Master node.

    2. Search for the port in the /etc/salt/master.d/_api.conf file.

    3. Verify that the Salt Master node is listening on that port:

      netstat -tunelp | grep <PORT>

    Caution

    In the 2019.2.4 update, by default, the HTTPS (SSL) NGINX URL is used as SALT_MASTER_URL. For long-running pipelines, it may lead to a timeout error, usually after 10 minutes. Therefore, specify the Salt API native non-SSL URL as described above to prevent timeout errors. Starting from the 2019.2.5 update, Salt API native non-SSL URL is used by default.
    SKIP_LIST_PATH Optional. Specifies the path to the tempest skip list file located in the repository specified in TOOLS_REPO. The default value is cvp-configuration/tempest/skip-list.yaml. For the offline mode, when cvp-rally image is used, define /var/lib/cvp-configuration/tempest/skip-list.yaml. If you use a custom image, specify the path to the custom skip list file.
    TARGET_NODES Specifies OpenStack control plane nodes that will be under the HA test. For example, ctl* will include all nodes that have ctl at the beginning of their names (usually controllers).
    TEMPEST_REPO Specifies the Tempest repository to clone and use. By default, it is the upstream Tempest. Though, you can specify your customized tempest in a local or remote repository. For the full offline mode, specify /var/lib/tempest (a path inside a container) and cvp-rally image.
    TEMPEST_TARGET_NODE

    Specifies the node to run the container with Tempest/Rally. Use the Jenkins slave as it has the Docker package.

    Note

    Starting from the MCP 2019.2.2 update, if the TARGET_NODE parameter is empty, the node with the gerrit:client pillar will be used, which is cid01 by default.
    TEMPEST_TEST_PATTERN Specifies the tests to run. See the Rally documentation for all available options.
    TEST_IMAGE

    Specifies the link to the Docker Rally-based image to use for running the container with testing tools. We recommend using the upstream Rally image, xrally/xrally-openstack:0.11.2.

    For the full offline mode, use the cvp-rally image from the local Docker images mirror (Registry) or pull docker-prod-local.docker.mirantis.net/mirantis/cvp/cvp-rally:<mcp_version> locally.
    TOOLS_REPO

    Includes the URL or path to the repository where testing tools, scenarios, and configurations are located. By default, it is https://github.com/Mirantis/cvp-configuration. Specify your MCP version here. For example, -b release/2019.2.0 for Q4`18.

    Caution

    For the Q4`18 MCP release, the branch name format is release/2019.2.0 for the cvp-configuraiton repository. The old 2019.2.0 format is deprecated.

    To customize the configuration, clone the cvp-configuration repository to your local Gerrit and commit the changes.

    For the offline mode, specify /var/lib/cvp-configuration/configure.sh. Alternatively, use this repository from the offline image.

    If your image is fully configured, leave the field empty.

  7. Click Build.

  8. Verify the job stages statuses in the Stage view section:

    • GREEN, SUCCESS

      Testing has beed performed successfully, no errors found.

    • RED, FAILURE

      Tempest run has failed during one of the job stages causing the pipeline abort. Review the console output.

updated: 2024-01-04 11:47