Configure load balancing for Horizon

Configure load balancing for Horizon

Starting from the Q4’18 MCP version, Horizon works in the load balancing mode by default. All requests to Horizon are terminated and forwarded to the Horizon back end by HAProxy bound on a virtual IP address. HAProxy serves as a balancer and manages requests according to the defined policy, which is round-robin by default, among proxy nodes. This approach allows for load reduction on one proxy node and spreading the load among all proxy nodes.

Note

If the node, which the user is connected to, has failed and the user is reconnected to another node, the user will be logged out from the dashboard. As a result, the The user is not authorized page opens, which is the expected behavior in this use case. To continue working with the dashboard, the user has to sign in to Horizon again from the Log In page.

This section provides the instruction on how to manually configure Horizon load balancing for the existing OpenStack deployments that are based on earlier MCP release versions.

To enable active-active mode for Horizon:

  1. Log in to the Salt Master node.

  2. Update to the 2019.2.0 Build ID MCP version or higher.

  3. Verify that the system.apache.server.site.horizon class has been added to your Reclass model. By default, the class is defined in the ./system/apache/server/site/horizon.yml file on the Reclass system level as follows:

    parameters:
      _param:
        apache_ssl:
          enabled: false
        apache_horizon_ssl: ${_param:apache_ssl}
        apache_horizon_api_address: ${_param:horizon_server_bind_address}
        apache_horizon_api_host: ${linux:network:fqdn}
      apache:
        server:
          bind:
            listen_default_ports: false
          enabled: true
          default_mpm: event
          modules:
            - wsgi
          site:
            horizon:
              enabled: false
              available: true
              type: wsgi
              name: openstack_web
              ssl: ${_param:apache_horizon_ssl}
              wsgi:
                daemon_process: horizon
                processes: 3
                threads: 10
                user: horizon
                group: horizon
                display_name: '%{GROUP}'
                script_alias: '/ /usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'
                application_group: '%{GLOBAL}'
                authorization: 'On'
              limits:
                request_body: 0
              host:
                address: ${_param:apache_horizon_api_address}
                name: ${_param:apache_horizon_api_host}
                port: 8078
              locations:
                - uri: /static
                  path: /usr/share/openstack-dashboard/static
              directories:
                dashboard_static:
                  path: /usr/share/openstack-dashboard/static
                  order: 'allow,deny'
                  allow: 'from all'
                  modules:
                    mod_expires.c:
                      ExpiresActive: 'On'
                      ExpiresDefault: '"access 6 month"'
                    mod_deflate.c:
                      SetOutputFilter: 'DEFLATE'
                dashboard_wsgi:
                  path: /usr/share/openstack-dashboard/openstack_dashboard/wsgi
                  order: 'allow,deny'
                  allow: 'from all'
              log:
                custom:
                  format: >-
                    %v:%p %{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %D %O \"%{Referer}i\" \"%{User-Agent}i\"
                error:
                  enabled: true
                  level: debug
                  format: '%M'
                  file: '/var/log/apache2/openstack_dashboard_error.log'
    
  4. Verify that the system.apache.server.site.horizon has been added to the Reclass system level in the ./system/horizon/server/single.yml file as follows:

    classes:
      - service.horizon.server.single
      - system.horizon.upgrade
      - system.horizon.server.iptables
      - system.apache.server.single
      - system.memcached.server.single
      - system.apache.server.site.horizon
    
  5. Verify that the definition for the system.haproxy.proxy.listen.openstack.openstack_web class has been added to the Reclass cluster level in the in the proxy nodes configuration file:

    parameters:
      _param:
        haproxy_openstack_web_check_params: check
      haproxy:
        proxy:
          listen:
            openstack_web:
              type: custom
              check: false
              sticks: ${_param:haproxy_openstack_web_sticks_params}
              binds:
              - address: ${_param:cluster_vip_address}
                port: ${_param:haproxy_openstack_web_bind_port}
              servers:
              - name: ${_param:cluster_node01_hostname}
                host: ${_param:cluster_node01_address}
                port: 8078
                params: ${_param:haproxy_openstack_web_check_params}
              - name: ${_param:cluster_node02_hostname}
                host: ${_param:cluster_node02_address}
                port: 8078
                params: ${_param:haproxy_openstack_web_check_params}
    
  6. Add the system.haproxy.proxy.listen.openstack.openstack_web class to the Horizon node configuration file, for example, cluster/<cluster_name>/openstack/dashboard.yml:

    classes:
      - system.haproxy.proxy.listen.openstack.openstack_web
    
  7. In the Horizon node configuration file (edited in the previous step), define the host names and IP addresses for all proxy nodes used in the deployment for the dashboard node and verify that the HAProxy checks the availability of Horizon.

    Congifuration example for two proxy nodes:

    parameters:
      _param:
        cluster_node01_hostname: ${_param:openstack_proxy_node01_hostname}
        cluster_node01_address: ${_param:openstack_proxy_node01_address}
        cluster_node02_hostname: ${_param:openstack_proxy_node02_hostname}
        cluster_node02_address: ${_param:openstack_proxy_node02_address}
        haproxy_openstack_web_bind_port: ${_param:horizon_public_port}
        haproxy_openstack_web_check_params: check inter 10s fastinter 2s downinter 3s rise 3 fall 3 check-ssl verify none
      horizon:
        server:
          cache:
            ~members:
            - host: ${_param:openstack_proxy_node01_address}
              port: 11211
            - host: ${_param:openstack_proxy_node02_address}
              port: 11211
    
  8. If the HTTP to HTTPS redirection will be used, add the following configuration to the Horizon node configuration file:

    parameters:
      haproxy:
        proxy:
          listen:
            openstack_web_proxy:
              mode: http
              format: end
              force_ssl: true
              binds:
              - address: ${_param:cluster_vip_address}
                port: 80
    
  9. Disable the NGINX servers requests for Horizon by replacing the NGINX class with the HAProxy class in the proxy node configuration file.

    Replace:

    - system.nginx.server.proxy.openstack_web
    

    with

    - system.haproxy.proxy.single
    
  10. Remove the nginx_redirect_openstack_web_redirect.conf and nginx_proxy_openstack_web.conf Horizon sites from /etc/nginx/sites-enabled/.

  11. Restart the NGINX service on the proxy nodes:

    salt 'prx*' cmd.run 'systemctl restart nginx'
    
  12. Verify that Keepalived keeps track on HAProxy by adding the haproxy variable for the keepalived_vrrp_script_check_multiple_processes parameter:

    parameters:
      _param:
        keepalived_vrrp_script_check_multiple_processes: 'nginx haproxy'
    
  13. Enable SSL for Horizon:

    parameters:
      _param:
        apache_ssl:
          enabled: true
          authority: ${_param:salt_minion_ca_authority}
          engine: salt
          key_file:  /srv/salt/pki/${_param:cluster_name}/${salt:minion:cert:proxy:common_name}.key
          cert_file: /srv/salt/pki/${_param:cluster_name}/${salt:minion:cert:proxy:common_name}.crt
          chain_file: /srv/salt/pki/${_param:cluster_name}/${salt:minion:cert:proxy:common_name}-with-chain.crt
    
  14. Define the address to be bound by Memcached in the cluster/<cluster_name>/openstack/proxy.yml file:

    parameters:
      _param:
        openstack_memcached_server_bind_address: ${_param:single_address}
    
  15. Verify that the Horizon Salt formula is updated to the the version higher than 2016.12.1+201812072002.e40b950 and the Apache Salt formula is updated to the version higher than 0.2+201811301717.acb3391.

  16. Delete the NGINX sites from the proxy nodes that proxy Horizon requests and possible redirection from HTTP to HTTPS.

  17. Apply the haproxy and horizon states on the proxy nodes:

    salt -C 'I@horizon:server' state.sls horizon
    salt -C 'I@horizon:server' state.sls haproxy