Access Tenant Telemetry services

The Tenant Telemetry services expose their APIs at different levels: some of them register public endpoints in the OpenStack Identity (Keystone) service catalog, one has no API at all, and one is reachable only from inside the cluster. This section describes how to reach each of them and which CLI clients are required.

Ceilometer (metering)

Ceilometer exposes no accessible API. Metrics it collects are queried through the metric-storage service instead.

Aodh (alarming)

Aodh exposes a REST API and registers itself in the Keystone service catalog as the alarming service type.

To view the registered endpoints:

openstack catalog show alarming

Example output:

+-----------+--------------------------------------------------------------+
| Field     | Value                                                        |
+-----------+--------------------------------------------------------------+
| endpoints | CustomRegion                                                 |
|           |   internal: http://aodh-api.openstack.svc.cluster.local:8042 |
|           | CustomRegion                                                 |
|           |   public: https://aodh.it.just.works                         |
|           | CustomRegion                                                 |
|           |   admin: http://aodh-api.openstack.svc.cluster.local:8042    |
|           |                                                              |
| id        | 8ae515d1e1ee41fb94edb509ae3dafe5                             |
| name      | aodh                                                         |
| type      | alarming                                                     |
+-----------+--------------------------------------------------------------+

To use the official CLI client, python-openstackclient must be installed together with the aodhclient plugin. For administrators, both are pre-installed in the standard keystone-client pod running in the openstack namespace. Therefore, run openstack alarm commands from that pod.

Aetos (metric-storage)

Aetos exposes a REST API that implements a minimal subset of the Prometheus API. It registers itself in the Keystone service catalog as the metric-storage service type.

To view the registered endpoints:

openstack catalog show metric-storage

Example output:

+-----------+----------------------------------------------------------------+
| Field     | Value                                                          |
+-----------+----------------------------------------------------------------+
| endpoints | CustomRegion                                                   |
|           |   admin: http://aetos-api.openstack.svc.cluster.local:8780/    |
|           | CustomRegion                                                   |
|           |   internal: http://aetos-api.openstack.svc.cluster.local:8780/ |
|           | CustomRegion                                                   |
|           |   public: https://aetos.it.just.works/                         |
|           |                                                                |
| id        | 23db5d17863b4fb289fc33db8ce1dac2                               |
| name      | aetos                                                          |
| type      | metric-storage                                                 |
+-----------+----------------------------------------------------------------+

To use the official CLI client, python-openstackclient must be installed together with the python-observabilityclient plugin. For administrators, all required packages are pre-installed in the standard keystone-client pod running in the openstack namespace. Therefore, run openstack metric commands from that pod.

Important

If the gnocchiclient plugin is installed, uninstall it. The two plugins clash on several openstack metric commands.

Because Aetos is an OpenStack-aware proxy, the set of available metrics depends on your scope of authorization. By default, metrics related to cluster-wide resources or to Prometheus itself are available to admin users only.

To list the available metrics:

openstack metric list -f value

Example output from a relatively unused cluster, as seen by an admin user:

ceilometer_dns_domain_exists
ceilometer_dns_zone_recordsets
ceilometer_dns_zone_serial
ceilometer_dns_zone_status
ceilometer_dns_zone_ttl
ceilometer_image_size
ceilometer_radosgw_api_request
ceilometer_radosgw_objects
ceilometer_radosgw_objects_containers
ceilometer_radosgw_objects_size
ceilometer_volume_provider_pool_capacity_allocated
ceilometer_volume_provider_pool_capacity_free
ceilometer_volume_provider_pool_capacity_total
scrape_duration_seconds
scrape_samples_post_metric_relabeling
scrape_samples_scraped
scrape_series_added
sg_total_ceilometer_metric_decode_count
sg_total_ceilometer_metric_decode_error_count
sg_total_ceilometer_msg_received_count
up

Prometheus

The Prometheus instance used by Tenant Telemetry is deliberately not exposed to external users. For debugging purposes, you can query the Prometheus API directly from a pod of the aetos-api Deployment in the openstack namespace:

aetos@aetos-api-5584b68cd6-sql6d:/$ curl -s http://openstack-prometheus-server/api/v1/labels | python3 -m json.tool
{
    "status": "success",
    "data": [
        "__name__",
        "counter",
        "dns",
        "image",
        "instance",
        "job",
        "project",
        "radosgw",
        "resource",
        "resource_name",
        "source",
        "type",
        "unit",
        "vm_instance",
        "volume"
    ]
}