Prometheus

Prometheus is deployed in PHX in order to store and analyze various metrics of the environment including CI jobs.

Since Prometheus actively crawls through discovered targets and CI jobs are short-living, a push gateway was deployed to cache metrics for Prometheus.

Sending data from Jenkins jobs

The endpoint to send metrics to is https://prom-gw.apps.ovirt.org/

Access to the endpoint is protected using certificate authentication. A client needs to supply a certificate with each request.

Required data is available as Jenkins credentials:

credential name in Jenkins description
prom-ca CA certificate
prom-crt client certificate
prom-key client private key

To send data add these credentials to a job and use the resulting environment variables as respective arguments for the HTTP client.

Here's a simple curl example that will send the test data to Prometheus:

echo "build_number $BUILD_NUMBER" | curl --data-binary @- --cacert $prom_ca --cert $prom_crt --key $prom_key https://prom-gw.apps.ovirt.org/metrics/job/$JOB_NAME

In this case, the build number is sent as the build_number metric and the job name is added as a label for future analysis. The certificate and key are provided by Jenkins as credentials.

Check out the official docs for more examples.

Accessing and visualising the data

The instance is deployed in OpenShift, data can be viewed and analyzed in Prometheus directly or through Kibana