Siddhi expands its hands to support Prometheus in event processing- Part 2

In the previous part of this blog, we learnt the functionality of prometheus-source which is a component of the Siddhi extension ‘siddhi-io-prometheus’. It was explained through a sample micro-service which exposes a Prometheus counter metric of total device power consumption. In this part, we will be analyzing the functionality of Prometheus-sink through a sample Siddhi app.

Prometheus-sink is a special type of Prometheus exporter which exposes the Siddhi events it receives as a set of metrics for Prometheus to scrape and monitor. The sink enables the user to specify the HTTP target URL, the mode of export and the other metric details through the sink definition. It supports four types of Prometheus metrics which are : counter, gauge, histogram and summary. The user can specify the mode of export through the ‘publish.mode’ option of the sink. The values that are supported by publish mode are ‘server’ and ‘ pushGateway’ where the server mode creates an HTTP server to expose the created metrics and the pushGateway mode enables Prometheus Push Gateway to push the metrics.

Functioning of siddhi-io-prometheus

prometheus-sink

To understand the functionality of prometheus-sink, we need to execute the sample provided with WSO2 product-sp. Prior to that, you must configure product-sp to support this extension by including the required 3rd party JARs. Please refer to part-1 of this blog and carry out the instructions given there.

Then, follow these instructions to run the sample.

1. Start the editor. {WSO2_SP_HOME}/bin/editor.sh
2. Open Stream Processor Studio with http://localhost:9390/editor.
3. File → Import Sample → PublishPrometheusMetrics → PublishPrometheusMetricsHTTPServer.siddhi → Open.
4. Follow the steps given in the sample.

Sample for prometheus-sink

Prerequisites

To check the metrics exported by sink, you need Prometheus server up and running in your machine with the configuration of targets. Therefore, Prometheus must be downloaded from https://prometheus.io/download/#prometheus prior to run the samples. The configuration of targets from which Prometheus will scrape the metrics must be included in the configuration file ‘prometheus.yml’ as follows,

- job_name: 'server'
  honor_labels: true
  static_configs:
     - targets: ['localhost:9080']

Then,you should navigate to the directory where Prometheus is installed through terminal and start Prometheus server using the following command,

./prometheus --config.file="prometheus.yml"

• If you use the ‘pushGateway’ publish mode in the sink, you must add the target configuration in the same way according to the host and port where Prometheus Push Gateway is enabled. And also, you must start the push gateway by using the following command before running the sample.

./pushgateway

How to run the sample?

Follow the instructions given in the sample to run and generate events through the ‘event simulator’ to create and update metrics using Prometheus-sink.

generating events through event simulator

Explanation for the app

In the Siddhi app, there are two sink streams that receives the events from a stream named ‘FooStream’. The ‘SweetProductionStream’ is the stream associated with prometheus-sink which creates a counter typed Prometehus metric with the attribute name as its label and updates the value of the metric by the value received in the attribute ‘amount’. The created metric is exported at the server.URL “http://localhost:9080/metrics” in a Prometheus specified format. The option ‘value.attribute’ in the sink definition specifies the attribute name of the stream which represents the value of the metric.

The exported counter metrics at the server URL

When the user generates events, the values of the metrics will be increased according to the values of the ‘value.attribute’ from received events. This behavior differs for each type of metrics the sink creates. For counter metrics, the metric values will be summed with previous values. For gauge metrics, the metric values are increased or subtracted from previous value according to its positive or negative nature. The received value of ‘value.attribute’ will be observed to generate the count, sum, quantile or bucket labels for summary and histogram metrics.

How the metrics are used by Prometheus?

As we configured Prometheus earlier to pull metric data from the target where the sink exposes the metrics, we will be able to read and analyze the metrics through Prometheus. Therefore it is possible for the user to query and visualize the metric data through the interface provided by Prometheus.

The exported metrics are queried at the Prometheus interface

Graph representation of the metric data provided by Prometheus

In this way, a Siddhi user gets a chance to create and manipulate data over Prometheus. This provides the benefits of complex event processing and the real-time streaming analytics to the multi-dimensional data collection model of Prometheus.

The extension targets the users who use Siddhi to perform their streaming analytics and Prometheus to monitor their systems. Through the extension, they will be able to improve the management of their system by interconnecting these two large platforms.

The implementation of the extension ‘siddhi-io-prometheus’ can be found here. As Siddhi and its extensions are open source, there is no restriction for anyone to use them under Apache 2.0 license and to contribute to the community. Further, reporting the issues found in the extension and contributing by sending fixes through pull requests are greatly appreciated.