Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. With its powerful querying language and data model, it has become a staple in the DevOps and SRE communities. One of the key features of Prometheus is its Remote Write API, which allows for the efficient forwarding of time series data to external systems. This article explores how to set up two Prometheus instances, where one instance sends its metrics to another using the Remote Write functionality.

Overview of the Remote Write API

The Remote Write API allows Prometheus to send scraped time series data asynchronously to other Prometheus servers or compatible systems. This can be particularly useful for various scenarios, such as centralizing telemetry data from multiple sources, scaling out Prometheus reads, or integrating with other monitoring solutions.

Setting Up the Environment

For this demonstration, we will have two Prometheus instances:

1. Prometheus-Source: This instance will scrape metrics from a defined target (for example, a sample application).
2. Prometheus-Destination: This instance will receive metrics via the Remote Write API.

Step 1: Configure Prometheus-Source

1. Install Prometheus: Download and configure Prometheus on your machine or server. You can follow the official Prometheus installation guide for detailed instructions.

2. Configure prometheus.yml: Create or modify the prometheus.yml file to include your scrape targets and the Remote Write configuration. Here’s a sample configuration:

   global:
     scrape_interval: 15s
   
   scrape_configs:
     - job_name: 'sample-app'
       static_configs:
         - targets: ['localhost:5000']  # Adjust this to point to your application
   
   remote_write:
     - url: 'http://localhost:9091/api/v1/write'  # URL of the Prometheus-Destination instance
   

Step 2: Configure Prometheus-Destination

1. Install Prometheus: Repeat the installation process for another instance of Prometheus, which will serve as the Destination.

2. Configure prometheus.yml: Unlike the source, the destination will primarily receive data. Use a configuration like this:

   global:
     scrape_interval: 15s
   
   rule_files:
     # Add alerting rules configuration if needed
   
   scrape_configs:
     # You can add other targets here if needed
   

   Note that the Prometheus-Destination does not require a special Remote Write configuration, but you could choose to scrape its own metrics or other targets.

Step 3: Start Both Instances

To start each Prometheus instance, navigate to their respective directories and use the following command in separate terminals:

./prometheus --config.file=prometheus.yml

Step 4: Verify Data Flow

1. Access the Prometheus-Source Dashboard: Open your web browser and go to http://localhost:9090. You should see the metrics from the defined scrape targets.

2. Access the Prometheus-Destination Dashboard: Navigate to http://localhost:9091, where you should be able to see the received metrics from Prometheus-Source.

3. Querying Data: You can verify that metrics are being sent correctly by constructing queries in both Prometheus UI instances. A simple query like http_requests_total should return results in the Prometheus-Destination if everything is set up correctly.

Conclusion

Using the Prometheus Remote Write API facilitates scalable metrics ingestion from multiple sources, allowing for robust monitoring and alerting architectures. Through this example, we’ve demonstrated how to configure two Prometheus instances, with one instance effectively sending metrics data to the other.

Additional Resources

• Prometheus Official Documentation
• Prometheus Remote Write
• Prometheus Community and Open Source Tools

By leveraging the Remote Write capabilities, teams can ensure more centralized observability and gain insights from their metrics more effectively.