How to install Prometheus on CentOS 8

Learn how to install Prometheus on CentOS 8 with this comprehensive guide. Follow our step-by-step instructions to set up and configure Prometheus for monitoring your system’s performance and metrics. #centlinux #linux #prometheus

What is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit originally developed by SoundCloud. It has since become a standalone open-source project and is maintained by the Cloud Native Computing Foundation (CNCF). Prometheus is designed for reliability and scalability, making it a popular choice for monitoring complex and dynamic environments, especially those that rely on microservices architecture.

The project is written in Go and licensed under the Apache 2 License, with source code available on GitHub, and is a graduated project of the Cloud Native Computing Foundation, along with Kubernetes and Envoy.

Key features of Prometheus include:

1. Multi-dimensional Data Model: Prometheus stores all data as time series, identified by metric names and key/value pairs called labels. This model allows for flexible and powerful queries.
2. Powerful Query Language (PromQL): Prometheus provides PromQL, a functional query language that allows users to select and aggregate time series data in real-time.
3. Pull-Based Model: Prometheus scrapes metrics from instrumented targets, which allows it to collect data at regular intervals. This pull-based approach makes it easy to integrate with various services and systems.
4. Service Discovery: Prometheus supports service discovery mechanisms to automatically find targets to scrape, including static configuration, DNS, and integrations with various orchestration systems like Kubernetes.
5. Alerting: Prometheus has a built-in alerting mechanism with its Alertmanager component. It allows users to define alert rules based on Prometheus query results and send notifications via various channels like email, Slack, or custom webhooks.
6. Visualization: Prometheus can generate basic visualizations of data. Additionally, it integrates well with Grafana, a popular open-source visualization tool, for creating more sophisticated dashboards.
7. Self-Contained: Prometheus is designed to run standalone without external dependencies, making it easy to deploy and manage.

Prometheus is widely used for monitoring cloud-native applications, infrastructure, and more due to its robustness, flexibility, and scalability.

How to use Prometheus?

A typical monitoring platform with Prometheus is composed of multiple tools.

• Multiple exporters that typically run on the monitored host to export local metrics.
• Prometheus to centralize and store the metrics.
• Alertmanager to trigger alerts based on those metrics.
• Grafana to produce dashboards.
• PromQL is the query language used to create dashboards and alerts.

Prometheus Alternatives

There are several alternatives to Prometheus for monitoring and alerting in various environments. Each has its own strengths and weaknesses, catering to different use cases and preferences. Here are some popular Prometheus alternatives:

Grafana Loki

• Description: Loki is a log aggregation system designed to work seamlessly with Grafana. While it’s primarily for logs rather than metrics, it integrates well with Grafana for a comprehensive monitoring solution.
• Strengths: Tailored for logs, efficient storage, and querying of log data, integrates seamlessly with Grafana.

InfluxDB

• Description: A time-series database optimized for fast, high-availability storage and retrieval of time series data.
• Strengths: High performance, SQL-like query language (Flux), part of the TICK stack (Telegraf, InfluxDB, Chronograf, Kapacitor).

Graphite

• Description: A monitoring tool that stores numeric time-series data and renders graphs of this data on demand.
• Strengths: Simple design, well-established, good at handling large amounts of data, wide range of integrations.

Zabbix

• Description: An enterprise-level monitoring platform designed for monitoring and tracking the status of various network services, servers, and other network hardware.
• Strengths: Extensive out-of-the-box monitoring templates, robust alerting, strong SNMP support, good for infrastructure monitoring.

Nagios

• Description: One of the oldest and most established monitoring systems, primarily used for monitoring hosts, services, and network hardware.
• Strengths: Flexible, strong community support, numerous plugins and integrations, reliable alerting.

Datadog

• Description: A cloud-based monitoring and analytics platform for infrastructure, application performance, and log management.
• Strengths: Comprehensive monitoring capabilities, user-friendly interface, seamless integration with various cloud providers, powerful dashboarding and visualization tools.

Sensu

• Description: A monitoring system designed for monitoring servers, services, and application health across complex infrastructure.
• Strengths: Scalable, API-driven, strong support for event processing and custom checks, good for dynamic environments.

Sysdig

• Description: A monitoring and security platform designed for cloud-native environments, with deep visibility into containerized applications.
• Strengths: Strong container and Kubernetes support, detailed system call visibility, security features integrated with monitoring.

Elastic Stack (ELK)

• Description: Comprises Elasticsearch, Logstash, and Kibana, used for searching, analyzing, and visualizing log data in real-time.
• Strengths: Powerful full-text search, robust visualization capabilities with Kibana, wide range of data source integrations.

Each of these alternatives has its unique features and is suited for different monitoring requirements. The choice of tool depends on factors such as the complexity of the environment, specific monitoring needs, integration capabilities, and personal or organizational preferences.

Recommended Online Training: Prometheus Alerting and Monitoring

Environment Specification

We are using a minimal RHEL 8 virtual machine with following specifications.

• CPU – 3.4 Ghz (2 cores)
• Memory – 2 GB
• Storage – 20 GB
• Operating System – Red Hat Enterprise Linux 8.4 
• Hostname – prometheus-01.centlinux.com
• IP Address – 192.168.116.238 /24

Update your Linux Operating System

Connect with prometheus-01.centlinux.com as root user by using a ssh client.

Update your Linux operating system by using dnf command.

# dnf update -y

If the above command updates your Linux Kernel, then reboot your Linux operating system with new Kernel.

# reboot

Verify the Linux operating system and Kernel versions, that are being used in this article.

# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)

# uname -r
4.18.0-305.12.1.el8_4.x86_64

Switch SELinux Target Policy to Permissive Mode

Prometheus currently does not includes an official SELinux policy. Therefore, your have to disable SELinux or switch it into permissive mode for proper functioning of your network monitoring software.

You can execute following commands at Linux Bash prompt to switch SELinux target policy to permissive mode.

# sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/sysconfig/selinux
# setenforce permissive

Create Prometheus User and Directories

Create a Linux user to own Prometheus software and processes.

# useradd --no-create-home -s /bin/false prometheus

Create required Prometheus directories and adjust the ownership.

# mkdir /etc/prometheus
# mkdir /var/lib/prometheus
# chown prometheus:prometheus /etc/prometheus
# chown prometheus:prometheus /var/lib/prometheus

Install Prometheus on CentOS 8

You can free download systems monitoring tool from Prometheus official website.

You can use wget command to download Prometheus tarball directly from the Linux CLI. If you don’t have wget command, then install Prometheus on CentOS 8 by using dnf command.

# wget https://github.com/prometheus/prometheus/releases/download/v2.29.2/prometheus-2.29.2.linux-amd64.tar.gz -P /tmp
--2021-09-05 14:30:50--  https://github.com/prometheus/prometheus/releases/download/v2.29.2/prometheus-2.29.2.linux-amd64.tar.gz
...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.111.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 73175122 (70M) [application/octet-stream]
Saving to: ‘/tmp/prometheus-2.29.2.linux-amd64.tar.gz’

prometheus-2.29.2.l 100%[===================>]  69.79M   300KB/s    in 5m 21s

2021-09-05 14:36:13 (222 KB/s) - ‘/tmp/prometheus-2.29.2.linux-amd64.tar.gz’ saved [73175122/73175122]

Extract the downloaded tarball into the /var/lib/prometheus directory.

# tar -xf /tmp/prometheus-2.29.2.linux-amd64.tar.gz -C /var/lib/prometheus/ --strip-components=1

Adjust the ownership of the extracted files.

# chown -R prometheus:prometheus /var/lib/prometheus

Move Prometheus configuration file to /etc/prometheus directory.

# mv /var/lib/prometheus/prometheus.yml /etc/prometheus/

Check the default configurations in prometheus.yml file.

# grep -v '#' /etc/prometheus/prometheus.yml
global:

alerting:
  alertmanagers:
    - static_configs:
        - targets:

rule_files:

scrape_configs:
  - job_name: "prometheus"


    static_configs:
      - targets: ["localhost:9090"]

Create symbolic links for Prometheus commands at /usr/bin directory to make them globally executable from any path.

# cp -s /var/lib/prometheus/prometheus /usr/bin
# cp -s /var/lib/prometheus/promtool /usr/bin

Create Systemd Service Unit for Prometheus

To enable autostart of Prometheus server, you are required to create systemd service unit.

# vi /usr/lib/systemd/system/prometheus.service

Add following directives in this file.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/bin/prometheus 
--config.file /etc/prometheus/prometheus.yml 
--storage.tsdb.path /var/lib/prometheus/ 
--web.console.templates=/var/lib/prometheus/consoles 
--web.console.libraries=/var/lib/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Enable and start Prometheus service.

# systemctl enable --now prometheus.service
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /usr/lib/systemd/system/prometheus.service.

Configure Linux Firewall

This network monitoring server uses default port 9090/tcp. Therefore, it is necessary to allow it Linux firewall to make it accessible accross the network.

# firewall-cmd --permanent --add-port=9090/tcp
success
# firewall-cmd --reload
success

Access Prometheus GUI

Open URL http://prometheus-01.centlinux.com in a web browser.

Open Status > Targets from top menu.

Install Node Exporter on CentOS 8

Node Exporter is a Prometheus exporter for server level and  level metrics with configurable metric collectors. It helps us in measuring various server resources such as RAM, disk space, and CPU utilization.

You need to install node_exporter on your Prometheus server to gather metrics.

Create directory for Node Exporter software.

# mkdir -p /var/lib/prometheus/node_exporter

Download Node Exporter tarball from Prometheus website.

# wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz -P /tmp
--2021-09-05 15:09:51--  https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.109.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8898481 (8.5M) [application/octet-stream]
Saving to: ‘/tmp/node_exporter-1.2.2.linux-amd64.tar.gz’

node_exporter-1.2.2.linux-amd6 100%[==================================================>]   8.49M  74.0KB/s    in 2m 12s

2021-09-05 15:12:07 (65.7 KB/s) - ‘/tmp/node_exporter-1.2.2.linux-amd64.tar.gz’ saved [8898481/8898481]

Extract downloaded tarball into /var/lib/prometheus/node_exporter/ directory.

# tar xf /tmp/node_exporter-1.2.2.linux-amd64.tar.gz -C /var/lib/prometheus/node_exporter/ --strip-components=1

Adjust directory ownership.

# chown -R prometheus:prometheus /var/lib/prometheus/node_exporter/

Create a symbolic link for node_exporter at /usr/bin directory.

# cp -s /var/lib/prometheus/node_exporter/node_exporter /usr/bin/

Enable autostart of node_exporter process, create a systemd service unit.

# vi /usr/lib/systemd/system/node_exporter.service

Add following lines of code in this file.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/usr/bin/node_exporter

[Install]
WantedBy=default.target

Enable and start Node Exporter service.

# systemctl enable --now node_exporter.service
Created symlink /etc/systemd/system/default.target.wants/node_exporter.service → /usr/lib/systemd/system/node_exporter.service.

Configure Linux firewall to allow node_exporter default port 9100/tcp.

# firewall-cmd --permanent --add-port=9100/tcp
success
# firewall-cmd --reload
success

Edit Prometheus configuration file in vim text editor.

# vi /etc/prometheus/prometheus.yml

Add the node_exporter endpoint configuration in this file.

 - job_name: 'node_exporter'
    static_configs:
    - targets: ['localhost:9100']

Restart Prometheus service to load new configurations.

# systemctl restart prometheus.service

Open Status > Targets or refresh the web page, if already opened.

If you want to learn more about Prometheus software than you should read Prometheus: Up & Running (PAID LINK) by O’Reilly Media.

Final Thoughts

Installing Prometheus on CentOS 8 can significantly enhance your ability to monitor and manage system performance and metrics. By following the steps in this guide, you can set up Prometheus efficiently and start leveraging its powerful features for your monitoring needs. If you need personalized assistance or encounter any challenges during the installation, I offer professional services on Fiverr. Check out my Fiverr gig for expert help with installing and configuring Prometheus on your CentOS 8 system.