Prometheus+Grafana+Alertmanager+Pushgateway安装

一、使用docker compose安装

选一个心仪的位置存放Prometheus等配置文件

mkdir /home/prometheus
mkdir /home/grafana
mkdir /home/alertmanager
mkdir /home/pushgateway

生成Prometheus的配置文件

cat > /home/prometheus/prometheus.yml << EOF
# my global config
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: "edocyun-prom-stack"

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:8094

rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

scrape_configs:
  - job_name: "prom-stack"
    static_configs:
      - targets:
          - 192.168.1.2:9090
          - 192.168.1.2:9091
          - 192.168.1.2:9093
          - 192.168.1.2:3000

  - job_name: "pve"
    static_configs:
      - targets:
          - 192.168.1.2:9100

  - job_name: "cloud"
    scheme: https
    tls_config:
      ca_file: /home/prometheus/www.rclandy.com.pem
    static_configs:
      - targets:
          - rclandy.com:9100
    basic_auth:
      username: cloud
      password: 123456  #此处填明文密码
EOF 

生成Alertmanager的配置文件

cat > /home/alertmanager/alertmanager.yml << EOF
global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'

route:
  group_by: ['cluster']
  receiver: team-a

receivers:
- name: 'team-a'
  email_configs:
  - to: 'team-a@example.org'
EOF 

在portainer新建一个stack，为了方便观察下载进度，可以提前pull好镜像

docker pull prom/prometheus:latest
docker pull prom/pushgateway:latest
docker pull prom/alertmanager:latest
docker pull grafana/grafana:latest

docker-compose.yml内容，端口号可以根据需要自行修改

services:

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    command: --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d
    user: 0:0
    ports:
      - 9090:9090
    volumes:
      - /home/prometheus:/etc/prometheus
      - /home/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - /home/prometheus/alerts/:/etc/prometheus/rules.d
      - /home/prometheus/data/:/prometheus
    network_mode: bridge
    restart: unless-stopped

  pushgateway:
    image: prom/pushgateway:latest
    container_name: pushgateway
    command: --persistence.file=/pushgateway/pushgateway.data
    ports:
      - 9091:9091
    volumes:
      - /home/pushgateway:/pushgateway
    network_mode: bridge
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    command: --config.file=/etc/alertmanager/alertmanager.yml
    ports:
      - 9093:9093
    volumes:
      - /home/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    network_mode: bridge
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - 3000:3000
    volumes:
      - /home/grafana:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=rc123456
    network_mode: bridge
    restart: unless-stopped

此时程序应该可以顺利跑起来，访问192.168.1.2:9090即可进入prometheus的控制面板查看相关信息，其中Status项下的Targets可以看到在prometheus配置文件中配置的各项子机的连接情况。

二、子机node_exporter安装

安装apache2-utils

apt install apache2-utils -y

执行命令后输入如123456，得到$2y$开头的加密密码，把密码复制出来备用（如果显示成如下的样子，要注意别多复制系统用户名比如root）

htpasswd -nBC 12 '' | tr -d ':\n'
New password: 
Re-type new password: 
$2y$12$QK8pna3SJoixhjCFHsxknuyYYCy2CD6Gp6vB.JD062MvtbKUmbfmmroot@debian:~#

取值为

$2y$12$QK8pna3SJoixhjCFHsxknuyYYCy2CD6Gp6vB.JD062MvtbKUmbfmm

新建systemctl控制文件

vim /etc/systemd/system/node_exporter.service

复制粘贴以下内容，ExecStart的执行地址需与后续node_exporter存放地址一致

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter/node_exporter --web.config.file=/usr/local/bin/node_exporter/config.yaml --web.listen-address=:9100

[Install]
WantedBy=multi-user.target

web.config.file ：使用TLS和AUTH通信的需使用该参数，局域网内用IP转发的可以不用

web.listen-address ：如果使用默认9100端口可以不加web.listen-address参数，使用其他端口，请在防火墙开放对应端口后修改以上参数

选一个心仪的位置下载存放node_exporter

cd /usr/local/bin/
mkdir node_exporter
cd node_exporter

需要使用tls和auth的，生成如下配置文件，并将对应的证书文件上传到配置文件所在文件夹

cat>/usr/local/bin/node_exporter/config.yaml<<EOF
tls_server_config:
  cert_file: rclandy.com.crt 
  key_file: rclandy.com.key
basic_auth_users:
  cloud: $2y$12$QK8pna3SJoixhjCFHsxknuyYYCy2CD6Gp6vB.JD062MvtbKUmbfmm
EOF

basic_auth_users里的cloud即是前面prometheus.yaml里的username，后面一串加密是password

新建执行脚本，便于后续更新

vim update-node-exporter.sh

复制粘贴脚本内容

#!/bin/sh

cd /usr/local/bin/node_exporter
tag=$(curl -sL https://api.github.com/repos/prometheus/node_exporter/releases/latest | jq -r ".tag_name")
echo "当前版本"$tag

if [ -f version.txt ]; then    
echo "文件存在"
else
cat>/usr/local/bin/node_exporter/version.txt<<EOF
new
EOF
fi

if [ $tag != $(cat /usr/local/bin/node_exporter/version.txt) ]; then
cat>/usr/local/bin/node_exporter/version.txt<<EOF
$tag
EOF
wget https://github.com/prometheus/node_exporter/releases/download/${tag}/node_exporter-${tag#*v}.linux-amd64.tar.gz
tar -xvf node_exporter-${tag#*v}.linux-amd64.tar.gz
systemctl stop node_exporter
rm -rf node_exporter
cd node_exporter-${tag#*v}.linux-amd64
mv node_exporter /usr/local/bin/node_exporter/node_exporter
cd /usr/local/bin/node_exporter
chmod a+x node_exporter
systemctl start node_exporter
rm -rf systemctl stop node_exporter-*
else
	echo "已是最新版本"
fi

保存赋权并运行

chmod a+x update-node-exporter.sh
./update-node-exporter.sh

node_exporter github地址

https://github.com/prometheus/node_exporter/releases