配置AlertManager，用于处理Prometheus发送的告警信息。
下载AlertManager
在Alertmanager Releases页面找到最新的下载版本:
https://github.com/prometheus/alertmanager/releases/download/v0.29.0/alertmanager-0.29.0.linux-amd64.tar.gz
1
| wget https://github.com/prometheus/alertmanager/releases/download/v0.29.0/alertmanager-0.29.0.linux-amd64.tar.gz
|
解压安装
1
2
3
| tar -xvf alertmanager-0.29.0.linux-amd64.tar.gz
sudo cp alertmanager-0.29.0.linux-amd64/alertmanager /usr/local/bin/alertmanager
sudo cp alertmanager-0.29.0.linux-amd64/amtool /usr/local/bin/amtool
|
创建用户
1
| sudo useradd --no-create-home --shell /bin/false alertmanager
|
配置启动
创建数据存储目录。
1
2
| mkdir -p /var/lib/alertmanager
sudo chown -R alertmanager:alertmanager /var/lib/alertmanager
|
创建配置文件/etc/alertmanager/alertmanager.yml。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'web.hook'
receivers:
- name: 'web.hook'
webhook_configs:
- url: 'http://127.0.0.1:5001/'
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
|
修改用户
1
2
3
| sudo chown -R alertmanager:alertmanager /etc/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/amtool
|
创建systemd服务文件/etc/systemd/system/alertmanager.service。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| [Unit]
Description=Alert Manager
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=alertmanager
Group=alertmanager
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yml \
--storage.path=/var/lib/alertmanager
Restart=always
[Install]
WantedBy=multi-user.target
|
启动服务
1
2
3
4
5
| sudo systemctl daemon-reload
sudo systemctl start alertmanager
sudo systemctl enable alertmanager
sudo systemctl status alertmanager
sudo systemctl restart alertmanager
|
故障排查
查看服务日志
1
2
3
| # 查看系统日志中关于 alertmanager 的详细信息
sudo journalctl -u alertmanager.service -f
sudo journalctl -u alertmanager.service --no-pager -n 50
|
关联Prometheus
再次编辑Prometheus配置文件prometheus.yml，添加AlertManager配置。
1
2
3
4
5
| alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
|