Enable EC2 Service Discovery with vmagent
This tutorial walks through setting up service disocvery for EC2 instances with vmagent.
Service Discovery
In the context of monitoring, Service Discovery refers to automatically detecting devices, services, or systems in a network that need to be monitored. Service discovery is significant in cloud environments that use auto-scaling and EC2 instances. These environments often have instances that change rapidly, making manual tracking infeasible from a monitoring point of view. This document lists steps to enable service discovery of EC2 instances so new instances can be monitored as they are created and decommissioned instances can be removed from monitoring, tackling false alerts.
This document assumes that the EC2 instance service discovery will be set up for vmagent to send metrics to Levitate via Remote Write.
Follow our guide for setting up vmagent on an EC2 machine.
Given that vmagent is successfully running on an EC2 Instance, we need to make provisions for vmagent to discover other EC2 instances, that is, scrape targets based on ec2_sd_config.
Create ec2-trustee
IAM role with assume role policy
Go to AWS Console → IAM → Roles → Create Role
- Select Trusted Entity
- Do NOT add any permissions and click next
- Name, Review and Create
Attach ec2-trustee
IAM role to vmagent EC2 Host
EC2 Instances > Select vmagent Instance > Actions > Instance Settings
- Modify IAM Role
- Select
ec2-trustee
IAM role and Update
Create vmagent-sd-role
IAM role
Go to AWS Console → IAM → Roles → Create Role
-
Select Trusted Entity > Custom Trust Policy with below trust policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::AWS_ACCOUNT_ID:role/ec2-trustee"
},
"Action": "sts:AssumeRole"
}
]
} -
Add Permissions
-
Create Custom Policy with the below policy
{
"Statement": [
{
"Action": "ec2:Describe*",
"Effect": "Allow",
"Resource": "*"
}
],
"Version": "2012-10-17"
} -
Select Policy and click next
-
Name, Review and Create
- Add
vmagent-sd-role
as the name of the role, review permissions and trusted entities and create role
- Add
-
Use the vmagent-sd-role
ARN in vmagent configuration
Update the scrape_configs
stanza in your vmagent.yaml
with the ec2_sd_configs
stanza as follows and restart vmagent.
# vmagent.yaml
# Check https://prometheus.io/docs/prometheus/latest/configuration/configuration for more details
scrape_configs:
- job_name: "node-exporter-sd"
ec2_sd_configs:
- region: ap-south-1
role_arn: "__role_arn_with_ec2_read_access__"
filters:
- name: tag:namespace
values:
- node-exporter
port: 9100
This will discover new EC2 instances automatically using the Service Discovery mechanism and their metrics will sent to Levitate from vmagent.