Skip to main content

How to enable EC2 Service Discovery with vmagent

This tutorial walks through setting up service disocvery for EC2 instances with vmagent.

Service Discovery​

In the context of monitoring, Service Discovery refers to automatically detecting devices, services, or systems in a network that need to be monitored. Service discovery is significant in cloud environments that use auto-scaling and EC2 instances. These environments often have instances that change rapidly, making manual tracking infeasible from a monitoring point of view. This document lists steps to enable service discovery of EC2 instances so new instances can be monitored as they are created and decommissioned instances can be removed from monitoring, tackling false alerts.

This document assumes that the EC2 instance service discovery will be set up for vmagent to send metrics to Levitate via Remote Write.

info

Follow our guide for setting up vmagent on an EC2 machine.

Given that vmagent is successfully running on an EC2 Instance, we need to make provisions for vmagent to discover other EC2 instances, that is, scrape targets based on ec2_sd_config.

Create ec2-trustee IAM role with assume role policy​

Go to AWS Console → IAM → Roles → Create Role

  • Select Trusted Entity. Select Trusted Entity
  • Do NOT add any permissions and click next. Add Permissions
  • Name, Review and Create. Create IAM Role Step 1 Create IAM Role Step 2

Attach ec2-trustee IAM role to vmagent EC2 Host​

EC2 Instances > Select vmagent Instance > Actions > Instance Settings

  • Modify IAM Role. Steps to update IAM role
  • Select ec2-trustee IAM role and Update. Modify IAM Role

Create vmagent-sd-role IAM role​

Go to AWS Console → IAM → Roles → Create Role

  • Select Trusted Entity > Custom Trust Policy with below trust policy. Custom Trust Policy

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::AWS_ACCOUNT_ID:role/ec2-trustee"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    }

  • Add Permissions

    1. Create Custom Policy with the below policy.

      {
          "Statement": [
              {
                  "Action": "ec2:Describe*",
                  "Effect": "Allow",
                  "Resource": "*"
              }
          ],
          "Version": "2012-10-17"
      }

    2. Select Policy and click next. Select Policy Step 1 Select Policy Step 2

    3. Name, Review and Create

      • Add vmagent-sd-role as the name of the role, review permissions and trusted entities and create role Add vmagent role

Use the vmagent-sd-role ARN in vmagent configuration​

Update the scrape_configs stanza in your vmagent.yaml with the ec2_sd_configs stanza as follows and restart vmagent.

# vmagent.yaml

# Check https://prometheus.io/docs/prometheus/latest/configuration/configuration for more details
scrape_configs:
  - job_name: 'node-exporter-sd'
    ec2_sd_configs:
      - region: ap-south-1
        role_arn: "__role_arn_with_ec2_read_access__"
        filters:
          - name: tag:namespace
            values:
              - node-exporter
        port: 9100

This will discover new EC2 instances automatically using the Service Discovery mechanism and their metrics will sent to Levitate from vmagent.