EC2 Instance

Send logs and hostmetrics from AWS EC2 instance using OpenTelemetry

Introduction

This guide will help you instrument your AWS EC2 instance with OpenTelemetry and smoothly send the logs and host metrics to a Last9.

Pre-requisites

1. You have a AWS EC2 instance and workload running in it.

2. You have signed up for Last9, created a cluster, and obtained the following OTLP credentials from the Integrations page:

   • endpoint
   • auth_header

3. Optional: Attach an IAM policy to the EC2 instance with ec2:DescribeTags permission. This is needed for resource detection processor to fetch additional tags associated with the EC2 instance which can be used as additional resource attributes.

4. Install Otel Collector. There are multiple ways to install the Otel Collector. One possible way of installing it using rpm is as follows. Every Collector release includes APK, DEB and RPM packaging for Linux amd64/arm64/i386 systems.

Note: systemd is required for automatic service configuration.

sudo rpm -ivh otelcol-contrib_0.103.0_linux_amd64.rpm

More installation options can be found here.

Note: We recommend installing otel-collector-contrib version 0.103.0.

Sample Otel Collector Configuration

The default path for otel config is /etc/otelcol-contrib/config.yaml.

You can edit it and update it with below configuration. The configuration is annotated with comments which should be addressed before applying the configuration. The configuration for operators is especially important to extract the timestamp and severity.

For JSON logs, you can use json_parser and use its keys for log attributes. For non-structured logs, use the regex_parser.

The configuration provdies sample example of both JSON parser and regex parsers.

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.logical.count:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
          system.memory.limit:
            enabled: true
      load:
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      network:
      paging:
      processes:
      process:
        mute_process_user_error: true
        metrics:
          process.cpu.utilization:
            enabled: true
          process.memory.utilization:
            enabled: true
          process.threads:
            enabled: true
          process.paging.faults:
            enabled: true
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Detailed configuration options can be found at https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver
  filelog:
    # File path pattern to read logs from. Update this to the destination from where you want to read logs.
    include: [/tmp/*.log]
    exclude: [/home/ubuntu/exclude/*.log]
    include_file_path: true
    # attributes:
      # A map of key: value pairs to add to the entry's attributes.
    # resource:
      # A map of key: value pairs to add to the entry's resource.
    retry_on_failure:
      enabled:
        true
    operators:
      # For logs in JSON format
      - type: json_parser
        severity:
          parse_from: attributes.level
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'
      # For plain text logs
      - type: regex_parser
        regex: '(?P<level>^[A-Za-z]+) (?P<time>[0-9]{4}-[0-9]{2}-[0-9]{2}.*[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{1,3})?)'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d-%H:%M:%S'
        severity:
          parse_from: attributes.level

processors:
  batch:
    timeout: 5s
    send_batch_size: 10000
    send_batch_max_size: 10000
  resourcedetection/ec2:
    detectors: ["ec2"]
    ec2:
      # A list of regex's to match tag keys to add as resource attributes can be specified
      tags:
        # This means you have a tag `Name` associated with the EC2 Instance.
        - ^Name$
        # This means you have a tag `app` associated with the EC2 Instance.
        - ^app$
  transform/ec2:
    error_mode: ignore
    log_statements:
      - context: resource
        statements:
          # Set Service name as the `Name` tag associated with the EC2 Instance. The format is `ec2.tag.<tag_name>`.
          - set(attributes["service.name"], attributes["ec2.tag.Name"])
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
  transform/hostmetrics:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["host.name"], resource.attributes["host.name"])
          - set(attributes["process.command"], resource.attributes["process.command"])
          - set(attributes["process.command_line"], resource.attributes["process.command_line"])
          - set(attributes["process.executable.name"], resource.attributes["process.executable.name"])
          - set(attributes["process.executable.path"], resource.attributes["process.executable.path"])
          - set(attributes["process.owner"], resource.attributes["process.owner"])
          - set(attributes["process.parent_pid"], resource.attributes["process.parent_pid"])
          - set(attributes["process.pid"], resource.attributes["process.pid"])

exporters:
  debug:
    verbosity: detailed
  otlp/last9:
    endpoint: "<last9_endpoint>"
    headers:
      "Authorization": "<last9_auth_header>"

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resourcedetection/ec2, transform/ec2, batch]
      exporters: [otlp/last9]
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection/system, transform/hostmetrics, batch]
      exporters: [otlp/last9]

Running otel collector

Run the otel collector using systemctl command.

sudo systemctl start otelcol-contrib
sudo systemctl status otelcol-contrib
sudo systemctl restart otelcol-contrib

Checking logs of otel collector

sudo journalctl -u otelcol-contrib -f

This will enable logs and host metrics to be sent to Last9. You can visit the Logs Dashboard and Metrics Dashboard to see the data in action.

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.