Skip to main content

Logs from S3 Bucket

Ingest logs from AWS S3 bucket to Last9

Introduction​

This guide will walk through the steps required to grant necessary permissions to Last9 to read logs from an S3 bucket. Last9 can ingest logs from an AWS S3 bucket using evented architecture as follows.

Ingestion from S3 bucket

This architecture ensures that

  • Last9 automatically receives notifications for new files being added or existing files being updated.
  • Last9 starts ingesting logs for the files in the S3 bucket that are created or updated after the integration is enabled.
  • For past data, Last9 can do one time ingestion of files in the S3 bucket based on filters similar to rehydration.

Pre-requisites​

  1. You have an AWS S3 bucket.
  2. You have signed up for Last9.

Notify Last9 when new files are created in the S3 bucket​

  • The S3 ingestion works on evented arhictecture using SQS queue.
  • Last9 team will share ARN of the SQS queue.
  • Enable event notification in the S3 bucket for new objects. Add Event Notification
  • Choose destination for event notification to be ARN of the Last9 SQS queue. Event Destination

Read more about S3 Event notification here.

AWS STS Role​

Attach the following Trust relationship to a role with Last9 prinicpal user ARN. You can create the role to delegate permissions as per the AWS documentation.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "<last9_principal_user_arn>",
                "Service": "s3.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
info

Get the <last9_principal_user_arn> from Last9 team.

S3 Access policy​

Attach the following IAM policy to the role created in above step so that Last9 ingestion pipeline can access S3 bucket contents.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::bucket_name",
                "arn:aws:s3:::bucket_name/*"
            ]
        }
    ]
}

Share the role ARN with the Last9 team and the S3 bucket ARN for seamless ingestion of logs.

Troubleshooting​

Please get in touch with us on Discord or Email if you have any questions.