Step-by-Step Guide to Export Compute Optimizer Logs of Multi Accounts to a Specific S3 Bucket in AWS Organization

Introduction

AWS Compute Optimizer is a powerful tool designed to analyze the configuration and utilization metrics of your AWS resources, providing recommendations to optimize performance and reduce costs. However, consolidating Compute Optimizer logs from multiple accounts into a central location for analysis and monitoring can be a challenging task.

In this blog post, I'll guide you through the process of exporting Compute Optimizer logs from multiple accounts within an AWS organization and storing them in a specific S3 bucket in another account. By centralizing these logs, you can easily track and analyze optimization recommendations across all your accounts, leading to better resource utilization and cost savings.

Preconditions:

• Ensure Compute Optimizer is enabled in AWS Organizations in the Management account.

• AWS resources reside in Member accounts.

• Create an S3 bucket in the target account where you want to export logs.

• Configure an S3 bucket policy in the target account.

Target Account Configuration

• Configure the following S3 bucket policy for exporting logs from Compute Optimizer.

  • You can also leverage wildcards in the bucket policy to enable dynamic prefixes, such as organizing exports by date or other criteria, for better file organization and easier analysis, such as "arn:aws:s3:::myBucketName/**/compute-optimizer/myAccountID/*"

Policy option 1: Using an optional prefix

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {"Service": "compute-optimizer.amazonaws.com"},
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::myBucketName"
        },
        {
            "Effect": "Allow",
            "Principal": {"Service": "compute-optimizer.amazonaws.com"},
            "Action": "s3:GetBucketPolicyStatus",
            "Resource": "arn:aws:s3:::myBucketName"
        },
        {
            "Effect": "Allow",
            "Principal": {"Service": "compute-optimizer.amazonaws.com"},
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::myBucketName/optionalPrefix/compute-optimizer/myAccountID/*",
            "Condition": {"StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control",
                    "aws:SourceAccount": "myAccountID",
                    "aws:SourceArn": "arn:aws:compute-optimizer:myRegion:myAccountID:*"
                }
            }
        }
    ]
}

• Replace myBucketName with the name of your bucket.

• Replace optionalPrefix with the optional object prefix.

• Replace myRegion with the source AWS Region.

• Replace myAccountID with the account number of the requester of the export job.

Note:

The compute-optimizer/myAccountID/ component isn’t part of the optional prefix. Compute Optimizer creates the optimizer/myAccountID/part of the bucket path for you that's added to the prefix that you specify.

Policy option 2: No object prefix

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {"Service": "compute-optimizer.amazonaws.com"},
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::myBucketName"
        },
        {
            "Effect": "Allow",
            "Principal": {"Service": "compute-optimizer.amazonaws.com"},
            "Action": "s3:GetBucketPolicyStatus",
            "Resource": "arn:aws:s3:::myBucketName"
        },
        {
            "Effect": "Allow",
            "Principal": {"Service": "compute-optimizer.amazonaws.com"},
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::myBucketName/compute-optimizer/myAccountID/*",
            "Condition": {"StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control",
                    "aws:SourceAccount": "myAccountID",
                    "aws:SourceArn": "arn:aws:compute-optimizer:myRegion:myAccountID:*"
                }
            }
        }
    ]
}

Reference :

• Amazon S3 bucket policy for AWS Compute Optimizer

Management Account Configuration

• Log in to the Computer Optimizer Management Console.

• Click on "Export Recommendations".

• Select the target bucket destination and region for exporting the logs.

• Enter the Object prefix - optional configured in the bucket policy. Note: Only if you use an optional prefix.

• Check "Include recommendations for all member accounts in the Organization".

• Choose the recommendation columns based on your requirements in the export file. Note: Export jobs might take time based on the number of recommendation columns.

• Click on "Export".

Target Account

Navigate to the Target Account's destination bucket and verify the successful export of logs.

Conclusion

In this blog post, we have walked through the step-by-step process of exporting Compute Optimizer logs from multiple accounts within an AWS organization to a specific S3 bucket in another account. By centralizing these logs, you can gain valuable insights and recommendations to optimize your AWS resources across all your accounts, leading to improved performance and cost savings. For further analysis and visualization of these logs, you can also leverage tools like Amazon Athena to query the data directly in S3 or Amazon QuickSight to create interactive dashboards and reports, helping you make informed decisions more effectively.

I hope this step-by-step guide has provided you with the knowledge and confidence to export Compute Optimizer logs across multiple accounts in your AWS organization. By implementing this solution, you are taking a significant step towards optimizing your compute resources and maximizing the value of your AWS infrastructure. Happy optimizing!