Visualizing Security Lake Log Data: Easy Steps with Athena and QuickSight for AWS Organizations
Step-by-Step Guide: Analyzing Security Lake Logs with AWS
Introduction
In today's rapidly evolving digital landscape, organizations face increasingly complex challenges in maintaining robust cybersecurity measures. One vital aspect of this endeavor is the ability to effectively analyze and visualize security log data, enabling timely threat detection and informed decision-making.
In this article, I will guide you through the process of visualizing security log data using QuickSight in AWS Organizations. By integrating AWS services such as Security Lake, Amazon S3, AWS Glue, AWS Athena, AWS Data Lake Formation, and QuickSight, you can gain valuable insights from your security logs. Whether you're an individual or an organization, this step-by-step guide will help you harness the power of AWS to enhance your cybersecurity efforts.
Prerequisites
The Basic Knowledge of Amazon Security Lake, Athena, Lake Formation, QuickSight, Amazon S3.
In the Source Account, Security Lake should be enabled in the Management Account within AWS Organizations and the security lake delegated admin account should be configured.
Security Lake Sources should be enabled in the Source Account.
The S3 bucket cross-account policy should be configured in the Source Account.
Amazon Athena should be configured in the Target Account.
Amazon QuickSight should be enabled in the Target Account.
Related Services
Amazon EventBridge
AWS Glue
AWS Lake Formation
AWS Lambda
AWS Resource Access Manager
Amazon S3
Reference Documentation:
Source Account Configuration
Enabling Sources in Security Lake and Configuring S3 Bucket Policy for Cross-account Access.
Log in to the Security Lake Management Console.
From the Left Navigation Pane, click on Sources and select the sources and region where you would like to enable.
- After enabling the sources, Security Lake will create a Data lake with an IAM role that has access to the S3 bucket. The bucket might appear as: aws-security-data-lake-us-east-1-xxxxxxxxxxxx
Configure the bucket policy to grant access to the Target account. (Please modify the action based on your needs)
"Sid": "CrossAccountAccessRole",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam:<Target Account>:root",
"arn:aws:iam:<Target Account>:role/XXXXXXXXXX"
]
},
"Action":"s3:*",
"Resource":[
"arn:aws:s3:::aws-security-data-lake-us-east-1-xxxxxxxxxxxx",
"arn:aws:s3:::aws-security-data-lake-us-east-1-xxxxxxxxxxxx/*"
]
}
]
}
Configuring Lake Formation Administrator and Cross-account Sharing.
Go to the AWS Lake Formation Management Console.
From the Left Navigation Pane, click on Administrative roles and tasks.
Click on Add Data Lake administrators.
Access type: Data Lake administrator
IAM users and roles: The role or user you would like to grant access.
Click on Confirm.
Granting permissions on tables.
Go to the Database section and click on the Security Lake Database.
In Principals, select External accounts. Enter the target account ID.
In LF-Tags or catalog resources, select Named Data Catalog resources.
In Databases, choose the database that Security Lake created.
In Tables, choose All tables.
For the Table permissions, grant Select and Describe for Table and Grantable permissions.
For Data permissions, select All data access.
If you only want to grant data access to specific columns, then choose Column-based access.
Granting permissions on the database.
Go to the Database section and click on the Security Lake Database.
In Principals, Select External accounts. Enter the target account ID.
In LF-Tags or catalog resources, select "Named data catalog resources".
In Databases, choose the database that Security Lake created.
For Database permissions, grant Describe for Database and Grantable permissions.
Viewing and Verifying the permissions.
Click on the Database and tables orderly and click on actions drop-down menu and click on view and verify.
Your permissions might look like as follows:
| Principal | Principal Type | Resource Type | Database | Table | Resource | Permissions | Grantable |
| Target Account Id | AWS Account/IAM role | Database | Database_Name | - | Resource_Name | Describe | Describe |
| Target Account Id | AWS Account/IAM role | Table | Database_Name | ALL_TABLES/Table_Name | ALL_TABLES/Table_Name | Describe | Describe |
| Target Account Id | AWS Account/IAM role | Column | Database_Name | ALL_TABLES/Table_Name | Included: All | Select | Select |
Principal Type: IAM Role (If the permission is granted to the specific IAM Role.)
Resource: Table_Name (If the permission is granted to the specific table.)
Target Account Configuration
Reviewing and Accepting RAM Invitation
- Log in to the AWS Management Console. In the Resource Access Manager (RAM) console, review and accept the invitations corresponding to the shared database and tables from the source account.
| Name | ID | Owner | Status |
| LakeFormation-XX-XXXXXXXXXX | XXXXX-XXXX-XXX-XX-XXXX | Source Account | Pending |
- After accepting the invitation, you can see the Database and Tables on the Target Account.
Creating Database Resource Link.
Open the Lake Formation Console in the Target Account.
Choose Create Database and create a resource link.
In the Resource link name, enter your preferred link name.
In the Shared database's region, select the region of the shared database.
In the Shared database, choose the shared database name of the source account.
Click Create.
Querying the Database and Tables from Athena.
Open the Athena Console in the Target Account.
You can now see and query the database and tables shared from the Source Account.
SELECT * FROM "database_name"."table_name" limit 10;
Integration and Visualization with Amazon QuickSight.
Necessary Configurations (Target Account)
Ensure the Athena IAM Role has the necessary permissions for Amazon QuickSight.
Ensure you have granted Database and Tables Permissions to QuickSight. (Lake Formation)
If you have created views in Athena, then ensure the necessary permissions are granted to QuickSight. (SELECT/DESCRIBE as view and grantable permissions.)
Grant database (DESCRIBE: Leave Tables blank when granting permission to the database only), tables(SELECT/DESCRIBE for the table and grantable permissions), and views permission to QuickSight. To grant permissions to QuickSight, the process is the same as granting permissions to the external account except In Principal section, ensure you have chosen "SAML users and groups". To confirm your QuickSight Arn:
[cloudshell-user@ip-10-5-99-172 ~]$ aws quicksight list-users --aws-account-id XXXXXXXXXXXX --namespace default --region us-east-1
In QuickSight, ensure you have selected the source S3 bucket and configured the Athena database, tables, and views for creating a dataset.
For a cross-account S3 bucket, manual entry of the bucket name is required. However, if the source S3 bucket is in the same account, manual entry of the bucket name is not necessary.
When creating a dataset from Athena, provide the necessary parameter values such as Data source name and Athena workgroup. If the permissions are correctly set up as in the above steps, you will be able to see the database, tables, and views in the data source.
By following this guidance, you can now create analyses and visualize the security lake data using Amazon QuickSight. Amazon QuickSight offers two query modes: SPICE and Direct Query. Select the query mode based on your workloads. For further information on Amazon QuickSight, please consult the AWS Documentation.