Monitor VPC Flow Logs in S3 Bucket

Overview

A virtual private cloud (VPC) is a virtual network dedicated to your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. To monitor the VPC Flow, logs of VPC which are stored in S3 bucket are gathered by sfPoller and displayed within SnappyFlow dashboard.

Prerequisites

1. To collect logs of VPC, it is necessary to have an IAM Role with S3 Bucket Read Only access and sfPoller set up within your AWS environment. Click here to learn more about setting up sfPoller in your AWS environment.

2. To effectively monitor VPC flow logs, follow these steps:

   • Begin by creating flow logs for your VPC. Select the VPC of interest and configure the destination to be an S3 bucket.

   • Ensure that you've already created the S3 bucket you intend to use for storing the flow logs before assigning it as the destination.

   • Once the S3 bucket is in place and properly configured, the flow logs will automatically be stored in the designated destination bucket.

3. Attach the policy to a dedicated IAM Role for read-only access.

   • Required Permission policies: AmazonS3ReadOnlyAccess

Configure sfPoller to collect VPC Flow logs

Follow the below step to add endpoints and plugins:

1. In the Application tab of sfPoller, navigate to your Project > Application.

2. Click on the Application, it will take you to the Endpoint page.

3. Click the Add Endpoint button, add the following data, and save.

   • Service Type: Select AWS Service
   • Account Name: Select an account name. Example: aws
   • Endpoint Type: Select VPCFlow
   • Name: Give an unique name to the endpoint
   • Instance Name: Name of the bucket that need to be monitored
   • Folder Name: Name of the folder or path to be monitored
   
   

4. In the Plugins window, click the +Add button.

5. In the Add Plugin window, add the below details to collect logs of VPC Flow.

   • Plugin Type: Select Logger
   • Plugin: Select cloudwatch-vpcflow-logs
   • parallaismperiod (optional): Choose the period as day, week, month or year. By default value is an empty string ""
   • parallaismfactor (optional): Choose an integer value. It indicates the number of threads to be run to collect the Older data.
   • ignoreolder (optional): Give True or False.
   • partitionstrategy (optional): Accepted value year/month/day, Considered if ignoreolder set to true only. (skips collecting the Older data from starting of the vpc bucket logs)
   • Interval: Choose an interval value. The minimum value for the interval is 300
   • Status: By default, the status is Enabled
   
   
   

6. Select the Save button.

7. Click the global Save button in the window's top right corner to save all the changes made so far.

View VPC Flow Logs

Follow the below steps to view the VPC Flow Logs collected from the S3 bucket.

1. Go to the Application tab in SnappyFlow and navigate to your Project > Application > Dashboard.

2. You can view VPC Flow Logs stored in the Log Management section.

   
   

3. To access the unprocessed data gathered from the plugins, navigate to the Browse data section and choose the Index: Log, Instance: Endpoint, Plugin, and Document Type.

Template Details

Template	Plugin	Document Type	Description
-	cloudwatch-vpcflow-logs	logStreams	Collects logs from s3bucket