Overview

I covered VPC Security Groups in Part III and VPC Network Access Control Lists in Part IV. Now it's time to discuss VPC flow logs.

Flow Logs

In short, flow logs are a free feature within AWS that lets you capture network traffic metadata to and from your network interfaces within you VPCs.

You can create flow logs for either a subnet, VPC or on the Elastic Network Interface (ENI) itself. It is also possible to create flow logs for other services. Some of them include; ELB, EDS, and Workspaces.

If you decide to monitor a subnet, flow logs will be captured from each network interface within that subnet. Likewise, if you choose VPC, it will collect from each network interface within that VPC. If you choose to create flow logs for a single ENI, it will capture traffic from that ENI only.

If you decide to add more instances to either your subnet (if you choose that) or VPC (if you choose that) it will automatically start collecting flow logs from that new network interface.

Each network interface will have its own flow log stream. Flow logs do not capture real-time log streams for your ENIs. They capture traffic in capture windows of anywhere between 10 - 15 mins, and then relay that data to Cloud Watch.

If you decide to create flow logs on subnets, and let's say you have 10 subnets, multiple flow logs can publish to the same log group within Cloud Watch. This would allow you to have all of your flow logs going to a single log group if you decided to do that.

After you create a flow log, if you wanted to modify it you cannot. You would need to delete it and make the modification while you create a new one.

To create flow logs you need a few pieces of information first at a minimum. I created a video below that shows you how to configure it.

• You need to decide where you want to attach (VPC, Subnet, or network interface (ENI))
• Create and Name your Cloud Watch Log Group. If you do not specify a log group, one will be created for you. You can have up to 5,000
• Create an IAM role that grants permissions to publish flow logs to the Cloud Watch log group
  • log:CreateLogGroup
  • log:CreateLogStream
  • log:PutLogEvents
  • log:DescribeLogGroups
  • log:DescribeLogStreams
• Modify the trust relationship to allow vpc-flow-logs the assume role permission

Cloud Watch

Flow logs are stored using Cloud Watch. For those of you unfamiliar with Cloud Watch, for now I will just copy/paste from the documentation, "Amazon CloudWatch monitors your Amazon Web Services (AWS) resources and the applications you run on AWS in real time. You can use CloudWatch to collect and track metrics, which are variables you can measure for your resources and applications."

When a flow log has been created, you can view it via Cloud Watch. I mentioned that flow logs are free, but there is a cost associated with cloud watch usage.

You can read more about CloudWatch pricing here, but in my us-east-1 region it's; $0.00 per request - first 1,000,000 requests, AmazonCloudWatch PutLogEvent's first 5GB per month of log data ingested is free, and AmazonCloudWatch TimedStorage-ByteHr's first 5GB-mo per month of logs storage is free.

Enabling Flow Logs

I went ahead and made a video showing you how to get flow logs configured. There would have been too many screen shots.

In the video I show you how to configure each of the items I mentioned above in the Flow Log section.

A quick point of clarification. Although, not in the video, because I did the voice over after I recorded it, you could have made the inline policy on the FlowLogs-Role more restrictive on the resource.

The resource element identifies (via ARN), the object or objects that the statement covers.

In the video I used.

"Resource": "*"

If you were going to have all of your flow logs go into the log group, FlowLogs, and region, us-east-1, you could have used a more restrictive resource like this.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:logs:us-east-1:aws-account-id:log-group:FlowLogs:*"
        }
    ]
}

Flow Log Structure

So now that you have your flow logs going into CloudWatch, what are they saying?

The flow log record, which represents a network flow in your flow log is constructed of the following metadata:

• Version
• Account Id
• Interface Id
• Source Address
• Destination Address
• Source Port
• Destination Port
• Protocol
• Packets
• Bytes
• Start Time
• End Time
• Action
• Log Status

Flow Traffic Not Captured

There is some traffic that is not captured by flow logs that you should be aware of as well.

• Traffic from instances going to Amazon DNS. It will capture DNS traffic if you are using your own DNS servers.
• Amazon Windows licensing activation traffic
• Traffic to/from 169.254.169.254 for instance metadata
• DHCP traffic
• Traffic to AWS reserved IP addresses.

Visual

And now we have our updated diagram, which depicts our subnet flow logs being sent to CloudWatch.

I've also gone ahead and added an RDS instance to our diagram because I moved my, MYSQL DB off the local Webserver instance.

I'll continue using this diagram of a mock environment in the future as we layer on more and more technologies.

Summary

Now you should have functional flow logging. In Part VI, I'll show you how to get these logs into Elasticsearch for processing and in Part VII I will show you how you can query exported flow logs in your S3 bucket using Athena.

If you find this post useful and educational you can donate via PayPal.Me here: $1, $2, $3, $4, $5 or Custom.

Enjoy!

References

Amazon Virtual Private Cloud User Guide