I do not consider these next steps complex but I still wanted to document it because I didn't really find much online except a few Stack Overflow comments here and there and I had it in my draft posts for a few days. So this is my effort to consolidate some information for others to use.

So what's Pyinstaller? According to their website, "PyInstaller is a program that converts (packages) Python programs into stand-alone executables, under Windows, Linux, Mac OS X, Solaris and AIX."

Simply put for those running Windows it takes your .py file and turns it into an EXE file so you can run Python files on your system without needing Python installed.

Unfortunately (or fortunately for job security purposes) malware takes advantage of Python's ease of use and it's ability to be packaged into an exe easily and then subsequently used to pwn people and organizations.

The sample I will look at now is: MD5 (test.exe) = 43adebf0983a5fc74d5b696bcbfd5f93, which isn't anything special. It just happened to be a sample that triggered on one of my VT Yara rules so I decided to take a look at it.

First you will need a couple tools for this little demo.

archive_viewer.py - This will allow you to "expand" the .exe file and see into it similar to how you would with an archive tool to look into a .zip, .7z, etc.

pyinstxtractor.py - This one works more often than archive_viewer.py, so it's good to have a couple to choose from.

I haven't run into any pieces of malware where either one of these didn't do their job. I'm sure there are samples, but I haven't found them yet. Please let me know if you do.

So now, let's take a look.

python pyinstxtractor.py test.exe

Successfully extracted Pyinstaller archive : test.exe

Now in the directory where you ran pyinstxtractor you should have some files. Let's ignore all of them except "test". Open it up in a text editor.

It looks like a blob of junk, yeah? Something similar to this. I've removed some code to conserve space.

from Crypto.Cipher import AES as DvMfw;from datetime import date;from base64 import b64decode as fBPSR;import struct, socket, binascii, ctypes, random, time;from datetime import datetime
exec(fBPSR("ZXhlYyhEdk=="))

Let's make it a bit more readable:

from Crypto.Cipher import AES as DvMfw;
from datetime import date;
from base64 import b64decode as fBPSR;
import struct, socket, binascii, ctypes, random, time;
from datetime import datetime
exec(fBPSR("ZXhlYyhEdk=="))

So we can see that fBPSR is actually base64decode, which makes since considering there is a big base64 blob in the code. Let's clean it up a bit more and then run it.

import struct, socket, binascii, ctypes, random, time
from Crypto.Cipher import AES as DvMfw
from datetime import date
from base64 import b64decode
from datetime import datetime
print(b64decode("ZXhlYyhEdk=="))

Now we have some more obfuscated code, but we are getting somewhere.

exec(DvMfw.new("@ZKWogab)$o8Gi1)cA6zo3(.P244jRdn").decrypt(fBPSR("oBpFHK")).rstrip('{'))

Let's fix it up again. I removed some of the base64 blob to preserve space.

from Crypto.Cipher import AES
from datetime import date
from base64 import b64decode
import struct, socket, binascii, ctypes, random, time
from datetime import datetime
print(AES.new("@ZKWogab)$o8Gi1)cA6zo3(.P244jRdn").decrypt(b64decode("oBpFHK")).rstrip('{'))

And now you have some readable code that you can continue to analyze. I actually watched this person upload this code to VT about 10 times. Ultimately they set the IP to some no-ip domain, but it appeared down/not responding when I attempted to infect my lab machine(s).

So yeah, in case you're curious how this works that's pretty much it. It's pretty simple. This is of course if the attackers aren't using some custom packaging tool and they are relying on stock pyinstaller to build their EXEs.

So I was reading Trend's blog post Without a Trace: Fileless Malware Spotted in the Wild and although not totally new, it got me thinking a bit. I downloaded some of the MD5s they posted and started infecting my lab box. I noticed a couple different behaviors when infecting my lab machine with two different samples.

• As mentioned in the blog: Creations of Rc4Encoded32 and Rc4Encoded64 registry values in HKEY_CURRENT_USER\Software\Microsoft\Active Setup\Installed Components{Bot GUID}
• A couple of the hashes created a Software entry: Software\ xsw\binaryImage32 and the data was an EXE.

How the malware uses these blobs of data is not the objective of this post. You can read the Trend blog for that information.

Both entries were binary blobs and were large relative to everything else I had in my test hives so I wondered how feasible it would be to recurse through a hive and return any entry larger than a user specified size.

I started writing some Python code and came up with: reghexdump (i'm terrible at coming up with script names). You can download the code here from my GitHub page.

Let's take a look at what I ended up with.

Help

python reghexdump.py -h

usage: reghexdump.py [-h] [--hive HIVE] [--size SIZE] [--write WRITE]
Parse Registry hive looking for malicious Binary data.
optional arguments:
  -h, --help     show this help message and exit
  --hive HIVE    Path to Hive.
  --size SIZE    Size in bytes.
  --write WRITE  Write the binary values out to a directory.

Output - No Write

You will see here we have Path, LastWrite MD5 along with VT lookups on the binary blobs of data. In one of the instances you can see it's hitting 35/57, and lastly I also included the data size and a hex preview.

You're able to change up the --size to any size you want to help reduce FPs.

python reghexdump.py --hive NTUSER.DAT.copy0 --size 20000

Path: CMI-CreateHive{6A1C4018-979D-4291-A7DC-7AED1C75B67C}\Software\ xsw\binaryImage32
LastWrite: 2015-04-21T14:17:17.642979Z
MD5: 5be923a9a323667dc6ae33fb2f4a80a6 - 35/57
Size: 223744
00000000   4D 5A 90 00 03 00 00 00 04 00 00 00 FF FF 00 00    MZ..............
00000010   B8 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00    ........@.......
00000020   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00000030   00 00 00 00 00 00 00 00 00 00 00 00 E8 00 00 00    ................
00000040   0E 1F BA 0E 00 B4 09 CD 21 B8 01 4C CD 21 54 68    ........!..L.!Th
00000050   69 73 20 70 72 6F 67 72 61 6D 20 63 61 6E 6E 6F    is program canno
00000060   74 20 62 65 20 72 75 6E 20 69 6E 20 44 4F 53 20    t be run in DOS
00000070   6D 6F 64 65 2E 0D 0D 0A 24 00 00 00 00 00 00 00    mode....$.......

Path: CMI-CreateHive{6A1C4018-979D-4291-A7DC-7AED1C75B67C}\Software\Microsoft\Active Setup\Installed Components\{72507C54-3577-4830-815B-310007F6135A}\Rc4Encoded32
LastWrite: 2015-04-21T14:17:26.051649Z
MD5: 26ef08eb9dd49c53e0526bf148d23e3d - None
Size: 54669
00000000   87 87 3F 5C D1 25 67 7D C8 47 0F 5A 9C B7 D1 3E    ..?..%g}.G.Z...>
00000010   0B 34 AB 0E 9D 2E 59 D6 A2 51 C7 66 18 54 5A C2    .4....Y..Q.f.TZ.
00000020   1D 6B C0 B8 17 F6 23 C3 7D CA B2 2F E3 10 82 5A    .k....#.}../...Z
00000030   C8 99 9C 83 C9 4C 58 FB C7 FC 14 3E 15 9C B4 70    .....LX....>...p
00000040   82 3B 35 AF E3 B9 B2 E3 34 47 7F 50 46 74 01 B6    .;5.....4G.PFt..
00000050   F2 72 D1 76 44 71 B2 F5 82 21 F6 79 0F FE EE 68    .r.vDq...!.y...h
00000060   CE 04 8E 0F 51 2D C3 FE 70 BC 78 BC 2C 6E 94 1D    ....Q-..p.x.,n..
00000070   E9 0C 40 C5 98 DD 2F 09 2D 27 7E 14 B6 DA 28 3C    ..@.../.-'~...(<

Path: CMI-CreateHive{6A1C4018-979D-4291-A7DC-7AED1C75B67C}\Software\Microsoft\Active Setup\Installed Components\{72507C54-3577-4830-815B-310007F6135A}\Rc4Encoded64
LastWrite: 2015-04-21T14:17:26.051649Z
MD5: 233ec524cd7b88e18661298d8de549dd - None
Size: 68909
00000000   87 46 53 6F 96 A7 6B AA E8 65 C2 DA AE 96 4C 98    .FSo..k..e....L.
00000010   43 B5 8E E5 99 2E 59 9E 2B CD E3 66 1C 54 5A 8A    C.....Y.+..f.TZ.
00000020   C1 5E 01 31 FF EE 27 8B F4 F7 0B D7 1C EF 7D 9F    .^.1..'.......}.
00000030   F4 D1 43 64 32 C5 E5 42 B5 D1 51 B5 50 94 3D 0A    ..Cd2..B..Q.P.=.
00000040   B0 A6 AC 36 F8 A2 D8 F8 8E 95 7F 05 9F 70 01 B6    ...6.........p..
00000050   86 4F 04 2E 46 F8 A6 F1 4B 56 7F 79 4F 77 18 05    .O..F...KV.yOw..
00000060   76 F8 71 F0 6C 93 7F FE 70 CE 89 BC 2C 6E 94 1D    v.q.l...p...,n..
00000070   EB 0C 40 C5 98 9A E4 99 D2 E4 84 9D 6D 9B A9 94    ..@.........m...

Path: CMI-CreateHive{6A1C4018-979D-4291-A7DC-7AED1C75B67C}\Software\Microsoft\Windows\CurrentVersion\Explorer\StartPage2\ProgramsCache
LastWrite: 2015-04-21T12:53:08.684547Z
MD5: 630f44e2a9ae05d72f5aecb471d608e3 - None
Size: 74419
00000000   13 00 00 00 C3 53 5B 62 48 AB C1 4E BA 1F A1 EF    .....S[bH..N....
00000010   41 46 FC 19 00 80 00 00 00 7E 00 31 00 00 00 00    AF.......~.1....
00000020   00 69 42 79 3F 11 00 50 72 6F 67 72 61 6D 73 00    .iBy?..Programs.
00000030   00 66 00 08 00 04 00 EF BE 69 42 50 3F 69 42 79    .f.......iBP?iBy
00000040   3F 2A 00 00 00 3F 01 00 00 00 00 02 00 00 00 00    ?*...?..........
00000050   00 00 00 00 00 3C 00 00 00 00 00 50 00 72 00 6F    .....<.....P.r.o
00000060   00 67 00 72 00 61 00 6D 00 73 00 00 00 40 00 73    .g.r.a.m.s...@.s
00000070   00 68 00 65 00 6C 00 6C 00 33 00 32 00 2E 00 64    .h.e.l.l.3.2...d

Write

You also have the option to write out the binary blobs to disk. When using the --write arguement you will also have an additional message included. In this example it says, "Writing Data: binaryImage32 to out/"

reghexdump.py --hive NTUSER.DAT.copy0 --size 20000 --write out/

Writing Data: binaryImage32 to out/
Path: CMI-CreateHive{6A1C4018-979D-4291-A7DC-7AED1C75B67C}\Software\ xsw\binaryImage32
LastWrite: 2015-04-21T14:17:17.642979Z
MD5: 5be923a9a323667dc6ae33fb2f4a80a6 - 35/57
Size: 223744
00000000   4D 5A 90 00 03 00 00 00 04 00 00 00 FF FF 00 00    MZ..............
00000010   B8 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00    ........@.......
00000020   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
00000030   00 00 00 00 00 00 00 00 00 00 00 00 E8 00 00 00    ................
00000040   0E 1F BA 0E 00 B4 09 CD 21 B8 01 4C CD 21 54 68    ........!..L.!Th
00000050   69 73 20 70 72 6F 67 72 61 6D 20 63 61 6E 6E 6F    is program canno
00000060   74 20 62 65 20 72 75 6E 20 69 6E 20 44 4F 53 20    t be run in DOS
00000070   6D 6F 64 65 2E 0D 0D 0A 24 00 00 00 00 00 00 00    mode....$.......

List Output

If you use the --write option it will write out and of the "hits" into the directory specified as seen below.

ls out/

• ProgramsCache
• Rc4Encoded32
• Rc4Encoded64
• binaryImage32

file out/*

• out/ProgramsCache: Spectrum .TAP data BASIC program
• out/Rc4Encoded32: data
• out/Rc4Encoded64: data
• out/binaryImage32: MS-DOS executable

Summary

I hope this is useful. I'll add some additional features over the next few days. I'm short on time these days. Let me know if you would like to see any additional features/functionality. I don't know how practical it is yet, but saw the blog and wanted a quick script to detect it if I needed.

You can download the code here from my GitHub page.

Blog Series

• Part 1: Introduction to Hardware and Embedded Forensics
• Part 2: UART Intro. and Sniffing UART with a Logic Analyzer
• Part 3: Home Automation Hub Forensics
• Part 4: TBD

Introduction to Hardware and Embedded Forensics

I started thinking about cell phones and how people acquire data from them. I read about hacking pin codes with electronic devices, rooting methods and their shelf life/practicality, chip off, JTAG, etc.

What? JTAG and chip off...? After a bit of time on Google I sent a Tweet on April 19th, 2015 asking, "JTAG opinions: RIFF, Smart-Clip2, Octoplus, or Sigma Box? Looking to "play" around a bit"? I got a few replies. I didn't know much about what they tweeted back to me. Below is what has happened since that tweet...

That's when I realized I had a knowledge gap. I don't know much about hardware and/or electronics and if I ever needed to do something with either one of those forensically I would need to ask for some help. But I also like new hobbies and learning new things so this is where my road begins.

So to learn more about hardware, hardware "hacking", and other techniques used to acquire forensic data off hardware devices (outside of desktop/laptop computers) I have decided to start poking around with some different consumer electronics. Specifically home automation. At least initially to get my feet wet, but more so because my cell phone is hundreds of dollars and it has some super small electronics on it. I figured I would start with something a bit bigger. So I hit up Amazon and ordered a few different home automation hubs to take a look at. I'm less interested in the "hacking" part, but if I find something interesting along the way I will write about it as well.

First things first. I need to buy some equipment before I get started.

My Shopping List

• Shikra - This will allow me to speak with various low(er) level data interfaces such as; JTAG, SPI, I2C, UART, and GPIO.
• JTAGulator - This will allow me to brute force the PIN configuration for UART and JTAG interfaces so we can the use something like Shikra (or the JTAGulator) to talk with those.
• Saleae Logic 8 - This is a logic analyzer and will allow us to more or less perform a "Wireshark"capture on various data interfaces so we can see what's being passed around the wire. It does this with the software, which has protocol analyzers that will decode the various protocols (SPI, I2C, serial, JTAG, etc.)
• J-Link EDU Debugger - This is the same as the J-Link Base, but I am not using it for commercial purposes so you save some money.
• DSLogic Pro Kit - Logic Analyzer
• Tekpower TP1803D Linear Digital Variable DC Power Supply
• Triplett 1101-B Compact Digital Multimeter
• Ultra-Efficient Desk Clamp-Mount 56 SMD LED Spring-Arm Magnifying Lamp
• Kendal 2 IN 1 SMD Rework Soldering STATION 852D++
• Olympia Tools 88-670 iWork 15-Piece Smart Phone Repair Tool Kit
• Klein Tools 32525 32-Piece Tamperproof Bit Set
• SMARTScope by LabNation
• Make: Electronics (Learning by Discovery)
• Hot Glue Gun
• Fishing lure tackle boxes - Keep all my stuff nice and neat so my wife doesn't kill me.
• Various Jumper wires, solder kits, bread board, parts, etc.
• Some random stuff here and there.

Things I already had

• Raspberry Pi 2 Model B
• Caldigit Thunderbolt Docking Station 2 - I ran out of USB ports...

Here is where I get to justify my purchases. It is about 50% of a graduate class and I will learn a hell of a lot more than I would there so I consider it justified. However, I couldn't stomach adding all of it up so you will need to do that on your own. Go home or go big I guess..

References

Below is a list of sites, links, videos, courses, books, etc. that I am finding useful or are on my to-do list. This will continue to grow as I add more and more sites. Feel free to send me links. Some of these are way above my head, but I keep them around for references more than anything. Maybe they wont be for others.

I suggest reading/watching the "free" stuff first before you start buying a bunch of books and what not. Do as I say, not as I did. :)

Protocol References

UART:

• How does the UART work? - https://www.youtube.com/watch?v=FQpbIvhY7es
• Serial Port Complete 2nd Ed: http://www.amazon.com/dp/193144806X (Also referenced in Books section)
• https://en.wikipedia.org/wiki/Universal_asynchronous_receiver/transmitter

SPI:

• https://learn.sparkfun.com/tutorials/serial-peripheral-interface-spi
• Serial Port Complete 2nd Ed: http://www.amazon.com/dp/193144806X (Also referenced in Books section)
• https://en.wikipedia.org/wiki/Serial_Peripheral_Interface_Bus (See references section as well)

I2C:

• http://www.8051projects.net/wiki/I2C_TWI_Tutorial
• https://en.wikipedia.org/wiki/I²C (See references section as well)

JTAG:

• https://en.wikipedia.org/wiki/Joint_Test_Action_Group (See references section as well)

SMBus:

• https://en.wikipedia.org/wiki/System_Management_Bus

Wikipedia

It's not the articles themselves. It's the references that contain the gold.

Hardware RE/Hacking Sites

• https://www.exploitee.rs
• forum.xda-developers.com
• http://www.devttys0.com

Reference Books (Google goes a long way here)

• Hacking the Xbox (Free) - http://www.nostarch.com/xboxfree
• Serial Port Complete 2nd Ed: http://www.amazon.com/dp/193144806X
• USB Complete: The Developer's Guide 5th Ed: http://www.amazon.com/USB-Complete-Developers-Guide-Guides/dp/1931448280

Educational Courses

• https://www.coursera.org/course/introtoelectronics
• https://www.coursera.org/course/hardwaresec
• https://www.newbiehack.com

Tools

• Jtagulator - http://www.grandideastudio.com/wp-content/uploads/jtagulator_slides.pdf
• Jtagulator - https://www.youtube.com/watch?v=GgMOBhmEJXA
• Jtagulator Part 1 - Discovering Debug Interfaces with the JTAGulator https://www.youtube.com/watch?v=4StBmkAsCr4
• Jtagulator Part 2 - Discovering Debug Interfaces with the JTAGulator https://www.youtube.com/watch?v=xi5jSN7NCi4

Overview Presentations/Videos/Blogs/Websites (in no order)

• http://www.slideshare.net/balgan/hardware-hacking-101
• http://www.slideshare.net/mattb/hardware-hacking-for-fun-and-profit-1049183
• http://www.slideshare.net/guest3bd2a12/advanced-hardware-hacking-techniques-presentation
• http://www.slideshare.net/Sudar/open-hack-2011hardwarehacks
• http://www.slideshare.net/fusionmkx/hackers-and-painters-hardware-hacking-101-10th-oct-2014
• http://www.slideshare.net/watsonsteve/arduino-forensics
• http://www.slideshare.net/watsonsteve/wearable-device-forensics
• http://www.devttys0.com/2012/10/jailbreaking-the-neotv
• http://www.devttys0.com/2014/05/hacking-the-d-link-dsp-w215-smart-plug
• https://www.newbiehack.com

ELF Binary Reversing

• http://www.hackercurriculum.org/elf
• http://althing.cs.dartmouth.edu/local/reverse-talk.pdf
• http://www.linuxsa.org.au/meetings/reveng-0.2.pdf
• https://raw.githubusercontent.com/corkami/pics/master/ELF101.png

Summary

So yeah, that's the "kit" I have and will be playing with. I'm not really sure where this "series" of blog posts are going to go, or even if i'll get anywhere. I really know nothing about hardware/electronics so i'm more or less putting this out to the public as a means to keep me on task and hopefully learn something along the way. I can already see some C coding in my future. At the very least I will keep notes and make a good record of where people can go to at least get started if they are also interested.

DISCLAIMER: EVERYTHING HERE WILL VOID YOUR PRODUCT WARRANTY

This is Part II in a series of blog posts I will be doing. The main tracking page is here.

Before we get into the process I used for sniffing a UART connection that ultimately lead to a root shell on a Home Automation hub we will go over UART quickly in an effort to understand what it is and what it is used for.

Overview of UART

UART stands for Universal Asynchronous Receiver (RX)/Transmitter (TX). It is a general-purpose component for transmitting and receiving data independent of the processor. Devices with asynchronous serial ports typically contain hardware UART. When devices communicate with each other via asynchronous communication they aren't relying on an external clock signal between the receiver or the transmitter. It uses a bitrate/baudrate.

Here is a picture of what UART PINs may look like on a PCB. Note the TX and RX PINs, which are used for communication. For this discussion we care more about the TX and RX PINs as that's what we will connecting to; however, we will also want to connect a ground wire to the ground PIN. We can ignore the Vcc pin for now.

Some products that contain UART are SMART card readers, home automation hubs, etc. Lots and lots of electronic devices contain UART so it is important to get a good understanding of what it is and how it works.

We will be using a configuration as such = 8 bits with no parity and one stop bit (8-N-1). [StartBit][0][1][2][3][4][5][6][7][StopBit]

Keep in mind that when data is sent it is sent least significant bit (LBS) first.

So when you add up the 8 bits of data and the start and stop bits you will have a total of 10 bits. If we would have sent 7 bits instead of 8 bits of data we would need to set a parity bit and configure it to be even or odd (0 or 1).

I mentioned above about bit rate/baud rate. Baud rate translates to symbols per second. So assuming a baud rate of 9,600, we will transmit data at 9.6 kilobits per second.

When UART is idle (not transmitting data) it's set to "high", or logic 1 and when it sends it's first bit (start bit) it will be set to "low", or logic 0.

I learn better from pictures, so let's take a look at what this all looks like using one of the home automation systems I sniffed using my Saleae Logic 8.

You will see right before we hit the +0.5 ms marker the transmission is idle, or "high" (logic 1), which makes sense because nothing has crossed the wire yet and as a result it's idle, but then you see it drop into its start bit ("low"), and then 8 bits of data are sent. In this case, 0100 1010, which is hex 0x4A, or ASCII J. Notice the "bars' when the data is sent. When a 0 is sent the wave pattern is set to low, and when a 1 is sent it's set to high. Now you can more or less read the patterns even without some special software.

Here is an example with the HEX and ASCII decoding turned on, which is normally what I do for ease of reading.

You will see here what actually crossed the wire was ASCII JFFS2, which is the magic header for journaling flash file system, which is used with flash memory devices. Now we know we are getting somewhere!

Sniffing with a Logic Analyzer

So now that we have a quick overview of what UART is and how it is used I figured I would show you how I went about sniffing the data transmissions.

Below is an image of the home automation system PCB. With the information above you're able to easily spot the UART pins on this particular board (VCC, GRD, TX, RX). The next thing to do is see if they are hot, which I used a multimeter for. It's a bit hard to show you here how I went about doing this so hopefully you will have some idea how a multimeter works. It's pretty simple. If not, read SparkFun's how-to on multimeters.

Now that we know the PINs are hot we can place our Logic Analyzer clamps on the PINs we want to monitor. In this case we want to hook up to GRD, TX and RX. More specifically TX, because TX is the transmit PIN so that's where the data will come from, but in either case we will connect to GRD, TX and RX. When you connect the logic analyzer it will look something like this.

Once the logic analyzer pins are connected to the UART pins we can start our capture. Once we start the capture power on the home automation hub. Once it's finished you will see something like this:

I call it (and i'm sure others do) the Wireshark for electronics. The screen output is really nice, but it doesn't scale. This particular Saleae product allows you to output the data in CSV format, which is nice, but it is also hard to read in its raw form. In it raw form it looks like this:

Time [s],Value,Parity Error,Framing Error
2.115247380000000,n,,
2.115420420000000,r,,
2.115593460000000,n,,
2.115766520000000,r,,
2.115939560000000,U,,
2.116112600000000,-,,
2.116285660000000,B,,
2.116458700000000,o,,
2.116631740000000,o,,
2.116804800000000,t,,
2.116977840000000,' ',,

In this case I really only care about the value data and after seeing this output I know I needed something better so I wrote a quick python script to parse the data and make it more readable.

import sys
with open(sys.argv[1], 'r') as f:
    value_list = []
    next(f)
    for line in f:
        initial_cleanse = line.split(',')[1].strip()
        value_list.append(initial_cleanse)
    final_cleanse = " ".join(value_list).replace(" ", '').replace("nr", "n").replace("n", "n").replace("r", "").replace("''", ' ')
print(final_cleanse)

And here is the output from saleaeparse.py. As you can see it is much easier to read. It's more or less the same as you would see if you were to connect to the serial port and watch the screen while spits out all the text.

python saleaeparse.py homehub.txt |head -20

U-Boot 1.1.3 (Jan 19 2012 - 17:57:52)
Board: Ralink APSoC DRAM:
Return Real size =67108864 !!
64 MB
relocate_code Pointer at: 83fa8000
Ralink UBoot Version: 3.4.0.4
Model:
ASIC 3883_MP (MAC to 100SW Mode)
DRAM component: 512 Mbits DDRCOMMA width 16
DRAM bus: 16 bit
Total memory: 64 MBytes
Flash component: NAND Flash
Date:Jan 19 2012 Time:17:57:52

Much easier on the eyes, yeah? You can download the code from my GitHub account.

Although not pasted above, while reviewing the full output of the 90 second collection I saw the following message, "Press the [f] key and hit [enter] to enter failsafe mode".

This is interesting, so let's take a deeper look using another piece of equipment called Shikra. This will allow me to speak with various low(er) level data interfaces. In this case we will be interfacing with UART as we already discussed.

Connecting to the Home Automation Hub's UART PINs

The Shikra is the device on the right hand side plugged into my USB port.

Pins layout

• Shikra: TX ---connected_to---> RX of Home Automation Hub
• Shikra: RX ---connected_to---> TX of Home Automation Hub
• Shikra: GRD --connected_to---> GRD of Home Automation Hub

Now that the pins are hooked up properly we can attempt to connect to the home automation hub through Shikra. To do this we will use the screen command, but before we can do that we need to know the path of our Shikra.

You can accomplish this via the following:

ls /dev/serial (on Mac)

• /dev/cu.usbserial-141
• /dev/tty.usbserial-141

python -m serial.tools.list_ports

• /dev/cu.Bluetooth-Incoming-Port
• /dev/cu.Bluetooth-Modem
• /dev/cu.usbserial-141
• 3 ports found

On Linux it will look like this:

• /dev/ttyUSB0

If you go the python route you will need to install pyserial via pip install pyserial

Now that we know our serial device information we can connect.

screen /dev/ttyUSB0 57600

Once we kick off the command we will turn on the home automation hub and once you start seeing a bunch of text we can hit either f + enter or simply hit enter a few times. Either one will drop us into root, although there are a few additional commands in failsafe mode.

Root Please?

So if everything worked as discussed we should now have our root shell and we can pretty much do anything we want at this point. We can dump memory, image mounted drives, grab the firmware, etc.

If you decided to go into failsafe mode your root prompt will look something like this: root@(none):/#

If you simply hit enter a few times your root shell will look like this.

enterBusyBox v1.17.3 (2012-01-09 12:40:42 PST) built-in shell (ash)
root@redacted:/#

The redacted parts show the serial number of my device, which I don't want published. I don't feel like buying a new one because someone locks me out.

Summary

So there we have it. Overview of UART, how to connect to the pins, and ultimately how we got root. I guess you could say, I got lucky, but I have to assume a lot of consumer electronics are configured in a similar fashion and you don't know what you don't know until you do it for yourself.

Again, my main point of this series was to look at the forensic evidence more than the hacking. In the next blog I will look at the forensics side of gaining root so this device and see if we can uncover any forensic evidence from the device.

It can be an expensive learning experience, but so far this is some really cool work. I've learned a lot just in the few days playing with this. Now I just need to figure out how I can do this day-in-day-out and get paid for it! Haha. #InTime... #InTime...

I've also added some more resource links to my into post so make sure to keep an eye on the original post of this series for the latest sites I found helpful.

References:
How does UART Work
Wikipedia - UART

In Part 1: Introduction to Hardware and Embedded Forensics I went over some tools I am using to perform hardware and embedded forensics analysis. In Part 2: UART Intro. and Sniffing UART with a Logic Analyzer we were able to get root on a home automation hub.

Now for Part 3: Home Automation Hub Forensics, which is this post we are going to go over getting root access on another home automation unit, and then proceed to perform some initial forensic analysis.

Why I decided to perform analysis on this particular unit is because it functions as both a home automation hub and also a wifi router. It seems more practical to me vs. purchasing a wifi router and a home automation hub. I hate duplicate devices so this all-in-one unit is nice.

Before we were able to obtain Root access I need to open the device and take a look at its inside. Right away I noticed the UART pinout. They were nice and labeled for me and allowed for easy access.

In this image I have already connected the female jumper wires to the TX, RX and GND pins.

And here you can see the connection is made to my Shikra so I am ready to talk to the device over serial.

So now that we are connected we can simply connect to the device once it powers up. In this case I kept hitting Enter and in due time I was dropped into a login prompt. The problem with this is that it was password protected. I tried some simple passwords like root, admin, etc. but none of them seemed to work.

Then I remembered when I setup the device there was a default password already configured in the device. I can't verify is these are unique, but my password was haars (if I recall correctly). So I decided to use root as the user and haars as the password and guess what?

If you change the internal password of your device that same password will follow forward so if you change yours from haars to helloworld you would use root as the username and helloworld as the password.

I also wanted to mention that at least on my device SSH seemed to be enabled by default and I was able to SSH into that same root account with the preconfigured user/password (root/haars) so you really wouldn't even need to go to the extent I did with the hardware to get the same level of access. In either case, we will continue as if SSH was not enabled.

Once connected I started to poke around a bit.

cat /proc/mtd

dev: size erasesize name
mtd0: 00100000 00020000 "uboot-env0"
mtd1: 00100000 00020000 "uboot-env1"
mtd2: 00100000 00020000 "sb0"
mtd3: 00100000 00020000 "sb1"
mtd4: 02e00000 00020000 "rootfs"
mtd5: 00600000 00020000 "kernel_standby"
mtd6: 02e00000 00020000 "rootfs_standby"
mtd7: 01000000 00020000 "rootfs_data"
mtd8: 00100000 00020000 "log"
mtd9: 00100000 00020000 "cal"
mtd10: 00100000 00020000 "man"
mtd11: 00500000 00020000 "SNIP_Data"
mtd12: 000e0000 00020000 "uboot0"
mtd13: 00100000 00020000 "uboot1"

I dumped mtd4, rootfs, which is the squashfs firmware file.

Once we have this we can unpack it and look at the filesystem contents and review all of the data.

I will look at this more in the next post.

I got this idea awhile back when I wrote a tool called, BARFF - Browser Artifact Recovery Forensic Framework. It was more or less a glorified SQLite parser, which is again, more or less what SLARF is, but it uses a plugin based architecture similar to the code I wrote for, python-regparse (written in Python).

Why SLARF?

It is extensible. If you come across a new SQLite database artifact all you need to do is copy a bit of plugin code and in 10 - 15 mins you have a new plugin to pull the data out.

Cell phones more or less equates to running various queries against a range of SQLite databases. Likewise with modern browsers and a whole bunch of other programs from Garmin GPS backups to everyday applications like Evernote, they all use SQLite databases to store information relevant to an investigation.

So rather than writing a tool for each application, or keeping a bunch of SQL queries pasted in a notebook for later use, it is much easier to have a set of ready to use plugins.

So let's take a look and see how it works.

Help

python slarf.py -h

usage: slarf.py [-h] [--plugin PLUGIN] [--listplugins] [--plugindetails]
                [--database DATABASE] [--format format]
                [--format_file format_file]
Parses SQLite Databases.
optional arguments:
  -h, --help            show this help message and exit
  --plugin PLUGIN       Specify plugin to run.
  --listplugins         Lists all of the available plugins.
  --plugindetails       Lists details available plugins.
  --database DATABASE   SQLite Database.
  --format format       Custom output.
  --format_file format_file
                        Custom output template.

Plugins

Right now I do not have a lot of plugins written for SLARF, but I will be adding to them over time. If there is one you want now let me know and I can write it for you, or you can simply pass me the SQL query and I can build the Python plugin for you.

The following plugins were some sample ones that I did to test out the tools effectiveness.

• dbinfo - prints information about the database
• nuvi2757lm_rs (see image below)
• safari_histody_db - parses safari history database
• unallocated - recovers data from unallocated space within the sqlite database

python slarf.py --listplugins

dbinfo
nuvi2757lm_rs
safari_history_db
unallocated

python slarf.py --plugindetails

NUVI2757LM_RS
	Plugin: 	NUVI2757LM_RS
	Author: 	Patrick Olsen
	Version: 	0.1
	Reference: 	http://sysforensics.org
	Print Fields: 	"{{ rank }}|{{ string }}"
	Description: 	Parses the searches table from the recent_searches.db on the Garmin Nuvi 2757LM GPS.

Templates

The HTML output uses predefined templates. You can see an example of one here on my GitHub.

The output when using templates look like this:

python slarf.py --plugin nuvi2757lm_rs --database recent_searches.db --format_file templates/nuvi2757lm_rs.html

User Defined Output

python slarf.py --plugin nuvi2757lm_rs --database recent_searches.db --format '{{rank}}|{{string}}'

0|Nordstrom
1|Washington DC, DC
2|Starbucks
3|Kroger
4|Radisson Hotel-Nashville International
5|St Louis, MO

Summary

So yeah, there it is. I have to write a bunch more plugins, and to be quite honest I don't do much analysis of SQLite DBs so if you have some ideas, or common queries, etc. by all means let me know and I'll see what I can do to create some for you.

I find myself short on time these days.

One more point. It really doesn't have to be just SQLite DBs... You could write your plugins to process any kind of database really. You will just need to add the respective imports.

You can download SLARF on my GitHub.

I am back online after a year. My previous employer did not allow blogging or code contributions, but that's behind me now so I am free to do what I want.

I actually debated whether or not to start writing again, but I found that I actually missed it over the past year. Blogging takes a lot of time and you rarely (if ever) get praise for it. On the positive side, it kept me on task and focused on learning new and exciting things at a much quicker pace.

The blogsphere is getting crowded, so if you can think of any topics you want covered by all means let me know. I am always up for suggestions, but rarely (if ever) get feedback from readers. If I know nothing about your suggestion I will learn about it. That's the fun part about having a blog.

I created a GitHub repo where people can submit blog ideas as issues. You can submit them here. This will give me a running list of ideas to write about.

Everything seems to be incident response and malware related these days since it's all the buzz so maybe I can get creative and find some other areas to write about.

I will write some new posts over the next few days. I am in the process of moving across the country.

Enjoy!

I started playing around with the Arduino Uno Rev 3 awhile back but never got around to documenting anything via a blog post (until now).

I read Steve Watson's slide series here on Arduino forensics, and decided to write up a blog on some of the stuff I was playing around with, too.

Overview

Most Arduino projects you find online seem to be more of a novelty and may not hold much forensic value overall, but there are some really neat projects and it is a rather easy process so I thought it is was worth writing down in case someone comes across the need in the future.

You never know when an Ardunio controlled drone will fall out of the sky and cause someone serious brain injury.

Arduino Uno Rev 3

So here we have the Arduino Uno. I will be using this one for demo purposes. I purchased mine from Adafruit for $24.95 + Shipping. Go get yourself a few of them. If only to keep them on your shelf and not touch them like most people do... You know I am right.. :)

Sources of Evidence

By looking at the schematics and reading the technical specifications I came up with a list of possible sources of evidence. There may be more depending on how the Arduino is configured and what it's purpose is.

• Arduino itself
• Microcontroller
• Onboard Storage
• Web Services (APIs)
• Charting, Messaging, etc. services
• Webserver hosting the services
• Router/Network Devices
• Internet communication via a breakout/shield board.
• Myriad of Arduino shields, sensors, etc.
• Development System(s) (Writing Code)

Onboard Storage

• Flash Memory 32 KB (ATmega328)
• 0.5 KB used by bootloader
• Static Random Access Memory (SRAM) 2 KB (ATmega328)
• Electrically Erasable Programmable Read-Only Memory (EEPROM) 1 KB (ATmega328)

You can read more about the Arduino memories here

Wiring up the Uno

To dump the Flash and EEPROM I used Newbie Hack's AVR User Programmer. I went this route because I had some issues with the Bus Pirate v4 and Shikra.

Sometimes it is just easier to throw money at the problem. In this case, $9.95.

So to get this working it's simple. I installed the usbasp drivers via CrossPack on my Mac system. I then installed Avrdude via Homebrew.

Once this was installed I wired up the Arduino. The 10 pin pinout and the device I am using can be found/purchased here.

If you want to reference the official schematics, which would also be useful when tracing the pins back to the MCU for "chip-off" acquisition as well.

The Arduino Uno Rev 3 SPI pinout can be found in Google Images quite easy as well. Here is an example if the above is too cryptic.

If you were to perform "chip-off" and analyze the chip via a breadboard you would need to have this reference of the chip pinout for the ATmega328P.

Now it's time to connect our USBAsp programmer to the Arduino. In this example I used a stripped down board to make it easier to see in the picture.

In the image you can see our AVR programmer and the ICSP pins connected up to the Uno. It really is as simple as that. Just follow the pinout diagrams above and you should be fine.

Dumping Contents

Now that we are wired up and have a solid connection let's attempt to dump the data for analysis.

We will be using avrdude to dump flash memory and EEPROM.

Linux: sudo apt-get install avrdude

Mac: brew install avrdude

EEPROM

EEPROM is non-volatile so it's not impacted by power off/on cycles like flash memory.

Per the data sheet we know that the EEPROM size is 1KB.

avrdude -p m328p -P usb -c usbasp -U eeprom:r:eeprom.bin:r

And this is what it looks like when it's dumping the contents.

avrdude: AVR device initialized and ready to accept instructions

Reading | ####################### | 100% 0.00s

avrdude: Device signature = 0x1e950f (probably m328p)
avrdude: reading eeprom memory:

Reading | ####################### | 100% 0.40s

avrdude: writing output file "eeprom.bin"
avrdude: safemode: Fuses OK (E:FD, H:D6, L:FF)
avrdude done.  Thank you.

So we successfully dumped the EEPROM as seen here.

-rw-r--r-- 1 root staff 1.0K Mar 11 19:03 eeprom.bin

There is also an EEPROM library that you can use to read data out of EEPROM.

Flash

Now to dump the flash contents.

avrdude -p m328p -P usb -c usbasp -U flash:r:flash.bin:r

And we can see here that we dumped the flash memory successfully. Flash memory - think USB drive.

avrdude: AVR device initialized and ready to accept instructions

Reading | ############################ | 100% 0.00s

avrdude: Device signature = 0x1e950f (probably m328p)
avrdude: reading flash memory:

Reading | ############################# | 100% 6.11s

avrdude: writing output file "flash.bin"
avrdude: safemode: Fuses OK (E:FD, H:D6, L:FF)
avrdude done.  Thank you.

SRAM

At present I do not have a solution for dumping SRAM from Arduino. If you have a solution let me know and I can update this post so everyone can take advantage of it. This would be the most artifact rich source of evidence.

I suspect this is more of a coding deficiency on my part, vs. it being difficult.

Analysis of the Data

Now it's time for analysis, which in this case isn't too difficult since we used a bare Arduino with nothing really running on it.

EEPROM

We can do a quick check with xxd to see if there are any contents available.

xxd eeprom.bin |head -1024

0000000: 69ff ffff ffff ffff ffff ffff ffff
0000010: ffff ffff ffff ffff ffff ffff ffff
0000020: ffff ffff ffff ffff ffff ffff ffff
0000030: ffff ffff ffff ffff ffff ffff ffff

Flash

Simply running strings against the flash contents produces some good information that could be relevant to a case.

MAC Address : 
Unable to retrieve MAC Address!
DNSserv: 
DHCPsrv: 
Gateway: 
Netmask: 
IP Addr: 
Unable to retrieve the IP Address!
Request DHCP
Connected!
Failed!
Check your wiring?
Initializing...
localhost
Free RAM: 
alucard_almond24 <-- SSID of my WLAN
<redacted_wifi_password> <-- WLAN password
www.adafruit.com <-- GET request I made
localhost

That's really about it. Not too exciting. I'm sure it's more exciting for more complex projects.

Summary

All and all that's about all of the information you get. Granted, I do not have any fancy code/configurations running on my Arduino, but this should be a good starting point for some deeper analysis.

For anyone running a drone or similar hardware setup and wants to donate their EEPROM/Flash dumps let me know. I will add the analysis to this post.

Enjoy!

Overview

I always thought JTAG was hard, then I tried it, and realized it was actually very easy (most of the time). Pretty much anyone can learn to do this in 8 hours of soldering practice. Really, it's dead simple. Don't let anyone fool you by trying to convince you it's not.

I said, most of the time, because there are going to be instances where the following may happen and prevent some people from being able to JTAG a phone:

1. Phone not supported by popular tools - Let's face it. For the majority of analysts if the tool doesn't work, they don't work. That's just life so we might as well accept that fact and list it first. Sooner or later you will get a phone that isn't supported and you likely wont be able to analyze it. The barrier of entry to write support for non supported chips is going to be too difficult for most people. That's fine. That's life. Analyze the other 95%+ of phones that come through your door and outsource the remaining 5%.
2. JTAG taps are unknown and you need to figure them out on your own. This can take time and more skills. Again, see first bullet (outsource the 5%).
3. Micro Controller (MCU) does not support JTAG requiring ISP/ICSP or Chip-off. This may or may not be possible in your environment. I would suggest additional training for Chip-off. Also, for ISP you will be soldering some crazy small wires. It's very doable, you will just need practice (eBay).
4. Encryption - Think new iPhones and new Samsung devices. This quite possibly spells the end of JTAG.

NOTE: I do not do this in my day job.  I purchased damaged phones off Ebay and played around with them. This is even more proof that anyone can learn how to do this.

What is JTAG?

First, let's define what JTAG is so we can better understand it going forward.

Joint Test Action Group (JTAG) is the group of companies that came together in 1985 to define a standard for boundary-scan testing of integrated circuits.

In 1990, the specification resulted in IEEE 1149.1, Standard Test Access Port and Boundary Scan Architecture. It's main purpose is to allow engineers to perform debugging and diagnostics of the system/device.

JTAG Interface Signals

UPDATED 10/02/2016 - Removed this section. This senr.io write up explains JTAG much better. No reason to repeat it.

Tools for JTAGing Phones

I use the following tools and have NO issues with JTAGing phones. You can buy better equipment if you want, but below is proof you do not need the best equipment.

I will assume you already have forensic software to process the data we will dump from the phone. If not, you can get a lot of good content with X-Ways, Net Analysis, Bulk Extractor, Carving Tools and Autopsy before diving into some of the more expensive commercial mobile forensic solutions (IEF, Cellebrite, Encase, etc.) But again, I will assume you have this already and if you're in Law Enforcement you likely already have at least Cellebrite and something like Encase.

• RIFF Box - $150
• Z3X Easy JTAG box - $250
• Hakko FX888D solder Iron - $98
• AmScope SE400-Z Stereo Microscope - $185
• Multimeter - I use Extech EX330 - $50
• Bench Power Supply - Tekpower - $75
• Flux - $10 - $12
• Solder - $25 - $35 (Lasts a long time) - Smaller the better.
• 20, 30 and 40 AWG Magnetic wire - $10 - $15
• Other wire as necessary (smaller the better) - $10 - $15 - Adjust the size as necessary.
• Alligator Clip Test Leads - $6 - $12
• Hakko CHP 3-SA Stainless Steel Non-Magnetic Precision Tweezers - $5
• Cell Phone Repair Tools $15 - $50
• Faber-Castell Perfection Eraser Pencil with Brush -
  $5 - Used to clean JTAG taps before soldering
• A Tooth Brush - $1 - Cleaning PCB/Taps
• Rubbing Alcohol - $2 - Cleaning PCB/Taps
• Kapton Tape - $10 - $15 - For those without steady hands, this can help cover surrounding components from getting solder on them.
• Hakko FA400-04 Fume Extractor - $68 - Or make one with a CPU fan and a Carbon filter for $20
• JPIN JTAG Molex Flex Kit - $60
• Hakko T18-BR02 - T18 Series Soldering Tip - $5
• Kohree 110V LED Digital 858D SMD Hot Air Rework Station Solder - $60 - Not needed for JTAG, but nice to have on your desk in case any JTAG taps are covered in epoxy

Total: For < $1,400 you can JTAG most phones from what research I have done. If you are in law enforcement and not doing this you are missing out. It is very simple.

There is another tool called, JTAGulator - $159 - that can help brute force JTAG tap layouts. This tool can be useful when you do not know what the layout of the taps are.

It's not necessary as most of the popular phones are supported and documented. I have used it and it worked on some phones, but not on others. It seems to be hit or miss. The Z3X also has a JTAG tap identification tool built into it so I recommend starting with the Z3X before shelling out another $159 for the JTAGulator.

If you do not want to invest in gear, but want something JTAG'd I can help you. Just ask.

Nokia Lumia 521

I decided on this phone after someone posted on the SANS mailing list asking how to acquire data from it.

I purchased two phones off Ebay. $15 each + shipping for a grand total of $40.69. The condition of the phone doesn't matter for JTAG if it powers on. So if you can confirm power (multi-meter), the broken screen doesn't matter and that will save you some money buying test phones.

Phone Research

• Phone Scoop - Nokia Lumina 521
• Internal Photos Per FCC ID
• GSM Forum via Google Search
• Google Image Search: "Nokia Lumia 521 JTAG" - Looking for JTAG pinout images

What we know

• 1 GHz Qualcomm Snapdragon S4 MSM8270
• Windows Version 8
• 8GB
• Labeled JTAG Pinouts
• JTAG Box Support

JTAG Box Verification

As we can see here the ZX3 supports the Nokia Lumina 521.

And here were can see the JTAG pinouts provided by the Z3X JTAG box.

Phone Disassembly

Now that we have confirmed that at least one of our tools supports our Nokia Lumina 521 let's move forward with phone disassembly.

Prep and Clean the Taps

First our Taps are under a heat shield. I use my SMD re-work station (hot air) to remove the shield. Be careful not to burn up the board. Use a swiping motion when using the rework hot air gun as not to apply direct heat to the board for too long.

In this step we want to make sure our JTAG taps are clean. I do this by scratching away the layer to expose the copper taps. If we leave the layer of coating (not sure the name) it will not attract the solder and you will not be able to solder it.

After scratching them off I cleaned them with a Q-tip and some alcohol. After you do this blow on the board to dry the alcohol (happens quickly).

Soldering our JTAG Taps

These are some shots under my Microscope. It's too small without the microscope so I do all of the soldering this way. Some people will use medical glasses (think Dentist). I like the Microscope.

Here are some completed taps that I have applied solder to. At this stage I need to scratch off a few more taps, and then solder them as well.

Wiring the Phone

After all of the solder was placed on the taps I moved forward with wiring up the phone per the wiring diagram above.

I have connected it to the Z3X JTAG box. The device in the middle between the phone and the Z3X is a custom connector I created via OSH Park. It's just an easy medium that allows me to interface with the JTAG box more easily. If you want the schematics let me know and I can send it. It's about $4 - $5 per device in parts. It is not necessary. The Z3X and Riff box will come with a small PCB to interface with.

Dumping Phone Contents

Here is a copy/paste from the console log.

CPU IDCODE: 0x4F1F0F0F 
Mfg: 0x787
Part: 0xf1f0
CPU Manufacturer: Samsung
CPU Name: ARM7GEN

JTAG device: MSM8227
CPU IDCODE: 0x207D00E1
Mfg.: 0x070 
Part: 0x07d0
CPU Manufacturer: QUALCOMM
CPU Name: MSM8227

EMMC 0:
ID: 0x004A0090
Name: H8G2d
Size: 7.2 G
Blocks: 15155200

EMMC 1
ID: 0x004A0090
Name: H8G2d
Size: 2.0 M
Blocks: 4096

eMMC Flash Device(s) found:
Device ID: 0x0090004a

The dump is being read from EMMC address 0x000000000000 -> 0x000200000000 and being saved to the following location:

C:\Users\sysforensics\Desktop\NOKIA_LUMIA_521_0x00000000_0x200000000.bin

The dumping process can take 2+ days with an 8GB phone. It's not something you can turn around to your department in an afternoon.

Now that we have the dump here:

NOKIA_LUMIA_521_0x00000000_0x200000000.bin

Let's begin analysis.

Analysis

You will have a single bin file. Simply load it up in your analysis tool of choice and have at it. In this example I loaded it up in X-Ways and started carving for photos.

Analysis Results

There were a few boob pictures, multiple selfie pictures, some porn, family birthday party and my favorite a selfie with the couple smoking a joint and holding a bag of pot.

Yes.... Some people really are that stupid.

Finding Help

So you read the blog and you are still stuck. Here are a few sites that have some good information.

• GSM Hosting Forum
• Google and Google Images
• Baidu (Chinese Google)
• Unlock Forum
• GSM India
• Phone Scoop
• FCC ID Search
• The RIFF and Z23 Box themselves - They have pictures of JTAG taps, etc.
• I'm sure you have internal police mailing lists. Ask around for supported devices.

Summary

I hope you learned how to JTAG. Assuming the phones are supported by the tools it's dead simple. Don't let the solder iron and wires scare you away. You really can do this. If you can't, or don't want to let me know. I would be happy to help.

Overview

Before I get into the real reason for this post, I want to quickly go over what kMDItemDateAdded is, and then I will discuss where a popular tool is missing the timestamp.

kMDItemDateAdded

kMDItemDateAdded is an HFS+ metadata attribute that also happens to be indexed by Spotlight. Spotlight is used to index your system. In simple terms, Spotlight allows you the ability to find things easily on your Mac.

The file structure for Spotlight is located at /.Spotlight-V100/Store-V2 (newer versions). There is no free/open source Spotlight Index database parser to my knowledge (if you know of one let me know). The Spotlight index database is also undocumented.

If you run mdimport (from a Mac):

mdimport -A |grep -i Date

You will get a list of the date attributes Spotlight has the capability of storing/using. If you just run, mdimport -A you will see all of the attributes available.

According to the output from mdimport, kMDItemDateAdded is the, "Date when this item was last moved". This date is updated when it's moved, and gets indexed by Spotlight (pretty much instantly). I'll show you an example below.

Date Added Example

NOTE:  This following example assumes Spotlight Indexing is enabled on your system.

Let's say I do the following from my Desktop:

• Create a folder called, TestFolder
• Create a file called, Testfile.txt
• Run mdls against Testfile.txt
• Move Testfile.txt into TestFolder
• Run mdls against Testfolder/Testfile.txt
• Compare the kMDItemDateAdded entry from the first mdls to the second (after the move)

After completing the commands below you will notice that the kMDItemDateAdded entry was updated when Testfile.txt was moved into another directory.

• mkdir Testfolder
• touch Testfile.txt
• mdls Testfile.txt

<snip>
kMDItemDateAdded = 2016-08-06 23:35:28 +0000
kMDItemDisplayName = "Testfile.txt"
<snip>

• mv Testfile.txt Testfolder/
• mdls Testfolder/Testfile.txt

<snip>
kMDItemDateAdded = 2016-08-07 00:16:15 +0000
kMDItemDisplayName = "Testfile.txt"

It is worth noting that kMDItemDateAdded attribute will only exist if the file/folder on the HFS+ volume is being indexed by Spotlight, but the HFS+ attribute (more on this below) will always exist.

Catalog File

The Spotlight index is not the only place to obtain the date added timestamp; however, some tools are not pulling the data out and presenting it to you. The age old, "If the tool tells me it's, ok then it must be, ok." Well, it's not in this case.

TN-1150 is Outdated

Most people reference TN-1150 as the sole-source of information for all things HFS/HFS+.

It is a great source of information, but it's not the most current, so you need to also look at some documented source code to find the latest information.

If you were to only look at TN-1150 you would see this ExtendedFileInfo C struct within the Catalog File section.

struct ExtendedFileInfo {
  SInt16    reserved1[4];
  UInt16    extendedFinderFlags;
  SInt16    reserved2;
  SInt32    putAwayFolderID;
};
typedef struct ExtendedFileInfo   ExtendedFileInfo;

Now here is where it gets exciting. If you look at an updated version of hfs_format.h you would see this FndrExtendedFileInfo C struct:

struct FndrExtendedFileInfo {
	u_int32_t reserved1;
	u_int32_t date_added;
	u_int16_t extended_flags;
	u_int16_t reserved2;
	u_int32_t reserved3;	
} __attribute__((aligned(2), packed));

Note the second 4 byte entry, date_added. Yep... That's right... It's also documented within the Catalog File, not just the spotlight index.

Furthermore, if you looked at the API documentation you would see this, which further labels some of the fields that were previously reserved/unused bytes. Specifically, document_id and write_gen_counter. I haven't had time to look into these two fields. On a few tests I have done they have been 00s (zeros).

struct FndrExtendedFileInfo {
    u_int32_t document_id;
    u_int32_t date_added;
    u_int16_t extended_flags;
    u_int16_t reserved2;
    u_int32_t write_gen_counter;
};

Catalog File Date Added Example

For this example I will be skipping over the details of a catalog file and diving right into the specifics of this blog post.

In this example I have a file called, If You're Not Helping Employees Learn You're Not Doing Your Job.pdf. This was a HBR article I downloaded to my Desktop as part of my MBA program.

You can see the hex of this catalog file record here:

In this example we get our MACB timestamps (in blue), which are parsed out just fine by common tools. So nothing really interesting there.

When we convert the hex we get the following times:

0xD3B854F3: 2016-07-22 22:55:47
0xD3B854F3: 2016-07-22 22:55:47
0xD3B854F4: 2016-07-22 22:55:48
0xD3BD57EA: 2016-07-26 18:09:46
0x00000000: 1904-01-01 00:00:00

In Sleuth Kit (using the istat command) we get:

Created:             2016-07-22 22:55:47
Content Modified:    2016-07-22 22:55:47
Attributes Modified: 2016-07-22 22:55:48
Accessed:            2016-07-26 18:09:46
Backed Up:           0000-00-00 00:00:00

So we know the times are all matching up, but we are missing one timestamp, date added.

In this image I highlighted; 0x5792A473, which is the date_added timestamp we have been looking for.

If you convert 0x5792A473 to an unsigned integer using Big-Endian you get an epoch timestamp of, 1469228147.

If you then convert it you will get:

• 2016-07-22 22:55:47

Compare mdls Output

Now we can compare the converted time from above to mdls.

mdls Desktop/If\ You’re\ Not\ Helping\ Employees\ Learn\,\ You’re\ Not\ Doing\ Your\ Job.pdf

And we get:

kMDItemDateAdded = 2016-07-22 22:55:47 +0000
kMDItemDisplayName = "If You’re Not Helping Employees Learn, You’re Not Doing Your Job.pdf"

In this example I used mdls to compare the timestamps. If Spotlight indexing wasn't enabled on my test system we wouldn't have been able to use mdls as a verification mechanism; however, this the date added timestamp is a HFS+ attribute it would have still been located within the Catalog file.

The Sleuth Kit (TSK)

It should be noted that I call out TSK here only because I use the tools all the time and I know they are used by a lot of people in industry. I created a feature/issue request at the same time I published this blog.

So where it goes wrong with TSK I believe is here under, CATALOG Record structures.

You can see that the first 16 bytes of the C struct are reserved (uinit8_t res1[8]).

The second set of 4 bytes should be date_added, with the first 4 bytes still reserved. I am not a C coder so I could be wrong, but I suspect this is the C struct that needs modified.

Current Source as of August 2016

typedef struct {
    uint8_t res1[8];      /* reserved 1 */
    uint8_t extflags[2];  /* extended finder flags */
    uint8_t res2[2];      /* reserved 2 */
    uint8_t folderid[4];  /* putaway folder id */
} hfs_extendedfileinfo;

Possible Solution as of August 2016

This is a possible solution to fixing the timestamp. Obviously more code will need to be added/changed as well, but all and all this should represent the updated C struct (I think).

typedef struct {
    uint32_t document_id;        /* Not sure?? */
    uint32_t date_added;         /* Date added */
    uint16_t extflags;           /* ext finder flags */
    uint16_t res2;               /* reserved 2 */
    uint32_t write_gen_counter   /* Not sure?? */
} hfs_extendedfileinfo;

If it was as simple as adding another line of code I would have done it, but I am not a C developer so the best I could do was find what I believe is the problem, read the source, write this blog post, and file a feature/issue request to the awesome TSK team.

Summary

So yeah, if we get the feature added to TSK this will give us another timestamp to include in our timelines. Also, in the meantime you can use mdls and/or hex to determine the timestamps while you wait.

For the other tool developers, have at it.

Enjoy!

Overview

I read a few blog posts and slide presentations discussing the forensic value of alias data within plists.

They typically consisted of running strings against the Data object to pull path data out.

Rather than relying on strings I decided to reverse engineer the Alias Version 3 format and provide a Pythonic means to parse it.

Alias Version 2 had been reversed and documented previously (I lost the link), but version 3 from what I could find had not. Version 3 has a different structure than Version 2. I got about 90% of it complete. There are still some bytes I am unsure about, but I believe I got most/all of the forensic relevant data reversed well enough to share and seek comments and allow for greater testing.

Tested on OS X 10.11.x

LoginItems Example

I will use LoginItems as an example, but from my testing it applies to any plist that contains alias version 3 data (sidebar items, etc.)

Assuming you have some loginitems you will have the following plist:

~/Library/Preferences/com.apple.loginitems.plist

Inside this particular plist there is an array of login items. We will look at Item 0, which in my case is Flux. You may have something different.

As you can see there is String called, Alias. The value of, which is a Data object.

The two common methods of reviewing this data is strings and hex dumps/editors. Let's look at those two options now.

Strings

Applications/Flux.app
Flux.app
OS X

Hex

With a hex editor we get:

Applications/Flux.app
OS X
Reference to H+

So with strings and a hex edited simply looking at the output doesn't give us that much really.

Better Way

The better way to obtain this information would be to reverse engineer the file type and figure out what is really inside.

Reversing Alias v3

Here is what the Alias v3 looks like after I reversed it and created a Synalyze It! template for parsing it. You can download the template in the, Play Time section below.

• VolumeCheckedDate- This matched up to the Volume Check Date timestamp in some testing. It was different in other testing. More testing needs to be done on this timestamp, but so far that's how it has matched up (see note below).
• Filesystem - Filesystem of the volume where the file/folder exists.
• Parent CNID - This points to the parent folder of the file/folder. In this example the parent CNID represents Applications
• Target CNID - This is the CNID of the target file/folder. In this case, it's the Flux.app
• File or Folder Creation Date - This is the creation date of the file (Flux.app). Keep in mind how timestamps are preserved across different/like filesystems.
• NameKey - The name of the file (Flux.app)
• VolName - Name of the volume where the file is (OSX)
• PathName - Path (Applications/Flux.app)

According to TN-1150 the volume header's checkedDate is, The date and time when the volume was last checked for consistency. Disk checking tools, such as Disk First Aid, must set this when they perform a disk check. A disk checking tool may use this date to perform periodic checking of a volume.

NOTE: The Volume Checked Date needs more testing.  It can be inconsistent at times, so it shouldn't be relied upon 100%.  I need more testing sample data, which is why I am releasing this 90% complete vs. 100%.  I only have a couple Mac VMs for testing.

Automation

I wrote some Python code to parse this data from within a plist. You will need to run it on a Mac.

This is just proof of concept code. You can import aliasv3_struct.py as such:

from aliasv3_struct import AliasV3Parser

It will return the following that you can sift through and print however you want.

return(cTime, vol_cTime, vol_fs_type, target_type, target_name, target_path, parent_cnid, target_cnid, ' '.join(cnids))

You can run the code as such:

aliasv3.py -f ~/Library/Preferences/com.apple.loginitems.plist

And here is some simple output.

2016-07-04 02:09:28,2015-10-02 07:23:53,H+,Folder,Flux.app,/OSX/Applications/Flux.app,78,727882

Summary

Make sure to validate my findings. Like anything else, there is likely room for improvement or modifications of the interpreted data, etc. It would be helpful if you could extract your volume header and provide the plists I discussed here. That will help with testing as I have limited samples.

Aliases it seems are being replaced by bookmarks, but they are still on newer Mac systems so it's worth knowing the structure of them. I've also reverse engineered the file format for bookmark data, but I haven't gotten around to documenting it. It's still in my head. I will try doing that over the next couple weeks. It's much more robust/complex.

Play Time

If you want to play around with the sample files in this blog you can download:

• Example plist here
• Exported Alias data here
• Grammar here
• Python code here

You can also find Alias data within your array of sidebar items.

~/Library/Preferences/com.apple.sidebarlists.plist 

I've attached a couple examples for my Mac VM here - Network and here - VMware.

Enjoy!

Overview

Understanding the HFS+ file system is an important step to analyzing Mac systems. It tracks all of the metadata associated with files and folders on a system among things.

Suggested Reading

• Technical Note TN1150
• Volume Header
• Open Source – Apple – bsd/hfs

NOTE: Any discussions about Mac systems will assume you are running 10.11.x That's all I have for testing on my Surface Pro 4 :)

HFS+ Structure

The core files (aka – “special files”) that make up the HFS+ filesystem according to TN1150 are listed below. This blog post will cover the Volume Header. I will cover the others in order as they are listed here in subsequest blog posts.

• Volume Header
• Catalog File
• Extents Overflow File
• Attributes File
• Allocation File
• Startup File

Table 1 – Special File Reservation C struct depicts the C struct code for the 16 reserved Catalog Node IDs (CNID/inode) for HFS+ special files per the TN1150

Table 1 – Special File Reservation C struct

enum {
    kHFSRootParentID           = 1,
    kHFSRootFolderID           = 2,
    kHFSExtentsFileID          = 3,
    kHFSCatalogFileID          = 4,
    kHFSBadBlockFileID         = 5,
    kHFSAllocationFileID       = 6,
    kHFSStartupFileID          = 7,
    kHFSAttributesFileID       = 8,
    kHFSRepairCatalogFileID    = 14,
    kHFSBogusExtentFileID      = 15,
    kHFSFirstUserCatalogNodeID = 16
};

Volume Header

Every HFS+ volume must have a volume header. The volume header is important to the overall filesystem because it provides the analyst with volume creation, modification, and checked dates. It specifics the block size, the locations of our 16 special files, boot files (boot.efi), bootable folder (CoreServices), number of files and folders on the volume and also lets us know if journaling is enabled among other details.

There are two volume headers. The volume header and the alternate volume header. The volume header is located at offset 0x400h, which is 1024 bytes from the start of the volume. The alternate volume header is located 1024 bytes before the end of the volume. Both are 512 bytes in size and are at fixed locations.

Extract Volume Header

In this example I show you how you can extract the volume header for analysis.

In our case I extract 4096 bytes, which is our first block. This will contain all of our volume header information that we can use during analysis.

In order to do the extraction, I used the following. If you prefer you can also open the entire disk with your hex editor. For size/portability reasons I will be using dd to extract the first 4096 bytes.

First we will look at the mapping of our disk.

Table 2 - mmls disk layout

sudo mmls /dev/rdisk1

We can see here that item 05 (HFSDemo) is what we want.

GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors

      Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Safety Table
001:  -------   0000000000   0000000039   0000000040   Unallocated
002:  Meta      0000000001   0000000001   0000000001   GPT Header
003:  Meta      0000000002   0000000033   0000000032   Partition Table
004:  000       0000000040   0000409639   0000409600   EFI System Partition
005:  001       0000409640   0030703575   0030293936   HFSDemo
006:  -------   0030703576   0030965759   0000262184   Unallocated

Table 3 - dd block 1 - header volume

sudo dd if=/dev/disk1 of=volheader bs=4096 count=1

After running dd we now have our 4096 byte dump, which will contain our volume header.

1+0 records in
1+0 records out
4096 bytes transferred in 0.000940 secs (4357055 bytes/sec)

Analyze Volume Header

Now that we have the volume header extracted we can move to offset: 0x400h with our hex editor (010 Hex Editor), which is sector 2 (0, 1) as depicted in Figure 1 – HFS+ Offset – 0400h (1,024 bytes). On a Mac, you can easily skip to an offset in 010 by using the Command + L keys. You could have also skipped these bytes using dd as well.

Figure 1 - HFS+ Offset - 0x400h (1,024 bytes)

To make things easier to see and separate out I created a Synalyze It! template for parsing the HFS+ volume header. The results of the grammar file can be seen in Figure 2 – HFS+ Synalyze It! Volume Header Grammar.

You can download the Synalyze It grammar file on my GitHub.

Figure 2 - HFS+ Synalyze It! Volume Header Grammar

NOTE: The volume header creation date is stored in local time, not GMT.

Table 3 – HFS+ Volume Header Output

Here is the parsed hex from Figure 2 above using the Synalyze It! Pro Grammar file I created.

Now that we have "manually" parsed the volume header using a hex editor let’s go over some of the more useful information that we were able to uncover.

Volume Header Analysis Results

• HFS+ filesystem
• H+
• Version 4 = HFS+
• A journaled filesystem - Journaling is enabled
• Creation Date (Local time) - 2016-07-23 11:08:47
• Modify Date(UTC) - 2016-07-23 18:09:36
• Backup Date (UTC) - 00 00 00 00
• Checked Date (UTC) - 2016-07-23 18:08:47
• File count – 66
• Folder count – 19
• Volume UUID - 37a4e9fdeafc14ea
• Block size is 4096
• Total Blocks – 3,786,742
• Free Blocks - 3,777,448
• Drive size = Total Blocks * Block Size (4096), which is 15,510,495,232 bytes, or 15 GB
• Location and Size of Special files (Extent, Catalog, Attributes, and Startup) via the Fork Data Structures.

To make converting the hex times easier I went ahead and wrote a simple Python method below to convert our times and print them out in a human readable format.

Table 4 - Python Datetime Conversion

>>> print mactimeConvert(0xD3B900BF)
2016-07-23 11:08:47
>>> print mactimeConvert(0xD3B96360)
2016-07-23 18:09:36
>>> print mactimeConvert(0xD3B9632F)
2016-07-23 18:08:47

The Easy Way

Now that we parsed the volume header the “hard” way, it is time to go over an easier method using one of my favorite tools.

This is accomplished via Sleuth Kit’s fsstat command.

fsstat displays details of a file system. The “f” set of tools within Sleuth Kit are file system specific (fls, fsstat, ffind, etc.) You could also use the native Mac tool, hdiutil (hdiutil fsid /dev/<rdisk#>).

You can see within Table 5 – fsstat - Sleuth Kit, fsstat provides us with all of the information we were able to obtain via hex analysis and then some.

It also provides us this information within a faction of a second vs. the time required for manual analysis/parsing like we did above.

Table 5 - fsstat - Sleuth Kit

You can run it via:

sudo fsstat /dev/rdisk2s2

And here are the results of using The Sleuth Kit's fsstat command.

FILE SYSTEM INFORMATION
File System Type: HFS+
File System Version: HFS+

Volume Name: HFSDemo
Volume Identifier: 37a4e9fdeafc14ea

Last Mounted By: Mac OS X, Journaled
Volume Unmounted Properly
Mount Count: 203

Creation Date: 	    2016-07-23 11:08:47 (PDT)
Last Written Date:     2016-07-23 11:09:36 (PDT)
Last Backup Date:      0000-00-00 00:00:00 (UTC)
Last Checked Date: 	2016-07-23 11:08:47 (PDT)

Journal Info Block: 117

METADATA INFORMATION
--------------------------------------------
Range: 2 - 108
Bootable Folder ID: 0
Startup App ID: 0
Startup Open Folder ID: 0
Mac OS 8/9 Blessed System Folder ID: 0
Mac OS X Blessed System Folder ID: 0
Number of files: 66
Number of folders: 19

CONTENT INFORMATION
--------------------------------------------
Block Range: 0 - 3786741
Allocation Block Size: 4096
Number of Free Blocks: 3777448

Summary

Keep in mind tools are not always 100% accurate.
I have personally written tools that parsed artifacts incorrectly. I do this all the time. Do not blindly rely on them for your analysis. Learn how to do things manually, then go about writing your own tools, or verifying other people's tools.

Enjoy!

So I was analyzing some bookmark files and noticed the volume creation time didn't make sense. The time was too old.

If you're at all familiar with Bookmark files you should know there are resource properties embedded inside them.

The particular property resource value that was throwing me off was NSURLVolumeCreationDateKey.

According to the Apple Developer documentation a NSURLVolumeCreationDateKey is a:

Key for the volume’s creation date, returned as an NSDate object, or NULL if it cannot be determined (read-only).

Simple enough. It should represent the creation time of the volume.

What didn't make sense was the creation date returned was August 2008 and that couldn't be right. My VM isn't that old.

NSURLVolumeCreationDateKey = "2008-08-22 14:48:36"

Some digging turned up a file called, preformattedHFSVolume.vmdk.

/Applications/VMware\ Fusion.app/Contents/Resources/preformattedHFSVolume.vmdk

When building Mac VM via VMWare Fusion, it appears VMWare uses a pre-formatted HFS+ volume.

If you open the preformattedHFSVolume.vmdk in a hex editor and search for, H+ or HFSJ you should see the following volume header.

Now if we take the hex and convert it into human readable format you get.

>>> print mactimeConvert(0xC4D48244)
2008-08-22 14:48:36

So if you ever wondered why you see this timestamp in your volume header, now you know.

Nothing earth shattering, but neat.

Enjoy!

Overview

This is going to be a multi-part series where I will cover various AWS concepts, terminology and technology. Subsequent posts will be more information security focused. There will be a bit of bias towards Incident Response (IR).

Follow along at the AWS Security Overview Table of Contents

Level Setting

There are exceptional companies (Netflix comes to mind) who are already doing really cool things in the cloud. Both from a Digital Forensics and Incident Response (DFIR) and Engineering perspective. There are also a couple good cloud security tools, a couple cloud security blogs, and I have even seen a couple CON talks on the topic.

Then there is the majority.

The individual(s) that got a call to respond to a compromised company only to find out the client's servers are in the cloud and the client's engineers didn't think their deployment through and no one knows anything about cloud. Oh yeah, and nothing is documented either. They give you console access, and say, "Here you go. Hook us up."

Then there is the Security Operations Center (SOC) team that was told during a morning meeting, "Hey, we are moving to the cloud, and oh yeah, we already have a dozen systems in there. A few of them are critical. Information security is a roadblock so we just deployed them."

This is my small part to try and close the gap a tad bit for the people that experience these kind of scenarios in their workplace.

NOTE:  For any definitions, I am going to use it more or less verbatim from the AWS documentation.  Their documentation is robust and usually pretty current.  So rather than citing everything, just assume I have gotten it directly from the respective AWS documentation.

Terminology

One of the challenges I had when first learning about cloud was the terminology was more or less all new and the concepts were foreign to me. I will start with the foundations in this post. In later posts I will get into more details for each respective technology.

Let's begin...

Regions

Each region is a separate geographic area. The list of currently supported regions and services can be found here. Again, this could change, especially with re:Invent 2017 around the corner. I will be using, Northern Virginia because that's where I have all my stuff located.

Some services are only located in certain regions, some are global and span all regions. For example, Identity and Access Management (IAM) does not support regions. It's global. Resources aren't replicated across regions. It is your responsibility to replicate them. There are fees associated with transferring data between regions.

Each region is completely independent, which achieves a high level of fault tolerance. It also ensures that you're meeting your legal compliance requirements. For example, maybe your organization conducts business in a country where you're not allowed to move citizen data outside of a particular country/region.

Choosing the right region can also; provide cost savings, decreases latency for users/customers, etc.

A region consists of two or more Availability Zones (AZs).

Availability Zones

Within a region, there are multiple (two or more - isolated) locations referred to as, Availability Zones, or "AZs".

An Availability Zone is represented by a region code followed by a letter identifier; for example, us-east-1a. The AZs are composed of one or more data centers. No one data center can be part of two AZs. This allows for independent failure zones.

Each AZ is isolated (physically separated), but the AZs in a region are connected through low-latency links. It should be noted that regions are connected over the "internet". Regions do not have low-latency links connecting them together like AZs do.

Virtual Private Cloud (VPC)

A VPC is a virtual network dedicated to your AWS account. It resembles the traditional networks we all use today. It's the networking layer for Amazon Elastic Compute Cloud (EC2). It's logically isolated from other virtual networks within AWS. This means you're network isn't going to cross paths with my network. I can't see your stuff.

As soon as you create an AWS account, you're assigned a "default VPC". This default VPC has everything you need to get started. You can launch an Amazon Machine Image (AMI) and have a web server up and running in just a few mins.

You can choose to keep this default VPC and make modifications to it to fit your environment, or you can create another VPC. When you create VPCs within your AWS account they are called, "non-default VPCs". You can also delete it if you want. If you delete it, and change your mind, you can recreate your default VPC within the VPC dashboard as well.

When you create a non-default VPC you will need to assign an IPv4 address range for the VPC (Ex. 10.0.0.0/16). You can, if you want, assign IPv6 CIDR block to your VPC as well.

Within your VPC, you can (just like a traditional network), configure an IP address range, specify/create private/public subnets, manage your route tables, implement network gateways, configure security settings among others.

Each VPC will have a unique ID (Ex. vpc-0cb12345).

VPCs span all the AZs. You can also add more than one subnet in a particular AZ. Subnets cannot span AZs.

We will discuss more about VPCs as we go on. A lot of other things make up a VPC. Those other things, will get their own sections (Ex. subnets).

Subnets

As mentioned above in the VPC section, you're assigned a, "default VPC".
You're also assigned a "default subnet". This default subnet is a public subnet, which means you can reach the internet out of the box.

When you create subnets within this default VPC, they are called, "non-default subnets". If you create an instances within those non-default subnets, they will be assigned IPv4 private addressing that's not routable to the internet (by default). They are; however, able to talk to other subnets via target local as you will see inside the respective route table. Each subnet that is created is assigned a unique ID (Ex. subnet-d123456d)

Subnets must reside within one Availability Zone and cannot span AZs.

If a subnet's traffic is routed to an internet gateway, the subnet is known as a public subnet. A subnet without a route to an IGW, is a private subnet. If the subnet has traffic routed to a virtual private gateway, it's referred to as a VPN-only subnet.

If you wanted a system within a public subnet to communicate to the internet it would need to have a public IP or a Elastic IP (EIP). Each instance that receives a public IP address is also given an external DNS hostname.

Each subnet must be associated with a route table. When you create a subnet, it will be automatically associated with the main route table of the VPC.

Subnets use Network Access Control Lists (ACLs) to manage inbound and outbound traffic for your subnet(s). They act as firewalls at the subnet level.

I'll write more about the security features in subsequent posts.

IP Addressing

I mentioned in the subnet section that, "If you wanted a system within a public subnet to communicate to the internet it would need to have a public IP or an Elastic IP (EIP)."

When you launch an instance into a VPC, your subnet has an attribute that determines whether instances launched into that subnet receive a public IP address.

This is configured here within your VPC dashboard. Subnet Actions -> Modify auto-assign IP settings

You can also enable or disable the public IP addressing feature during launch, which overrides the subnet's public IP addressing attribute.

By default, Amazon assigns a public IP address to instances launched in the default VPC, and Amzon does not assign a public IP address to instances launched in a non-default subnet.

The public IP address is not assigned to your account. If you stop (or terminate) your instance(s), you will receive a new public IP address when it's restarted. If you decide later to assign an Elastic IP (EIP) address to your instance, you will not be given the already assigned public IP address. It will give you a new one. If you change your mind an no longer want an EIP, when you disassociate that EIP you will receive a new public IP.

If you want a persistent public IP address for your systems, use EIP.

I bring this section up because I can foresee a case where someone has their web server popped, they panic, stop the instance, fiddle around a bit, then restart it. Then the security team is reviewing logs and they wonder why there is two different public IP addresses for this web server they are analyzing.

More on Network ACLs in a subsequent post.

Internet Gateways (IGW)

Amazon says the IGW is, "a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in your VPC and the Internet." Amazon manages the IGW. It just, works.

It serves two purposes. 1) provide a target in your VPC route tables for Internet-routable traffic 2) perform network address translation (NAT) for instances that have been assigned public IPv4 addresses.

IGWs are attached to the VPC. The default VPC that's created when you open an AWS account already has an IGW associated with it. If you create non-default VPCs, and you want them to talk to the internet you will need to associate an IGW.

You can then create routes with your associated subnets and create a route that points to the target, igw-xxxxxxxx.

Public Subnet Route Example. Note the route to the 0.0.0.0/0 target, igw-xxxxxxxx. This indicates that this is subnet can route out to the internet. If you do not have this route, you're system wont be able to communicate out to the internet.

Private Subnet Route Example. Notice the missing route to the igw.

You can also have VPN-only subnets if traffic is routed to a virtual private gateway (vgw-xxxxxxxx). I don't have an example of this within my account. You would use something like this for example to connect your corporate environment (or house) to your AWS environment.

NAT Gateways

You can use a NAT gateway to enable instances within a private subnet to connect to the internet (or other AWS services) without having to expose them to the internet via a public subnet. This can be useful for software patches (Ex. yum/apt-get updates), or to isolate a web server in a public subnet behind a NAT gateway.

The NAT gateway prevents the internet from connecting to the instances within that private subnet.

If you use IPv6, you will need to use an egress-only internet gateway as IPv6 is not supported with a NAT gateway (at the time of this post).

You can monitor you NAT Gateway via CloudWatch, which are sent in intervals of 1 min. Some of the metrics sent are: PacketsOutToDestination, PacketOutToSource, PacketsInFromSource, PacketsInFromDestination, BytesOutToDestination, etc. Look at the VPC documentation for specifics about those metrics.

These metrics could be useful in an exfiltration scenario. You can also use flow logs to capture the traffic for both, NAT Gateways and NAT Instances. Flow logs will be a separate post.

There are also NAT Instances, which vary a bit from NAT Gateways. You can read more about them here.

Route Tables

Just like physical networking, a route table within AWS; contains routes (or rules) that route traffic to its respective route (or direction).

"A subnet can only be associated with one route table at a time, but you can associate multiple subnets with the same route table. If you don't explicitly associate a subnet with a particular route table, the subnet is implicitly associated with the main route table."

You must update the route tables for your subnets that use gateways or connections. For example, if you setup an IGW, you will need to modify the route table to traffic knows to use the IGW.

You can determine what routes a subnet is associated with my going to the VPC Dashboard -> Subnets -> Select your subnet -> Route Table tab (next to the Summary tab).

You can also, determine how many and which subnets are explicitly associated with a route table.

VPN Connections

Like a traditional VPN, you can connect your VPC to remote networks, whether it's your home or your corporate network (or both). In AWS, a VPN connection is defined as, "the connection between your VPC and your own network."

Setting up a VPN is quite simple. You will need to setup a; Virtual Private Gateway (VPN concentrator) and a Customer Gateway, which is the device (HW or SW) on your end of the connection. You must then update your route table to include the routes used by the VPN.

It uses a 32 bit pre-shared key (PSK) to establish the initial Initial Key Exchange (IKE) Security Association between the virtual private gateway and customer gateway.

If your key is compromised you can replace it. You will have to delete your VPN connection and then create a new one using the same virtual private gateway. Then you will need to configure the new keys on your customer gateway.

If a subnet doesn't have a route to an IGW, but has traffic routed to a virtual private gateway, it's called a VPN-only subnet.

You can monitor the VPN tunnel metrics using CoudWatch. They are recorded for a period up to 15 months. VPN metric data is automatically sent to CloudWatch.

Visual

So here we have a visual depicting most of the topics we covered briefly above. I have not added any computing instances yet. I will build on this diagram in future posts to help visualize all this terminology. Some of the services above are an additional fee (VPN and NAT Gateway), so I do not have them configured in my environment (hence, not on the diagram).

In Summary

Now that we have laid the networking foundations, I will start getting into more specifics about each of these in subsequent posts. There is A LOT to cover. It can get long winded, so separate posts are necessary.

Enjoy!