Overview
Before I get into the real reason for this post, I want to quickly go over what kMDItemDateAdded is, and then I will discuss where a popular tool is missing the timestamp.
kMDItemDateAdded
kMDItemDateAdded is an HFS+ metadata attribute that also happens to be indexed by Spotlight. Spotlight is used to index your system. In simple terms, Spotlight allows you the ability to find things easily on your Mac.
The file structure for Spotlight is located at /.Spotlight-V100/Store-V2 (newer versions). There is no free/open source Spotlight Index database parser to my knowledge (if you know of one let me know). The Spotlight index database is also undocumented.
If you run mdimport (from a Mac):
mdimport -A |grep -i Date
You will get a list of the date attributes Spotlight has the capability of storing/using. If you just run, mdimport -A you will see all of the attributes available.
According to the output from mdimport, kMDItemDateAdded is the, "Date when this item was last moved". This date is updated when it's moved, and gets indexed by Spotlight (pretty much instantly). I'll show you an example below.
Date Added Example
NOTE: This following example assumes Spotlight Indexing is enabled on your system.
Let's say I do the following from my Desktop:
- Create a folder called, TestFolder
- Create a file called, Testfile.txt
- Run mdls against Testfile.txt
- Move Testfile.txt into TestFolder
- Run mdls against Testfolder/Testfile.txt
- Compare the kMDItemDateAdded entry from the first mdls to the second (after the move)
After completing the commands below you will notice that the kMDItemDateAdded entry was updated when Testfile.txt was moved into another directory.
- mkdir Testfolder
- touch Testfile.txt
- mdls Testfile.txt
<snip>
kMDItemDateAdded = 2016-08-06 23:35:28 +0000
kMDItemDisplayName = "Testfile.txt"
<snip>
- mv Testfile.txt Testfolder/
- mdls Testfolder/Testfile.txt
<snip>
kMDItemDateAdded = 2016-08-07 00:16:15 +0000
kMDItemDisplayName = "Testfile.txt"
It is worth noting that kMDItemDateAdded attribute will only exist if the file/folder on the HFS+ volume is being indexed by Spotlight, but the HFS+ attribute (more on this below) will always exist.
Catalog File
The Spotlight index is not the only place to obtain the date added timestamp; however, some tools are not pulling the data out and presenting it to you. The age old, "If the tool tells me it's, ok then it must be, ok." Well, it's not in this case.
TN-1150 is Outdated
Most people reference TN-1150 as the sole-source of information for all things HFS/HFS+.
It is a great source of information, but it's not the most current, so you need to also look at some documented source code to find the latest information.
If you were to only look at TN-1150 you would see this ExtendedFileInfo C struct within the Catalog File section.
struct ExtendedFileInfo {
SInt16 reserved1[4];
UInt16 extendedFinderFlags;
SInt16 reserved2;
SInt32 putAwayFolderID;
};
typedef struct ExtendedFileInfo ExtendedFileInfo;
Now here is where it gets exciting. If you look at an updated version of hfs_format.h you would see this FndrExtendedFileInfo C struct:
struct FndrExtendedFileInfo {
u_int32_t reserved1;
u_int32_t date_added;
u_int16_t extended_flags;
u_int16_t reserved2;
u_int32_t reserved3;
} __attribute__((aligned(2), packed));
Note the second 4 byte entry, date_added. Yep... That's right... It's also documented within the Catalog File, not just the spotlight index.
Furthermore, if you looked at the API documentation you would see this, which further labels some of the fields that were previously reserved/unused bytes. Specifically, document_id and write_gen_counter. I haven't had time to look into these two fields. On a few tests I have done they have been 00s (zeros).
struct FndrExtendedFileInfo {
u_int32_t document_id;
u_int32_t date_added;
u_int16_t extended_flags;
u_int16_t reserved2;
u_int32_t write_gen_counter;
};
Catalog File Date Added Example
For this example I will be skipping over the details of a catalog file and diving right into the specifics of this blog post.
In this example I have a file called, If You're Not Helping Employees Learn You're Not Doing Your Job.pdf. This was a HBR article I downloaded to my Desktop as part of my MBA program.
You can see the hex of this catalog file record here:
In this example we get our MACB timestamps (in blue), which are parsed out just fine by common tools. So nothing really interesting there.
When we convert the hex we get the following times:
0xD3B854F3: 2016-07-22 22:55:47
0xD3B854F3: 2016-07-22 22:55:47
0xD3B854F4: 2016-07-22 22:55:48
0xD3BD57EA: 2016-07-26 18:09:46
0x00000000: 1904-01-01 00:00:00
In Sleuth Kit (using the istat command) we get:
Created: 2016-07-22 22:55:47
Content Modified: 2016-07-22 22:55:47
Attributes Modified: 2016-07-22 22:55:48
Accessed: 2016-07-26 18:09:46
Backed Up: 0000-00-00 00:00:00
So we know the times are all matching up, but we are missing one timestamp, date added.
In this image I highlighted; 0x5792A473, which is the date_added timestamp we have been looking for.
If you convert 0x5792A473 to an unsigned integer using Big-Endian you get an epoch timestamp of, 1469228147.
If you then convert it you will get:
- 2016-07-22 22:55:47
Compare mdls Output
Now we can compare the converted time from above to mdls.
mdls Desktop/If\ You’re\ Not\ Helping\ Employees\ Learn\,\ You’re\ Not\ Doing\ Your\ Job.pdf
And we get:
kMDItemDateAdded = 2016-07-22 22:55:47 +0000
kMDItemDisplayName = "If You’re Not Helping Employees Learn, You’re Not Doing Your Job.pdf"
In this example I used mdls to compare the timestamps. If Spotlight indexing wasn't enabled on my test system we wouldn't have been able to use mdls as a verification mechanism; however, this the date added timestamp is a HFS+ attribute it would have still been located within the Catalog file.
The Sleuth Kit (TSK)
It should be noted that I call out TSK here only because I use the tools all the time and I know they are used by a lot of people in industry. I created a feature/issue request at the same time I published this blog.
So where it goes wrong with TSK I believe is here under, CATALOG Record structures.
You can see that the first 16 bytes of the C struct are reserved (uinit8_t res1[8]).
The second set of 4 bytes should be date_added, with the first 4 bytes still reserved. I am not a C coder so I could be wrong, but I suspect this is the C struct that needs modified.
Current Source as of August 2016
typedef struct {
uint8_t res1[8]; /* reserved 1 */
uint8_t extflags[2]; /* extended finder flags */
uint8_t res2[2]; /* reserved 2 */
uint8_t folderid[4]; /* putaway folder id */
} hfs_extendedfileinfo;
Possible Solution as of August 2016
This is a possible solution to fixing the timestamp. Obviously more code will need to be added/changed as well, but all and all this should represent the updated C struct (I think).
typedef struct {
uint32_t document_id; /* Not sure?? */
uint32_t date_added; /* Date added */
uint16_t extflags; /* ext finder flags */
uint16_t res2; /* reserved 2 */
uint32_t write_gen_counter /* Not sure?? */
} hfs_extendedfileinfo;
If it was as simple as adding another line of code I would have done it, but I am not a C developer so the best I could do was find what I believe is the problem, read the source, write this blog post, and file a feature/issue request to the awesome TSK team.
Summary
So yeah, if we get the feature added to TSK this will give us another timestamp to include in our timelines. Also, in the meantime you can use mdls and/or hex to determine the timestamps while you wait.
For the other tool developers, have at it.
Enjoy!