Getting More Out of ESE Databases

The Extensible Storage Engine (ESE) is a database technology developed by Microsoft. ESE databases have featured in Microsoft Windows operating systems for some time now and aren’t always the nicest database format to deal with. The Windows.edb and Contacts.edb are among many of the ESE databases found in Windows that you may be familiar with.

With Windows 8 and Windows Phone 8, even more of these ESE databases exist for us to find evidence in. Internet Explorer 10 and 11 also uses an ESE database (called WebCacheV##.dat) instead of index.dat files to store history records.

Repair or Recovery

Depending on what tool you use to view and examine an ESE DB, you may be familiar with the term ‘Dirty Shutdown’. If you are using a tool that uses the ESE API, you’ll need to change the state of the database before you can open it. The easiest and probably most common way of fixing this is to use the ‘repair’ option with the ‘esentutl’ utility with the command:

esentutl /p "c:\esedbfilename.edb"

However, as you can see below, this isn’t the only option:

esentutl1

Instead of using the repair option, we can use the recovery option. This probably won’t come as news to anyone with a System Admin background, as this appears to be a common procedure for fixing Microsoft Exchange and Active Directory databases. This recovery option should update the database from the database log files that are usually stored in the same location. Thus, when extracting the ESE database file, we also need to extract the other files from the same directory; these will usually include .log, .jrs and.chk files. For the most success using the recovery option, I used the syntax:

esentutl /r E00 /l "c:\LogFileDirectory" /s "C:\CheckpointFileDirectory" /d "C:\ESEDBDirectory"

It should be noted here that the paths in this command are not the full path of the ESE database or log files, but the path to the directory in which they reside. ‘E00’ are the first three characters of the log files (and usually the checkpoint files). ‘/l’ is followed by the path of the directory in which the logs files reside. ‘/s’ is followed by the path of the directory in which the checkpoint files reside, and ‘/d’ is the path of the directory in which the ESE database resides. In most cases, all three of these paths will be the same.

Does the ‘Recovery’ option always work? No, but there may be a reason for this. Whilst I was attempting (and failing) to use the ‘Recovery’ option on an ESE database called ‘store.vol’ from a Windows Phone 8 device, I thought I would have a quick look at my Windows Event Logs. These Event Logs (as well as some online research), suggested that this might be an issue with the ESE database version; I was using Windows 7, and this ESE database was from a Window Phone 8 device.

By using the command below, we might be able to get a better idea of what is going on:

esentutl /mh "c:\esedbfilename.edb"

This command shows us the header information from the ESE database file. Part of this information can be seen in the screenshot below:

esentutl_mh1

As can be seen, these version numbers don’t match, and suggest (at least to me) that the ESE database engine used in Windows Phone 8 is newer than the version used in my Windows 7 workstation (using the ‘Repair’ option doesn’t appear to be an issue with this version mismatch).

However, using a workstation with the Window 8 OS installed, we can see that the same version of the ESE database engine appears to be in use:

win8_mh2

Using the ‘Recovery’ option, with the same command as above, now appears to work. We now have a database in a ‘Clean Shutdown’ state that we can open.

The ‘store.vol’ database from Windows Phone 8 stores a number of different records such as messages, contact’s information and information relating to the different account types in use i.e. email accounts.

In this instance, using the ‘Recovery’ option instead of the ‘Repair’ option hasn’t provided us with any additional messages or contacts. The number of records within the “Folder” table has however increased.

WinPhoneFolderRepair2
After the ‘esentutl /p’ (Repair) command is run.
WinPhoneFolderRecovery2
After the ‘esentutl /r’ (Recovery) command is run.

In this instance, these additional ‘Folders’ mostly related to the Skype account that was being used on the phone.

However, using the ‘Recover’ option isn’t always a good thing; in some cases, it may actually reduce the number of records within the database. This is presumably because those records have been deleted for one reason or another and the database file hasn’t yet been updated to reflect this record deletion.

Data Types

Despite the ESE database format supporting many different column data types, it would seem that data isn’t always stored or presented in a way you would expect. For example, the screenshot below shows some data from Internet Explorer 10 cache records stored with a ‘WebCacheV01.dat’ file.

IE_Cache1

As you can see, the data within the ‘ResponseHeaders’ column isn’t readable text (all the ESE database viewing/export tools I’ve used displays this data in this format). This data appears to be a string of hexadecimal:

ContentViewer

If however, we convert this hexadecimal sequence to ASCII encoded text, we can see something that actually makes sense:

ContentViewerascii2

Examples like this can be found in many ESE databases. The ‘store.vol’ database from a Windows Phone 8 device also contains examples of this behaviour. In one of the columns from a record in the ‘Contact’ table, we are presented with the following string:

630063006C00740065007300740035000000

If we convert this hexadecimal to UTF-16 encoded text, we get the following:

ccltest5

This is actually the name of the contact this record relates to. The name ‘ccltest5’ actually already exists in a number of other columns for this record, so we haven’t really learnt anything new from this.

This behaviour is presumably mostly due to the result of viewing these databases in a generic ESE DB viewer. This textual data may not necessarily be physically stored differently from other textual data, it’s just that they may not be stored within a column that is marked as being of the data type ‘Text’ or ‘Long Text’; therefore the ESE DB viewer will not retrieve the data as text. It could for example, be stored as the data type ‘Long Binary’. Obviously, the programmer of the application using the ESE database will know exactly what this data is supposed to represent and process it accordingly.

It is also worth noting that this data may not always represent textual data. A good example of this is in a database called ‘ModernPhoto.edb’ from Windows 8. A screenshot is shown below:

ModernPhoto.edb

As you can see, the column named ‘MegastripThumbData’ contains a string of hexadecimal. The beginning of this hexadecimal string (\xFF\xD8\xFF…) should look familiar; yes, stored within this column is a JPEG image!

Many of these phenomena were encountered during the development of internal tools; both a generic ESE database viewer and a tool designed to deal with Windows Phone 8 artefacts, including the ‘store.vol’. As the development of both these tools is still on going, I’m sure we will encounter many more ways in which data can appear to be obfuscated.

When examining ESE databases, don’t disregard them too soon; you never know where the evidence may be hiding.

Oliver Hartshorn (Senior Digital Forensic Analyst)

Sign up

Sign up to receive the latest news and insight from CCL Group.