Over the last few months I’ve been working on getting the new version of Epilog ready for release. It’s always an exciting time as, inevitably, I’ve been adding to my wish-list of new features and tweaks almost relentlessly since the last version was released. Add to this the really valuable feedback and requests we have received from our user-base – it’s a lot of work to get everything implemented, but I hope that you’ll find that version 1.2 of epilog is the most useful, and powerful version yet.
I wanted to take the time to put together a blog detailing some of the new features and my thoughts behind them which will hopefully provide some insight into their intended use and help you get even more out of epilog.
New Signature Format and Signature Builder
The signature search in epilog has always been the most targeted algorithm, allowing epilog to recover records not only from database files, but also from arbitrary blobs of data such as unallocated space or mobile device dumps. That being said, the form that the signatures took in previous versions was tightly tied to SQLite’s internal description of data types – the Serial Type Codes and also betrayed Epilog’s roots in Python (the original prototypes for Epilog were written in Python, and the “square bracket and numbers” layout of the signature maps directly into the Python “list of lists” structure that the prototypes used to represent signatures).
Example Legacy Style Signature: ,[1,2,3,4],,[0,-2],[0,-2],[1,8,9],,[1,8,9]
Although I still think it is beneficial for those working with SQLite data to have some understanding of how serial type codes work and what they mean, when it came to generating signatures I recognise that this acts as a barrier for many users. Furthermore, this close tie-in to SQLite’s internal structure meant that representing other types of validation of recovered data were not feasible.
With this in mind, I have redesigned the signature format – the data types used by SQLite internally are still present, however they are now dealt with in such a way the user is not required to know the meaning of the serial type codes off by heart in order for a signature to be built. Beyond this, the new signature format allows additional validation of recovered values – minimum and maximum values for numerical data, lengths of blob and text data, and regex validation of text fields. Careful use of this additional validation means that the number of false positives can be further reduced, which is especially useful when working in unallocated space or with hex dumps.
Although the new signature format is already more powerful and easier to work with, I wanted to make the generation of Signatures even easier. With the old, legacy signature format, the user was forced to leave epilog in favour of a text editor in order to write signatures, I wanted to remove this break in workflow, so epilog 1.2 introduces a brand new, integrated signature builder.
Accessed from the “Tools” menu, the Signature Builder allows you to create and edit signature files in a graphical interface – removing the requirement to open the signature files in a text editor (although you are free to continue working like this if you prefer – the new format is still XML based).
The Signature Builder can also import the old signature format – useful if you want to add the new validation capabilities to any old signatures you wish to use. More importantly, the Signature Builder can auto-generate a signature file from a live database. The quality of the signature will be greatly improved if the live database is well populated – though it is still recommended that you check the auto-generated signature for any improvements you can make (epilog does a better job than ever in this respect – but a human will always win when it comes to contextualising the data).
Both old-style and new-style signatures can be used in epilog’s signature search – so don’t worry if you have written signatures in the legacy format – they still work just as well as they always did, but purely from a usability stand-point, I highly recommend using the new format for new signatures you create.
Write Ahead Log Timelining
I wrote about the Forensic Implications of the SQLite Write Ahead Log in a previous blog post: The Forensic Implications of SQLite’s Write Ahead Log so I will hold back on the technical details here (I do recommend that you have a read of that post if you do any work with SQLite files though).
epilog’s Write Ahead Log Timelining tab takes a “-wal” file and its associated database and examines all of the records in the WAL and orders them based on the criteria described in the blog post mentioned above. Each record is categorised as one of: “First Occurrence” (records on a page that has not previously been encountered in the WAL), “New” (records when they are first encountered on a page which has previously been encountered in the WAL), “Update” (records that have previously been encountered but that how contain altered data) and “Delete” (when a page that was previously encountered no longer contains a record that was previously present). By right-clicking a record in the WAL Timelining tab you can jump to the next (or previous) event associated with that record.
This allows you to follow the flow of changes made to the database contained within the WAL. An area that I have explored with this feature is working with messaging databases and determining when the status of a message has changed (unread to read for example) or when a message has been deleted. By examining the flow of the events occurring either side of this record it may be possible to give a time-range within which the change occurred – something that would not previously have been possible.
Brute Force Live Record Recovery
One type of request which I have assisted with a number of times since the release of the last version relates to the recovery of live records in databases which are corrupt (usually after being recovered from unallocated space). These databases will often not work or only partially work when viewed in traditional SQLite viewers. The Database Forensics tab in Epilog has gained a new option: “Attempt Brute Force Live Record Recovery” which, when enabled, will recover records from live sections of a database, enabling live records in a corrupt database to be recovered.
One potential frustration encountered with previous versions of epilog was when multiple instances of the same record were recovered, for example when running multiple algorithms against a database, or when working with a WAL file (where duplication of data is rife). epilog 1.2 has an option to enable De-Duplication of records during recovery. epilog’s de-duplication algorithm is sensitive to keeping the “best” version of a record. For example, if the same record has been recovered once without a ROWID, and once with, the “superior” version with the ROWID will be retained.
Other new features
Alongside the features above a number of other new features and tweaks have been added:
- Signature Search ROWID Recovery – The signature search algorithm in all recovery modes can now optionally attempt to recover the ROWID of the record. This is especially useful when the ROWID is the field upon which table relationships are built
- INSERT Statement Export Improvements – The INSERT statement export dialog can now auto-detect the most likely tables of the recovered records and auto-populate the table names, allowing the user to export multiple tables at once, streamlining the process
- Database Rebuilder Additional Controls – Users can now select to copy live data from individual tables rather than all or none
I hope that you enjoy using the new version of epilog and that it continues to be useful in your investigations, as always if you have any questions, feedback or suggestions please get in touch on firstname.lastname@example.org.
Sign up to receive the latest news and insight from CCL Group.