Wear Levelling
Most flash drives are NAND EEPROM devices capable of 100,000 to 1 million erase and write cycles. The lifetime of the flash drive depends on endurance of the flash chip. To extend the life of flash drives, manufacturers often implement wear-levelling (also referred as wear-leveling).
Wear-levelling mechanism spreads write cycles across a flash chip, thus reducing continual usage of the same areas of the flash chip, and as a result promotes even usage of all memory cells.
What this means for forensic examiners? The content of a file that is no longer exists from the point of view of the file system may have been fully or partially changed by the wear-levelling algorithm. On many NAND flash memory devices this occurs upon writing the new data.
NAND flash drives are not very efficient at random writes due to the requirement of an application to locate a free block, before it can write to it. If such block is not available, the block must be fully erased which takes additional time, thus reducing the efficiency of the device. Different manufacturers are taking different approaches to tackle this problem. Some implement additional controllers or/and memory into their flash drives. Some change the software (firmware) and wear-levelling algorithms that shuffles "unallocated" free space every time the device is read, so when the application is about to write the new data, free blocks are already available to the application for writing.
Acquiring these devices require an additional step that from my experience is rarely taken. The standard procedure is to simply connect such USB device to a forensic machine via hardware or software write blocking device and let the forensic software to do the acquisition and verification. There are two problems with this approach.
- Most forensic tools verify (calculate MD5 or SHA1 hash) of the device, then acquire the data followed by MD5 or SHA1 verification of the image. There is no verification of the physical device after that. So, we essentially rely on the write blocker to prevent any changes.
- Some USB devices (approximately one in every ten from my experience) will produce different cryptographic hash every time you calculate it, despite the fact that no write is allowed. So, by simply reading such devices, we are changing something inside these drives.
So, what is actually changed on the drive and how to deal with this issue? The good news is that existing files are not changed and this can be easily confirmed by comparing hash values of files from two images of the same device taken at a different time. X-Way forensics is probably the best tool for this task.
By utilising the above mentioned tool and its terminology we can see that changes occurred in 'Free space' and 'previously existed files'. It is up to the forensic examiner to deal with admissibility of the data/evidence extracted from 'Free space'. Taking an additional image of the device, extracting (carving) files and comparing these files with the files from the first image is one of these techniques. There will be many files that are changed by the sector shuffling, thanks to the wear-levelling algorithm.
Deletion/wiping
Additionally, because of the wear-levelling mechanism and dynamic mapping of logical to physical sectors, some file artefacts may be left behind even after "secure wiping" of the USB flash drive.Ordinary hard disks in general do not have wear-levelling implemented; however this may soon change due to becoming increasingly popular in notebooks solid-state drives.
27 February 2009
Yes,
The issue does exist despite some people finding it hard to believe, and it is here to stay for some time. The only way to deal with this is through the correctly devised procedures that in general can be described as:
1. Identifying the device with the specific wear-levelling behaviour (via hashing before and after the procedure for example).
2. Isolate the existing (not marked as deleted) from the deleted files. Verify the integrity of the existing files.
3. Deal with the deleted files in a way that the accurate and verifiable data can be presented in court.
------------------------------------------------------------------------------------------
"Knowledge is dynamic in nature, today's knowledge may well become tomorrow's ignorance if an individual or organisation fails to update knowledge as environmental conditions change."
Turban, E., Leidner, D., Mclean, E., Wetherbe, J., Information Technology for Management: Transforming Organizations in the Digital Economy. Wiley; 6 edition (March 5, 2007)
----------------------------------------------------------------------------------------------------------
March 2009
Here is the link to a series of youtube videos of 'DEFCON 16' presentation by Scott Moulton who does a good job of explaining how the concept works.
5 November 2009
Another good article about SSD and NAND flash technology.
4 comments:
Thanks for this very interesting article. Can you tell me the specific flash media you have experienced which generated different hash value from two images of the same device taken at a different time?
Here are two examples: EagleTec 8Gb Turbo Series and 16GB PQI U310 USB flash drives.
Hello Andre,
In your research did you identify any tools or a methodology to maximise the potential to recover files from unallocated space given the wear-leveling (fragmentation and over-writing) that can occur on each read?
No, I haven’t. The devices are a part of various ongoing investigations, so it was too risky to give them too much stress :-) . May be one day some R&D guys would do that and share it with us.
Post a Comment