Saturday, May 10, 2014

Distributed Processing Notes

I tested distributed case processing and password cracking today by adding Amazon EC2 instances to the local processing resources. Purpose - tmp improve processing (& decryption) speed with security and budget in mind. I used Amazon "compute optimised" instances "c3.8xlarge", each with 32 Virtual CPU; 60GB RAM; 2 x 320 (SSD) and 10 Gigabit Network. "c3.8xlarge" instance costs around $3 USD per hour. My Internet link was a bottleneck, because it only supports 15.62 Mbps (15615 kbps). I used 'soon to be decommissioned' Free LogMeIN service, participating nodes were setup as "MESH" Network. Free account only supports up to five nodes in Mesh network.

In order to use LogMeIN Hamachi, it must be installed on each computer. Windows Server 2008 OS had issues with failing to connect to tunnelling engine, so to save time Windows Server 2003 OS was used instead. No problems encountered with installing and connecting the instances to the Mesh. Perhaps Windows Server 2008 R2 or later OS would also work without too much tinkering. I called my network "Passware", but had to use network ID instead to connect each instance.

Firstly, Passware Password Recovery Kit Forensic was used to decrypt SAM + System registry files. The tool has its own integration with Amazon Cloud, however I always had connectivity issues with it. Additionally,  the purpose of this exercise was to make sure no data is transferred between the local network and external resources in a clear.

After the 1st EC2 instance was connected to the local workstation running Passware, I saw a significant improvement in speed compared to two (1CPU) local workstations at brute-forcing NTLM hashes.

When three remaining EC2 instances were connected, each added 153 000 000 pwd/sec to the pool.

AccessData Password Recovery Toolkit (PRTK), oclHashcat v1.20 and ElcomSoft Distributed Password Recovery, all produced decent results. I would be even more happier if I could make the new EC2 NVIDIA GRID GPU driver to work.

Distributed processing of evidence with X-Ways Forensics, NUIX and Forensic Toolkit (FTK) also produced some good results, however I would not call these tests scientific :-). 

X-Ways Forensics calls the process "Distributed Volume Snapshot Refinement". It splits  the task of processing different evidence objects of the same case between multiple computers, that must live on the same network to be able to process the evidence at the same time. 

NUIX is a completely deferent beast, it is the fastest evidence processing solution I ever worked with. My network has quickly became a bottleneck with NUIX, which was expected. 

Setting up FTK distributed processing engine took me some time ( configuring shares, DB connectivity etc.). Processing of 50 Gb E01 image was completed about 30% faster compared to a stand-alone machine.

Several Amazon EC2 instances provide 10 Gigabit Network option, which is more than enough for distributed processing. With faster internet connection at the lab it makes sense to use Amazon EC2 resources when time is limited or demand for these resources is higher then usual. Configuration time is relatively short, CPU overhead (due to encryption) is only slightly noticeable and Security should not be the issue with VPN tunnel established between all processing nodes. 

No comments: