josh moulin: designing a mobile digital forensic lab on a budget

Designing a Mobile Digital Forensics Lab on a Budget of 18 Joshua S. Moulin – GSEC,CCENT,GCFA,CFCE,CAWFE,DFCP,ACE,CEECS http://JoshMoulin.com

Designing a Mobile Digital Forensics Lab on a Budget Developing a Business Justification When I began investigating cyber crimes and seizing digital evidence, it was rare to seize more than ten items of digital evidence from a residential search warrant. Usually a suspect would have a desktop and laptop computer, a cellular phone, and some loose media like floppy disks or CDs. It was easy to identify the digital evidence and the capacity was relatively small, allowing for faster forensic imaging and analysis. As technology became more affordable and accessible with larger storage capacities, my digital forensics laboratory began feeling the effects. A typical residential search warrant started to yield dozens of digital devices, all requiring a forensic examination by trained analysts. I remember one search warrant that was served at a home in a child sexual exploitation case where over 80 items were seized. Devices such as computers, tablets, smart phones, CDs, DVDs, USB devices, camera cards, and network storage devices started becoming commonplace. New operating systems and increased security controls and encryption along with the sheer volume of evidence being seized placed an even higher demand on the few digital forensic examiners available. Every crime imaginable has a nexus to electronic evidence. Couple this fact with an increase in electronic evidence being seized at every crime scene and it doesn’t take long to watch the backlog and turnaround time of a forensics lab grow exponentially. Since digital evidence is unique from other traditional evidence in that it can be the instrumentality to commit a crime (child exploitation, network intrusions), it may be the fruit of the crime (stolen in a burglary), or it may contain evidence of a crime it had nothing to do with (think of a suspect who may write a journal), cyber crime investigators are finding themselves needed in all types of investigations. To add to the monumental task of managing an increasing caseload and having a reasonable turnaround time, new techniques and technologies continue to be developed. One example of a paradigm shift in digital forensics is the collection of volatile evidence from a device, such as the contents of Random Access Memory (RAM). When I began in digital forensics, the standard protocol was to pull the power plug from the back of a running computer and transport it to the forensics lab. Doing anything other than pulling the plug was seen as destructive and against all standard practices. Now, first responders and forensic practitioners are being taught quite the opposite to save critical evidence. Forensic examiners are now taught that pulling the plug destroys evidence that may contain inculpatory or exculpatory evidence. Most law enforcement agencies don’t have the funding or time to train and equip patrol officers and detectives in the collection of volatile memory, so managers have to make a risk-‐based decision; continue pulling the plug, or provide the expertise to seize digital evidence properly.


As a digital forensic lab director and law enforcement manager, I decided to make my forensic examiners (detectives) available during seizures to perform tasks such as capturing volatile memory and assisting with the search and seizure of digital evidence. The pros of this decision was that we were getting evidence that otherwise would have been lost. We also were able to be more selective on what digital evidence was seized at scenes and could identify some digital storage devices that non-‐technical investigators didn’t realize may contain evidence. The cons of the decision included having examiners out of the lab frequently, resulting in increased backlogs and turnaround times. The legal landscape of digital forensics also began to change, mostly as the result of law enforcement mishandling digital evidence in some high profile cases. Judges began to be less tolerant of the length of time it was taking digital forensic labs to provide the results of their analysis. In some cases individuals and businesses that had data seized during an investigation were waiting months and even years without their data and criminal cases were taking forever to reach adjudication. Although it was my lab’s standard practice to explain in both state and federal affidavits that due to the highly technical process of digital forensics and the lack of trained forensic examiners there was a delay in analyzing evidence, judges began putting time limits on us. We even started to see search warrants written that required the onsite preview of digital evidence and immediate triage with instructions that only devices that had data related to the investigation could be taken offsite for additional analysis. As I began watching these changes, both technical and administrative, I realized that something had to be done to make my lab both more efficient and most importantly, more effective. My answer to this dilemma was the creation of a business plan to justify a mobile digital forensics laboratory. It was my opinion that if we had the ability to take our forensics lab to the crime scene, my lab could begin collecting evidence, imaging evidence, and even doing some forensically-‐sound analysis in the field all while within a controlled and secure environment. I hypothesized that if my lab could respond to a crime scene or warrant location with all of our tools and equipment, we would be able to provide immediate feedback to the investigators and reduce our overall operating costs. The reduction in expenses would come from faster case adjudications, less evidence supplies being consumed, and less evidence space being needed. In the end, I was able to prove all of these. In 2009 when this business plan was created, there were no other mobile digital forensic laboratories in my state. The only exposure to these vehicles I had was images on the Internet of custom-‐built vehicles. After obtaining a few quotes for these vehicles, it became quickly apparent that buying a pre-‐made mobile digital forensics lab was out of the question. My agency had no budget for this type of expense, so an alternative plan was created. In order to move forward with the project, the vehicle requirements had to be documented. For a mobile digital forensics lab to be successful, it had to:

1. Be secure 2. Have adequate room for two or three people to work 3. Be mechanically reliable


4. Have both AC and DC power available internally and externally with the capacity to power multiple high-‐end computers

5. Have climate control 6. Be able to run for long periods of time while not introducing exhaust fumes into the passenger

compartment 7. Have adequate internal and external lighting 8. Have storage space for digital evidence and equipment 9. Have the ability to network equipment inside

As these requirements were reviewed, I began considering all of the existing vehicles available that could meet the above requirements. I looked at delivery trucks (UPS, FedEx, etc.), bread trucks, and small recreational vehicles. Then, the perfect vehicle came to mind, an ambulance. I happened to know all about ambulances since I spent eight years as a firefighter and EMT, working three years on a transport ambulance before I started my law enforcement career. A local non-‐profit ambulance company was known to donate their ambulances when they reached their cycle period, so I reached out to them about my need. Within about two hours of my phone call to the ambulance company, I had an ambulance parked in my agency’s parking lot, completely free of charge. An ambulance is perfect for a mobile digital forensics lab because it meets all of the requirements, but also is already setup as an emergency vehicle. It has emergency lights, siren, radio, antennas, and is sure to be maintained in excellent condition.

Vehicle in its original condition when it was donated.

With some interior remodeling, the ambulance was quickly transformed into a working digital forensics laboratory. To help keep costs down, I contacted various companies in the area and received several


donations. A local body shop agreed to remove the decals and paint the vehicle for free, a tire shop agreed to provide all new tires, a Whelen representative provided new LED lights at cost, a graphic design company provided new custom graphics at cost, and a local graphics shop agreed to apply the graphics for free, a local cabinetry maker agreed to remodel the interior for cost, and an upholstery shop agreed to reupholster the vehicle and tint the windows for a reduced fee. To express our appreciation to these businesses, the names of these businesses were placed on the rear of the vehicle with the words “This vehicle was made possible by” above the business names. Interior Design Dimensions were taken of the interior of the vehicle and a design was created. The bench seat on the passenger’s side was removed and this is where the forensic workstations were installed. On the driver’s side, the cabinetry was perfect for storing forensic equipment such as write blockers, cables, USB devices, hard drives, keyboards, etc. This was kept as-‐is, with the exception of a void area near the rear doors that was used to hold folding stretchers. A new cabinet was specified for this area with adjustable shelving to hold additional equipment.

Original condition of interior


The cabinetry work was the first thing to be done since it was going to be the largest project and create the biggest mess inside the vehicle.

Remodeled condition of mobile forensics lab

The above image shows the completed interior remodel. The work surface on the passenger’s side gave two examiners plenty of room to work with two custom-‐built forensic workstations between them. Fasteners were placed on either end of the work surface so a bungee cord could clip to each end, pushing the chairs up against the area keeping them secure when the vehicle was in motion. The original flooring was wood with a sandpaper-‐like layer glued to the wood for traction. The top layer was manually removed and bare wood was exposed. Anti-‐static carpet tiles were selected for the new flooring. This type of flooring reduced the noise inside the vehicle, was more comfortable, and the carpet tiles are easy to pop up and remove in the event one becomes damaged or stained.


Cabinetry on driver’s side

The existing cabinets were left, just cleaned and labeled. New plastic organizing bins were purchased and labeled and equipment was stored logically in the cabinets. The new additional cabinet that was built as part of the remodel can be seen on the far left of the photograph. This setup allowed a forensic examiner to sit at their workstation and simply spin around on the office chair to access everything needed; write blockers, cables, notepads, and more were all at their fingertips.


Communications Area

The area shown above already existed in the vehicle and was repurposed for use in digital forensics. This area provided AC and DC power, so a printer and charger for portable radio batteries was installed. This area had a police radio installed so the examiners in the back could hear radio traffic and talk on the radio if needed. The control panel shown in the top of the photograph provided the ability to control the air conditioning and heating, the interior lights, and other functions. The entire vehicle was equipped with a secure, encrypted Bluetooth network. This allowed examiners to send documents to the Bluetooth printer shown above and print directly on scene. We were able to


make property receipts or print evidence found on a computer during a forensic preview and give it to the investigators conducting a suspect interview.

Forensic workstations

The above picture shows the forensic work area. The remodel included the three storage cabinets above the examiner work surface which were used to store evidence supplies, notepads, pens, and other miscellaneous items. The top of the work surface was laminate, allowing it to be scratch resistant and easy to clean after putting dirty hard drives and other equipment on it.


Top of workstation and monitors

The design included a large hole cut in the top of the work surface, directly in the center. Several cables were run through here to give an examiner immediate connectivity to the forensic workstations. The cables included USB, eSata, Firewire 800, and power cords for Tableau write blockers. 23” Acer monitors were selected for this vehicle and mounted directly on the wall. Due to the limited space, the monitors included USB ports and built-‐in speakers, eliminating the need for standalone speakers and USB hubs for the examiners. Dongles could easily be plugged in to the monitor for forensic applications, still leaving USB ports on the front of the workstations for additional connectivity. A large stainless steel power strip was installed at the base of the work surface, giving examiners plenty of outlets to plug in devices. It was not uncommon for examiners to have write blockers plugged in, cell phones charging, and laptops powered on.


The entire vehicle was networked with Cat 6 cables. In the above image Ethernet cables can be seen coming up from the two forensic workstations and plugged into outlets in the base of the newly built cabinets. In the top of the storage cabinet labeled “7” in the far left of the above photograph, a Network Attached Storage (NAS) head was installed and attached to a NAS device. An internal workgroup was created and computers could attach to the workgroup and access the NAS. The NAS was a multi-‐terabyte storage device and it was formatted as a Redundant Array of Independent Disks (RAID) in level 5. The NAS was further partitioned with the largest partition used as evidence storage and the smaller partition used to store documents. Examiners had the ability to access our forms, such as evidence receipts, search warrant templates, exigent circumstance forms, and other important documents and create and print them at the scene. Digital evidence could be forensically imaged directly to the NAS and the NAS could then be unplugged from the mobile forensics lab and transported into the forensics lab and downloaded to the in-‐house SAN.

Forensic Workstations


To continue saving costs on this project, I built the forensic workstations. These workstations had 64bit Windows 7 Ultimate operating systems, plenty of internal hard drive space, and lots of RAM. To ensure the computers did not tip while the vehicle was in motion, a closet rod was placed in front of them with padding on it.

Interior lighting

The ambulance came with standard halogen interior lights that were dimmable and could be turned on as either a bank of three, or all six lights. The halogen lights were removed and replaced with these Whelen LED interior lights. I selected three white dimmable LED lights for general purpose and then three red LED lights. The red LED lights allowed examiners to work in the back of the vehicle and not lose their night vision during nighttime operations. It also made it nearly impossible to see into the vehicle at night through the tinted windows when the red lights were on.


Faraday Box

To address the growing need of mobile device forensics, a Faraday box was installed in the mobile digital forensics vehicle. One of the existing shelves was converted to a sliding shelf so while the vehicle was in motion or the Faraday box was not being used, it could be put away. When needed, the shelf could be pulled out and the top opened, allowing an examiner to conduct an analysis of a mobile device. In the cabinet above the Faraday box was a CelleBrite unit.


Interior Cab

This vehicle sat on a Ford E-‐450 diesel chassis that was in excellent condition. As part of the remodeling process, all chairs were reupholstered. The interior cab was already equipped with a control panel, radio, and siren. The control panel allowed the passengers in the cab to control the


climate and lighting in the back as well as all of the emergency lights and scene lights on the exterior of the vehicle. Exterior Design There was not much that had to be done to the exterior except for cosmetically. All of the emergency lights were converted from strobe and halogen lights to LED as a matter of preference, however it wasn’t absolutely necessary to do in order for the vehicle to be placed in service.

Sideview of mobile digital forensics lab

The exterior of the vehicle provided a great deal of scene lighting, which was excellent during evening search warrants or crime scene investigations. This vehicle began responding to major assaults, deaths, vehicle collisions, and other incidents as part of a regional major crime team. We were able to


immediately search witness and suspect cell phones at the scene of crimes and provide real-‐time information to the investigators, or pull digital video footage of a crime to help in the investigation.

Rear of mobile digital forensics lab

The rear of the vehicle also provided additional scene lighting and emergency lighting. All windows were tinted with limousine tinting to provide the most amount of privacy and safety to those inside the vehicle. This also prevented bystanders from seeing the content being displayed on the forensic workstation monitors.


Front of mobile digital forensics lab

Equipment Housed Within the Vehicle Our lab did not have the budget to replicate all of the equipment that was in our laboratory inside of this vehicle. When the vehicle was needed at a scene, certain items from the lab had to be moved to the mobile lab. To ensure equipment was not missed, a checklist was created. The vehicle was equipped with multiple Tableau write blockers, Logitech Talon devices, all types of computer-‐related cables, wiped hard drives, empty USB thumb drives, network equipment, CDs, DVDs and thumb drives containing forensic software, notepads, pens, evidence bags, a portable heat sealing device for evidence, permanent markers, cameras, batteries, keyboards, mice, flashlights, adapters, and forcible entry tools. Anytime we responded somewhere with the vehicle we would always add at least one CelleBrite unit, some laptop computers, and additional write blockers. On a monthly basis the forensic workstations inside the mobile digital forensics lab were patched for any vulnerabilities (although they were not on the Internet) and all updates for our forensic software were applied as well.


Before and After

Effectiveness and Efficiencies Gained The success of the mobile digital forensics vehicle was better than anticipated. Responding to a crime scene with this vehicle displayed professionalism and it rapidly became a sought after resource. My lab was able to take digital evidence into this vehicle, forensically preview the evidence, and provide investigators with immediate feedback during their investigations. On numerous occasions, we were able to find evidence, print the evidence, and hand it to an investigator who was interviewing a suspect. This feedback was invaluable and occasionally led the investigation in a new direction. The vehicle was particularly helpful during incidents that involved several witnesses, such as an officer involved shooting. The mobile digital forensics lab would arrive and witnesses who captured evidence with their cell phone were able to sign a consent to search form, wait fifteen or twenty minutes for us to image their cell phone, and then get it back. It was also helpful for those individuals who normally


would not consent to being without their phone for a few days while a traditional lab examined the device, but would consent to giving it up for a few minutes. We did see a reduced amount of evidence being seized at crime scenes. Often, our examiners could quickly rule out a digital device and leave it at the scene. In the past, every item of digital evidence was seized, packaged, stored, and examined within a laboratory. Now, we could triage onsite and leave items that didn’t have evidence and take back only those items that we knew were involved in the investigation. This translated to a reduction in the use of costly anti-‐static evidence bags, evidence tape, barcode labels, and the physical storage space needed to store the evidence. In all, the vehicle and the related equipment stored within the vehicle came to just under $13,000. Without the donations received, the cost probably would have been closer to $25,000 to $30,000. There were some upgrades done to this vehicle that were not absolutely necessary, but were functional enhancements, which could be eliminated if a budget didn’t allow for them. For what we gained, $13,000 was a minimal expense and over time this vehicle not only saved us money but it served the needs of the public. I was also able to leverage this vehicle and our capabilities to successfully obtain grant funding and financial partnerships with outside law enforcement agencies, actually making us money in the end.

josh moulin: designing a mobile digital forensic lab on a budget

Technology

digital forensics laboratory

ceecs http

digital forensic lab

electronic evidence

new techniques

larger storage capacities

new operating systems

usbde network storage