RAID Rebuilding S/A Daniel Dickerman Technical Advisor to the Director, Electronic Crimes Program IRS - Criminal Investigation daniel.dickerman@ci.irs.gov
Objectives Brief introduction to RAID technology and the issues you need to be aware of to properly perform the acquisition and rebuilding of data stored on a RAID array, for subsequent analysis. •What is a RAID? •Hardware vs. Software RAID •RAID Attributes •RAID Levels
Objectives (cont.) •RAID rebuilding 101 •Rebuilding Tools •RAID Reconstructor •X-Ways Forensics/WinHex (Specialist or Forensic license) •Encase •SMART
What is RAID? • Redundant Array of Inexpensive/Independent Disks • Multiple disks functioning as one for: – Fault Tolerance (Data Protection) – Increased Performance – Increased Capacity
Hardware RAID • Hardware RAID is controlled by a RAID controller. • The OS is typically unaware that it is writing/reading to/from multiple disks.
Hardware RAID • What the forensic examiner sees (physically)….
Hardware RAID • What the OS “sees”…a 273GB primary disk and two 2,235 GB Disks…
Hardware RAID • The physical drives that are actually present…3-136GB array disks and 1-136Gb hot spare, plus 14 – 400GB IDE disks in an Apple X-Serve RAID (not shown in screenshot).
Hardware RAID • What your imaging tool might see‌
* The above screenshot is for the sole purpose of demonstrating examples of RAID volume detection and does not necessarily depict the RAID volume detection capabilities of all versions of the above shown tool. The disks and volumes detected will vary depending on the version of your imaging tool and the controller drivers incorporated into your bootable disk.
Hardware RAID • What your imaging tool might see‌
* The above screenshot is for the sole purpose of demonstrating examples of RAID volume detection and does not necessarily depict the RAID volume detection capabilities of all versions of the above shown tool. The disks and volumes detected will vary depending on the version of your imaging tool and the controller drivers incorporated into your bootable disk.
Hardware RAID • What your imaging tool might see‌
* The above screenshot is for the sole purpose of demonstrating examples of RAID volume detection and does not necessarily depict the RAID volume detection capabilities of all versions of the above shown tool. The disks and volumes detected will vary depending on the version of your imaging tool and the controller drivers incorporated into your bootable disk.
Hardware RAID • What your imaging tool might see‌
* The above screenshot is for the sole purpose of demonstrating examples of RAID volume detection and does not necessarily depict the RAID volume detection capabilities of all versions of the above shown tool. The disks and volumes detected will vary depending on the version of your imaging tool and the controller drivers incorporated into your bootable disk.
Software RAID • Software RAID is controlled by the OS or software running in the OS. – On a PC, the bootable system drive is not part of the Software RAID, but usually contains the information required to load/access the software RAID. – Many multi-drive external storage devices are actually Linux software RAIDs “behind the scenes”, where the device has a Linux OS on it’s firmware that controls disk read/write operations to the multiple disks.
Software RAID • Notice the “X: drive” is a 4471 GB Windows Server 2003 striped volume made up of two 2235 GB physical disks…which are actually each made up of 7 – 400GB IDE disks set up as RAID 5 hardware RAID volumes. (a software RAID 0 striped across two hardware RAID 5 volumes = RAID 50.)
RAID Attributes • • • •
Disk Order Stripe Size RAID Header Parity – Dedicated vs. Distributed – Parity Type/Rotation – Parity Delay
RAID Attributes • Disk Order – The order of the disks that make up the array – This may seem like a very simple one, but when pulling individual drives from a RAID, it is easy to get them out of order or mislabel the image names for each disk image. – Always double check yourself, especially when putting the disks back into the server to ensure they are in the correct order.
RAID Attributes • Stripe Size • How much data is written to each disk before moving to the next disk to write the next block of data. • Typical stripe sizes: – 8,16, 32, 64, and 128 kilobytes per stripe – you may occasionally see other sizes
RAID Attributes • RAID Header – Static block of data at the beginning of each array disk. – May be identical (or nearly identical), making you initially think it’s a “mirror” – Usually has a byte that identifies the disk # for the array, which gives you your “Disk Order” – Header size and disk # usually found by performing a comparison of the disks. – Compaq/HP servers usually = 1088 sector header size
RAID Attributes • Parity – Rebuilding information created by XOR’ing together bytes from each disk containing RAID data, the result of which gets stored as a parity value on the “parity disk”. – The drive on which this calculated parity data is stored will depend on the type of “Parity Rotation” used. • Parity Rotation described in more detail later in presentation
– RAID4 = Dedicated parity disk – RAID5 = Distributed parity disk
RAID Levels • • • •
RAID 0 (Striping) RAID 1 (Mirroring/Duplexing) RAID 5 (Striping w/ Distributed Parity) Multi-RAID levels – RAID 1+0 (a stripe of mirrors) – RAID 0+1 (a mirror or stripes) – RAID 1+5, 5+1, 0+5, 5+0, etc. • Other non-RAID multi-disk setups: – Disk Spanning – JBOD (Just a Bunch Of Disks)
RAID 0 • No fault tolerance – Single disk failure = array failure
• Fastest performance • Capacity of array = total capacity of individual disks combined • Items needed for rebuilding: – Disk Order – Stripe Size – RAID header size* * Not all RAIDs have a RAID header
RAID 1 • Fault tolerance (via data replication) • Increased read performance, same write performance as writing to single disk • 50% of disk capacity used for data redundancy • Items needed for rebuilding: – Typically no rebuilding necessary – …unless RAID header exists*
* Not all RAIDs have a RAID header
RAID 5 • Fault tolerance (via parity data) • Increased read and write performance • 1/Nth reduction in disk capacity, used for parity, where N = # of array disks. – Minimum of 3 array disks needed for any RAID level with parity
RAID 5 • Rebuilding components: – Disk order – Stripe size – RAID header size* – Parity rotation – Parity delay**
* Not all RAIDs have a RAID header ** Only used in Backward Delayed Parity
RAID 5 • Parity Rotation – Backward Delayed Parity (Compaq/HP)*
* Example shown using a parity rotation delay of 4, meaning parity stays on it’s current disk for 4 stripes, then moves for the next 4 stripes and so on.
RAID 5 • Parity Rotation – Backward Dynamic Parity (AMI) • Probably the most common type
RAID 5 • Other Parity Rotations – Backward Parity • (Adaptec)
– Forward Parity
RAID Rebuilding 101 • The “goal” in RAID rebuilding it to put back together the data that has been spread out across multiple disks and may include parity information, depending on the RAID level. • This is done by re-pasting the striped data back together into one disk/image and removing the parity as you go.
Individual RAID 5 disks/images Disk 0
Disk 1
Disk 2
Disk 3
Disk 4
Stripe1 T
H
I
S
Parity
Stripe2 A
S
Parity
W
Stripe3 R
A
Parity
A
Stripe4 !
Parity
I
D
!
RAID 5 rebuilt into single disk Disk 0 THIS WAS A RAID!!
RAID Rebuilding 101 • The more you document about the RAID onsite, the less you have to manually try to figure out later! – Boot RAID server into RAID Controller BIOS configuration utility during Power On Self Test (POST)
– View array configuration and write down the RAID level, disk order, stripe size, disk & array configuration, controller type, etc!!!
RAID Rebuilding 101
RAID Rebuilding 101
RAID Rebuilding 101 • Any of the information you are unable to determine onsite during the imaging of the RAID disks will have to be either manually determined or possibly via some guesswork. • Manual interpretation of the striped data on RAID disks is not difficult if you have an in-depth understanding of how data structures are laid out on a non-RAID disk, including: • MBR and Partition Table • Boot Sectors/Records • FAT tables, Root Dirs, etc. • MFT records, INDX entries, etc. Unfortunately, it is not possible to cover manual data interpretation in this one hour presentation.
RAID Rebuilding Tools • RAID Reconstructor (Runtime Software) http://www.runtime.org/raid.htm
• X-Ways Forensics/WinHex (X-Ways Software Technology AG) http://www.x-ways.net/forensics/index-m.html
• Encase (Guidance Software) http://www.guidancesoftware.com/products/ef_index.aspx
• SMART (ASRData) http://www.asrdata2.com/ ***There are a few other RAID rebuilding tools out there but as of the writing of this presentation, the above tools were the only ones I had available to include.
RAID Reconstructor • Step #1 – chose RAID type, number of drives, add drives images (in correct order), select block size and parity rotation.
RAID Reconstructor • Step #2 – analyze data to attempt to determine correct RAID parameters.
RAID Reconstructor • Step #3 - write out a new “rebuilt” single image from the multiple images.
RAID Reconstructor • Pros – Tests numerous combinations of RAID parameters to try and “Guess” settings using entropy testing. Useful when you don’t know the parameters. – Works with up to 14 RAID disks for RAID 5. – Will rebuild RAID 5, from parity, with one missing disk/image. • Cons – Can only do a 2-disk RAID 0 – Doesn’t do Backward Delayed Parity RAIDs – Requires you to actually “rebuild” a new image before you can check to see if you actually have the correct settings. Only after the rebuild can you open the new image in your forensic tools. – Does not recognize .e01 or other image formats, must convert images to raw bit.
X-Ways Forensics/WinHex • Step #1 – Open each individual disk image and “Interpret Image File as Disk” from the Specialist menu.
X-Ways Forensics/WinHex • Step #2 – Select “Assemble RAID system” from the Specialist menu. Open each disk component in the correct order, enter the header size, select the parity rotation type and stripe size and click OK.
X-Ways Forensics/WinHex • If you entered the correct RAID parameters, the RAID volume is “virtually” reconstructed, allowing you to map out the file system.
X-Ways Forensics/WinHex • Pros – Performs a “virtual” rebuild in RAM to allow you to see the results right away. File system mapping errors indicate if you have the wrong parameters. – Works with up to 10 RAID disks for RAID 5 or RAID 0. – Will rebuild RAID 5, from parity, with one missing disk/image. – The only tool that does Backward Delayed Parity (Compaq/HP). – Reads .e01 or raw bit images. • Cons – Does not use entropy or do any “guesswork” for you.
EnCase (Software RAID)
EnCase (Software RAID)
EnCase (Hardware RAID)
EnCase • Pros – Can be used to “virtually” reconstruct Windows Software RAIDs and some hardware RAIDs. – Reads .e01 and raw bit images. – Can rebuild RAID 5, from parity, with a missing image. • Cons – Only rebuilds “Right or Left handed stripe” RAIDS. (Not sure what Parity rotation types these refer to, but they are not in line with the correct industry terminology used by other vendors.) – Lacks features for RAID headers and Delayed Parity.
SMART
1 3
4
2
SMART
4
1 2 3
3
1 2
SMART • Pros – Can be used to “virtually” reconstruct RAIDs. – The only tool that does RAID4. – Allows removal of RAID header when importing images (prior to RAID rebuilding steps). – Reads .e01 and raw bit images. – “Guesses” using entropy to try to determine settings for you. • Cons – Only rebuilds Right Symmetric or Left Symmetric parity RAID5 (no Backward Dynamic or Backward Delayed). – Relies on Linux OS it is running on for driver support (i.e. MD raid driver). Device detection may be more complex and require more user interaction or configuration. Linux drivers are not available for all controller cards. – Requires Linux knowledge/familiarity.
The End
Questions??? Concerns??? Confusion???