What happened to my Drobo
My Drobo started resync of data for some reason (It does not tell why….).
I accessed data on it during resync operations and switched off the Drobo….
Resync times of more then 100 hrs (may be 500 hrs) where displayed in dashboard…. It never finished. Result: Drobo died. Dashboard could not find it any more after some time…
What was still possible
I could copy some of the data from the Drobo in very time consuming operations. It took several days. OK, I had to sleep during the night and to work during the day to earn money for another hard disk device…
I could not copy the data from the volume used for TimeMachine backup… So my backup history was lost.
What I did next
I did a factory reset on the Drobo to get back to a working environment. This kills all data. All disks remained unchanged in Drobo. I bought another hard disk and connect it to my Mac mini to save from my Drobo whatever is possible. Then I decided to run tests to understand better what is going on…
Drobo 2nd generation with 4 slots
– disk1: 1TB
– disk2: 1TB
– disk3: 2TB
– disk4: 2TB
Drobo Dashboard shows 3,62 TB total disk capacity.
Firmware and dashboard are latest versions.
I use a Mac mini with Mountain Lion with latest patches. Drobo is connected via Firewire 800.
Testing with Drobo
Initial Drobo Settings
Drobo is filled with four disks. 2 volumes with 2 TB each are created with Drobo dashboard. 1 volume is partitioned with OSX disk utility (2 times 1 TB). About 200 GB of data: Time Machine backup and some other stuff spread over the volumes.
All tests are executed after a factory reset (see above).
Drobo is always changed to standby and then power is cut off before changing the disks.
No data accesses on Drobo during resync operations – never again !!!.
I wonder how professional users survive this, when they depend on the Drobo data; recovery took up to 12 ???? hours with my little data…
I never switched off the computer (nor the Drobo) during the resync operations.
I executed the following tests:
Test 1: Drobo recovery after disk1 removal (bay1, top position – 1 TB)
Remove disk from bay1. Drobo dashboard shows the status: Data protection in progress (yellow/green flashing). Drobo starts to resync data. Dashboard shows about 3(?) TB total disk capacity. Dashboard indicates 4 hours duration for resync of data
Result: Drobo is back in sync after the indicated time. Data is accessible.
Test 2: Put disk1 back into the slot
Drobo status: operational (green)
Result: Dashboard shows about 3,62 TB total disk capacity
Test 3: Drobo recovery after disk removal from bay 4 (2 TB)
Preparation: Copy 760 GB of data to the volume with 2 TB capacity (takes some time….) via rsync (backuplist+).
Remove disk from bay 4. Drobo dashboard status: Data protection in progress (yellow/green flashing). Drobo starts to resync data. Dashboard shows about 1,8 TB available disk capacity for data. Additional 930 GB are reserved for extension. The dashboard shows 17 hours duration for resync of data (changes to 19 hrs after some time…)
The resync started at 02:00 in the morning (computers must work all night, while humans can take a sleep – lucky human beings:-). At 12:00 (day 2) Drobo dashboard claims another hour to resync data. Finally resync is finished at 12:30, which means that recovery took 10 hrs 30 min.
Result: Drobo is back in sync. Data is accessible. Time Machine can use it again
Test 4: Drobo starts recovery on its own
Drobo starts the recovery process arbitrarily. There were no changes made to the Drobo. I just added an additional drive to Time Machine, which resides on a Drobo volume. The respective Drobo volume was portioned, using the OSX disk utility. I forced Time Machine via the OSX task bar to execute the backup. The resync process is finished after just 1 minute. There is no visible cause for the recovery.
Result: Drobo is back in sync. Data is accessible.
Test 5: Fill Drobo with Data and re-insert disk4
I started copying 760 GB of data to the 2 TB volume Drobo2 via Forklift. It fails after about 60 GB with “error -36 ioErr IO error (bummers)”.
The Drobo2 volume is not accessible via Finder. The “wheel” in Finder just keeps turning, no files get displayed. The new folder, being copied to the volume is “greyed out” in OSX Finder: Finder information shows 53.939.979.701 Byte (53,98 GB on Volume) for 17.589 Objects. The Drobo Dashboard keeps loading, it does not find the Drobo anymore… The lights on the Drobo are “GREEN”… OSX disk utility keeps checking on the volume…; it hangs. I guess I have to power off the whole thing…
Next steps: Remove the Drobo_TM dive from the Time Machine settings. Then I killed OSX disk utility and Drobo dashboard after some time. Time Machine makes another backup to another hard disk, which is not on the Drobo. OSX disk utility check on the source drive of the initially copy action completes without error.
OSX disk utility can not unmount the Drobo volumes. OSX Finder is stalled. I got to kill it. It takes a few minutes until Finder restarts. I shutdown the Mac mini. It hangs with a grey screen. I unplug the Drobo from power and the Mac mini continues to shutdown normally.
After restart of the mini, Drobo dashboard is green. I am able to access data on the Drobo volumes (what a surprise…). The volume check in OSX disk utility shows no error.
Result: Not clear to me… What I see: Drobo is green. The new copied data is not accessible. The data, which was on the Drobo before the copy operation, is still accesible.
Remark: Drobo still works with three disks; disk4 is not yet back in the system. OSX disk utility checks all Drobo volumes OK.
Test 6: Continue copying data to Drobo
First I removed the folder, created in test 5 to start over. This time I use rsync via backuplist+ with incremental folders to fill up the Drobo_2 volume. It is about 50 GB.
The copy operation completed without any problem. Drobo dashboard shows red flashing and claims that disk2 is corrupted !!! The data on the Drobo_2 volume is still accessible via Lightroom 4 – even though it is very slow. A Time Machine backup to Drobo is very very slow: 500kB in about 5 minutes; I am going to stop it. Remark: Drobo has executed a resync during the previous copy operation. No hint why it did this. This is what is not acceptable. There is an exception, Drobo does something and does not give info on the initial cause !!!
Result: Drobo is flashing red. The data on the Drobo volumes is still accessible; it is very slow. Drobo claims the HD in bay 2 (from top) is corrupted and asks to replace it.
Test 7: Shutdown Drobo and replace Disk in Bay 2
First attempt to put Drobo ins standby failed due to running OSX disk utility: Standby of Drobo is successful after closing OSX disk utility. What I am going to do next: Move the 2 TB drive, which was previously in bay 4 to bay 2.
Drobo dashboard flashed red and claims for a moment, that I have to place another disk in bay 4. Then the status changes to flashing amber. The ongoing resync shall last 11 hours. After a few moments it goes doen to 9 hours. The Drobo capacity is now 2,71 TB.
The resync started at 21:45 in the evening (computers must work all night, while humans can take a sleep – lucky human beings:-). The resync finishes at 05:35 the other morning (day 3), which means that recovery took about 7 hours10 hrs 30 min.
Result: Drobo is back in sync. Data is accessible. Time Machine can use it again.
Test 5: Fill Drobo with Data and remove Disk from Bay 2.
Day 3 in the evening
I start to copy about 760 GB of data to the volume with 2 TB capacity (takes some time….) via rsync (backuplist+).
Come back to see the rest of the story. Copying will take some time to FW800 Drobo (about 30 GB in 30 min).