To get in front of failover problems the Sky team might encounter, we need to test the scenario where a node goes down. So, set up a multiple-pm cluster to use S3 (can be VMs on a local box, doesn't matter), use gluster as the DFS to replicate & make available all data under SM's storage dir, get things running then nuke a node and see what happens.

The Storagemanager stuff should be able to handle it, the question is mainly whether ProcMon/Mgr are going to do the right thing, and consistently. It hasn't been tested yet but we think it should work.