Failure Diagnosis for Distributed Systems Using Targeted Fault Injection
Abstract: This paper introduces a novel approach to automating failure diagnostics in distributed systems by combining fault injection and data analytics. We use fault injection to populate the database of failures for a target distributed system. When a failure is reported from production environment, the database is queried to find “matched� failures generated by fault injections. Relying on the assumption that similar faults gene generate rate similar failures, we use information from the matched failures as hints to locate the actual root cause of the reported failures. In order to implement this approach, we introduce techniques for (i) reconstructing end-to-end end execution flows of distrib distributed uted software components, (ii) computing the similarity of the reconstructed flows, and (iii) performing precise fault injection at pre pre-specified specified executing points in distributed systems. We have evaluated our approach using an OpenStack cloud platform, a popular p cloud infrastructure management system. Our experimental results showed that this approach is effective in determining the root causes, e.g., fault types and affected components, for 71-100 100 percent of tested failures. Furthermore, it can provide fault lt locations close to actual ones and can easily be used to find and fix actual root causes. We have also validated this technique by localizing real bugs that occurred in OpenStack.