Adaptive Network Coded Clouds High Speed Downloads and Cost-Effective Version Control
Abstract: Although cloud systems provide a reliable and flexible storage solution, the use of a single cloud service constitutes a single point of failure, which can compromise data availability, download speed, and security. To address these challenges, we advocate for the use of multiple cloud storage providers simultaneously using network coding as the key enabling technology. Our goal is to study two challenges of network coded storage systems. First, the efficient update of the number of coded fragments per cloud in a system aggregating multiple clouds in order to boost the download speed of files. We developed a novel scheme using recoding with limited packets to trade-off storage space, reliability, and data retrieval speed. Implementation and measurements with commercial cloud providers show that up to 9x less network use is needed compared to other network coding schemes, while maintaining similar download speeds and reliability. Second, the ability to update coded fragments from a linear erasure code when the original file is modified. We exploit code structure to provide efficient representations of the evolution of the file. Evaluations using file changes on
software library repositories show that a five-order of magnitude reduction in network and storage use is possible compared to state-of-the-art. Existing system: We have no knowledge of any version control systems to support erasure codes. The amount of research is limited and state-of-the-art systems follow a costly approach. For example, deals with both repair and update of data. However, it limits discussion to a particular 2 dimensional product code and focuses on sizepreserving block updates. Very recently, covered the topic from a theoretical perspective, including operations that required a modification in the size of the stored data. The latter is an unpublished work, submitted to an online archive following the initial submission of this paper. The novelty of our work consists in a set of techniques that pre-transform data to enable a cost-effective erasure coded version control system possible. Furthermore, we propose solutions for situations in which the size of the stored data changes. Our proposed system is built on top of facilities provided by existing solutions, such as Gift . We consider systems such as Git as state of the art and use it as a baseline to evaluate the overhead in storage and computation of our proposed solution. Proposed system: However, this lacks the benefits offered by erasure codes and does not allow for a seamless update of the file, without first decoding the original file to later apply the history of changes. We propose a mechanism that exploits the linear operations of erasure codes to provide a means to update coded fragments. Our solution does not send the original data and supports version control in coded storage systems while reducing network use and storage use of the overall system. The mechanism supports parts of files being added, deleted, or individual bytes being modified. This report details how these operations are carried out for supporting these three features. Our mechanism is useful in distributed storage and cloud systems using linear erasure codes and is fully compatible with random linear network codes and other network code variants. To analyze the effectiveness of the proposed technique, we used a Git repository with the source code for an active software library.
Advantages: CLOUD storage is widely adopted as it offers a cost-effective solution to storing enterprise data, with the advantages of increased reliability as well as arguably decreased technical complexity and business agility compared to on-site, personalized storage solutions. Its adoption among end users is growing as well thanks in part to the free storage offered by major IT players, e.g., Amazon, Google, Microsoft, Apple, and specialized cloud storage companies, e.g., Drop box, Box, and Sugar Sync. A limitation of these solutions is that users are typically tied to a single cloud storage provider and, thus, dependent on the provider’s offered reliability and availability of the data. The impact of an outage of a single provider can be significant. Disadvantages: The latter, close to optimal performance in the former, but also a reduced encoding and recoding effort. We define this problem formally and give it the recoding bandwidth problem name. We also devised and implemented a dense recoding mechanism and two simple replication–based approaches to use as comparison to our sparse solution. Section 4 defines the optimal distribution problem formally and gives an analytic formula to solve it. It then introduces the recoding bandwidth problem. It proposes four possible solutions to this problem and evaluates their effectiveness using several metrics. It proposes four possible solutions to this problem and evaluates their effectiveness using several metrics. Section 5 presents the problem of updating coded data and proposes a solution to it. It details three different operations to support making changes to coded files and contains an evaluation of their effectiveness. Modules: System Description: Our work focuses on distributed storage solutions using RLNC. In this paper, we present a system that employs commercially available clouds to store files reliably. Given the similar challenges, our solutions can be adapted for use with other types
of storage nodes as well. Our proposed system is comprised of a client application that uploads and downloads data to the storage nodes and handles all computations related to encoding, decoding and recoding. The storage nodes have no other functionality besides storing the data, which makes employing recoding techniques that involve encoding on the nodes impossible. This is a limitation in several commercial clouds. We have built a model without this limitation and explored other types of recoding separately to this work . Figure 1 shows an overview of the system, both the initial distribution of data as well as retrieval (top and bottom part, respectively). The original file is divided into a number of uncoded packets by the client, which are then linearly combined using random coefficients to generate a somewhat larger number of coded packets. The packets are then distributed by the client evenly to N (5 in the example) clouds. Recoding bandwidth problem: This subsection focuses on the resource allocation and adaptation process needed to optimize performance in terms of download time, while keeping the adaptation traffic low. This is in fact somewhat similar to the repair bandwidth problem, which deals with minimizing the number of packets that must be transferred to keep the reliability criterion met in case one of the storage nodes becomes unavailable. This is the reason for naming it the recoding bandwidth problem. We formulate it by introducing three metrics which should be minimized: _ B: the number of packets transferred for adaptation – adaptation cost_ X: the number of extra packets needed during data retrieval to reconstruct the original file – linear dependence in the system _ S: the time required to retrieve and reconstruct the original file in seconds – retrieval performance The first and third are rather straightforward.
Proposed solution - implementation So far we have given a short description of the problem and the proposed solution as well as an example for the operation of handling individual modifications in a file. These translate relatively well into practice with a few exceptions. This part of the paper presents the implementation details and introduces two; slightly more complex operations that are needed to handle changes in files: deleting and adding
elements. First, a general overview of the three operations is presented. This is followed by a more detailed presentation of the individual processes and algorithms that make up these operations. The first difference between the proposed mathematical model and the implementation arises from the fact that files cannot always be divided into rows of equal length to form a matrix; therefore most files will have an incomplete last row. In the mathematical model, this can easily be overcome by padding the file with 0s; however this would be wasteful in practice. Because of this, all algorithms that we present deal with non-padded files. The second difference also stems from the desire to make the operations more effective. We store and transmit the coded differences using a compressed form. To make this feasible, we transform the way data is represented. Deleting elements: Deleting is the process of removing an arbitrary number of elements from arbitrary locations in the original data. The basic idea is to signal deletions with the 0 value in M0, then performing the modification operation. However, implementation-wise there are aspects that make it differ significantly from this. A general overview is shown on Figure 10. First, the original data is remapped and transformed. Based on this and a vector of deleted indexes Del, the changed data Mt0 r is calculated. This is then used to calculate the difference. It is possible to calculate the difference without creating the deleted data by only modifying the removed elements individually in ^M T r, as all other values are 0. Encoding and applying the difference and decoding are identical to the previous operation. Following this the matrix is transformed back. A mask Del (a vector of the indices of the deleted elements) of the deleted elements is created. This is applied after unmapping to actually remove the deleted elements.