Framework to fi nd hairball structure in enterprise data integration repositories

Abstract

In the current changing trends, collaboration between di erent organizations or consolidation between applications of the same organization has become a common phenomenon. In order to achieve strategic business objectives it is necessary to have a uni ed view of data, which is given by Enterprise Data integration (DI). Based on the requirements of di erent organizations, large number of tools and technologies are available in the market. Some organizations are already using di erent integration techniques from past many years. However due to the emerging era of big data and cloud enterprises want to shift from their old integration methods to the new and advance techniques. In this dissertation, a framework has been developed to retrieve the structure of the enterprises integration repositories and present them visually, so that the enterprises can take completely informed decisions, as they cannot change what they do not understand. This framework can be used to nd the connection link information, connection location and the frequency of the repetition of the same sources and links. This tool can also be used to nd the amount of data transferred from one geographical area to the other, which will help the organizations in measuring the bandwidth requirements across networks. We conducted extensive experimental study on the available datasets of di erent organisations and found that approximately 90% of sources and 80% of connections are repeated in an integration environment. We discovered the main reason behind this repetition is the end-to-end connectivity between the creators to the consumers. To the best of our knowledge, our proposed framework is a unique tool of this type. We have also implemented a Log Stitching utility, which can stitch the logs of a speci c duration from various remote and local locations into one le that will help the existing integration tools in faster debugging of their errors which will indirectly reduce the down time for the applications. To achieve this, we have designed a unique method of k-way merges by using Java NIOs, and priority queue.