Improving software maintenance ticket resolution using process mining

Abstract

Software maintenance refers to the modification of software product after delivery and is required to correct faults, to improve the performance or other attributes, or to adapt the product to a modified environment. It is a crucial activity in the software industry and consumes a major portion of the expenditure on software. It is known that the performance of an organization can be improved by improving the process. Therefore, given the importance and cost involved, there is need to continuously improve the software maintenance process. This thesis focuses on analyzing and improving the software maintenance ticket resolution process by exploring novel applications of process mining. We decided to study the ticket resolution process as it is an important part of a software maintenance process. Process mining consists of mining event logs generated from business process execution supported by information systems. To identify the potential opportunities for improvement in software process management by mining data repositories, we first conducted qualitative interviews and surveys of more than 40 managers in a large global IT company. The survey provided us with a list of more than 10 maintenance process challenges encountered by practitioners, and benefits that might accrue by addressing them. This thesis addressed a few of the identified challenges pertaining to the software maintenance ticket resolution process. We studied different types of software maintenance tickets, that is, software bug tickets and IT support tickets. As identified from the survey, there is a need to analyze the data generated during the ticket resolution process to capture process reality and identify the process inefficiencies. Hence, we proposed a framework to analyze the data for ticket resolution process from diverse perspectives, by applying process mining techniques. Using the proposed framework, we discovered the process model that captured the control flow, timing and frequency information about events. We then studied inefficiencies such as self-loops, back-forth, ticket reopen, timing issues, delay due to user input requests, and effort consumption. We also analyzed the degree of conformance between the designed and the runtime (discovered) process model. The data-driven insights helped to make process improvement decisions. For example, using the proposed framework on IT support ticket data for a large global IT company we found that around 57% of the tickets had user input requests in the life cycle, causing significant delays in user-experienced resolution time. Therefore, we proposed a machine learning based-system that preempts a user at the time of ticket submission with an average accuracy of around 94% to provide additional information that the analyst is likely to ask, thus mitigating delays due to later user input requests. XIII Also, we explored unstructured data generated during process execution to derive insights that could not be obtained solely from structured data (event logs). To achieve this, we extracted topical phrases (keyphrases) from the unstructured data using an unsupervised graph-based approach. The keyphrases were then integrated into the event log, which got reflected in the discovered process model. To resolve a ticket, some code changes were made, which led to anomalies such as regression bugs. We aimed to detect whether ticket resolution caused some anomalous behavior so as to reduce the post-release bugs, one of the important challenges identified from the survey. To achieve this, we proposed an approach to discover an execution behavior model for the deployed and the new version using the execution logs that is, runtime print statements. Differences between the two models were then identified, which allowed programmers to detect anomalous behavior changes, that is, not consistent with code changes thereby identifying potential bugs that might have been introduced during code change. We applied the proposed framework and solution approaches on a series of case studies on data sets of commercial and open source projects. Through the aforementioned contributions, we explored the potential of applying process mining using various data sources to improve various aspects of the software maintenance ticket resolution process. Such analysis usually focuses on identifying the inefficiencies, but as we observed in the thesis, it can also lead to automation opportunities to make the ticket resolution process more efficient.