Total quality management and integration of heterogeneous sewer inspection databases for asset management

Abstract

Understanding the current condition of sewer pipe networks is a critical step in improving national wastewater systems. Several studies have attempted to develop deterioration models for sewer pipes, and a common concern raised by those studies was data availability and reliability. Due to quality issues in current data collection practices, approximately one-third of the data is not usable. These data records are often associated with severe defects in the pipes that cause the interruption of the inspection process. The most common data cleaning process was to eliminate missing and duplicate data which helps to avoid misinterpretation of the data. To address the reliability problems, data quality evaluation tools were developed. Data quality evaluation is a multi-dimensional concept that includes both subjective perceptions and objective measurements to be evaluated. Five data quality metrics were defined to assess different quality dimensions of the sewer inspection data including Accuracy, Consistency, Completeness, Uniqueness, and Validity. Moreover, sewer pipes condition assessment databases with more than 90,000 inspections provided by different municipalities across the nation were examined to develop a data quality assurance tool. The quality assurance process consists of three steps: 1) Formulating a quality assurance framework, 2) Detecting problematic data, and 3) Resolving problematic data. The results show that, by applying the developed quality assurance tool, the percentage of good quality inspection data increased from 50%-75% (pre-process) to 95% (post-process). Also, it has been noticed that the data has been collected in so many different formats. As a result, a data mapping tool was developed to address this problem by transforming data into the PACP data structure while keeping the integrity of the database. By implementing this tool one of the major issues in the industry will be addressed and the data can be viewed, modified, and analyzed independently from the generating software. Also, a normalized dimensional Sewer Inventory Schema (SIS) was developed for integrated national sewer inventory. This research contributes to the overall body of knowledge by providing a robust data quality evaluation and integration process for sewer pipes inspection data, which will result in quality data for sewer asset management endeavors.