AN EXPLORATION OF TRANSFORMER ARCHITECTURE FOR SHALE SEM IMAGES INTERPRETATION AND FOR TIME-SERIES ANOMALY DETECTION

Abstract

This dissertation explores the application of Transformer-based architectures for two criticaltasks in the oilfield domain: Scanning Electron Microscopy (SEM) image analysis and anomaly detection in multivariate time-series production data. The study addresses key challenges in both domains, including the scarcity of labeled datasets, the complexity of microstructural image features, and the irregular nature of production data, which often limits the effectiveness of conventional analytical methods. For SEM image analysis, this work is organized into two complementary but distinct projects. The first project focuses on shale SEM image segmentation, where numerical representations are learned through a self-supervised framework that integrates Vision Transformers (ViT), trained using the Self-Distillation with No Labels (DINO) approach, into the Self-supervised Transformer with Energy-based Graph Optimization (STEGO) segmentation pipeline. To address limitations in existing implementations, a Hierarchical- STEGO model is proposed, replacing the original k-means clustering component with hierarchical agglomerative clustering. This modification improves feature representation and enhances segmentation accuracy for complex shale microstructures. The model is evaluated on both synthetic and real SEM datasets, demonstrating improved performance in separating organic matter, inorganic matter, and pore structures compared to conventional backbone approaches. The second project focuses on classifying SEM images according to their geological sources. Rather than relying on fully supervised training, this study leverages the representations learned from the self-supervised framework and performs classification using a k-Nearest Neighbors (k-NN) classifier. The results demonstrate high classification accuracy and strong clustering structure in the learned feature space across different geological formations, despite the absence of labeled training data. For time-series anomaly detection, this study compares a Transformer-based machine learning model (AnomalyBERT) with two deterministic approaches grounded in the Matrix Profile framework: the Standard Matrix Profile and a modified Baseline Matrix Profile. The Standard Matrix Profile follows the original formulation, while the Baseline Matrix Profile is adapted to handle imbalanced datasets through a normal reference-based detection strategy. Experiments conducted on the imbalanced 3W well production dataset show that model performance varies with anomaly characteristics. AnomalyBERT demonstrates more consistent performance in detecting short-window anomalies, whereas Baseline Matrix Profile approaches are more effective for long-window anomalies. In contrast, on the balanced electrocardiogram (ECG) dataset, which is dominated by short-window anomalies, the Standard Matrix Profile achieves the strongest overall performance. Beyond detection accuracy, the study provides a comparative analysis of computational cost, interpretability, data requirements, and deployment considerations. The results indicate that while Transformer-based models offer strong adaptability and automated feature learning, deterministic methods remain highly competitive due to their simplicity, transparency, and lower computational overhead. Overall, this research demonstrates that integrating Transformer architectures with self-supervised learning provides a scalable and effective approach for both image-based and time-series analysis in specialized domains.