Explainable AI: Debugging the Black Box

This project was okay. It came directly from an IF4070 Knowledge Representation and Reasoning assignment, and there was no dramatic origin story behind our choice of sentiment analysis.

What interested me was the beauty of the techniques themselves. Machine-learning models are naturally difficult to inspect, yet LIME, SHAP, and Integrated Gradients approach that black box from completely different directions and still produce something a person can reason about.

LIME was the most intuitive method to me because it feels close to traditional machine learning. It perturbs the input, observes how the model reacts, and fits a simpler local model around that one prediction.

SHAP frames influence through feature coalitions, while Integrated Gradients follows the model internally from a baseline toward the real input. Seeing those differences taught me that there is no single universal meaning of an explanation.

Integrated Gradients was the hardest method for me. Its mathematics was less immediate than perturbing words and observing predictions, and it required us to work inside RoBERTa's embedding space.

Surprisingly, the backpropagation work from my earlier neural-network projects helped. Following influence backward through a model no longer felt entirely foreign, even if applying that idea to a transformer was still challenging.

Derwin and I worked together across the project. We implemented the required methods from scratch, then compared them with established LIME and SHAP libraries to check whether our reasoning produced sensible results.

The methods did not always agree, which was part of the lesson. For obvious sentences such as “This is bad,” however, the word “bad” consistently appeared among the strongest influences. The agreement gave us a simple sanity check without pretending every method should explain the model identically.

The most useful debugging moment happened when the model misclassified a sentence and assigned unusually strong negative influence to a word we understood as positive. Instead of treating the prediction as another model failure, we followed the explanation back to the training data.

The example had been mislabeled. Explainability did not merely tell us what the model noticed. It helped reveal that the model had learned from an incorrect answer we had given it.

We explored BERT, RoBERTa, and XLM-RoBERTa experimentally before choosing the model that performed best for our setting. We also had to rebalance the sentiment data so its label distribution would not dominate what the classifier learned.

Performance became another tradeoff. SHAP was conceptually elegant, but its repeated model evaluations made it noticeably slow. An explanation can be valuable and still be too expensive for every interaction.

This was not the most dramatic project I built, but it changed how I think about the word “explanation.” LIME can describe behavior around one example without opening the model, while Integrated Gradients uses the model's own internal path. Both can be useful without answering exactly the same question.

The lasting lesson came from the mislabeled example: explainability is not only for making an AI appear more understandable. It can be a practical debugging tool for the model, the prediction, and even the data used to train it.