Explainable AI — Opening the Black Box of Machine Learning

[Leejohnston]

Thread 7 — Explainable AI (XAI): Opening the Black Box of Machine Learning

Understanding Why AI Makes Its Decisions

As AI becomes more powerful, transparency becomes essential. 
Explainable AI (XAI) aims to reveal why models behave the way they do.

---

1. Why We Need XAI

Without explanation, AI can be:
• untrustworthy 
• biased 
• opaque 
• difficult to debug 

This is dangerous in:
• medicine 
• law 
• finance 
• scientific research 

---

2. Local vs Global Explanations

Global — how the entire model behaves. 
Local — why a specific decision was made.

Example:
Why did the model reject this loan application?

---

3. Key XAI Techniques

• SHAP values 
Shows how each feature contributed to the output.

• LIME 
Perturbs input slightly to measure influence.

• Saliency maps 
Visual highlights of what influenced an image prediction.

• Integrated gradients 
Measures contributions along the path from baseline to input.

---

4. Interpreting Neural Networks

Tools analyse:
• neuron activations 
• attention patterns 
• network internal structure 
• feature embeddings 

Helps uncover how models “think.”

---

5. Challenges in XAI

• Complex models resist simple explanations 
• Explanations can mislead 
• Interpretability is subjective 
• Some systems (like deep LLMs) are massively high-dimensional 

---

6. The Future of XAI

Research focuses on:
• mechanistic interpretability 
• transparent architectures 
• self-explaining models 
• safety-critical auditing 

---

Final Thoughts

Explainable AI bridges the gap between raw model power and human understanding. 
It ensures that AI remains safe, fair, and transparent — essential for the future of intelligent systems.