The recent advancements in Artificial Intelligence (AI) models, particularly in finance, transportation, and autonomous systems, have resulted in a growing demand for the development of models that are both accurate and interpretable. Numerous contemporary state-of-the-art XAI methodologies, like LIME, SHAP, and gradient-based visualisations, offer intuitive elucidations of model behaviour; nonetheless, they are predominantly heuristic and lack robust mathematical foundations or possess tenuous underpinnings. This limitation raises questions about the interpretability and generalisability of explanations derived from deep learning models. To overcome these restrictions, we present a systematic theory of interpretability and explainability, incorporating axioms and computable properties that precisely delineate the optimal framework for transparent models. It regards interpretability as a structural characteristic of the functional mappings and information flow inside the model, examined from information-theoretic, topological, and optimisation viewpoints. It delineates four essential criteria: fidelity, stability, consistency, and completeness. Such assumptions ensure that an explanation accurately characterises the behaviour of a model within the permissible errors dictated by Lipschitz continuity. We present a Soft Unified Interpretability Score that integrates mutual information relevance, stability against input perturbations, and distortion criteria to assess explanation quality. Analytical parameterisation utilising linear models, decision trees, and a neural network indicates that interpretability diminishes as model complexity increases, although this trade-off may be quantified by the suggested approach. This initiative establishes a foundation for verifiably reliable AI by anchoring the explanation in mathematical principles, while also presenting a compelling open question regarding future standardised measurements of explainability.