Mathematical word problem (MWP) solving is a complex domain that is characterized by a wide range of complex problems, ranging from rule-based systems to pre-trained language models. In this context, we introduce the External Equation Generator (ExEqGen), a novel model that combines SymPY, a sophisticated Python equation generator, with the state-of-the-art RoBERTa language model. Our approach leverages external symbolic solvers for expression generation and mathematical computations, demonstrating enhanced accuracy in solving math word problems. The study also demonstrates the positive impact of external symbols on improving numerical reasoning performance, particularly in mitigating false positives associated with longer reasoning programs. The findings provide valuable insights into enhancing the performance of Natural Language Understanding Models in complex tasks, contributing to the ongoing quest for more accurate, interpretable, and applicable natural language understanding models.