Neural network optimisation is a major problem in deep learning. It needs algorithms that work well with millions of parameters and keep convergence stable and generalisation. Stochastic gradient descent (SGD) and other traditional first-order methods are fast to compute, but they take a long time to converge and are sensitive to learning rate schedules. Second-order methods use curvature information to speed up optimisation, but they are too expensive for large networks. Adaptive methods like Adam and RMSProp are more stable and converge faster, but they don’t always work as well as SGD. In this paper presents a thorough mathematical examination of weight optimisation in neural networks, encompassing gradient-based methods, regularisation techniques, theoretical perspectives on loss landscapes, and comparative performance evaluations.