Transfer learning has emerged as a pivotal technique for biomedical image classification, where labeled data is limited and computation cost of model training is very high. By leveraging pre-trained neural networks, transfer learning accelerates convergence and enhances model precision on domain-specific tasks. The efficiency and performance of transfer learning is governed by various factors that need to be fine-tuned. Despite its wide success, the mathematical understanding of transfer learning behavior remains limited. In this paper, we systematically studied the mathematical aspect of transfer learning as a regularized optimization problem, where adaptation to a target domain is balanced against deviation from pre-trained parameters, which plays an important role in classification of datasets including small and imbalanced datasets invariably found in medical diagnosis. We validated this model using controlled regularization across a simulated neural network and three models of deep convolutional networks pre-trained on two different domains for classification of ultrasound scans used in diagnosis of polycystic ovary syndrome. Our experimental results suggest that regularization improves generalization and preserves inductive bias, due to a principled adaptation-retention tradeoff, thereby bridging mathematical modeling and practical implementation of transfer learning in domain-adaptive medical imaging applications.