Mango ranks among the fruits that are of high economic value, and the leaf diseases like anthracnose, powdery mildew, bacterial canker, and sooty mold have great impact on the productivity. Efficient and accurate diagnosis of the disease is very essential for improving the output and sustaining a good agricultural process. Recent advancements in the field of computer vision and deep learning have depicted promising features in plant disease diagnosis. Convolutional neural networks (CNNs) and transfer learning models have shown significant improvement in the process of leaf disease recognition among fruits like mangoes. However, most of the models have depicted generic architecture that could not effectively focus on leaf texture and vein pattern features, which are very essential in disease recognition in complex backgrounds. To overcome the limitations, a new Leaf Texture and Vein pattern-aware Convolutional Neural Networks (LTVC-Net) model specifically targeting the process of mango leaf disease recognition has been proposed in this research. LTVC-Net incorporates dual-scale dilated convolution and channel attention module in its architecture in order to improve multi-scale feature learning while retaining the disease-causing vein and texture features. Experiments on the proposed model have been performed on a publicly available dataset of mango leaf disease that contains multiple disease classes and normal ones. Extensive experiments on the proposed architecture have been performed while comparing the result with five efficient deep learning models that are MobileNetV2, ResNet50, EfficientNetB0, DenseNet121, and InceptionV3 models respectively. Based on the results in terms of ‘accuracy evaluation,’ ‘loss convergence,’ and ‘ROCAUC evaluation,’ the proposed LTVC-Net architecture clearly outperformed all other models in terms of efficiency and minimizes computational cost that could be effectively suitable for real-world applications in agriculture.