Background: Predatory journals are fraudulent publications that feature unethical and inaccurate research works. Objective: This study aimed to check the ability of a machine learning tool to identify predatory journals when compared to manual methods. Methods and Material: A cross-sectional observational study design was opted for. After data collection and screening of 23 journals, 19 journals were chosen for evaluation. These journals featured research in pediatric dentistry between 2016 and 2023 from three western states of India. Each journal was analyzed through two manual methods [Predatory Rate (PR) and Patwardhan Protocol (PP)] and by a machine learning tool [(Academic Journal Predatory Checking system (AJPC)]. Statistical analysis was carried out using Orange Data Mining Software (3.36.0). A confusion matrix was drawn to calculate accuracy, precision, recall, and F1 score. A Cohen’s Kappa was calculated to evaluate the agreement of results beyond chance, followed by Mcnemar’s test of significance. Results: An F1 value of 0.96 suggests that AJPC nearly replicates the results of PP. However, this cannot be said when compared with PR, where an accuracy of only 68.42% is statistically significant (p