Image encryption algorithms based on permutation-based algorithms are being common since they are simple and fast to compute but their security is a major issue when subjected to a high level of cryptanalysis. This paper provides a detailed discussion of the permutation-only schemes, in terms of their mathematical representation, and structural weaknesses. The systematized research questions are the patterns of vulnerabilities caused by deterministic pixel rearrangements, which do not possess diffusion qualities. The cryptanalytic methods such as known-plaintext attacks (KPA) and chosen-plaintext attacks (CPA) are discussed in order to illustrate the effectiveness of retrieval of permutation keys using little computing power. In addition, optimization techniques are suggested to expedite key reconstruction with heuristic and reduction of search space techniques. Through experimental analysis, it is demonstrated that permutation only schemes are highly vulnerable, recovering key with more than 90% accuracy and with a lesser level of computation. The results indicate that to achieve robustness, it is important to incorporate diffusion processes, which can be helpful in creating secure image encryption systems.