The escalating global demand for sustainable energy solutions has prompted an increasing focus on solar photovoltaic (PV) systems. This research paper comprehensively investigates the modelling, and techno-economic analysis of an optimal combination of sets of PV and a BIPV system based on floor areas of residential buildings. The study employs advanced simulation techniques to develop a precise mathematical and optimization model and considers the levelized cost of electricity (LCOE) during the installation of PV and BIPV systems while addressing the challenge of variations in generated energy. The research contends that a comprehensive evaluation of LCOE is instrumental in determining the economic viability of optimal combinations for different sets of PV and BIPV installations based on floor area of the building while considering sustainable approaches to fulfilling energy needs. The outcomes of this research contribute significantly to the burgeoning field of renewable energy by providing a robust framework for the assessment and optimization of a combined PV BIPV system leading to decision-making towards adoption of sustainable energy solutions.