To address the issues of low efficiency and insufficient accuracy in monitoring fish populations in aquaculture, this study proposes a real-time fish monitoring method based on an omnidirectional scanning sonar and the You Only Look Once (YOLO) model using tilapia (Oreochromis sp.) as the research object. The proposed method used an omnidirectional scanning sonar to collect underwater fish shoal image data. By using the YOLOv8 algorithm combined with real-time monitoring, the proposed method achieved target recognition and analysis. Euclidean distance-based spatial analysis algorithms were used to merge and exclude anomalous data points to obtain the number and spatial distribution of fish schools. Experiments were conducted to evaluate fish schools of varying sizes (50, 100, 150, and 200 individuals) and achieved monitoring accuracies of 93.5, 94.5, 89.6, and 85.8%, respectively, with an average accuracy of 90.9%. This method substantially enhanced the real-time monitoring and accuracy of fish school population assessments. This provides an efficient solution for monitoring fish schools in aquaculture towards optimizing aquaculture management, improving production efficiency, and promoting the sustainable development of ecological aquaculture.