Detection and Classification of Abnormal Passenger Behaviour in Self Driving Buses Using In-Vehicle Camera Systems

Shaik Sidra Mehrin; Shaik Rubeena; Shaik Azmathulla; Shaik Sameena; M Jyoshna

doi:10.5281/zenodo.15209623

Authors

Shaik Sidra Mehrin Department of Computer Science and Engineering, Annamacharya Institute of Technology and Sciences, Kadapa, Andhra Pradesh, India
Shaik Rubeena Department of Computer Science and Engineering, Annamacharya Institute of Technology and Sciences, Kadapa, Andhra Pradesh, India
Shaik Azmathulla Department of Computer Science and Engineering, Annamacharya Institute of Technology and Sciences, Kadapa, Andhra Pradesh, India
Shaik Sameena Department of Computer Science and Engineering, Annamacharya Institute of Technology and Sciences, Kadapa, Andhra Pradesh, India
M Jyoshna Department of Computer Science and Engineering, Annamacharya Institute of Technology and Sciences, Kadapa, Andhra Pradesh, India

DOI:

https://doi.org/10.5281/zenodo.15209623

Keywords:

Autonomous Vehicle, Activity Recognition, Unusual Behavior Detection, Deep Learning Model, Machine Vision System

Abstract

With the rapid advancement of self-driving technology in public transportation, ensuring passenger safety remains a critical challenge. Identifying and responding to unsafe or inappropriate passenger behaviors is essential, particularly in autonomous vehicles operating without direct human oversight. This study presents a novel approach for detecting and classifying abnormal passenger activities within a bus environment. Unlike conventional human activity recognition methods, the proposed system utilizes an overhead vision-based framework to reduce occlusion and improve detection accuracy. A specialized action recognition network is designed to process top-view images, effectively capturing both spatial and temporal dynamics for enhanced classification. To facilitate real-world implementation, a dedicated dataset, BUS-HAR, has been developed, containing diverse activity samples for robust model training. Experimental evaluations on real-world data demonstrate the superior performance of the proposed method over existing techniques, highlighting its potential for improving safety in autonomous public transport

References

Buehler, R. (2018). Can public transportation compete with automated and connected cars? Journal of Public Transportation, 21(1), 7–18.

Bridgelall, R. (2022). Using artificial intelligence to derive a public transit risk index. Journal of Public Transportation, 24, 100009. https://doi.org/10.5038/2375-0901.24.1.1

He, J., Wu, X., Cheng, Z., Yuan, Z., & Jiang, Y.-G. (2021). DB-LSTM: Densely-connected bi-directional LSTM for human action recognition. Neurocomputing, 444, 319–331. https://doi.org/10.1016/j.neucom.2021.03.024

Lee, H., Kim, Y.-S., Kim, M., & Lee, Y. (2021). Low-cost network scheduling of 3D-CNN processing for embedded action recognition. IEEE Access, 9, 83901–83912. https://doi.org/10.1109/ACCESS.2021.3087776

Carreira, J., & Zisserman, A. (2017). Quo vadis, action recognition? A new model and the Kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6299–6308). https://doi.org/10.1109/CVPR.2017.667

Tseng, C.-H., & Lin, H.-Y. (2022). A vision-based system for abnormal behavior detection and recognition of bus passengers. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC) (pp. 2134–2139). IEEE. https://doi.org/10.1109/ITSC55140.2022.9922289

Ramanujam, E., Perumal, T., & Padmavathi, S. (2021). Human activity recognition with smartphone and wearable sensors using deep learning techniques: A review. IEEE Sensors Journal, 21(12), 13029–13040. https://doi.org/10.1109/JSEN.2020.3045991

Ehatisham-Ul-Haq, M., et al. (2019). Robust human activity recognition using multimodal feature-level fusion. IEEE Access, 7, 60736–60751. https://doi.org/10.1109/ACCESS.2019.2915552

Popoola, O. P., & Wang, K. (2012). Video-based abnormal human behavior recognition—a review. IEEE Transactions on Systems, Man, and Cybernetics, Part C, 42(6), 865–878. https://doi.org/10.1109/TSMCC.2011.2178594

Helbing, D., & Molnar, P. (1995). Social force model for pedestrian dynamics. Physical Review E, 51(5), 4282–4286. https://doi.org/10.1103/PhysRevE.51.4282

Mehran, R., Oyama, A., & Shah, M. (2009). Abnormal crowd behavior detection using social force model. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (pp. 935–942). IEEE. https://doi.org/10.1109/CVPR.2009.5206737

Basharat, A., Gritai, A., & Shah, M. (2008). Learning object motion patterns for anomaly detection and improved object detection. In 2008 IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8). IEEE. https://doi.org/10.1109/CVPR.2008.4587478

Delgado, B., Tahboub, K., & Delp, E. J. (2014). Automatic detection of abnormal human events on train platforms. In NAECON 2014 - IEEE National Aerospace and Electronics Conference (pp. 169–173). IEEE. https://doi.org/10.1109/NAECON.2014.7000085

Sun, X., Zhu, S., Wu, S., & Jing, X. (2018). Weak supervised learning-based abnormal behavior detection. In 2018 24th International Conference on Pattern Recognition (ICPR) (pp. 1580–1585). IEEE. https://doi.org/10.1109/ICPR.2018.8546099

Nayak, R., Pati, U. C., & Das, S. K. (2021). A comprehensive review on deep learning-based methods for video anomaly detection. Image and Vision Computing, 106, 104078. https://doi.org/10.1016/j.imavis.2020.104078

Zhou, S., et al. (2016). Spatial–temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Processing: Image Communication, 47, 358–368. https://doi.org/10.1016/j.image.2016.06.007

Yan, C., et al. (2015). Recognizing driver inattention by convolutional neural networks. In 2015 8th International Congress on Image and Signal Processing (CISP) (pp. 680–685). IEEE. https://doi.org/10.1109/CISP.2015.7408012

Tu, I., et al. (2017). Deep passenger state monitoring using viewpoint warping. In Image Analysis and Processing – ICIAP 2017 (pp. 137–148). Springer. https://doi.org/10.1007/978-3-319-68560-1_13

Tu, I., et al. (2018). Dual viewpoint passenger state classification using 3D CNNs. In 2018 IEEE Intelligent Vehicles Symposium (IV) (pp. 2163–2169). IEEE. https://doi.org/10.1109/IVS.2018.8500517

Velastin, S. A., & Gómez-Lira, D. A. (2017). People detection and pose classification inside a moving train using computer vision. In Advances in Visual Informatics (pp. 319–330). Springer. https://doi.org/10.1007/978-3-319-70010-6_29

Kao, S.-F., & Lin, H.-Y. (2021). Passenger detection, counting, and action recognition for self-driving public transport vehicles. In 2021 IEEE Intelligent Vehicles Symposium (IV) (pp. 572–577). IEEE. https://doi.org/10.1109/IV48863.2021.9575887

Madapuri, R. K., & Senthil Mahesh, P. C. (2017). HBS-CRA: Scaling impact of change request towards fault proneness: Defining a heuristic and biases scale (HBS) of change request artifacts (CRA). Cluster Computing, 22(S5), 11591–11599. https://doi.org/10.1007/s10586-017-1424-0

Dwaram, J. R., & Madapuri, R. K. (2022). Crop yield forecasting by long short‐term memory network with Adam optimizer and Huber loss function in Andhra Pradesh, India. Concurrency and Computation: Practice and Experience, 34(27), e7310. https://doi.org/10.1002/cpe.7310

Ahmed, S. T., Sivakami, R., Banik, D., Khan, S. B., Dhanaraj, R. K., Mahesh, T. R., & Almusharraf, A. (2024). Federated learning framework for consumer IoMT-edge resource recommendation under telemedicine services. IEEE Transactions on Consumer Electronics.

Fathima, A. S., Basha, S. M., Ahmed, S. T., Khan, S. B., Asiri, F., Basheer, S., & Shukla, M. (2025). Empowering consumer healthcare through sensor-rich devices using federated learning for secure resource recommendation. IEEE Transactions on Consumer Electronics.

Ahmed, S. T., Patil, K. K., Shanraj, R. K., Khan, S. B., Alzahrani, S., & Rani, S. (2024). 6GTelMED: Resources recommendation framework on 6G enabled distributed telemedicine using Edge-AI. IEEE Transactions on Consumer Electronics.

Ramaiah, N. S., & Ahmed, S. T. (2022). An IoT-based treatment optimization and priority assignment using machine learning. ECS Transactions, 107(1), 1487.

Pasha, A., Ahmed, S. T., Painam, R. K., Mathivanan, S. K., Karthikeyan, P., Mallik, S., & Qin, H. (2024). Leveraging ANFIS with Adam and PSO optimizers for Parkinson's disease. Heliyon, 10(9).