A COMPUTER VISION PIPELINE FOR AUTOMATED BIRD BEHAVIOR RECOGNITION IN A FIXED-CAMERA AVIARY SETTING

Abstract

Understanding the social behavior of house sparrows (Passer domesticus) is funda-mental to studying how dominance hierarchies form, how nest site competition shapes breeding success, and how affiliative and aggressive interactions drive the structure of avian social networks. Characterizing these behaviors requires continuous observation of multiple individuals simultaneously in naturalistic settings, a task that is fundamen- tally beyond the capacity of manual annotation, which captures only a small fraction of behavioral events for a few individuals at a time and cannot scale to the population- level monitoring. This thesis presents an automated bird behavior classification pipeline for detect-ing, tracking, and classifying the behaviors of house sparrows from fixed-camera aviary footage, enabling population-level behavioral analysis at a scale and temporal reso- lution that manual observation cannot achieve. The pipeline combines a fine-tuned YOLOv8 bird detector achieving a recall of 0.997 and mAP@0.5 of 0.979, a custom multi-object tracking system with novel ghost track logic for maintaining house sparrow aviary tracking continuity during nest box occlusion, and a temporal behavior recogni- tion model built on a Vision Transformer backbone with a causal temporal transformer and a spatial feature vector encoding positional, motion, and social context informa- tion. The temporal model achieves an overall accuracy of 0.9958 and macro F1 of 0.9844 across four behavior classes - on box, at hole, in box, and none. A specialized interaction detector extends the pipeline to side-by-side affiliative behavior classifica- tion, achieving mAP@0.5 of 0.984. The system is developed and evaluated on footage of house sparrows housed inoutdoor aviaries at the University of Oklahoma, using five synchronized GoPro cam- eras per enclosure and behavioral annotations provided by expert observers from the Department of Biology. The pipeline directly extends the 3D tracking infrastructure of prior work from our research group, which established where birds are in the aviary by addressing the complementary question of what behaviors those birds are performing.