This paper presents a unified deep learning framework for automated sperm counting and motility assessment using Computer-Aided Semen Analysis. Conventional semen analysis is predominantly manual, making it time consuming, subjective, and susceptible to inter-observer variability, whereas existing automated systems often struggle in the presence of debris and morphological abnormalities. To address these limitations, the You Only Look Once version 8 (YOLOv8) Nano object detection model was fine-tuned using carefully annotated clinical data to accurately detect and count only valid sperm cells that are characterized by the presence of both the head and tail, while effectively suppressing non-sperm objects in noisy and unclean semen samples. In addition, a tracking-free approach for motility assessment is proposed by reformulating the problem as a video-level regression task. Instead of relying on computationally expensive and error-prone tracking-by-detection pipelines, the proposed method predicts the overall sperm motility directly from video sequences by modeling global optical flow patterns and temporal dynamics. To achieve this, three spatiotemporal deep learning architectures were developed and systematically compared: a convolutional neural network (CNN)-based optical flow model, CNN–long short-term memory (LSTM) temporal model, and semi-supervised 3D autoencoder designed to learn latent motion representations from limited labeled data. All models were trained and validated on a real-world dataset acquired from a private fertility clinic, reflecting clinically relevant imaging conditions. The experimental results demonstrated that the proposed framework achieved robust sperm detection and reliable motility estimation (with the CNN–LSTM model achieving the best performance). The proposed approach offers an effective alternative to manual analysis.