The vibrating flip-flow screen (VFFS) is extensively employed for screening moist fine-grained minerals. However, prevalent uniform-amplitude VFFSs often suffer from material accumulation and suboptimal screening efficiency. In this paper, a dynamic model of the multi-mass VFFS was established using the multi-mass dynamics theory, and the amplitude regulation in different regions of the multi-mass VFFS was achieved by adjusting the mass of the floating screen frame (FSF), thereby revealing the regulation mechanism of the variable-amplitude VFFS. Leveraging the coupling advantages of the finite element method (FEM) and discrete element method (DEM), a simulation model of the interaction between multiple flip-flow screen panels and minerals was constructed. The kinematic characteristics of particles and the screen panel under constant and variable amplitude conditions for the VFFS were analyzed, respectively, and the impact of dynamic parameters on screening performance was investigated. A three-factor and three-level orthogonal experimental design, combined with response surface optimization, was used to assess the influence of screen panel inclination, angular frequency, and relative amplitudes of the main screen frame (MSF) and FSF on the screening performance and time. The results indicated that the optimal solutions considering both screening efficiency and screening time can be achieved when the relative amplitude of the three regions of variable-amplitude VFFS are 4 mm, 6 mm, and 8 mm, respectively, the screen inclination was 11°, and the angular frequency was 76 rad· s^-1.