The appropriate selection and optimization of flotation reagents are essential for enhancing the quality of flotation coal, reducing reagent consumption, and lowering production costs in coal preparation plants. Currently, most coal preparation plants utilize compound reagents in their production processes. However, these compound reagents often fail to optimize both bubble formation and collection efficiency simultaneously, resulting in excessive reagent consumption during production. This study uses the compound reagent (XZ) employed by Xinzheng Clean Coal Company as a case study to thoroughly investigate the causes of its high reagent consumption and to perform optimization experiments. First, the differences in flotation performance between the XZ compound agent and the kerosene-sec-octanol system were compared. Next, the causes of high XZ agent consumption were identified using infrared spectral analysis, and finally, the response surface methodology (RSM) was applied to optimize the reagent system. Experimental results indicated that the effectiveness of the XZ agent primarily depended on its ester and ether groups, although its foaming performance was inadequate. The addition of sec-octanol significantly enhanced its foaming performance. Response surface optimization revealed that when the XZ agent dosage was 1000 g/t, sec-octanol dosage was 275 g/t, and slurry concentration was 90 g/L, the recovery of combustible matter peaked at 98.95%. When the XZ agent dosage was 997.33 g/t, sec-octanol dosage was 191.27 g/t, and slurry concentration was 60 g/L, the flotation perfection index reached its highest value of 58.12%. Further analysis revealed that a moderate froth layer height improves flotation performance, whereas excessively high or low froth layers negatively impact the flotation performance.