Coal slurry foam image enhancement based on multiscale convolutional network

: Collecting information on the flotation foam characteristics is important for controlling flotation production conditions. Foam images acquired during coal slurry flotation are affected by factors such as ambient lighting, contributing to uneven grayscale images with low brightness and contrast. Brightness enhancement of foam images is often required when using network models to extract feature information from the images. The paper proposes a foam image brightness enhancement algorithm based on a multiscale convolutional neural network. The method employs a skip connection structure based on a summation connection design based on logarithmic functions and introduces a loss function based on logarithmic transformation in the network. At the same time, branching networks of different complexity are designed in the network to further help alleviate the gradient vanishing problem. The experimental results show that when evaluating the quality of images after brightness enhancement of foam images and the public dataset MIT, the numerical results of using the proposed skip connection structure in the proposed network are overall better than using the resblock structure, and the proposed loss function is better than is better than using the L2 loss function. The proposed network greatly improves the visual effect of flotation foam images and lays the foundation for feature extraction of flotation foam images and intelligent flotation production.


Introduction
Coal slurry foam flotation is one of the most effective methods for achieving fine coal sorting based on the difference in hydrophobicity between the coal particles and gangue in the slurry.During the flotation production process, according to the difference in the wetting of the coal slurry surface, hydrophobic coal particles adhere to air bubbles and float to the slurry surface, with hydrophilic gangue remaining in the slurry as tailings.The flotation staff often make subjective judgments and perform manual operations on production variables, such as inflation volume and drug dosage, after obtaining information on the surface characteristics of the foam.However, frequent fluctuations in flotation production indices, high chemical consumption rates, and low resource recovery rates may occur depending on factors such as site lighting and personnel experience (Zarie et al., 2020;Wen et al., 2021;Aldrich et al., 2022;Pawlik et al., 2022;Cao et al., 2022).Therefore, using deep learning-based machinevision technology to monitor and identify the foam surface characteristic information during the flotation production process in real-time can guide the adjustment of relevant production elements and improve the mineral resource utilisation and economic efficiency of coal washing plants.Owing to factors such as uneven illumination and water vapour generation, foam images obtained at the production site often have low brightness and contrast, and uneven greyscale distributions, hindering the recognition and extraction of features (Zhang et al., 2019;Ju et al., 2022;Zhang et al., 2022).Zhang et al. (2019) employed the SSR algorithm (Jobson et al., 1997), which uses the original foam image minus the low-frequency components of the image obtained after log transformation.After resolving the uneven grayscale distribution and poor brightness and contrast of the foam images, they used a watershed segmentation algorithm with optimal markers to segment the foam images.Ju et al. (2022) applied grayscale processing, the top-hat transform, and the filter transform before morphological feature extraction and segmentation to improve the brightness and contrast of the acquired foam images and solve the problem of uneven grayscale distribution.However, traditional low-light image enhancement methods such as grayscale processing, the top-hat transform, and the filter transform are susceptible to pixel-to-pixel relationships during image detail enhancement when processing foam images (Peng et al., 2023).
Therefore, the paper proposes a multiscale convolutional neural network-based brightness enhancement method for foam images.The proposed method solves the problems of the uneven grayscale distribution and low image brightness and contrast of foam images, improves the visual effect of foam images, and lays the foundation for feature extraction and grade analysis of coal slurry flotation foam images.The contributions of this study are as follows: (1) In order to smooth out the changes in the image information features and at the same time increase the sensitivity to the changes, a designed logarithmic function is introduced in the skip connection structure.Meanwhile, the designed connection hopping structure can be used as a plug and play module for building target networks for image tasks.The designed connection skip structure can be used as a plug and play module to construct the target network of image tasks.
(2) A new loss function based on a logarithmic transform is proposed.The effectiveness of the proposed network architecture and loss function in foam image brightness enhancement is experimentally verified.
(3) In order to reduce the information loss, branches with different levels of complexity are designed in the deep learning-based brightness enhancement method.Experiments have proved that the network can effectively improve the brightness and contrast of the image and enhance the contour, edge, and detail information of the flotation image, thus laying the foundation for intelligently adjusting the production elements and guiding the flotation production process.

Traditional methods
The problems of low brightness and contrast and uneven grayscale distribution in low-light images can be solved by traditional image-processing methods, such as the hue mapping algorithm (Cao et al., 2021;Pang et al., 2021;Zhu et al., 2022) and histogram equalisation (Hu et al., 2022;Xiao et al., 2022;Zhu et al., 2022).Guo et al. (2022) used histogram equalisation and filtering to preprocess coal slurry foam images for extracting coal slurry foam velocity features.Jiang et al. (2023) used the NSST method to identify the flotation working conditions of processed images for model training and testing.However, the hue-mapping algorithm may result in the loss of image details due to incomplete consideration of the relationship between pixels, whereas, in histogram equalisation, image details may be lost owing to over-enhancement (Peng et al., 2023).Therefore, image detail information must be carefully controlled in traditional low-light image enhancement methods.

Deep-learning methods
In recent years, deep-learning methods have been successfully applied in low-light image brightness enhancement tasks (Ma et al., 2022;Fan et al., 2022;Guo et al., 2023;Nguyen et al., 2023;Wang et al., 2023).Self-calibrated illumination (SCI) involves low-light image brightening using a weight-sharing cascade mode learning process.In SCI, a self-calibration module converges the results of each stage to enhance the learning effect while relieving the computational pressure of the network structure using the cascade mode.Simultaneously, the self-calibration module constrains the unsupervised training loss defined in the SCI to enhance low-light image brightness enhancement.Moreover, supervised training losses were constrained to increase the luminance in low-light images (Ma et al., 2022).HWMNet (Fan et al., 2022), which uses half-wavelet attention blocks (HWABs) in each layer of the bilinear downsampling of the image to fully capture the image features in the wavelet domain, achieved good results in the image-exposure-adjustment task.Moreover, HWMNet uses bilinear downsampling in the gate post path to reduce image feature information losses and reconstructs the images using SKFF (Zamir et al., 2020).

Materials and methods
The network used in this study is based on a multiscale neural-network architecture, with the specific network design shown in Fig. 1, where the network becomes finer as the image resolution increases, while the output of the coarse network at the previous scale is the input of the fine network at the next scale.The network improves the image resolution by upsampling between scales, which helps the finer network focus on the main image features.Meanwhile, branches of varying complexity designed in the network can mitigate the problem of vanishing gradients.

Skip connection structure
As shown in Fig. 2, the skip connection structure designed in this study consists of two convolutional layers and a summation skip connection designed according to the logarithmic function.Inspired by the ResBlock structure (He et al., 2016), ResNeXt structure (Xie et al., 2017), DenseBlock structure (Huang et al., 2017), and inception family of architectures (Szegedy et al., 2015;Ioffe et al., 2015;Szegedy et al., 2016;Szegedy et al., 2017), the skip connection structure mitigates the gradient disappearance problem by summing the outputs of different network layers to obtain the gradient information of each network layer.
The designed skip connection structure is expressed as given by Eq. ( 1): where the image feature information x after the first convolutional layer is denoted as p(x), respectively.The image feature information after the skip connection structure is denoted as H(x).The added logarithmic function makes the change of image information smoother while improving the sensitivity of the skip connection structure to the image information, while according to the nature of the logarithmic function, the addition of the function helps to eliminate the covariance and anisotropy between the connected image information without altering the original relationship between the pixels.

Loss function
The logarithmic transform in digital image processing can enhance the low-grey value details of an image by expanding and compressing the low-and high-grey value parts of the image, respectively, as given by Eq. ( 2). S and r represent the output and input grey values, respectively, C is a constant, and r ≥ 0.
Inspired by the logarithmic transform, a loss function based on the logarithmic transform is proposed, as given by Eq. ( 3), where the resolution of the foam image is m×n, and ro and rp represent the gray value of the target foam image after image normalisation and that of the foam image after brightness enhancement via image normalisation, respectively.
The effectiveness of the proposed loss function in enhancing the brightness of foam images in the designed network structure is confirmed.The experimental results indicated that the loss function could effectively enhance image brightness.

Foam image acquisition
Foam image data were obtained from the flotation production site of a coal-washing company in the Inner Mongolia Autonomous Region, China.The structure of the foam image acquisition device is shown in Fig. 3 and that of the site acquisition device is shown in Fig. 4.
During the 8 d of filming, it was collected the foam images from flotation cells 1 and 2 of the flotation production plant every 20 min for 11 h per day.

Experimental procedure and results
The experimental equipment used in the experiments had a graphics processing unit of 2080Ti, a central processor of i7-12700KF, the optimiser used in the network was Adam, and the learning rate was set to 0.0001.
The MIT dataset (Bychkovsky et al., 2011) contains 5000 low-light images and corresponding brightness-enhanced images after processing.It was randomly selected 1000 images from this dataset for the network model comparison test.Under the same experimental conditions, the proposed method was compared with SCI (Ma et al., 2022), HWMNet (Fan et al., 2022), the network proposed by Hu et al. (2023), and the PSENet (Nguyen et al., 2023) network used for enhancing the brightness of low-light images.

Coal slurry foam image dataset
For the coal slurry foam image dataset, three image quality evaluation indices-standard deviation, contrast, and entropy-were selected to objectively evaluate the image brightness enhancement.The standard deviation reflected the contrast between brightness and darkness, the contrast reflected the degrees of brightness stretching and image clarity, and the entropy reflected the complexity of the feature information in the image; a larger image quality evaluation index value corresponds to a better image enhancement effect.The image evaluation indices obtained from the experiments are presented in Table 1, with the corresponding visual effects presented in Table 2.Among the selected image quality evaluation indexes, the proposed method achieves the highest results in other evaluation indexes except that the value of standard deviation is 0.006 lower than that of PSENet.(Table 1).Moreover, compared with the original image and other network-enhanced images (Table 2), more image details, such as coal slurry foam contours and edges, could be observed after the brightness enhancement of the coal slurry foam images.In addition, the images had better colour balance and visual effects.

Traditional watershed division effect display
Segmentation of coal foam flotation images before and after network enhancement using the most common conventional watershed algorithm to verify whether the enhanced froth images are useful for further image tasks, such as froth image feature extraction, to guide the flotation production process.
In image processing, the watershed algorithm is often used to segment image edges (He et al., 2022;Sharma et al., 2022;Chowdhury et al., 2023).This algorithm can be understood as rainwater landing on a mountain surface and flowing down the terrain; if the water lands on two different points belonging to the same region, it will eventually flow to the same local nadir and form a segmented region.Water falling on a ridge has the same probability of flowing into the surrounding areas; therefore, the ridge is a watershed.The segmentation results are presented in Table 3. From the Table 3, it can be seen that more image information is not presented when the foam image is enhanced using the traditional contrast stretching method.There is more loss of image segmentation details when the foam image is enhanced using the traditional histogram equalization method.The foam edge details were clearer after the foam image was brightened by the proposed network.Additionally, the segmentation effect was closer to the foam edge, as judged by visual observation, when the traditional watershed segmentation algorithm was used to segment the foam edge.Thus, the brightness enhancement of the foam image by the proposed network helped the machine extract the foam feature information more efficiently.

MIT dataset
The MIT dataset (Bychkovsky et al., 2011) is a widely used image dataset containing 5000 original lowlight images and five sets that form 5000 pairs of images with the original images obtained via brightening processing by five professionals.For the MIT image dataset, the image quality evaluation indices selected in this study were the peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and mean squared error after image brightness enhancement.Larger PSNR and SSIM values correspond to a better enhancement effect, while a smaller MSE value corresponds to a higher degree of similarity between the images before and after enhancement.The image evaluation indices obtained from the experiments are presented in Table 4, with the corresponding visual effects presented in Table 5.The proposed network exhibited the largest PSNR and SSIM values and smallest MSE values, indicating that the brightness-enhanced MIT images had good similarity and image detail display.Moreover, after the brightening process, the MIT images had better image details, sharpness, and colour balance than the original image and the images processed by other networks, with the processed graph closest to the labelled image presented in Table 5.

Skip connection structure
Ablation experiments on the possible begging and skip connection methods for the designed skip connection structure are conducted using the coal slurry foam image test set.As shown in the figure 3, the experimental skip connection structures for the test were defined as x + p(log(|p(x) − x| + 1) + p(x)/.The experimental results on the publicly available MIT dataset are presented in Table 6, the experimental results on the coal slurry foam dataset are presented in Table 7.The designed summation connection achieves better results in the quality evaluation metrics for the MIT dataset (Table 6), and the quality evaluation metrics for the coal slurry foam dataset are only 0.006 lower on the metric of Standard deviation (Table 7).Therefore, the designed skip connection structure can help the network to achieve a better image brightness enhancement effect, however, whether this skip connection structure can achieve good results in other network structures for image processing tasks remains to be further explored.

Loss function
The proposed loss function is compared with the L2 loss function commonly used in image tasks to verify its effectiveness.When the skip connection structure in the network is Resblock, the evaluation results for the MIT dataset are presented in Table 8, while those for the coal slurry foam dataset are presented in Table 9.
As shown in Tables 8 and 9, When the skip connection structure in the network is Resblock, the designed loss function is worse than using the L2 loss when evaluating the MIT dataset only on the image quality assessment metric MSE, and when evaluating the coal slurry foam dataset, the results of the image quality assessment metrics were only 0.001 lower than the results using L2 loss on the quality evaluation metric Standard deviation.

+ Output Log Conv
The proposed loss function is compared with the L2 loss function commonly used in image tasks to verify its effectiveness.When the skip connection structure in the network is as shown in Fig. 3, the evaluation results for the MIT dataset are presented in Table 10, while those for the coal slurry foam dataset are presented in Table 11.As shown in Table 10 and 11, when the skip connection structure as shown in Fig. 3 is used in the network, the designed loss function is worse than using the L2 loss only for the image quality assessment metric MSE when evaluating the MIT dataset, and worse than using the L2 loss only for the image quality assessment metric contrast when evaluating the coal slurry foam dataset.
As shown in Tables 8, 9, 10, and 11, the use of the designed loss function in the network is a better choice to help in the enhancement of brightness and contrast of the image and achieve a good result.However, further experiments are needed to investigate the effectiveness of the proposed loss function in building other network architectures for image-processing tasks.

Conclusion
The paper proposes a deep-learning method for the brightness enhancement of flotation foam images to solve the problems of uneven grayscale distribution and low brightness and contrast of foam images.A summation skip connection based on a logarithmic function and propose a loss function based on a logarithmic transformation is proposed.In addition, ablation experiments on the proposed summation skip connection structure and loss function to verify the effectiveness of the network in image brightening.The segmentation effect of the foam image before and after brightening was demonstrated using the traditional watershed algorithm.The improved visual effect of the foam image confirmed the effectiveness of the designed network for brightening the image.Overall, this method lays the foundation for effectively extracting foam image features and for intelligent adjustment of production factors in guiding the flotation production process.Further research is needed to evaluate the effectiveness of the proposed skip connection structure and loss function in other network architectures and image tasks.Additional research will be conducted to lay the foundation for effective extraction and grade analysis of coal slurry flotation foam image features.

Fig. 3 .
Fig. 3. Structure diagram of the acquisition device

Table 1 .
Evaluation results for images of coal slurry foam after brightness enhancement

Table 2 .
Coal slurry foam images after brightness enhancement

Table 3 .
Foam image segmentation results

Table 4 .
Results of image evaluation after brightness enhancement for the MIT dataset

Table 5 .
Visual effects of MIT images after image enhancement