In this paper, we propose a feature-aware network (FaNet) for a few shot defect classification, which can effectively distinguish new classes with a small number of labeled samples. In our proposed FaNet, we use ResNet12 as our baseline. The feature-attention convolution module (FAC) is applied to extract the comprehensive feature information from the base classes, as well as to fuse semantic information by capturing the long-range feature relationships between the upper and lower layers. Meanwhile, during the test phase, an online feature-enhance integration module (FEI) is adopted to average the noise from the support set and query set defect images, further enhancing image features among the different tasks. In addition, we construct a large-scale strip steel surface defects few shot classification dataset (FSC-20) with 20 different types. Experimental results show that the proposed method achieves the best performance compared to state-of-the-art methods for the 5-way 1-shot and 5-way 5-shot tasks. 

This article proposes a pyramid feature fusion and global context attention network for pixel-wise detection of surface defect, called PGA-Net. In the framework, the multiscale features are extracted at first from backbone network. Then the pyramid feature fusion module is used to fuse these features into five resolutions through some efficient dense skip connections. Finally, the global context attention module is applied to the fusion feature maps of adjacent resolution, which allows effective information propagate from low-resolution fusion feature maps to high-resolution fusion ones. In addition, the boundary refinement block is added to the framework to refine the boundary of defect and improve the result of the prediction. The final prediction is the fusion of the five resolutions fusion feature maps. The results of evaluation on four real-world defect datasets demonstrate that the proposed method outperforms the state-of-the-art methods on mean intersection of union and mean pixel accuracy (NEU-Seg: 82.15%, DAGM 2007: 74.78%, MT_defect: 71.31%, Road_defect: 79.54%).

Kechen Song and Yunhui Yan. A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects [J].Applied Surface Science, 2013, 285: 858-864. (database)

Kechen Song, Shaopeng Hu, Yunhui Yan and Jun Li. Surface defect detection method using saliency linear scanning morphology for silicon steel strip under oil pollution interference[J]. ISIJ International, 2014, 54(11):2598-2607 .

Kechen Song and Yunhui Yan. Micro surface defect detection method for silicon steel strip based on saliency convex active contour model[J].Mathematical Problems in Engineering, 2013,  (paper)

SONG Kechen, YAN Yunhui, PENG Yishu, DONG Dewei. Convex Active Contour Segmentation Model of Strip Steel Defects Image Based on Local Information[J].JOURNAL OF MECHANICAL ENGINEERING，2012，48(20)：1-7. (Chinese)

We propose a neural network named collaborative learning attention network (CLANet) for no-service rail surface defect inspection. The proposed method consists of three main stages: feature extraction, cross-modal information fusion, and defect location and segmentation. A multimodal attention block is proposed to highlight complex defect object with a new cross-modal fusion strategy. Furthermore, dual stream decoder enriches the representation of advanced features and avoids the dilution of information in the decoding stage. Suffering from the scarcity of defective data, an industrial RGB-D dataset NEU RSDDS-AUG is built. Finally, ablation studies verify the effectiveness of our proposed method.

A novel image-level weakly supervised segmentation formulation is proposed for no-service rail surface defects. These defects are decomposed into three sub-categories (strip-shaped, spot-shaped, block-shaped) according to the size prior information (area and shape). Then, a method is presented with a pooling combination module. The pooling combination module makes full use of the size attributes of the sub-category by utilizing different pooling functions for different sub-categories. Experimental results demonstrate that our method is effective and outperforms the state-of-the-art methods.

Defu Zhang, Kechen Song, et al. Two Deep Learning Networks for Rail Surface Defect Inspection of Limited Samples with Line-Level Label [J]. IEEE Transactions on Industrial Informatics, 2021,17(10),6731-6741.  (paper)

Shuai Ma, Kechen Song, et al. Cross-scale Fusion and Domain Adversarial Network for Generalizable Rail Surface Defect Segmentation on Unseen Datasets [J]. Journal of Intelligent Manufacturing, 2022 . (paper)

Menghui Niu, Kechen Song, et al. An Adaptive Pyramid Graph and Variation Residual-Based Anomaly Detection Network for Rail Surface Defects [J]. IEEE Transactions on Instrumentation and Measuremente, 2021,70,5020013 . (paper)

RENet is proposed for accurate and robust pavement crack detection. The rectangular convolution pyramid module is first built on deep layers so that the features can describe defects with different structures. The optimized contextual information and features of shallower layers are gradually merged into three resolutions. Subsequently, the hierarchical feature fusion refinement module and the boundary refinement module are applied to each branch. These two modules effectively promote the seamless fusion of features at various scales and make the model pay more attention to boundaries. Finally, the outputs of the three branches are integrated to obtain the final prediction map.

This paper proposes a relevance-aware and cross-reasoning network (RCN) for anomaly segmentation of pavement defects, which can segment defects using merely non-defective images for training. A relevance-aware transformer-based encoder is first devised to model intrinsic interdependencies across local features, thus improving representations of complex non-defective images. Next, a dual decoder strategy is proposed to remap the encoder-generated latent dependencies at the local semantic and global detailed levels, respectively. Specifically, a cross-reasoning refinement module is built in the local decoder to reason the crossrelationship between spatial and channel dimensions. Finally, a context-aware abnormal distillation measurement is developed to evaluate the semantic reconstruction deviations during the inference. Under the guidance of semantic affinity, this measurement allows our model to highlight defective areas adaptively. Extensive experimental results on four datasets indicate that RCN outperforms other leading anomaly segmentation methods.

We propose a variable illumination dataset named VI-RGBT1500 for RGBT image SOD. This is the first time that different illuminations are taken into account in the construction of the RGBT SOD dataset. Three illumination conditions, which are sufficient illumination, uneven illumination and insufficient illumination, are adopted to collect 1500 pairs of RGBT images.

The U-shaped encoder–decoder architecture based on CNNs has been rooted in salient object detection (SOD) tasks, and it have revealed two drawbacks while driving the rapid development of saliency detection. (1) The inherent characteristics of CNNs dictate that it is difficult to learn long-range dependencies and model global correlations. (2) For the common purpose of improving the performance of saliency detection, the encoder and decoder should complement each other and work together. However, the existing encoder–decoder architecture treats encoder and decoder independently of each other. Specifically, the encoder is responsible for extracting features and the decoder fuses multi-level or multi-modal features to produce prediction maps. That is, the encoder alone needs to be responsible for the decoder, while the valuable information after the decoder fusion will not facilitate feature extraction. Therefore, we propose a unidirectional RGB-T salient object detection network with intertwined driving of encoding and fusion to solve the above problems.

Yanqi Bao, Kechen Song, et al. Visible and Thermal Images Fusion Architecture for Few-shot Semantic Segmentation [J]. Journal of Visual Communication and Image Representation, 2021, 80, 103306.  (paper)(code and datasets)

Mingzheng Feng, Kechen Song, et al. Learning Discriminative Update Adaptive Spatial-Temporal Regularized Correlation Filter for RGB-T Tracking [J]. Journal of Visual Communication and Image Representation, 2020, 72, 102881.  (paper)

Industrial robots frequently encounter transparent objects in their work environments. Unlike conventional objects, transparent objects often lack distinct texture features in RGB images and result in incomplete and inaccurate depth images. This presents a significant challenge to robotic perception and operation. As a result, many studies have focused on reconstructing depth data by encoding and decoding RGB and depth information. However, current research faces two limitations: insufficiently addressing challenges posed by textureless transparent objects during the encoding-decoding process and inadequate emphasis on capturing shallow characteristics and cross-modal interaction of RGB-D bimodal data. To overcome these limitations, this study proposes a depth perception network based on orientation-aware guidance and texture enhancement for robots to perceive transparent objects. The backbone network incorporates an orientation-aware guidance module to integrate shallow RGB-D features, providing prior direction. In addition, this study designs a multibranch, multisensory field interactive texture nonlinear enhancement architecture, inspired by human vision, to tackle the challenges presented by textureless transparent objects. The proposed approach is extensively validated on both public datasets and industrial robotics platforms, demonstrating highly competitive performance. 

Robots have always found it challenging to grasp in cluttered scenes because of the complex background information and changing operating environment. Therefore, in order to enable robots to perform multi-object grasping tasks in a wider range of application scenarios, such as object sorting on industrial production lines and object manipulation by home service robots, we innovatively integrated segmentation and grasp detection into the same framework, and designed a simultaneous instance segmentation and grasp detection network (SISG-Net). By using the network, robots can better interact with complex environments and perform grasp tasks. In order to solve the problem of insufficient fusion of the existing RGBD fusion strategy in the robot field, we propose a lightweight RGB-D fusion module called SMCF to make modal fusion more efficient. In order to solve the problem of inaccurate perception of small objects in different scenes, we propose the FFASP module. Finally, we use the AFF module to adaptively fuse multi-scale features. Segmentation can remove noise information from the background, enabling the robot to grasp in different backgrounds with robustness. Using the segmentation result, we refine grasp detection and find the best grasp pose for robot grasping in complex scenes. Our grasp detection model performs similarly to state-of-the-art grasp detection algorithms on the Cornell Dataset. Our model achieves state-of-the-art performance on the OCID Dataset. We show that the method is stable and robust in real-world grasping experiments. The code and video of our experiment used in this paper can be found at: 

https://github.com/meiguiz/SISG-Net

It is challenging for robots to detect grasps with high accuracy and efficiency-oriented to multi-object clutter scenes, especially scenes with objects of large-scale differences. Effective grasping representation, full utilization of data, and formulation of grasping strategies are critical to solving the problem. To this end, this paper proposes an antipodal-points grasping representation model. Based on this, the Antipodal-Points-aware Dual-decoding Network (APDNet) is presented for grasping detection in multi-object scenes. APDNet employs an encoding–decoding architecture. The shared encoding strategy based on an Adaptive Gated Fusion Module (AGFM) is proposed in the encoder to fuse RGB-D multimodal data. Two decoding branches, namely StartpointNet and EndpointNet, are presented to detect antipodal points. To better focus on objects at different scales in multi-object scenes, a global multi-view cumulative attention mechanism, called Global Accumulative Attention Mechanism (GAAM), is also designed in this paper for StartpointNet. The proposed method is comprehensively validated and compared using a public dataset and real robot platform. On the GraspNet-1Billion dataset, the proposed method achieves 30.7%, 26.4%, and 12.7% accuracy at a speed of 88.4 FPS for seen, unseen, and novel objects, respectively. On the AUBO robot platform, the detection and grasp success rates are 100.0% and 95.0% on single-object scenes and 97.0% and 90.3% on multi-object scenes, respectively. It is demonstrated that the proposed method exhibits state-of-the-art performance with well-balanced accuracy and efficiency. 

Chongyan Sun, Kechen Song, et al. A Multi-Exposure Fusion Method for Reflection Suppression of Curved Workpieces [J]. IEEE Transactions on Instrumentation and Measuremente,2022,71,5021104. (paper) (code & dataset)