mmSeg: Leveraging mmWave Radar for Fine-grained Human Semantic Segmentation

What is it about?

Human semantic segmentation facilitates the recognition of different parts of the human body and is essential for applications such as sports analysis and fall detection. To integrate human semantic segmentation into the domain of radio front-end sensing, this paper introduces mmSeg, an innovative system that leverages commercial millimeter-wave radar for human semantic segmentation. However, the inherent propagation characteristics of mmWave signals often result in highly sparse point clouds with limited semantic information and the entanglement of temporal-topological features, making human semantic segmentation a challenging task. To address these challenges, mmSeg (i) first introduces a radar cross-section (RCS) calculation method suitable for commercial millimeter-wave radar to enhance the semantic information of radar point clouds at a coarse granularity; (ii) further designs a temporal-topological decoupling network to obtain the fine-grained human semantic segmentation results; (iii) constructs an efficient loss function for end-to-end training, based on an adjacency matrix graph to improve the segmentation performance. We evaluate mmSeg on our self-built millimeter-wave dataset HSS and a public dataset MM-Fi. mmSeg achieves an average point cloud segmentation accuracy of 87.74% on the HSS dataset and 84.18% on the MM-Fi dataset, outperforming the existing methods in both cases.

Featured Image

Photo by Marco Bianchetti on Unsplash

Photo by Marco Bianchetti on Unsplash

Why is it important?

This paper presents mmSeg, a novel system for human semantic segmentation using commercial mmWave radar. The system architecture of mmSeg consists of four key modules: (1) Data collection and preprocessing, (2) RCS-based semantic enhancement, (3) temporal-topological feature encoding, and (4) graph-constrained semantic segmentation output. We first build a data collection platform to obtain the mmWave point cloud of the subjects, then we use the preprocessing module to achieve spatiotemporal alignment and noise filtering. The processed radar point clouds are then fed into the RCS-based semantic enhancement module. Although mmWave point clouds are inherently sparse, our key observation reveals significant differences in the RCS threshold ranges across different human body parts, which provides coarse-grained guidance for initial human semantic segmentation. Subsequently, the enhanced 6D mmWave point clouds are fed into the temporal-topological feature encoder to mine more fine-grained semantic features. To address the unordered nature of point clouds and capture intra-frame topological structures, mmSeg introduces an innovative spatial attention mechanism based on symmetric functions for topological feature encoding. Furthermore, temporal sequence learning is employed to model the coupled temporal-topological relationships within point cloud sequences, thereby generating discriminative fine-grained human semantic features. The human semantic segmentation output module produces body part labels (e.g., head, torso, and limbs) for each point. During training, we propose a graph-theory-based custom loss function that leverages a weight matrix between points and segmentation labels to define graph connectivity, effectively assisting the model in reconstructing topological features during optimization.