Spatial Transformer for 3D Point Clouds

ArXiv 2019


Deep neural networks can efficiently process the 3D point cloud data. At each layer, the network needs to partition points into multiple local patches, and then learn features from them, in order to understand the geometric information encoded in the 3D point cloud.Previous networks adopt all the same local patches for different layers, as they utilized the same fixed original 3D point coordinates to define local neighborhoods. It is easy to implement but not necessarily optimal. Ideally local patches should be different at different layers so as to adapt to the specific layer for efficient feature learning. One way to achieve this is to learn different transformations of the original point cloud at each layer, and then learn features from local patches defined on transformed coordinates. In this work, we propose a novel approach to learn different non-rigid transformations of the input point cloud for different local neighborhoods at each layer. We propose both linear (affine) and non-linear (projective and deformable) spatial transformer for 3D points. With spatial transformers on theShapeNet part segmentation dataset, the network achieves higher accuracy for all categories, specifically with 8% gain on earphones and rockets. The proposed methods also outperform the state-of-the-art methods in several other point cloud processing tasks(classification, semantic segmentation and detection). Visualizations show that spatial transformers can learn features more efficiently by altering local neighborhoods according to the semantic information of 3D shapes regardless of variations in a category.

Public Video


Code and Models


  author    = {Jiayun Wang and
               Rudrasis Chakraborty and
               Stella X. Yu},
  title     = {Spatial Transformer for 3D Points},
  journal   = {CoRR},
  volume    = {abs/1906.10887},
  year      = {2019},
  url       = {},