ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection
三维物体检测中弱监督领域自适应的自训练
3D detectors developed on one specific domain (i.e. source domain) might not generalize well to novel testing domains (i.e. target domains) due to unavoidable domain-shifts arising from different types of 3D sensors, weather conditions and geographical locations, etc
由于不同类型的3D传感器、天气条件和地理位置等导致的不可避免的领域变化，在一个特定领域（即源领域）开发的3D检测器可能无法很好地推广到新的测试领域（即目标领域）。

Therefore, approaches to effectively adapting 3D detector trained on labeled source domain to a new unlabeled target domain is highly demanded in practical applications.

因此，在实际应用中，有效地适应在标记源域上训练的3D检测器到一个新的未标记目标域是非常需要的方法

ST3D

• 总体
• • Random Object Scaling (ROS)
  • • 目的
    • 步骤
  • Pseudo label Generation with Quality-aware Triplet Memory Bank (QTMB)
  • • 目的
    • 组成
    • 步骤
    • • Proxy-pseudo Label {[L^it]k}i=1nt\{[\widehat{L}^t_i]_k\}^{n_t}_{i=1}{[Lit​]k​}i=1nt​​
  • Curriculum Data Augmentation (CDA)
• Algorithm
• • 符号说明
  • 输入：源域的点云和标签{(Pis,Lis)}i=1ns\{(P^s_i,L^s_i)\}^{n_s}_{i=1}{(Pis​,Lis​)}i=1ns​​，目标域的点云{Pit}i=1nt\{P^t_i\}^{n_t}_{i=1}{Pit​}i=1nt​​
  • 输出：目标域的三维目标检测模型。

总体

Random Object Scaling (ROS)

随机目标缩放策略

目的

overcome the bias in object size on the labeled source domain
克服了标记源域上对象大小的偏差。

ROS可以有效地模拟具有不同对象大小的对象，以解决大小偏差，从而有助于训练大小鲁棒的检测器，为后续的自我训练产生更精确的初始伪盒

步骤

给定3D bounding box with size大小 (l,w,h)(l, w, h)(l,w,h), center中心 (cx,cy,cz)(c_x, c_y, c_z)(cx​,cy​,cz​) and heading angle θ\thetaθ , random scale factors随机规模因子 (rl,rw,rh)(r_l ,r_w ,r_h )(rl​,rw​,rh​)
有npn_pnp​个点, 分别是{pi}i=1np\{p_i \}^{n_p}_{i=1}{pi​}i=1np​​, 其坐标是(pix,piy,piz)(p_i^x,p_i^y,p_i^z)(pix​,piy​,piz​)

First, we transform the points to the local coordinate system of the box along its length, width and height dimensions via

(pil,piw,pih)=(pix−cx,piy−cy,piz−cz)⋅R(p_i^l,p_i^w,p_i^h)=(p_i^x-c_x,p_i^y-c_y,p_i^z-c_z)\cdot R(pil​,piw​,pih​)=(pix​−cx​,piy​−cy​,piz​−cz​)⋅R
R=[cos⁡θ−sin⁡θ0sin⁡θcos⁡θ0001],R=\begin{bmatrix}\cos\theta&-\sin\theta&0\\ \sin\theta&\cos\theta&0\\ 0&0&1\end{bmatrix},R=​cosθsinθ0​−sinθcosθ0​001​​,
where · is matrix multiplication.
通过该公式将这些点转化为局部坐标系

Second, to derive the scaled object, the point coordinates inside the box are scaled to be (rlpil,rwpwi,rhpih)(r_lp^l_i, r_wp^wi, r_hp^h_i )(rl​pil​,rw​pwi,rh​pih​) with object size (rll,rww,rhh)(r_ll, r_ww, r_hh)(rl​l,rw​w,rh​h).
框内点坐标被缩放为(rlpil,rwpwi,rhpih)(r_lp^l_i, r_wp^wi, r_hp^h_i )(rl​pil​,rw​pwi,rh​pih​)

Third, to derive the augmented data {piaug}i=1np\{p^{aug}_i \}^{n_p}_{i=1}{piaug​}i=1np​​, the points inside the scaled box are transformed back to the ego-car coordinate system and shifted to the center (cx,cy,cz)(c_x, c_y, c_z)(cx​,cy​,cz​) as
piaug=(rlpil,rwpiw,rhpih)⋅RT+(cx,cy,cz)p_i^{\text{aug}}=(r_lp_i^l,r_wp_i^w,r_hp_i^h)\cdot R^T+(c_x,c_y,c_z)\quadpiaug​=(rl​pil​,rw​piw​,rh​pih​)⋅RT+(cx​,cy​,cz​)
移回中心

Pseudo label Generation with Quality-aware Triplet Memory Bank (QTMB)

利用QTMB(质量感知的三重存储库)生成伪标签

目的

integrating historical pseudo labels via ensemble and voting, to reduce pseudo label noise and stabilize training.
通过ensemble和voting集成历史伪标签，以减少伪标签噪声和稳定训练

组成

1. an IoU-based box scoring criterion (to directly assess the quality of pseudo boxes)
   一个基于IoU的盒评分标准,来直接评估伪盒的质量
2. a triplet box partition scheme (to avoid assigning pseudo labels to ambiguous examples)
   一个三重盒分区方案,来避免分配伪标签变成模糊的例子
3. a memory bank,ensemble and voting
   一个记忆库，集成和投票

步骤

总体：

1. 通过PitP^t_iPit​(目标域点云)，输出BitB^t_iBit​（a group of predicted boxes是一组预测框，the object predictions for a scene称之为场景的对象预测）
2. BitB^t_iBit​ 包含 category confidence scores类别置信分数，regressed box sizes回归框大小, box centers回归框中心 and heading angles
3. {[L^it]k}i=1nt\{[\widehat{L}^t_i]_k\}^{n_t}_{i=1}{[Lit​]k​}i=1nt​​(proxy-pseudo label )是由{Bit}i=1t\{B^t_i\}^t_{i=1}{Bit​}i=1t​ 通过 IoU-based scoring criterion策略产生
4. {[Mit]k}i=1nt\{[M^t_i]_{k}\}^{n_t}_{i=1}{[Mit​]k​}i=1nt​​ 是包含所有目标域的伪标签pseudo labels。 {[Mit]k}i=1nt\{[M^t_i]_{k}\}^{n_t}_{i=1}{[Mit​]k​}i=1nt​​ 是由{[L^it]k}i=1nt\{[\widehat{L}^t_i]_k\}^{n_t}_{i=1}{[Lit​]k​}i=1nt​​ 和{[Mit]k−1}i=1nt\{[M^t_i]_{k-1}\}^{n_t}_{i=1}{[Mit​]k−1​}i=1nt​​ 通过ensemble和voting集成历史伪标签。

Proxy-pseudo Label {[L^it]k}i=1nt\{[\widehat{L}^t_i]_k\}^{n_t}_{i=1}{[Lit​]k​}i=1nt​​

Curriculum Data Augmentation (CDA)

progressively increasing the intensity of augmentation, to guarantee effective learning at the beginning
逐步增加增强的强度，以保证在开始时有效学习，
gradually simulate hard examples to improve the model, preventing it from overfitting to easy examples – pseudo-labeled data with high confidence

并逐步模拟难的示例来改进模型，防止其对简单的伪标记数据的过拟合

Algorithm

符号说明

{(Pis,Lis)}i=1ns\{(P^s_i,L^s_i)\}^{n_s}_{i=1}{(Pis​,Lis​)}i=1ns​​：nsn_sns​个的源域点云和其标签
{Pit}i=1nt\{P^t_i\}^{n_t}_{i=1}{Pit​}i=1nt​​：ntn_tnt​个的目标域点云(最终就是要得到其标签)
LisL^s_iLis​: LisL^s_iLis​ 包含第i个点云中每个对象的类别和3D bounding box信息，每个box有其大小(l,w,h)(l,w,h)(l,w,h)、中心(cx,cy,cz)(c_x,c_y,c_z)(cx​,cy​,cz​)和heading angle θ

输入：源域的点云和标签{(Pis,Lis)}i=1ns\{(P^s_i,L^s_i)\}^{n_s}_{i=1}{(Pis​,Lis​)}i=1ns​​，目标域的点云{Pit}i=1nt\{P^t_i\}^{n_t}_{i=1}{Pit​}i=1nt​​

输出：目标域的三维目标检测模型。

1. 使用ROS在{(Pis,Lis)}i=1ns\{(P^s_i,L^s_i)\}^{n_s}_{i=1}{(Pis​,Lis​)}i=1ns​​上预训练目标检测
2. 利用当前模型为每个目标域样本PitP^t_iPit​生成预测框[Bit]k[B^t_i]_k[Bit​]k​（k为生成的次数）
3. 在给定[Bit]k[B^t_i]_k[Bit​]k​的情况下，使用Triplet Box Partition生成伪标签[L^it]k[\widehat{L}^t_i]_k[Lit​]k​
4. 通过集成和投票(MEV)使用[L^it]k[\widehat{L}^t_i]_k[Lit​]k​更新[Mit]k−1[M^t_i]_{k-1}[Mit​]k−1​（第i个目标域样本的label memory bank，初始状态为空），维护[Mit]k[M^t_i]_{k}[Mit​]k​的状态
5. 在上{(Pit,[Mit]k)}i=1nt\{(P^t_i,[M^t_i]_{k})\}^{n_t}_{i=1}{(Pit​,[Mit​]k​)}i=1nt​​使用CDA训练模型。
6. 回到步骤2直到模型收敛