Consistency Regularization

Exponential Moving Average (EMA)

$\tilde{z} = \frac{Z}{1-\alpha^t}$

$Z$ 被初始为 $\mathbf{0}_{N\times C}$$z$ 是每个 epoch 模型对于每个 sample 输出，$t$ 是epoch，$\tilde{z}$是经过 bias correction 的 target vector。

Virtual Adversarial Training (VAT)

Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation.