汪航,李宝婷,张旭翀,等.深度神经网络在线训练硬件加速器的数据量化综述[J]. 微电子学与计算机,2024,41(3):1-11. doi: 10.19304/J.ISSN1000-7180.2023.0145
引用本文: 汪航,李宝婷,张旭翀,等.深度神经网络在线训练硬件加速器的数据量化综述[J]. 微电子学与计算机,2024,41(3):1-11. doi: 10.19304/J.ISSN1000-7180.2023.0145
WANG H,LI B T,ZHANG X C,et al. A review of data quantization of deep neural network online training hardware accelerator[J]. Microelectronics & Computer,2024,41(3):1-11. doi: 10.19304/J.ISSN1000-7180.2023.0145
Citation: WANG H,LI B T,ZHANG X C,et al. A review of data quantization of deep neural network online training hardware accelerator[J]. Microelectronics & Computer,2024,41(3):1-11. doi: 10.19304/J.ISSN1000-7180.2023.0145

深度神经网络在线训练硬件加速器的数据量化综述

A review of data quantization of deep neural network online training hardware accelerator

  • 摘要: 随着算法和数据的爆炸式增长,深度神经网络(Deep Neural Network, DNN)逐渐在实际应用中扮演愈发重要的角色。然而,真实场景中的数据与线下训练数据之间往往并不满足独立同分布假设,导致预训练DNN模型在实际应用中性能严重下降。所以,在资源供给相对有限的平台上进行DNN模型在线训练成为其有效应用的保证。为了满足真实场景对DNN模型质量与速度的多维度性能要求,如何在保证算法精度的同时显著降低计算复杂度是在此类应用中部署DNN的关键。数据量化是降低计算复杂度的主流优化技术之一,能够通过降低模型参数、中间值等数据的位宽来减少硬件加速器的资源耗费。因此,从软件和硬件两个方面对深度神经网络训练加速器中关于数据量化的研究进行总结。对国内外最新发表的相关文献进行归纳总结。首先,从软件的角度总结了不同的量化方法,包括简单映射数据量化和复杂映射数据量化;其次,从硬件的角度总结了DNN加速器对网络在线训练各计算步骤的量化支持;再次,阐述了数据量化对加速器设计的影响,包括存储单元和计算单元;最后,对本领域的研究进行总结,并展望了未来本领域的发展方向。文章提出的分类方法有助于对之前的DNN加速器在数据量化方面的工作进行分类。

     

    Abstract: With the explosive growth of algorithms and data, the Deep Neural Network (DNN) gradually plays an increasingly important role in practical applications. However, it is difficult for the real scene and offline training data to meet the assumption of independent and identical distribution, resulting in a serious decline in the performance of the pre-training DNN model in practical applications. Therefore, online training of DNN model on the platform with relatively limited resources becomes the guarantee of its effective application. Hence, it is necessary to significantly reduce the computational complexity while ensuring the accuracy. Data quantization is one of the mainstream optimization technologies to reduce computational complexity. Accordingly, we summarize the researches on data quantization of DNN online training accelerator. Firstly, data quantization based on direct fixed-point representation and complex mapping are summarized from the perspective of software. Secondly, the quantization of DNN accelerator for each training step is summarized from the perspective of hardware. Then, the influence of data quantization on accelerator design is described, including memory unit and processing unit. Finally, the researches in this field are summarized and the future development directions of this field are prospected. The classification method proposed in this paper is helpful to classify the previous work of DNN accelerator in data quantization.

     

/

返回文章
返回