Study and Realization of Heterogeneous Learning Architecture for Knowledge Distillation based Classification

Abstract

Study and Realization of Heterogeneous Learning Architecture for Knowledge newlineDistillation based Classification newlineHickam s dictum: Deep Neural Networks can have as many parameters and/or newlineconfigurations as the developer may please. newlineThe Deep learning frameworks have progressed beyond human recognition newlinecapabilities, but one faces difficulty to deploy those models in real-time applications. newlineDespite their effectiveness, these models are built upon large scale data sets and newlineusually have parameters in the billion scale. Various compression techniques and newlinealgorithms are developed, yet no comprehensive approach is available to transfer newlinethem to handheld devices. Now, it is the perfect opportunity to either optimize newlinethem or develop architectures for implementation on the embedded platforms. newlineKnowledge distillation provides efficient and effective teacher-student learning newlinefor a variety of different visual recognition tasks because a lightweight student newlinenetwork can be easily trained under the guidance of the high-capacity teacher newlinenetworks. The present deep learning architectures support learning capabilities newlinebut they lack flexibility for applying learned knowledge on the tasks in other newlineunfamiliar domains. A huge gap still exists between developing a fancy algorithm newlinefor a specific task and deploying the algorithm for online/offline production run. newlineThis work tries to fill this gap with the deep neural network based solution for object newlineclassification/detection in unrelated domains with a focus on the reduced footprint newlineof the developed model. The teacher-student architecture is developed with binary newlineclassification problem and knowledge distillation based transfer learning approach newlineshows a 20% improvement in terms of classification accuracy with respect to newlinebaseline teacher model. newlineVarious hardware frameworks have their inherent characteristics which effect newlinethe algorithms efficiency. The CPU based implementation wins in term of latency, newlineix newlinewhile GPU wins in terms of computation. Other types of hardware like custom DSP newlineor SoC are