Algorithm-hardware co-optimization for cost-efficient ML-based ISP accelerator

Abstract

We present an advanced algorithm-hardware co-optimization method to design an efficient accelerator architecture for image signal processing with deep neural networks. Based on the systolic-array structure, we introduce two evaluation metrics for performing the target network model, each of which is dedicated to fairly representing either the processing speed or the energy consumption. Several array scaling methods are presented to find the most cost-efficient array structure from the initial array, which showed the best score of overall metric with the given number of multipliers. In addition, the original ML model is adjusted to further increase the overall efficiency with subtle quality drops of image outputs. Implementation results in 28-nm CMOS technology show that the proposed co-optimization method successfully finds the cost-efficient accelerator architecture for ISP applications, improving the energy efficiency by 51% compared to the straightforward array design.