Bidirectional Motion Estimation for Motion Compensated Frame Rate Up-Conversion

Abstract

Motion compensated frame rate up-conversion has been used for several applications such as film to video conversion, TV standard conversion for different frame rates, and the temporal redundancy reduction for image data in video coding. Recently, it has been used to reduce the motion artifact of hold-type displays such as LCD TVs by increasing the frame rate.This technique consists of three primary components: motion estimation (ME), motion vector smoothing (MVS), and motion-compensated interpolation (MCI). Among all components, ME is the most important because the accuracy of the motion vector estimation most affects the ability of the total system to generate an exact interpolated frame. Several ME algorithms have been proposed, and the block matching algorithm is the most popular, since its hardware and software implementations are simple. However, it has some problems such as overlapped areas and holes when generating an interpolated frame, thereby causing image quality degradation.Bidirectional ME is the most effective ME algorithm. It estimates motion vectors for blocks in the interpolated frame by using the temporal symmetry between previous and current blocks. Therefore, the motion vector is always calculated for the interpolated block and assigned to the block, thereby avoiding the problems of overlapped areas and holes. However, in some cases, bidirectional ME has a problem. For example, if there is spurious temporal symmetry in a patterned (or uni-color) background, or similar (or identical) objects in the previous and current frames, ME can produce a false motion vector.This paper proposes three ME algorithms to solve these problems: dual ME algorithm, extended bilateral ME (EBME) algorithm, and multiframe-based bilateral ME (MBME) algorithm. The dual ME algorithm uses the unidirectional and bidirectional matching ratios of the previous and current blocks, and performs a motion vector validity check to reduce erroneous motion vector estimation. The EBME algorithm utilizes an expanded motion trajectory to enhance the accuracy of motion vector estimation. The MBME algorithm uses additional frames when computing the motion vector to improve motion vector estimation.In the experimental results, the proposed ME algorithms enhance the image quality of the interpolated frame compared to existing ME algorithms. Specifically, in the objective evaluation, the average PSNRs of three proposed ME algorithms were up to 2.272 dB, 1.865 dB, and 1.804 dB higher than those of the existing algorithms, respectively. The average SSIMs of the proposed ME algorithms were up to 0.048, 0.044, and 0.019 higher than those of the conventional ME algorithms. Among the proposed ME algorithms, the dual ME improved the average PSNR by up to 0.620 dB and 1.163 dB and the average SSIM by up to 0.017 and 0.024 compared to the EBME and MBME algorithms, respectively.