Design of a Quantitative Evaluation Methodology for Government-sponsored R&D Programs using a Performance-Effectiveness-Efficiency Matrix

Abstract

This study proposes an evaluation methodology for government-sponsored R&D programs to manage and operate them effectively through objective and quantitative analysis. The proposed methodology considers three aspects: performance, effectiveness, and efficiency of R&D programs. First, the performance evaluation is defined as the examination of various performance indicators in the results achieved by R&D programs. In this evaluation, weighted-sum model is employed. This model considers the characteristics of three research types (basic, applied, development) and three performance criteria (science and technology, economy, society) with weights. Normalization is also applied to adjust different units of evaluation indicators by using z-scores in this model. Second, effectiveness evaluation measures an extent to what a program has achieved its objectives. In this evaluation, portfolio model is employed. This model compares R&D programs' targets which are set before starting R&D activities and the outputs which are really performed in R&D programs. Third, efficiency evaluation implies a measurement of how efficiently a system uses R&D programs' resources to generate outputs. In this evaluation, The BCC model in DEA is employed because R&D activity shows not only constant returns to scale but also varible returns to scale. AR model is also employed because the weight flexibility in DEA can produce unrealistic efficiency scores. Then a Performance-Effectiveness-Efficiency Matrix (PEE-Matrix) is developed based on the results of three evaluations to diagnose states and suggest guidelines of R&D programs. The PEE matrix can compare and analyze every program in terms of the score and rank.In order to apply the proposed methodology in practice, case study was performed in 16 R&D programs supported by a government foundation in Korea. And to validate the effectiveness of the proposed methodology, an expert interview was carried out targeting nine program directors and two managers of the Frontier support center. The survey result showed that the proposed methodology is useful to evaluate and manage government-sponsored R&D programs. In last, this study provided a balanced view of R&D evaluation to analyze quantitatively the performance of government-sponsored R&D programs using the PEE matrix.