Introduction

The evaluation of proposals submitted for Research Development and Grant Submission (RDGS) is a crucial process. This evaluation determines which projects receive funding and ensures the effective allocation of resources to high-impact research. Third-party evaluation plays a vital role in maintaining objectivity, enhancing credibility, and providing an expert assessment of the feasibility, potential, and merit of proposed projects. Evaluating RDGS proposals requires the consideration of multiple factors that influence both the processes and outcomes of the evaluation. Understanding the complexities of third-party evaluation in RDGS proposals is essential for funders, researchers, and evaluators. This article explores five key considerations in navigating this complexity: defining evaluation criteria, managing evaluator bias, aligning the evaluation with the goals of the RDGS, assessing interdisciplinary research, and ensuring transparency and accountability in the evaluation process.

5 Key Considerations

1. Defining Evaluation Criteria

The first critical consideration in third-party evaluation is the establishment of clear and comprehensive evaluation criteria. The criteria set the foundation for the entire evaluation process and provide a systematic approach to assessing the merits of RDGS proposals. Well-defined evaluation criteria ensure that evaluators can objectively measure the quality and relevance of proposals based on consistent benchmarks.^[1] These criteria encompass factors such as the scientific quality of the proposed research, the experience of the research team, the innovation and originality of the research, and the potential impact of the research on the field. In the context of RDGS, these criteria must be adaptable to a wide variety of research areas and funding objectives. A nuanced and domain-specific approach is needed to ensure that evaluators are considering the right factors for each proposal.^[2] Moreover, the alignment of the evaluation criteria with the strategic objectives of the RDGS is critical.^[3] Clear and well-articulated criteria also help to mitigate any potential confusion or inconsistency in the evaluation process. Without precise guidelines, evaluators might assess proposals based on subjective preferences or incomplete information, leading to discrepancies in how proposals are rated.^[4]

2. Managing Evaluator Bias

Another key consideration in third-party evaluation is managing evaluator bias. Bias can emerge in various forms during the evaluation process, and it can significantly influence the outcomes of the evaluation.^[5] Evaluators bring their own experiences, preferences, and assumptions to the table, so it is imperative to recognize and mitigate these biases to ensure fair and objective assessments of RDGS proposals. Research has shown that unconscious bias can affect the way evaluators interpret and score proposals. This bias can undermine the credibility of the evaluation process and result in unfair outcomes.^[6] To mitigate evaluator bias, one approach is to employ multiple evaluators for each proposal, ensuring that different perspectives and expertise are brought to the evaluation. Additionally, it is important for evaluators to undergo training that raises awareness of potential biases and educates them on how to conduct unbiased assessments. This training can help evaluators recognize their own biases and apply the evaluation criteria consistently.^[7] Also, anonymizing the proposals during the review process can also help reduce bias by eliminating any preconceived notions about the applicants based on their reputation or institutional affiliation.^[8]

3. Aligning Evaluation with RDGS Goals

The third key consideration is the alignment of the evaluation with the overarching goals of the RDGS. RDGS funding is often aimed at promoting specific research agendas, such as innovation, collaboration, or addressing pressing societal challenges.^[9] The evaluation process must be designed in a way that reflects and supports these strategic objectives. It is essential for evaluators to understand the broader goals of the RDGS and ensure that their evaluation is aligned with these goals.^[10] To ensure that the evaluation process reflects RDGS goals, it may be helpful for funders to provide evaluators with a clear framework or set of priorities that reflect these goals. This framework can help guide evaluators in their decision-making and ensure that proposals are assessed in the context of the RDGS’s strategic objectives.^[11] Aligning the evaluation with RDGS goals also helps foster greater transparency and accountability in the process.

4. Assessing Interdisciplinary Research

Interdisciplinary research often involves the integration of knowledge, methods, and perspectives from multiple disciplines, making it more difficult to evaluate using traditional, discipline-specific criteria.^[12] It is vital for third-party evaluators to develop appropriate strategies for assessing such proposals. The interdisciplinary nature of the research requires evaluators to have a broader understanding of the various disciplines involved so the evaluation of interdisciplinary research often requires a collaborative approach, where multiple evaluators with expertise in different areas assess various aspects of the proposal. This can ensure that all facets of the research are properly evaluated and that the interdisciplinary nature of the work is fully appreciated.^[13] Evaluators must be able to assess not only the scientific rigor of the proposal but also its potential for real-world application and broader societal relevance.^[14] Therefore, evaluators need to be trained to recognize the unique value of interdisciplinary research and develop appropriate criteria for its assessment.

5. Ensuring Transparency and Accountability

Finally, ensuring transparency and accountability in the evaluation process is crucial. Transparency involves making the evaluation process clear and accessible to all stakeholders, including applicants, funders, and the general public.^[15] Accountability ensures that the evaluation process is conducted with integrity, fairness, and in accordance with the stated criteria and goals. Transparency can be achieved by providing clear guidelines on the evaluation process, including how proposals will be assessed, the criteria that will be used, and the roles and responsibilities of evaluators. Additionally, evaluators should provide detailed feedback to applicants, explaining the reasons behind their decisions and how the proposals were rated according to the established criteria.^[16] This feedback can help applicants improve their proposals for future submissions and foster a greater sense of fairness in the evaluation process. Evaluators must be held to high ethical standards, and any potential conflicts of interest should be disclosed and managed.^[17] 

Conclusion

Third-party evaluation of RDGS proposals is a complex and multifaceted process that requires careful consideration of several factors. Defining clear and comprehensive evaluation criteria, managing evaluator bias, aligning the evaluation with RDGS goals, assessing interdisciplinary research, and ensuring transparency and accountability are all critical to ensuring a fair and effective evaluation process.

Take Away

This article provides 5 key considerations for third-party evaluation for RDGS proposals. By addressing these key considerations, funders and evaluators can help to navigate the complexity of RDGS evaluations and ensure that resources are allocated to research projects with the highest potential for impact and success.

[1]Patton, M. Q. (2008). Utilization-focused evaluation. SAGE.

[2] Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. Sage publications.

[3] Poh, K. L., Ang, B. W., & Bai, F. (2001). A comparative analysis of R&D project evaluation methods. r&D Management, 31(1), 63-75.

[4] Chen, H. T., & Rossi, P. H. (1987). The theory-driven approach to validity. Evaluation and program planning, 10(1), 95-103.

[5] Tversky, A., & Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157), 1124-1131.

[6] Reich, D. A., Green, M. C., Brock, T. C., & Tetlock, P. E. (2007). Biases in research evaluation: Inflated assessment, oversight, or error-type weighting?. Journal of Experimental Social Psychology, 43(4), 633-640.

[7] Kaplan, B., & Shaw, N. T. (2004). Future directions in evaluation research: people, organizational, and social issues. Methods of information in medicine, 43(03), 215-231.

[8] Chen, H. T., & Rossi, P. H. (1987). The theory-driven approach to validity. Evaluation and program planning, 10(1), 95-103.

[9] Guyadeen, D., & Seasons, M. (2018). Evaluation theory and practice: Comparing program evaluation and evaluation in planning. Journal of Planning Education and Research, 38(1), 98-110.

[10] Vinkenburg, C. J., Ossenkop, C., & Schiffbaenker, H. (2022). Selling science: optimizing the research funding evaluation and decision process. Equality, Diversity and Inclusion: An International Journal, 41(9), 1-14.

[11] Poh, K. L., Ang, B. W., & Bai, F. (2001). A comparative analysis of R&D project evaluation methods. r&D Management, 31(1), 63-75.

[12] Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. Sage publications.

[13] Chen, H. T., & Rossi, P. H. (1987). The theory-driven approach to validity. Evaluation and program planning, 10(1), 95-103.

[14] Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. Sage publications.

[15] Kaplan, B., & Shaw, N. T. (2004). Future directions in evaluation research: people, organizational, and social issues. Methods of information in medicine, 43(03), 215-231.

[16] Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. Sage publications.

[17] Chen, H. T., & Rossi, P. H. (1987). The theory-driven approach to validity. Evaluation and program planning, 10(1), 95-103.