Following an extensive review by a task force to determine if the merit review criteria used by the National Science Foundation (NSF) to evaluate all proposals remain appropriate, the National Science Board (NSB) yesterday released its report — National Science Foundation’s Merit Review Criteria: Review and Revisions — recommending that NSF “better define the two criteria for the benefit of the science community.”

According to the press release announcing the report:

The NSB task force was put in place in February 2010, with a charge to examine the two merit review criteria and their effectiveness in achieving the goals for NSF research support of science, engineering and education. Based on the task force’s analyses, the NSB concluded that the two current merit review criteria of intellectual merit and broader impacts remain appropriate for evaluating NSF proposals, though with revisions. The revisions to the criteria are described in the report…

 

In addition [to the recommendation of better defining the two criteria], the report contains three principles governing NSF’s approach to utilizing these criteria and guidance addressing several issues associated with their implementation…

From the executive summary of the report (after the jump):

In February 2010, the National Science Board (NSB, Board) established a Task Force on Merit Review, and charged it to review how well the current Merit Review Criteria used by the National Science Foundation (NSF) to evaluate all proposals were serving the agency. The two Criteria had been in place since 1997 with only one significant modification in 2007 (to include mention of potentially transformative concepts). The Task Force conducted a thorough review of data collected from multiple sources, which included extensive outreach to many stakeholder groups.

 

Based on the Task Force’s analyses, NSB concluded that the two current Merit Review Criteria of Intellectual Merit and Broader Impacts remain appropriate for evaluating NSF proposals (the Board also recognized that the America COMPETES Reauthorization Act of 2010 included a provision mandating the retention of the Broader Impacts criterion). However, the Board concluded that revisions were needed; both to draw a clearer connection of the Criteria to core principles and to better articulate the essential elements of each criterion…

 

Merit Review Principles

 

Given that the NSF is the primary federal agency charged with nurturing and supporting excellence in basic research and education, the following three principles apply:

 

• All NSF projects should be of the highest quality and have the potential to advance, if not transform, the frontiers of knowledge.
• NSF projects, in the aggregate, should contribute more broadly to achieving societal goals. These “Broader Impacts” may be accomplished through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by, but are complementary to, the project.
• Assessment and evaluation of NSF funded projects should be based on appropriate metrics, keeping in mind the likely correlation between the effect of broader impacts and the resources provided to implement projects. If the size of the activity is limited, evaluation of that activity in isolation is not likely to be meaningful. Thus, assessing the effectiveness of these activities may best be done at a higher more aggregated level than the individual project.

 

Merit Review Criteria

 

When evaluating NSF proposals, reviewers should consider what the proposers want to do, why they want to do it, how they plan to do it, how they will know if they succeed, and what benefits would accrue if the project is successful. These issues apply both to the technical aspects of the proposal and the way in which the project may make broader contributions. To that end, reviewers are asked to evaluate all proposals against two criteria:

 

• Intellectual Merit: The intellectual Merit criterion encompasses the potential to advance knowledge; and
• Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.

The following elements should be considered in the review for both criteria:

 

1. What is the potential for the proposed activity to (a) advance knowledge and understanding within its own field or across different fields (Intellectual Merit); and (b) benefit society or advance desired societal outcomes (Broader Impacts)?
2. To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?
3. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success?
4. How well qualified is the individual, team, or institution to conduct the proposed activities?
5. Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities?

According to the press release, “NSF plans to develop a detailed implementation plan that will lead to the inclusion of the revised criteria in the next version of NSF’s Proposal and Award Policies and Procedures Guide and the Proposal & Award Manual.”

For more on yesterday’s report, check it out here.

(Contributed by Erwin Gianchandani, CCC Director)