OMB Preliminary Investment Criteria for Basic Research
Background
Predicting the outcome of worthwhile basic research should not be easy. Serendipitous results are often the most interesting and ultimately may have the most value. Taking risks and working towards difficult-to-attain goals are important aspects of good research management.
However, there is no inherent conflict between the difficulty of predicting the success of basic research programs and the call for R&D investment criteria and budget-performance integration in the President's Management Agenda. Bringing clearer information about program performance to bear upon resource allocation decisions lies at the heart of these initiatives. The Administration will focus on improving the management of basic research programs, not on predicting the unpredictable. Reinforcing good management practices and the adoption of best practices by all basic research programs across the federal government is the goal. Not all programs will meet all the criteria initially, but we expect that over time they will.
COSEPUP previously recommended Quality, Relevance, and Leadership as appropriate metrics for basic research programs.
The Army Research Laboratory (ARL) selected Quality, Relevance, and Productivity as the relevant metrics for both basic and applied research programs.
Principles behind proposed OMB investment criteria for basic research programs
OMB proposes using the guidelines: Quality, Relevance and Performance, combining both the COSEPUP and ARL models. OMB retains COSEPUP's Leadership concept, but as a potential indicator to demonstrate Quality and not as an independent goal. Adapting the ARL Productivity metric provides a means of coupling investment criteria for basic research programs to the President's Management Agenda initiative for budget and performance integration.
The criteria include clear distinctions between prospective assessment and retrospective evaluation. Research agencies fund a mix of contracts, grants, and in-house activities, which means that program management often entails placing "bets," monitoring contractors, and managing internal research activities. It is tremendously important that basic research programs are able to demonstrate responsible management of their inputs, in addition to clearly articulating and demonstrating progress towards expected outputs. Yet, outcomes still matter. Retrospective review of whether investments were productive is essential for validating program design and instilling confidence that future investments will be wisely invested as well. Retrospective reviews should address both technical excellence and the relevance of program outputs to others. In practice, Quality, Relevance, and Performance are more readily demonstrated separately for prospective information but are highly interrelated in the retrospective analysis.
The following pages provide preliminary criteria - with brief discussions - to address Quality, Relevance, and Performance.
Next Steps
Draft Criteria for Basic Research
All basic research programs must meet all of the following criteria. (Appropriate levels of applicability remain to be determined.)
1. Quality
B. Retrospective Expert Review of Program Quality
Programs should maximize quality of the research they fund and periodically examine whether their portfolio of projects produces scientific and technical excellence.
Programs must use a clearly stated, defensible method for awarding a significant majority of their research grants and contracts. The customary method for promoting research quality is the use of a competitive, merit-based process. NSF's process for the peer-reviewed, competitive award of its research grants is a good example. Exceptions must be well justified, but they may include concerns for timeliness (e.g., research grants for rapid response studies of Pfisteria) or a proven record of outstanding performance (e.g., performance-based renewals).
Programs must assess and report on the quality of current and past research. For example, NSF's Committees of Visitors, which reviews NSF directorates, are one implementation of these reviews. Benchmarking programs in one agency against other federal programs is encouraged, as is international benchmarking, which provides a measure of leadership in a field of research. Leadership is the result of a well-defined, high-quality, well-managed program, one able to make identifiable contributions to a field within the confines of available resources. Not delivering world-class performance within existing resources is indicative of poor program execution, timidity, or an overly broad portfolio.
A. Definition of Program Direction and Relevance
B. Retrospective Outcome Review to Assess Program Design and Relevance
Research programs need to set a general direction for their investments. These plans must identify and prioritize research opportunities within the program. The Joint DOE/NSF Nuclear Sciences Advisory Committee's Long Range Plan and the Astronomy Decadal Surveys are the products of good planning processes. Workshops may be an acceptable planning mechanism for "small science" programs.
Program objectives and goals should be assessed by their relevance to agency missions, national needs, and the field(s) of study the program intends to address. Review committees should provide an answer to the question: "Does the agency's research address subjects in which new understanding could be important in fulfilling the agency's mission?"
An example of a responsibility that goes beyond agency bounds is the operation of scientific facilities for the use and benefit of the entire research community. Committees will address whether programs are fulfilling these responsibilities appropriately or whether other stakeholders should bear those burdens.
(Metrics and assessments should be reflected in the agency's annual performance plan.)
A. Prospective Assessment of Program Inputs and Output Performance Measures
B. Demonstration of Performance
Programs must demonstrate attentiveness to the health of their research enterprise and an ability to manage their programs in a manner that produces identifiable results.
Input statistics help demonstrate to oversight bodies that agency heads are managing the inputs of their research enterprise. The range of these statistics is highly variable and should be tailored to address issues of concern to agency management and OMB.
Agencies should establish a set of high priority, multi-year research objectives with annual performance outputs that show how one or more outcomes will be reached. ARL's "Top 5 Deliverables" metric serves as a good model. Agencies and their OMB representatives will negotiate the appropriate level of applicability and frequency for each bureau. At each level negotiated, programs must develop and report on semi-quantitative metrics for that level's three to five highest priorities. These metrics shall be included in the budget request as GPRA performance metrics.
Construction projects and facility operations will require additional performance metrics. Cost and schedule earned value metrics for the construction of R&D facilities must be tracked and reported. Formalized independent reviews for DOE's Office of Science of technical cost, scope, and schedule baselines and project management of construction projects ("Lehman Reviews") are a widely recognized "best practice" for discovering and correcting problems involved with complex, one-of-a-kind construction projects.
Any set of specific output milestones is unlikely to cover 100 percent of a program's research portfolio, nor should it. OMB will assume that basic research programs reporting on the results of specific output milestones as developed above combined with reporting the results retrospective portfolio reviews will have satisfied the requirement of GPRA.
- Quality is most often assessed using mechanisms of peer review, a method commonly applied throughout the scientific community to the work of laboratories and individuals.
The latter two metrics operate at the portfolio level and can be assessed usefully every 3-5 years.
- Relevance is assessed by panels of expert peers, experts in related fields, and potential users of research results to answer the question, "Does the agency's research address subjects in which new understanding could be important in fulfilling the agency's mission?"
- Leadership is proposed as a potentially effective evaluation criterion to test whether research is being performed at the forefront of scientific and technologic knowledge on an international level.
- Quality is interpreted to assess whether the work is being done at a level that could be called at or beyond the state of the art.
- Relevance addresses whether the work being performed responds to some bona fide requirement of a customer. In other words, does anyone care about the work being done?
- Productivity is a measure of the progress of a given project or lab as a whole towards some specified goal. Is ARL delivering some product the customer can use in a timely fashion?
Like COSEPUP, the ARL approach uses portfolio level metrics, which can be assessed usefully every 3-5 years, along with annual reporting of progress on high priority technical objectives. ARL uses a semi-quantitative approach for reporting on progress towards meeting annual technical objectives.1 - Discuss criteria with stakeholders in government, academia, and industry.
- Validate general criteria against relevant programs at specific agencies.
- Develop larger strategy and framework for an integrated approach assessing basic and applied programs across the agencies.
- Determine how, where, when, and at what levels the criteria will be applied.
- Work with agencies to determine how best to implement criteria at each agency.
- Provide more detailed guidance to agencies to help them develop more meaningful performance metrics.A. Prospective Review of Awards
- Clearly define how much of the requested funding will be directed to specific research performers, open to a limited subset of research performers through merit-reviewed competitions, or open to all potential research performers through merit-reviewed competitions (following definitions in OMB Circular A-11, Section 84).
- Provide a compelling justification for research funding that is to be directed to specific performers or open to a limited subset of research performers.
- Clearly define a plan for regular, external reviews of the quality of the program's research and research performers. Explain how the results from these reviews will be used to guide future program decisions. Rolling reviews performed every 3-5 years by advisory committees can satisfy this requirement. Benchmarking is an effective means of assessing program quality relative to other programs, other agencies, and other countries.
- Provide a clear response to prior external reviews, including whether and how the program has responded or will respond to recommendations from those reviews.Discussion
2. Relevance
- Set and provide details on the direction of the program. Plans must identify and prioritize research goals within the program. Even programs that fund unsolicited proposals for investigator-initiated research should be able to clearly articulate what new knowledge, understanding, technology, or tools might result from the investment.
- Demonstrate linkages between planned program activities and relevant goals in national research plans, agency strategic plans, and appropriate external reviews, such as those conducted by the National Academies. Programs should also document any linkages to national priorities identified by the President. Define how program investments will yield outputs that will help researchers address these goals and priorities.
- Provide details on any shifts in program design from prior reports.
- For capital projects, include schedules with milestones for major future competitions, decisions, and termination points. Highlight any changes from previous schedules.
- Provide a clear plan for external reviews of the program's relevance to future research, including how results from these reviews will be used to guide future program decisions. Committees should be composed of peers, experts in related fields, and potential users of research results. Retrospective reviews should be conducted on a 3-5 year, rolling basis.
- If research programs have roles and responsibilities beyond the confines of their parent agencies, direct external committees to identify, provide justification for, and assess program success in fulfilling these responsibilities. Committees may also identify roles that would be better executed by academia, industry, or other agencies. Discussion
3. Performance
- Report annually on relevant program inputs, including overhead rates, intramural versus extramural spending, and human capital statistics.
- For programs that include significant capital investment, provide data on development costs and potential costs associated with the maintenance and repair of existing infrastructure. Where possible, programmatic risk on individual capital projects should be quantified, compared to prior projects, and appropriately managed within a program's portfolio of investments. Include any discrepancies from prior estimates.
- Provide multi-year research objectives with annual performance outputs that show how the program will improve the state-of-knowledge, state-of-technology, scientific understanding, or research capabilities towards the program's goals over time. Agencies may define new research objectives as a program progresses, but not without showing a program's past progress against prior objectives (see criterion 3B). Metrics must be representative of the program's major goals.
- If a proposal includes significant capital projects, provide a clear plan for how progress against cost estimates will be assessed and incorporated in future program decisions.
- For operational facilities, define appropriate metrics for the dependability, effectiveness, and use of those facilities over time.
- Document performance against previously defined output metrics, including any success in reaching one or more multi-year objectives.
- If a proposal includes significant capital projects, provide external, independent cost and schedule estimates. Highlight any discrepancies from previously reported estimates, especially if program has failed to meet any cost or schedule goals.
- For operational facilities, report on metrics for dependability, effectiveness, and use.Discussion
1Objectives are evaluated as to whether they are met (1 point), not met (0 points), or demonstrated progress short of completion (0.5 point). The 80% goal for ARL's "Top 5 Deliverables" metric permits the use of stretch goals.