Foundational Work
Hunter R. Rawlings III became the president of AAU in June 2011. At his first meeting with staff, he expressed a strong desire for AAU to undertake an initiative to improve STEM education at the undergraduate level at AAU universities. This desire meshed nicely with activities already underway and being actively pursued by Toby Smith, AAU’s Vice President for Policy under the leadership of Bob Berdahl, the previous AAU president.
These activities included ongoing conversations with Carl Wieman, a Nobel Prize-winning physicist with a passion for improving undergraduate STEM education, who had recently been appointed Associate Director for Science at the White House Office of Science and Technology Policy (OSTP); with Linda Slakey, Director of the Division of Undergraduate Education in the Education and Human Resources (EHR) Directorate of the National Science Foundation (NSF); and with the President’s Council of Advisors on Science and Technology (PCAST) who had created a working group and were in the midst of conducting their own report on the need to improve undergraduate STEM education.
Moreover, Toby had previously conducted a survey of AAU member universities asking them to highlight noteworthy programs in undergraduate STEM education. Results of the survey indicated that most efforts occurred in out-of-classroom or co-curricular activities (e.g., undergraduate research opportunities, bridge programs, living-learning communities, etc.), which, although important, do not address the within-class experiences and the effectiveness of faculty members’ instructional practice and student engagement.
During the summer of 2011, Josh Trapani, AAU’s Associate Vice President of Research and Policy Analysis, prepared a “discussion draft” in the form of a white paper of a proposal for a five-year initiative to improve undergraduate STEM Education. The discussion draft laid out the problem, highlighting the fact that “improving undergraduate teaching is integral to meeting the pressing national need for more STEM majors.”
The white paper pointed to previous work showing that many students interested in STEM switched to other majors during the first two years of college, and that teaching was one of the main causes for this shift. Cultural factors at research universities worked against instructors incorporating more active learning pedagogy in their classes, even though the evidence base supporting active learning pedagogy held appeal for researchers who were also teachers. The discussion draft laid out five goals, which have continued to guide the Initiative through its subsequent activity.
From the beginning, AAU staff recognized that such an ambitious initiative would not succeed if it were simply an add-on to existing AAU activities. Staff lacked both expertise and bandwidth to perform the work, and the development of a demonstration program required an external funding source.
Coincident with producing this discussion draft, AAU staff began to identify individuals who might serve on an advisory committee for the Initiative and help provide expert guidance. The initial membership of the advisory committee included practitioners and leaders in aspects of work related to the goals of the Initiative.
The Initiative was announced publicly in September 2011, and the advisory committee held its first meeting, by phone, in October. Many of the individuals on the advisory committee provided critiques of the discussion draft.
Kristen Hodge-Clark, a AAAS Fellow at AAU, gathered data to map major association and disciplinary society efforts in STEM reform and to identify areas of overlap among various organizations. As a part of this effort, AAU developed a matrix of STEM undergraduate education reform efforts at research institutions.
Additionally, AAU drew on the expertise of federal officials and colleagues at other organizations, including associations, disciplinary societies, and funders, in honing the plan for the Initiative. In particular, staff coordinated with two advisory committee members, S. James Gates and Jo Handelsman, who at the time were also serving as co-chairs of the PCAST working group on issues around improving undergraduate STEM education.
PCAST’s report “Engage to Excel” was released in February 2012 and provided a national-level vision that was consistent with the goals of the AAU Initiative, especially in its focus on the importance of reforming teaching. The advisory committee held its first in-person meeting, also in February. In May, the National Academies released its report on discipline-based education research, which further supported the emphasis on active pedagogy in the Initiative.
AAU staff began compiling information that was envisioned as useful to developing a framework for institutions to assess and improve the quality of STEM teaching and learning. In placing framework development as the first goal, staff benefited from the example and experience of the Association of Public and Land-grant Universities’ (APLU) Science & Mathematics Teacher Imperative (SMTI).
The framework was initially envisioned as a lengthy, detailed document and set of resources. The first rough outline of what would be included in a framework was produced in November, 2011. This version received feedback from members of the advisory committee and benefited from the information contained in the PCAST report. It was further revised during the spring and summer, and sent out to AAU member campuses for feedback in October, 2012 [see Online Appendix for request for feedback on the Framework].
During this time-period, the AAU Initiative continued to gain momentum. In April 2012, AAU member university presidents and chancellors were asked at their spring meeting to provide a campus primary point of contact for the Initiative. This person was to be someone to whom the presidents and chancellors would rely on and engage with regarding the AAU Initiative.
During that spring, AAU had conversations with numerous potential funders of the demonstration program. The Research Corporation for Science Advancement (RCSA) sponsored a workshop for funders interested in learning more about the AAU Initiative and other initiatives focused on improving undergraduate STEM education. This workshop was held at AAU in May, and attendees included The Leona M. and Harry B. Helmsley Charitable Trust, who were beginning to move into the space of higher education.
Over the summer, AAU staff, along with two members of the advisory committee – Linda Slakey and James Fairweather, who served as a co-PI – produced a proposal for The Helmsley Charitable Trust to conduct a demonstration project on AAU campuses. The same group also developed a proposal focused on metrics and evaluation for NSF. Ultimately, both proposals were successful. AAU announced the Helmsley Charitable Trust grant in October, 2012, and the NSF WIDER grant in May, 2013.
Implementation
In fall of 2012, with The Helmsley Charitable Trust funding secured, AAU hired a full-time project manager for the Initiative. Emily Miller joined AAU in November. Collectively, Toby Smith, Jim Fairweather, Linda Slakey, Josh Trapani, and Emily Miller became the core AAU team responsible for the implementation of the AAU Undergraduate STEM Education Initiative and the corresponding goals. This group has met bi-weekly for a one-hour teleconference, and at least twice a year for a one or two-day working session in Washington, DC, since fall of 2012.
The initial draft Framework was refined based on feedback provided by 42 different AAU member campuses. In response to this feedback, the Framework was transformed from a detailed and prescriptive set of instructions to a concise conceptual document. The final Framework provides a set of key institutional elements that need to be addressed to bring about sustainable change.
From a multi-institutional perspective, the Framework provides a shared vision for change, one that includes a common understanding of the challenges and an agreed upon set of institutional elements that must be addressed to bring about sustainable change. Along with the five goals in the discussion draft, the Framework became central to subsequent work.
Collectively the project team developed and implemented a process for selecting project sites funded by The Helmsley Charitable Trust. From the beginning, AAU’s approach to the demonstration project involved a balance: taking advantage of the competitive nature of AAU’s member universities with one another without creating a two-tiered system that might exclude some institutions to the extent of discouraging them from acting.
The project sites were envisioned as one part of a larger STEM Network that included all interested AAU member universities. In February 2013, AAU put out a call for concept papers from schools interested in being part of the demonstration project [see Online Appendix for project site selection process materials]. Thirty-one AAU universities submitted concept papers, which were reviewed using a rubric and narrowed to 11 institutions, who were asked for more detailed plans of work. From these 11, eight project sites were selected in June 2013.
The eight project sites–Brown University; Michigan State University; The University of Arizona; University of California, Davis; University of Colorado Boulder; University of North Carolina at Chapel Hill; University of Pennsylvania; and Washington University in St. Louis–served as laboratories to implement the key elements of the Framework and represented the first phase of encouraging AAU universities to take a systems approach to reform of undergraduate teaching practices. Each of the eight project sites received $500,000 to seed fund projects (Year 1: $250K, Years 2-3: $125K).
It is important to note that these eight universities were not selected because they were the most advanced in terms of activity already focused on improving undergraduate STEM education. AAU made a deliberate decision to create balance among the schools not only in terms of public/private status, region, size, and other factors, but also in terms of proposal objectives and how developed existing STEM teaching reform efforts on campus were. The hope was that the eight project sites could serve as potential models for other institutions, no matter where these others were in terms of emphasis and activity.
AAU’s Initiative also recognized that large-scale improvements require a network through which ideas can travel, be tested, modified, and improved in a continuous cycle of growth. Research by Fairweather and by Eckel and Kezar strongly suggests that evidence alone is not sufficient for sustainable reform. Rather, peer relationships and institutional as well as interpersonal networks are crucial factors in changing ideas and practices.
Thus, AAU also built into its approach the AAU STEM Network – a collaborative network that would allow AAU institutions that were not project sites to participate in the Initiative. It became clear as the process moved forward that several institutions – both project sites and non-project sites – were serious about advancing educational reforms and interested in learning from other AAU institutions that were tackling similar challenges on their own campuses.
AAU has hosted three STEM Network conferences — in 2013, 2014, and 2015 — with attendees including administrators, faculty members, postdocs, and students from AAU universities. These conferences have given attendees opportunities to showcase their work and learn about the work of others, discuss common themes and challenges, and build relationships across campus roles and institutions.
AAU intentionally built a multi-institutional network aimed at improving the quality and effectiveness of undergraduate STEM education in the nation’s top research universities.
AAU intentionally built a multi-institutional network aimed at improving the quality and effectiveness of undergraduate STEM education in the nation’s top research universities.
Additionally, AAU frequently hosts workshops to provide in-person forums for all AAU institutions to engage in the Initiative, and AAU brings together on occasion key stakeholders to address specific challenges to implementing institutional efforts to reform STEM teaching and learning for undergraduate students. For example, AAU hosted a two-day workshop in April 2015 for STEM department chairs and faculty members, “Improving Undergraduate STEM Teaching & Learning: The Role of the Department Chair.” This workshop, which was sponsored by Elsevier, provided over 100 department chairs, faculty members, and university administrators with an opportunity to discuss introductory curriculum redesign efforts, staffing models for introductory STEM courses, evaluation of department innovations in teaching and learning, and metrics for rewarding teaching.
To date, 55 (out of 62) AAU member universities have participated in undergraduate reform activities hosted by AAU with involvement from more than 275 faculty members and institutional leaders. In addition, AAU recently awarded 12 mini-grants to universities who are engaged in the AAU STEM Network but not part of the original eight project sites.
AAU has learned through its STEM Initiative that grants awarded by AAU to member institutions can have powerful symbolic implications that can help campuses facilitate change.
Toby Smith was invited to speak about the AAU Initiative during the 2012 RCSA Cottrell Scholars conference. At the conference, Toby discussed with current and former Cottrell Scholars a major barrier to improving the quality of undergraduate education: the predominant use of student-based evaluations to assess teaching quality at colleges and universities. While effective at assessing faculty popularity, these student evaluations often fail to accurately reflect teaching quality and student learning.
The two groups proposed a joint project, which RCSA subsequently agreed to support, aimed at identifying new and innovative means to evaluate and reward teaching quality and effectiveness.
To help to accomplish the goals of this joint project, AAU and a subset of scholars from the RCSA Cottrell Scholars Collaborative held a joint workshop in January 2014 that brought together leading research-active faculty members as well as higher education scholars and practitioners to understand the landscape of established and emergent means to reward teaching more accurately than traditional measures such as student evaluations and to assess how research universities do or do not reward teaching in promotion and tenure decisions. The workshop culminated in a workshop report, Searching for Better Approaches: Effective Evaluation of Teaching and Learning in STEM and an article in Nature titled, University Learning: Improve Undergraduate Science Education.
The Cottrell Scholars and AAU were awarded a second collaborative project by RCSA and in May 2016 brought together leading higher education scholars and practitioners as well as research-active faculty members to develop specific recommendations and guidance to value, assess, and reward effective teaching.
In addition to AAU’s efforts to try to find ways to better assess and evaluate teaching, AAU has also worked to better leverage existing NSF broader impacts requirements to improve the quality of undergraduate STEM education and to achieve meaningful and long-lasting cultural change.
Currently, all NSF grant proposals are evaluated on two broad-based criteria: their intellectual merit, which encompasses the potential to advance knowledge, and their broader impacts. The broader impacts criterion
encompasses the potential of the work being done associated with the grant to benefit society and contribute to the achievement of specific, desired societal outcomes.
As a part of 2010 legislation reauthorizing the America COMPETES Act, Congress included eight specific goals for NSF broader impacts criteria. Two of these goals, “improving undergraduate education” and “increasing public scientific literacy,” directly relate to the AAU Initiative, and can be used as a tool to encourage faculty to improve the quality of their undergraduate STEM teaching.
There are, however, many challenges in getting faculty members to utilize improving their classroom teaching practices to fulfill their NSF broader impacts requirements. These challenges include the fact that many NSF researchers are unfamiliar with the broader impacts criteria and unaware that improving the quality of their classroom teaching, if done correctly, can count. Still others are skeptical that NSF review panels will view improving how they provide in-classroom instruction in the discipline or field of their NSF funded research as broader impacts.
This concern has some legitimacy because the NSF itself is not very prescriptive in telling review panels how to interpret and assess broader impacts. Additionally, NSF review panels are not always well instructed by their program officers on what counts and how to effectively evaluate broader impacts. Indeed, some faculty members who have included improving in-class teaching of undergraduate students as a broader impact have received negative feedback from NSF reviewers who have specifically suggested in their comments that since ‘teaching’ is a required part of their faculty work, activities that help to improve instruction should not be included or counted as broader impacts.
To help to address this concern, AAU successfully added clarifying language to the American Innovation and Competitiveness Act (AICA). This legislation, approved by the Congress in late December 2016 and signed into law by the President January 6, 2017, updated the goals for NSF’s broader impacts criteria. The new language clearly states that ‘improved undergraduate education and instruction’ are means by which NSF researchers can achieve a broader impact.
Building on this change in the legislation, AAU is working though the National Alliance for Broader Impacts (NABI), the NSF, other scientific organizations and its member institutions to deliver a clear message that the adoption and usage by an NSF awardee of evidence-based and/or active and engaged teaching practices proven to enhance undergraduate learning and understanding of core STEM concepts in disciplines relating on the principal investigator’s NSF funded research award should be recognized by NSF review panels as an acceptable form of meeting NSF broader impacts criteria.
Evaluation
Evaluation is a key component of the Initiative. AAU is assisting member universities in tracking the progress of their reform efforts in addition to evaluating the overall impact of the Initiative. In this process, AAU has distinguished between measures that are most meaningful at the department level and those most useful in documenting cross-institutional effects.
To support local assessment, AAU has developed a resource guide, Essential Questions and Data Sources for Continuous Improvement of Undergraduate STEM Teaching and Learning. The guide provides a set of key questions designed to engage institutional leaders and faculty members in discussions about teaching and learning. The guide also profiles data sources and analytical tools available to answer these questions and inform decision-making. It further provides guidance to address shared challenges in evaluating the quality and effectiveness of undergraduate education.
The Essential Questions and Data Sources guide is a complementary resource to the Framework for Systemic Change in Undergraduate STEM Teaching and Learning.
To document cross-institutional effects, AAU collected data from all project sites over a three-and-a-half-year period, beginning in the Fall 2013. Common data collection included a survey of instructors in participating departments; department chair narratives on policy and practice to assess teaching in the promotion and tenure process; and campus and department level assessment of learning spaces [see Online Appendix for project site common data collection materials].
To begin developing these cross-institutional quantitative measures, AAU convened a working group of experts on metrics and evaluation in July 2013. Following this meeting, AAU project staff developed a set of research questions mapped to the AAU Framework. AAU decided to collect information on physical infrastructure (using a portion of the PULSE Vision & Change Rubric) and to ask for written descriptions of the role of teaching in promotion and tenure by project leads and department chairs.
To assess instructor attitudes and practices, AAU project staff assembled an instrument from existing tools that would be used to survey instructors. Through an iterative conversation with individuals at project sites, AAU arrived at a final instrument, as well as a collective understanding of how AAU would use the data. Results were obtained at two points in time – early 2014 (for the 2013-14 year) and in fall of 2016 (for the 2015-16 year).
Beyond this baseline data request, AAU asked project sites to provide additional information in annual reports. Site-specific data included data on student learning outcomes. AAU asked each project site to provide evidence for enhanced student learning, but did not require the same indicators or metrics to be collected in the same way across the sites [see Online Appendix for project site annual and final report requests].
Integrated with the collection of baseline measures and annual reports, AAU conducted two site visits at each of the eight project sites to allow for a more qualitative evaluation of project implementation and progress [see Online Appendix for project site interview protocols]. In total, AAU met and talked with 325 individuals across the eight project sites. Teams met with campus project leaders, department chairs and deans, and Provosts, using these visits to identify challenges and possible solutions to implementing project activities, and to look at subsequent changes.
Site visits also built trust between AAU and project sites. While designed for project site campuses, the components of this evaluation are useful for any institution interested in assessing its progress.
In September 2014, AAU announced it had received a second grant from NSF, this one to examine the role a national association can play in expanding reform efforts aimed at improving the quality of undergraduate teaching and learning in STEM fields at its member institutions. This project is being conducted in partnership with Adrianna Kezar, Professor, Rossier School of Education and Co-Director, Pullias Center for Higher Education at the University of Southern California.
Sustaining the Momentum of the Initiative
AAU found that certain steps taken prior to supporting specific plans for implementation were essential to the Initiative’s success. These included developing a shared understanding of the Initiative’s goals; collectively developing the Framework; agreeing that multiple-strategies are possible to achieve the Initiative’s goals; determining how to assess the Initiative’s overall impact; and aligning this effort within the national landscape.
Ultimately the foundational work required to generate buy-in and engagement from member campuses in an initiative of this magnitude is critical. Member universities need the opportunity for teams to participate in the development of a shared vision and framework to ensure that each campus can see their local context within the conceptual change model. Time is also critical to build the trust necessary for campuses to participate in a collective effort, develop a functioning network, and agree to a common data collection effort aimed at measuring aggregate impact.
AAU has successfully established an infrastructure to help align the AAU membership toward a common goal. Each AAU member campus has designated at least one individual as a campus point of contact for the Initiative who acts as a liaison between their campus and AAU on the project. To support the Initiative’s activities and goals, the dedicated project director situated at AAU engages in continuous dialogue with AAU member campuses along with senior university administrators and designated campus liaisons to coordinate and manage the project. The project director also plays an essential role in writing grants to secure funding for the various activities in support of the Initiative’s goals.
Andrea Follmer Greenhoot, Professor of Psychology, Director of the Center for Teaching Excellence and Gautt Teaching Scholar at the University of Kansas, facilitating a knowledge exchange among faculty members around evidence-based educational improvement at a TRESTLE meeting.
AAU also continuously brings to the attention of university leadership (President or Chancellor, and Provost) results from Initiative activities and successful strategies to improve undergraduate STEM education, drawing attention to their own campus-based success stories supporting the AAU Initiative (of which they are often unaware). AAU recognizes that it is critical to have continuous communication and create spaces for relationships and trust among peers to develop.
At the same time, AAU leadership and staff have also made it known to campus leaders when they have observed relative inactivity from their particular campus in engaging with the Initiative and/or in focusing attention on improving the quality of undergraduate STEM education.
AAU also collaborates with other national associations, organizations, funders, and industry partners to coordinate activities relating to undergraduate STEM reform and to develop effective means to disseminate promising and effective programs, approaches, methods, and strategies. The Initiative engages multiple stakeholders to promote long-lasting reform to undergraduate STEM education, and it works to address the cultural and policy barriers within research universities that hamper educational improvement and innovation.
In 2016 AAU received two major awards. A grant in the amount of approximately $1 million over four years from the Northrop Grumman Foundation will support institutional mini-grants to further advance and coordinate existing efforts aimed at improving undergraduate STEM teaching and learning. AAU will award two rounds of twelve mini-grants designed to further existing efforts to improve undergraduate education.
The first cohort was announced in January 2017 and will fund specific improvements in individual departments or across colleges at the selected universities. Efforts include creating learning communities for STEM faculty members involved in reform efforts, establishing programs to train graduate students and undergraduate teaching assistants or peer advisors in active learning practices, developing college-wide teaching evaluation programs, implementing an educational analytics program for the university, and supporting STEM course redesigns.
A grant in the amount of approximately $700K over three years from the NSF will allow AAU to examine the institutional landscape in which STEM innovations take place to better understand how universities align their various projects to promote long-lasting reform to undergraduate STEM education. This project led by Emily Miller, James Fairweather and Mary Deane Sorcinelli is designed in recognition of the reality that many AAU universities are advancing multiple department-level as well as institution-wide efforts to improve undergraduate STEM teaching and learning.