Pedagogy

Extent of Participation
 
Direct participation of departments, courses, and students (in terms of student-seats) in the AAU Initiative grew at the eight project sites over the three years of the project. At the same time, AAU encouraged other non-AAU project site institutions and departments to undertake reforms.

At the project sites, the total number of participating departments increased from 28 (Year 1) to 37 (Year 2) to 39 (Year 3). The total number of courses involved increased from 69 (Year 1) to 143 (Year 2) to 162 (Year 3). The total number of student-seats increased from around 38,000 (Year 1) to 45,000 (Year 2) to nearly 56,000 (Year 3). These numbers are continuing to rise because the reforms are expanding to more sections and additional courses.

The table below depicts the number of faculty members by type of appointment engaged in course reforms.

It is important to note that each project site institution had its own plan for the project, and those who included more departments and courses are not necessarily “better” than those which included less. However, the primary goal of the Initiative is to spread evidence-based pedagogical practices as widely as possible. 
 
Implementation

All project sites showed evidence of dissemination beyond the initial target courses and faculty members. All eight project sites increased the number of courses targeted for reform based on evidence-based pedagogy (some expanded the number of sections of the same course) and all increased the number of faculty members (tenure-track and non-tenure track) participating. One-half of project sites expanded their reach to additional departments. One-half of project sites developed and disseminated common tools used to assess teaching and instruction, in some cases adopted by the university as a whole. Several project sites linked co-curriculuar activities with reformed courses to increase retention in STEM majors.

At Brown, innovative pedagogy formats used during the three years of the project have now became the departmental norm for 13 courses. Instructors of some of the introductory physics classes are committed not only to innovative pedagogy, but also to assessing students’ learning gains every semester and building a longitudinal understanding of how the outcomes of their instruction are evolving with time. Also, the number of academic units involved has grown. In the Fall 2013 semester, only the departments of Chemistry and Physics were involved in the AAU STEM project. Now, the Division of Applied Mathematics, the School of Engineering, and the Division of Biology and Medicine and their respective staffs are engaged in offering innovative pedagogy to their students in introductory and mid-level undergraduate courses.

One of the key findings from the work undertaken at CU Boulder was the need to shift from individual faculty and administrator consultations to departmental working groups. CU Boulder developed a new model for this work, called a Departmental Action Team (DAT). In the DATs departmental members work collectively, addressing unit-defined issues meant to improve undergraduate education in a sustainable manner. The project has facilitated a total of six DATs to date, which have often dealt with priority needs specific to each department. Examples include creating curriculum coordinator positions to better link sequences of courses, addressing diversity issues, and improving use of data in assessments.

One measure of success of the DAT approach is the expansion from the original departments—Interactive Physiology and Physics—to include Ecological and Evolutionary Biology, Mathematics, Electrical and Computer Engineering, and Information Science. There is now more interest by departments in forming a DAT than there are resources to support that effort. CU Boulder sought and was awarded NSF IUSE funds to support the expansion of DATs, both locally and at a second partner campus (Colorado State University). DATs will expand into Geological Sciences, Atmospheric and Oceanic Sciences, and Computer Science at CU Boulder in the fall of 2017.

WashU’s efforts have expanded far beyond the original scope of the project, including into non-introductory and non-STEM courses. Work has continued into a fourth year, with 14 departments/programs, 71-course sections, and 50 faculty implementing active learning. All courses that adopted active-learning continued to implement it in later semesters, which provides evidence of sustainability. Additionally, one faculty member in the School of Law has expressed interest in the clicker program, which, if implemented in their course, would expand these efforts to a new school and to graduate-level students.

The UPenn’s SAIL program continues to grow. In departments where individual instructors, rather than the department, choose to teach SAIL courses, all instructors plan to continue offering their courses in the SAIL format. The Office of the Provost, through the Center for Teaching and Learning, has sent out the call for more SAIL course development grant proposals, and the number of SAIL courses is expected to continue to grow over the next academic year. Some departments have expanded their efforts beyond introductory sequences to include upper-level courses. 

Efforts have also expanded to additional STEM departments and some nonSTEM departments. For example, the SAIL version of one of the introductory economics courses, reaching approximately 600 students annually, is now in its fifth semester, and has featured iterative improvements based on student feedback and learning outcomes. This course offers the opportunity to explore ways to scale this approach for larger classes, and the number of students allows for the evaluation of specific course elements.

Beyond SAIL, there has been an increase in other active learning efforts. One such mechanism is the design of active, student-centered recitations to complement larger lecture courses. The Math, Chemistry, and Psychology Departments launched versions of their introductory courses with active recitations this year. Lessons from the SAIL initiative have been helpful in other endeavors, as instructors can use previous student feedback on activities and group work to shape their approach.
 
At UNC-Chapel Hill, a noticeable change has occurred in the culture of the departments involved in the AAU project. There is also evidence of spill-over effects to other departments. There have been additional course redesign efforts in Chemistry and Biology. The Department of Mathematics hired a lecturer knowledgeable in evidence-based methods and has begun to redesign its calculus sequence. The Department of Anthropology initiated a department-wide course redesign effort that was influenced by the AAU project, and the Departments of Economics and Psychology have continued to expand their use of evidence-based teaching methods during the AAU project. Other faculty members within the participating units have begun to incorporate many of the ideas that have developed from the project. At least one example of a mentoring relationship was established with a faculty member at another university.

UC Davis has expanded its efforts in several departments.  Introductory biology underwent a major and long-lasting change as a result of a combined Gates Foundation and AAU effort to improve student outcomes. Teaching assistant training resulted in a highly structured discussion environment that emphasized group work and high-order cognitive skills development. Combined with a modeling-focused active lecture, students are now able to successfully complete substantially higher order problems, as based on Bloom’s taxonomy. In addition, the changes in the introductory course are having measurable effects on subsequent course performance.

In the Chemistry Department the introductory chemistry restructuring project expanded substantially beyond the rethinking of the initial course sequence. Three major components emerged including: 1) the methodical measurement of learning as a function of the instructional approach for the introductory courses, 2) a major overhaul of the chemistry preparation pathways, and 3) an entirely new three quarter sequence for life science majors.
 
For the first component, it was shown that instructional approach could be reliably measured using the Classroom Observation Protocol for Undergraduate STEM (COPUS) instrument via the Generalized Observation and Reflection Platform (GORP) tool, which was developed by the CEE team. Additionally, it was found that active learning instructional approaches yielded improved learning compared to traditional approaches, especially with less prepared student populations.
 
For the second component, the Assessment and Learning in Knowledge Spaces (ALEKS) preparatory chemistry adaptive learning summer self-paced course was tested, replicated, and fully incorporated to help prepare students for immediate entry into the chemistry sequence. In summer 2016, over 600 students could enter the chemistry sequence without needing a three-unit, non-credit bearing, preparatory course in their first quarter. This approach better prepares students, saves them time and money, and improves initial chemistry course success. The overall success of this program has inspired the Math Department to try a similar approach.

For the third component, this approach focused on conceptual knowledge and life science application with an entirely revised laboratory that emphasizes open-ended problems. The Chemistry Department chose to adapt the UA Chemical Thinking curriculum for several of the courses and currently is examining the impact.

MSU’s AAU project focused primarily on the main lecture sequences in general chemistry, introductory biology, and introductory physics. However, other course committees and instructors have begun to use the three dimensions developed by MSU (scientific practices, crosscutting concepts, and core ideas) and adaptations of the three dimensions as the framework for transformation efforts. These courses include a pre-general chemistry course that focuses on connections between chemistry and mathematics, organic chemistry, the general chemistry laboratory sequence, one section of an upper-division physical chemistry course, second-tier yet foundational biology courses including ecology and evolution, calculus for life science students, and introductory physics for life science students.

The goals of MSU’s Biology Initiative and its AAU project overlapped substantially, including an overall focus on scientific practices and core ideas in courses and degree programs. Additional TAs and LAs provide the resources necessary to facilitate more student-centered and active approaches in class meetings as well as implement more frequent assessments and feedback. Curriculum coordinator positions have also been implemented for several courses, to develop and maintain a shared vision for the courses based on a core set of learning goals and to develop and evaluate new course materials and to coordinate assessment of student learning across sections.
 
In the Chemistry Department, implementation of transformed courses led to a permanent increase of resources including the creation of two new permanent instructional faculty positions: a lecturer for general chemistry and a Director of General Chemistry Laboratories. These positions allowed the Director of General Chemistry to focus on the transformation effort, coordinate the new materials, and provide support for faculty who rotate into the new course.

The success of the transformed course in physics has led to faculty support for expanding the course to additional sections, piloting a second-semester course, developing an integrated lecture-lab model for those students having to take both, and further development of the transformed laboratory. The Physics faculty voted overwhelmingly to embrace these changes to the extent that the physics budget can support them. The department has committed to supporting a single instructor line that is devoted the expansion and continued development of the transformed course, and additional resources for development of both transformed lecture and laboratories have been endorsed by the faculty and requested from the college and Provost.

All of the UA’s originally proposed course reforms were expanded within target departments. Additionally, in chemistry, the work on Chemical Thinking has expanded to:

• The development of a version of the curriculum for Honors students.
• The creation of a pilot preparatory course to better support students with weak academic backgrounds before they enroll in Chemical Thinking.
• One faculty member in Organic Chemistry using active learning instructional approaches and teaching in one of the Collaborative Learning Spaces.
• The use of the Chemical Thinking Curriculum at other three other  U.S. universities.

In physics, seven different instructors have used active learning to teach one introductory class. The active learning, student-centered teaching approach has been taken in five other physics courses, with reform of a sixth beginning this fall. The department has also started its own Faculty Learning Community focused on improving instruction and student learning in undergraduate physics courses.
 
Evidence Related to Student Learning

Each project site included a plan for assessing the effectiveness of the classroom interventions they proposed to use or test. These varied in form from site to site. AAU encouraged individual sites to use these data to inform ongoing practice, and to publish results as appropriate. AAU did not ask the sites to report learning outcomes data in a form that could be aggregated as with the responses to AAU’s instructional survey. Summarized here are results shared in the sites’ annual reports that point to trends across the Initiative toward improved learning gains, decreased failure rates, improved persistence from introductory to later courses, and shrinking achievement gaps for previously underserved students.

Across the project sites, a variety of tools were used to measure content or skill mastery. These included validated concept inventories developed by academic disciplines, inventories developed locally, new assessments developed following departmental adoption of learning goals, common exams administered across multiple sections of a course, and course grades. In addition to measures of content and skill mastery, most sites tracked additional student outcomes indicative of improved instruction often in line with institutional priorities. 

Although many of the interventions are still in process with data gathering and analysis at an early stage, some initial (and substantial) trends are evident. Every site reported some improvement in student learning outcomes. The magnitude and significance varied according to the different stages of the reform process across the institutions and across departments in some institutions. Dramatic reductions in achievement gaps especially for women, under-represented minorities, and first-generation students were observed in some sites. Reports of decreased DFW (D grades, F grades, and withdrawals from a course) rates were common, as were increased persistence to the next or later courses and success in later courses as measured by grade performance. Improved performance on exams sponsored by disciplinary societies was observed, as was stronger performance on disciplinary concept inventories. Some sites also have tracked the effects of instructional interventions on more general psychological factors, such as self-efficacy, metacognition, and student attitudes toward science.

MSU included an explicit strategy for changing the ways that student learning is assessed. Use of the 3D-LAP allowed the project team to explicitly identify assessment tasks that require students to use their knowledge in the context of scientific practices. Early research results from MSU’s work are very promising, and provide the team with insights about factors that affect transformation of large-scale courses. Research studies on student learning in chemistry show that compared to students in traditional courses, students in the transformed courses show significantly increased understanding and use of core ideas in chemistry. This, coupled with a decrease in the rate of DFW, on average, around 450 more students per year now pass the course with a grade of C or better.

John Pollard, Associate Professor of Practice, teaching a Chemical Thinking course at the University of Arizona.

At UNC-Chapel Hill, the academic year average D/F rates in redesigned courses dropped from 11.5% in 2013 prior to the AAU project to 9.5% in 2016. The learning gains in HSAL courses were 13% higher than in traditional courses. Teaching Assistants at UC Davis trained to use active learning practices and adaptive learning technology from Carnegie Mellon were able to raise student outcomes by half a letter grade (and/or increase the probability of passing exams by 66%) in introductory biology.

As a result of the Initiative at WashU, 57 STEM faculty, from 9 of our 11 STEM departments, have taught 49 different courses (76 course sections) using personal response systems (i.e., “clickers”). WashU’s evaluation studies found that the clicker-based active learning used in three of the courses, which were high-enrollment introductory science courses, was positively associated with exam performance, even when students’ cognitive characteristics (ACT, AP test scores, scores on course pre-tests) and class attendance were accounted for.

CU Boulder’s Departmental Action Teams each worked toward department level consensus on learning goals, pedagogical approaches, and assessments aligned with the learning goals as a way to build widespread and lasting change. In the physics department, CU Boulder conducted pre/postconceptual assessments of four courses (Calculus-based General Physics I and II and Algebra-based General Physics I and II) over five semesters of the AAU project. The results averaged across semesters indicated that students from all four courses had higher posttest scores between 25% and 30% in reformed courses.

UA found improved learning gains in sections of physics with calculus taught with active learning approaches, compared to the traditional lecture class. The reformed course final exam scores outperformed the traditional course on every exam item. The redesigned Chemical Thinking has demonstrated that students’ performance on the American Chemical Society standardized exam were not significantly different from those in the traditional course. However, students in Chemical Thinking performed significantly better on a conceptual chemistry questionnaire (55.3% vs. 44.3%) and had significantly more positive attitudes toward chemistry than the traditional group.

Not surprisingly, since the Initiative overall intended to catalyze change, much of the assessment work was used formatively over the several years of the interventions. Reports included multiple accounts of fine-tuning of approaches based on student outcomes, with significant growth and insights into factors that make a difference in impact such as student readiness and interest, physical settings for instruction, whether students are taking the course for the first time or repeating it, and effective practices for managing group dynamics for group work. UC Davis tested among other things the use of free online textbooks in two disciplines, and found students did as well with them as with traditional textbooks (a significant financial benefit for the students).
 
The first published reports of specific impacts can be found in the “AAU STEM Project Site Scholarship” section.