One strategy to successfully institutionalize reform is to embed instructional expertise within the department to provide educational leadership and to support all faculty members in the adoption and use of evidence-based pedagogy. Although the types of appointments of individuals with this type of expertise vary widely (e.g., tenure-track, non-tenure track, junior and senior ranks), these individuals all have in common an understanding of the discipline and how students learn best within the discipline. When used most effectively, these individuals are well positioned to provide educational leadership to the department.
Individuals in these roles can help redesign courses, co-teach courses with other faculty members, take on the responsibility for departmental-wide educational improvement, and produce scholarship for the broader higher education community based on the evaluation of reformed courses. These individuals often will also engage in efforts with other STEM departments to achieve broader institution-wide systemic STEM teaching reforms. Acceptance and support of individuals with instructional expertise—even those in tenure-track positions—by departmental leaders and by tenure-track faculty members is also an essential element for long-lasting change. In Biological Sciences, Physics and Astronomy, and Chemistry, The UNC-Chapel Hill has made several teaching-related hires with backgrounds in evidenced-based STEM teaching since the start of the AAU project. Over time, these personnel have been seen by chairs and their faculties as strategic hires whose help in achieving overall departmental instructional objectives is paramount. UNC-Chapel Hill staff (and AAU staff) have found these hires fully entrenched within the respective departments and are beginning to have “lasting impact on the teaching culture within departments. As another example, MSU hired a curriculum coordinator for introductory biology courses and added staff to develop inquiry-based laboratories in Chemistry. As a third example, WashU is embedding educational specialists in STEM departments. WashU also hired a coordinator for General Chemistry and appointed a new Vice Dean for Education in the College of Engineering.
Individuals hired to enhance the instructional expertise of their programs and departments—even tenure-track discipline-based educational researchers—can find acceptance by their departmental colleagues difficult. Even though UC Davis is part of the University of California System, which has a designated employment category for teaching-oriented positions endowed with the possibility of tenure (LPSOE - lectures with the potential of security of employment), some of its STEM departments have found it difficult to fully utilize and accept LPSOE hires. Cultural differences between departments make it quite difficult to see uniform change across UC Davis in this respect. The Center for Education Effectiveness (CEE) is working with departments and colleges to clarify promotion and tenure criteria and create tools to help all faculty members including LPSOE hires document their evidence-based practices.
In addition to adding positions with specific teaching expertise, enhancing the value placed on teaching at research universities starts with stated expectations for teaching when new tenure-track faculty members are hired. At UNC-Chapel Hill, for example, new STEM hires (including those with tenure-track positions) must have experience in the use of evidence-based teaching and be prepared to participate in the mentor-apprenticeship model. Also, MSU substantially enhanced the visibility of STEM educational reforms by hiring several tenure-track endowed chairs focused on STEM educational reforms.
Institutional Data Analytics and Visualizations
Data analytics and visualizations played a role in most of the project sites. Data focused on both faculty practices and student outcomes, the latter including academic performance and progress toward the degree. Much data collection and visualization about faculty instructional practices focused on classroom observation protocols.
The UC Davis developed the Generalized Observation and Reflection Platform (GORP), which has been used with the Classroom Observation Protocol for Undergraduate STEM (COPUS) protocol to observe interactions between students and faculty in large STEM classrooms. This tool has been disseminated through the Tools for Evidence-Based Action network and is now in use by over 100 universities.
At the CU Boulder, classroom observations were conducted with modified versions of the Teaching Dimensions Observation Protocol (TDOP) and COPUS. CU Boulder’s classroom observation protocol has been branded as VIP/OPLE (Visualizations in Instructional Practice & Observation Protocol for Learning Environments). The university’s Office of Information Technology is working on a scalable version of the TDOP that will provide faculty, on a voluntary basis, with observation-based visualizations of what happens across class periods. This service should benefit faculty members interested in gaining new insights into the patterns of their class activities, documenting changes as they try out new methods of teaching, and finding new ways to communicate about their teaching with colleagues.
WashU is studying faculty use of clickers using both a quantitative observation tool, the Observation Protocol for Active Learning (OPAL), and qualitative interviews. Observational data are analyzed to identify patterns and classify instructors into four categories based on how they used active learning with clickers in their classrooms. OPAL, which was developed as part of the AAU Initiative, has been institutionalized at WashU by the Teaching Center. In addition to the 200+ classroom observations made as a part of AAU research studies, the Teaching Center has integrated data from the tool into faculty consultations. A visual timeline of the data can be generated, providing faculty with an overview of the activities that occurred in their classroom. The timeline data can be used in addition to a more traditional classroom observation by a faculty developer, an approach WashU calls Multimodal Observation for Scholarly Teaching (MOST).
MSU has developed the Three-Dimensional Learning Assessment Protocol (3D-LAP), a set of criteria that can be used to determine if an individual assessment item (or cluster of items) aligns with a scientific practice, crosscutting concept, or disciplinary core idea (the “three dimensions”). They have used the 3D-LAP to characterize almost 5,000 assessment items over the course of the AAU project at MSU, fully representing all 200+ course sections of the eight relevant gateway lecture courses in biology, chemistry, and physics.
Institutions have looked at a range of student-level information. UC Davis developed Ribbon, which allows departments and campus administrators to look at migration in and out of STEM departments and between departments. More than one-third of the STEM departments and several nonSTEM departments now use the tool with many faculty members and professional advisors using it repeatedly. By 2017, more than 100 universities nationwide and internationally were using the tool to visualize student migrations as part of the Helmsley-funded Tools for Evidence-based Actions (TEA) project.
The Ribbon Tool developed by CEE at UC Davis visualizes undergraduate students’ educational pathways through courses and programs to inform department decision making.
As the work of the Center for Educational Effectiveness (CEE), much of which is related to reforms begun under the aegis of the AAU project, has become known across the UC Davis campus, additional departments have started collaborations with this unit. For example, the College of Agriculture has engaged in multiple studies of undergraduate student outcomes in programs including animal science, human development, environmental management, and agricultural economics. These studies focus on pre-requisites, troublesome large introductory courses, longitudinal course outcomes, retention of under-represented groups, development of course and program learning objectives, and online/hybrid course development. The departments of Communication, Economics, and Psychology have been involved in the development of hybrid and online courses with expanding plans for instructional research in the 2017-18 academic year.
The CU Boulder has undertaken an initiative called Data Analytics for Student Success and Educational Effectiveness (DASSEE). The tools developed in this project make it possible for departments and administrators to track impacts of courses and pathways of students across courses and address campus priorities for improving student retention, and diversity. Using Tableau as a visualization platform, queries of the student information system can be performed focusing on a single course over time, a suite of courses in a given department or unit for a given semester, or a dashboard of individual student course grades by major over term. Key visualizations in the dashboard include representation of enrollment, grade distributions, pathways into a given course and following course, retention/persistence of students from a given course over time (in the major and institution). Subgroups can be selected by term, faculty instructor, demographics (including: an individual instructor offering of a course, gender, first generation status, ethnic/racial identification), and for subgroups receiving a given grade in a course.
Efforts to visualize student data have involved partnerships with Institutional Research (IR) and the Office of Information Technology to implement visualizations in Tableau. The goal is to create a suite of easily accessible data sets and associated visualizations of educational outcomes for undergraduates. Although these data already exist, they are not available in an easily accessible way. Similarly, at the UPenn, the Office of Institutional Research & Analysis (IR&A) is working with the Center for Teaching and Learning to establish a protocol for data requests from instructors assessing their courses. By developing assessment tools in the Center for Teaching and Learning and making them available to course instructors, UPenn has reduced the “time barrier” and encouraged more faculty members to use evidence-based instruction.
Keys to Successful Use of Data Analytics
- Distinguish between the types of data useful for individual faculty members designing and assessing their courses from the types of data used in departmental decision-making. The key data influencing the faculty members (and chair) in the Chemistry and Molecular Biology Department at UA to adopt Chemical Thinking as the preferred introductory sequence in General Chemistry were the success of these students in subsequent courses; detailed within-course data on student learning outcomes demonstrated in the course were not important in the departmental decision. As another example, UNC-Chapel Hill and UPenn demonstrated decreases in Ds and Fs and course withdrawals from reformed courses. Departmental and university leadership were very interested in such data, particularly when underrepresented minorities showed differential improvement on these metrics.
- Ease and efficiency of use are keys to broad acceptance of teaching-related metrics. CU Boulder DATs have emphasized commonly-used metrics on student outcomes, which makes dissemination and use in decision-making more likely. Linking IR into the data sharing process is also important.
- Data must be seen as part of the policy and decision-making process rather than as sufficient in their own right. Among the more important lessons learned on the use of data in educational reform took place at UC Davis. Initially, the data analytics developed by UC Davis—which have broad appeal nationally and are quite widely used—were assumed to be sufficient to show faculty and administrators what worked and why. Over time it became clear to the CEE that passive presentation of data were not effective in promoting the redesign of courses and curricula. Rather, the CEE had to change its role to work more actively with academic units to devise strategies based on data analytics.
UA has devoted funding to redesigning classrooms into collaborative learning spaces (CLSs), including new furniture, additional projection equipment and other technology, carpeting, paint, and facilities work (including electrical upgrades, mounting projectors, etc.). Senior administrators have committed to three additional years of funding up to $1 million per year for new CLSs and other classroom and lecture hall improvements. These new spaces are needed to accommodate the desire for large collaborative classrooms (rooms for 130-250 student seats), since the institution has already targeted most of the large flat classrooms that can be converted to a CLS. The most popular CLSs are the larger rooms (90 to 264 seats). The university currently has ten CLSs ranging in size from 30 to 264 student-seat spaces, and its funded plan for summer, 2017 calls for ten additional CLSs ranging in size from 25 to 135.
The next new building on UA’s campus is likely to be an engineering building, and the Dean of Engineering wants to include space for large collaborative classrooms. Administrators are also working to ensure that renovation of the Old Chemistry Building will include CLSs. Several department heads in STEM fields, including in chemistry and biochemistry, molecular and cellular biology, and several engineering departments, have become involved in space redesign with two associate heads on the leadership team. Several others have themselves taught in CLSs. For example, the department head of Chemical and Environmental Engineering co-taught with an engineering faculty member in a CLS; he supports team teaching in this facility.
UPenn created several active-learning classrooms to accommodate growing demand. At the start of the AAU project and the SAIL program, UPenn had only one 42-seat active learning room since then, they have added six more active classrooms, including three that accommodate between 72 and 90 students. All are fully scheduled throughout the year. One more collaborative classroom is planned, and faculty committees have asked the university to investigate further additions. The demand for and use of active learning rooms continues to grow, prompting the development of an additional active learning room in the Biomedical Library.
UNC-Chapel Hill is experimenting with new classroom furniture and configurations designed to make it easier for faculty members to use interactive learning methods. These classrooms facilitate eye contact between students, instructor access to students, and transitions between lecture, class discussion, and small group work. A series of classroom renovations are in progress across the campus to support active learning efforts. Since 2010, UNC-Chapel Hill has created sixteen active learning classrooms for general-purpose use. Early work featured small and mid-size classrooms including two smaller (45 seats) and one larger (90 seats) studio classrooms (larger round tables for student interaction). In 2015, the university’s first interactive lecture hall (a 100-seat classroom renovation for active learning) was completed.
Others are in progress, including a large SCALE-UP classroom, indicating a significant institutional trend toward support of active learning.77 To date, four primary designs have been used for active learning classrooms.
At CU Boulder, the Provost’s Learning Spaces Committee in 2016 developed design principles for educational spaces that were informed by the AAU Initiative. These guidelines have been inserted in the “stage-gate” design process, before plans are made, as the concept for new space is considered by campus.
Although redesigned learning spaces are not required for all forms of active learning, it can affect some applications directly. UC Davis, as an example, noted that implementation of active biological modeling in an introductory calculus course was highly negatively impacted by poorly designed instructional space. Discussion rooms that did not have tables where students could work together made group work and modeling exercises via computers logistically difficult and made it more difficult for students to complete the assignments.
Institutions have utilized both graduate and undergraduate students as teaching and learning assistants. Inclusion of students in instructional roles has benefits for institutions at the level of the course or section. With more trained individuals in the room, the capacity to facilitate and evaluate active learning activities increases. Including students also benefits the students themselves. Not only must they master the course material and develop familiarity with the objectives of various evidence-based learning techniques, but they also must put into play skills including leadership, evaluation and analysis, facilitation, and public speaking. Undergraduate learning assistants can play different roles in the classroom, with different levels of formality.
Across the eight project sites, use of graduate and undergraduate assistants in active learning classes more than doubled, from 740 to 1,676, during the three years of the AAU project.
WashU utilizes Peer-Led Team Learning (PLTL) in its general chemistry and calculus classes, and is translating the model for engineering courses. PLTL groups typically consist of 6-8 students who work together to solve problems, and are facilitated by a Peer Leader. Peer Leaders are undergraduate students who have previously taken and performed well in the course.
WashU’s model of PLTL is designed to help students become conscious of the problem-solving process. It also helps students develop important collaboration skills, including how to approach problems effectively as a group, how to communicate well, and how to exchange and critique ideas in a collaborative environment. Participation in PLTL is voluntary, and students who do participate commit, through a contract, to meeting specific obligations to their group. Peer Leader training is an important component of the program. All Peer Leaders enroll in two one-credit courses. In these two courses, Peer Leaders learn how to be strong mentors for their groups, and they form a collaborative group of their own to help one another address common PLTL challenges.
At UPenn, undergraduate learning assistants have been utilized in a similar way. For example, in introductory biology, two instructors restructured their large lecture into a hybrid active learning course with the help of undergraduate learning assistants. The hybrid biology course features traditional lecture two days a week, and a problem-based active-learning session once a week where students apply concepts or are introduced to new topics. The success of this format encouraged these instructors to further adapt the course; they plan to integrate shorter active moments into the lecture portions of the course, and formalize student group assignments to encourage more productive group work.
UA utilizes trained undergraduate learning assistants to help encourage student discussion (without giving the answer) and to help the instructor with formative assessment of student learning. At UA, as at several other institutions, undergraduate learning assistants are often students who have previously taken and done well in the course. They are given credit for acting as learning assistants and further develop their leadership skills, but they do not need to be paid.
One example of how UA uses undergraduates as learning assistants involves a measurement instrument called Fine-grained Evaluation of Active Learning (FEAL), which is designed for large classes in collaborative learning spaces. FEAL is an activity quality measurement instrument that can be quickly administered by undergraduate learning assistants to record key measures of activity success such as student engagement, student success, activity difficulty, activity time, and associated lecture time. The instrument is designed to require minimal training and minimal effort within the classroom for recording observations. Quick administration of the instrument is critical because the learning assistants recording with FEAL are primarily tasked with engaging the students with the given activities. A key difference between FEAL and other tools (such as COPUS) is that the learning assistants must have expertise on the subject matter, as the intent is to evaluate activity quality. Moreover, relevant information such as the concepts covered by the activity are recorded and analyzed. The instrument is further applied to code exam questions and is used to correlate student performance for the same concepts.
MSU is increasing the number of graduate and undergraduate students who assist in teaching active learning classes. Funds for increased numbers of graduate and undergraduate learning assistants in the main introductory biology courses (cell and molecular biology, and organismal and population biology) have been made permanent. These additional assistants provide the resources necessary to facilitate more student-centered approaches in class meetings, as well as to implement more frequent assessments and feedback, especially on constructed-response tasks. In chemistry, resources for undergraduate learning assistants to support interactive engagement and the use of some constructed response assessment items in the lecture sections of general chemistry have been written into the departmental budget. This will be extended to organic chemistry in Fall 2017.
At UNC-Chapel Hill, the computer science department began utilizing undergraduate learning assistants to provide academic support to their peers in lectures, labs, office hours, and recitations, providing more individualized support to students while strengthening their own understanding of the course material in the process. According to the department’s website: “Within two years, the learning assistants became the linchpin of the department’s introductory courses.” The department has introduced the Learning Assistant Award, which recognizes an undergraduate learning assistant who has shown outstanding dedication to helping their fellow students understand and master challenging course material.
Department Support Structures
Another key to successful institutionalization of undergraduate instructional reforms is to align relevant administrative units, such as Centers for Teaching and Learning, with department-based instructional improvement efforts. For some institutions, this approach included linking advising and co-curricular units with classroom instruction. For others, it meant consolidating units to form a more effective organization. Providing college or campus-wide structures to support departmental reform efforts increased the likelihood of institutionalization in AAU project sites.
Structural innovations offer one type of solution. One alternative is to embed pedagogical expertise in each department by hiring, for example, LPSOEs at UC Davis. Another is to develop a teaching-oriented center staffed by respected senior tenure-track faculty members engaged in STEM reform such as MSU’s CREATE for STEM Institute and WashU’s CIRCLE. Brown provides another example, shifting the locus of control for STEM reforms from its Sheridan Center for Teaching and Learning (which remains highly engaged) to its academic departments.
More culturally-oriented reforms also can assist in enhancing the value placed on instructional improvement. CU Boulder uses DATs to move away from appeals to individual faculty members and toward group consensus for reform. UA’s FLCs have been fundamental in promoting improved pedagogy and in institutionalizing reforms. This cross-department approach is especially important when many departments lack sufficient numbers of faculty members interested or trained in improving instruction and reforming curricula.
The AAU instructor survey also asked about use of on-campus and off-campus professional development opportunities focused on improving teaching. These percentages increased between the two survey administrations, showcasing instructors’ additional interest in improving their teaching.