A Discussion of “A Randomized Assessment of Online Learning”
I teach at a small, private college which traditionally has offered a low teacher-student ratio as one of our main amenities. It’s why I teach there because teaching is my favorite part of being a professor.
Then the pandemic struck this spring and we all shifted to “Remote Learning” in a week. Not face to face anymore but not really an online class since that is a class designed to be taught asynchronously.
I have taught both face to face (aka “residential”) and online before this crazy spring. While I prefer being in the classroom, both have their strengths. In fact, I learned a lot about how to do a better job in each format from that sudden shift to remote learning.
Whether we teach face to face or online or somewhere in the middle, what really matters in the end is what helps the students learn. How can we determine what works best for the students?
Gathering data from various classes to assess the differences in outcomes is a possibility but we cannot be sure what any differences may mean. Students who choose to take an online class may be characteristically different from those who choose to take a face to face class; thus, any observed differences in results could come from differences in students.
A Randomized Assessment
Alpert, Couch, and Harmon (2016) fix that issue by randomly assigning students to three different formats. No idea how they got away with randomly assigning students, but I am happy to read the results!
The data collected is for four consecutive 16-week semesters of principles of microeconomics classes taught by one instructor at a “large, public university in the Northeast.” (p. 379) Each semester the students interested in taking the class had to agree to be randomly assigned to one of three sections, each capped at 35 students.
From the students’ perspective then they decide that micro fits in their schedule this semester, but they must accept that they will be assigned to a section that meets twice a week, once a week, or purely online.
Part of the article explored the issues around which format had the most students drop out, but that part did not interest me as much. (It was the purely online.) I liked finding out what seemed to make a difference to the learning outcomes of the students, which is being measured by the cumulative final exam score.
They first analyze the students in each section to make sure they really are randomized. Across multiple characteristics, they conclude this was in fact a random assignment, so we do not have to worry about the usual issues like selectivity bias.
To understand the results, we need a little more detail then on how each format worked, what they had in common, and what was different.
- Face to face: Students in this class met two times a week for 75 minutes each session. One session was a lecture and the other was a discussion. Power Points were available online.
- Blended: Students met one time a week for 75 minutes for a discussion. Power Points were available online as well as “online lecture materials” (described in a minute).
- Online: Instead of meeting in person, students participated in asynchronous online discussion forums. Part of their course grade depended on their participation demonstrated by posting or responding to questions, posting links to helpful articles and videos, and responding to the instructor-posted questions. Power Points were available online as well as online lecture materials.
OK, what are “online lecture materials”? (p. 379)
- Recordings of the live lectures given in the face to face section using Power Point slides.
- A 20 minute “compact” recording of the lecture using some of the Power Point slides. This version is closed captioned in English and Spanish. The closed captioning is keyword searchable, which could be helpful for later review.
- Enhanced Power Points which are broken into 5 “manageable chunks.” They are narrated, include animated graphs, and a drawing tool is provided to get students to practice drawing the graphs they are learning.
Those are some really great materials for teaching yourself!
What I love about this study is what it allows us to compare and contrast.
- If the face to face comes out significantly better than the other two formats, the unique characteristic is the in-class lectures. That would seem to indicate that lecturing is a valuable tool.
- If the blended comes out significantly better than the other two formats, then the lack of in class lectures is not a problem. Using the online lecture materials is apparently an improvement. But, the in-person discussion would appear to be important because that is missing from the online method.
- If the online comes out significantly better than the other two formats, then it would appear in class time is not really needed.
Results
- The online students average 4.2 points lower on the final exam than the face to face students.
- There was no significant difference in scores for the blended class compared to the face to face students.
Essentially, then the learning outcomes for the students is the same whether they are in class one time or two times a week.
The two formats include in-class discussion but differ in how the students learn the material. Face to face students are taught through a lecture while the blended students use the online lecture materials to accomplish the same thing on their own. However, not being in person at all and missing out on an in-class discussion lowered the learning outcome.
Their results are in line with the few similar studies the authors cite. Randomized assignments of students in a college-length course had only been published in 3 previous studies according to the authors at the time they wrote.
- One found that students who attended live lectures outperformed the purely online student by 2–3 points.
- Another found no significant differences on the average final exam score between those who were in face to face versus those in a blended format.
- And the third one, looking at statistics classes instead of principles of microeconomics, also found no significant differences in outcomes between face to face and blended.
Alpert, Couch and Harmon (2016) conclude
These results suggest a potentially promising avenue for economizing on teaching resources while maintaining student outcome is to use blended teaching models of instruction for economics principle that reduce instructor contact time in a classroom setting. (p. 381)
I have to say, this make my heart sing a bit as we head into the fall. We have been told our face to face classes will be hybrid. Half of the students come one day, and the second half are given access to a recording of that session. Then the second half comes the second day, while the first half is given access to a recording of that session.
As a proponent for active learning activities over passive learning such as lectures, this study demonstrates class discussion is an important learning activity.
If we focus on providing high level online lecture materials to all students and engage in classroom discussions with each group for the one day they are in class, then perhaps learning outcomes will be the same as when we were all face to face.
Active Learning: Myth or Reality?
References:
Alpert, William T., Kenneth A. Couch, and Oskar R. Harmon, 2016. “A Randomized Assessment of Online Learning.” AEA Papers and Proceedings, 106(5): 378–382.
By Ellen Clardy, PhD on .
Exported from Medium on December 15, 2022.
