A Discussion of “(Dis)Organization and Success in an Economics MOOC”
MOOCs will change higher education forever!
That was a common statement in 2012 when I was early in my professorial career so I was alarmed by claims my industry was coming to an end.
Banerjee and Duflo (2014) said it was “a fast cycle of hype and disappointment” because by the end of 2013 the Washington Post was asking, “Are MOOCs Already Over?”
First, what is a MOOC, besides being a fun acronym to say?
Massively Open Online Class.
Some of the top universities launched these classes which were available for free for anyone in the world. Some offered an option to pay a small fee to earn a certificate of completion, usually requiring a final test or other assessments.
Thanks to technology, it was argued that now anyone can access high quality classes. No longer did geography or lack of funds have to mean lack of access to knowledge.
So what happened?
A study by the Graduate School of Education at UPenn reported on the completion results of the first 17 UPenn courses offered on the Coursera platform. (p. 514)
- Only 5% of students who registered did complete the course.
OK. Results like that are not going to spell the doom of higher education as we know it!
Why such a low completion rate?
The authors consider some reasons. For one, they say not everyone who registers for a class actually goes on to attend it in any way. They say this might be “shopping around” where students register for every mildly interesting class and then later only attend a fraction of them. (p. 514)
Still, if you look just at the ones who attend the class, not just register, only 2%-14% of attendees are still active by the last week of the class.
The authors cite another source of MOOC data that showed similar outcomes.
In their analysis of outcomes from MIT-based edX course “Circuits and Electronics,”…76% of students account for only 8% of the time spent on the course while 7% of the participants accounted for 60% of the time spend on the course.” (p. 514)
Translation: only a few students actively engaged in the course.
Banerjee and Duflo (2014) examine reasons for this using a MOOC they taught for MIT in spring 2013, “The Challenges of Global Poverty.” (p. 515)
Interestingly, this was simultaneously taught to the general public as a MOOC and to MIT students. Since the class was taught as a flipped classroom, both groups accessed the lectures and homework online. The MIT students, however, worked in groups in class to prepare and present case studies. It is not clear what the MOOC students did instead of this sort of work, if anything.
For the MOOC students, there were 10 modules to complete with each one including 3 hours-worth of video lectures, though they were broken down into 6–10 minutes per video.
In addition, there were some short comprehension quizzes between the videos, available slides, and automatically graded homework assignments.
To earn a certificate required a final average of 50 or higher. The average was calculated using the highest 7 homeworks (out of a maximum of 9) and the highest 19 quizzes (out of a maximum of 22).
Thus not every homework or exercise had to be completed, but certainly most did to earn at least a 50 average.
So overall then there is quite a large time commitment in watching videos each week as well as completing at least 7 homework assignments and 19 quizzes in order to earn a certificate.
First, how many students took the class?
- 42,314 students registered, 26,140 started (viewed a page), 12,947 were active (attempted an assignment) and 4,597 earned a certificate
- Which means, the retention rate is 11% of registrants, 18% of starters, and 36% of active participants.
Thus this class had better retention than the average MOOC.
Demographic data collected on 4600 students in the middle weeks of the class showed its wide reach.
- The class reached people in 194 countries (isn’t that all the countries?)
- Only 28% were from the US. India was second at 10%.
- English was the first language for about 50% of the surveyed students.
In addition, the report that the students had a “relatively advantaged background” (p. 516)
- 99% had computers at home
- 87% had 25+ books
- 98% had a high school degree
- 82% had a bachelor’s degree
- 75% had parents who had at least had some college.
The authors’ analysis mainly focused on one aspect— the date of enrollment.
They found no difference in the final grade between those who enrolled early and those who enrolled on the final day. But since enrollment stayed open after the class started, they found that those who enrolled late performed significantly worse.
There is something significantly different about those who enroll a day late. Those who enroll later get even lower grades. (p. 516)
Focusing on those who enrolled in a narrow window of 15 days around the enrollment date, on average 45% of the students received a certificate, but for those who enrolled late, only 29% earned a certificate, an overall 37% reduction in attaining a certificate.
Likewise, their grade was lower. The late enrollees had an average 10.7 points lower than the overall average grade of 42.5, an overall 25% lower average.
The authors said that this is not due to the late enrollees missing assignments because there were no grades given in the first week, so they had time to view the material and catch up.
Their hypothesis: late enrollees perform worse because of disorganization. (p. 517)
They examine the data and say there is no demographic differences between the late enrollees and those who enrolled by the cut off date. They said the only variable that stood out at all was whether English was the native language. It was not significant when looking at this group limited to 15 days around the enrollment date, but it was if you included all the students.
Further, they say it is not due to a difference in motivation. They collected survey data on their motivation. Controlling for their self-reported motivation, the different outcomes for the 2 groups narrows, but still remains. Instead of being 16% less likely to achieve a certificate, they were 9% less likely once adjusted for lower motivation.
Finally, they say it is not ability differences. Completed homework grades showed no significant differences in grades for the late enrollees compared to the average. The late enrollee on average completed 1.146 fewer assignments, however.
Thus they conclude, late registration does not signal lower ability but instead lower willingness to commit.
From all of this, the authors conclude that outcomes could be improved for all students, but particularly for the late enrollees, by providing more structure.
They suggest that in future offerings of the course they want to offer a random group of students an option to set aside regular times to study, reinforced by the knowledge this can be verified by log in data. Also, they may include the option for nagging reminders.
Presumably, if these added optional increases in structure led to better performance, then we would see that more structure could improve outcomes in MOOCs.
Interesting proposition, but I do not see that these authors later published any additional work in this area. Typically in academia, inconclusive results tend not to be published so perhaps the suggested increases in structure did not pay off. Or perhaps their schedules changed and they no longer taught this class.
The idea of structure improving the online experience does sound persuasive. The recent switch to remote learning in Spring 2020 thrust a lot of students online, many who did not desire to take classes online, in classes that were not originally designed for that format.
The complaint I heard the most is the challenge of getting started on the work because they had lost their structure.
- The regular class time was gone because we did not keep the classes synchronous.
- They had lost their study routine and favorite study locations.
- And, they lost the social pressure to prepare for class, for fear of being called on and looking bad in front of everyone.
I think these issues also apply to MOOCs and why they did not take over the world as predicted. People need more than information. Maybe it’s structure. Maybe it’s community. But it does seem time together matters.
Beyond the Lecture: Time to Blend
MOOCs roared on to the scene with hype saying it was the end of traditional education. While they have shown us that technology can make knowledge more available than ever, human beings seem to need more than availability. Studies like this one show us that for now, traditional education does not seem destined to end any time soon.
References:
Banerjee, Abhijit V. and Esther Duflo (2014). “(Dis)Organization and Success in Economics MOOC.” American Economic Review, Papers & Proceedings, 104 (5): 514–518.
By Ellen Clardy, PhD on .
Exported from Medium on December 15, 2022.
