Introduction

Many disciplines aim to develop students’ decision making skills in real-world situations characterized by risk, uncertainty and constant change. Our innovation is to use a collaborative forecasting tool called a prediction market to create real world decision scenarios which are linked to module learning outcomes. This facilitates students in the development of decision making and information literacy skills in situations which mimic those they will face in their careers.

In their simplest form, prediction markets utilise contracts whose value depends on a future uncertain event to forecast outcomes. For example, a project manager may wish to evaluate whether a project will be completed on time. A contract is created which returns €100 if the project is completed on time and €0 otherwise. This contract is offered for sale to project employees at €50 (based on an initial 50:50 probability), typically on an electronic market. Market participants who believe the project will be completed on time will buy the contract, causing its price to rise. If they believe the contrary, they will sell the contract, reducing its price. The price of the contract is used as an estimate of the group’s collective assessment of the probability of the project being completed on schedule.

This simple approach can be easily extended to allow large groups of individuals to make forecasts about any future event. Our innovation is to use this tool in an educational context. Since 2010, we have successfully used our technique across a range of disciplines, including Finance, Economics, Risk Management, Tax, and Politics. We integrate it into modules as an online continuous assessment project. Our forecasting markets run continuously for days or weeks, allowing students to make and revisit decisions in response to changing information. Our approach prompts active learning in large groups, without imposing prohibitive administrative overheads.

Synopsis

Objective

Our goal was to create a tool that can be used effectively in the large classes that have become the norm in higher level education. Subject to this size constraint, we had three key objectives.

First, we wanted to improve student engagement and motivate active learning. Prediction markets present students with a competitive, dynamic, group activity. As a continuous, online learning activity, it matches the expectations of the millennial generation. It is a social activity, with market prices continuously moving in response to the trading decisions of participants, who can also communicate using inbuilt messaging tools. The social element of the activity, as well as the competitive desire to provide better forecasts than their peers, increases student motivation and engagement.

Second, we wanted to create a learning tool that improved students’ discipline specific knowledge. For example, a learning outcome in a Tax module might be “To understand how national tax policy is formulated.” To address this learning outcome, we ask students to forecast what policy changes are likely to be introduced in the next national budget. To make accurate forecasts in this context, students need to research how policy is made and what new tax policies are being considered. To engage effectively, they need to research and engage with a wide range of information sources both inside and beyond the classroom environment.

The necessity for students to engage in research also allows us to meet our third objective; to enhance information literacy and critical thinking. Students need to research and evaluate the reliability of information sources. When reading a study or a news article on tax policy, they need to consider questions such as “Is this what the author thinks will happen, or what should happen?” and “How knowledgeable and trustworthy is this information source?”

Approach

We have integrated prediction markets into numerous modules as continuous assessment projects. Students are provided with an endowment of virtual cash which they use to make forecasts on an online platform. The forecasts are tied to the modules’ specific learning outcomes. For example, in a Finance module, students are asked to forecast what the closing price of financial assets such as gold will be on a weekly basis, whereas in a Politics module, students may be asked to forecast the results of an election. It is important to note that as well as buying or selling contracts, students must provide a text rationale for the trade they made.

When students make successful predictions, they receive additional virtual cash, whereas poor forecasts result in a loss of investment. Students are ranked based on the amount of virtual cash they hold at the close of the market. This introduces a competitive element to the activity, as students can directly compare their performance with their peers. It can also be used to facilitate evaluation and marking.

When appropriate, we design the prediction markets to reset and repeat on a weekly basis. For example, students may be asked to forecast the weekly closing price of the Dow Jones Industrial Average. At the end of every week, the student’s endowment is reset. This avoids students becoming discouraged if they perform poorly in a given week. This structure has the additional advantage of creating a feedback loop that encourages critical thinking. Students receive feedback on their forecasts at the end of the week. If they perform poorly, the reset allows them to critically evaluate their decision making paradigm before starting again. Providing this feedback loop enables the reflection and introspection necessary for improving decision making and critical thinking skills.

Engagament

A key enabler of learning, which is often difficult to scale, is feedback. Our approach results in a range of scalable feedback channels for students. First, they receive feedback from the operation of the market itself both at the close of the market and throughout its operation by virtue of observing the changing probabilities of forecasts. If at any stage students see that the collective estimate of a particular outcome is different from their own, they are prompted to re-evaluate the decisions they have made. They can also read the rationales provided by others for the forecasts they have made. Crucially, prediction markets allow students to change their decisions at any stage during the operation of the market by buying and selling contracts in response to this feedback and/or newly revealed information.

The prediction market system automatically generates individual emails detailing weekly performance and identifying correct and incorrect outcomes. Lecturers can integrate the project into traditional lectures by presenting and analysing the anonymised decisions of both successful and unsuccessful students. They can use the textual rationales provided to illustrate successful and unsuccessful strategies. None of these feedback mechanisms carry a prohibitive administrative burden and they scale easily to very large groups.

In general, student engagement with these projects is excellent. Its unique approach captures and holds students’ attention. We have used prediction markets in undergraduate and post graduate modules across a wide range of disciplines, in classes ranging in size from 30 to over 400 students. Furthermore, we have shared the same project across similar modules in different Universities. This scalability speaks to the potential of prediction market utility for modern teaching environments such as MOOC’s.

Impact

Overall, we have achieved our objectives. These projects have a positive effect on student engagement. In general, students describe them as unique and interesting. For example, one student commented “I thought it was a brilliant way to learn – Pure genius”. When running these projects, one phenomenon we invariably observe is misdirection. Students will post false rationales or spurious web links in order to prompt others to make poor decisions. Obviously, this behaviour is ethically dubious, but is also a realistic reflection of the real world, and is another driver of improved information literacy. More pertinently, we see this behaviour as an indicator of genuine engagement by students.

To forecast well, students must access a wider range of information sources then required by a traditional module. They must attend to the news and consult sources such as government reports and surveys. One student noted “It made you read the newspapers and watch the news on a daily basis.” This has the ultimate effect of improving students’ general disciplinary knowledge, meeting our second objective.

We also see improved information literacy in students. These projects prompt them to synthesise information from a wide variety of sources of varying reliability in order to inform their decision making. As well as demonstrating how academic theories and research can be applied to real world contexts, they prompt students to deal with modern phenomena such as information overload and biased information sources.

The impact of this innovation has been validated in a number of ways. It received the Jennifer Burke Award for Innovation in Teaching in 2013, a national teaching award in Ireland. We have conducted empirical research validating its effectiveness and have published a number of articles in highly ranked journals such as Computers and Education.

Next Steps

Our work to date on the use of prediction markets in higher level education has developed a proven, mature pedagogical innovation that can be successfully used in a wide variety of contexts and situations. Our published research provides both practical guidance on how others can use prediction markets in their modules and evidence of their effectiveness in meeting learning objectives.

Nonetheless, the evolution of this technique is far from complete. Two distinct work programmes need to be pursued. First, a more nuanced examination of the impact of prediction market participation on learning is needed. Issues such as class context (i.e. large or small classes, undergraduate or postgraduate classes, and so on) or how individual personality traits impact upon the efficacy of the intervention need to be examined. This work will allow practitioners to better target their interventions to improve learning outcomes.

Second, we believe the great strength of our tool is its scalability. To date, we have largely deployed it within the confines of a discrete module. However, we can create larger learning communities. Prediction markets that extend across modules and Universities are possible, as are learning communities that include working professionals. Ultimately, we envisage creating semi-permanent prediction markets that are constantly running, providing a learning resource that lecturers and students can dip into. These markets would be a valuable resource for teachers around the world, and an ideal complement for emerging learning environments like MOOC’s.

The exposure and funding that we would gain from a Reimagine Education award would enable further work aimed at meeting these objectives and further enhance the utility of our tool as a learning resource for academics around the world.

Other Information

PDF - Click to download
PDF - Click to download
Project Website - Sample Support Site for Students with Videos
Project Website - Video introducing prediction markets as pedagogical tools
Project Website - Prediction Market Platform provided by Cultivate Labs
Other - Journal Article: Buckley, P., Garvey, J. & McGrath, F., 2011, A Case Study on Using Prediction Markets as a Rich Environment for Active Learning, Computers and Education, 56(2
Other - Journal Article: Buckley, P., & Doyle, E. (2014). Gamification and student motivation. Interactive Learning Environments, (ahead-of-print), 1-14.
Other - Journal Article Buckley, Patrick, and Elaine Doyle. "Using web-based collaborative forecasting to enhance information literacy and disciplinary knowledge." Interactive Learning Environments ahead-of-p
Other - Journal Article Garvey, J., & Buckley, P. (2010). Teaching the concept of risk: Blended learning using a custom-built prediction market. Journal of Teaching in International Business, 21(4), 346-357.
Other - Journal Article: Garvey, J., & Buckley, P. (2011). Using technology to encourage critical thinking and optimal decision making in risk management education. Risk Management and Insurance Review, 14(2)
Return to Awards