Creating and Running Online Courses: Webinar Series for the Indian Government

Webinar
Timeline:
to
Location:
Zoom
John Floretta, J-PAL Global Director of Policy and Communications, delivering a lecture at the Lal Bahadur Shastri National Academy of Administration, India.

J-PAL South Asia in collaboration with J-PAL Global organised a series of webinars through June 2020 for officials from the Indian government. This series was geared to support the Indian government as it transitions to a centralised online learning platform to build the capacity of their officers. 

Webinar Series

One webinar was held every Wednesday between 17th June- 1st July 2020. Details are below:

1. How to Plan your Course Launch: A Case Study 

This webinar  provided an overview of the best practices while planning an online course, including aspects such as roles and responsibilities of the course development team and the types of content required to be developed. The speaker touched  upon key questions such as: how do your modules work together? How will you measure student learning & the effectiveness of your teaching innovations?

2. Best Practices on Creating an Online Course

This webinar shared guidance on creating lecture and exercise segments, finalising course details and beta-testing. Participants learned about creating lecture and exercise segments that follow First Principles of Instruction techniques and best practices when testing content and preparing for course launch. 

3. Running an Online Course

Through this webinar, participants learned about how to manage an online course during its run. This webinar touched upon topics such as best practices for TAs, proctored examinations and using learner analytics. 

Responses to FAQs 

This section outlines responses to the most frequently asked questions during the course of this webinar series. The responses from speakers are based on their experience running the Data, Economics and Development Policy (DEDP) MicroMasters programme. For more details on the programme, please visit their webpage>>

Course Creation and Launch

1. Are there any standard softwares for course designing?

While there exist many options for course design (such as edX), CTIs will be using the Content Authoring tool on iGOT to do so.

2. How much is the anticipated average cost in developing a course given the best practices highlighted in the webinars?

The cost can vary depending on the amount of content that will be developed and also the existing resources available. For the DEDP team, being able to draw on existing materials was particularly helpful, through access to DEDP filming and studios. For overseeing the course development for a course of similar length, this would likely be a half or full-time role of a staff member over 6-9 months.

3. What is the most challenging part of the course creation process?

The most challenging part of the entire course creation process was to remain flexible and keep the bigger picture in mind. It is important to zoom out while designing smaller units to ensure both that they fit into the larger ideas and learning objectives of the course and that all components continue to move forward during course development. 

4. How can online courses be sequenced? 

Sequencing of courses depends on the learning objectives and whether the learners require any foundational knowledge in order to achieve their learning goals. Sequencing should be done in a way such that learning is scaffolded and the learners begin with foundational competencies and move onto more complex competencies. Courses on more advanced topics can incorporate reference materials from earlier introductory courses as appropriate.

5. How can case studies be embedded into the content?

The MITx team uses some case studies in their courses. For these case studies, the written content replaces the video content and there are questions in between (some multiple choice, some open response) to substitute for the in-person discussions. The case study module can be structured to begin with an introduction and then use structured reading and questions on the case studies. It is always useful to end the module with a conclusion and then open the discussion forum for additional comments. 

6. How do subject-matter experts and instructional designers collaborate to create online courses? What does the workflow entail?

Subject-matter experts create the material (lecture slides, questions, etc.) and instructional designers translate that into an online course setting. With the MITx courses, most of our material (created by subject-matter experts) is pulled from the residential courses and the rest of the course team uploads the materials and edit and rewrite material as needed to better suit the online format.

7. Should videos of in-person teaching be used as a substitute to PowerPoint presentations with a voice over?

This completely depends on the course design and the resources available. The MITx team uses videos of the course instructors from their lectures on campus at MIT and these videos are edited in parts to show the slides used by the lecturers. 

8. Who can be appointed as beta testers for a course?

Beta testers can be anyone who has taken the course in the past, people familiar with the content but who have not taken the course, and/or people who are interested in the course content with little expertise.The variety of knowledge and familiarity with the course subject will catch more errors in the beta testing process: those familiar with the course content will be able to flag mistakes and those unfamiliar with the content will be able to point out if topics need further clarification or if questions need to be rephrased. They should not be people directly involved with the creation of the exercises or videos because their familiarity with the online course content may cause them to overlook things.

9. What should be the maximum and minimum duration of an online course?

11 weeks is the length of the MicroMasters courses and that is a good maximum, especially if there is no break in between. There is flexibility in deciding on the minimum duration based on the content and the learning objectives. 

Running the course post-launch

1. What is the ideal number of learners for an online course?

The biggest difference between online and residential courses is that there is no need for a cap on enrollment for online courses. A large number of enrollees, however, may require more effort from the course team to moderate the discussion forum and answer questions.

2. Instead of offering a course just a few times in a year, can it be kept open at all times?

This is an option that can be particularly useful if you want your course to be accessible at all times and can provide learners with more flexibility. Some tradeoffs to consider include the engagement of your learners, both with other learners as part of a cohort in the course, and with the course material and keeping on track if there will not be a set schedule or deadlines. This would also require continuous TA and course team support.

3. Once the online course is launched, how easy or difficult is it to update or amend to keep the course relevant? Who will be responsible for these changes?

There are two types of changes that the DEDP team is responsible for implementing; the first pertains to changes that can be implemented in real time and are usually minor changes such as typographical errors, and the second type is usually reserved for the end of the semester, especially if they could significantly impact the learning experience of the current participants. For example, if the majority of learners have difficulty with a question or exercise, the DEDP team might prefer to change this into an ungraded question for the time being and make required changes at the end of the semester. The course team is also responsible for reviewing the course material to ensure it is accurate and up to date, which may include adding new case studies, worked examples, or recent papers.

Course Run Team Composition 

1. What is the difference between a course assistant and a teaching assistant?

The teaching assistant or TA is meant to provide support to learners on the content of the course by responding to their questions. The course assistant responds to queries about the grading system or any other issues that relate to the operational aspects of taking the course.

2. What are the required credentials for a TA position?

The TA should be someone who is familiar with the course content. Whether they are required to have a doctorate degree is dependent on the level of the course, for instance, if it is a Masters level program. DEDP sometimes hires non-PhD students who have taken the course in the past and scored well.

3. How can we assess the TAs’ competencies?

 It is important to test whether the course assistants or TAs are familiar with the platform and are able to address basic issues such as how to register and reset grades. For content knowledge or skill based training, the DEDP team typically looks at resumes and makes sure the TAs have mastered the material by taking the course and performing well or teaching a similar course before.

4. Who is responsible for onboarding the TA on the platform’s features?

The course manager should be responsible for onboarding the TAs in their roles and responsibilities and the platform features. If the course assistant is well versed in the functionalities of the platform, they could potentially take on this role instead of the manager. 

5. What are the roles of a course manager, course assistant and teaching assistant for an asynchronous course?

Asynchronous courses use pre-recorded lectures that are not delivered in real time and learners have the flexibility to move through the course at their own leisure. The roles of the course manager, course assistant and TAs as outlined in the webinar remain the same and the discussion forums pertain to specific modules. Responses to learners’ questions posted by TAs are segregated by modules and can be viewed by all learners. This enables the team to respond to learners adequately based on where they are in their learning process. 

6. What are the roles of the lecturers in online courses?

Senior lecturers play a minimal role in the design of the online course and are consulted for guidance at a later stage during the design of the proctored assessments. In-person lectures delivered by the senior lecturers are recorded and any on-going monitoring of learning lies primarily with the course manager, course assistant and TA. 

7. What professional development provisions exist for the TAs to stay up to date with changes in the course content?

Most MITx courses do not get upgraded too often and remain fairly consistent and hence, there is not much required by way of keeping the team updated. DEDP tries to hire TAs that have relevant work and / or teaching experience, and hence would be familiar with current trends in the courses.

8. Is it possible to run a course without a TA?

 It is possible to run a course without a TA but it is not advisable to do so as someone would need to take on the role of the TA and respond to learners’ queries. The course manager could potentially fulfill this role, however, that depends on how many courses the manager is responsible for and how many learners they are catering to. 

9. What are the minimum requirements for the staffing of course teams given that there may be varying levels of resource constraints across different contexts? 

Minimum requirements for the roles and responsibilities of the staff can vary according to resource constraints and the context. The course team members could perform various roles and do not need to strictly adhere to the team structure as has been outlined in the webinar. For example, the course manager or teaching assistant could perform the tasks of the course assistant. AI powered proctored assessments work out to be more cost efficient in the longer run than individual human proctors. 

Promoting Peer Learning 

1. How do you plan and run group exercises in an online course?

This is not a feature that is used in the DEDP courses, but different platforms may have functions that can support this. For example, you may be able to enroll learners in cohorts with group discussion forums that can facilitate such assignments. 

Keeping Learners Engaged

1. How can government stakeholders be engaged in active learning online?

It could be useful to break down the functions or activities performed by government stakeholders into smaller units and understand what steps are needed to learn how to perform them better. Questions can be created to enable learners to define these steps themselves. 

2. Can we replace finger exercises with breakout rooms?

For live lectures given over Zoom, breakout rooms can be used to have learners discuss the segments and use the polling feature to have them answer questions.

3. How can we make the process of teaching facts interesting?

Finger exercises are a great way to make things interesting. It is a way to increase engagement and allow learners to do an activity.

4. How can learners stay engaged in an online course and how can we generate interest around online learning?

Breaking down a lesson into smaller segments assists can generate continued interest amongst learners. These segments should not be longer than 10-12 minutes and should be in digestible formats. Finger exercises also help maintain interest and keep learners engaged. Encouraging participation during lectures through questions also helps engaging learners. 

5. How often should finger exercises be incorporated into the lesson? Is there a way to retain learners if they fail to solve these exercises correctly?

There is no definitive number of finger exercises that should be used in a segment of a lesson, however, they should be used adequately in order to generate interest and participation from the learners. Implications of failing to answer a finger exercise correctly can be decided upon by the course instructor, such as allowing learners multiple attempts and adding hints.

6. Are there any alternatives to homework as giving government officials homework assignments, given their busy schedules, may be challenging?

Lecture segments allow for a great alternative to time consuming learning tasks as the officials can take their time to work through the learning module, at their own convenient pace. Finger exercises are very effective in these situations as they are meant to be quick. Homework assignments can also be un-graded or broken into smaller segments to allow for flexibility. 

Assessments and Academic Integrity

1. Is there a recommended software for analysing learner data?

DEDP typically uses STATA but even Excel, R or any other software that best fits the organisation’s resources and capacity can be used.

2. What is the value addition of proctored assessments? 

For online courses, it is difficult to ensure that the assessments reflect what the learner has achieved as there could be instances of cheating. However, proctored examinations allow the learner to take the test when they reach a certain threshold in the course, in a controlled environment, where there is a higher level of certainty about the accuracy of the test results. 

3. Should every course have its own Honour Code or should there be a general code for the entire platform?

It is advisable to maintain the same Honour Code for the entire platform as it reinforces the tenets of the code and makes it easier for learners to abide by them. However, some aspects of the code may differ across courses based on the requirements of the course. 

4. Is there a possibility that a person may get falsely accused of cheating while appearing for the proctored assessments?

The course manager, course assistant or TA could always review the video of the learners taking the assessment and check for the flags of cheating to ascertain whether they are valid. However, this may be extremely time consuming and not possible to conduct if the number of learners is very large. This is not an exact science and there are trade-offs, however, to ensure authenticity of the certification, cheating behaviours have to be accounted for. 

5. How reliable are cheating detection softwares?

Several softwares such as HIIT, CAMEO, and CAPSO are reliable in the detection of cheating, two of which are used by the MITx platform. They have been extremely reliable in providing accurate data on cheating behaviours and extensive reports that flag where and when the cheating occurred.

6. Do the structures to avoid cheating limit or demotivate the learners and discourage learning?

Discussions and peer learning are encouraged on the platform and they have been found to be effective ways to motivate learners. However, the honour code is very clear on what constitutes cheating and group-work on proctored examinations is prohibited. This has not been found to inhibit learning and it is the responsibility of the course design team to articulate the honor code that communicates its intention adequately. 

Speakers

Sarah Gault

Sarah Gault
Sarah Gault is a Senior Research, Education, and Training Associate at J-PAL Global, where she works on the MITx MicroMasters in Data, Economics, and Development Policy and J-PAL training courses including Executive Education: Evaluating Social Programs.   

Previously, she worked as a research associate at the Urban Institute focusing on topics relating to state and local public finance. Sarah also worked as a consultant to the World Bank’s Development Impact Evaluation Unit on a randomized evaluation of Nepal’s Agriculture and Food Security Project.

Sarah holds an MPA from Brown University with a focus on global and comparative policy and a BA in economics from the College of William and Mary.

 

Elizabeth Cao

Elizabeth Cao 
Elizabeth Cao is an Education Associate at J-PAL, where she works on the development and data analysis of two of J-PAL's education initiatives: the MITx MicroMasters in Data, Economics, and Development Policy (DEDP) and the blended MIT Master's in DEDP. Prior to joining J-PAL, Elizabeth worked as a research assistant on projects related to literacy and teacher retention.

Elizabeth holds an EdM in international education policy from the Harvard Graduate School of Education and a BA in international studies and mathematics from the University of Wisconsin-Madison. Prior to graduate school, she was a Peace Corps Volunteer in the Republic of Georgia where she worked on education and girls' leadership projects.

 

Maya Duru

Maya Duru
Maya joined J-PAL as the Associate Director of Education in 2018. She is responsible for the management and development of the MITx MicroMasters in Data, Economics, and Development Policy as well as the Department of Economics’ blended Master’s in Data, Economics, and Development Policy.

Maya holds a PhD in political science from University of California San Diego. She worked as the Senior Monitoring and Evaluation Manager for One Acre Fund prior to joining J-PAL.