Brainstorm meeting, 22 March 2012

At 22 March a brainstorm meeting was organised with all team members. The project team is quite multidisciplinary, which was appreciated by all participants:

• Jeroen Donkers, Jean van Berlo: Knowledge engineer, data scientist
• Arno Muijtjens: statistician, psychometrics
• Danielle Verstegen and Bill Wrigley: educational scientist
• Guido Tans: study advisor
• Robert Peperkamp, Peter Bex, Eric Sol: developers

(Erik Sol and Peter Bex are hired from CaseBuilders. They developed the user interface of the ProF system and will, during this project, make some changes to the system to collect usage data.)

Expected usage patterns

During the meeting we discussed what usage patterns could exist for students and what factors can influence those.  We distinguished the following usage patterns:

1.       Quick orientation (look at the main page and browse through a few details)
2.       Study the profiles (look at the momentary scores for all categories or disciplines at different points of time).
3.       Search for issues (systematically browse through all categories and disciplines)
4.       Look at test-making strategy (look at different score-types: correct-score, questionmark score)
5.       Look for knowledge development (use the cumulative scores, compare different background populations).

Factors that could influence the use of these patterns are:

1.        What is at stake for the student. (For some students progress test is a bottle neck at the end of the bachelor and threatens to prevents them to enter the master. Those students will use ProF much more seriously than others.)  This factor is important, and can be computed from previous progress test results. The rules, however, differ per university.
2.        The study year of the student
3.        The level of the student (we take the score of the student at the progress test)
4.        The medical school

A complication factor is that sometimes a student logs on to the system together with a study advisor or mentor. In this case, usage will be quite different from normal usage. It will be difficult to filter out these occasions.

Next to looking at usage patterns during a session, it is interesting to know how often and when students use ProF.

We discusses also that we could also look at the usage patterns of staff members.


Finally, we agreed that the usage of the accompanying website at prof.ivtg.nl. I worthwhile to investigate, using standard web analysis. The problem then is that this website is public so we cannot link it to individual students.

Technical issues

With the technical subgroup (Robert, Peter, Erik and Jeroen) we discussed how we could log the usage in such way that these patterns can be studied.  For this, we have to know which student during which session views at what point of time what pages with what options selected.

Possible options are to use the web access log, to create a special table in the ProF database, to use CAM schema. We decided, however, to look into the open source web analytics tool Piwik. It appears feasible to link the ProF application to Piwik. The advantages are that the logdata is separate from the ProF application, that Piwik allows for user-defined data (such as the user-id we need for linking to progress test data), that Piwik offers a dashboard and API that we could use, in future, to visualize patterns to students and staff. We decided to perform a desk research and to discuss the findings in the first week of April.

Future directions

During the meeting we also looked forward to future directions. An important concern is that usage data alone is not enough to know why a student is using ProF in a certain way. Interviews and observations (with think-aloud protocols) with students and study advisers would be needed to find out more about this. Moreover, to measure the effect of using ProF in a certain way needs longitudinal study. The patterns that we hope to find in this project might help us to set up detailed research projects.

The results of this project could also lead to changes in the ProF system itself. If it appears that some pages are never used, we might decide to remove those. It could also mean that the instruction of using ProF has to be improved.

We also speculated at how results might be presented to users. For now it is very unclear how students could profit themselves, but Guido indicated that he would very much like to see the ProF usage pattern of students that visit him about progress test problems.

Welcome

Welcome to our project blog.  ProfAnalytics is a project of Maastricht Unversity funded by the Surf innovation program Learning Analytics.  The project runs from March till October 2012.


In this project we relate usage data of the already existing feedback system for progress testing, ProF, with the results that medical students obtain in these progress tests. We want to distill whether a certain type of usage of the system is related to better or worse results in the progress test. This knowledge can lead to advice to students, study advisers and academic organisations. 


To be able to achieve this, we have to change the ProF system in such way that it can collect the required data. Next, data will be collected during a period of two months. During the analysis phase, techniques from data mining and statistics are applied to search for possible links between usage patterns and results.  Finally, the patterns and models found will be translated, whenever possible, into practical advises. Follow-up projects will be needed to further investigate how these patterns can be visualized interactively to students and teachers.


The project team consists of: Jeroen Donkers, Arno Muijtjens, Jean van Berlo, Daniëlle Verstegen, Robert Peperkamp, Guido Tans, Bill Wrigley, Eric Sol, and Peter Bex.

A screenshot of the Prof system.