As indicated in previous blogs, we intended to study the data using Process Mining. We were able to transform the Piwik data into the XES format in several ways and to read the data into PROM6.1. The next job was to figure out which of the many analysis tools in PROM were the most suited for our data. After reading the book "Process Mining" by Wil van der Aalst (ISBN 978-3-642-19344-6), we decided for Fuzzy Mining (Günther and van der Aalst, 2007). This technique makes it possible to abstract processes from spaghetti-like traces, like the ones we produced in ProF. We generated datasets for three types of students: depending on their results at the progress test. However, the fuzzy miner was not able to generate a meaningful process from this. Wat we saw were pictures like:
These pictures were clear representations of the navigational structure of the ProF application - the arrows indicating roughly the preferred navigation routes. In short, this did help us not so much in understanding the data.
We then decided to switch back to R and analyze the data classically by counting, neglecting the order of the pages viewed.This, fortunately, produced more useful results. To summarize the results, the most important finding was that students with poor results (blue below) used ProF less often than others (red and green), but when they used ProF, they tended to stay longer in the application. They also looked a bit longer into score details.
Other findings were that too many students only watched a single page in ProF, that students did not use the cumulative view of their results too often, and that they did not study the detailed scoretypes enough.
During an expert meeting we discussed the findings and came up with a number of advices for improvement. The first one was that we need to change the opening page of ProF, so that students are stimulated to look further. Second, we need to develop instruction materials to teach students how to make better use of the possibilities of ProF.
The story does not end here. We intend to do a follow-up research in which we ask students their opinion of using the tool and its effectiveness. In the meantime we are continuing harvesting data using Piwik so that longitudinal studies will become able.
