Automatic Detection of Reflective Thinking in Mathematical Problem Solving based on Unconstrained Bodily Exploration

Temitayo Olugbade, Joseph Newbold, Rose Johnson, E Volta, P Alborno, R Niewiadomski, M Dillon, G Volpe, N Bianchi-Berthouze
in Journal article


For technology (like serious games) that aims to deliver interactive
learning, it is important to address relevant mental experiences such as
reflective thinking during problem solving. To facilitate research in this
direction, we present the weDraw-1 Movement Dataset of body movement sensor
data and reflective thinking labels for 26 children solving mathematical
problems in unconstrained settings where the body (full or parts) was required
to explore these problems. Further, we provide qualitative analysis of
behaviours that observers used in identifying reflective thinking moments in
these sessions. The body movement cues from our compilation informed features
that lead to average F1 score of 0.73 for automatic detection of reflective
thinking based on Long Short-Term Memory neural networks. We further obtained
0.79 average F1 score for end-to-end detection of reflective thinking periods,
i.e. based on raw sensor data. Finally, the algorithms resulted in 0.64 average
F1 score for period subsegments as short as 4 seconds. Overall, our results
show the possibility of detecting reflective thinking moments from body
movement behaviours of a child exploring mathematical concepts bodily, such as
within serious game play.