A camera sees the finished letter. A motion-sensing pen sees something stranger and richer: a stream of accelerations and rotations, most of which is not writing at all. Turning that stream back into legible strokes is one of the hardest, and most useful, problems in handwriting technology.
When you write, the pen tip spends a surprising amount of time off the page, lifting between letters, gliding to the next word, hovering while you think. An inertial sensor inside the barrel records all of it indiscriminately. To rebuild the path the tip actually traced on paper, a system first has to answer a deceptively simple question at every instant: was the pen writing, or just moving?
Get that wrong and the reconstructed page dissolves into scribble, phantom lines connecting letters that were never joined, or real strokes erased as if they were hovers. Get it right and you can recover handwriting from nothing but motion, on any surface, with no special paper or screen underneath.
One model can't do two jobs well
The instinct is to train a single neural network to map raw sensor data straight to pen-tip positions. It works, but only up to a point, because touching and hovering are genuinely different physical regimes. A tip pressed to paper is constrained, damped, and relatively slow; a tip flying through the air between words is fast, free, and dominated by drift. Asking one model to be fluent in both means it ends up mediocre at each.
A 2025 study in Pattern Recognition by Florent Imbert and colleagues at IRISA / INSA Rennes takes a cleaner approach: a mixture of experts. Instead of one network, they train two specialists, one dedicated to the touching trajectory and one to the hovering trajectory, and combine them. Each expert only has to master a single regime, and each can be improved with context and examples tailored to its job. Splitting the problem along the seam where the physics actually changes turns out to reconstruct the written line far more faithfully than a single undivided model.
Why this matters beyond the lab
Reconstruction from motion is what lets a pen work on ordinary paper, the way people already write, instead of on a tablet that dictates posture and feel. It also preserves something a photograph throws away: the order and timing of every stroke. That temporal signal is the raw material for measuring how someone writes, not just what they wrote, which is exactly where handwriting analysis gets its depth.
Key takeaways
- Most of a pen's motion is not writing; separating touch from hover is the core challenge.
- A single model handles both regimes poorly because their physics differ.
- A mixture-of-experts, one network for touching, one for hovering, reconstructs the written line more faithfully.
- Motion-based reconstruction frees writing from special surfaces and preserves stroke order and timing.
For Vahini, this is the layer beneath everything. Today the analyser reads a photo of the finished page. The sensor pen adds the live motion view, and the better that motion becomes legible strokes, the more honestly we can score the twenty factors that make handwriting clear, consistent and confident.
Mixture-of-experts for handwriting trajectory reconstruction from IMU sensors
Florent Imbert, Eric Anquetil, Yann Soullard, Romain Tavenard. Pattern Recognition, vol. 161 (2025), 111231.
Read the paperSummary and interpretation are our own. We link the original work so you can read it in full; we don't reproduce it.