MouseKey
- Role
- Software Engineer
- Timeline
- In progress
- Skills
- Swipe decode
- Windows overlay
- On-device ML
- Privacy
Tech Stack
- C++
- Windows
- ONNX Runtime
- Python
- CTest
Coming soon
Coming soon
Coming soon
Coming soon
Overview
This is MouseKey, a personal offline English swipe keyboard for Windows. You drag across a floating QWERTY overlay and it decodes a word into the app that has focus.
It stays on the machine. No text or gesture leaves. What runs today is the geometric decoder and the overlay. A public neural release is still blocked.
Contributions
- Built the floating DPI-aware QWERTY overlay and the geometric swipe decoder.
- Kept low-confidence or unmatched gestures as suggestions instead of injecting them.
- Stored personalization locally, with a pause switch and no upload path.
- Left the neural runtime and training pipeline behind explicit public-release blockers.
Challenges
Windows input is the constraint. SendInput will not reach an elevated app if MouseKey is at a lower integrity level, and some games and secure desktops reject synthetic Unicode. I treated those as unsupported rather than fighting every surface.
The other constraint is what I am allowed to ship. The heuristic decoder still loads legacy word and bigram files whose provenance is not approved for a public package. The training pipeline can build lexicon and model assets with a ledger. Until those replace the legacy files, there is no public build.
Design Decisions
Decode is geometric and local. A live preview stays on the overlay. A low-confidence or unmatched swipe becomes a suggestion, not an automatic inject. If mouse capture drops, the gesture cancels.
Learning is opt-in and stored in a local user model. Nothing is uploaded. The collector for prompted traces is a separate binary so ordinary typing is never recorded.
I used C++ and a DPI-aware overlay because this has to sit on top of other apps, not inside a browser.
Reflection
The bar is whether I can swipe a word into the app I am already in, without an account and without a server.
A public neural keyboard is a later gate. Provenance and traces come first. Until then this stays a local tool.