Skip to content
Back to work

MouseKey

Role
  • Software Engineer
Timeline
  • In progress
Skills
  • Swipe decode
  • Windows overlay
  • On-device ML
  • Privacy

Tech Stack

  • C++
  • Windows
  • ONNX Runtime
  • Python
  • CTest

Coming soon

Coming soon

Coming soon

Coming soon

Overview

This is MouseKey, a personal offline English swipe keyboard for Windows. You drag across a floating QWERTY overlay and it decodes a word into the app that has focus.

It stays on the machine. No text or gesture leaves. What runs today is the geometric decoder and the overlay. A public neural release is still blocked.

Contributions

  • Built the floating DPI-aware QWERTY overlay and the geometric swipe decoder.
  • Kept low-confidence or unmatched gestures as suggestions instead of injecting them.
  • Stored personalization locally, with a pause switch and no upload path.
  • Left the neural runtime and training pipeline behind explicit public-release blockers.

Challenges

Windows input is the constraint. SendInput will not reach an elevated app if MouseKey is at a lower integrity level, and some games and secure desktops reject synthetic Unicode. I treated those as unsupported rather than fighting every surface.

The other constraint is what I am allowed to ship. The heuristic decoder still loads legacy word and bigram files whose provenance is not approved for a public package. The training pipeline can build lexicon and model assets with a ledger. Until those replace the legacy files, there is no public build.

Design Decisions

Decode is geometric and local. A live preview stays on the overlay. A low-confidence or unmatched swipe becomes a suggestion, not an automatic inject. If mouse capture drops, the gesture cancels.

Learning is opt-in and stored in a local user model. Nothing is uploaded. The collector for prompted traces is a separate binary so ordinary typing is never recorded.

I used C++ and a DPI-aware overlay because this has to sit on top of other apps, not inside a browser.

Reflection

The bar is whether I can swipe a word into the app I am already in, without an account and without a server.

A public neural keyboard is a later gate. Provenance and traces come first. Until then this stays a local tool.