SeeYouLater

Native, low-latency mouse control with hand gestures or iPhone TrueDepth gaze tracking.

Important

SeeYouLater is experimental software that can generate real mouse input. Start in diagnostics mode, keep Ctrl+Alt+G available as the emergency lock, and do not use it where an unintended click could cause harm or data loss.

SeeYouLater is a native C++20 application for Windows 11. The hand controller combines YOLO person detection with MediaPipe-style palm detection and 21-point hand landmarks. A separate Windows/iPhone pair controls the pointer from TrueDepth gaze and intentional blinks. Python is not used at runtime or for the desktop UI.

Highlights

• Local USB-camera processing; camera frames are not uploaded or saved.
• Right-hand pointer control, left/right click, drag, multi-monitor mapping and calibration.
• CPU, DirectML and TensorRT-RTX inference backends with a safe CPU default.
• Automatic preservation and regeneration of incompatible TensorRT caches.
• Latest-frame capture architecture that avoids building up input latency.
• One Euro filtering, gesture hysteresis and click-point locking for stable input.
• Native Windows tray application, diagnostics view and per-user autostart.
• Optional iPhone TrueDepth gaze controller that sends numeric tracking data, never video.
• Portable x64 build with no Python, Visual Studio or separate MSVC Runtime requirement.

Requirements

Hand control

• Windows 11 x64.
• A USB or built-in camera. A stable 720p/30 FPS camera is recommended.
• CPU mode works without a dedicated GPU.
• DirectML requires a compatible Windows graphics driver.
• TensorRT-RTX requires a compatible NVIDIA RTX GPU and driver.

Eye control

• Windows 11 x64.
• An iPhone with a TrueDepth front camera and iOS 17 or newer.
• A Mac with Xcode for installing the included SwiftUI app.
• The iPhone and PC on the same trusted private network.

Quick start from a portable ZIP

Use a release ZIP when one is available, or create one with the build instructions below.

1. Extract the complete archive to a permanent directory such as C:\SeeYouLater. Do not run the application from inside the ZIP.
2. Connect the camera and allow desktop applications to access it in Windows privacy settings.
3. Run Start-Diagnostics.cmd. This mode performs recognition but cannot move or click the mouse.
4. Confirm that the camera, hand skeleton and inference backend are healthy.
5. Exit from the tray icon, then run Start-Control.cmd.
6. Hold an open right palm for about 0.45 seconds to enable gesture input.

The distributed binaries are currently unsigned, so Windows SmartScreen may show an "Unknown publisher" warning. Verify the archive checksum before choosing Run anyway.

Launcher commands

Start-Diagnostics.cmd
Start-Control.cmd
Start-Diagnostics.cmd -GpuBackend directml
Start-Control.cmd -GpuBackend tensorrt
Start-Diagnostics.cmd -RepairCache

The launcher verifies Windows 11 x64, required DLLs, model SHA-256 hashes and duplicate processes before startup. -RepairCache moves generated GPU caches into %LOCALAPPDATA%\SeeYouLater\CacheRecovery; it does not delete configuration or calibration.

Hand gestures

Gesture	Action
Open right palm for 0.45 s	Enable gesture control
Move the index finger	Move the pointer
Touch thumb and index finger	Left click
Hold thumb and index finger together	Drag
Touch thumb and middle finger	Right click
Hold a fist for 0.55 s	Disable gesture control
Press Ctrl+Alt+G	Immediately lock or unlock all synthetic input

The click target is frozen as the fingers begin to close, before the click is emitted. CURSOR LOCKED in diagnostics confirms that hand movement can no longer pull the pointer away from the target. During a drag, movement follows the palm center. Losing the hand immediately releases any held mouse button; a longer loss disables gesture control.

Calibration and configuration

Open the tray menu and select Calibrate. Point at the comfortable top-left, top-right, bottom-left and bottom-right limits of the working area, pressing Space at each point. Press Esc to cancel.

Configuration, logs, calibration and generated caches live in:

%LOCALAPPDATA%\SeeYouLater

Pointer tuning can be changed in config.ini and takes effect after restart:

cursor_smoothing_ms=28.0
pointer_min_cutoff=0.72
pointer_beta=2.50
pinch_arm_ratio=0.70
pinch_confirmation_ms=65

• Increase cursor_smoothing_ms for a smoother but more delayed pointer.
• Increase pinch_arm_ratio to lock the click target earlier.
• Change values in small increments and validate them in diagnostics mode.

The normal control mode enables per-user Windows autostart by default. It can be disabled from the tray menu without administrator rights.

Inference backends

Backend	Intended use	Notes
cpu	Maximum portability	Default; no GPU provider setup or generated GPU cache
directml	Cross-vendor GPU acceleration	Depends on the installed Windows graphics driver
tensorrt	Lowest latency on compatible NVIDIA RTX systems	Provider preparation and the first model compilation can take longer

If TensorRT reports an incompatible serialized engine, SeeYouLater preserves the old cache, rebuilds it once and falls back to CPU if the retry still fails. A backend override applies only to that launch; diagnostics displays the backend actually used by the vision pipeline.

iPhone TrueDepth eye control

The eye-control mode is deliberately separate from the USB-camera application. The iPhone estimates gaze, head pose and eyelid openness locally and sends only numeric packets to the Windows receiver. A deliberate two-eye closure performs a left click after calibration.

.\dev.cmd gaze-diagnostics
.\dev.cmd gaze-test
.\dev.cmd gaze

Installation, positioning, two-display calibration and safety details are documented in docs/EYES.md. The UDP protocol is not encrypted or authenticated; use it only on a trusted private LAN and allow the Windows firewall rule only for private networks.

Build from source

Install Visual Studio 2022 or newer with Desktop development with C++, CMake 3.28+ and Ninja. Windows PowerShell 5.1 or PowerShell 7 can run the scripts.

The shortest development workflow is:

.\dev.cmd                 # incremental build and safe diagnostics
.\dev.cmd control         # incremental build and real gesture input
.\dev.cmd test            # build and run all tests
.\dev.cmd test -Clean     # clean rebuild and all tests

The script discovers the Visual Studio toolchain and downloads pinned, SHA-256-verified dependencies on first use. Manual commands are also supported:

.\scripts\bootstrap.ps1
cmake --preset windows-release
cmake --build --preset windows-release
ctest --preset windows-release
cpack --config .\out\build\windows-ninja-release\CPackConfig.cmake

The portable archive and its .sha256 file are created in the repository root. Runtime models, downloaded NuGet contents, build directories, logs and archives are intentionally excluded from Git.

To create the separate Windows/iPhone eye-control development kit:

.\scripts\package-eyes.ps1

Architecture

flowchart LR
    Camera["USB camera"] --> MF["Media Foundation capture"]
    MF --> Latest["Latest frame only"]
    Latest --> YOLO["YOLO person detection"]
    Latest --> Hands["Palm + 21 hand landmarks"]
    YOLO --> Gestures["Gesture state machine"]
    Hands --> Gestures
    Gestures --> Input["Windows SendInput"]

    iPhone["iPhone TrueDepth"] --> Metrics["Gaze / head / eyelid metrics"]
    Metrics --> UDP["Private-LAN UDP"]
    UDP --> Gaze["Calibration + gaze controller"]
    Gaze --> Input

Loading

Main directories:

src/app/          hand-control Windows application
src/capture/      Media Foundation camera capture
src/inference/    Windows ML / ONNX Runtime pipeline and cache recovery
src/vision/       landmarks, filtering and gesture state machine
src/gaze/         Windows TrueDepth receiver and gaze controller
ios/              native SwiftUI TrueDepth sender
tests/            C++ and PowerShell regression tests
scripts/          dependency bootstrap and packaging

Troubleshooting

Symptom	What to do
No camera frames	Close other camera apps, check Windows camera privacy settings, reconnect the camera and restart the PC if the driver is stuck
Skeleton is visible but the pointer does not move	Open the safety lock with Ctrl+Alt+G, then hold an open right palm until control activates
CURSOR LOCKED appears	This is expected while a click is being confirmed
TensorRT cannot deserialize an engine	Run Start-Diagnostics.cmd -RepairCache, then retry diagnostics
Launcher reports a missing DLL or invalid model	Extract a fresh complete ZIP; do not copy only the EXE
Development build is already running	Exit the existing instance from its tray icon before rebuilding

When filing a bug, attach the diagnostics text and the relevant tail of %LOCALAPPDATA%\SeeYouLater\SeeYouLater.log. Remove usernames, local IP addresses or other private data before posting.

Privacy and security

• Hand-camera inference is local; SeeYouLater does not intentionally store or transmit frames.
• Eye tracking sends numeric data over unauthenticated UDP and must stay on a trusted LAN.
• Mouse injection is restricted by the normal Windows integrity model; the application does not request administrator privileges.
• Model and dependency downloads are pinned and checked with SHA-256 before installation.
• Report security issues according to SECURITY.md, not in a public issue.

Contributing

Bug reports and focused pull requests are welcome. Read CONTRIBUTING.md before changing behavior that affects mouse safety, calibration, inference or packaging.

License and third-party software

Project source code is licensed under the GNU Affero General Public License v3.0. Third-party models, libraries and generated provider components retain their own licenses; see THIRD_PARTY_NOTICES.md and models/README.md.

In particular, Ultralytics publishes YOLO26 under AGPL-3.0 and an Enterprise option. Review the applicable Ultralytics terms before commercial use or redistribution of a portable build containing yolo26n.onnx.