VTON Footwear: Markerless AR Shoe Try-On

VTON Footwear is my final-year Computer Science research project at NSBM Green University: an Android augmented reality (AR) application that lets users preview virtual shoes on their feet, estimate foot length, and receive a shoe-size recommendation using only a compatible smartphone camera.

Unlike marker-based solutions, it requires no printed calibration target, external depth sensor, or cloud inference. The research explores a difficult engineering question: can a mid-range Android phone deliver a useful combination of markerless tracking, physically meaningful sizing, stable shoe placement, and convincing self-occlusion?

My role: Independent researcher and developer
Project type: Final Year Research / Android AR Prototype
Research period: 2026-01 to 2026-09 Platform: Android (ARCore-compatible devices)

What the Application Does

Markerless foot tracking: Detects ankle, heel, and toe landmarks with MediaPipe Pose, processed directly on the phone.
AR footwear visualization: Positions and scales 3D shoe models over the tracked feet using Unity and ARCore.
Foot-length estimation: Projects image-space landmarks into 3D using camera intrinsics, ARCore planes, and gated depth information.
Calibrated size recommendations: Converts estimated foot length to EU, UK, US, and Mondopoint sizes using an affine calibration and ISO/TS 19407 conversion relationships.
Temporal stabilization: Uses quality gates, tracking-state logic, a One Euro filter, and Kalman-based depth/distance refinement to limit jitter.
Occlusion-aware rendering: Uses a custom analytic leg-occlusion method to hide virtual shoe fragments that should appear behind the wearer's lower leg.
Privacy-first execution: Runs inference locally, without sending foot images or landmarks to a remote service.

The Research Challenge

A virtual shoe overlay is only convincing if it gets three things right: where the foot is, how large it is, and what should appear in front of the shoe.

Each is harder on a regular smartphone than it first appears. A 2D landmark detector does not directly provide metric 3D coordinates. Frame-to-frame confidence and pose changes introduce visible jitter. ARCore depth can be unreliable around the silhouette of a person's leg, while whole-person segmentation cannot distinguish the foot from the leg for self-occlusion.

Instead of treating those problems as independent effects, I built a hybrid pipeline connecting computer vision, AR geometry, temporal filtering, calibration, and custom shader rendering.

How the System Works

Camera + pose inference: Capture frames on Android and run the MediaPipe Pose Landmarker locally to obtain heel, toe, and ankle landmarks.
Quality validation: Reject implausible, low-confidence, duplicated, or frozen landmark tracks before placement.
3D coordinate recovery: Use camera intrinsics and ARCore plane raycasts; accept raw-depth refinements only when they pass confidence and geometry checks.
Temporal filtering: Stabilize estimated positions with moving averages, adaptive One Euro filtering, and Kalman-assisted estimation.
Sizing + registration: Apply a ruler-based affine calibration to the measured landmark span, map it to commercial sizes, and scale the virtual shoe accordingly.
AR compositing: Render the footwear with a mobile-friendly custom shader and a ray-cylinder model of the lower leg for self-occlusion.

Key Engineering Contributions

1. Metric foot measurement without a marker

ARCore geometry supplies the world-space reference, while a fitted affine calibration corrects the systematic difference between the MediaPipe landmark span and the physical heel-to-toe length. This is more appropriate than simply adding a fixed number of millimeters to every foot.

2. Custom self-occlusion beyond standard depth masks

During evaluation, AR environment depth and whole-person segmentation did not provide a reliable foot-leg boundary at the tested distance. I implemented an analytic ray-cylinder occluder in the rendering pipeline as a geometry-based alternative, avoiding dependence on a clean depth mask for that boundary.

3. Evidence-driven debugging and robustness

The system was refined through device-based experiments, diagnostic overlays, automated checks, and regression fixtures. Failure cases were used to revise the measurement and rendering approaches rather than being hidden behind visual demos.

Research Results

Evaluation	Result	Interpretation
Affine calibration RMSE	3.35 mm	In-sample fit across five ruler-measured participants
Calibration maximum residual	4.01 mm	Maximum error within the five-point calibration sample
Calibration R²	0.9767	Fit between measured landmark spans and ruler measurements
Leave-one-out MAE	5.42 mm	Out-of-sample estimate; slightly above the 5 mm research target
Automated checks	142 cases	Engine-independent test fixtures reported in the final thesis

Affine foot-length calibration graph showing ruler-measured foot lengths, fitted regression line, confidence band, and five subjects

Calibration evidence from the final thesis. The training-sample fit is promising, but the sample is too small to claim general-device or general-population accuracy.

Prototype Screenshots

The images below are cropped from actual Android prototype screenshots, showing the virtual footwear overlay without exposing the test participants' faces.

Markerless AR footwear try-on showing tracked virtual sneakers on the user's feet — Foot-tracked 3D sneaker overlay on an Android device.

Second research test showing virtual shoes aligned with a participant's feet — Another test subject demonstrating footwear registration and scaling.

Screenshots were captured with research diagnostics enabled; those debugging overlays are not intended as final consumer-facing UI.

Technology Stack

Area	Technology
Application & 3D engine	Unity 6, C#
AR tracking & geometry	AR Foundation, Google ARCore
On-device pose estimation	MediaPipe Pose Landmarker, TensorFlow Lite GPU delegate
World-space reconstruction	Camera intrinsics, plane raycasting, raw-depth trust gating
Tracking stabilization	Moving average, One Euro filter, Kalman-based refinement
Footwear visualization	3D GLB/GLTF models, Unity Universal Render Pipeline
Occlusion & shading	Custom HLSL/URP shader, analytic ray-cylinder intersection
Sizing logic	Affine calibration, ISO/TS 19407 conversion relationships
Target platform	Android, with on-device processing

What I Learned

This project required more than building an AR interface. I worked across real-time computer vision, coordinate transformations, 3D graphics, shader development, signal filtering, measurement calibration, Android deployment, automated testing, and experimental research design.

The most important lesson was to validate each subsystem against actual device behavior. A convincing-looking overlay is not proof of millimeter-level accuracy, and a working rendering technique is not automatically reliable under different camera geometries.

Research Scope & Limitations

This is an evaluated research prototype, not a production-ready footwear fitting or medical measurement tool. Device testing was performed on one mid-range Android handset, with five participants in the primary calibration set. Broader hardware validation, a formal usability study, and instrumented frame-rate benchmarking remain future work. The leave-one-out measurement error was also slightly above the original 5 mm target.

Potential extensions include larger and more diverse calibration studies, improved foot-specific landmark and segmentation models, performance profiling across Android devices, and richer shoe-model fitting for width and instep shape.