Gesture Based Authentication System

UX & AI Class Project | Jan’25 - May’25

Project Overview

Project: A gesture based authentication system that allows users to seamlessly login using hand gestures without the hassle of typing in annoying captchas or double factor authentication.

Goal: Designing AI-driven, password less security that feels human.

Role: This was my final project for my UX and AI class

Location: New York

Problem Area

We live in a world where our identities are constantly being verified — logging into work accounts, university portals, or even recipe websites. Each time, we face the same digital gatekeepers: passwords, CAPTCHAs, and Two-Factor Authentication (2FA). Whether it’s searching frantically for your phone to enter a six-digit code, or squinting at a CAPTCHA that can’t decide whether an image contains a “motorcycle” or not, the process feels exhausting.

As a designer, I began to ask:

"If security is supposed to make us feel safe, why does it make us feel so frustrated?”

Problem Statement

How might we design an AI based authentication system that ensures security while reducing user friction?

Timeline

The project was divided into 2 phases

The project was divided into two big phases- The UX Research Phase and the AI Model Development Phase. In the Research Phase, I did user research on security authentication systems and users frustration points around it. I also brainstormed different solutions and prototyped the final solution. In the second phase, I researched about existing AI gesture based models and built on them for my final soution.

Research

To understand the emotional and functional pain points, I combined user interviews, sentiment analysis of online forums (like Reddit), and observation studies during login sessions.

I wasn’t just looking for what people struggled with — but how they felt when systems failed them.

Mapping problems faced by Users

Needing to enter 'one-time codes' frequently is annoying, especially when sites don't remember login information despite claiming to keep users logged in

Needing 2FA when the phone is in another room interrupts the user's focus and comfort

Having to enter a six-digit number every time to access applications is annoying

Rotating CAPTCHAs: Perceived as unnecessary and irritating.

Frustration arises from having to complete CAPTCHAs too often, especially when they fail on the first try. Multiple attempts are common (up to 70% needing a second try). This leads to closing the browser and restarting.

Account Locked: Experienced with a Facebook account being permanently locked due to 2FA issues, creating caution about choosing 2FA methods.

Losing your phone can prevent you from logging into anything that requires a code, including banking, social media, and even topping up a card to use laundry machines

Email OTP Preference: Generally prefers email OTP over SMS OTP.

CAPTCHA Replacement Desire: Prefers 2FA over CAPTCHAs, despite the noted drawbacks.

Needing to enter 'one-time codes' frequently is annoying, especially when sites don't remember login information despite claiming to keep users logged in.

Awareness that CAPTCHAs are becoming more difficult due to advances in AI and OCR, leading to a negative feedback loop.

Captchas have become obsolete

Irrelevant Security: The interviewee questions the need for CAPTCHAs on sites where security seems unnecessary (e.g., Chegg, recipe sites).

Annoyance with frequent automatic logouts, requiring repeated 2FA authentication, even on the same device.

Phone-Based Security: Believes phone-based 2FA (SMS OTP) is the most secure method because the phone is always with them.

App-Based 2FA (e.g., NYU Duo): While preferred over CAPTCHAs, still seen as tedious. The need to switch to a separate app (Duo Mobile) for approval adds an extra step.

SMS OTP Delays: OTP delays are a significant frustration.

Some users believe that 2FA and 6-digit codes are no longer secure and can be bypassed by attackers

Losing a phone can create a nightmare scenario where it's impossible to log into anything, even with the correct passwords, and can require getting a new phone number if the phone is lost

Affinity Mapping

Needing to enter one-time codes frequently is annoying.
Having to enter a six-digit number every time to access applications is annoying.
Needing 2FA when the phone is in another room interrupts focus and comfort.
App-based 2FA (NYU Duo) adds extra steps.
Frequent automatic logouts require repeated 2FA on the same device.
Side tab is logged in, but new tab won’t login.

Device Dependency & Risks

Losing a phone creates a nightmare scenario: can’t log into anything, even with correct passwords.
Losing a phone may require getting a new phone number to restore access.
Phone-based security makes users feel “locked out” of essential services (banking, social, laundry machines).
SMS OTP delays create frustration.
Account locked permanently due to 2FA issues → high risk.

CAPTCHA Pain Points

CAPTCHAs perceived as irrelevant on low-security sites (Chegg, recipe sites).
Awareness that CAPTCHAs are harder due to AI/OCR, leading to a negative feedback loop.
Frustration from frequent CAPTCHA failures (70% needing multiple tries).
Users sometimes abandon sessions (close browser/restart).
Rotating CAPTCHAs seen as unnecessary and irritating.
Desire for CAPTCHA replacement.
Belief that CAPTCHAs are obsolete.

Perceptions of Security

Some users believe 2FA and 6-digit codes are no longer secure (hackable).
Phone-based 2FA (SMS OTP) is perceived as most secure because the phone is “always with them.”
Preference for email OTP over SMS OTP.

Trust & Reliability Issues

Sites don’t remember login info despite “keep me logged in” options.
Annoyance with frequent automatic logouts.
SMS OTP delays undermine trust.

Emerging Themes

Over-friction in authentication → logins feel like a constant chore.

Device fragility → if phone is lost/dead, digital life is locked.

CAPTCHA frustration → seen as unnecessary, outdated, and ineffective.

Security paradox → users want safety but don’t trust current methods.

Reliability gaps → ldelays, failed CAPTCHAs, and unkept “remember me” promises create distrust.

Cognitive Offloading Map

To understand how to balance control between humans and AI in the authentication process, I used
the Cognitive Offloading Map .

Through this mapping, I explored how AI can handle repetitive verification tasks while humans provide emotional and situational input.  

AI’s role becomes one of contextual intelligence — learning when to trust, while humans remain in charge of final consent.

Ideation

Since this project is part of my UX and AI class, I needed to develop an AI-driven solution. The following ideation outlines the concepts generated for this solution.

Gesture Based Security Authentication System

Gesture-based security authentication system that allows users to verify their identity using intuitive gestures, reducing reliance on passwords and traditional two-factor authentication. By leveraging computer vision and motion detection, the system could recognize predefined gestures—such as peace sign, thumbs up, etc in a specific pattern.

Haptics Based Security Authentication System

Using advanced Haptic Rendering Models, the system generates personalized vibration sequences based on pressure, motion, and intensity, ensuring each user experiences a distinct pattern. To log in, the user must mimic the exact feedback sequence through touch gestures, much like repeating a rhythm in a memory-based game. This approach transforms authentication into an intuitive and engaging process, replacing traditional passwords with a tactile signature that is both secure and playful.

ML5.js (for gesture recognition and training models) WebHaptics API / Vibration API (for haptic feedback on supported devices)

AI-Based Adaptive Authentication System

An AI-driven contextual authentication system that automatically logs in users based on trusted locations, network patterns, and behavioral biometrics—eliminating the need for traditional 2FA while maintaining security. Using Federated Learning: Allows AI to learn user patterns across devices without compromising privacy. Gait & Keystroke Dynamics Recognition: Machine learning models analyze user-specific behavior patterns.

TensorFlow
Google Cloud Vision API for landmark recognition
Google Maps API / OpenStreetMap (for GPS-based detection)
WiFi Fingerprinting with ML (for indoor location tracking) TensorFlow / Scikit-Learn (for training the model)

Final Solution

Gesture Based Authentication System

Why Gestures?

My approach to this project stems from a strong interest in observational user research, specifically analyzing how individuals naturally interact with their environment and leveraging those insights to develop intuitive and user-friendly technological solutions. I think if you really get that, you can make tech way easier and less annoying. So, when I was brainstorming about security, I wanted to find ways to prove you're a real person and not a bot, but also that doesn't drive users crazy.

And, you know, as someone from Gen Z, we use gestures all the time to communicate – like the peace sign or thumbs up. Plus, I'm seeing gesture recognition pop up in more and more tech. So, I thought, 'Why not use gestures for security?' It could be kind of fun and playful, but also really secure, since robots can't really do human gestures convincingly. For me, it was a perfect match – solving the security problem while also making things more accessible.