• Project name
  • Handle
• Team members / who will do what
  • Aadarsh Jha, Raahul Natarrajan
• Abstract/problem definition
  • Motivation: Continuous hand gesture recognition (HGR) is essential in human-computer interaction, with a specific application in Augmented, Mixed, Virtual, and Extended Reality systems so as to allow users to engage in digital and virtual assets in an immersive fashion.
  • Problem: In research, Deep Learning applied to continuous HGR is a new problem, and the lack of real-world datasets presents a challenge. Thus, accurate models in deployable, productionized systems is not yet at optimal performance due to lack of model exploration and data diversity.
  • Goal: The goal of this project can be described in three-fold: 1) engage in a novel study of varying and proposing new models of deep learning in making HGR more accurate compared to baseline results; 2) diversify our models through training and evaluation on several types of open-source datasets; and 3) produce an end-to-end application to allow the user to, in real-time, perform HGR on custom data.
• Approach you will take (or at least an initial starting point)
  • To achieve Goal [1], we plan on finding existing deep learning frameworks that have baseline results for the task of HGR (ResNeXt-101, C3D, ResNet-50, etc).
  • Then, on understanding different frameworks and how they compare, we will make variational changes and experiment with their end performance.
  • We will source our gesture dataset from different sources that account for four problems as aforementioned, as best as possible:
  • Iterate training process, exploring tuning hyperparameters and making architectural changes until competitive performance is achieved.
  • Create a deployable application, where real-time video is classified with hand detection signals.
• Data you will use / how you will get it (publicly available? Permission? How will you collect the data if you’re collecting it yourself)
  • http://www.nlpr.ia.ac.cn/iva/yfzhang/datasets/egogesture.html
  • https://gibranbenitez.github.io/IPN_Hand/
• Success metrics – how will we know this project went well?
  • Results, Inference
• Timeline/milestones
  • On Presentation
• References
  • Build Up

Note:

In class presentations – so no written report, just a few slides:

• Aim for 8 minutes per team presentation
• 6 minutes presentation time + 2 min discussion