An ARKit app to help people with learning disabilities

CSCI-5413 Augmented Reality Project using ARKit

Team members:
Shivendra Agrawal 
Saumya Sinha
Shayon Gupta
Daniel Szafir (Professor)

Overview

We developed a proof-of-concept augmented reality ARKit app that can parse text in the wild and visually represent it for the ease of reading. We also built a basic text editor on top of it designed again around ease of reading. [Github Link]

Preparing ARKit

Spatial Mapping: First, spatially map the surrounding environment. The yellow dots represent feature points indicating that ARKit is perceiving and mapping the environment.


Parsing and Visualizing Text in a Scene

Text Recognition and Visualization: EduAR utilizes the LSTM mode of Tesseract OCR to read text from paper. It visualizes this by displaying 3D text on a floating plane in AR, anchored to the real-world text location.


Visual Aide Text Editor

Interactive Text Editing: The Visual Aide Text Editor allows learners to generate visually anchored 3D text for all texts in a scene. Users can tap desired text segments to input visual aids.

Text Editor Features

  • Standard Options: Copy, share, and define text.
  • Unique Read Mode: Focus on one paragraph, sentence, or word at a time, with swipe navigation.
  • Real-Time Font Size Adjustment: Modify font size on-the-fly for convenience.


3D Objects Integration

Enhancing Visual Learning:

  • Double Tap to Visualize: Users can double-tap anywhere on the screen to spawn a 3D object on a detected plane.
  • Object Interaction: The spawned 3D objects can be rotated and resized using pinch gestures (movement is restricted).
  • Object Database: Objects are selected from a predefined database and are contextually linked to the scanned text, aiding visual learners in grasping concepts better.


Getting Started

  1. Download the Code: Clone the repository to your local machine:

    git clone https://github.com/ShivendraAgrawal/EduAR.git