AR-Based Tangible Programming Language

Yue Yuan --- Developer
Chen Xiang --- Developer
Yate Ge ---  Researcher
3D Designer
Visual Designer
UX Designer
US - China Hackathon Finalist
Augus 2018 


The Challenge

Coding, STEM, robotics; these have become buzzwords for what a 21st century education should be. How to make programming education into a more engaging experience and make it accessible to all kids, even the ones who cannot afford the huge budget for Programming Kit to have the equal chance to learn and make.


Our solution is developing an AR-based  programming language, Where kids can implement the programming logic and build the scenario by jointing colorful cards. Also,  immersive and interactive learning experience can be achieved with minimal budget.





In China, the changes in population patterns such as '4 + 2 + 1 family structure' and 'educational consumption upgrade' have made the overall demand for the early childhood education market increase rapidly, and more parents have a general demand for children's education.


The "nurturing care" and "game entertainment" have shifted to a more systematic and comprehensive "parent education" and "professional early education". Among them, the "children's programming education" market has received special attention.

Children's Programming Education Received Special Attention

Immersive Programming Learning Experience

Programming itself is a very complex subject, requiring a certain logic, abstraction, and mathematical foundation.


Compared with the traditional programming education based on graphical language learning, Creating an immersive experience is critical to programming learning. Learners guide characters through input code instructions, starting with zero threshold and increasing difficulty. The process contains both rich sensory experience and cognitive experience.

Augmented Reality Brings New Possibilities

New interaction methods


voice recognition

camera recognition + AR gestures

image recognition...

New medium

Combined with real-world spatial computing, rich in 3d content

Frame the Problem Statement 

How to Use AR Technology for Programming Learning?

Concept 1 - Programming for Waterdrop Adventure


Our hero is a raindrop. Kids can give instructions (via simplified programming language) to help raindrop move and choose different channels until arrive at one floating island. 

Three Floating Islands 

IF Succeed ...

( I can do it! (ง •̀_•́)ง

Three Floating Islands 

IF Fail ... 

( I can't ...  (ಥ﹏ಥ) 

Water droplets at different states

Sectional view

Concept 2  - Multi-modular Tangible Programming

Create a personalized immersive AR experience with programming cards. Design your own AR games, AR stories, interactive experiences. And share and communicate with your family or friends.

AR + Programming Blocks


Logic programming through physical logic cards
Set user interaction, data, logic, etc.


Build scenes from space and object cards and connect to logical cards


Interact with virtual content through handheld devices (screen interactive controls, voice, accelerometers...) 

  • Help the programming process with AR (using AR features to guide programming)

  • Create AR interactive experiments (sound, screen interaction, camera, accelerometer, etc.) with device features

  • Create multi-person AR interactive games together (multi-users create together and share experiences)

  • Interactive AI technology learning (encapsulation of AI algorithms, users experience machine learning and other ai technologies in ar)

  • Immersive scene package (users can build small physical experiments, simulate unmanned vehicles, robot control, shooting games, etc. according to different themes)

Create a personalized immersive AR experience with programming cards. Design your own AR games, AR stories, interactive experiences. And can share and communicate

Interaction Flow

Block Design

Use different color blocks to represent different functional modules

Control card

Entity card

  • Spatial physics experiment of multimodal interaction

  • Multiplayer interactive game

  • Agent control

  • AI technology learning

voice input


Flow-based programming is designed with block-based programming language.

Spatial programming with physical cards.

  • Data flow control, calculation, transfer, and execution through splicing

  • Each card takes the generated texture for image recognition

  • Visual design, using icons, colors, interface arrows to help semantic understanding

  • The hexagonal design is suitable for both splicing and easy expansion.

Users create new modules through hand-drawn and software platforms

voice input

Boolean inversion



Boolean inversion



Cards jointing

View help information

Virtual button

Cards interaction

© 2020  by  Scarlett  Wu - All rights reserved.