event
Synthux Hackathon 2024

Am I a ditto on live

Voice-activated live filter to enhance the virtual experience of socializing
Unreal Engine
Kinect
Camera feed
Text to Speech
prompt

Coaches & collaborators

Roey Tsemah
institute
Design Academy Eindhoven
Creating to enhance virtual socializing and education experiences real-time, the voice-activated live filter can do things below: 1) set up customized virtual character before you go live (you can go wild as being a shoe, designer chair, or customized based on your own appearance). 2) detect certain words to make related changes to your character 3) detect certain emotions to alter voice for a short second. 4) detect name/pop culture terminology to show images/explanations on screen. Initially tailored for design podcasts, it has the potential to expand to game streaming, online courses,etc.
No items found.
the tech

How does it work?

Starting with rigging in C4D, the rigged object (mango in this case) is transported into Unreal Engine, tracks a human face, and shows the same human expression on it. After that, the speech-to-text and word capture functions will help recognize certain words during speech. Nest, it come up with word-related changes to the virtual character based on the algorithm. However, considering the limited timeframe, I switched to Spark AR to create a rough prototype to see what it looks like before getting deep into Unreal Engine. In spark AR, I made filters with face distortion, floating objects, face masks, and voice distortion. Besides, I also created a sample podcast content talking about "trends, social belong, and stanley cups" that I will continue developing to test the tech part.
Open source code
Click here to grab this project's code
Unreal Engine
Game engine, which is a type of software that provides a framework for creating video games
Kinect
Line of motion sensing sensor
Camera feed
Text to Speech
About the instrument
we're using cookies to run our site. privacy policy.