What are the Benefits of Meta’s Surface Touch Typing Technology that Mark Zuckerberg Praised?
Can you believe that you can now type on any flat surface just by touching it? It’s true! Researchers are working on developing surface touch type technology, and Meta has recently achieved a significant breakthrough in this field.
In a report by The Verge, Mark Zuckerberg, CEO of Meta, revealed an impressive typing speed of 100 words per minute (WPM) while wearing a virtual reality (VR) headset. Even more remarkable is Meta’s claim that it can turn “any flat surface” into a virtual keyboard capable of speeds of up to 120 WPM. This breakthrough represents a significant leap from Meta’s previous technology, as evidenced by their 2020 “PinchType” method, which averaged just 12 WPM. However, in the same year, their “touch typing” reached an average speed of 73 WPM.
Surface Touch writing technology
Meta’s latest development demonstrates its dedication to developing text input methods for VR and Augmented Reality (AR) environments. According to Meta’s blog. groundbreaking text decoding technology that enables touch writing on a flat, non-instrumented surface. This method eliminates the need for physical keyboards or capacitive touch interfaces. Touch writing is based on hand movement recorded with hand tracking technology. This motion data is then extracted directly into text characters, resulting in a seamless and efficient typing experience.
Meta uses a temporal convolutional network that acts as a motion model that converts hand motion—a set of hand position features—into text input. One of the key challenges for Meta’s researchers was to take into account the irregular writing movement caused by finger drift, since haptic feedback from physical keys has not been obtained. To solve this, the company integrated the language model as a text prior and used ray tracing algorithms to intelligently combine the motion and language models. This fusion enables accurate decoding of text from both ambiguous and irregular hand movements.
To validate their approach, Meta collected a dataset from 20 touch typists and ran various benchmarks on their model, including touch-based text extraction and a traditional physical keyboard. The results speak volumes: their proposed method utilizes continuous hand placement data to outperform contact-based techniques in terms of text decoding accuracy. An offline study showed parity with typing on a physical keyboard, achieving a speed of 73 WPM with an impressive 2.38% uncorrected error rate.
