Chinese smartphone giant Huawei has launched a new Android app that leverages AI tools such as image recognition and optical character recognition(OCR)to translate popular children’s books into sign language.
StorySign was developed in a collaboration between Huawei,the nonprofit European Union for the Deaf,Penguin,and animation gurus Aardman.
The app is available to download from Google Play and Huawei’s own AppGallery in 10 European markets from today.While it officially works on most phones with Android 6.0 and over,Huawei said it is optimized for its own AI-infused phones,such as the Huawei Mate 20 Pro.
How it works
Given that sign language varies from country to country,the app asks you to choose your preferred sign language variant. At launch,StorySign will support sign language in British,French,German,Italian,Spanish,Dutch,Portuguese,Irish,Belgian Flemish,and Swiss-German.
Above:Select your sign language
When you’re in the main section of the app,the first thing you’ll probably notice is that there is only one book supported at launch—Where’s Spot?,the classic children’s story published by Eric Hill in 1980.
And this hints at one of the app’s core problems—it feels like it has launched a little prematurely.Unless you have a physical copy of Where’s Spot?,you won’t be able to use this app.
Above:StorySign:There is only one book available at launch
If you do have the book,you or your kid can hold your phone up to the pages,and the avatar—called Star—will sign out the words in real time as you flip the pages.
Above:Where’s Spot?Huawei’s StorySign app
Using image recognition technology,the app is able to detect words even if the phone is positioned at an awkward angle in relation to the book,while OCR enables greater accuracy,according to Huawei.
“At Huawei,we believe in the power of AI and that technology can make a positive difference in the world,”said the company’s Western Europe CMO,Andrew Garrihy.“We created StorySign to help make it possible for families with deaf children to enjoy an enriched story time.”
Potential
On the surface,the StorySign app holds a great deal of potential—there are some 460 million people with“disabling hearing loss”globally,according to the World Health Organization(WHO),of which 34 million are children.Against that backdrop,an app such as StorySign could offer millions of early learners the chance to enjoy books before they are able to read standard written text themselves.But it would be infinitely more useful if it worked with any book in real time.
Indeed,a scenario where you could translate anything from the Gruffalo to The Wonky Donkey into sign language would be a pretty powerful display of AI.Offering support for just one book at launch feels a little scant,and the fact that it has to roll out support on an incremental book-by-book basis makes it just a little underwhelming.
However,alongside a bunch of other recent initiatives,StorySign does provide further evidence of how technology could help bridge the gap between the hearing and non-hearing worlds.
Earlier this year,students from New York University demonstrated a prototype app that meshes augmented reality(AR)and machine learning to translate both ways between sign language and written English.And last year the University of California,San Diego developed a glove that transforms sign language into text.
StorySign fits into that broader exploration—and lays a foundation for future developments.
“We’re very hopeful that it will make a significant impact in the deaf community,helping more deaf children learn how to read at the same level as hearing children,”added Mark Wheatley,executive director at European Union of the Deaf.“We also hope the launch of StorySign will support a wider conversation about ensuring equality in every aspect of their lives for deaf people across Europe.”
