Sign2Word AI App for the Deaf Learners
The seeds of our idea for an AI App for Deaf children started way back at our very first Young Coders MeetUp (YCM), in January 2019. Imran joined this event, our first deaf young coder who has been such an inspiration to us all. Our meetups are youth-led and focus on diversity, inclusion and community. After a period of agile development we decided that we want to explore Al and Machine Learning to celebrate the first MeetUp. In the spirit of collaboration, Imran and Femi, a founding member, buddied up together to engage in a workshop on Machine Learning for Kid using IBM Watson. As the afternoon progressed Imran told Femi about the difficulties that deaf people have with their literacy skills, so whilst they were progressing through the session they came to the conclusion that, perhaps with machine learning they could help other deaf students with this problem. So, making the most of the limited features, they built a quick prototype using fingerspelling signs - capturing still images through their laptops and going through the supervised learning of their model. However, they found that this was quite hard as some of the signs required movements, they were not just static signs.
After meeting up a few times, talking about their ideas to other members of the YCM, they were encouraged to take their idea to the iOS community, and were invited by Skillsmatter to present their ideas at the iOS Con 2019 in March. There they got some great advice from industry professionals on how they could use Swift to create the app on the iPhone and iPad. They managed to make some awesome contacts in the Swift community. Tim Condon, who had worked on the BBC iPlayer app and runs Swift workshops in the industry, was so impressed that he offered to help and later ran a two day workshop for the YCM, at the Tate Modern. Although they learned a lot about the uses of Swift and why it was so awesome for machine learning, they weren’t able to get much support in creating the machine learning side of the app.
However, it was great because a team of young coders got interested in the project. Over the first 6 months the YCM ran workshops on User Stories, building narratives, collaborative working and designing websites. This enabled the group to come together when they found out about the Longitude Explorers Prize.Femi was lucky enough to be invited by UAL Creative Computing Institute in July 2019, to be a guest participant on their Masterclass on Machine Learning for the Creative Industry. There, he learned all about neural networks and machine learning models. He was given some great advice from the facilitator, Marco Marchesi about the project idea. He was pointed towards Rebecca Firebrink and her amazing Wekinator project as well as being guided towards Dynamic Time Warping as a way to solve the problem of detecting moving signs. Femi was then able to explore Wekinator and dive deeper into machine learning throughout the summer, and he got a chance to contact Rebecca Firebrink for help with the idea. With her Wekinator project he made a very simple prototype of the app, able to determine a few distinguished signs from each other. In the meantime Imran helped to collect video clips of people signing individual words for data, to use for training a neural network.
In November Femi went to a talk by Professor Richard Harvey from Gresham University, an expert on artificial intelligence, machine learning and signal processing, and asked him about the approach that they should be taking towards the idea. He told Femi that Dynamic Time Warping is fairly experimental and not nuanced enough and advised that Hidden Markov Models would be a better way to go. He also said that this sort of project has been researched before and that the expert on computer vision, machine learning and Sign Language Recognition would be Professor Richard Bowden, University of Surrey.
Inspired by the Nesta Longitude Explorers Prize competition Femi and Imran invited other YCM coders to form a team to enter the Sign2Word AI app for the competition. What has been fantastic is that the YCM’s focus on collaborative working and knowledge sharing, has meant that an awesome team, each young person with unique strengths, knowledge and skills have joined forces and it’s been amazing. They have all been to the YCMs so have a shared experience and enjoy being together. Since the news came that they had got through to the Semi-finals, they have been meeting regularly, and after brushing up on his machine learning and AI skills, Femi delivered a knowledge sharing session to the rest of the group to get everyone on a level playing field. Imran is too old to join the competition but he still holds a central role of being the Product Client and representing the Deaf community. Mutsa has been amazing with her design skills - interpreting the user stories from Imran and translating them into wireframes so the Swift App can be designed to meet the needs of the users. Malaika has been collaging all the information and is building a website and managing the content. Nishka has been documenting the journey and giving support with the learning and looking into the mathematics linked to the project (Statistics & Linear Algebra). Thomas has been exploring the options for the build - so we can incorporate the neural network into the Swift app in the most efficient way.
We are currently at the phase of looking at research in the field of Computer Vision and understanding the complexities of Sign Language Recognition. Having made the initial contact with Academia and continuing to learn more about Machine Learning, our next stage is to follow up, seek mentoring and learn more about Hidden Makov models. We are really excited about collaborating together, being on a journey of discovery to make our Sign2Word AI App a reality - providing the very thing that Imran found missing as a young Deaf person as he struggled with written language.
Team members:
Femi
My name is Femi. I started coding when I was 8 and since then I've run coding workshops to hundreds of young people across the UK and globally. In February 2018 I was part the Young Coders Conference and together with some other young coders have founded the YCM. The YCM is an awesome place for young people to meet up collaborate and develop 21st century skills. The wide variety of topics covered means we get a taste of all aspects of coding, from web design to AR and VR. I love coding as it is really cool when you can make something that helps lots of people so easily. One awesome thing about the YCM is that it is youth-led meaning that we get to choose what we want to learn about. Also the diversity, inclusion and gender parity is amazing and really adds to the diverse young coders community.
Mutsa
My name is Mutsa and I am part of the Young Coders MeetUp. YCM is a place for anyone who is part of it, to learn, collaborate and grow. However, it provides something unique to each person. For me, Young Coders MeetUp has allowed my to find a new love for computing and a different perspective on the subject itself. Due to a different style of teaching in school, I viewed computing as a restricted subject where you really just had to follow a straight road, with maybe with a few bumps along the way. And I view myself as someone who just loves making something my own way and just being creative, so as a result I didn’t enjoy computing very much. So when it came to choosing my GCSE options, computing was surprisingly out of the running. However due to YCM, I have gotten a completely different perspective. Each week I learn something I had never known before from each mentor that comes in to give us a session. I have learnt that you can do a lot with coding and be as creative as you like using many different platforms. After each session I enjoy coming home and experimenting with the new skills I have learnt. YCM has made me really enjoy coding. In the future, I hope that many more kids will get to have an experience like YCM and way more people learn to code. A growing number of things in the world are becoming digitalised and will need people to create and code them. YCM puts you one step closer to becoming that person.
Nishka
Hi I’m Nishka. I enjoy coding as it is creative and has the power to transform the way we do things, for instance, through AI. When I was 8 we moved to the UK from India and I started learning Python at our local code club. I joined my school Robotics team as the coder – we went up to the nationals and then to the World Finals in Kentucky! Two years back I was invited to be part of the Young Coders Conference, and this opened up a whole new world for me. I met other young people like me and became part of the young coder community. I started running workshops to teach coding to other young people, especially those who aren’t lucky enough to have easy access – both in the UK and on my summer visits to India, and have so far run 40 workshops teaching around a thousand participants directly. It has taught me so much. In mid-2018 the idea of Young Coders Meet Ups was born and I took part in the development sessions. Other young people were invited and there were monthly meet-ups over the next six months. We learnt new ideas and concepts every time. Our community has grown stronger and we continue to work together, helping and supporting each other, and making a difference to the wider community.
Thomas
Hi, I'm Thomas R. My first experience with coding was when I was around 6 or 7 and had to use the Linux command line to get Minecraft working on the original Raspberry Pi. I did my first Linux installation when I was 7 and got into programming in Python through the Raspberry Pi and Minecraft. I started going to a Raspberry Jam where I met Femi. We set up our own Raspberry Jam with an aim to be inclusive towards people with autism and Tourettes. This slowly evolved into the Young Coders Conference where young coders from across the UK delivered coding workshops to the public in the Tate Modern. Many of the young coders from the YCC (including me) went on to start the Young Coders Meetup. We then met Imran, a deaf young coder and realised the challenges deaf young people face in their day-to-day life. Many deaf people have trouble translating their thoughts and ideas into English so we decided to use AI to make an app that translates sign language into English. Outside (sort of) of the YCM, I am also into electronics and physical computing and recently 3D printing. I have also been fascinated by the power of AI and neural networks and own so many Raspberry Pis I can now pretty much just pull one out of my drawer whenever I need one.
Malaika
Hi, my name is Malaika. I started coding when I was about 7, both at home with help from my dad (who is a professional coder) and at school. However, I was still interested in other aspects outside of what I was learning, and very curious about how things worked, and how they had been coded. I was then invited to the Young Coders Meetup, and was really interested in all the different subjects we covered, from Scratch to Microbit to Artificial Intelligence. I could then combine all I had learnt into fun and interesting projects, such as the racing game with lights I created. At first I was just coding for fun and for my benefit, but then I got to the the point where I was better than many others in my class. I found that I could come to school after a Sunday of YCM and help people in my class who found coding more difficult. I then had a new reason to code - not just for fun, but also to help others. Now, I want to take my coding beyond school and help communities which don’t have the same resources to learn how to code, or to aid a global problem such as global warming. Overall, I greatly enjoy coming here, both to learn new things myself and to help others learn.