Facebook

'Image may contain… cat?' – now Facebook can talk you through your photos

‘Image may contain… cat?’ – now Facebook can talk you through your photos

<!–[if IE 9]><![endif]–>Cat with moustache.

Cat with moustache.
Photograph: GNM

If you post a picture of a cat on Facebook, a bunch of people will immediately identify it as a cat – but now Facebook automatically knows it’s a cat, too. This could be seen as another invasion of our online privacy, but for the visually impaired it’s incredibly useful: with the help of Apple’s VoiceOver software, phones and tablets using the Facebook app can now tell you what might be in the picture. “Image may contain: cat,” the audio software says when I tap on the photo. Correct! So I try a dog. “Image may contain: grass, outdoor, nature,” it says. Ah. It missed a huge golden retriever. Still, nice try.

The visually impaired usually rely on pictures being tagged so that screen reading software can tell them what’s there. On a regular website it’s down to the web designer, and on social media it’s down to the person posting the photo – but of course people often can’t be bothered. So Facebook is now automating that process by using artificial neural networks to recognise concepts within pictures, and in many cases it’s pretty accurate. Cats, snow, beaches, pizza and sky pose no problems, but responses can be vague (a picture of some salad was described as “plant, food”, an elephant merely “outdoors”) and occasionally very wrong (an arid desert scene was “outdoors, water”). For its part, Facebook claims that it can identify at least one concept with 80% accuracy in more than half the photos it analyses.

Image recognition software is getting cleverer but still struggles with things humans find easy. For example, Facebook can be fairly sure when a photo contains “one person”, but is unable to differentiate between Ed Sheeran and Judi Dench, let alone name them. Imagga, a competing image recognition product, helpfully offers percentage assessments of its abilities – but it is only 46% sure that a picture of Jay-Z is male, and 23% certain that he’s white. So while artificial intelligence gets a grip on who is who and what is what, the responsibility for tagging and captioning our pictures still lies with us. Because it’s not always obvious to everyone that a cat’s a cat.