a picture’s worth a thousand hashtags: how image recognition will power the future of analytics

A picture’s worth a thousand hashtags:

How image recognition will power the future of analyticsDavid BerkowitzChief Strategy OfficerSysomosdberkowitz@sysomos.comwww.sysomos.com@sysomos / @dberkowitz

About this presentation

This talk was presented to Marketing Week Live in London in March 2017. A Texan version of this was delivered to W2O Group’s Pre-Commerce Summit during SXSW that month. If you prefer Frito pie to bangers and mash, I will gladly send you the W2O edition.

Sources for information shown in slides are presented as links in the bottom-left corner. Further details about sources, where applicable, are also included in the notes field when downloading this presentation.

To share your feedback or discuss this further, please contact me at dberkowitz@sysomos.com.

Thank you.

David

And now for something completely standard: an agenda

• A brief history of nearly everything visual search

• Why visual search matters

• How Google, Pinterest, and others are deploying it

• How marketers can use it

• Numerous gratuitous references to all things British

?

Not British, but just a Chunnel ride away

How would you #HASHTAG

the Mona Lisa?

#YOLO

Mona Lisa #Hashtags

• #MonaLisa

• #art

• #painting

• #woman

• #lady

• #smile

• #smug

• #Italian

• #DaVinci

• #epic

• #outdoors

• #masterpiece

• #sky

• #France

• #LaGioconda

• #badhairday

• #beautiful

• #Louvre

• #Renaissance

• #portrait

This is a futile exercise. One can’t simply capture the Mona Lisa in hashtags. It points to the need for better ways to

identify and analyze visual content. Text and hashtags alone don’t cut it.

Textual Media

Visual Media

We are here.

Drop in a bucket visual

The Visual Data GapRecall how pictures are worth a thousand words? There is

so much more data in images (‘the sun’) than there is in text (represented metaphorically by the planets).

A Short History of Nearly Everything Visual

Dual British/ American citizen

A challenge of Shakespearean proportions:

“His reasons are as two grains of wheat hid in two bushels of chaff: you shall seek all day ere you find them, and when you have them, they are not worth the search.”-Bassanio, The Merchant of Venice

Number of object categories out there

15,000There are 15,000

object categories

Source: IEEE

Source: Computer Vision by Richard Szeliski

For fun, I included a few examples of early attempts at machine-

powered object recognition.

Source: Computer Vision by Richard Szeliski

An inflection point waiting to happen

Jason Goldberg, Razorfish: “I’m strongly bullish on visual search. It solves a real problem consumers have… In the not-too-distant future, it’ll become a heavily used mainstream feature. I think the inflection point is at least a year away, but not two years." We’re approaching the

inflection point, but it has taken longer than expected.

This report is from November 2014.

Scanning products = Cool

Facial scanning = Creepy

This is simply an enlarged, cropped

version of the highlights from the

previous chart.

You can’t always get what you want (with text search)

• 74% of consumers say text-based keyword searches are inefficient for helping them find the right products online

• 67% of consumers say quality of product images is very important in selecting and purchasing products

• 90% of information transmitted to the brain is visual

• Visual information is processed 60,000x faster than text

Source: Slyce

The dress that inspired Google Image Search in 2001

The dress that inspired Google Image Search in 2001

“…People wanted more than just text. This first became apparent after the 2000 Grammy Awards, where Jennifer Lopez wore a green dress that, well, caught the world’s attention. At the time, it was the most popular search query we had ever seen. But we had no surefire way of getting users exactly what they wanted: J Lo wearing that dress. Google Image Search was born.”-Eric Schmidt, Executive Chairman, Google

Source: Project Syndicate

Solving the Clarissa problem My wife gave me this reference. As a kid, she always wanted to identify and shop for whatever Clarissa wore.

Applications for image recognition

Source: Facebook

Source: The Verge

An introductory framework for visual search

Layers of image recognition

A Deep Learning algorithm is presented with the images made up of simple pixels.

The algorithm discovers simple regularities that are present across many/all images, like curves & lines.

The algorithm discovers how these regularities are related to form higher-level concepts

The system gains a high-level understanding of the original image… all automatically

Source: GazeMetrix

A framework for visual search

Scene

Identification Intelligence

ObjectIdentification Intelligence

LogoIdentification Intelligence

ImageIdentification Intelligence

CategoryIdentification Intelligence

This notes some of the most important processes within visual search. Also note that identification and intelligence

are two separate approaches. Examples follow.

What follows is an example using a real photo from Agnes, a

Chinese tourist to the UK.

Here’s her photo. In each subsequent slide, you can see how the framework

plays out and a sample finding that can be derived. Note the intelligence

examples that follow are for illustrative purposes only; feel free to cite the framework, but not the data itself.

Category Identification:This is food and drink

Category Intelligence:7.2% of images posted at museums include food or

drinks

Logo Identification:There is a Fanta logo, and

the text in the top-right says Starbucks

Logo Intelligence:Fanta logos are rarely paired with Starbucks; Fanta logos

are most often seen with Coca-Cola and Adidas

Object Identification:This looks like fish and chips with a can of Fanta Lemon

Object Intelligence:Fanta Lemon is the fourth most popular soda when paired with fish & chips

Image Identification:This is the same photo that

appears on Agnes_Cin’s Flickr and public Facebook

pages

Image Intelligence:This image hasn’t been

shared in any media outlets and hasn’t been shared

publicly

Scene Identification:This photo seems to be taken

outdoors during the day

Scene Intelligence:94% of photos at the British Museum are shot indoors,

compared to 87% of museum photos worldwide

Spotlight: Google

This section is drawn (no pun intended) from quickdraw.withgoogle.com.

Spotlight: Pinterest

Pinterest: one of the world’s biggest search engines

• 150 million monthly users• 75 billion pins• 2 billion searches/month• 97% of searches are unbranded

Source: Pinterest

Pinterest: search pins from real-world images

Source: Pinterest

Examples from Pinterest’s new Visual Discovery follow. In the downloadable

version of this talk, the next few visuals play as GIFs, and you can read more at

the source below.

Pinterest Visual Discovery

Source: Pinterest

Browse images and buy from them

Source: Pinterest

I really liked the circled image here: Pinterest draws a connection between Big Ben and the Blue Mosque. Such

errors are often more revealing.

Spotlight: Emerging Technologies

Toys R Us offers Slyce image detection for its catalog

Source: Slyce

Visual search to complement textual search

eMarketer: Do you think [visual search] will replace some types of searches, or do you think it will augment existing searches?

Gierhart: It will probably augment. It’s adding a new utility to what was there before... There will still be contexts for both.

Source: eMarketer (see a related video on YouTube)

Blippar: scan images for surprises about Planet Earth

Source: Blippar

Source: Houzz

Houzz has a Visual Match offering akin to

Pinterest.

TheTake uses AI to identify products, locations in video

Source: TheTake

What’s Possible with Image Analytics

The images that follow are sample reports drawn from Sysomos. The data is again for illustrative purposes. Reach out if you want to dive deeper into any of

this.

Visual analysis: understanding visual characteristics

Logo recognition

Object recognition

Scene recognition

Food recognition

Color detection

OCR: Search text within images

Visual analysis: understanding visual characteristics

Logo recognition

Object recognition

Scene recognition

Food recognition

Color detection

OCR: Search text within images

Visual analysis: understanding visual characteristics

Logo recognition

Object recognition

Scene recognition

Food recognition

Color detection

OCR: Search text within images

Audience analytics show growth and spikes

Brand affinities can highlight cross-promotion opportunities

Identify most popular objects, scenes

B2B applications for visual search

Creative optimization

Influencer marketing

Rights manageme

nt

Crisis manageme

nt

Partnership ideation

Competitive intelligence

Customer service

Any questions?

So long, and thanks for all the fish.Let’s take tea!

David BerkowitzChief Strategy OfficerSysomosdberkowitz@sysomos.comwww.sysomos.com@sysomos / @dberkowitz