A breakdown of my submission for Twitters Innovation Challenge #Promote

Introduction

This is the last post in the series whereby I’ve been documenting the build effort for this year’s entry to the Twitter Innovation Challenge 2017 #Promote. The following is covered:

• key features of the solution
• overview of the process
• main components
• results and evidence

For reference, here are links (in chronological order) to all posts related this:

1. The Innovation Challenge is back for 2017
2. Tapping into the Twitter Firehose
3. Real-time updates using SignalR
4. Building a dashboard with SignalR, JQuery and Bootstrap
5. Twitter API, SignalR and Microsoft Cognitive Services
6. Identifying sales leads using Microsoft Cognitive Services
7. Generating a Tailored Audience List for Twitter using C#
8. Segmenting Customers and performing market research using Microsoft Cognitive Services
9. Twitter, AI, Lead Generation and #Promote
10. #Promote and Azure

Key Features

• Twitter Ads API integration
• Turns signals on Twitter into sales leads using AI probabilities and ML
• Dynamic Tailored Audience generation
• POS (Part of Speech) Tagging to enrich analytics
• Dynamic promoted tweet generation when a new Campaign is created
• Creative module to create ads and associated copy
• Performs sentiment analysis against tweets to determine positive/negative emotion
• Sends email alerts when sentiment spikes or drops
• Dashboards and visualisations

History

The origins of this can be traced back to Master’s degree research I undertook a few years ago, the subject of the dissertation was:

“Sentiment Analysis and Data Mining Social Networks using Bayesian Theorem”

This resulted in software that performed sentiment analysis against tweets.  It has since evolved from a simple .NET Windows application to the web application I’ve called SocialOpinion.

All of this has been done entirely in my own time which has been tough to manage.  Evenings and weekends, at airports whilst waiting between flights – basically whenever I could get a minute.

Timeline

2013

• Researched machine learning, natural language processing, sentiment analysis and Bayesian theorem for Master’s degree. Produced dissertation.

2014

• Wrote .NET Windows application that extracted tweets and performed sentiment analysis based on dissertation research.

2015

• Migrated logic from the .NET Windows application into an API (sentiment analysis engine)
• Introduced location / GPS functionality and Google Maps
• Built skeleton web application to house the sentiment analysis engine

2016

• Feedback from early adopters on Reddit
• Identified that users on Reddit were using the product to monitor stock market symbols/sentiment and attempting to identify correlation
• Entered Twitter #Promote Ads API Challenge
• Extended the API and web application to incorporate the Twitter Ads API

2017

• Investigated Microsoft Bot Framework and potential use cases
• Integrated Microsoft Cognitive Services to enrich AI features and built sales lead identification functionality
• Entered Twitter Innovation Challenge #Promote
• Validated AI engine was successfully identifying “commercial intent” in Twitter data

***breaking news***

• Acquired for $1 million (ok I made that up)

Technology

For the nerds, some insight into the tech stack and tooling.

• ASP.NET
• Bootstrap
• Bayesian classification (proprietary sentiment analysis implementation)
• Cloud Server(s)
• C#
• Google Maps
• JavaScript
• JQuery
• Microsoft .NET
• Microsoft Azure
• Microsoft Cognitive Services
• OAuth
• REST APIs
• Postman
• POS Tagging (Stanford NLP)
• SQL Server
• Twitter API
• Twitter Ads API
• TopShelf
• Visual Studio Team Services
• Windows Services

Overview

Before getting into the detail, as the old saying goes, a picture often depicts a thousand words.  The Visio diagram provides an overview of the service that processes Twitter data to identify:

• User Location
• Part of Speech (POS) Tags
• Sales leads and commercial intent

For context, the keyword “iPhone” has been used in the Campaign configuration throughout the remainder of this post.

Main Components

The main components of the solution are:

• User Profile
• Social Opinion Campaign
• Social Opinion Ad Campaign
• Creatives Module (not fully integrated with the Ads API yet)
• Twitter Ads Campaign
• Promoted Tweets
• Tailored Audience(s)

User Profile

From this screen, you can which Twitter Ads Account should be associated with the SocialOpinion Account. The solution interrogates any available accounts via the Ads API and imports then into the SocialOpinion database.  Subsequent Twitter Ads related activities within SocialOpinion are then ran within the context of the selected account.

Campaign

A Campaign is the container for most activities in SocialOpinion.  In this screen, you can view, create and modify Campaigns.  You specify keywords, phrases or hashtags that you want platform to listen for.

In this screenshot, you can see a Campaign called iPhone has been defined to extract tweets that contain the keywords iPhone.

You can also view the Dashboard associated with a Campaign.

Dashboard

The dashboard contains feeds that let you view the sentiment in the form of:

• Grids
• Real-time line graphs
• Bar charts
• Heatmaps

These allow you to identify how people feel about your product, brand or service:

AdTech Implementation

When a SocialOpinion Campaign is created, SocialOpinion Ad Campaign metadata is also generated.  This metadata is then used to create a Twitter Ads Campaign on the Twitter end.  For example, the Twitter Ads Campaign name is generated dynamically by using the SO Campaign Name and prefixing it with the text SO_AdCampaign_

Other items created dynamically on the Twitter Ad server during this phase include:

• Line Item(s)
• Funding Instrument
• Promoted Tweet
• Start Time of the Ad Campaign
• Ad Campaign Daily Budget
• Tailored Audience Placeholder

All the above happens within the context of an Ads Account which is inferred via the SocialOpinion User Profile.

In the screen shots below you can see an extract of the SocialOpinion database table for the iPhone Campaign and the Twitter Ads Campaign that has been created.  You’ll see the Campaign names match.

(Social Opinion database extract)

(Twitter Ads Campaign)

Promoted Tweet

A default Promoted Tweet is created on the Twitter Ads server with accompanying copy.  Again, SocialOpinion Campaign meta data is used to drive this.

You can see the sample Tweet on the Twitter Ads manager console below:

Ad Campaigns and Promoted Tweets aren’t much use unless you have users to target and this is where the following components come in:

• TON File Path
• Tailored Audiences

TON File Path

The TON File Path is a location on the Twitter Ads server where you can place binary content.  In the image below you can see that SO has persisted the following TON metadata for the iPhone Campaign.

TON metadata is used to create a placeholder on the Twitter Ads server for the saving of Twitter user handles.  This bring us onto the concept of Tailored Audiences.

Tailored Audiences

A Windows Service runs 24×7 collecting Twitter user ids which are transmitted to Twitter Ad servers using the Ads API.  This collection of user ids is known as a Tailored Audience.

How does SocialOpinion decide which user ids to add to the Tailored Audience?

User ids are identified by leveraging the following:

• Social Opinion Campaign configuration
• Sales lead probability (driven by AI)
• Part of Speech (POS) Tagging

 

In the code extract below, the engine has identified there is a of 0.70 probability of this user being a sales lead.

The probability threshold 0.70 (or 70%) is hard coded for the time being but naturally this could be configurable.

This results in a hyper-targeted collection of user ids that have been tweeting about iPhones in the last 15-30 minutes and expressing commercial intent.

User ids and tweet metadata are added to the Tailored Audience.  You can see a selection user ids for the iPhone Campaign below:

OK, but what can I do with these?

Time to discuss JiT (Just-in-Time) Marketing.

JiT Marketing

The JiT Audience screen is split into three sections

• Ad Campaign Config
• Tailored Audience List
• Tailored Audience Config

Each of these allow you to:

• View Campaign / Ad Campaign metadata
• Create a new Promoted Tweet
• View a sample segment of the Tailored Audience for the selected Campagn/Ad Campaign
• View TON / Audience metadata
• Upload the Tailored Audience that SocialOpinion has generated directly to Twitter

You can see all three sections for the iPhone Campaign below:

Ad Campaign Config

This section contains Campaign and Ad Campaign metadata.  For #Promote, default values were used for the Start Time, Daily Budget, Ad Type and Ad Placement.

From here you can new Promoted Tweet on the Twitter Ads server for this Ad Campaign by supplying the copy and clicking Save.

…and the newly create Promoted Tweet appears on the Twitter Ads Manager Console:

Tailored Audience List

This grid displays a selection of Twitter users than have been identified by the AI module as being potential sales leads that will form the current Tailored Audience for this Ad Campaign.

Tailored Audience Config

This section displays Tailored Audience metadata that has been generated as a result of the SocialOpinion Campaign.

Clicking on the button uploads all user ids from the Tailored Audience List directly to the Twitter Ads server.

You can see Ads Audience Manager is processing an upload of user ids from SocialOpinion for the “iPhone” Campaign.

Important points regarding Tailored Audiences:

• It can take a hours for the user ids to be ingested on the Twitter Ads server, this is why the Status value = Processing
• The minimum size of a Tailored Audience is 500 user ids
• User ids must be hashed (SocialOpinion takes care of this!)
• Users in the Tailored Audience must be active on Twitter in the last 30 days

Creatives Module

Having a hyper-targeted Tailored Audience is all fine and well but you need to be able to serve them with ads.  This is where the Creatives Module comes into play. Using these screens, you can create the following types of creative:

• Website Card
• App Card
• Photo
• Video Card

Creating and viewing a list of Creatives

Previewing a Website Card

*At the time of writing, the Creatives module is not fully integrated with the Ads API.  The focus has been on AI and ML dynamic Tailored Audience generation.

Results

Finally, the results for the iPhone Campaign:

• Extracted 5000+ tweets related to the key word “iPhone”
• Configured the application to identify and extract sales leads where the probability of a potential sales lead was greater than 80%
• Generated a Tailored Audience with 622 user accounts
• Associated cost of invoking Azure services for 3 days was no more than 10GBP

The table below contains sample tweets that form part of the Tailored Audience of 622.

I’ve included the Status ID as evidence and the probability of the tweet / user being a sales lead.

There were many of false positives in the dataset but the AI component can be further trained and this will gradually reduce these.

Finally

This is it for a while on the development of this and I’m also taking a month off from the weekly posting, that may change when the winner is announced! 😉