I’ve been working with the new Twitter Labs APIs recently. In this blog post I introduce an open source API I’ve built that makes it easy for .NET developers to consume these APIs.
For reference, the entire collection of APIs that I’ve been looking at are:
- Filtered Stream
- Metrics
- Recent Search
- Tweets
- Users
Each of these APIs gives you programmatic access to Twitter data and you get a lot quality data straight out of the box.
Overview of the APIs
Key offerings from the above APIs are as follows:
Filtered Stream
Let’s you open an always on connection that can extract 500,000 Tweets a month using upto 10 Powerful Querying Rules.
Metrics
Returns the number of Likes, Comments, Retweets etc for a single Tweet or collection of Tweets.
Recent Search
Let’s you search over the last 7 days using powerful queries and return upto 500,00 Tweets per month.
Tweets and Users
Let’s you perform Tweet and User Lookups that contain richer insights that v1 of the Twitter API
Extended Datasets and Richer Insights
Most of these APIs also return additional nodes as part of the JSON payload called Context Annotations, Entities and Domains.
These let you surface loads of insights such as: people being discussed, products, brands or services being mentioned along with calculated probabilities of said item being present in the Tweet.
For example, here you can see some examples of Context Annotations and Domains related to a query “iphone”:
In the past I would have implemented something like Part of Speech tagging (POS) or Named Entity Recognition (NER) to identify such things. Now you get this out the box.
Social Opinion API
I’ve encapsulated all functionality from the APIs mentioned above into an easy to use API. I’ve called this the Social Opinion API.
Some key features of the Social Opinion API include:
- written in C#
- targets .NET Core
- open source, available on NuGet and GitHub
- handles User Authentication (OAuth 1) & Application Authentication (OAuth 2)
- gives you a rich set of strongly typed objects to work
- makes it easy to understand and process Twitter data
- easy to integrate with existing software projects or services
Note: I introduced aspects of the Social Opinion API in a dedicated blog post that detailed Filtered Stream integration. You can read more about that here. I also introduced aspects of the Social Opinion API integration with the Twitter Labs Metrics API. You can find out more about that here.
Social Opinion API – under the hood
Each implementation contains a Client and Service class. The Client class is responsible for making the underlying requests, processes and parsing the JSON response from the Twitter API endpoints to DTOs. The Service class applies any necessary business logic and converts the DTOs to Models.
The following UML Sequence diagram shows you this in action within the context of the Recent Search Service being invoked by a client application:
Under the hood the Recent Search Service is performing the following steps:
- performs authentication using the developer keys
- invoking the Social Opinion API Recent Search Service
- the Recent Search Client constructs the GET request, setups authorization and adds the relevant parameters mandated by the Twitter API
- Request Builder adds all parameters and sends the GET request
- The Twitter API is invoked and returns a JSON response which contains 1-N “pages” of data (max of 100 Tweets per page)
- The Social Opinion API serializes Twitter API JSON to Social Opinion DTO’s
- Social Opinion converts DTOs to Models for use by the client application
With the over out of the way, let’s look at an example implementation of each API and some sample data each Social Opinion Service returns.
Authentication
First, we need to look at authentication. Authentication is performed using OAuth 1 (User Context) and OAuth (Twitter Application Context). User level context can give you access to certain user level data whereas Application Context may not have access to this.
To setup authentication you need to send in a bunch of keys. This is a standard process across all five API implementations which you can see here below.
string _ConsumerKey = ConfigurationManager.AppSettings.Get("ConsumerKey");
string _ConsumerSecret = ConfigurationManager.AppSettings.Get("ConsumerSecret");
string _AccessToken = ConfigurationManager.AppSettings.Get("AccessToken");
string _AccessTokenSecret = ConfigurationManager.AppSettings.Get("AccessTokenSecret");
OAuthInfo oAuthInfo = new OAuthInfo
{
AccessSecret = _AccessTokenSecret,
AccessToken = _AccessToken,
ConsumerSecret = _ConsumerSecret,
ConsumerKey = _ConsumerKey
};
After you have these you then send in the OAuth object to the respective Social Opinion API Service constructor. This lets each service in the Social Opinion API perform the necessary steps to make an authenticated request.
Filtered Stream Service
First up in the Filtered Stream, I covered aspects of this in earlier blog post but will include here for completeness.
After we have our OAuth data, we create an instance of the Filtered Stream Service:
FilteredStreamService filteredStreamService = new FilteredStreamService(oAuthInfo);
Before using the Filtered Stream, you need to create 1 or more Rules. A Rule is a query that is used to fetch Tweets that match your criteria.
Here I am using the Create Rule method and telling the Filtered Stream to listen for “#iphone”. The text “testing #iPhone” is a human readable description for the Rule:
After the method has executed, a model FilteredStreamRule is returned which contains a unique ID for the Rule generated by Twitter.
As this is a real-time interface that maintains a connection the Twitter API the Social Opinion API uses the concept of events. As a Tweet is fetched and processed from the stream an event is raised.
We need to wire up this event and that’s what we can see here:
filteredStreamService.DataReceivedEvent += FilteredStreamService_DataReceivedEvent;
The client application then must handle the event and use the processed Tweet which gets sent as part of the event arguments. You can see this here:
private static void FilteredStreamService_DataReceivedEvent(object sender, EventArgs e)
{
FilteredStreamService.DataReceivedEventArgs eventArgs = e as FilteredStreamService.DataReceivedEventArgs;
FilteredStreamModel model = eventArgs.FilteredStreamDataResponse;
}
Finally, with everything in place, we can start the stream by calling the following method:
filteredStreamService.StartStream("https://api.twitter.com/labs/1/tweets/stream/filter? tweet.format=detailed", 10, 5)
In the code above we are supplying the Filtered Stream Endpoint, the number of Tweets we want returned (10) and the number of network retry attempts to retry (5) in the even of a network failure.
We can run this in Visual Studio and examine some of the data:
A closer look at the data shows programming and hobbies are being discussed:
Metrics Service
I introduced this API in another blog post too. Unlike the Filtered Stream, we don’t need to setup an event. We use a simple request/response pattern. We supply our OAuth data then simply supply a single Tweet ID or collection of Tweets:
List<string> ids = new List<string>();
ids.Add("1258736674844094465");
TweetMetricsService service = new TweetMetricsService(oAuthInfo);
List<TweetMetricModel> metricModels = service.GetTweetMetrics(ids);
After we run this, we can see valuable Tweet Metrics for the supplied Tweet Id:
In the above we can see this Tweet has received 61 impressions and 1 Like.
Recent Search Service
Here we can see the Recent Search API Service being invoked.
I create an instance of the Social Opinion API Recent Search Service passing in the OAuth keys as usual. In this example am telling the API to fetch 100 Tweets that contain the text iphone.
RecentSearchService searchService = new RecentSearchService(oAuthInfo);
List<RecentSearchResultsModel> resultsModels = searchService.SearchTweets("iphone", 100);
When Search Tweets has finished executing a collection of results are available for processing. We can examine this in the Visual Studio Debugger. Here are the root level objects that form the initial response generated by the Social Opinion API:
You can see in this page of data we have 100 Tweets to process (the data object). There is also some user data in the Incudes collection property which gives you quick access to all media and users found in this list of Tweets.
We can expand the data property and see all Tweets that matched our initial query “Iphone”:
We can drill a bit further into this data by looking into the Context Annotations, Entities and Domains being discussed in these Tweets. For example, here we can see two lookups the show a Domain of type “Product” and Entity of type “Apple-iPhone” are being discussed:
I’m currently feeding this data into Azure as its identified to create Lookup and Reference Tables in real-time. This is great for reporting as when the list is created newly incoming Tweets will be categorised automatically using unique ids.
Tweet Service
This contains lookup functionality that lets you send in a Tweet ID or collection of Tweets. Like most of the others we supply our authentication data and it uses a request and response pattern. Here we are sending in a single Tweet ID and getting data back out the user, public metrics and the Tweet copy itself:
TweetService tweetsService = new TweetService(oAuthInfo);
TweetModel tweetModel = tweetsService.GetTweet("1258736674844094465");
User Service
This also contains lookup functionality but for users. I’ve implemented functionality to let you perform lookups for a single user or collection of users. Here you can how to call the single get user method and some of the basic account data the Social Opinion API will return:
We can take a deeper look at this structured data by looking into the Entities object:
If I browse to my account on Twitter, we can see these match:
A lot of active users on Twitter do this to improve the searchability and visibility of their accounts so being able to surface these insights is beneficial for many reasons.
Where can you get the Social Opinion API?
It’s early days for the API but I’ve uploaded it to GitHub and published it on NuGet.
You can download it for free, can contribute to the development of API or if you want to change it to suit your business needs that’s cool also!
At the time of writing, it’s on version 0.0.6 and has been available for 6 days, has received 100+ downloads and is receiving regular updates.
Uses for the Social Opinion API?
I’m using the Social Opinion API to extract data and push it into Azure for reporting and analytical purposes. Here you can see a dashboard written in ASP.NET Core that displays some data from the Social Opinion Filtered Stream Service:
I’m putting the finishing touches to functionality that will let users create Campaigns. When a Campaign is created with associated keywords, the Recent Search Service works its magic and starts to populate databases in Azure. This data can then be visualised in multiple dashboards.
Some ideas for the API, dashboards and data might include:
Ad-Tech
Using a combination of user metadata and Tweet insights such as Context Annotations, Entitles and Domains you can develop ad-tech solutions that help surface potential sales or advertising opportunities. Alternatively, use the data to gauge the strength of a products, company, brand, or service. Take this further by tapping into the location query operators to further segment filter Tweets or user data by geolocation.
Analytics
Use a combination of the Recent Search Clients Filtered Stream Clients (not the Services) to process Twitter data at scale. Push this into a scalable and cost effecting database solution such as Azure Table storage. Bypassing the respective Service classes and hooking straight into the Client classes will expose the underlying JSON as soon as the request is returned from the Twitter API . This will remove all the conversion between object types. You can then push this into NoSQL databases at maximum speed
Chatbots
Place a chatbot in front of data and insights that Social Opinion API returns. Use technology such as Microsoft Bot Framework to developer a reusable chatbot component such as a Skill. Users can pull the Skill off the shelf and have a chatbot which interacts with Twitter data.
Web Services
Build a REST API and expose the Social Opinion API as web service. Create a web application that lets user’s setup configurations which kick of instances of the Recent Search or Filtered Stream. Consumers of web services can get programmatic access to real-time data with minimal fuss. Use this data to augment existing insight from Twitter data.
What Next?
There are two APIs left to fully integrate: The Sampled Stream and Hide Replies APIs. When this is done, the Social Opinion API will have full coverage for the Twitter Labs APIs. I need to find time to plan this in but am aiming to get this done in the next 30 days. Or, if you’re reading this and fancy implementing them, then get in touch and I’ll add you as a contributor to the GitHub repo.
Summary
In this blog post I’ve introduced a new open source API I’ve built that can help developers quickly consume Twitter data using the new Labs APIs.
You can download the Social Opinion API from NuGet or take the code from GitHub.
Feel free to get in contact if you want to contribute to the API, have ideas or have more questions about it.
Leave a Reply