{"id":1758,"date":"2020-04-05T16:43:47","date_gmt":"2020-04-05T15:43:47","guid":{"rendered":"http:\/\/www.jamiemaguire.net\/?p=1758"},"modified":"2020-04-05T16:43:47","modified_gmt":"2020-04-05T15:43:47","slug":"connecting-to-the-twitter-labs-filtered-stream-with-c","status":"publish","type":"post","link":"https:\/\/jamiemaguire.net\/index.php\/2020\/04\/05\/connecting-to-the-twitter-labs-filtered-stream-with-c\/","title":{"rendered":"Connecting to the Twitter Labs Filtered Stream with C#"},"content":{"rendered":"<p style=\"font-weight: 400;\">In my last post, I introduced the <a href=\"https:\/\/developer.twitter.com\/en\/docs\/labs-v1\/filtered-stream\/overview\">Twitter Labs Filtered Stream API<\/a>.<\/p>\n<p style=\"font-weight: 400;\">In that post, I explored the main features of the API and the rich data you get back. I looked at the rich JSON output and shared some ideas around use cases for the API.<\/p>\n<p style=\"font-weight: 400;\">To towards the end of that blog post, I mentioned an API I had been working on that connects to the Twitter Filtered Stream API.<\/p>\n<p style=\"font-weight: 400;\">This was called the\u00a0<strong>Social Opinion API<\/strong>.\u00a0 This API can extract process and parse data from the Detailed JSON payload that is returned by the Twitter Filtered Stream API.\u00a0 It\u2019s been implemented a reusable .NET Core class library that can be used by multiple projects.\u00a0 \u00a0I shared a cut of the data the Social Opinion API can surface with the Twitter DevRel Team along with some other mislabelled datasets.<\/p>\n<p style=\"font-weight: 400;\">In this blog post, I follow on from the last and share:<\/p>\n<ul>\n<li style=\"font-weight: 400;\">Overview of the Social Opinion API and Filtered Stream API integration<\/li>\n<li style=\"font-weight: 400;\">How the Social Opinion API connects to the Filtered Stream API<\/li>\n<li style=\"font-weight: 400;\">Insight into some of the data Social Opinion API can surface from the Filtered Stream API<\/li>\n<li style=\"font-weight: 400;\">Telemetry, Metrics, Resource Usage and Diagnostics details<\/li>\n<li style=\"font-weight: 400;\">Snapshot of the data the Social Opinion API can extract and persist<\/li>\n<\/ul>\n<p style=\"font-weight: 400;\">First things first, an overview of the process and main components.<\/p>\n<h2>Overview<\/h2>\n<p style=\"font-weight: 400;\">The Twitter Filtered Stream API needs at least one\u00a0<strong>Rule<\/strong>\u00a0to be configured before you can use it. You also need to create your own client \/ service \/ code to make the connection and start listening for data.<\/p>\n<p style=\"font-weight: 400;\">I thought about how this might work and quickly mocked this up using a mobile app called Lucid Charts which you can see here:<\/p>\n<p style=\"font-weight: 400;\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-1760\" src=\"http:\/\/www.jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image002.jpg\" alt=\"\" width=\"617\" height=\"1235\" srcset=\"https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image002.jpg 617w, https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image002-375x750.jpg 375w, https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image002-600x1200.jpg 600w\" sizes=\"auto, (max-width: 617px) 100vw, 617px\" \/><\/p>\n<p style=\"font-weight: 400;\">The above diagram gives you an overview of the main components and there is more going on under the hood.\u00a0 Boxes in blue site at the Twitter domain whereas everything else sits within the Social Opinion API end.<\/p>\n<h3>Social Opinion API (Data Processing)<\/h3>\n<p style=\"font-weight: 400;\">The\u00a0<strong>Social Opinion API (Service)<\/strong>\u00a0section is a class library. This can be housed in an Azure Web Job, \u00a0Web API, Windows Service \u2013 anything really that can call .NET code. It will run 24&#215;7 extracting, ingesting, transforming data an as surfacing actionable insights.<\/p>\n<h3>Manual Input (Rules Administration)<\/h3>\n<p style=\"font-weight: 400;\">The\u00a0<strong>Manual Input<\/strong>\u00a0section is a smaller collection of APIs that sit within the Social Opinion API which let you administer Filtered Stream Rules. You could house this functionality within a console application, Web API or even build a web front end to let you administer Rules.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-1761\" src=\"http:\/\/www.jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image003.jpg\" alt=\"\" width=\"753\" height=\"354\" srcset=\"https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image003.jpg 753w, https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image003-750x353.jpg 750w\" sizes=\"auto, (max-width: 753px) 100vw, 753px\" \/><\/p>\n<p style=\"font-weight: 400;\"><strong>(Sample Web UI for Rule Administration)<\/strong><\/p>\n<h2>How does the Social Opinion API connect to the Twitter Filtered Stream API?<\/h2>\n<p style=\"font-weight: 400;\">I\u2019m a big fan of encapsulation and aim to create easy to use APIs. The Social Opinion API lets you connect to the Filtered Stream API in just a few lines of code.<\/p>\n<p style=\"font-weight: 400;\">In the code segment below, you can see the Filtered Stream API connection can be made with less than 3 lines of code:<\/p>\n<pre class=\"prettyprint\">class\u00a0Program\r\n\u00a0\u00a0\u00a0 {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0static\u00a0void\u00a0Main(string[] args)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 SocialOpinionAPI.Core.Labs.FilteredStream.Logic.LabsFilteredStreamDataLogic logic =\u00a0\u00a0new\u00a0LabsFilteredStreamDataLogic(true,\u00a0true);\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 logic.LabsFilteredStreamStartStream();\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 }<\/pre>\n<p style=\"font-weight: 400;\">Under the hood, it\u2019s a different matter.\u00a0 There\u2019s a lot to consider such as:<\/p>\n<ul>\n<li style=\"font-weight: 400;\">Creating OAuth Headers using the Twitter App Secret Keys\/Tokens<\/li>\n<li style=\"font-weight: 400;\">Building the Http Requests<\/li>\n<li style=\"font-weight: 400;\">Processing the JSON response from the Filtered Stream and mapping to the Social Opinion data model<\/li>\n<li style=\"font-weight: 400;\">Data access\/persistence<\/li>\n<li style=\"font-weight: 400;\">Handling errors and adhering to the Twitter Terms of Service<\/li>\n<\/ul>\n<p style=\"font-weight: 400;\">I won\u2019t go into all of this but calling this one method does\u00a0<strong>all<\/strong>\u00a0the above.<\/p>\n<h3>Under the Hood<\/h3>\n<p style=\"font-weight: 400;\">Under the hood, I call a private method that loads in a series of configuration settings for the interface.<\/p>\n<p style=\"font-weight: 400;\">These settings let me set the number of \u201cTweets to fetch\u201d or \u201cMaximum Number of Errors\u201d the interface should tolerate before telling the service to \u201cback off\u201d from the Filtered Stream API.<\/p>\n<p style=\"font-weight: 400;\">You\u2019ll also see there is a setting that lets you enable\/disable the service too:<\/p>\n<pre class=\"prettyprint\">if(!config.ServiceActive)\r\n{\r\n  _log.Info(\"Service not Active. Abandoning start\");\r\n}\r\nelse\r\n{\r\n  LabsFilteredStreamStartStream(config.AddressAndFilter, config.TweetsToFetch, \r\n  config.MaxNetworkErrorRetryAttempts, config.MaxParseErrorTolerence);\r\n}<\/pre>\n<p style=\"font-weight: 400;\">From the code snippet above, you\u2019ll see a further call is made to\u00a0<strong>LabsFilteredStreamStartStream<\/strong>.<\/p>\n<p style=\"font-weight: 400;\">This is a private method that does a lot of the heavy lifting in terms of parsing the JSON from the Filtered Stream API to custom C# objects (and a whole load of other things).<\/p>\n<h3>Live Debug Session<\/h3>\n<p style=\"font-weight: 400;\">You can see a live debug session of a lower level Social Opinion API method call in action here:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-1762\" src=\"http:\/\/www.jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image004.png\" alt=\"\" width=\"753\" height=\"429\" srcset=\"https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image004.png 753w, https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image004-750x427.png 750w\" sizes=\"auto, (max-width: 753px) 100vw, 753px\" \/><\/p>\n<p style=\"font-weight: 400;\">From the screenshot above, you can see that the\u00a0<strong>Hashtag\u00a0<\/strong>(COVID2019) and A<strong>nnotations\u00a0<\/strong>(Person) have been identified in the Tweet being processed.<\/p>\n<h2>Events<\/h2>\n<p style=\"font-weight: 400;\">The Social Opinion API also raises events as Twitter data is fetched and saved.\u00a0 This lets you handle responses from the Filtered Stream API whichever way you want.\u00a0 For example, here I am setting up a few event handlers and telling Social Opinion to start listening to the Filtered Stream API:<\/p>\n<pre class=\"prettyprint\">private void button17_Click(object sender, EventArgs e)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 SocialOpinionAPI.Core.Labs.FilteredStream.Logic.LabsFilteredStreamDataLogic logic =\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0new\u00a0LabsFilteredStreamDataLogic(true,\u00a0true);\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 logic.FilteredStreamDataReceivedEvent += Logic_FilteredStreamDataReceivedEvent;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 logic.FilteredStreamDataSavedEvent += Logic_FilteredStreamDataSavedEvent;\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 logic.LabsFilteredStreamStartStream();\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }<\/pre>\n<p style=\"font-weight: 400;\">Here I am taking the\u00a0<strong>data<\/strong>\u00a0(the Filtered Stream API JSON payload mapped to a custom DTO) and outputting this to console:<\/p>\n<pre class=\"prettyprint\">\u00a0private\u00a0static\u00a0void\u00a0Logic_FilteredStreamDataReceivedEvent(object\u00a0sender, EventArgs e)\r\n{\r\n    TweetReceivedEventArgs eventArgs = e\u00a0as\u00a0TweetReceivedEventArgs;\r\n    FilteredStreamDataResponse dataResponse = eventArgs.filteredStreamDataResponse;\r\n    Console.WriteLine(\"Data received from filtered stream: \"\u00a0+ dataResponse.data.ToJson());\r\n}<\/pre>\n<p style=\"font-weight: 400;\">Finally, I am listening to the \u201cdata saved event\u201d that my API raises:<\/p>\n<pre class=\"prettyprint\">private\u00a0void\u00a0Logic_FilteredStreamDataSavedEvent(object\u00a0sender, EventArgs e)\r\n{\r\n    StreamDataSavedEventArgs streamDataSavedEventArgs = e\u00a0as\u00a0StreamDataSavedEventArgs;\r\n    LabsFilteredStreamData\u00a0data\u00a0= streamDataSavedEventArgs.filteredStreamDataReceived;\r\n\r\n    Console.WriteLine(\"Data saved with StreamDataUid:\"\u00a0+\u00a0data.StreamDataUid +\u00a0\". Content:\"\u00a0+ data.ToJson());\r\n}<\/pre>\n<p style=\"font-weight: 400;\">When this happens, I know that Social Opinion has pushed the Filtered Stream data into the database.\u00a0 You can see from the above code; the Social Opinion API shields you from the low-level calls you need to make if you want to consume the Filtered Stream API and can be used with just a few lines of code!<\/p>\n<p style=\"font-weight: 400;\">Let\u2019s look at a subset of the data the Social Opinion API will return when it processes real-time data from the Filtered Stream API.<\/p>\n<h3>Filtered Stream Data<\/h3>\n<p style=\"font-weight: 400;\">This is the root level object for a processed Tweet. It contains many fields which can be valuable for reporting, analytics and social listening purposes:<\/p>\n<pre class=\"prettyprint\">public\u00a0class\u00a0LabsFilteredStreamData\r\n\u00a0\u00a0\u00a0 {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0LabsFilteredStreamData();\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0string\u00a0source {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0string\u00a0lang {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0bool\u00a0possibly_sensitive {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0int\u00a0quote_count {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0int\u00a0like_count {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0int\u00a0reply_count {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0int\u00a0retweet_count {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0Stats stats {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0public\u00a0Models.Entities Entities {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0List&lt;FilteredStreamReferencedTweet&gt; referenced_tweets {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0string\u00a0in_reply_to_user_id {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0string\u00a0author_id {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0string\u00a0text {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0DateTime created_at {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0string\u00a0id {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0int\u00a0RequestId {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0long\u00a0StreamDataUid {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0List&lt;FilteredStreamContextAnnotation&gt; context_annotations {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0string\u00a0format {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0 }<\/pre>\n<h3>Annotation<\/h3>\n<p style=\"font-weight: 400;\">This is one of my favourite objects that I\u2019ve added to the Social Opinion API. Arriving at this kind of insight in the past would have meant implementing my own custom Part of Speech (POS) Tagger or Named Entity Recognition (NER) API, now I have a lot of this out of the box.<\/p>\n<pre class=\"prettyprint\">public\u00a0class\u00a0Annotation\r\n{\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0int\u00a0start {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0int\u00a0end {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0double\u00a0probability {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0string\u00a0type {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0public\u00a0string\u00a0normalized_text {\u00a0get;\u00a0set; }\r\n}<\/pre>\n<p style=\"font-weight: 400;\">Let\u2019s look at how this is used.\u00a0 Consider the following Tweet:<\/p>\n<p style=\"font-weight: 400;\"><em>\u201cUK could face &#8216;indefinite lockdown&#8217; until vaccine is found. They know where they can go on this idea. #coronavirusuk #COVID2019\u00a0 https:\/\/t.co\/HkQF1BOQ6z\u201d<\/em><\/p>\n<p style=\"font-weight: 400;\">The Social Opinion API integration with the Twitter Filtered Stream API is identifying\u00a0<em>UK\u00a0<\/em>as a\u00a0<strong>Place\u00a0<\/strong>and an\u00a0<strong>Annotation<\/strong>\u00a0with a high confidence score.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-1763\" src=\"http:\/\/www.jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image005.jpg\" alt=\"\" width=\"752\" height=\"163\" srcset=\"https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image005.jpg 752w, https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image005-750x163.jpg 750w\" sizes=\"auto, (max-width: 752px) 100vw, 752px\" \/><\/p>\n<p style=\"font-weight: 400;\">The API will also identify\u00a0<strong>Products<\/strong>,\u00a0<strong>Organisations,<\/strong>\u00a0and\u00a0<strong>People<\/strong>.\u00a0 With the information in this format, you can start to build some innovative solutions.<\/p>\n<h3>Domain<\/h3>\n<p style=\"font-weight: 400;\">Here you can see some of the Domains that are being collected:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-1764\" src=\"http:\/\/www.jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image006.jpg\" alt=\"\" width=\"640\" height=\"242\" \/><\/p>\n<p style=\"font-weight: 400;\">These are just a few examples of the data that\u2019s being captured by the Social Opinion API.<\/p>\n<h3>TweetReceivedEventArgs and StreamDataSavedEventArgs<\/h3>\n<p style=\"font-weight: 400;\">Finally, the objects we\u2019ve just looked at (and others) are passed back in custom\u00a0<strong>Event Args<\/strong>.\u00a0 These contain data for your client application to handle.\u00a0 You can see an example of some of this data here:<\/p>\n<pre class=\"prettyprint\">public\u00a0class\u00a0Data\r\n{\r\n    public\u00a0string\u00a0id {\u00a0get;\u00a0set; }\r\n    public\u00a0DateTime created_at {\u00a0get;\u00a0set; }\r\n\u00a0   public\u00a0string\u00a0text {\u00a0get;\u00a0set; }\r\n    public\u00a0string\u00a0author_id {\u00a0get;\u00a0set; }\r\n    public\u00a0string\u00a0in_reply_to_user_id {\u00a0get;\u00a0set; }\r\n    public\u00a0List&lt;ReferencedTweet&gt; referenced_tweets {\u00a0get;\u00a0set; }\r\n    public\u00a0Entities entities {\u00a0get;\u00a0set; }\r\n    public\u00a0Stats stats {\u00a0get;\u00a0set; }\r\n    public\u00a0bool\u00a0possibly_sensitive {\u00a0get;\u00a0set; }\r\n    public\u00a0string\u00a0lang {\u00a0get;\u00a0set; }\r\n    public\u00a0string\u00a0source {\u00a0get;\u00a0set; }\r\n    public\u00a0List&lt;ContextAnnotation&gt; context_annotations {\u00a0get;\u00a0set; }\r\n    public\u00a0string\u00a0format {\u00a0get;\u00a0set; }\r\n}\r\n\r\npublic\u00a0class\u00a0MatchingRule\r\n{\r\n   public\u00a0string\u00a0id {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0public\u00a0string\u00a0tag {\u00a0get;\u00a0set; }\r\n}\r\n\r\npublic\u00a0class\u00a0FilteredStreamDataResponse\r\n{\r\n    public\u00a0Data data {\u00a0get;\u00a0set; }\r\n\u00a0\u00a0\u00a0\u00a0public\u00a0List&lt;MatchingRule&gt; matching_rules {\u00a0get;\u00a0set; }\r\n}<\/pre>\n<h2>Storing Filtered Stream API data<\/h2>\n<p style=\"font-weight: 400;\">The JSON payload from the Filtered Stream API contains a lot of data points so there the number of tables that are needed to support it.<\/p>\n<p style=\"font-weight: 400;\">Here you can see a subsection of a database in Azure where I tell the Social Opinion API to push processed data into:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-1765\" src=\"http:\/\/www.jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image007.jpg\" alt=\"\" width=\"753\" height=\"130\" srcset=\"https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image007.jpg 753w, https:\/\/jamiemaguire.net\/wp-content\/uploads\/2020\/04\/image007-750x129.jpg 750w\" sizes=\"auto, (max-width: 753px) 100vw, 753px\" \/><\/p>\n<p style=\"font-weight: 400;\">The actual number of tables required to support the Detailed JSON payload goes into the double-digits. I\u2019ve mapped them all!\u00a0 Those weekends were busy!<\/p>\n<h2>Housing the Social Opinion API<\/h2>\n<p style=\"font-weight: 400;\">The Social Opinion API is housed in Azure running 24&#215;7 and is processing data as I write this blog. The API has a feature that lets you switch it on or off and detailed error logging to help diagnose errors. I may create a web UI that lets me search for errors.<\/p>\n<p style=\"font-weight: 400;\">For the time being, I currently write direct SQL to the database to a system standard logging table that\u2019s used by the Social Opinion API.<\/p>\n<h2>Fault Tolerance and Twitter ToS<\/h2>\n<p style=\"font-weight: 400;\">In addition to parsing and processing the Detailed JSON payload to custom C# POCOS, the Social Opinion API features built-in configuration and error handling.<\/p>\n<p style=\"font-weight: 400;\">It will also let you configure how many tweets to fetch and when to exit on the existence of a given number of errors.<\/p>\n<p style=\"font-weight: 400;\">This also makes sure any calls made to the Filtered Stream API adhere to the Twitter Terms of Service.<\/p>\n<h2>Telemetry, Metrics, Resources and Diagnostics<\/h2>\n<p style=\"font-weight: 400;\">During testing, I tracked the following key data points when <strong>1 Rule<\/strong> was set up and approximately\u00a0<strong>300k Tweets<\/strong>\u00a0were processed.<\/p>\n<ul>\n<li style=\"font-weight: 400;\">1,500 Tweets processed every 5 minutes<\/li>\n<li style=\"font-weight: 400;\">2.8% error rate for every 1,000 Tweets processed<\/li>\n<li style=\"font-weight: 400;\">3GB database storage usage (Azure SQL)<\/li>\n<li style=\"font-weight: 400;\">Maximum of 60% database resource usage at any given time<\/li>\n<li style=\"font-weight: 400;\">421,609 Context Annotations identified<\/li>\n<li style=\"font-weight: 400;\">160,931 Entity Annotations identified<\/li>\n<li style=\"font-weight: 400;\">44 Domains Identified<\/li>\n<li style=\"font-weight: 400;\">1,727 Entities Identified<\/li>\n<\/ul>\n<p style=\"font-weight: 400;\">With these sorts of metrics, the Social Opinion API will quickly hit the limit of 500,000 Tweets per month.\u00a0 In a recent video call with the Twitter Product Team, one of the things I politely asked for was more data. &#x1f609;<\/p>\n<h2>Summary<\/h2>\n<p style=\"font-weight: 400;\">This has been a follow on from last week\u2019s blog post and we\u2019ve seen how you can connect to the Twitter Labs Streaming API.<\/p>\n<p style=\"font-weight: 400;\">We\u2019ve looked at an interface that lets you process data from the Filtered Stream API, the data you can expect to get back and some of the in-built features and metrics.\u00a0\u00a0I\u2019m working on a set of dashboards and visualisations for this data.<\/p>\n<p style=\"font-weight: 400;\">Another idea I have is around placing a chatbot in front of the data and insights that are being surfaced by the Twitter Filtered Stream API and Social Opinion API.<\/p>\n<ul>\n<li style=\"font-weight: 400;\">Got a question?<\/li>\n<li style=\"font-weight: 400;\">Want to know more?<\/li>\n<li style=\"font-weight: 400;\">Interested in this interface?<\/li>\n<\/ul>\n<p style=\"font-weight: 400;\">Drop me a message below or reach out on social.<\/p>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<p style=\"font-weight: 400;\">\n","protected":false},"excerpt":{"rendered":"<p>In my last post, I introduced the Twitter Labs Filtered Stream API. In that post, I explored the main features of the API and the rich data you get back. I looked at the rich&#8230;<\/p>\n","protected":false},"author":1,"featured_media":1759,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","footnotes":""},"categories":[59,6,8,35],"tags":[],"class_list":["post-1758","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-asp-net-core","category-c","category-prototyping","category-twitter"],"_links":{"self":[{"href":"https:\/\/jamiemaguire.net\/index.php\/wp-json\/wp\/v2\/posts\/1758","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jamiemaguire.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jamiemaguire.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jamiemaguire.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jamiemaguire.net\/index.php\/wp-json\/wp\/v2\/comments?post=1758"}],"version-history":[{"count":0,"href":"https:\/\/jamiemaguire.net\/index.php\/wp-json\/wp\/v2\/posts\/1758\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/jamiemaguire.net\/index.php\/wp-json\/wp\/v2\/media\/1759"}],"wp:attachment":[{"href":"https:\/\/jamiemaguire.net\/index.php\/wp-json\/wp\/v2\/media?parent=1758"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jamiemaguire.net\/index.php\/wp-json\/wp\/v2\/categories?post=1758"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jamiemaguire.net\/index.php\/wp-json\/wp\/v2\/tags?post=1758"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}