Software Architect / Microsoft MVP (AI) and Pluralsight Author

AdTech, C#, Prototyping, Social Media, Twitter

#Promote – Drinking from the Twitter Firehose

Introduction

In order to get access to real-time in Twitter data there are a few options.  You can consume the REST APIs directly at the Twitter end.  You can use 3rd party products such as LINQ to TwitterTweetInvi or you can pay a few grand a month to GNIP and get jacked into the Twitter Matrix.

In this short post you’ll see how to extract data directly from the real-time Streaming API by using TweetInvi.

Main Steps

The main steps are as follows:

  1. Generate your Twitter Application
  2. Download and install TweetInvi
  3. Create a Console Application in Visual Studio
  4. Write code to connection directly to the Steaming API

In the console application you also need to setup authentication Tokens and Secrets and pass these into TweetInvi.  When this is done, you can create a FilteredSteam.

Show me some code!

The following code extract is a working method that connects directly to the Twitter Streaming API and pulls down hundreds of tweet a second for anything that referenced “brexit”:

private static void Stream_FilteredStreamExample()
{
    StringBuilder sb = new StringBuilder();

    for (;;)
    {
        try
        {
            ExceptionHandler.SwallowWebExceptions = false;
            var stream = Tweetinvi.Stream.CreateFilteredStream();

            stream.AddTrack("brexit");

            stream.AddTweetLanguageFilter(Tweetinvi.Core.Enum.Language.English);

            var tweetCount = 0;
            var timer = Stopwatch.StartNew();

            stream.MatchingTweetReceived += (sender, args) =>
            {

                tweetCount++;
                var tweet = args.Tweet;

                totalTweets += 1;

                Console.WriteLine("Location:-" + tweet.CreatedBy.Location);

                if (timer.ElapsedMilliseconds > 1000)
                {

                    Console.ForegroundColor = ConsoleColor.Green;
                    Console.WriteLine("\n{0}: {1} {2}", "ID:" + tweet.Id + " ", "(" + tweet.Language.ToString() + ")", "Text:" + tweet.Text);

                    Console.ForegroundColor = ConsoleColor.White;

                    timer.Restart();
                    Console.WriteLine("\tTweets/sec: {0}", tweetCount);
                    tweetCount = 0;
                }
                else
                {
                    Console.ForegroundColor = ConsoleColor.Green;
                    Console.WriteLine("\n{0}: {1} {2}", "ID:" + tweet.Id + " ", "(" + tweet.Language.ToString() + ")", "Text:" + tweet.Text);
                }
                Console.ForegroundColor = ConsoleColor.Yellow;
                Console.WriteLine("Total tweets:" + totalTweets.ToString());


            };

            stream.StartStreamMatchingAnyCondition();
        }
        catch (Exception ex)
        {
            Console.WriteLine("Exception: {0}", ex.Message);
        }
    }
}

The Output

Here you can see the console application pulling several tweets a second.

firehose
The Twitter object graph contains lots of useful metadata which can be interrogated to derive actionable insights.

Ideal for marketing and ad-tech purposes.

More on that later!

 

JOIN MY EXCLUSIVE EMAIL LIST
Get the latest content and code from the blog posts!
I respect your privacy. No spam. Ever.

Leave a Reply