How to Create an AI Social Listening Tool using Azure and the Twitter API Part 2: Creating a Function to Fetch Data

In Part 2 of this mini-series, we create the Azure Function “TwitterFunction”.

This function will be responsible for fetching data using the Twitter API Recent Search endpoint.

The Social Opinion API is a wrapper for the Twitter API V2 endpoints and makes it simple and quick to fetch tweets.

Tweet data is then stored in a container raw-tweets.

Prerequisites

To implement this, you will need:

• Empty Solution in Visual Studio 2022
• Twitter API Developer Account
• Azure Subscription

Read Part 1 of this mini-series if you need more information about these.

Process Overview

The sequence diagram below shows how this Azure Function (TwitterFunction) operates:

A timer trigger sends request to the Twitter API Recent Search API endpoint via the Social Opinion API/SDK.

A list of strongly typed tweet objects is then returned.  This data is then serialized to JSON and pushed into a blob storage account in Azure raw-tweets.

Twitter Function Logic

The logic for the Azure Function is relatively simple.

1. Supply OAuth credentials
2. Instantiate Social Opinion API Recent Search Service
3. Authenticate
4. Supply search criteria
5. Send Request
6. Parse response from Twitter API
7. Generate unique JSON filename with data for each tweet and write file data
8. Push file to Azure Blob Storage

You also need your Twitter API access secret, access token, consumer key, and consumer secret.

Twitter Function Code

To implement this Azure Function, references to the following NuGet packages are needed:

• Storage.Blobs
• NET.Sdk.Functions
• SocialOpinionAPI

Let’s dig into the main parts of the Azure Function.

First, the blob storage container name and container connection string are defined as private properties at the class scope in the Azure Function:

private static string containerName = "raw-tweets";

private static string containerConnectionString = "{connString";

The main method in the function GetRecentSearchData:

[FunctionName("GetRecentSearchData")]

public static void Run([TimerTrigger("*/30 * * * * *")] TimerInfo myTimer, ILogger log)
        {
         RecentSearchService searchClient = new RecentSearchService(new OAuthInfo
            {
                AccessSecret = "",
                AccessToken = "-",
                ConsumerKey = "",
                ConsumerSecret = ""

            });

       log.LogInformation($"C# executed at: {DateTime.Now}");

        // invoke Recent Search API to fetch 100 tweets that mentioned Microsoft
           var response = searchClient.SearchTweets("Microsoft lang:en", 100, 3);

            // loop through each page of data
            foreach (var tweet in response)
            {
                // and each tweetData item in the page
                foreach (var data in tweet.data)
                {
                  // raw tweet data and meta data
                  var tweetJson = JsonConvert.SerializeObject(data);

                  // construct a unique filename for the tweet in blob storage
                  var name = "tweet_" + data.id + "_" + data.author_id + ".json";

                   // write to blob storage
                    UploadToBlobStorage(name, tweetJson, log);
                }
            }
        }
     }

The above code contains a method UploadToBlobStorage:

private static void UploadToBlobStorage(string name, string jsonData, ILogger log)
{

BlobContainerClient blobContainerClient = new    BlobContainerClient(containerConnectionString, containerName);   using (MemoryStream memoryStream = new MemoryStream(Encoding.ASCII.GetBytes(jsonData)))
 {
   blobContainerClient.UploadBlob(name, memoryStream);
   log.LogInformation($"Tweet data uploaded to container at: {DateTime.Now}");
  }
}

UploadToBlobStorage takes serialized tweet JSON data and creates a unique file per tweet in the blob storage container in Azure.

Azure – Blob Storage Container

A storage account tweet-data is created in Azure:

The storage account (tweetdata) has a container raw-tweets :

Raw-tweets contains the raw and unprocessed serialized tweet data that TwitterFunction creates.

Testing

Twitter Function can be tested in Visual Studio. In this test, the following query is passed into Twitter Function:

// invoke Recent Search API to fetch 100 tweets that mentioned Microsoft

var response = searchClient.SearchTweets("Microsoft lang:en", 100, 3);

This call will search Twitter for any tweets that mention Microsoft that are written in English.

100 tweets are to be returned.

3 attempts are allowed at fetching this data in the event the connection drops and to prevent the Twitter Developer Account from hitting rate limits.

When the Azure Function Timer Trigger kicks into action, the console signals that data is being generated and move to Azure Blob Storage:

We can examine the data in the container in Azure:

You can further check the contents by opening a JSON file to test if the content (Microsoft) is mentioned in the tweet: