Looking back on 2018
This post is recap of 2018, the projects and the respective tech I’ve been using. The main themes in 2018 were:
- Building intelligent chatbots using the Microsoft Bot Framework v3
- Rewriting v3 chatbots to use Microsoft Bot Framework v4!
- Becoming well versed in the Microsoft Cognitive Services and Bing APIs
- Continuing to develop with the Twitter APIs
- Submitting an entry for Twitters annual event #Promote
- Getting to grips with the Facebook and Instagram APIs
- Collaborating with the National Geographic Photo Ark Team
- Building a new API that extracts cryptocurrency discussions and news from sources such as Bing News, Reddit, Stock Twits and Twitter
Chatbots and the Bot Framework
I done A LOT with the Microsoft Bot Framework v3 in 2018. With the release of BFv4 I had to unlearn most of BFv3 half way through the year!
I rebuilt chatbots from using BFv4 but happy to report it gives developers a better coding experience and the architecture is much easier to work with. Some of the key differences are found in the following areas:
- Dialogue and conversation management
- State management
- New components to handle the NLP aspects of bot development (LUIS)
Each time I found a topic of interest I’d add it to OneNote; this list of topics grew quite a bit as I worked my way through the Bot Framework.
I now have a list of topics that I might cover in a set of future blog posts. These could form a set of bot development tutorials. Writing these into a series of posts helps firm up my knowledge and if you’re about to get into chatbot development and AI, it’ll help you too!
Twitter ran promote again in 2018, I didn’t have much time to build out a new feature for Social Opinion so drafted in help from a contact.
We extended existing functionality to build a new feature that was able to listen for commercial intent signals then automatically contact users with a contextual advert, direct message or mention them. All of which helped digital marketers contact the right person, at the right time, with the right message.
Putting my personal Twitter Account on Autopilot
Over the course of a weekend I wrote a Twitter Bot to manage content on my personal Twitter account. It auto-retweets, publishes and likes specific tech related content from accounts that I find interesting.
When building this I had to make sure the Twitter Bot kept within the strict automation guidelines (I did get black listed a few times!) but was able to find a reasonable threshold of being able to carry out roughly 600 interactions on Twitter per month without getting blacklisted.
In this screenshot below you can see the bot had a decent impact on the impressions, mentions and increased my followers.
It now runs on autopilot 90% of the time but I still check periodically to see what’s happening.
University of Michigan | National Geographic
I was contacted by a Professor at a University North America and asked if I could build an API that extracted social media data and signals such as Likes, Retweets, number of comments and other KPIs to help identify potentially viral content. This formed part of several activities as part of National Geographic’s Photo Ark Project which strives to help promote environmental and animal welfare.
I was able to leverage APIs I’d built in the past and tap into previous knowledge to accelerate this but extracting data from Facebook and Instagram was much more difficult than Twitter. Data security and application review procedures are tight, no doubt because of recent data scandals like Cambridge Analytica!
For example, in the screenshot below, you can see a small extract of the permissions you can apply for access to:
After you’ve tested your Facebook or Instagram application, you also need to record a video of how the permissions and data will be used.
In this screenshot below, you can see me testing the Facebook and Instagram query editor, extracting the number of “likes” and comments for an image I uploaded to an Instagram account:
The image has 1 Comment and 3 Likes which you can see match the image data on the Instagram site:
This is only the tip of the iceberg with these APIs and they follow a similar pattern to others I’ve used in the past.
Twitter Premium API Interface
I continued to dive deeper into the Twitter APIs, specifically the Premium offerings. The Premium APIs gave me access to more insights than their standard counterparts. Paying £1 per request meant that I had to optimise my code!
Whilst you still consume the Premium APIs over REST there are some “gotchas”. For example, the query parameters need to get issued in the JSON Body in some requests and paging through JSON responses is handled a little differently. I had to reach out to the Twitter Support Team to help me figure this out, something probably worth a separate blog post to detail how you can do this.
Twitter Audience Insights Interface
I put the finishing touches to an interface that returns valuable insights for users on Twitter. Some of the data it can return can include, but is not limited to:
- consumer buying styles
- education levels
- home value
- household income
- occupation types
You can drill down into breaks downs of each data point, for example, 75% of users are educated to BSc level, 45% of users have property with values between £200,000 – £300,000.
In the screenshot above you can see a live debug session and just a handful of the data points the interface I built returns. This information can then be blended with other datasets to give digital marketers more understanding of the audience they seek to serve.
Cryptocurrency, Day Trading and Stock Twits
Stock Twits is a social media network for financial traders. The platform is like Twitter and they offer developer APIs that let you tap into various types of finance related datasets. I was interested in mining cryptocurrency conversations on the platform and surfacing potential buy or sell signals.
I extended an existing sentiment analysis API I had built for Social Opinion and by adding additional layers of intelligence with Microsoft Cognitive Services and the Stanford POS Tagger, was able to surface valuable insights related to cryptocurrency discussions taking place in real-time.
Throughout the year I shared screen shots of visualisations that rendered these insights. For example, in the screen shot below you can see both Twitter and Stock Twits data being displayed related to #bitcoin.
Real-time line graphs plot the current mood of #bitcoin stock as people discuss it on Twitter and Stock Twits. This data is processed 24×7 (until you delete or disable your Campaign).
- top hashtags
- main keywords
- location data
- ..and much more
The solution (which I named ‘Market Falcon’) is a centralised dashboard for all your cryptocurrency news. It can help you find out what’s trending and can notify you of potential spikes or dips in public mood of a given stock, thereby giving you a heads-up on your cryptocurrency investments.
News from other data sources help give you additional context – all in one place and easy to understand.
The underlying API was developed in a modular fashion so adding other interface integrations such as Bing News, Google News or Reddit is straightforward.
Guest Writing and Microsoft MVP
Toward the end of 2017 I received nominations to become an AI Microsoft MVP (thank you!), the nominations were sent over to Redmond but that’s where it ended! A key piece of feedback was “more community involvement”.
In 2018 I wrote several blog posts on other websites to remedy that and work towards achieving MVP status. Most of these posts were related to artificial intelligence, natural language processing (NLP), chatbots, Microsoft Cognitive Services and Bing. You can read some of these here.
Microsoft Cognitive Services
Microsoft Bot Framework
I also setup a GitHub account to house code experiments and supplement some of the articles that were published which you can find here.
Useful Tools and Services
Some useful tools and services I used throughout 2018, some of these might surprise you or be obvious!
- Excel (yes)
- Microsoft Cognitive Services
- Visual Studio
- VS Code
Audio, Books, Blogs and Podcasts
Finally, it’s good to get away from the 1s and 0s…some interesting content creators:
- James Altucher – entrepreneurship, peak performance and ideas
- Jordan B Peterson – clinical psychologist and professor of psychology
- Jocko Willink – retired navy SEAL, author and podcaster
- Joe Rogan – comedian and podcaster
- Robert Greene – author
- Seth Godin – marketing and business
That’s about it, let’s see what 2019 brings!