Integrating IBM Watson with the Twitter API (Node.js)


My team at Hack Reactor is currently building a Node application that integrates the new IBM Watson Service with the Twitter API (using the Twit API client). The application is based on Personify.js--an open source JavaScript library we built to allow for easier integration of these tools.

You might remember Watson from his (?) appearance on Jeopardy in 2011. In October 2014, IBM opened up Watson's cognitive computing power to the public with a menu of free beta API services. One of the most interesting, and the service which our team is integrating with the Twitter API, is Watson's User Modeling.

With User Modeling Watson "uses linguistic analytics to extract cognitive and social characteristics, including Big Five personality, values, and needs, from text." Our team thought it would be really interesting to tap into Twitter's Search API, which contains an archive of all publicly available tweets, to compare an analysis of tweets between individual states and the US as a whole.

Here is how the application works in a nutshell: The end user selects a specific state and passes a topic of interest, e.g., #nike. On the backend, the topic passes through the Twitter API, and from there the return tweets pass through Watson who (?) then gives back the User Modeling analysis.

Before moving forward, I would suggest taking a look at IBM's Watson documentation, which is quite detailed, in order to help you get your application ready for the Watson integration.

Integration

In order to create a successful integration, you will need to modularize your code as much as possible as there are many functions in both the Twitter API and Watson.

First things first, open a free account with IBM Bluemix, which you will use to connect to Watson's services and to deploy your application. Next, it would be a good idea to have a look at the sample code from the IBM Watson documentation.

Below is the basic setup of our application. While the code examples are Node.js-specific, IBM does provide examples for other languages. I have also commented inline in the code examples.

I first created a "server" folder which includes three files: app.js, twitter.js, and watson.js. Here is what the post request inside app.js looks like:

var twitter = require('twitter');

app.post('/', function(req, res){
  twitter.twitterData(req, res);
}); 

And the twitter.js:

var Twit = require('twit');
var watson = require('watson');

module.exports.twitterData = function(req, res) { 
var analyzeData = function () {
      dataFromTwitter = "";
      Tweet.get('search/tweets', { q: ''+req.body.subject+' since:2014-10-01', 
                               count: 5000, lang: 'en' },
                               function(err, data, response) {
        for(var i = 0; i < data.statuses.length; i++) {
          // accumulate the data (each tweet as a text) received from twitter
          dataFromTwitter += data.statuses[i].text;
        }
        // we pass Twitter data to Watson after fetching Twitter data completed
        watson.watson(dataFromTwitter, res);
      });
    };
    analyzeData();
};

Finally the watson.js

module.exports.watson =  function(data, res) {

  var parts = url.parse(service_url.replace(/\/$/,''));
  var profile_options = { host: parts.hostname,
    port: parts.port,
    path: parts.pathname + "/api/v2/profile",
    method: 'POST',
    headers: {
      'Content-Type'  :'application/json',
      'Authorization' :  auth }
    };

  create_profile_request(profile_options, data, res)(function(error,profile_string) {
    console.log('dataFromTwitterUS', data);
    if (error) console.log(error);
    else {
      // parse the profile and format it
      var profile_json = JSON.parse(profile_string);
      var flat_traits = flatten.flat(profile_json.tree);

      // Extend the profile options and change the request path to get the visualization
      // Path to visualization is /api/v2/visualize, add w and h to get 900x900 chart
      var viz_options = extend(profile_options, { path :  parts.pathname + "/api/v2/visualize?w=900&h=900&imgurl=%2Fimages%2Fapp.png"})

      // create a visualization request with the profile data
      create_viz_request(viz_options,profile_string)(function(error,viz) {
        if (error) res.render('index',{'error': error.message});
        else {
          //Here we get the results from Watson and send it back to the client
          res.send(flat_traits);
        }
      });
    }
  });
};