MIT predicts trending topics on Twitter

Associate Professor Devavrat Shah and his student Stanislav Nikolov will unveil a new algorithm in November at the Interdisciplinary Workshop on Information and Decision in Social Networks. With a 95 percent accuracy it is reported to predict which topics will trend on Twitter by an average of an hour-and-a-half advance. MIT says that the social network could make use of the algorithim and even charge extra for ads that are linked to the future popular topic. At the same time, it can be implemented to determine ticket sales for movies, duration of bus rides and possibly stock prices.

The latter might gather some interest from Wall Street, but judging by the presentation that MIT made so far it is not at the top of the list. The idea behind the algorithm is to find meaningful patterns from a sample set, which in Twitter's case is represented by topics that were not popular as well as previously trending ones.

Shah says that instead of assuming a model, he and his student "just let the data decide". In their experiments the training set implemented used data from 200 Twitter topics that trended and 200 that did not. The algorithm was "set loose on live tweets" and delivered a 95 percent accuracy and a 4 percent false-positive rate.

Despite the small training set, Shah says that the results are relevant and can improve with more input data. Even if the computational resources increase with the training set, it is still suited for large cloud services and can be split among machines for optimization. The algorithm appears suited for social media where it can be used with a more precise set of input data, as it requires to filter less noise in the historical data.

Photo Credit: Anneka/Shutterstock