Intelligence from Social List Hashtags Can Power Human Search

Microsoft Researchers collectively with professors from IIT Kharagpur are working towards developing a system that can form the basis for a deeper, more meaningful search engine. Microsoft’s Senior Applied Researcher Manish Gupta recently partnered with Ankan Mullick, Prof. Pawan Goyal, and Prof. Niloy Ganguly from IIT Kharagpur, to conduct a study on extracting meaningful information from social conversations to help search engines answer social list queries better. by deploying artificial intelligence and machine learning.

While search engine algorithms are great at working with fact-based queries and providing structured answers, they are surprisingly ineffective at answering subjective and personal questions. Queries based on human experiences and personal opinions are difficult for a standard search engine to comprehend. Therefore, they fail to answer questions such as “How to make small talk with new friends,” “People’s favourite memories from school,” “How does it feel to immigrate to a new country?” or “The songs that defined the 80s.”

The team used multi-word hashtags, also called “idioms” from Twitter, to conduct an in-depth study to make search results much more personal and human. The researchers collected around 4 million hashtags that were trending between January 2015 and June 2015, and used a SVM (Support Vector Machine) classifier to conduct this research. The classifier worked on deeply personal and human hashtags such as #foreveralone, #awkwardcompanynames, #childhoodfeels, and #africanproblems using factors such as duration of hashtag popularity, related hashtags, URLs, and related hashtags to detect context and classify the social lists accurately. The algorithm used to conduct this study forms the basis for a better search engine for social platforms which can assist users looking for subjective information and trusted opinions.

The study eventually proved that, by analysing multi-word hashtags search engines can now scour social networks and detect valuable insights from public conversations, using a highly effective and precise algorithm to provide a deeper, more meaningful search experience.