IBM Launches Watson Discovery – AI-Powered Search Tool, Now Supports Hindi Language

* Leading Global Brands including EY use Watson to Infuse Intelligence into Workflows.

* IBM Research innovations in natural language processing, automation and building trust to be commercialized in Watson products

IBM (NYSE: IBM) today announced innovative new capabilities planned for IBM Watson designed to help scale the use of AI by businesses.

Developed by IBM Research, the new capabilities are designed to improve the automation of AI, provide a higher degree of precision in natural language processing, and foster greater trust in outcomes derived from AI predictions. They include:

* Reading Comprehension is based on an innovative question-answering (QA) system from IBM Research. Currently in beta in IBM Watson Discovery, it is planned as a new feature that can help identify more precise answers in response to natural language queries from vast troves of complex enterprise documents. It also provides scores that indicate how confident the system is in each answer.

* FAQ Extraction uses a novel NLP technique from IBM Research to automate the extraction of Q&A pairs from FAQ documents. Currently in beta in IBM Watson Assistant’s search skill, it is planned as a new feature to help businesses keep virtual assistants up-to date with the latest answers and reduce the time-consuming process of manual updates.

* A new intent classification model is now available in IBM Watson Assistant. It is designed to more accurately understand an end-user’s goal or intent when engaging with a virtual assistant and to enable administrators to train the system faster. The model provides more accurate results from less data versus compared commercial systems1. This can help businesses go live with virtual assistants in a few days while achieving high accuracy.

* Watson Discovery now includes support for 10 new languages including Hindi. With this capability, it can now natively understand the meaning of Hindi text written in Devanagari script and extract useful information from it such as business keywords, named entities, relationships, perform sentiment analysis, and other such NLP features. IBM Research - India collaborated with its AI Horizon Network partner – IIT Bombay to lead this effort.

Advancing AI Governance Capabilities to Help Build Trust and Transparency in Business Outcomes

IBM also announced plans to commercialize IBM Research-developed ‘AI Factsheets’ in Watson Studio in Cloud Pak for Data throughout the next year. Like nutrition labels for foods or information sheets for appliances, AI Factsheets are designed to provide information about a product’s important characteristics. Standardizing and publicizing this information will help build trust in AI services across the industry.

To complement this, IBM Services for AI at Scale, a new consulting offering, provides businesses with a framework, methodology, and underlying technology to guide organizations on their journey to  trustworthy and ethical AI. IBM Cloud Pak for Data also has new capabilities to provide a complete foundation for AI that can run on any cloud. Read more about the AI Governance updates here.

'Today's announcement reiterates IBM's focus on helping clients innovate across all three foundations of AI – natural language processing, building trust, and automation. IBM Research India has been key partner and leader in IBM’s AI advancements with a focus on Trustworthy AI initiatives and along with US team (IBM Research Yorktown) conceptualized the core idea of AI FactSheets. In the area of natural language processing - by using a combination of AI-deep learning, shallow semantic understanding models we have enabled Watson to understand Hindi language natively. For instance, Watson Discovery can natively understand documents in Hindi and extract meaning, without having to resort to any translation to English -- an outcome of our collaboration with IIT Bombay’s Center for Indian Language Technology as a part of  IBM’s AI Horizon Network. - said Gargi Dasgupta –Director, IBM Research India.

"India has taken positive steps to establish a leadership role in AI and IBM-IITB AIHN (AI Horizon Network) project is an important step - both for science and technology and for India's progress. The collaboration is focused on automation of cognition and perception, with emphasis on cutting edge research, high quality publications and open and widely usable AI resources. There are social and commercial needs, whose servicing requires user interaction and information dissemination in Indian languages (IL), however the complexity of Indian languages, low Indian language corpora and other constraints makes adoption and adaptation of English centric NLP very difficult for IL-NLP. Through our partnership with IBM, we have been able to use machine learning for IL-NLP and address challenges related to low resource, understanding of Hindi language sense, intent, sentiment and more. In two years, the endeavour has seen stellar publications coming out of the collaboration and creation of quality software (in NLP, Speech and Multimodal AI). " – said Prof Pushpak Bhattacharyya, Professor - Department of Computer Science and Engineering, IIT- Bombay.

Leading global brands are using IBM Watson to infuse intelligence into their workflows.  For instance, a global leader in assurance, tax, strategy and transactions and consulting services, EY is transforming the M&A due diligence process using IBM Watson Discovery. Following the global expansion of its alliance with IBM earlier this summer, EY launched Diligence Edge, which includes a custom NLP model trained on EY proprietary M&A language, to help deal makers identify and leverage valuable insights throughout the transaction’s lifecycle, as well as make informed decisions on deal value opportunities to gain an advantage in competitive deal processes.

Today’s news is another step from IBM to advance its AI for business strategy: from commercializing advanced NLP capabilities from Project Debater; expanding its AI-powered automation capabilities with the launch of IBM Watson AIOps, acquisitions of WDG Automation and Instana2 and expanded ecosystem partnerships including ServiceNow; and helping businesses, healthcare organizations, governments and academic institutions use IBM Watson Assistant to get information they deem trustworthy on the pandemic to stakeholders.

For more information on today's announcement, please visit: https://newsroom.ibm.com/new-innovative-capabilities-for-watson

[1] Qi et. al, 2020: Benchmarking Intent Detection for Task-Oriented Dialog Systems https://arxiv.org/pdf/2012.03929.pdf

[2] Expected to close within several weeks.

About IBM Watson

Watson is IBM’s AI technology for business, helping organizations to better predict and shape future outcomes, automate complex processes, and optimize employees’ time. Watson has evolved from an IBM Research project, to experimentation, to a scaled, open set of products that run anywhere. With more than 30,000 client engagements, Watson is being applied by leading global brands across a variety of industries to transform how people work.