A recent study by Mckinsey & IDC found that 20% of a knowledge worker’s time was used up by searching for market information. On average, they calculate that for every 1,000 knowledge workers employed in an organization, this represents approximately 3 million USD / year wasted looking for market insights. In total, this costs the Fortune 1000 companies 2.5 billion USD a year in searching for data that cannot be found.
IDC refers to this as the “Knowledge Deficit”. When considering that for 2019, ESOMAR estimated 45 $B in Market Research Data and Services, this deficit represents a major issue for companies to overcome.
AI applied to Market Intelligence is more than ever being recognized as key to tackling the ever growing insight gap, with AI being pushed to the forefront of strategic recommendations.
AI is a powerful technology that can read thousands of articles per minute, understand important concepts, link topics together, highlight contradictions, in other words an AI properly trained is able to takeover knowledge workers’ repetitive, time consuming and complex tasks an complete it in couple minutes where it would typically take us hours, when not days.
The table below illustrates the performance difference between deep learning algorithm vs human intervention, emphasizing a x30,000 factor improvement when using an algorithmic approach.
| Human | Machine Learning |
| 1 article / min | 1,000 articles / s. |
Deep learning, which is a facet of machine learning, is capable of self- learning from data that is unstructured or without metadata. The advantage of using this type of technology is that it learns on its own from a large amount of training data – in ReportLinker’s case training data is made of economic articles, analysis and reports – and it is capable of reiterating through this process as many times as it is required to achieve acceptable precision.
Despite the advantages machine learning offers, effective & robust insight retrieval requires rigorous methodology when it comes to implementation, in order to avoid common pitfalls such as over-fitting.
We have modeled the tasks undertaken by our data scientists. Below is an example of the steps they follow to detect industry & sector specific insights:
We prepared an annotated corpus of 100,000+ sentences providing typical insights from a range of 20+ categories, spanning from Market forecast, to Merger and Acquisitions, to consumer insights.
We prepared a list of potential features the model learns from, such as word embedding, article topics, named entities found in the text etc … All together we ended up with a list of 400 features
We built a data pipeline based on Tensorflow, using python and libraries such as Keras, Pandas… and we conducted the learning task on a set of 80,000 sentences.
The 20,000 remaining annotated sentences were kept safely outside of the learning process as a separate test corpus. This way we were able to quickly assess precision and recall of each iteration and adapt the features accordingly. This allowed us to eliminate the risk of overfitting our model. We finally came up with a model providing the required performance we needed.
Once implemented into production, results obtained with this method were much better than manual approaches, such as Boolean research techniques.
AI enables us to detect emerging trends and early signals thanks to our trained learning algorithm.
ReportLinker provides AI driven market intelligence solutions that automate insights discovery from large amounts of data.
This type of technology offers enormous potential to facilitate the role of MI professionals including an increased productivity and accuracy.
It is becoming the new norm because of the benefits it represents such as: