With new technologies appearing more rapidly than ever before, how do you pick the ones that will deliver competitive advantage and spot the ones that could threaten your business? Dr Letizia Mortara, Senior Research Associate in the IfM’s Centre for Technology Management, says companies need comprehensive technology intelligence (TI) systems to make sure they spot relevant new technologies and understand how effective they are at doing so.

 

Artificial intelligence, augmented and virtual reality, nanotechnologies, synthetic biology, 3D printing, driverless cars and interplanetary travel: new potentially disruptive technologies and speculation about how they will evolve appear in the media on an almost daily basis. But how do you know which of the emerging technologies will present an opportunity or threat to you and your company? 

 

Picking up the early warning signs and understanding the implications of new technological trends is not a trivial or straightforward task. Just ‘Googling it’, for example, will not deliver sufficient insights for the identification of new technology opportunities and support the implementation of an effective strategy for businesses in today’s competitive market.

 

TI activities aim to identify early technology breakthroughs and trends that could create long-term competitive advantage or could impact negatively on the business. As firms come under increasing pressure to maintain a rapid pace of innovation they are dedicating more resources to developing TI systems that can efficiently capture information from the external environment in order to develop insights that support a variety of decision-making and strategic planning activities.

 

TI activities span a firm’s functions and roles and are often informal or carried out ad hoc. These may include scouting networks, patent mining tools, calls for information via idea competitions or working with external intermediaries or consultants that search and communicate the new trends for you. Although most firms carry out some form of TI, relying on ad hoc approaches makes it hard to evaluate if a TI system is making an effective contribution to the company’s performance. 

 

Furthermore, even if there is extensive and specific knowledge within a company about approaching technologies, getting that information efficiently and accurately to decision-makers can sometimes be difficult. So how does a company know if the technology intelligence system it has put in place is performing well? 

 

Evaluating the quality of technology intelligence: effective and efficient TI 

We wanted to develop a framework that companies could use to evaluate their TI system regardless of what type of TI activities they are undertaking. To this end, we reviewed what companies are doing in practice and integrated our findings with an academic understanding of the development of impact measurements. Twelve global companies ranging in size from 800 to more than 250,000 employees participated in this research. We asked the companies how they undertook and measured TI – both formally and informally.

 

We used their responses to develop the TI evaluation matrix (see right) which incorporates the main strategies being used by companies to measure TI performance and suggests that companies should use a combination of these metrics to appraise TI activities. This evaluation matrix can help companies do a number of things: contextualise their TI performance analysis; structure and organise their TI measuring and evaluation strategy; understand the limitations of some metrics and encourage them to adopt more than one method for measuring TI performance. 

 

The evaluation matrix combines measures of the intensity of the TI activity (how much TI work has been done) with the TI impact (the quality of the outcome of TI). These can be reviewed in the short-term (the success of a project) or in the long-term (the health of a company). 

 

The evaluation matrix encompasses four types of metrics: 

• Activity-based and project-specific: These are the most easily quantifiable metrics and are less subject to personal bias. An example of this metric is a measure of how much work has been done to review the trends and implications of new technologies for a project. For example, how many patents have we reviewed? These metrics are useful for understanding how far we have got in the review of the information available, but stays away from measuring what the information is telling us. This type of metric is useful when we want to avoid forming an opinion too early, based on only partial data and hence it is useful for TI reviews which take a long time. However, as the extent of the TI effort is not necessarily reflected in the final outcome, other metrics are needed to complement these. 

 

• Activity-based and firm-specific: This type of metric evaluates how TI is benefiting the whole company. It looks, for example, at how established and integrated the TI activity is with the rest of the organisation. For example, are we capable of using TI insight to guide the firm? Can we do patent analysis? Are we aware of what the cumulative TI insights are telling us? Can we embed these in our decisions and planning systematically? 

 

• Outcome-based and project-specific: Some metrics are employed after a project is completed and evaluate the success of the project for which TI was gathered. However, as a project’s success can be impacted by many other factors beyond the quality of TI, such as the resistance of a decision-maker to act on the insight, this measure cannot be used in isolation. 

 

• Outcome-based and firm-specific: This is important for measuring the long-term effectiveness of TI for the company. Data needs to be collected over a long time period to understand whether TI activities are suitable for the company needs and are helping it survive and prosper. Very few of the companies surveyed measured TI in this quadrant and it is often subjectively measured as a ‘sense’ or ‘feel’ about how well TI is delivering value to the firm. However, if short-term metrics are not used in combination with these long-term ones, the TI officers might struggle to demonstrate the value of their work.

 

 

 

 

 

 

 

 

TI evaluation matrix (Y.W. Loh and L. Mortara) with some examples of metrics in each category.

 

 

 

 

 

 

Communicating technology intelligence 

This evaluation matrix is useful both as a structured way of capturing and assessing the range of TI activities within an organisation and as a way of communicating the importance of TI to decision-makers. 

 

This can be something of a challenge. All the companies we interviewed reported difficulties in communicating their TI insights to senior decision-makers. Even when the insights are clear, relevant and well-presented, they are not always acted upon. Much depends on the inevitably subjective perceptions and personalities of the decision-makers who sometimes tend to underestimate the depth and complexity of the TI activity. One of our interviewees said: 

“…you do get people in management who think that everything is on the internet, where you can just Google [it]. We are trying to prove that in fact this is not the case.” 

 

If TI is to be a successful endeavour it needs to be well structured and well communicated. The evaluation matrix can help with the former and our practical guide, Communicating Technology Intelligence, can help with the latter. Download the report.

 

Related short courses 

Technology and Innovation Management

Find out more