5 best practices on collecting competitive intelligence data

Competitive intelligence data collection is a challenge. In fact, according to our survey of more than 1,000 CI professionals, it’s the toughest part of the job. On average, it takes up one-third of all time spent on the CI process (the other two parts of the process being analysis and activation).

A consistent stream of sound competitive data—i.e., data that’s up-to-date, reliable, and actionable—is foundational to your long-term success in a crowded market. In the absence of sound data, your CI program will not only prove ineffective—it may even prove detrimental.

By the time you’re done reading, you’ll have an answer to each of the following:

• Why is gathering competitive intelligence difficult?
• What needs to be done before gathering competitive intelligence?
• How can you gather competitive intelligence successfully?

Let’s begin!

Why is gathering competitive intelligence difficult?

It’s worth taking a minute to consider why gathering intel is the biggest roadblock encountered by CI pros today. At the risk of oversimplifying, we’ll quickly discuss two explanations (which are closely related to one another): bandwidth and volume.

Bandwidth

CI headcount is growing with each passing year, but roughly 30% of teams consist of two or fewer dedicated professionals. 7% of teams consist of half a person—meaning a single employee spends some of their time on CI—and another 6% of businesses have no CI headcount at all.

When the responsibility of gathering intel falls on the shoulders of just one or two people—who may very well have full-time jobs on top of CI—data collection is going to prove difficult. For now, bandwidth limitations help to explain why the initial part of the CI process poses such a significant challenge.

Volume

With the modern internet age has come an explosion in competitive data. Businesses’ digital footprints are far bigger than they were just a few years ago; there’s never been more opportunity for competitive research and analysis.

Although this is an unambiguously good thing—case in point: it’s opened the door for democratized, software-driven competitive intelligence—there’s no denying that the sheer volume of intel makes it difficult to gather everything you need. And, obviously, the challenges of ballooning data are going to be compounded by the challenges of limited bandwidth.

Key steps before gathering competitive intelligence

Admittedly, referring to the collection of intel as the initial part of the CI process is slightly misleading. Before you dedicate hours of your time to visiting competitors’ websites, scrutinizing online reviews, reviewing sales calls, and the like, it’s imperative that you establish priorities.

What do you and your stakeholders hope to achieve as a result of your efforts? Who are your competitors, and which ones are more or less important? What kinds of data do you want to collect, and which ones are more or less important?

Nailing down answers to these questions—and others like them—is a critical prerequisite to gathering competitive intelligence.

Setting goals with your CI stakeholders

The competitors you track and the types of intel you gather will be determined, in part, by the specific CI goals towards which you and your stakeholders are working.

Although it’s true that, at the end of the day, practically everyone is working towards a healthier bottom line and greater market share, different stakeholders have different ways of contributing to those common objectives. It follows, then, that different stakeholders have different needs from a competitive intelligence perspective.

Generally speaking:

• Sales reps want to win competitive deals.
• Marketers want to create differentiated positioning.
• Product managers want to create differentiated roadmaps.
• Customer support reps want to improve retention against competitors.
• Executive leaders want to mitigate risk and build long-term competitive advantage.

Depending on the size of your organization and the maturity of your CI program, it may not be possible to serve each stakeholder to the same extent simultaneously. Before you gather any intel, you’ll need to determine which stakeholders and goals you’ll be focusing on.

Segmenting & prioritizing your competitors

With a clear sense of your immediate goals, it’s time to segment your competitive landscape and figure out which competitors are most important for the time being.

Segmenting your competitive landscape is the two-part job of (1) identifying your competitors and (2) assigning each one to a category. The method you use to segment your competitive landscape is entirely up to you. There’s a number of popular options to choose from, and they can even be layered on top of one another. They include:

• Direct vs. indirect vs. perceived vs. aspirational competitors
• Sales competitiveness tiers
• Company growth stage tiers

We’ll stick with the first option for now. Whereas a direct competitor is one with which you go head-to-head for sales, an indirect competitor is one that sells a similar product to a different market or a tangential product to the same market. And whereas a perceived competitor is one that—unbeknownst to prospects—offers something completely different from you, an aspirational competitor is one that you admire for the work they’re doing in a related field.

Once you’ve categorized your competitors, consider your immediate goals and ask yourself, “Given what we’re trying to do here, which competitors require the most attention?” The number of competitors you prioritize largely depends on the breadth of your competitive landscape.

Identifying & prioritizing types of intel

One final thing before we discuss best practices for gathering intel: You need to determine the specific types of intel that are required to help your stakeholders achieve their goals.

To put it plainly, the types of intel you need to help sales reps win deals are not necessarily the same types of intel you need to help product managers create differentiated roadmaps. Will there be overlap across stakeholders? Almost certainly. But whereas a sales rep may want two sentences about a specific competitor’s pricing model, a product manager may want a more general perspective on the use cases that are and are not being addressed by other players in the market. In terms of gathering intel, these two situations demand two different approaches.

It’s also important to recognize the trial-and-error component of this process; it’ll take time to get into a groove with each of your stakeholders. Hopefully, their ongoing feedback will enable you to do a better and better job of collecting the data they need. The more communicative everyone is, the more quickly you’ll get to a place where competitive intelligence is regularly making an impact across the organization.

5 best practices for gathering competitive intelligence

Now that we’ve covered all our bases, the rest of today’s guide is dedicated to exploring five best practices for gathering competitive intelligence in a successful, repeatable manner.

1. Monitor changes to your competitors’ websites

[According to the State of CI Report, 99% of CI professionals consider their competitors’ websites to be valuable sources of intel. 35% say they’re extremely valuable.]

You can make extraordinary discoveries by simply monitoring changes on your competitors’ websites. Edits to homepage copy can indicate a change in marketing strategy (e.g., doubling down on a certain audience). Edits to careers page copy can indicate a change in product strategy (e.g., looking for experts in a certain type of engineering). Edits to customer logos can indicate opportunities for your sales team (e.g., when a competitor appears to have lost a valuable account).

The examples are virtually endless. No matter which specific stakeholders and goals you’re focused on, frequenting your competitors’ websites is a time-tested tactic for gathering intel.

2. Conduct win/loss analysis

[According to the State of CI Report, 96% of CI professionals consider win/loss analysis to be a valuable source of intel. 38% say it’s extremely valuable.]

Although win/loss analysis—the process of determining why deals are won or lost—is a discipline in its own right, it’s often a gold mine of competitive intelligence. The most effective method of collecting win/loss data is interviewing customers (to find out why they bought your solution) and prospects (to find out why they didn’t buy your solution). You’ll find that these conversations naturally yield competitive insights—a customer mentions that your solution is superior in this respect, a prospect mentions that your solution is inferior in that respect, etc.

Through the aggregation and analysis of your customers’ and prospects’ feedback, you’ll be able to capitalize on some tremendously valuable intel.

3. Embrace internal knowledge

[According to the State of CI Report, 99% of CI professionals consider internal knowledge to be a valuable source of intel. 52% say it’s extremely valuable.]

This may seem counterintuitive, but it’s true: Your stakeholders themselves are amazing sources of competitive intelligence. In fact, as you read above, more than half of CI pros say internal knowledge (a.k.a. field intelligence) is an extremely valuable resource. 

Sales reps are often speaking with prospects, and marketers, customer support reps, and product managers are often speaking with customers. Across these conversations with external folks, your colleagues learn about your competitors in all kinds of useful ways—product features, pricing models, roadmap priorities, sales tactics, and so on.

Some of the best ways to gather internal knowledge include listening to calls with prospects and customers, reviewing emails and chat messages, and combing through CRM notes.

4. Find out what your competitors’ customers are saying

[According to the State of CI Report, 94% of CI professionals consider their competitors’ customers’ reviews to be valuable sources of intel. 24% say they’re extremely valuable.]

If you found yourself wondering how one might fill in the gaps between pieces of internal knowledge, look no further: By reading reviews written by your competitors’ customers, you can uncover tons of previously unknown intel.

And if your initial instinct is to head straight for the scathing reviews, make no mistake—there’s just as much to learn from your competitors’ happy customers as there is from their unhappy customers. Let’s say, for example, that nearly every single positive review for one of your competitors makes mention of a specific feature. This is a critical piece of intel; as long as you’re lacking in this area, your rival will boast a concrete point of differentiation.

5. Keep your eye on the news

[According to the State of CI Report, 96% of CI professionals consider news to be a valuable source of intel. 38% say it’s extremely valuable.]

Product launches, strategic partnerships, industry awards—there’s no shortage of occasions that may land your competitors in the news. Typically, media coverage is the result of a press release and/or other public relations tactics, but that may not always be the case. (In certain industries, media coverage is very common—whether it’s solicited or not.)

Regardless of why a competitor is in the news, it’s almost always an opportunity to gather intel. In the case of a product or feature launch, you can learn about the positioning they’re trying to establish. In the case of a partnership, you can learn about the kinds of prospects they’re trying to connect with. And in the case of an award, you can learn about the ways in which they’re trying to present themselves to prospects.

Author: Conor Bond

Source: Crayon

Data interpretation: what is it and how to get value out of it? Part 1

Data interpretation: what is it and how to get value out of it? Part 2

If you haven't read part 1 of this article yet, you can find it here!

Common Data Analysis And Interpretation Problems

The oft-repeated mantra of those who fear data advancements in the digital age is “big data equals big trouble.” While that statement is not accurate, it is safe to say that certain data interpretation problems or “pitfalls” exist and can occur when analyzing data, especially at the speed of thought. Let’s identify some of the most common data misinterpretation risks and shed some light on how they can be avoided:

1) Correlation mistaken for causation: our first misinterpretation of data refers to the tendency of data analysts to mix the cause of a phenomenon with correlation. It is the assumption that because two actions occurred together, one caused the other. This is not accurate as actions can occur together absent a cause and effect relationship.

• Digital age example: assuming that increased revenue is the result of increased social media followers… there might be a definitive correlation between the two, especially with today’s multi-channel purchasing experiences. But, that does not mean an increase in followers is the direct cause of increased revenue. There could be both a common cause or an indirect causality.
• Remedy: attempt to eliminate the variable you believe to be causing the phenomenon.

2) Confirmation bias: our second data interpretation problem occurs when you have a theory or hypothesis in mind but are intent on only discovering data patterns that provide support to it while rejecting those that do not.

• Digital age example: your boss asks you to analyze the success of a recent multi-platform social media marketing campaign. While analyzing the potential data variables from the campaign (one that you ran and believe performed well), you see that the share rate for Facebook posts was great, while the share rate for Twitter Tweets was not. Using only the Facebook posts to prove your hypothesis that the campaign was successful would be a perfect manifestation of confirmation bias.
• Remedy: as this pitfall is often based on subjective desires, one remedy would be to analyze data with a team of objective individuals. If this is not possible, another solution is to resist the urge to make a conclusion before data exploration has been completed. Remember to always try to disprove a hypothesis, not prove it.

3) Irrelevant data: the third data misinterpretation pitfall is especially important in the digital age. As large data is no longer centrally stored, and as it continues to be analyzed at the speed of thought, it is inevitable that analysts will focus on data that is irrelevant to the problem they are trying to correct.

• Digital age example: in attempting to gauge the success of an email lead generation campaign, you notice that the number of homepage views directly resulting from the campaign increased, but the number of monthly newsletter subscribers did not. Based on the number of homepage views, you decide the campaign was a success when really it generated zero leads.
• Remedy: proactively and clearly frame any data analysis variables and KPIs prior to engaging in a data review. If the metric you are using to measure the success of a lead generation campaign is newsletter subscribers, there is no need to review the number of homepage visits. Be sure to focus on the data variable that answers your question or solves your problem and not on irrelevant data.

4) Truncating an Axes: When creating a graph to start interpreting the results of your analysis it is important to keep the axes truthful and avoid generating misleading visualizations. Starting the axes in a value that doesn’t portray the actual truth about the data can lead to false conclusions. 

• Digital age example: In the image below we can see a graph from Fox News in which the Y-axes start at 34%, making it seem that the difference between 35% and 39.6% is way higher than it actually is. This could lead to a misinterpretation of the tax rate changes. 

• Remedy: Be careful with the way your data is visualized. Be respectful and realistic with axes to avoid misinterpretation of your data. 

Data storytelling: 5 best practices

Learn how to hone both your verbal and written communication skills to make your insights memorable and encourage decision-makers to revisit your research.

You’ve spent months collecting data with your insights team or research vendors, and you’ve compiled your research into a presentation that you think is going to blow your audience away. But what happens after you’ve finished presenting? Do your stakeholders act on the insights you’ve shared, or do they move on to their next meeting and quickly forget your key takeaways and recommendations?

If you want to avoid the latter, it’s important to consider how you can make the biggest possible impact while presenting and also encourage your stakeholders to revisit your research after the fact. And that requires you to hone both your verbal and written communication skills.

In other words: practice your storytelling.

Research shows that combining statistics with storytelling results in a retention rate of 65-70%. So, how do you take advantage of this fact when presenting and documenting your insights?

Below are five best practices to help you present insights through stories – and encourage your stakeholders to revisit those stories as they make business decisions.

Tailor the message to your audience

To maximize the impact of your story, you have to consider who’s hearing it.

When you’re presenting to someone in finance, try to cover how your findings can help the company save money. When you’re talking to Marketing or Sales, explain how the information can drive new leads and close more deals. When you’re talking to the product development team, explain how they can deliver a better solution.

The more you can address your audience’s concerns in the language they use and the context they understand, the bigger the impact your story will have.

Ask yourself:

1. How much does my audience already know about the subject I’m covering?
2. How much time do they have to listen to what I’m saying?
3. What are their primary concerns?
4. What type of language do they use to communicate?
5. Are there preconceptions I need to address?

If your insights are applicable to multiple groups across the organization, it’s worth thinking about how you can tweak the story for each audience. This could mean writing different sets of key takeaways and implications for different groups or altering the examples you use to better align with each audience’s interests.

Follow the structure of a story

While stories come in various shapes, sizes, tones, and genres, they all have a few things in common – one of those being a similar structure.

Think about how a movie is typically divided into three acts. Those acts follow this general structure:

1. Setup: We’re introduced to the protagonist, and they experience some kind of inciting incident (i.e., the introduction of conflict or tension) that propels the story forward.
2. Confrontation: The protagonist works to achieve a goal but encounters obstacles along the way.
3. Resolution: The protagonist reaches the height of their conflict with an antagonist and achieves some kind of outcome (whether it’s the protagonist’s desired outcome or not will depend on the type of story).

Here’s a (fictional) example of an insights-driven story that follows this structure:

1. The insights team for a beverage company shares a recorded interview with a real customer, who we’ll call Raquel. Raquel talks about how she loves getting together for backyard barbecues with friends. She says that she used to always drink beer at these barbecues but has recently decided to stop drinking.
2. Raquel goes on to say that she doesn’t really like soda because she thinks it’s too sweet, but she will often pick one up at barbecues because she wants to have a drink in her hand.
3. After playing this interview, the insights team presents findings from their latest study into young women’s non-alcoholic beverage preferences. They use Raquel’s story to emphasize trends they are seeing for canned beverages with lower sugar or sweetener contents.

By framing your data and reports in this narrative structure, you’re more likely to keep your audience interested, make your findings memorable, and emphasize how your findings relate to real customers or consumers. This is a great way to get business decision-makers to invest in and act on your insights.

Put your editor’s hat on

When you have managed or been directly involved with a research project, it can be tempting to include every fascinating detail in your presentation. However, if you throw extraneous information into your data story, you’ll quickly lose your audience. It’s important to put yourself in the mindset of your audience and ruthlessly edit your story down to its essential ingredients.

According to Cinny Little, Principal Analyst at Forrester Research, you should focus on answering the audience’s two primary questions: “What’s in it for me?” and “Why do I need to care?”

You should also keep your editor’s hat on when documenting your key recommendations or takeaways for a report. Studies show that people can only hold about four items in their conscious mind, or working memory, at any one time. If you include more than three or four recommendations, your audience will have a harder time retaining the most important information.

Find your hook

When presenting, don’t think you can start slow and build up excitement – research suggests you only have about 30 to 60 seconds to capture your audience’s attention. After that, you’ve lost them.

And getting them back won’t be easy.

That’s why you need a hook – a way to start your story that’s so engaging and compelling your audience can’t help but listen.

According to Matthew Luhn, a writer, story consultant, and speaker who has experience working with Pixar, The Simpsons, and more, a compelling hook is at least one of the following:

• Unusual
• Unexpected
• Action-filled
• Driven by conflict

When sharing your research, you could hook your audience by leading with a finding that goes against prevailing assumptions, or a specific example of a customer struggling with a problem that your product could solve. Find a hook that evokes a strong emotion so that your story will stick with listeners and drive them to make decisions.

Experiment with your story medium

If you present your research to a room (or Zoom meeting) full of stakeholders once and then move on, you’re limiting the reach, lifespan, and value of that research. At a time when so many teams have become decentralized and remote work is common, it’s more important than ever to preserve your data stories and make them accessible to your stakeholders on demand.

At the most basic level, this could mean uploading your presentation decks to an insights management platform so that your stakeholders and team members can look them up whenever they want. However, it’s also worth thinking about other mediums you can translate your stories into. For example, you might publish infographics, video clips from customer interviews, or animated data visualizations alongside your reports. Think about the supporting materials you can include to bring the story to life for anyone who wasn’t in the room for the initial presentation.

Conclusion

​​By applying the best practices above, you can take the data and reports that others often find dry (no matter how much you disagree) and turn them into compelling, engaging, and persuasive stories.

This process of developing and distributing insights stories will enable you and your team to have a more strategic impact on your company as a whole by demonstrating the potential outcomes of making decisions based on research.

Author: Madeline Jacobson

Source: Greenbook

Overcoming data challenges in the financial services sector

Powering the Future of Healthcare: Microsoft BI Takes the Lead

New modifications are important to be done in the distribution methods of healthcare all over the globe. They need to reduce their operating costs, improve the management of their human resources, update and strengthen their internal procedures, and work toward providing better care for their patients. These businesses need to exercise greater guidance if they are to continue adding value to society in light of the rapid pace at which government regulations are updated and the growing expectations of the general public. 

Introduction to Microsoft Power BI

The development of healthcare technology has made it possible to provide greater treatment for patients, and healthcare analytics is no exception to this trend. Microsoft Power BI is a robust application that has fundamentally altered the process of data interpretation and analysis in the healthcare industry. Insights that were previously unattainable for healthcare practitioners have become available as a result of advances in data collection, analysis, and sharing capabilities that operate in real-time.

You can facilitate (BI) for healthcare by putting into place an up-to-date business solution. Power BI Developers assist businesses in extracting additional worth from the data they have collected. Unlock your data and display it in a framework that makes it simple for everyone to identify patterns and outliers. Implementing a comprehensive solution for managing the revenue cycle will allow you to obtain additional information regarding your claims, customers, or suppliers.

The current Healthcare Industry is facing several Challenges

1. Huge data

A large number of medical data presents a challenge for those working in the healthcare sector. It presents an important obstacle in terms of organizing and evaluating the data, although it makes it possible for professionals in the healthcare industry to access more information than ever before. When there is such a massive quantity of data, it can be challenging to recognize patterns. For the industry to effectively control this expansion, it requires advanced data analytics solutions that can manage enormous amounts of data.

2. Errors in medical reports

The difficulty of preventing healthcare errors caused by a shortage of data is a significant problem that faces those who work in the medical field. Healthcare professionals can’t make rational decisions about a patient’s treatment if they do not have access to reliable data. This can result in errors, incorrect diagnoses, and even mortality in some cases. The absence of data may be attributable to a variety of factors, including insufficient instruction, inadequate recordkeeping, and outmoded technology.

3. Data privacy and security

It is impossible to overstate the significance of data in healthcare. The complications of healthcare data can be difficult to navigate, which can be a huge challenge. The data that pertains to healthcare is frequently disorganized, stored in silos, and difficult to access. The delivery of healthcare presents its own unique set of challenges, not the least of which is ensuring that statistics are accurate and comprehensive. Because patient information is considered to be extremely confidential and is subject to numerous stringent regulation requirements, data privacy, and security are also significant concerns in the healthcare business.

4. Employee data administration

Because there is a growing volume of employee data that needs to be saved, handled, and evaluated, it can be challenging to ensure that the data is accurate and that it is secure. If documents on the number of physicians, nurses, and other staff are not readily available, it may be difficult to prepare for emergencies, designate responsibilities, and ensure that operations are carried out effectively. For this purpose, they need a combination of specialized knowledge, cutting-edge technology, and data analytics to guarantee the precision of employee records.

How Power BI can revolutionize healthcare analytics

The field of healthcare produces a significant amount of data. There is a plethora of information that is readily accessible, ranging from electronic health records to invoicing data, that can assist medical professionals in making decisions regarding patient treatment and business operations that are more educated. Having said that, the difficulty lies in being able to evaluate and understand these facts accurately. This is where Microsoft Power BI comes in.

1. Identify patterns and trends

You may be able to recognize trends and patterns in your data with the support of Power BI. The capability of creating data models within Power BI is one of the program’s most useful features because it enables users to conduct various types of analyses on their data. You can quickly connect to multiple data sources and integrate them into a singular perspective thanks to the data connections that are built right in.

2. Study of patient comments

The ability of providers to recognize areas that could use development and implement the necessary changes to improve the experience of patients can be gained through the analysis of remarks received from patients. Obtaining information regarding customer grievances, suggestions, and assessments of satisfaction is made simple for healthcare organizations who use Power BI. By using this tool healthcare practitioners can gain a deeper understanding of the requirements of their patients and more successfully modify their services to satisfy those requirements.

3. Proper data modeling

Data modeling is the method of building a graphic representation of data and the interactions between that data to gain a better understanding of how that data can be used to support healthcare targets. Power BI comes equipped with robust data modeling capabilities, which make it possible for medical professionals to construct intricate data models that can then be put to use in advanced analytical procedures.

4. Epidemic trend breakdown

The evaluation of outbreak trends with Power BI is a vital tool that is used by health specialists to monitor the spread of illnesses and discover prospective breakouts before they develop into significant disasters. Researchers can determine potential areas and take precautionary measures to mitigate the spread of disease by evaluating data on historical pandemics as well as patterns that are occurring currently in the world. Tracking a variety of variables, such as the number of cases, dissemination pathways, and demography of those who are afflicted is a vital part of the process of analyzing the pattern of an outbreak.

5. Improved communication

Power BI is consistently updated with new features by Microsoft, which contributes to the program’s robustness and versatility. The power BI solution allows for every person involved to have access to pertinent reports, assessments, documents, folders, etc., and it is also beneficial to collaboration between physicians, surgeons, and other medical professionals. All of that is accomplished while accurately, thoroughly, and controllably representing high-level data.

Bottom Line

The field of healthcare is one in which Power BI Developers play a particularly significant part. These software developers can be of assistance to healthcare organizations by developing interfaces and reports that offer information about improvements for patients, the financial performance of the organization, and other important measures.

In general, using Power BI for healthcare statistics opens up a world of unimaginable opportunities. Healthcare organizations can remain ahead of the curve if they adopt the latest trends and advancements in the industry and embrace them.

Author: Daniel Jacob

Source: Datafloq

Solutions to help you deal with heterogeneous data sources

With enterprise data pouring in from different sources; CRM systems, web applications, databases, files, etc., streamlining data processes is a significant challenge as it requires integrating heterogeneous data streams. In such a scenario, standardizing data becomes a pre-requisite for effective and accurate data analysis. The absence of the right integration strategy will give rise to application-specific and intradepartmental data silos, which can hinder productivity and delay results.

Consolidating data from disparate structured, unstructured, and semi-structured sources can be complex. A survey conducted by Gartner revealed that one-third of respondents consider 'integrating multiple data sources' as one of the top four integration challenges.

Understanding the common issues faced during this process can help enterprises successfully counteract them. Here are three challenges generally faced by organizations when integrating heterogeneous data sources, as well as ways to resolve them:

Data extraction

Challenge: Pulling source data is the first step in the integration process. But it can be complicated and time-consuming if data sources have different formats, structures, and types. Moreover, once the data is extracted, it needs to be transformed to make it compatible with the destination system before integration.

Solution: The best way to go about this is to create a list of sources that your organization deals with regularly. Look for an integration tool that supports extraction from all these sources. Preferably, go with a tool that supports structured, unstructured, and semi-structured sources to simplify and streamline the extraction process.

Data integrity

Challenge: Data Quality is a primary concern in every data integration strategy. Poor data quality can be a compounding problem that can affect the entire integration cycle. Processing invalid or incorrect data can lead to faulty analytics, which if passed downstream, can corrupt results.

Solution: To ensure that correct and accurate data goes into the data pipeline, create a data quality management plan before starting the project. Outlining these steps guarantees that bad data is kept out of every step of the data pipeline, from development to processing.

Scalability

Challenge: Data heterogeneity leads to the inflow of data from diverse sources into a unified system, which can ultimately lead to exponential growth in data volume. To tackle this challenge, organizations need to employ a robust integration solution that has the features to handle high volume and disparity in data without compromising on performance.

Solution: Anticipating the extent of growth in enterprise data can help organizations select the right integration solution that meets their scalability and diversity requirements. Integrating one data point at a time is beneficial in this scenario. Evaluating the value of each data point with respect to the overall integration strategy can help prioritize and plan. Say that an enterprise wants to consolidate data from three different sources: Salesforce, SQL Server, and Excel files. The data within each system can be categorized into unique datasets, such as sales, customer information, and financial data. Prioritizing and integrating these datasets one at a time can help organizations gradually scale data processes.

Author: Ibrahim Surani

Source: Dataversity

The 4 steps of the big data life cycle

Simply put, from the perspective of the life cycle of big data, there are nothing more than four aspects:

1. Big data collection
2. Big data preprocessing
3. Big data storage
4. Big data analysis

All above four together constitute the core technology in the big data life cycle.

Big data collection

Big data collection is the collection of structured and unstructured massive data from various sources.

Database collection: Sqoop and ETL are popular, and traditional relational databases MySQL and Oracle still serve as data storage methods for many enterprises. Of course, for the open source Kettle and Talend itself, big data integration content is also integrated, which can realize data synchronization and integration between hdfs, hbase and mainstream Nosq databases.

Network data collection: A data collection method that uses web crawlers or website public APIs to obtain unstructured or semi-structured data from web pages and unify them into local data.

File collection: Including real-time file collection and processing technology flume, ELK-based log collection and incremental collection, etc.

Big data preprocessing

Big data preprocessing refers to a series of operations such as “cleaning, filling, smoothing, merging, normalization, consistency check” and other operations on the collected raw data before data analysis, in order to improve the data Quality lays the foundation for later analysis work. Data preprocessing mainly includes four parts

1. Data cleaning
2. Data integration
3. Data conversion
4. Data specification

Data cleaning refers to the use of cleaning tools such as ETL to deal with missing data (missing attributes of interest), noisy data (errors in the data, or data that deviates from expected values), and inconsistent data.

Data integration refers to the consolidation and storage of data from different data sources in a unified database. The storage method focuses on solving three problems: pattern matching, data redundancy, and data value conflict detection and processing.

Data conversion refers to the process of processing the inconsistencies in the extracted data. It also includes data cleaning, that is, cleaning abnormal data according to business rules to ensure the accuracy of subsequent analysis results.

Data specification refers to the operation of minimizing the amount of data to obtain a smaller data set on the basis of keeping the original appearance of the data to the maximum extent, including: data party aggregation, dimension specification, data compression, numerical specification, concept layering, etc.

Big data storage

Big data storage refers to the process of using memory to store the collected data in the form of a database in three typical routes:

New database cluster based on MPP architecture: Using Shared Nothing architecture, combined with the efficient distributed computing model of MPP architecture, through column storage, coarse-grained indexing and other big data processing technologies, the focus is on data storage methods developed for industry big data. With the characteristics of low cost, high performance, high scalability, etc., it has a wide range of applications in the field of enterprise analysis applications.

Compared with traditional databases, its PB-level data analysis capabilities based on MPP products have significant advantages. Naturally, MPP database has also become the best choice for a new generation of enterprise data warehouse.

Technology expansion and packaging based on Hadoop: Hadoop-based technology expansion and encapsulation is aimed at data and scenarios that are difficult to process with traditional relational databases (for storage and calculation of unstructured data, etc.), using Hadoop open source advantages and related features (good at handling unstructured and semi-structured data), Complex ETL processes, complex data mining and calculation models the process of deriving relevant big data technology.

With the advancement of technology, its application scenarios will gradually expand. The most typical application scenario at present is to support the Internet big data storage and analysis by expanding and encapsulating Hadoop, involving dozens of NoSQL technologies.

Big data all-in-one: This is a combination of software and hardware designed for the analysis and processing of big data. It consists of a set of integrated servers, storage devices, operating systems, database management systems, and pre-installed and optimized software for data query, processing, and analysis. It has good stability and vertical scalability.

Big data analysis and mining

From visual analysis, data mining algorithms, predictive analysis, semantic engine, data quality management, etc., the process of extracting, refining and analyzing the chaotic data.

Visual analysis: Visual analysis refers to an analysis method that clearly and effectively conveys and communicates information with the aid of graphical means. Mainly used in massive data association analysis, that is, with the help of a visual data analysis platform, the process of performing association analysis on dispersed heterogeneous data and making a complete analysis chart. It is simple, clear, intuitive and easy to accept.

Data mining algorithm: Data mining algorithms are data analysis methods that test and calculate data by creating data mining models. It is the theoretical core of big data analysis.

There are various data mining algorithms, and different algorithms show different data characteristics due to different data types and formats. But generally speaking, the process of creating a model is similar, that is, first analyze the data provided by the user, then search for specific types of patterns and trends, and use the analysis results to define the best parameters for creating a mining model, and apply these parameters In the entire data set to extract feasible patterns and detailed statistics.

Data quality management refers to the identification, measurement, monitoring, and early warning of various data quality problems that may be caused in each stage of the data life cycle (planning, acquisition, storage, sharing, maintenance, application, extinction, etc.) to improve data A series of quality management activities.

Predictive analysis: Predictive analysis is one of the most important application areas of big data analysis. It combines a variety of advanced analysis functions (special statistical analysis, predictive modeling, data mining, text analysis, entity analysis, optimization, real-time scoring, machine learning, etc.), to achieve the purpose of predicting uncertain events.

Help users analyze trends, patterns, and relationships in structured and unstructured data, and use these indicators to predict future events and provide a basis for taking measures.

Semantic Engine: Semantic engine refers to the operation of adding semantics to existing data to improve users’ Internet search experience.

Author: Sajjad Hussain

Source: Medium

Two modern-day shifts in Market Research

In an industry that is changing tremendously, traditional ways of doing things will no longer suffice. Timelines are shortening, as demands for faster and faster insights increase, and, in addition, we are seeking these insights in such a vast sea of data. The only way to address the combination of these two issues is with technology.

The human-machine relationship

One good example of this shift is the whole arena surrounding computational text analysis. Smarter, artificial intelligence (AI)-based approaches are completely changing the way we approach this task. In the past, the human-based analysis only allowed us to skim the text, use a small sample and analyze it with subjective bias. This kind of generalized approach is being replaced by a computational methodology that incorporates all the text while throwing away what the computer views as non-essential information. Sometimes, without the right program, much of the meaning can be lost. However, this machine-based approach can work with large amounts of data quickly.

When we start to dive deeper into AI-based solutions, we see that technology can shoulder much of the hard work to free up humans to do what we can do better. What the machine does really well is finding the data points that can help us tell a better, richer story. It can run algorithms and find patterns in natural language, taking care of the heavy lifting. Then the human can come in, add color and apply sensible intelligence to the data. This human-machine tension is something I predict that we’ll continue to see as we accommodate our new reality. The end goal is to make the machine as smart as possible to really leverage our own limited time in the best ways possible.

Advanced statistical analysis

Another big change taking place surrounds the statistical underpinnings we use for analysis. Traditionally we have found things out by using the humble crosstab tool. But if we truly want to understand what’s driving, for example, differences between groups, it is simply not efficient to go through crosstab after crosstab. It is much better to have the machine do it for you and reveal just the differences that matter. When you do that, though, classical statistics break down because false positives become statistically inevitable.

Bayesian statistics do not suffer this same problem when a high volume of tests are required. In short, a Bayesian approach allows researchers to test a hypothesis and see if it holds given the data, rather than the more commonly used tests for significance which test that the data is right in the face of a given hypothesis.

There are a host of other models that are changing the way we approach our daily jobs in market research. New tools, some of them based in a completely different set of underlying principles (like Bayesian statistics), are giving us new opportunities. With all these opportunities, we are challenged to work in a new set of circumstances and learn to navigate a new reality.

We can’t afford to wait any longer to change the way we are doing things. The industry and our clients’ industries are moving too quickly for us to hesitate. I encourage researchers to embrace this new paradigm so that they will have the skill advantage. Try new tools, even if you don’t understand how they work, many of them can help you do what you do (better). Doing things in new ways can lead to better, faster insights. Go for it!

Author: Geoff Lowe

Source: Greenbook Blog

Understanding Natural Language Processing Terms

This post provides a concise overview of 18 natural language processing terms, intended as an entry point for the beginner looking for some orientation on the topic.

At the intersection of computational linguistics and artificial intelligence is where we find natural language processing. Very broadly, natural language processing (NLP) is a discipline which is interested in how human languages, and, to some extent, the humans who speak them, interact with technology. NLP is an interdisciplinary topic which has historically been the equal domain of artificial intelligence researchers and linguistics alike; perhaps obviously, those approaching the discipline from the linguistics side must get up to speed on technology, while those entering the discipline from the technology realm need to learn the linguistic concepts.

It is this second group that this post aims to serve at an introductory level, as we take a no-nonsense approach to defining some key NLP terminology. While you certainly won't be a linguistic expert after reading this, we hope that you are better able to understand some of the NLP-related discourse, and gain perspective as to how to proceed with learning more on the topics herein.

So here they are, 18 select natural language processing terms, concisely defined.

1. Natural Language Processing (NLP)

Natural language processing (NLP) concerns itself with the interaction between natural human languages and computing devices. NLP is a major aspect of computational linguistics, and also falls within the realms of computer science and artificial intelligence.

2. Tokenization

Tokenization is, generally, an early step in the NLP process, a step which splits longer strings of text into smaller pieces, or tokens. Larger chunks of text can be tokenized into sentences, sentences can be tokenized into words, etc. Further processing is generally performed after a piece of text has been appropriately tokenized.

3. Normalization

Before further processing, text needs to be normalized. Normalization generally refers to a series of related tasks meant to put all text on a level playing field: converting all text to the same case (upper or lower), removing punctuation, expanding contractions, converting numbers to their word equivalents, and so on. Normalization puts all words on equal footing, and allows processing to proceed uniformly.

4. Stemming

Stemming is the process of eliminating affixes (suffixed, prefixes, infixes, circumfixes) from a word in order to obtain a word stem.

5. Lemmatization

Lemmatization is related to stemming, differing in that lemmatization is able to capture canonical forms based on a word's lemma.

For example, stemming the word "better" would fail to return its citation form (another word for lemma); however, lemmatization would result in the following: better → good

It should be easy to see why the implementation of a stemmer would be the less difficult feat of the two.

6. Corpus

In linguistics and NLP, corpus (literally Latin for body) refers to a collection of texts. Such collections may be formed of a single language of texts, or can span multiple languages -- there are numerous reasons for which multilingual corpora (the plural of corpus) may be useful. Corpora may also consist of themed texts (historical, Biblical, etc.). Corpora are generally solely used for statistical linguistic analysis and hypothesis testing.

7. Stop Words

Stop words are those words which are filtered out before further processing of text, since these words contribute little to overall meaning, given that they are generally the most common words in a language. For instance, "the," "and," and "a," while all required words in a particular passage, don't generally contribute greatly to one's understanding of content. As a simple example, the following panagram is just as legible if the stop words are removed: The quick brown fox jumps over the lazy dog.

8. Parts-of-speech (POS) Tagging

 
POS tagging consists of assigning a category tag to the tokenized parts of a sentence. The most popular POS tagging would be identifying words as nouns, verbs, adjectives, etc.

9. Statistical Language Modeling

Statistical Language Modeling is the process of building a statistical language model which is meant to provide an estimate of a natural language. For a sequence of input words, the model would assign a probability to the entire sequence, which contributes to the estimated likelihood of various possible sequences. This can be especially useful for NLP applications which generate text.

10. Bag of Words

Bag of words is a particular representation model used to simplify the contents of a selection of text. The bag of words model omits grammar and word order, but is interested in the number of occurrences of words within the text. The ultimate representation of the text selection is that of a bag of words (bag referring to the set theory concept of multisets, which differ from simple sets).

Actual storage mechanisms for the bag of words representation can vary, but the following is a simple example using a dictionary for intuitiveness. Sample text:

The resulting bag of words representation as a dictionary:

   {
      'well': 3,
      'said': 2,
      'john': 1,
      'there': 4,
      'james': 1
   }

11. n-grams

n-grams is another representation model for simplifying text selection contents. As opposed to the orderless representation of bag of words, n-grams modeling is interested in preserving contiguous sequences of N items from the text selection.

An example of trigram (3-gram) model of the second sentence of the above example ("There, there," said James. "There, there.") appears as a list representation below:

   [
      "there there said",
      "there said james",
      "said james there",
      "james there there",
   ]

12. Regular Expressions

Regular expressions, often abbreviated regexp or regexp, are a tried and true method of concisely describing patterns of text. A regular expression is represented as a special text string itself, and is meant for developing search patterns on selections of text. Regular expressions can be thought of as an expanded set of rules beyond the wildcard characters of ? and *. Though often cited as frustrating to learn, regular expressions are incredibly powerful text searching tools.

13. Zipf's Law

Zipf's Law is used to describe the relationship between word frequencies in document collections. If a document collection's words are ordered by frequency, and y is used to describe the number of times that the xth word appears, Zipf's observation is concisely captured as y = cx^-1/2 (item frequency is inversely proportional to item rank). More generally, Wikipedia says:

''Zipf's law states that given some corpus of natural language utterances, the frequency of any word is inversely proportional to its rank in the frequency table. Thus the most frequent word will occur approximately twice as often as the second most frequent word, three times as often as the third most frequent word, etc.''

 14. Similarity Measures

There are numerous similarity measures which can be applied to NLP. What are we measuring the similarity of? Generally, strings.

• Levenshtein - the number of characters that must be deleted, inserted, or substituted in order to make a pair of strings equal
• Jaccard - the measure of overlap between 2 sets; in the case of NLP, generally, documents are sets of words
• Smith Waterman - similar to Levenshtein, but with costs assigned to substitution, insertion, and deletion

15. Syntactic Analysis

Also referred to as parsing, syntactic analysis is the task of analyzing strings as symbols, and ensuring their conformance to a established set of grammatical rules. This step must, out of necessity, come before any further analysis which attempts to extract insight from text -- semantic, sentiment, etc. -- treating it as something beyond symbols.

16. Semantic Analysis

Also known as meaning generation, semantic analysis is interested in determining the meaning of text selections (either character or word sequences). After an input selection of text is read and parsed (analyzed syntactically), the text selection can then be interpreted for meaning. Simply put, syntactic analysis is concerned with what words a text selection was made up of, while semantic analysis wants to know what the collection of words actually means. The topic of semantic analysis is both broad and deep, with a wide variety of tools and techniques at the researcher's disposal.

17. Sentiment Analysis

Sentiment analysis is the process of evaluating and determining the sentiment captured in a selection of text, with sentiment defined as feeling or emotion. This sentiment can be simply positive (happy), negative (sad or angry), or neutral, or can be some more precise measurement along a scale, with neutral in the middle, and positive and negative increasing in either direction.

18. Information Retrieval

Information retrieval is the process of accessing and retrieving the most appropriate information from text based on a particular query, using context-based indexing or metadata. One of the most famous examples of information retrieval would be Google Search.

Author: Matthew Mayo

Source: KDnuggets

Using Hierarchical Clustering in data analysis

This article discusses the analytical method of Hierarchical Clustering and how it can be used within an organization for analytical purposes.

What is Hierarchical Clustering?

Hierarchical Clustering is a process by which objects are classified into a number of groups so that they are as much dissimilar as possible from one group to another group and as much similar as possible within each group.

For example, if you want to create four groups of items, these items  should be as similar as possible in terms of attributes of the items in each group, and items in group 1 and group 2 should be as dissimilar as possible. All items start in one cluster, and are then divided into two clusters. The data points within one cluster are as similar as possible, and the data points in other clusters are dissimilar from the other clusters being analyzed. For each cluster, we repeat the process until the specified number of clusters is reached (four in this case).

This type of analysis can be applied to segment customers by purchase history, segment users by the types of activities they perform on websites or applications, to develop personalized consumer profiles based on activities or interests, and to recognize market segments, etc.

How does an organization use Hierarchical Clustering to analyze data?

In order to understand the application of Hierarchical Clustering for organizational analysis, let us consider two use cases.

Use case one

Business problem: A bank wants to group loan applicants into high/medium/low risk based on attributes such as loan amount, monthly installments, employment tenure, the number of times the applicant has been delinquent in other payments, annual income, debt to income ratio etc.

Business benefit: Once the segments are identified, the bank will have a loan applicants’ dataset with each applicant labeled as high/medium/low risk. Based on these labels, the bank can easily make a decision on whether to give loan to an applicant and how much credit to extend, as well as the interest rate the applicant will be given, based on the amount of risk involved.

Use case two

Business problem: The enterprise wishes to organize customers into groups/segments based on similar traits, product preferences and expectations. Segments are constructed based on customer demographic characteristics, psychographics, past behavior and product use behavior.

Business benefit: Once the segments are identified, marketing messages and products can be customized for each segment. The better the segment(s) chosen for targeting by a particular organization, the more successful the business will be in the market.

Hierarchical Clustering can help an enterprise organize data into groups to identify similarities and, equally important, dissimilar groups and characteristics, so that the business can target pricing, products, services, marketing messages and more.

Author: Kartik Patel

Source: Dataversity