Organizations today don’t use previous generation architectures to store their big data. Why would they use previous-generation BI tools for big data analysis? When looking at BI tools for your organization, there are 10 “Commandments” you should live by.

First Commandment: Thou Shalt Not Move Big Data
Moving Big Data is expensive: it is big, after all, so physics is against you if you need to load it up and move it. Avoid extracting data out into data marts and cubes, because “extract” means moving, and creates big-data-sized problems in maintenance, network performance additional CPU — on two copies that are logically the same. Pushing BI down to the lower layers to run at the data is what motivated Big Data in the first place.

Second Commandment: Thou Shalt Not Steal!...Or Violate Corporate Security Policy
Security’s not optional. The sadly regular drumbeat of data breaches shows it’s not easy, either. Look for BI tools that can leverage the security model that’s already in place. Big Data can make this easier, with unified security systems like Ranger, Sentry and Knox; even Mongo has an amazing security architecture now. All these models allow you to plug right in, propagate user information all the way up to the application layer, and enforce a visualization’s authorization and the data lineage associated with it along the way. Security as a service: use it.

Third Commandment: Thou Shalt Not Pay for Each User, Nor Every Gigabyte
One of the fundamental beauties of Big Data is that when done right, it can be extremely cost effective. Putting five petabytes of data into Oracle could break the bank; but you can do just that in a big data system. That said, there are certain price traps you should watch out for before you buy. Some BI applications charge users by the gigabyte, or by gigabyte indexed. Caveat emptor! It’s totally common to have geometric, exponential, logarithmic growth in data and in adoption with big data. Our customers have seen deployments grow from tens of billions of entries to hundreds of billions in a matter of months, with a user base up by 50x. That’s another beauty of big data systems: Incremental scalability. Make sure you don’t get lowballed into a BI tool that penalizes your upside.

Fourth Commandment: Thou Shalt Covet Thy Neighbor’s VisualizationsSharing static charts and graphs? We’ve all done it: Publishing PDFs, exporting to PNGs, email attachments, etc. But with big data and BI, static won’t cut it: All you have is pretty pictures. You should be able let anyone you want interact with your data. Think of visualizations as interactive roadmaps for navigating data; why should only one person take the journey? Publishing interactive visualizations is only the first step. Look ahead to the Github model. Rather than “Here’s your final published product,” get “Here is a Viz, make a clone, fork it, and this is how I derived at those insights, and see what other problem domains it applies to.” It lets others learn from your insights.

Fifth Commandment: Thou Shalt Analyze Thy Data In Its Natural Form
Too often, I hear people referring to big data as “unstructured.” It’s far more. Finance and sensors generate tons of key value pairs. JSON — probably the trendiest data format of all — can be semi-structured, multi-structured, etc. MongoDB has made a huge bet on making sure data should stay in this format: Beyond its virtues for performance and scalability reasons, expressiveness gets lost when you convert it into the rows and tables. And lots of big data is still created in tables, often with thousands of columns. And you’re going to have to do relational joins over all of it: “Select this from there when that...” Flattening can destroy critical relationships expressed in the original structure. Stay away from BI solutions that tell you “please transform your data into a pretty table because that’s the way we’ve always done it.”

Sixth Commandment: Thou Shalt Not Wait Endlessly For Thine ResultsIn 2016 we expect things to be fast. One classic approach is OLAP cubes, essentially moving the data into a pre-computed cache, to get good performance. The problem is you have to extract and move data to build the cube before you get performance (see Commandment #1). Now, this can work pretty well at a certain scale... until the temp table becomes gigantic and crashes your laptop by trying to materialize it locally. New data will stop analysis in its tracks while you extract that data to rebuild the cache. Be wary of sampling too, you may end up building a visualization that looks great and performs well before you realize it’s all wrong because you didn’t have the whole picture. Instead, look for BI tools that make it easy to continuously change which data you are looking at.

Seventh Commandment: Thou Shalt Not Build Reports, But Apps Instead
For too long, ‘getting the data’ meant getting a report. In big data, BI users want asynchronous data from multiple sources so they don’t need to refresh anything — just like anything else that runs in browsers and on mobile devices. Users want to interact with the visual elements to get the answers they’re looking for, not just cross-filtering the results you already gave them. Frameworks like Rails made it easier to build Web applications. Why not do the same with BI apps? No good reason not to take a similar approach to these apps, APIs, templates, reusability, and so on. It’s time to look at BI through the lens of modern web application development.

Eighth Commandment: Thou Shalt Use Intelligent ToolsBI tools have proven themselves when it comes to recommending visualizations based on data. Now it’s time to do the same for automatic maintenance of models and caching, so your end user doesn’t have to worry about it. At big data scale, it’s almost impossible to live without it, there’s a wealth of information that can be gleaned from how users interact with the data and visuals, which modern tools should use to leverage the data network effects . Also, look for tools that have search built in for everything, because I’ve seen customers who literally have thousands of visualizations they’ve built out. You need a way to quickly look for results, and with the web we’ve been trained to search instead of digging through menus.

Ninth Commandment: Thou Shalt Go Beyond The Basics
Today’s big data systems are known for predictive analytical horsepower. Correlation, forecasting, and more, all make advanced analytics more accessible than ever to business users. Delivering visualizations that can crank through big data without requiring programming experience empowers analysts and gets beyond a simple fixation on ‘up and to the right.’ To realize its true potential, big data shouldn’t have to rely on everyone becoming an R programmer. Humans are quite good at dealing with visual information; we just have to work harder to deliver it to them that way.

Tenth Commandment: Thou Shalt Not Just Stand There On the Shore of the Data Lake Waiting for a Data Scientist To Do the WorkWhether you approach Big Data as a data lake or an enterprise data hub, Hadoop has changed the speed and cost of data and we’re all helping to create more of it every day. But when it comes to actually using big data for business users, it is too often a write-only system: Data created by the many is only used by the few.

Business users have a ton of questions that can be answered with data in Hadoop. Business Intelligence is about building applications that deliver that data visually, in the context of day-to-day decision making. The bottom line is that everyone in an organization wants to make data-driven decisions. It would be a terrible shame to limit all the questions that big data can answer to those that need a data scientist to tackle them.

 Source: Datanami

The questions that help you find out why market research reports are often not that helpful

It’s always easy to blame someone else.  It’s far more difficult to do an honest self-evaluation and see the extent to which you may have contributed to a problem. On November 5, in a GreenBook Blog post by Mike Sherman and Neil Gains, Do Market Research Agencies Produce Poor Quality Reports?, the authors present clear evidence that research vendors often think more highly of their reports than end clients do. Reports are too long, lack practical answers and recommendations, and don’t address business problems.

In multiple decades in the insights business (both as a client and a vendor), I’ve produced hundreds of reports and seen many, many more. Mike and Neil are entirely right: research reports too often just aren’t helpful to the end client. They present data but not actual insights.

But there are two important questions:  

1. Why is this?  
2. What are the solutions?

Whether you’re the writer or the recipient of reports, I’m going to ask you a really critical third question:  How much of this is your own fault?  

Questions for the report writer

Let’s start with the folks actually producing the research reports.

1. How did you start the project?

When you began the project, did you start with the methodology, timeline, and budget? Or was your first question, 'What is the business problem you’re trying to solve?'

Let’s be blunt: that’s why you exist. If you’re not helping clients solve business problems, you have no reason to be in business.  

2. Are you by nature curious?

If not, then you have a serious obstacle to becoming an effective report writer. Hmmm… the data shows long-term customers are less satisfied than newer customers with customer service. Many report writers stop after pointing this out, thinking their job is done. But aren’t you curious why this is? Certainly, your client will be curious! What else in the data would provide some insight into the issue? Just reporting the facts without attempting to understand and present the meaning behind them is inexcusable for a report writer.

3. Is your report produced for your company or your client?

Is your reporting customized to each client’s needs and company culture? Or do you simply use your standard template and hand over the results?  

Every client is different. I’ve had clients who want just a one-page summary, and others who want to wring every last bit of data from the study, so there’s no such thing as a report that’s 'too long'. Some value visuals; others want a narrative. This shouldn’t just be the preference of the individual client, but the client’s knowledge of how the information will be most useful within the organization. If you haven’t discussed this, you’re missing out on a chance to produce something that will actually be used, making you more valuable to your client.

4. Is there a broken link in the communication chain?

Especially in large research agencies, the account manager and the actual report writer may be entirely different people. The account manager gets to know the client; the report writer looks at a bunch of data in a vacuum and just writes based on what the data says. It’s no different than a caterer learning about the client’s food allergies and preferences, but the chef just cooking whatever dishes can be made from whatever’s in the fridge. Are you fully involving the report writer in client communications? If not, the person who most needs the knowledge isn’t receiving it.

5. Do you have clients or partners?

My company has multiple clients we’ve been serving since the 90s. We seek clients with whom we become partners; solving problems together.

If you as a vendor are nickel-and-diming clients, complaining when the situation requires last-minute changes, being unresponsive when they have needs, pushing them toward more profitable 'solutions', and basically treating the client like a bother or a paycheck, you’re not building partnerships. If you’re not in a true partnership with your clients, how can you expect them to provide you with all the background, detail, and insight you require to address their needs effectively rather than just providing the data?

Questions for the Client

Okay, you’re not happy with the reports you’re getting. I hear it all the time: It’s hard to find good market research vendors! But maybe it’s time to do a challenging self-evaluation: Are you as the client creating some (or maybe even much) of the problem?

1. What drives your vendor decisions?

In any purchase, your decision factors determine what you get. If your choices are driven by cost, you’ll be sure to get the lowest-cost vendor.  But that may not be the vendor who can best understand and help solve your business problem.

Sometimes it’s not even cost. We pitched a client and they gave us an RFP with little detail beyond their business problem. We designed a proposal to address that problem. Their response was both exhilarating and depressing. Exhilarating: 'This is fantastic! You captured exactly what we were looking for, far more than any other vendor. You obviously understand our business problem, and the pricing was great!' Depressing: 'We’re giving the project to someone else because you’re not in our vendor system.  The report is due in six weeks and it’ll take longer than that just to get you registered'.  

Whatever drives your vendor choice will be the determining factor in the product you get.

2. Are you choosing vendors or creating partners?

The more we treat each other as partners, the more I can know you and your organization and the more helpful I can be.

But the more you try to whittle down my pricing, fail to respond to my questions, take forever to pay invoices, miss deadlines (and then expect me to work overtime to stay on track), refuse to consider my suggestions, and otherwise treat me as 'just a vendor', the less I’ll ever be your partner. I’ve had clients who regularly joked about how much they abuse their vendors. Tell me how that helps me perform at my highest level for you?

A partnership also involves two-way communication. When you start with: 'We want to do six online focus groups among our customers to test new logo options', you limit me to being an order-taker. When you start with: 'We need to evaluate some new logo options', you allow my experience to collaborate with your knowledge to make the project better.

3. Are you giving your report writers what they need to succeed?

I can’t tell you how many times clients start a research conversation by discussing the methodology, timeline, or budget. Or get offended when I offer alternatives or different ways of thinking about the project. Or become frustrated when I ask deeper questions about their business needs. Or how many times clients can’t even express a clear business need.

We were bidding a project for a large bank. I asked questions about how they planned to use the data, what existing customer and transaction data we could pair with the survey data, etc. The client became quite exasperated and demanded, 'Why are you asking all these questions? The other vendors didn’t have these questions!' Right then, I knew we had no chance at the project, and that (other than financial implications) I didn’t actually want it.

If you expect me to help solve your business problem, you need to share the background, information, and detail to allow me to do that. You can’t withhold information and then blame the vendor when the report isn’t helpful.

4. Do you really know what you want?

I’ve given a short, succinct summary to a client and been told:  'We paid a lot of money for this study, where’s all the detail?' The next time I provided a detailed report to the same client and was told:  'No one is going to read all this, just give me a three-page summary!'

So how do I provide you what you want when you don’t know what you want?

It’s a two-way street

From the research Mike and Neil did, it’s obvious that many report writers feel all is good while their clients think otherwise. Unfortunately, it also appears that many clients simply blame their vendors for less-than-helpful reports, while report writers shift blame to uncooperative clients.  

Until both sides take a good, hard look at the possibility of their own contributions to the problem, it isn’t likely to be solved any time soon.

Author: Ron Sellers

Source: GreenBook Blog