Data Lakes necessary for advanced market intelligence

When The Weather Company wanted to up its game in the forecasting world, executives knew the answer was to analyze even more data. However, the company's data warehouse was too constricting; it accepted only structured data and required a

data lakess long as six months to develop appropriate schemas.

"Our goal was to inject data into our businesses as fast as possible to be able to see new opportunities," says Bryson Koehler, executive vice president, CTO and CIO of The Weather Company. "It's not realistic for a business to go dark on a project for any extended period of time just to clean up data. So much changes on a daily basis -- so many new sources of data -- that that journey would never be complete."

Koehler wanted to bring in data from anywhere it originated, including personal weather stations and Internet of Things sensors, to enrich analysis. With traditional data warehouses, this would have been near impossible because of the unstructured nature of the new data, the volume, and the lengthy development time necessary to process and validate it.

"We get data from a lot of startups, and I can't ask these companies to create a specialized format for us," Koehler says. "They would go somewhere else that would take it [as is],and that would take away a competitive advantage."

To ward off that potential, two years ago The Weather Company became an early adopter of data lakes. This approach allows enterprises to ingest, analyze and store unstructured, semi-structured and structured data in an agnostic manner, providing a more flexible repository than traditional data warehouses.

Author: Sandra Gittlen

Source: Computerworld