2 items tagged "data automation"

  • Be careful when implementing data warehouse automation

    DWHAAutomation can be a huge help, but automating concepts before you understand them is a recipe for disaster.

    The concept of devops has taken root in the world of business intelligence and analytics.

    The overall concept of devops has been around for a while in traditional IT departments as they sought to expand and refine the way that they implemented software and applications. The core of devops in the world of analytics is called DWA (data warehouse automation), which links together the design and implementation of analytical environments into repeatable processes and should lead to increased data warehouse and data mart quality, as well as decreased time to implement those environments.

    Unfortunately, for several reasons the concept of data warehouse automation is not a silver bullet when it comes to the implementation of analytical environments.

    One reason is that you really shouldn't automate concepts before you fully understand them. As the saying goes, don't put your problems on roller skates. Automating a broken process only means that you make mistakes faster. Now, while I often advocate the concept of failing faster to find the best solution to an analytical problem, I don't really agree with the concept of provisioning flawed database structures very quickly only to rebuild them later.

    Another issue with applying devops to analytical practices is that the software development community has a 10-15 year head start on the analytical community when it comes to productizing elements of their craft.

    oftware developers have spent years learning how to best encapsulate their designs into object-oriented design, package that knowledge, and put it in libraries for use by other parts of the organization, or even by other organizations. Unfortunately, the design, architecture, and implementation of analytical components, such as data models, dashboard design, and database administration, are viewed as an art and still experience cultural resistance to the concept that a process can repeat the artistry of a data model or a dashboard design.

    Finally, there is the myth that data warehouse automation or any devops practice can replace the true thought processes that go into the design of an analytical environment.

    With the right processes and cultural buy-in, DWA will provide an organization with the ability to leverage their technical teams and improve the implementation time of changes in analytical environments. However, without that level of discipline to standardize the right components and embrace artistry on the tricky bits, organizations will take the concept of data warehouse automation and fail miserably in their efforts to automate.

    The following is good advice for any DWA practice:

    • Use the right design process and engage the analytical implementation teams. Without this level of forethought and cultural buy-in, the process becomes more of an issue than it does a benefit and actually takes longer to implement than a traditional approach.
    • Find the right technologies to use. There are DWA platforms available to use, but there are also toolsets such as scripting and development environments that can provide much of the implementation value of a data warehouse automation solution. The right environment for your team's skills and budget will go a long way to either validating a DWA practice or showing its limitations.
    • Iterate and improve. Just as DWA is designed to iterate the development of analytical environments, data warehouse automation practices should have the same level of iteration. Start small. Perfect the implementation. Expand the scope. Repeat.

    Source: Infoworld

  • Data warehouse automation: what you need to know

    data warehouseIn the dark about data warehousing? You’re not alone

    You would be forgiven for not knowing data warehousing exists, let alone that it’s been automated. It’s not a topic that gets a lot of coverage in the UK, unlike in the USA and Europe. It might be that Business Intelligence and Big Data Analytics are topics that have more ‘curb’ appeal. But, without data warehousing, data analytics would not generate the quality of business intelligence that organisations rely on. So what is a data warehouse and why did it need to be automated?

    Here’s what you need to know about data warehouse automation.

    In its most basic form a data warehouse is a repository where all your data is put, so that it can be analysed for business insight, and most business have one. Your customers will most likely have one because they need the kind of insight data analysis provides. Business Insight or Intelligence (BI) helps the business make accurate decisions, stay competitive and ultimately profitable.

    In retail, for example, the accurate and timely reporting of sales, inventory, discounts and profit is critical to getting a consolidated view of the business at all levels and at all locations. In addition, analysing customer data can inform businesses which promotions work, which products sell, which locations work best, what loyalty vouchers and schemes are working, and which are not. Knowing customer demographics can help retailers to cross or upsell items. By analysing customer data companies can tailor products to the right specification, at the right time thereby improving customer relations and ultimately increasing customer retention.

    Analysing all the data

    But, this is only part of the picture. The best intelligence will come from an analysis of all the data the company has. There are several places where companies get data. They usually have their own internal systems that have finance data, HR data, sales data, and other data specific to its business. In addition, most of your customers will now also collect data from the internet and social media (Big Data), with new data coming in from sensors, GPS and smart devices (IoT data). The data warehouse can pull any kind of data from any source into one single place for analysis. A lack of cross-pollination across the business can lead to missed opportunities and a limited corporate view.

    Previously, to get the data from its source (internal or external) into the data warehouse involved writing code by hand. This was monotonous, slow and laborious. It meant that the data warehouse took months to build, and then was rigidly stuck to the coding (and therefore design) it had been built with. Any changes that needed to be made, were equally slow and time consuming creating a frustration for both the IT and the Business. For the business, the data often took so long to be produced that it was out of date by the time they had it.


    Things have moved on since the days of the traditional data warehouse and now the design and build of a data warehouse is automated, optimised and wizard driven. It means that the coding is generated automatically. With automation, data is available at the push of a button. Your customers don’t have to be an IT expert to create reports and employees don’t need to ask head office if they want information on a particular product line. Even more importantly, when you automate the data warehouse lifecycle you make it agile, so as your business grows and changes the warehouse can adapt. As we all know, it’s a false economy to invest in a short-term solution, which in a few years, will not be fit for purpose. Equally, it’s no good paying for excellent business intelligence tools and fancy reporting dashboards if the data underneath is not fully accessible, accurate and flexible.

    What does this mean for the channel?

    So now you know the importance of a data warehouse for data analytics, and how automation has brought data warehousing into the 21st century. So, what next? What does this mean for the channel?

    Not everyone in the channel will be interested in automation. Faster more efficient projects might not look like they will generate the immediate profit margins or revenue of a longer, slower one. But, innovative channel partners will be able to see that there are two clear advantages for them. One is that the projects, whilst shorter, never really end. This means there is a consistent stream of income. Secondly, by knowing about and offering your clients data warehouse automation the channel partner shows their expertise and consultancy abilities.

    The simple fact is that most companies have a data warehouse of some kind, from the giant supermarkets such as Tesco and Sainsbury, to smaller businesses like David Lloyd or Jersey Electricity. You don’t want to be the channel partner who didn’t know about or didn’t recommend the best, most efficient solution for your client. This could impact more than just the immediate sales. By educating your customers about the benefits of data warehouse automation you will bring them a wealth of efficiencies to their company, and most likely a wealth of future recommendations to yours.

    Source: ChannelPro

EasyTagCloud v2.8