Approaching the Transformation phase in ELT processes

Much has been written about the shift from ETL to ELT and how ELT enables superior speed and agility for modern analytics. One important move to support this speed and agility is creating a workflow that enables data transformation to be exploratory and iterative. Defining an analysis requires an iterative loop of forming and testing these hypotheses via data transformation. Reducing the latency of that interactive loop is crucial to reducing the overall time it takes to build data pipelines.

ELT achieves flexibility by enabling access to raw data instead of predefined subsets or aggregates. The process achieves speed by leveraging the processing power of Cloud Data Warehouses and Data Lakes. A simple, dominant pattern is emerging: move all of your data to cost effective cloud storage and leverage cloud compute for transformation of data prior to analysis.

What this takes:

1. Extract your data from source systems
2. Load your data into the cloud platform
3. Transform your data in the cloud!

There are a few different approaches to doing the transformation work as part of the ELT process.

Code only solutions

Individuals and teams proficient in languages such as SQL or Python can write transformations that run directly against the cloud data warehouse or data lake. Tools such as DBT and Dataform provide infrastructure around code to help teams build more maintainable pipelines. Code gives its authors ultimate flexibility to build anything the underlying system supports. Additionally, there are large communities of Python and SQL developers as well as a wealth of examples, best practices, and forums to learn from. Of course, there are many individuals in organizations that do not know how to write code or simply prefer not to but still need to efficiently transform data.

Visual only solutions

Individuals and teams that prefer visual transformation can leverage visual tools to build their pipelines. To gain the benefits of ELT, visual tools increasingly execute these pipelines directly  in Cloud Data Warehouses and Lakes instead of in proprietary run times. Visual solutions appeal to a large community of data pipeline developers who need to produce data assets but don’t necessarily want to or know how to code. These solutions often provide more automated approaches to build pipelines, increasing efficiency for many use cases. However, visual only approaches can at times be not as flexible as coding in the underlying system: certain use cases are not performant enough or simply not possible in the tool.

Visual + code solutions

We believe increasingly that modern data transformation tools will support both approaches to enable a broader community to collaborate and scale how this work is done in organizations. Use a visual approach where it makes sense but also enable users to leverage code where it makes sense. This best of both worlds approach has two major benefits:

1. Increased efficiency for individuals: While some have strong preferences for doing their work in code or in a visual tool, we find that many people just want to get the job done as efficiently and effectively as possible. Each approach has advantages and disadvantages – providing flexibility allows an individual to choose the right tool for the job.
2. Collaboration across the organization: Organizations have some users who prefer to code and some prefer not to. Solutions providing support for both have the potential to enable collaboration across users with varied skill sets and use cases.

Approach at Trifacta
Moving forward, Trifacta is increasingly investing in the two areas to enable collaboration across teams and individuals working in both code + user interfaces:

1. Running code within Trifacta: Most of Trifacta's customers primarily leverage it's visual interface to build transformations, but many of them also use the existing functionality for running SQL queries from directly within Trifacta, building pipelines with SQL and/or our internal DSL. Soon, Trifacta plans to support other languages such as Python.
2. Generating code with Trifacta: The pipelines customers build in Trifacta are built on top of our internal transformation language. This language is then translated into code that can run across a variety of different platforms. Today Spark, Google Dataflow, Photon (Trifacta's engine for in-browser computation) are supported and common transformations pushed down into databases and cloud data warehouses like Snowflake, Big Query and Redshift. To date, this code is ran on behalf of the customers, but Trifacta received many requests to take the code that is generated and use that completely outside of Trifacta.  

Author: Sean Kandel

Source: Trifacta

Helping your employees deal with change by communicating it the right way

Last year, 70% of employees we surveyed indicated that they faced an increasing amount of change over the prior year. In addition, Communications executives identified employee change fatigue as the most pressing challenge they faced. Of course, 2020 was a year unlike any other, but before attributing these figures to the dramatic events of that year, consider this: from 2017 to 2019, employee change fatigue had already topped of the list of communications leaders’ challenges. Clearly, this is not a new phenomenon, and does not appear to be going away any time soon.

VUCA Environment

More and more leaders are using the term “VUCA” (volatile, uncertain, complex, and ambiguous) to describe the current environment that employees are facing. VUCA is an acronym coined by the US army in the 1980s, but over the past several years, it has migrated into business lexicon to describe a constantly ongoing state of change. For communicators, it can be instructive to frame the communications strategy that will effectively engage workers in this brave new world.

Given this context then, how can communications leaders engage employees to help them navigate this ever-changing world? First, they need to reconsider the nature of the changes that employees are facing. But they must also acknowledge the limitations of the traditional approach to change communications, which prioritizes large-scale, organizational transformations like a merger or acquisition, the launch of a new corporate strategy, or introduction of a new CEO.

While all of these events still necessitate a communications response, they represent a small fraction – just 4% – of the total number of changes that employees face in a given year. This leaves a gaping hole in the way that leaders help employees through the other 96% of changes – everything from rolling out new technology in the workplace to getting a new manager – that may be individually smaller in scope but collectively have the potential to be quite disruptive.

Communications’ response: adopting an always-on strategy

When we studied how communications leaders were most successfully responding to this environment, we found that they were fundamentally shifting how their employees think about changes; not as one-off events, but by talking about change as an ecosystem. In short, they were adopting an ‘always-on change strategy,’ where they deliver regular, overarching messages that are not specific to any one organizational change, but rather discuss how the organization operates in a consistently changing state.

Broadly speaking, an always-on change strategy includes two components:

• Providing employees with regular information about how decisions have been made and the implications of a VUCA environment on the organization
• Providing employees with access to self-serve networks and resources that they can use for just-in time support and to build their resiliency

Adopting an always-on strategy dedicated to supporting employees in an environment of ongoing change has several benefits when compared to the traditional, one-off approach:

• First, the regular information about VUCA helps reset employee expectations of stability. Information that is initially shared with employees to help them understand change often becomes outdated as business conditions change. As such, the always-on strategy can help employees become better primed to expect that their organization is in a consistent period of change.
• Second, visibility into how change decisions are made helps employees follow the organization’s change journey and prompts them to seek out opportunities to consider how they can contribute to that journey.
• Third, access to self-serve networks and resources helps employees find long-term success in a VUCA environment, which ultimately builds resiliency.

One final advantage of adopting an ‘always-on change strategy’ is that it helps to mitigate the need for communications leaders to respond to every single change that occurs across the organization. On the contrary, an effective always-on communications strategy has the potential to be self-sustaining, as it provides employees the ability to access self-serve networks and resources in real-time, drawing upon peer-to-peer engagement to help each other navigate the ongoing VUCA environment.

Author: Emmett Fitzpatrick

Source: Gartner