Data work

Data work is the set of activities involved in collecting, preparing, managing, analysing and communicating data so that it can inform decisions or power processes. It covers the full lifecycle, from capturing or sourcing data, through integration, cleaning, modelling, storage and governance, to analysis, reporting and turning insight into action.

Sorry about that content. That was Claude. Claude is Datatype employee [1]. There is someone named Nick. He’s employee [0]. And now there’s this new person named Paul. Paul is employee [2]. So, Claude knows a lot of stuff, but he’s kinda boring at this point. We’ll teach Claude some skills to make it more interesting.

When we say ‘do data’, we’re talking about all that work that needs to be done to make data useful. We call this data work. We emphasize this because it really does take work. This ship* is hard. It really is.

*note: to avoid using any sort of profanity in this knowledge base, we will be substituting any inappropriate terms with suitably unoffensive language.

A bit more about data work

At the core of data work is the deceptively simple process of extracting, transforming and loading data. Often referred to as ETL. The fundamental pattern of ETL hasn’t changed….ever. So, let’s ground ourselves in that.

For now, we can park all the noise around ETL vs ELT, data warehouses vs lakehouses, etc. These are all extensions and elaborations on the fundamentally basic pattern of ETL. Here’s a diagram to demonstrate how simple it is.

We say this is deceptively simple because it is. There is work to be done before within each of these steps as well as before and after each one.

Extract, Transform and Load (and stuff)

Here are some some thought starters to gently ease you into the wild world of data work. A good way to get a taste of these is to frame them as questions. Here are some questions.

Pre-Extract (pre-E) - what data do we need? why do we need it? where can we get it?

Extract (E) - how will we connect to the data? how do we authenticate? Do we have the right authorization? what format will the data come in? What do we know about the schema? How reliable is it? Is it incremental? How do we manage networking issues?

Transform (T) - do we need to combine this data with other data? How is all of the data related? What does each record or row represent? is the schema dynamic? Should we enforce data types? How will the data be used downstream? What sort of data model should we build? Do we need to quarantine data that doesn’t look right?

Load (L) - who or what will be consuming this data? who should have access to what data and what drives this? Will this be published into a report for business users? How will business users be accessing the report, from their laptops or mobile phones? Will they be mobile or working in remote locations with poor network access? Does data need to be cached so that it can work offline? Will users want to write back?

Post-Load (post-L) - is everything working as expected? are users engaging with the outputs? What is useful? What isn’t? How can we improve it? Do users need training? are they aware this exists? Are we achieving solution uptime SLAs? Do we even have a way to monitor whether or not we are achieving solution uptime SLAs?

Summary

Right? This ship is hard. There’s a lot to it. And that is just a 20-minute brain dump from a real person that does data work.