In recent weeks, I've had several discussions with colleagues and students about the role of Data Engineer (DE). Questions about the necessary skills and what to look for in a role to develop these skills are common. Here, I aim to share my perspective on DE roles and how this role is distinct from other data-related roles.
Data Engineering is a relatively new role, and a not well defined, yet. So, even if I have my own view on the matter, its hard to give advice to someone who will join a team and most probably will embrace their culture. Because I deeply believe that most people could have a decent career in whatever they pick to do if they put their efforts into it.
The fact that DE is relatively new in town, is not the only cause of its poor definition. There are more aspects of it such as the fact that technology is moving blazing fast in that space. As a result, what a DE should be capable of doing is a lot different in 2024 than in 2014, and most probably will be different in 2034 as well. But, In the end of the day, the fundamentals have been the same, and most probably will stay the same for long.
Fundamentals of data management is a big part of the role. The skill of transforming data, moving them around and set up ways to distribute them was a need in the market of 2014 and still remains in 2024. One big difference is that in 2024, there are cases that a no-code/low-code tool may do the trick. And that's happens because that kind of tools have been developed and are now available, and there are more and smaller organizations chasing value from their data which brings some simpler scenarios.
So, that environment has led us in need of roles who are fully committed on utilizing such tools in order to do the data management functions. The question is, are those roles DE or something different?
My view on the topic is, No. Those roles aren't a good fit for someone who wants to pursue a career in DE. In fact, I believe that we should use another term for making it more clear, such as Data Management Specialist or something. Also I think this is a big opportunity for people who love working data, but think code sucks. Maybe I post in future a blog post about the skills they should have.
On the other hand, Data Engineering is a new branch of Software Engineer, like others such as Web, Mobile, Game etc. Data Engineer is frequently mentioned as the person of doing the data management, but this is half the truth. The proper way of describing should be the person who creates software in order to do data management.
In conclusion, I would like to find similar examples elsewhere to grasp better an idea, and here is one for the idea of this blog post. Would you call someone who can create a Shopify Store a web developer with specialization in ecommerce? If the answer is no, then you shouldn't confuse a Data Engineering with Data Management role as well.