5 data warehouse myths that hold development back

Businesses now live and die by data. From managing resources and analyzing performance to hyper-targeted marketing and behavioral analytics, data has become essential to the way companies operate. The demand for data has never been higher -- and it’s only increasing.

This pressure passes directly onto the people who help meet this demand. Data warehouse (DW) and analytics professionals are having to develop new data sets and produce insights at a record pace, all whilst keeping costs down. It’s clear DW acceleration is critical and this is acknowledged across the industry: according to WhereScape and TDWI’s research published this year, 63 percent of data warehouse professionals consider DW acceleration extremely important.

In order to achieve this, DW teams are turning to agile methodologies such as SCRUM and rapid prototyping. These are going some way in helping DW teams meet increased data demand but there are still some myths and misconceptions when it comes to acceleration.

If we want the sector to move forward, we need to name, interrogate and dispel these myths once and for all.

Data warehouse myths...

1) Software automation is stealing developers’ jobs

Contrary to opinion, we’re not in a battle with machines. It may sound irrational when you say it out loud but it can be hard to shake the false belief that a consequence of automation will be that human jobs are replaced. But it’s crucial we move past this falsehood because believing this myth, combined with worry over job losses, can stop organizations from embracing tools that could transform their work.

Software automation is simply there to make developers’ lives easier, not to replace them. Leaning on automation frees up developers’ time by relieving them of repetitive, manual tasks, allowing them to concentrate on higher-level design tasks and boosting their productivity. It also enables developers to consider new and more agile development methods. This all results in development acceleration, allowing developers to deliver data solutions sooner.

2) Only development needs automation

Automation and acceleration shouldn’t just be the preserve of DW development. Automation can help data warehouse professionals in testing, monitoring, documentation, deployment and administration, similarly taking over repetitive tasks and giving them more time for higher level work. Using special automation software for this repetitive work can also help improve accuracy and make data compliance easier.

It’s encouraging to see that this myth is on the way out: according to WhereScape and TDWI’s report, 83 percent of data warehouse professionals disagree with the idea that development is the only DW aspect that needs automation and acceleration.

3) Automation is the enemy of creativity

Another myth is that the greater the levels of automation, the lower the capacity for creativity, flexibility and customization. This myth split respondents in the research, with 55 percent correctly identifying this idea as false and the remainder being unsure or believing it to be true.

In reality, automation can enhance creativity -- by freeing up professionals’ time for more creative, design-led tasks, as discussed before. And even if a tool is built to work in a certain way, developers have the ability (and should be looking to) adapt the way it’s used to a business’s individual needs and requirements.

4) Automation only works with certain limits

A fourth myth is that automation has its limits and will only be able to properly work with moderate volumes of data, organized in traditional data structures. In fact, the complete opposite is true and most automated software solutions are built to work with a very wide range of data sources, types, structures, containers and latencies. They’re also designed to cope with large data sets -- the industry isn’t in danger of outgrowing automation.

… and a misconception to clarify

One last misconception to clear up is around the affordability of automation tools. Investing in automation to improve DW acceleration doesn’t need to hurt the business bank account. In fact, compared to the cost of bringing on new staff to serve additional data needs, it’s a more affordable option, as payroll costs are typically higher than the expense of software licensing. As well as saving them time, automation can actually save companies money too. The knock on effect of increased efficiency means harder and faster results, freeing up employees to focus on higher value tasks, consequently means increased profits.  

The exponential growth of the data industry is exciting to behold but it needs extra help to reach its full potential. Automation tools allow data warehouse professionals to work at the greater speed that’s necessary, as well as drive innovation by giving them more time for design tasks and creative thinking. And the sector can’t let outdated myths hold us back. Although the majority of data warehouse professionals have the right idea about automation, it’s important we don’t let misconceptions make us miss out on a wealth of opportunities.

Image credit: [email protected]/depositphotos.com

Simon Spring is Account Director EMEA, at WhereScape. Simon joined WhereScape nearly ten years ago and throughout this time has worked effectively with hundreds of organizations looking to utilize data analytics and data warehouse automation to transform their business.