We helped with
Energy data platform – VEKA
To be able to adjust and create new energy policies the Flemish Energy and Climate Agency (VEKA) needs data to be able to make well informed decisions. To do this VEKA needs complete and high-quality data, this data mainly consist of data about green energy production and the energy performance of Flemish buildings. Apart from making policies VEKA also needs to report this data to its employees, other government organizations, the energy sector and the inhabitants of Flanders. The main goals of building a new data platform were:
- Streamlining internal and external data sources.
- Promoting higher data quality.
- Making reporting accurate and consistent.
- Creating a uniform data model.
- Automating data processing to become more efficient.
- Setting up security measures when it comes to data access, especially focusing on GDPR.
VEKA quickly chose to set up their new data platform on the cloud. They made this decision because of the many advantages the platform brings, the main advantages are:
- Security: The data is protected against theft or loss, the suppliers of cloud platforms make sure your data is stored safely and make back-ups of your data in case something happens to the hardware or software.
- Innovation: the providers of these cloud platforms are continuously developing new services and improving the existing ones, this means you can easy start using the newest technologies.
- Flexibility: using the cloud makes it easy to change or add data sources, add storage space or computing power.
- Cost efficient: making use of cloud databases means you don’t have to purchase any hardware or software and you don’t have to manage these yourself. This saves a massive amount of time and resources.
This is how VEKA’s new Data Warehouse as a Service (DWaaS) was born. All internal and external data gets loaded onto this platform, then users can log into the platform to access data and reports. In the next part of this article we will zoom in on the figure below.
Let’s start at the start, the data input. We have both internal and external data entering the DWaaS platform. This data is either uploaded regularly (every day, hour, week, month…) or gets uploaded in real-time. Files get uploaded through the data portal, this portal facilitates the exchange of files between VEKA and its suppliers. After getting uploaded the data will move on to the file landing area where an original copy is saved.
The incoming data gets processed as little as possible and gets saved in the first layer of the data warehouse (the staging layer). This way we can always access the original data. Now the first transformations will take place, we use ETL tools to do this. ETL stands for extract, transform and load. The data we need will be extracted from the source, then this data will be transformed to the right format. This way we can guarantee high quality data and uniformity throughout the data warehouse. Finally, the data is loaded into the second layer of the data warehouse (operational data store).
In the third layer of the data warehouse the data is transformed again to be able to use it in reports and visualizations. Reports are made automatically in Power BI every time new data gets uploaded. Users can now log into the platform to view the reports and data they have access to. They can filter this data to get the information that is relevant for them, someone working for the province of Antwerp can use a location filter to only get the data for this province.
In the middle of the figure there are two blocks we haven’t discussed yet. The first one is the sandbox; this is a space for employees to experiment with the data to gain new insights. The last block is the artificial intelligence space, VEKA experiments with different AI algorithms to try and gain more insight into their data. All these new insights can in turn be used to adapt and adjust policies.
VEKA’s main goal was to create a platform where all data is stored and processed in a uniform way. DWaaS is the perfect answer, the uniformity ensures everyone inside and outside VEKA sees the same data visualized in the same way which means there will only ever be one version of the truth.
Another hugely important goal was to be more efficient. A good example of how DWaaS achieves this by supporting VEKA’s staff when compiling the EPB-report that every new building in Flanders must obtain when applying for a building permit. This report used to be made by hand but now the tables and visuals will be generated automatically in Power BI. Staff will no longer waste time by imputing data, but they will be able to use this time to interpret this data.