To be able to adjust and create new energy policies the Flemish Energy & Climate Agency (VEKA) needs data to be able to make well informed decisions. To do this VEKA needs complete and high-quality data, this data mainly consists of data about green energy production and the energy performance of Flemish buildings. Apart from making policies VEKA also needs to report this data to its employees, other government organizations, the energy sector and the inhabitants of Flanders. The main goals of building a new data platform were:
Streamlining internal and external data sources.
Promoting higher data quality.
Making reporting accurate and consistent.
Creating a uniform data model.
Automating data processing to become more efficient.
Setting up security measures when it comes to data access, especially focusing on GDPR
Solution
An Azure Synapse Analytics data platform with ELT flows designed according to medallion architecture (based on the core principles of our own data framework). Refinement of data through the Bronze, Silver, and Gold layers To enable extensive scalability. Furthermore, parallel notebook processing with notebook dependencies were configured.
A golden dataset which can be used in all dashboards and reports (Power BI), ensuring that all visualizations are based on data that has been validated and transformed for optimal performance and insight.
By analogy with best practices in software engineering, CI/CD is set up using Github.
Reusable components and extensive documentation for future developers to quickly adopt the way-of-working.
Result
The main objective was to create a platform where all data is streamlined into one place and processed in the same way. This uniformity ensures that everyone inside and outside VEKA, according to their access, gets to see the same data in the same way. Thus, there is little room for misunderstanding or other interpretation.
Methodologies
The methodologies used at VEKA, as outlined in the reference case, include:
Medallion Architecture: A data architecture approach that organizes data processing and storage into layers, typically including raw data, refined data, and aggregated data layers.
Data Streamlining: Integrating internal and external data sources to promote a seamless flow of information.
Quality Assurance: Implementing processes to ensure high data quality for accurate policy-making and reporting.
Reporting and Visualization: Utilizing Power BI to create consistent and accurate reports and visualizations.
Data Modeling: Establishing a uniform data model to standardize data storage and processing.
Automation: Automating data processing to enhance efficiency and reduce manual intervention.
Security Measures: Visualize sensitive data and enforce access controls and compliancy rules to protect sensitive information.
Data Framework Principles: The setup of all components followed the principles of Cloubis’ own data framework.
Infrastructure-as-code: provided support with the deployment of IaC-templates
These methodologies were crucial in achieving VEKA’s goals of efficiency, uniformity, and data-driven decision-making.
Technologies involved
Following technologies were utilized:
Microsoft Azure: used as cloud platform to host the data platform
Azure Synapse Analytics: A cloud-based analytics service that combines big data and data warehouse technologies for comprehensive analytics capabilities.
CI/CD with GitHub Actions
Power BI (Embedded): Utilized for creating automated reports and visualizations.
Easymorph: Employed for extracting, transforming, and loading data to ensure high quality and uniformity.
Github: used for version control and automated deployments.
Cloud Experience: Leveraged for the flexibility, security, and cost-efficiency of data management.
Data Portal: Facilitated the exchange of files between VEKA and its suppliers.
Business Intelligence: Implemented through data analysis and reporting.
Artificial Intelligence: Explored for gaining deeper insights from the data.
These technologies collectively contributed to the creation of a robust and efficient data management system for VEKA.
