Modern Data Platform - Scott Technology
Scott Technology (Scott) is a multi-national organisation that designs and manufactures advanced automation solutions. Industries serviced include Meat Processing, Mining, Materials Handling, Appliance, and Industrial Automation. Over the past two decades, Scott has been on a journey of diversification and expansion through acquisition, creating a group of companies with largely independent IT systems. Scott is an innovative and technologically advanced organisation, with the ability to adapt and evolve in response to changes in the business climate it operates within.
The Problem
Scott uses several ERP systems to run various aspects of their business in NZ and Worldwide. The systems are a mix of on-premise and cloud-based solutions. The absence of integration between the different systems means that data often needs to be entered more than once across systems, or manually exported out of one system and then imported into another. There is also an increased reliance on Excel for integrated data reporting and process monitoring.
Scott wanted a solution that would:
Improve operational integration between the various systems they use.
Create a platform for data integration and warehousing that will:
help improve process integration and/or automation
fill gaps in ERP systems reporting functionality
enable data sharing between Scott companies, business units, and other parties
enable reporting on different levels (consolidated, regional)
adapt to changing ERP systems landscape and assist in providing continuity during transitions.
Automate the extraction of data from data sources into data integration and warehouse platform.
The Solution
The solution is an Azure-based data integration and warehouse platform that consolidates ERP systems data. The platform uses Azure Data Factory to automate ELT pipelines that move data from on-premise and cloud-based sources into a data warehouse hosted on Azure SQL where further transformation, integration and other data operations can be performed.
The data integration and warehouse platform implement metadata-driven pipelines to automate the ingestion of data from the source through to the various data layers in the data warehouse. Metadata-driven pipelines provide Scott with the ability to rapidly ingest new data sources through configuration changes with minimal pipeline development.
Two environments were set up, one for development and another for production workloads. DevOps is used for continuous integration and deployment across environments.
The Outcome
The initial benefit of having deployed the cloud-based data platform is that they are now able to reliably report on their ERP data in Power BI.
In the long term, the data platform has given Scott Technology a way to integrate and analyse its data in a secure, reliable, and accessible platform that can adapt to its ever-changing systems integration and reporting needs.
