This is one of 4620 IT projects that we have successfully completed with our customers.How can we support you?
Monitoring of ETL / ELT processes in the Business Intelligence environment
Brief description
The aim of the project is to monitor all ETL or ELT processes within the framework of data management of an Enterprise Data Warehouse (EDWH). In terms of quality assurance for both the development and the operation of an EDWH, the integration processes are to be made transparent in order to identify sources of error at an early stage. The PTA designs and implements dashboards in order to visualise selected metrics across components in a clear form according to the drill-down principle.
Supplement
PTA identifies relevant or appropriate metrics to monitor the ETL or ELT processes. For this purpose, the metadata is extracted from the Oracle Repository Database of the ETL/ELT tool Oracle Data Integrator (12c) at regular intervals and imported as metrics into the designated timeseries database InfluxDB. Analogous to the existing data warehouse architecture in the form of a Business Object Model (BOM), the PTA creates dedicated Grafana dashboards based on the metrics for all components. In addition, each component is visualised as a tile in an overview dashboard, which shows the current status based on the metrics and defined threshold values in green, yellow or red. The detailed dashboard of the respective component or sub-components can be called up at any time via drill-down.
Subject description
The monitoring stack used, based on Grafana and InfluxDB, enables a clear presentation of all components and ETL/ETL processes in accordance with the specified architecture of the data warehouse (DWH). In addition, the metrics can be analysed at different levels of detail (component/subcomponent) over time in order to identify anomalies.
Overview
- Industries traffic / transport / logistics
- Specialist tasks Administration, business intelligence, data warehouse, ETL, Monitoring
- DV tasks Data Warehouse
- System environment Business Intelligence, Cloud Computing, technical data processing
- Procedure model Kanban
- Databases InfluxDB, Oracle Data Integrator, Oracle Database Enterprise Edt., Teradata
- Development/ test tools Aqua Data Studio
Project start01.04.2020
Contact
PTA GmbH Head Office
Weberstraße 2-4
D-68165 Mannheim
© Copyright 2024 PTA GmbH | All rights reserved | Imprint | Privacy | Legal notice | Values & Code of Conduct
