Database as decoupling between source and analysis systems
Brief description
Data from various sources that are connected via special middleware systems, including Kafka Streaming, must be processed and analyzed. The analysis programs were previously connected directly to the source systems, which led to a high susceptibility to errors, maintenance and communication costs. To solve these problems, Kaffka middleware and a database are interposed so that source and target systems are decoupled and can act asynchronously.
Supplement
A relational database is used (SQL Server). The source code is managed, deployed and versioned in a .NET database project. We want to be able to react to changes in the source systems as flexibly and configuratively as possible in the database schema. Depending on the task, prepared data can be made available to data consumers via views.
Subject description
Data consumers should be able to configure themselves which data they want to obtain from a system without having to adapt the middleware that supplies this data.