Self-service: In the traditional setup, the IT department built everything in a data warehouse. However, modern DW architecture enables splitting the responsibility between IT and business by allocation as per ownership and stewardship frameworks. The IT department does the heavy lifting by ingesting data from core operations and standardizing architecture building blocks like tables, schemas, data dictionary, and business dictionary, and then providing access to these tables, views, and schemas to data scientists, data explorers, data analysts, and business data consumers of various departments. From there, business units are provided ownership of data. They can generate reports if they have the skills, desire, and needs. Business units are made effective by focusing on the strength of the analytical skills of their analysts so that they can focus on core skills, and IT data engineers focus on data warehouse engineering skills by focusing on data preparation. Similarly, data scientists are provided access to raw data in the landing area or in a purpose-built sandbox where they can mix raw corporate data with external data to create predictive or AI models. This capability along with providing data catalogs and governance is important for encouraging innovation capabilities across the enterprise for digital transformation.
Adaptable: In a modern enterprise data warehouse architecture system, data flows smoothly from source systems to business end users. The purpose of the modern data warehouse architecture is to manage data flow by creating a series of multi-directional, dependent, and interconnected data pipelines that serve various changing business needs. These pipelines are built using data views, snapshots, and deltas, including reference and master data. These architecture building blocks are continuously reused, repurposed, and replenished to ensure the steady flow of high-quality and relevant data to the business.
Low Latency: Speed and cost of implementation are interrelated and are important factors for shifting to a modern data warehouse architecture. Data ops is used for debug fixing, DevOps for migration, activities like customizable template library for writing/creating/exporting metadata. Having automatic code/ ETL scripts/programs/jobs running schedule generation using a metadata repository reduces a lot of development time and effort, and reduces risk of mistakes. The ability to self-observe the data warehouse combined with easy access controls to restrict or open traffic to the database nodes and easy database node resizing so you can scale your database resources up or down as needed
reduces database admin/maintenance time. Another dimension is low latent processing; e.g., data grids of in-memory databases provide a distributed reliability, scalability and consistency that is shareable across multiple distributed applications.
These applications perform concurrent transactional and/or analytical operations in the low-latency data grid, thus drastically reducing the use of traditional high-latency storage. Some of the advantages are data grid consistency, availability, and durability via replication, partitioning, and persistent storage.
Simplicity: In principle, simple architecture is the best architecture. Given the complexity involved and diversity of the technical and business requirements of modern data warehouses, it is important to simplify these requirements and spend some time on design and approach; e.g., look at the big picture, long term, and create a multi-year road map. Take into consideration internal and external factors, such as size of enterprise, vision, organization culture, business environment, and trends. Plan ahead, and if required minimize data movement and data redundancy, and create standardization across enterprises. Sometimes traditional BI tools are sufficient at the time for small organizations, but not as they grow. At the end of the day, it boils down to cost and benefit analysis. Map all benefits you want to come out of BI tools with their technical and non-technical capabilities.
