Data engineering helps make data more useful and accessible for consumers of data. To do so, ata engineering must source, transform and analyze data from each system. For example, data stored in a relational database is managed as tables, like a Microsoft Excel spreadsheet.
We are happy to announce Data Engineering with real-time use case Knowledge Sharing sessions
๐ Syllabus
๐ฏ ๐๐๐ซ๐ญ-๐:๐๐ง๐๐๐ซ๐ฌ๐ญ๐๐ง๐๐ข๐ง๐ ๐๐จ๐ฎ๐ซ๐๐ ๐๐ฒ๐ฌ๐ญ๐๐ฆ
๐ Have taken Retail Banking as case study and made it easy to understand for beginners and intermediate level.
๐ Why? This will in-turn ensure, have good clarity while providing solution and designing
๐ฏ ๐๐๐ซ๐ญ-๐:๐๐๐๐ฅ ๐ฐ๐จ๐ซ๐ฅ๐ ๐๐๐ญ๐ ๐๐ง๐ ๐ข๐ง๐๐๐ซ๐ข๐ง๐ ๐๐๐ฌ๐ ๐ฌ๐ญ๐ฎ๐๐ข๐๐ฌ
๐ Requirement - Will define case study, provide scenario-base problem
๐ Why? This will help us to focus on architecture, design and provide solution
๐ฏ ๐๐๐ซ๐ญ-๐:๐๐ฏ๐๐ซ๐ฏ๐ข๐๐ฐ ๐จ๐ ๐๐๐ญ๐ ๐ฆ๐จ๐๐๐ฅ๐ข๐ง๐
๐ Design simple data warehouse with dimension and facts
๐ Design Data Lake
๐ Azure SQL and SQL Server
๐ฏ ๐๐๐ซ๐ญ-๐:๐๐๐ญ๐ ๐๐ง๐ญ๐๐ ๐ซ๐๐ญ๐ข๐จ๐ง
๐ Batch and Real time streaming - ingestion, transformation, delivery and movement of data
๐ Batch includes Azure Data Factory
๐ Real Time streaming includes - Azure streaming
๐ Data Lake ingestion includes - Data Bricks
๐ Walk though on Kafka
๐ฏ ๐๐๐ซ๐ญ-๐:๐๐จ๐ง๐ข๐ญ๐จ๐ซ๐ข๐ง๐ ๐๐จ๐ง๐ญ๐ซ๐จ๐ฅ ๐๐ง๐ ๐๐๐ญ๐ ๐ ๐จ๐ฏ๐๐ซ๐ง๐๐ง๐๐
๐ Error handling
๐ Monitoring and control
๐ Testing
๐ Data Quality and Data Governance - Tools in market
๐ฏ ๐๐๐ซ๐ญ-๐:๐๐ข๐ฌ๐ฎ๐๐ฅ๐ข๐ณ๐๐ญ๐ข๐จ๐ง
๐ Do we have all the data? in desired quality?
๐ Visualization using Power BI
๐ Understand value to business, customer related service and decision making
Loved it?
๐๐ข๐ญ ๐ ๐ฌ๐ก๐จ๐ฐ โค๏ธ ๐๐๐ ๐๐จ๐ฎ๐ซ ๐๐ซ๐ข๐๐ง๐ โค๏ธ
