Augmenting ETL Systems with GoldenGate's Change Data Capture Capabilities

To compete effectively in an age of growing customer expectations, organizations have been demanding business intelligence that draws on increasingly fresher data. However, the ETL-based data integration systems that many employ within their business intelligence environments process data in scheduled batches, which requires processing windows, and companies must stay within these shrinking batch windows while data volumes are growing exponentially.

This paper discusses several options for addressing this dilemma, with a particular focus on augmenting existing ETL systems with real-time change data capture (CDC) technology. GoldenGate's real-time, transactional CDC technology can effectively co-exist with ETL systems, leveraging existing ETL investments while enabling fresh business intelligence without relying on batch windows.

Table of Contents:

• Challenges in Today's ETL Environments
• Architectural Options
  • Customizing the ETL System to Extract Multiple Intra-Day Batches
  • Customizing the ETL System to Extract Only the Changed Data in "Mini Batches"
  • Replacing the ETL System with a Real-Time Data Integration Solution
  • Augmenting the ETL System with Change Data Capture Capabilities
• Requirements for Integrating ETL and CDC Technologies
• GoldenGate Solutions for Augmenting ETL Systems
  • GoldenGate Technology Overview
  • Using Staging Tables
  • Using Direct Flat File Integration
  • Using Message-Queue Infrastructure
  • Using GoldenGate in Parallel with an ETL System
• Conclusion

Download the PDF