Apache Iceberg – The Definitive Guide

Foreword by Gerrit Kazmaier
Foreword by Raghu Ramakrishnan
Foreword by Rick Sears
Preface
About This Book
Why We Wrote This Book
What You Will Find Inside
How to Use This Book
Feedback and Questions
Conventions Used in This Book
Using Code Examples
O’Reilly Online Learning
How to Contact Us
Acknowledgments

I. Fundamentals of Apache Iceberg
1. Introduction to Apache Iceberg
How Did We Get Here? A Brief History
Foundational Components of a System Designed for OLAP Workloads
Bringing It All Together
The Data Warehouse
A Brief History
Pros and Cons of a Data Warehouse
The Data Lake
A Brief History
Pros and Cons of a Data Lake
Should I Run Analytics on a Data Lake or a Data Warehouse?
The Data Lakehouse
What Is a Table Format?
Hive: The Original Table Format
Modern Data Lake Table Formats
What Is Apache Iceberg?
How Apache Iceberg Came to Be
The Apache Iceberg Architecture
Key Features of Apache Iceberg
Conclusion

2. The Architecture of Apache Iceberg
The Data Layer
Datafiles
Delete Files
The Metadata Layer
Manifest Files
Manifest Lists
Metadata Files
Puffin Files
The Catalog
Conclusion

3. Lifecycle of Write and Read Queries
Writing Queries in Apache Iceberg
Create the Table
Insert the Query
Merge Query
Reading Queries in Apache Iceberg
The SELECT Query
The Time-Travel Query
Conclusion

4. Optimizing the Performance of Iceberg Tables
Compaction
Hands-on with Compaction
Compaction Strategies
Automating Compaction
Sorting
Z-order
Partitioning
Hidden Partitioning
Partition Evolution
Other Partitioning Considerations
Copy-on-Write Versus Merge-on-Read
Copy-on-Write
Merge-on-Read
Configuring COW and MOR
Other Considerations
Metrics Collection
Rewriting Manifests
Optimizing Storage
Write Distribution Mode
Object Storage Considerations
Datafile Bloom Filters
Conclusion

5. Iceberg Catalogs
Requirements of an Iceberg Catalog
Catalog Comparison
The Hadoop Catalog
The Hive Catalog
The AWS Glue Catalog
The Nessie Catalog
The REST Catalog
The JDBC Catalog
Other Catalogs
Catalog Migration
Using the Apache Iceberg Catalog Migration CLI
Using an Engine
Conclusion

II. Hands-on with Apache Iceberg
6. Apache Spark
Configuration
Configuring Apache Iceberg and Spark
Configuring the Catalogs
Starting Spark with All the Configurations (AWS Glue Example)
Data Definition Language Operations
CREATE TABLE
ALTER TABLE
Alter a Table with Iceberg’s Spark SQL Extensions
DROP TABLE
Reading Data
The Select All Query
The Filter Rows Query
Aggregation Queries
Using Window Functions
Writing Data
INSERT INTO
MERGE INTO
INSERT OVERWRITE
DELETE FROM
UPDATE
Iceberg Table Maintenance Procedures
Expire Snapshots
Rewrite Datafiles
Rewrite Manifests
Remove Orphan Files
Conclusion

7. Dremio’s SQL Query Engine
Configuration
Data Definition Language Operations
CREATE TABLE
ALTER TABLE
DROP TABLE
Reading Data
Using the SELECT Query
Filtering Rows
Using Aggregated Queries
Using Window Functions
Writing Data
INSERT INTO
COPY INTO
MERGE INTO
DELETE
UPDATE
Iceberg Table Maintenance
Expire Snapshots
Rewrite Datafiles
Rewrite Manifests
Conclusion

8. AWS Glue
Configuration
Creating a Glue Database
Configuring the Glue ETL Job
Create a Table Using the Glue Data Catalog
Read the Table
Insert the Data
Conclusion

9. Apache Flink
Configuration
Prerequisites
Start the Flink Cluster and Flink SQL Client
Data Definition Language Operations
CREATE CATALOG
CREATE DATABASE
CREATE TABLE
ALTER TABLE
DROP TABLE
Reading Data
Flink SQL Batch Read
Flink SQL Streaming Read
Metadata Table
Writing Data
INSERT INTO
INSERT OVERWRITE
UPSERT
Flink DataFrame and Table API with Apache Iceberg Tables
Prerequisites
Configuring the Flink Job
Starting the Cluster and Building the Package
Running the Job
Conclusion

III. Apache Iceberg in Practice
10. Apache Iceberg in Production
Apache Iceberg Metadata Tables
The history Metadata Table
The metadata_log_entries Metadata Table
The snapshots Metadata Table
The files Metadata Table
The manifests Metadata Table
The partitions Metadata Table
The all_data_files Metadata Table
The all_manifests Metadata Table
The refs Metadata Table
The entries Metadata Table
Using the Metadata Tables in Conjunction
Isolation of Changes with Branches
Table Branching and Tagging
Catalog Branching and Tagging
Multitable Transactions
Rolling Back Changes
Rolling Back at the Table Level
Rolling Back at the Catalog Level
Conclusion

11. Streaming with Apache Iceberg
Streaming with Spark
Streaming into Iceberg with Spark
Streaming from Iceberg with Spark
Streaming with Flink
Streaming into Iceberg with Flink
Example of Streaming into Iceberg with Flink
Streaming with Kafka Connect
The Iceberg Kafka Sink
Streaming with AWS
Conclusion

12. Governance and Security
Securing Datafiles
Securing Files: Best Practices
Hadoop Distributed File System
Amazon Simple Storage Service
Azure Data Lake Storage
Google Cloud Storage
Securing and Governing at the Semantic Layer
Semantic Layer Best Practices
Dremio
Trino
Securing and Governing at the Catalog Level
Nessie
Tabular
AWS Glue and Lake Formation
Additional Security and Governance Considerations
Conclusion

13. Migrating to Apache Iceberg
Migration Considerations
Three-Step In-Place Migration Plan
Four-Phase Shadow Migration Plan
Migrating Hive Tables to Apache Iceberg
The Snapshot Procedure
The Migrate Procedure
Migrating Delta Lake to Apache Iceberg
Migrating Apache Hudi to Apache Iceberg
Migrating Individual Files to Apache Iceberg
Using the add_files Procedure
Migrating from Delta Lake or Apache Hudi Without Preserving History
Migrating from Anywhere by Rewriting Data
Migrating Data to a New Iceberg Table
Migrating Data into an Existing Iceberg Table
Conclusion

14. Real-World Use Cases of Apache Iceberg
Ensuring High-Quality Data with Write-Audit-Publish in Apache Iceberg
WAP Using Iceberg’s Branching Feature
Running BI Workloads on the Data Lake
Land the Raw Data into the Data Lake
Curate Virtual Data Marts/Data Products
Create a Reflection to Accelerate Our Dashboard
Connect Our View to Our BI Tool
Benefits of Running BI Workloads on the Data Lake
Implementing Change Data Capture with Apache Iceberg
Create Apache Iceberg Tables
Apply Updates from Operational Systems
Create the Change Log View to Capture Changes
Merge Changed Data in the Aggregated Table
Conclusion

Index
About the Authors