AThoroughparisonofDeltaLake,Icebergand

HudiJunjie

ChenAboutMe?

Software

engineer

at

Tencent

Data

Lake

Team?

Focus

on

big

data

areafor

yearsAgendaIntroduction

to

DeltaLake,

Apache

Icebergand

Apache

HudiKey

FeaturesparisonMaturityConclusion?

Tooling?

Transaction?

Integration?

Performance?

Data

mutation?

StreamingSupport?

Schema

evolutionWhatfeaturesare

expectforthedatalake?Unified

Batch&

StreamingDataMutationScalableMetadataTransaction(ACID)DataLakeData

QualityStoragePluggableIndependenceof

EnginesDelta

LakeDelta

Lakeis

an

open-sourcestoragelayer

that

bringsACIDtransactionsto

Apache

Spark?and

big

data

workloads.ApacheIcebergAntableformatforhugeanalyticdatasetswhichdelivershighquery

performancefortableswithtensofpetabytesofdata,alongwithatomicmits,concurrentwrites,andSQLpatibletableevolution.InteractiveQueriesSpark

Batch&StreamingAI&ReportingStreamingStreamingAnalyticsDFS/Cloud

StorageApache

HudiApacheHudiingests&managesstorageoflargeanalyticaldatasetsoverDFSA

Quickparison2020-05Delta

Lake

(open

source)Apache

IcebergApache

HudiTransaction

(ACID)MVCCYYYYYNYYYTime

travelYYSchema

EvolutionData

MutationStreamingYYY

(update/delete/merge

into)Y

(upsert)Sink

and

source

for

spark

structstreamingSink

and

source(wip)

for

Sparkstruct

streaming,

Flink(wip)DeltaStreamerHiveIncrementalPullerFile

FormatParquetParquet,

ORC,

AVROAPI

available

(Spark

Action)DSv2,

InputFormatParquetpaction/CleanupIntegrationManualManual

and

AutoDSv1,

InputFormatDSv1,

Delta

connectorMultiple

language

supportStorage

AbstractionAPI

dependencyScala/java/pythonYJava/pythonYJava/pythonNSpark-bundledSpark,

presto,

hiveNative/Engine

bundledSpark,

hiveSpark-bundledDeltaStreamerData

ingestionTransactionDelta

Lake?

Model?

TransactionLog(DeltaLog)?

Optimisticconcurrencycontrol?

Checkpoint

changesintoparquet?

AtomicityGuarantee?

HDFSrename?

S3

filewrite?

Azurerenamewithout

overwrite?

TimeTravel?

timestamp?

versionnumberApacheIceberg?

Model?

Snapshot?

OptimisticconcurrencycontrolWR?

AtomicityGuarantee?

HDFSRename?

Hive

metastorelockS1S2S3S4?

TimeTravel?

snapshot

id?

timestampApache

Hudi?

Model?

Timeline?

Optimisticconcurrencycontrol?

AtomicityGuarantee?

HDFSrename?

TimeTravel?

Hoodiemit_timeDataMutationDelta

Lake?

Copy

on

Write

mode?

Step

1:

find

files

to

delete

according

to

filter

expression?

Step

2:

load

files

as

dataframe

and

update

column

values

in

rows?

Step

3:

save

dataframe

tonew

files?

Step

4:

logs

thefiles

to

delete

and

add

into

JSON,mit

to

table?

Tablelevel

APIs?

update,

delete(condition

based)?

merge

into

(upserta

source

into

target

table)Apache

Hudi?

CopyonWritetable?

Step1:

read

out

records

from

parquet?

Step2:

merge

records

accordingto

passingupdate

records?

Step3:

write

merged

recordsto

files?

Step4:mit

to

tablemitActionExecutor?

MergeonReadtable?

Store

delta

records

into

AVRO

format

log

file?

Scheduledpaction?

Indexing?

Mapping

Hudi

recordkey

(in

metadata

column)

to

file

group

and

file

id?

In-memory,

bloom

filter

and

HBase?

TablelevelAPIs?

upsertApacheIceberg?

Copy

on

Write

Mode?

File

leveloverwriteAPIsavailable?

Merge

onRead

mode?

PositionbaseddeletefilesandequalitybaseddeletefilesStreamingSupportDelta

Lake?

Deeply

integratedwithSpark

Structured

Streaming?

Asa

streaming

source?

Streamingcontrol:

maxBytesPerTrigger,

maxFilesPerTrigger?

DoesNOThandlenon-append(ignoreDeletesor

ignoreChanges)?

Asa

streaming

sink?

Appendmode?pletemodeApache

Hudi?

DeltaStreamer?

Exactlyonceingestionof

newevent

fromKafka?

Support

JSON,

AVRO

orcustomrecordtypes?

Managecheckpoints,

rollback&recovery?

Support

forpluggingin

transformations?

IncrementalQueries?

HiveIncrementalPuller?

AsSparkdatasource(beginInstantTime)ApacheIceberg?

Support

spark

struct

streaming?

As

streaming

source

(WIP)?

Ratelimit:

max-files-per-batch?

Offsetrange?

As

streaming

sink?

Appendmode?pletemode?

Supportflink

(WIP)TableSchemaEvolution?

Delta

Lake?

Use

Spark

schema?

Allow

Schema

merge

and

overwrite?

Apache

Hudi?

Use

Spark

schema?

Support

adding

new

fields

in

stream,

column

delete

is

not

allowed.?

Apache

Iceberg?

Independent

ID-based

schema

abstraction?

Full

schema

evolution

and

partition

evolutionMaturityIntegrations?

Delta

Lake?

DSv1?

Delta.io

connector

enable

Apache

Hive,

Presto?

Apache

Iceberg?

DSv2,

InputFormat,

Hive

StorageHandle

(WIP)?

Flinksink(WIP)?

Apache

Hudi?

InputFormat,

DSv1?

DeltaStreamer

for

data

ingestingQueryPerformanceOptimization?

Delta

Lake?

Vectorizationfrom

Spark?

Dataskippingviastatistic

from

Parquet?

Vacuum,

optimize?

ApacheHudi?

Vectorizationfrom

Spark?

Dataskippingviastatistic

from

Parquet?

Autopaction?

ApacheIceberg?

Predicatepushdown?

Nativevectorizedreader(WIP)?

Statistic

from

Icebergmanifest

file?

HiddenpartitioningTooling?

Delta

Lake?

CLI:

VACUUM,

HI