The kinds of databases:
β’ Fancy files
β’ 1970s stays winning
β’ This would be so cool if it worked at scale
β’ Oh, that's why google has a monopoly on search
I've been playing around with backfill a lot. Dan's tool is giving me 40-50k records per second now.
Increasingly leaning toward "capture the backfill in an LSM or similar" and then build your database from that