The term "data lake" was coined in 2010 by James Dixon, then CTO of Pentaho. A data mart, he said, is bottled water, cleaned and packaged for easy drinking. A data lake is the body of water in its natural state, which people can come to and sample as they like.
It's a good metaphor, and it hides the part everyone learns the hard way. Natural water is also where things go to rot.
Lake vs. warehouse
The difference comes down to when you decide what the data means.
A data warehouse decides up front. Data is cleaned, typed and loaded into tables with a schema before anyone queries it. That's called schema-on-write. Queries are fast and the numbers agree with each other, but every new source is a project, and anything that doesn't fit the schema doesn't get in.
A data lake decides later. Everything lands as-is: CSV exports, JSON event streams, application logs, Parquet files, images, whatever. The structure is applied when you read it, which is schema-on-read. Getting data in is cheap, and you can keep things you don't have a use for yet. Making sense of it is now the reader's problem.
The first lakes were Hadoop clusters, with HDFS on racks of commodity servers and MapReduce jobs to query them. Today the storage is almost always object storage, S3 or something that speaks its API, and the compute is separate. That's the real economic shift. Storage is billed by the terabyte, compute only runs while a query does, and the two scale independently.
Why you'd want one
- Cheap, durable storage for everything. Object storage costs a few cents per gigabyte per month. Keeping raw data you might need later is suddenly affordable.
- You don't have to know the question yet. The raw data is still there when somebody asks something new. A warehouse can only answer the questions its schema anticipated.
- One place for machine learning. Training pipelines want raw, detailed data, not quarterly summaries. A lake is where that data naturally sits.
- Separation from the systems that produce it. Analysts stop running reports against the production database at 9 a.m. on Monday.
How it turns into a swamp
"Store everything and figure it out later" has a predictable failure mode. Three years in, there are four hundred buckets of files nobody can identify. Nobody knows which copy of the customer table is current, whether a field means what its name says, or who's allowed to see it. People stop trusting the data, then stop using it, and the storage bill keeps growing anyway.
The usual causes:
- No catalog. If you can't search what's in the lake, what each dataset means and who owns it, then you don't have a lake. You have a very large shared drive.
- No table semantics. A folder of Parquet files isn't a table. Two jobs writing at once can corrupt it, a failed job leaves half its files behind, and there's no way to see what the data looked like yesterday.
- No ownership. Every dataset needs a person or team responsible for it. "The data team" isn't an owner.
- No retention. "We might need it someday" is how a lake grows forever. Decide what you keep and for how long, then delete the rest on schedule.
The lakehouse and the table format fight
The fix for the table-semantics problem is an open table format: a metadata layer that sits on top of files in object storage and makes them behave like database tables. Writes are atomic, schemas can evolve safely, and "time travel" lets you query the table as it was last Tuesday. There are three main ones:
- Apache Iceberg, started at Netflix and an Apache top-level project since 2020.
- Delta Lake, from Databricks.
- Apache Hudi, from Uber.
Put one of those on object storage, add a query engine like Spark, Trino or DuckDB, and you get what the industry calls a lakehouse: lake prices with most of a warehouse's reliability.
Which format wins became a very expensive question this month. On June 3, Snowflake announced Polaris, an open source catalog for Iceberg, during its annual summit. On June 4, Databricks announced it was buying Tabular, the company founded by Iceberg's original creators, for a reported one to two billion dollars. Two of the biggest vendors in the business spent the same week trying to own the open format their customers had picked so they wouldn't be locked in.
That's the good news, actually. When the vendors fight over who supports the open standard best, customers win. Your data sits in your bucket in an open format, and you can point a different engine at it next year.
Security: the lake is the target
A lake puts all your data in one place, which is the point, and it also makes the lake the single most valuable thing to steal. This month is making that case loudly. Mandiant is investigating a campaign against Snowflake customers. Accounts at around 165 organizations were accessed with usernames and passwords stolen by infostealer malware, some of them years old, on accounts without multi-factor authentication. Snowflake itself wasn't breached. Its customers' logins were.
The lessons apply to any lake, on any platform:
- Require MFA for every human login and use short-lived credentials for every machine.
- Apply least privilege at the table and column level, not the bucket level. Most people need some of the data, not all of it.
- Restrict network access so the lake only answers from places it should.
- Log every read. If someone exports a billion rows, you want to know that day, not when the data turns up for sale.
Building one yourself
You don't need a vendor to build a lake. An open source stack can do it:
- Storage: Ceph's RADOS Gateway or MinIO for S3-compatible object storage on your own hardware, or any cloud bucket.
- Table format: Iceberg, which every major engine can read and write.
- Query: Trino for interactive SQL across large datasets, Spark for heavy processing, and DuckDB for a laptop pointed at a few gigabytes of Parquet, which is a surprisingly large share of real analytics work.
- Catalog: Something to hold the Iceberg metadata and tell people what's there. This is the piece that keeps it from becoming a swamp, so don't treat it as optional.
Start small. One well-cataloged, well-owned dataset that people actually use beats a petabyte nobody trusts.
Since then
Added October 2026. Iceberg won. Polaris went to the Apache Incubator later in 2024. Databricks made Iceberg a first-class format alongside Delta. AWS launched S3 Tables in December 2024, managed Iceberg built into S3 itself. And one name on my self-hosted list above is a cautionary tale: MinIO stripped the admin console out of its community edition in 2025 and moved the open source project into maintenance mode by the end of the year. If you're building on someone's open source, check who controls it. I wrote more about that in Software You Rent vs. Software You Own.