DEV Community

Cover image for Your Cloud Data Platform POC Has an Expiry Date
Krishna Tangudu
Krishna Tangudu

Posted on Edited on

Your Cloud Data Platform POC Has an Expiry Date

Hi Folks,

I had shared a post on LinkedIn comparing a few Snowflake and Databricks capabilities with what was available around mid-2023. Our earlier warehouse POC covered Athena, Dremio, Redshift and Snowflake. Going through the documentation again, there are a few things I would revisit before using those findings for another decision.

I will keep this to the same three areas: join elimination, result downloads and Iceberg.

Note: The current capabilities below are from public documentation checked on November 13, 2026, not a fresh benchmark. The earlier POC observations apply to the configurations we tested then; we have not tested the newer capabilities and integrations described here.

1. Avoiding joins that the report doesn't need

Let us take a sales report which only needs total revenue. The BI tool may still generate a join between the sales and customer tables because that relationship exists in its model.

If the query doesn't use any customer attributes, the database may be able to skip that join. It needs valid key relationships to establish that doing so won't change the result. Duplicate customer keys, for example, could multiply the sales rows.

Snowflake had documented join-elimination improvements by July 2022. Its RELY constraints let the optimizer use declared relationships for qualifying optimizations. Snowflake documentation, 2022 release history.

For the Databricks pattern I discussed, I couldn't find documented support at the mid-2023 point. Primary-key RELY is now documented from Runtime 14.2, released in November 2023. Foreign-key RELY followed in Runtime 15.4 and the September 2024 SQL release. Requirements, Runtime 14.2, SQL release history.

So this is something I would no longer keep as a Snowflake-only advantage. Redshift also documents using primary and foreign keys to remove redundant joins. I haven't established when that was introduced. Redshift documentation.

The supported Databricks compute and table types need checking; classic SQL warehouses are excluded. Also, declaring a key doesn't make the data valid. Incorrect key declarations can lead to incorrect results.

2. Paying for the time taken to download results

For a large BI extract, query execution may finish while the client is still downloading rows. We looked at this during the earlier evaluation because the compute behavior during that download can affect cost.

Snowflake persists completed results separately from the warehouse cache. Its result-storage and suspension behavior allow later retrieval without keeping warehouse compute active solely for that retrieval. Auto-suspend settings and other running work still matter. Persisted results, warehouse suspension.

Databricks has Cloud Fetch to download larger results directly from cloud storage. This was available before mid-2023. Its current monitoring documentation also says SQL warehouses remain active while queries are running or fetching results. Cloud Fetch, warehouse monitoring.

When we tested Redshift during the earlier evaluation, we had to keep the compute running until the client finished downloading the results. We were therefore paying for the full extract duration, not just query execution.

We have not repeated that test on current Redshift or Databricks configurations. The earlier Redshift finding applies to the setup we tested; today’s behavior and cost need checking with the warehouse, connector and workload we intend to use.

3. Reading and writing Iceberg tables

Iceberg was one of the formats we were following in 2023. Snowflake Iceberg Tables became generally available in June 2024. Since then, the available integration options have expanded. Snowflake release.

The important detail is where the table's catalog is managed:

Scenario What the documentation supports
Another engine accessing Snowflake-managed Iceberg Read and write through Horizon Catalog
Snowflake accessing supported external Iceberg REST catalogs Read and write through the documented integration
Another engine accessing Unity Catalog-managed Iceberg Read and write through the Iceberg REST Catalog API
Databricks accessing foreign Iceberg tables managed outside Unity Catalog Read-only through the native foreign-table path
Redshift accessing supported Iceberg tables registered in AWS Glue Create, insert, update, delete and merge, with the documented requirements

Snowflake-managed access, external-catalog writes, Databricks Iceberg, Redshift Iceberg SQL.

Multiple engines do not necessarily need to write to the same table. The distinction matters when a team wants to keep its existing catalog and use another engine for transformations. Databricks’ native foreign Iceberg integration is documented as read-only; external engines can read and write Unity Catalog-managed Iceberg through its REST API. Whether that restriction matters depends on the intended architecture.

Redshift has changed considerably here. Iceberg querying became generally available in November 2023, append-only writes in November 2025, and UPDATE, DELETE and MERGE in April 2026. That old “not supported” entry needs another look. Querying, inserts, row-level changes.

Snowflake also added on-demand table registration for catalog-linked databases on November 3, 2026. A remote table can be registered when referenced, without waiting for the next catalog sync. Release note.

What else I would consider now

Dremio was part of the original POC. SAP completed its acquisition on July 6, 2026, and has described how Dremio technology fits into SAP Business Data Cloud's Iceberg and open-catalog direction. For teams combining SAP and non-SAP data, I would include that in another evaluation. Acquisition, SAP update.

Microsoft Fabric could be considered as well, particularly by teams already using Power BI. It wasn't in the original POC. Its Warehouse and Lakehouse experiences around OneLake give us another option to evaluate. Microsoft overview.

With AI in the mix, I feel product teams are shipping features faster than ever. As an architect, I still need to look further ahead: how the company works, the people who will maintain the platform, business direction, contracts and the overall financial commitment.

I wouldn't change platforms just because a feature was released. But I also wouldn't use a limitation from an old POC to rule one out without checking it again.

This comparison follows our earlier Glue-to-Horizon lakehouse journey, where team skills and operational ownership mattered alongside platform features.

If you have recently tested these result-download or Iceberg scenarios on Databricks or Redshift, please share your experience. I would be interested to know what you observed.

Top comments (0)