DEV Community

Cover image for Direct Lake: Do You Still Need to Refresh a Semantic Model?
Sai Prudhvi Neelakantam
Sai Prudhvi Neelakantam

Posted on Originally published at geekindata.com AI-assisted

Direct Lake: Do You Still Need to Refresh a Semantic Model?

I used AI tools to help draft and edit this post. I checked the facts and the wording, and the original lives at geekindata.com. The diagrams are on that page.

Three levers: build, refresh, and read mode

Key takeaways

  • Document build, model refresh or framing, and storage mode as three levers with owners.
  • Orchestrate lake transforms, then trigger refresh only when dependents changed.
  • Prefer Direct Lake for large facts when the Delta layout fits; avoid SQL-endpoint comfort food.
  • Put MLV refresh in the DAG; partial freshness is worse than honest staleness.
  • Use a fixed identity for unattended Direct Lake and Import refresh.
  • Keep an incremental or skip-if-unchanged policy for Import and lake builds.

Fresh gold tables are useless if the semantic model still points at yesterday, or if refresh burns an hour on a fact you could have incrementalized. Refresh is an architecture problem: orchestration, identity, and storage mode. Treat it as a portal checkbox and you invent a second process nobody can explain.

Building curated tables, refreshing the semantic model, and choosing how the model reads data are three different levers. Fixing one does not fix the others. Mixing them produces slow facts, surprise fallbacks, and overnight jobs that succeed while reports stay stale.

Named model: three levers

  • Build curated tables: notebooks, pipelines, Spark jobs, and materialized lake views write or maintain Delta.
  • Refresh or frame the model: schedule, on-demand, or post-pipeline trigger. Import still reloads; Direct Lake often frames.
  • How the model reads: Import, DirectQuery, Direct Lake, or SQL endpoint views. This decides how expensive lever (2) becomes.
  • Contract: a green notebook is not a fresh semantic model. Document all three levers or you debug the wrong layer under pressure.

Three levers: build curated tables, refresh or frame the semantic model, and choose how the model reads data

Figure 1. Build, refresh or frame, and read mode are three levers. Fixing one does not fix the others. If your diagram has one arrow labeled Refresh, split the ownership.

Context and constraint

A common failure on Fabric lakehouse stacks: bronze and silver look healthy while reports still tell last night's story. Transforms are fine. The semantic refresh contract is not.

Direct Lake changes the freshness conversation. When the model reads Delta in OneLake without a classic Import copy, refresh stops meaning "reload every row into VertiPaq." It means the lake is correct, framing is current, and the connection identity can see the files. Import and hybrid shapes still need scheduled or triggered model refresh. Skipping checks on framing, MLVs, and connection policy is false confidence.

Figure 2 is the Learn framing for Direct Lake: the semantic model connects to Delta in OneLake rather than carrying a full Import copy of every heavy fact.

Direct Lake overview connecting a semantic model to Delta in OneLake

Figure 2. Direct Lake connects the semantic model to Delta in OneLake. Freshness becomes lake correctness, framing, and identity, not a VertiPaq reload of every row. Source: Microsoft Learn: Direct Lake overview.

Design

1. Orchestrate build, then refresh

Pipelines should finish lakehouse transforms, validate outputs when you care about parity, then trigger model refresh only when something the model depends on changed. Blind nightly full refresh is a default, not a design. For Import tables, that trigger still matters. For Direct Lake heavy facts, land correct Delta and optionally nudge framing rather than reloading history. If gold did not move, do not burn capacity pretending it did. Figure 3 is that ownership order.

Pipeline flow: lake transforms, validate outputs, skip-if-unchanged gate, then model refresh or framing

Figure 3. Finish transforms, validate when it matters, then refresh only if dependents changed. Independent model schedules that ignore pipeline success invent stale or incomplete semantics.

In my demo, demo_refresh_pipeline does exactly that order. Notebook1 builds gold, validate_gold is a Lookup that reads the first row of gold.value_by_segment, and if_gold_changed runs the native Semantic model refresh activity on demo_value_model only when this expression is true:

@greater(int(activity('validate_gold').output.firstRow.total), 0)
Enter fullscreen mode Exit fullscreen mode

The False branch has no activities. In one on-demand run all four activities succeeded: Notebook1 in 53 seconds, validate_gold in 11, if_gold_changed in 24, and the refresh in 22 (Figure 4). The gate is deliberately simple: it checks that gold has a positive total, not that it changed. A real skip-if-unchanged gate compares against what the model last saw.

Three stacked strips of one Fabric pipeline. First, demo_refresh_pipeline with Notebook1, validate_gold and if_gold_changed chained, and its Output tab showing all four activities Succeeded with durations 53s, 11s, 24s and 22s. Second, the Semantic model refresh activity inside the True branch, set to workspace demo_ws and semantic model demo_value_model. Third, the if_gold_changed expression, truncated by the UI, with one activity under True and none under False

Figure 4. Build, validate, then refresh, from one demo run. Three strips from separate screens of the same pipeline: the run output, the refresh activity inside the True branch, and the If gate. The UI truncates the expression; the full text is above. Run and model IDs are blurred.

2. Prefer Direct Lake for large facts when the layout fits

If a heavy fact is read through a SQL endpoint view because it worked in the lab, ask whether Direct Lake on the Delta table is the intended path. Wrong mode shows up as slow queries, odd refresh behavior, and capacity burn that looks like a DAX problem. Direct Lake wants clean Delta, sensible types, and a model shaped for that mode. SQL views in front of Direct Lake often reintroduce the latency you left Import to escape. Mixing Import dimensions with Direct Lake facts can be intentional; without a written reason it becomes tribal knowledge.

3. Treat Materialized Lake Views as their own freshness class

MLVs are not ordinary gold tables with a nicer name. They have their own refresh and dependency story. If a semantic model points at an MLV or at gold that depends on one, your pipeline DAG must include that refresh path. Hoping the view "is just SQL" yields correct notebooks and wrong reports. Write down who refreshes the MLV, what fails open versus fails closed, and whether the model can see a partially updated graph. Partial freshness is worse than honest staleness.

The MLV refresh is its own run with its own status. In my demo, one manual run refreshed silver in 16 seconds and gold in 5, 36 seconds end to end (Figure 5). That status is what your pipeline should wait on before it touches the model.

Fabric materialized lake view lineage for one completed refresh run: bronze.table_a feeds silver.table_a_clean, completed in 16 seconds, which feeds gold.value_by_segment, completed in 5 seconds, with run details showing a manual run named demo_run_2, 36 seconds end to end, status Completed

Figure 5. One MLV refresh run in a demo lakehouse: bronze.table_a feeds silver.table_a_clean, which feeds gold.value_by_segment. Fabric records each view's duration and the run's own status. That chain is the refresh path your pipeline has to finish before the model reads gold.

4. Identity for refresh must match the connection

Interactive SSO that works in the browser is not the same as a service principal or managed identity refreshing overnight. Design the connection the way production will authenticate. Microsoft docs strongly recommend a fixed identity for Direct Lake cloud connections. Personal accounts fail on vacation, Conditional Access, or MFA. Map workspace roles, lakehouse permissions, and the model connection to that same identity. "It works when I click Refresh" is not an operations proof.

In my demo, the Direct Lake model's cloud connection maps to the default, Single Sign-On (Entra ID), with "Keep your Direct Lake data up to date" On. Creating a cloud connection instead offers Service Principal and Workspace identity among the auth methods. Either one gives refresh a fixed identity instead of whoever is signed in. I did not bind one in the demo, so it still runs as SSO (Figure 6).

Two panels from Fabric. Left: semantic model settings where the cloud connection for the OneLake source maps to Default: Single Sign-On (Entra ID), with Create a connection as the alternative, and Keep your Direct Lake data up to date switched On. Right: the New connection form whose authentication methods are Key, OAuth 2.0, Shared Access Signature (SAS), Service Principal and Workspace identity

Figure 6. The identity options for a Direct Lake model's cloud connection. Left, the demo model as it stands: default SSO, automatic Direct Lake updates On. Right, the auth methods a new connection offers, including Service Principal and Workspace identity. No fixed identity was bound in the demo; data source paths and hostnames are blurred.

Figure 7 is the ownership split.

Anti-pattern interactive SSO versus production fixed identity for unattended Direct Lake and Import refresh

Figure 7. Browser click success is not overnight proof. Use a fixed identity for unattended Direct Lake and Import, mapped to the same workspace and lakehouse rights.

5. Know when classic refresh still matters

Direct Lake does not delete the need for refresh architecture. Import tables still need it. Composite models still need a clear story per table. Framing and metadata operations still belong in the runbook. Calculated tables and hybrid patterns can reintroduce load work even when the big fact is Direct Lake. Keep incremental discipline for Import: full materialization of a huge fact every run will eventually own your capacity calendar.

Figure 8 is the Learn framing for the lakehouse as the analytical surface. The model still has to be honest about how it reads that surface.

Lakehouse as the analytical surface in Microsoft Fabric

Figure 8. The lakehouse is the analytical surface. Build the lake, then make the model honest about how it reads it. Source: Microsoft Learn: lakehouse overview.

Common mistakes

Pipeline green, model stale.
Transforms finished; nobody triggered the model or framing step.

Independent schedules.
Model refresh ignores pipeline success, so you refresh incomplete gold or skip after a late land.

SQL endpoint comfort food.
Heavy facts stay on views because prototyping was easy. Production pays in query time and confused Direct Lake expectations.

MLV ignored in the DAG.
Definition exists; refresh ownership does not. Downstream gold and semantics drift on quiet nights.

Interactive identity for unattended work.
Works on the author's session. Fails overnight under a different auth path.

Mixed modes without a legend.
Nobody can say why dim is Import and fact is Direct Lake, so the next change guesses.

Use as-is or adapt

Use as-is

  • What builds tables versus what refreshes or frames models, with owners.
  • Storage mode per table or model, with a one-line reason.
  • Identity used for unattended Direct Lake and Import refresh.
  • Incremental or skip-if-unchanged policy for Import and lake builds.
  • MLV and dependent-gold refresh placement in the pipeline.
  • Alerting when the pipeline succeeds and semantic freshness checks fail.

Adapt before you use

Tighter orchestration means more pipeline authoring and more alerts. You will negotiate with capacity owners about burning CU on full Import refresh versus Direct Lake framing. Direct Lake reduces classic refresh pain for large facts but raises the bar on Delta hygiene, permissions, and model design. MLVs add elegance and dependency debt: every materialized hop needs a DAG owner, or freshness becomes folklore.

Try it yourself

The MLV file ingestion demo in my public tech-demos repo builds a files to bronze, silver and gold chain of materialized lake views in one lakehouse, so you can run a refresh and watch the lineage and per-view status yourself.

Primary references: Direct Lake overview, Lakehouse overview, and Microsoft Learn on semantic model refresh and Direct Lake connections.

Build the lake. Make the model honest about how it reads that lake. Prove the overnight identity can do both without you watching. If your diagram has one arrow labeled Refresh, split build, mode, and model operations across owners.

Related

Top comments (0)