Perf: Render scope upstream snapshots

docs/examples/sqlmesh_cli_crash_course.md

@@ -738,6 +738,11 @@ You'll use these commands as needed to validate that your changes are behaving a
 
 This is a great way to verify that your model's SQL is looking as expected before applying the changes. It is especially important if you're migrating from one query engine to another (ex: postgres to databricks).
 
+In large projects, add `--use-project-index` to load only the model being rendered and its upstream
+dependencies. To enable this behavior by default, set
+[`render.use_project_index`](../reference/configuration.md#render) to `true` in the project
+configuration.
+
 === "SQLMesh"
 
     ```bash
@@ -1254,4 +1259,4 @@ If you notice you have a lot of old development schemas/data, you can clean them
 
     ```bash
     tcloud sqlmesh janitor
-    ```
+    ```

docs/guides/configuration.md

@@ -90,6 +90,7 @@ The `config` sub-module API documentation describes the individual classes used
     - [Connection configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/config/connection.html) (separate classes for each supported database/engine)
     - [Scheduler configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/config/scheduler.html) (separate classes for each supported scheduler)
 - [Plan change categorization configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/config/categorizer.html#CategorizerConfig): `CategorizerConfig()`
+- [Render configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/config/render.html): `RenderConfig()`
 - [User configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/user.html#User): `User()`
 - [Notification configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/notification_target.html) (separate classes for each notification target)
 
@@ -331,7 +332,7 @@ The cache directory is automatically created if it doesn't exist. You can clear
 
 #### Project index
 
-The `--use-project-index` option on supported commands maintains a persistent model dependency index in the cache directory. Each project writes a file named `<project>_<hash>_model_index.json`.
+The `--use-project-index` option on the `lint` and `render` commands maintains a persistent model dependency index in the cache directory. Each project writes a file named `<project>_<hash>_model_index.json`. The option can be enabled by default for each command with `linter.use_project_index` or `render.use_project_index`, respectively.
 
 A full project load with the option enabled creates or refreshes the index. SQLMesh invalidates it when relevant configuration, gateway, macro, audit, or signal metadata changes, or when the set of model files changes. If the index is missing, invalid, or stale, SQLMesh safely falls back to a full project load and rebuilds it.
 
@@ -1503,6 +1504,35 @@ SQLMesh provides a linter that checks for potential issues in your models' code.
 
 Learn more about linting configuration in the [linting guide](./linter.md).
 
+### Rendering
+
+By default, `sqlmesh render` loads every model in the project. In large projects, you can use the
+persistent project index to load only the model being rendered and its transitive upstream
+dependencies. Enable indexed rendering for an individual command with `--use-project-index`, or
+make it the project default with the `render.use_project_index` configuration option.
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    render:
+      use_project_index: true
+    ```
+
+=== "Python"
+
+    ```python linenums="1"
+    from sqlmesh.core.config import Config, ModelDefaultsConfig, RenderConfig
+
+    config = Config(
+        model_defaults=ModelDefaultsConfig(dialect="duckdb"),
+        render=RenderConfig(use_project_index=True),
+    )
+    ```
+
+`Context.render` uses the configured value when `use_project_index` is omitted. Passing
+`use_project_index=False` explicitly disables indexed rendering for that API call. See the
+[`render` CLI reference](../reference/cli.md#render) for the other rendering options.
+
 ### Debug mode
 
 To enable debug mode set the `SQLMESH_DEBUG` environment variable to one of the following values: "1", "true", "t", "yes" or "y".

docs/reference/cli.md

@@ -447,6 +447,10 @@ Options:
                               only they will be expanded as raw queries.
   --dialect TEXT              The SQL dialect to render the query as.
   --no-format                 Disable fancy formatting of the query.
+  --use-project-index         Use the persistent project index to load and
+                              render only the target model and its upstream
+                              dependencies. Can also be enabled with
+                              render.use_project_index.
   --max-text-width INTEGER    The max number of characters in a segment before
                               creating new lines in pretty mode.
   --leading-comma             Determines whether or not the comma is leading

docs/reference/configuration.md

@@ -67,6 +67,12 @@ See all the keys allowed in `model_defaults` at the [model configuration referen
 | `linter.enabled`          | Whether linting is enabled (Default: `False`)                                                                                                      | boolean | N        |
 | `linter.use_project_index` | Whether to use the persistent project index for linting. Targeted linting loads selected models and their upstream dependencies. (Default: `False`) | boolean | N        |
 
+### Render
+
+| Option                     | Description                                                                                                                                                       | Type    | Required |
+|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|----------|
+| `render.use_project_index` | Whether to use the persistent project index when rendering. Only the target model and its upstream dependencies are loaded. (Default: `False`)                    | boolean | N        |
+
 ### Variables
 
 The `variables` key can be used to provide values for user-defined variables, accessed using the [`@VAR` macro function](../concepts/macros/sqlmesh_macros.md#global-variables) in SQL model definitions, [`context.var` method](../concepts/models/python_models.md#global-variables) in Python model definitions, and [`evaluator.var` method](../concepts/macros/sqlmesh_macros.md#accessing-global-variable-values) in Python macro functions.

sqlmesh/cli/main.py

@@ -141,8 +141,8 @@ def cli(
         if ctx.invoked_subcommand in SKIP_LOAD_COMMANDS:
             load = False
 
-    # Unlike the other commands above, lint can scope its own load for multi-project contexts.
-    if ctx.invoked_subcommand == "lint":
+    # These commands can scope their own load for multi-project contexts.
+    if ctx.invoked_subcommand in ("lint", "render"):
         load = False
 
     configs = load_configs(config, Context.CONFIG_TYPE, paths, dotenv_path=dotenv)
@@ -284,6 +284,12 @@ def init(
     help="The SQL dialect to render the query as.",
 )
 @click.option("--no-format", is_flag=True, help="Disable fancy formatting of the query.")
+@click.option(
+    "--use-project-index",
+    is_flag=True,
+    default=None,
+    help="Use the persistent project index to load and render only the target model and its upstream dependencies. Can also be enabled with render.use_project_index.",
+)
 @opt.format_options
 @click.pass_context
 @error_handler
@@ -297,19 +303,20 @@ def render(
     expand: t.Optional[t.Union[bool, t.Iterable[str]]] = None,
     dialect: t.Optional[str] = None,
     no_format: bool = False,
+    use_project_index: t.Optional[bool] = None,
     **format_kwargs: t.Any,
 ) -> None:
     """Render a model's query, optionally expanding referenced models."""
-    model = ctx.obj.get_model(model, raise_if_missing=True)
-
     rendered = ctx.obj.render(
         model,
         start=start,
         end=end,
         execution_time=execution_time,
         expand=expand,
+        use_project_index=use_project_index,
     )
 
+    model = ctx.obj.get_model(model, raise_if_missing=True)
     format_config = ctx.obj.config_for_node(model).format
     format_kwargs = {
         **format_config.generator_options,

sqlmesh/core/config/__init__.py

@@ -37,6 +37,7 @@
 from sqlmesh.core.config.naming import NameInferenceConfig as NameInferenceConfig
 from sqlmesh.core.config.linter import LinterConfig as LinterConfig
 from sqlmesh.core.config.plan import PlanConfig as PlanConfig
+from sqlmesh.core.config.render import RenderConfig as RenderConfig
 from sqlmesh.core.config.root import Config as Config, DbtConfig as DbtConfig
 from sqlmesh.core.config.run import RunConfig as RunConfig
 from sqlmesh.core.config.scheduler import BuiltInSchedulerConfig as BuiltInSchedulerConfig

sqlmesh/core/config/render.py

@@ -0,0 +1,13 @@
+from __future__ import annotations
+
+from sqlmesh.core.config.base import BaseConfig
+
+
+class RenderConfig(BaseConfig):
+    """Configuration for rendering model queries.
+
+    Args:
+        use_project_index: Whether to use the persistent project index when rendering.
+    """
+
+    use_project_index: bool = False

sqlmesh/core/config/root.py

@@ -35,6 +35,7 @@
 from sqlmesh.core.config.naming import NameInferenceConfig as NameInferenceConfig
 from sqlmesh.core.config.linter import LinterConfig as LinterConfig
 from sqlmesh.core.config.plan import PlanConfig
+from sqlmesh.core.config.render import RenderConfig
 from sqlmesh.core.config.run import RunConfig
 from sqlmesh.core.config.dbt import DbtConfig
 from sqlmesh.core.config.scheduler import (
@@ -141,6 +142,7 @@ class Config(BaseConfig):
         format: The formatting options for SQL code.
         ui: The UI configuration for SQLMesh.
         plan: The plan configuration.
+        render: The render configuration.
         migration: The migration configuration.
         variables: A dictionary of variables that can be used in models / macros.
         disable_anonymized_analytics: Whether to disable the anonymized analytics collection.
@@ -183,6 +185,7 @@ class Config(BaseConfig):
     format: FormatConfig = FormatConfig()
     ui: UIConfig = UIConfig()
     plan: PlanConfig = PlanConfig()
+    render: RenderConfig = RenderConfig()
     migration: MigrationConfig = MigrationConfig()
     model_naming: NameInferenceConfig = NameInferenceConfig()
     variables: t.Dict[str, t.Any] = {}
@@ -208,6 +211,7 @@ class Config(BaseConfig):
         "ui": UpdateStrategy.NESTED_UPDATE,
         "loader_kwargs": UpdateStrategy.KEY_UPDATE,
         "plan": UpdateStrategy.NESTED_UPDATE,
+        "render": UpdateStrategy.NESTED_UPDATE,
         "before_all": UpdateStrategy.EXTEND,
         "after_all": UpdateStrategy.EXTEND,
         "linter": UpdateStrategy.NESTED_UPDATE,

sqlmesh/core/context.py

@@ -1199,6 +1199,7 @@ def render(
         end: t.Optional[TimeLike] = None,
         execution_time: t.Optional[TimeLike] = None,
         expand: t.Union[bool, t.Iterable[str]] = False,
+        use_project_index: t.Optional[bool] = None,
         **kwargs: t.Any,
     ) -> exp.Expr:
         """Renders a model's query, expanding macros with provided kwargs, and optionally expanding referenced models.
@@ -1211,11 +1212,23 @@ def render(
             expand: Whether or not to use expand materialized models, defaults to False.
                 If True, all referenced models are expanded as raw queries.
                 If a list, only referenced models are expanded as raw queries.
+            use_project_index: Whether to use the persistent project index to load and
+                render only the target model and its transitive upstream dependencies. If
+                omitted, the value of ``render.use_project_index`` is used.
 
         Returns:
             The rendered expression.
         """
         execution_time = execution_time or now()
+        use_project_index = (
+            self.config.render.use_project_index if use_project_index is None else use_project_index
+        )
+
+        if not self._loaded:
+            target_fqns = (
+                {self._node_or_snapshot_to_fqn(model_or_snapshot)} if use_project_index else None
+            )
+            self.load(model_fqns=target_fqns, use_project_index=use_project_index)
 
         model = self.get_model(model_or_snapshot, raise_if_missing=True)
 
@@ -1245,7 +1258,19 @@ def render(
             )
             return next(pandas_to_sql(t.cast(pd.DataFrame, df), model.columns_to_types))
 
-        snapshots = self.snapshots
+        if use_project_index:
+            # Only the target model and its transitive upstream dependencies can be referenced
+            # by the rendered query, so there is no need to create snapshots for the rest.
+            upstream_fqns = {model.fqn, *self.dag.upstream(model.fqn)}
+            upstream_models: UniqueKeyDict[str, Model] = UniqueKeyDict(
+                "models", {fqn: m for fqn, m in self._models.items() if fqn in upstream_fqns}
+            )
+            snapshots = self._snapshots(
+                upstream_models,
+                include_standalone_audits=False,
+            )
+        else:
+            snapshots = self.snapshots
         deployability_index = DeployabilityIndex.create(snapshots.values(), start=start)
 
         return model.render_query_or_raise(
@@ -3009,9 +3034,13 @@ def _get_engine_adapter(self, gateway: t.Optional[str] = None) -> EngineAdapter:
         return self.engine_adapter
 
     def _snapshots(
-        self, models_override: t.Optional[UniqueKeyDict[str, Model]] = None
+        self,
+        models_override: t.Optional[UniqueKeyDict[str, Model]] = None,
+        include_standalone_audits: bool = True,
     ) -> t.Dict[str, Snapshot]:
-        nodes = {**(models_override or self._models), **self._standalone_audits}
+        nodes: t.Dict[str, Node] = dict(models_override or self._models)
+        if include_standalone_audits:
+            nodes.update(self._standalone_audits)
         snapshots = self._nodes_to_snapshots(nodes)
         stored_snapshots = self.state_reader.get_snapshots(snapshots.values())
 

tests/cli/test_cli.py

@@ -2111,6 +2111,19 @@ def test_render(runner: CliRunner, tmp_path: Path):
 
     assert expected in cleaned_output
 
+    indexed_result = runner.invoke(
+        cli,
+        [
+            "--paths",
+            str(tmp_path),
+            "render",
+            "sqlmesh_example.full_model",
+            "--use-project-index",
+            "--no-format",
+        ],
+    )
+    assert indexed_result.exit_code == 0
+
 
 @time_machine.travel(FREEZE_TIME)
 def test_signals(runner: CliRunner, tmp_path: Path):

tests/core/test_config.py

@@ -16,6 +16,7 @@
     ModelDefaultsConfig,
     BigQueryConnectionConfig,
     MotherDuckConnectionConfig,
+    RenderConfig,
     BuiltInSchedulerConfig,
     EnvironmentSuffixTarget,
     TableNamingConvention,
@@ -70,6 +71,13 @@ def python_config_path(tmp_path_factory) -> Path:
     return config_path
 
 
+def test_render_config() -> None:
+    config = Config.parse_obj({"render": {"use_project_index": True}})
+
+    assert config.render == RenderConfig(use_project_index=True)
+    assert Config().update_with(config).render.use_project_index is True
+
+
 def test_update_with_gateways():
     gateway0_config = GatewayConfig(connection=DuckDBConnectionConfig())
     gateway1_config = GatewayConfig(connection=DuckDBConnectionConfig(database="test"))