Feature/spark connect interpreter - #5446
Conversation
Adds a new spark-connect module that enables Apache Zeppelin to connect to remote Spark clusters via the Spark Connect gRPC protocol (Spark 3.5+). This eliminates the need for a local Spark installation on the Zeppelin host. The module includes four interpreters: - %spark-connect: SQL execution (default) - %spark-connect.sql: SQL with optional concurrent scheduler - %spark-connect.pyspark: PySpark via Py4j bridge - %spark-connect.ipyspark: IPython variant of PySpark Key design decisions: - Per-user session quota enforcement to prevent resource exhaustion - Per-notebook fair ReentrantLock for sequential query execution - Dependency shading to avoid Netty/gRPC conflicts - Token-based authentication and SSL support - Python bridge via Py4j for shared Java SparkSession Includes comprehensive unit tests (SparkConnectUtilsTest) and integration tests (gated by SPARK_CONNECT_TEST_REMOTE environment variable) for SparkConnectInterpreter, SparkConnectSqlInterpreter, and PySparkConnectInterpreter.
…ython interpreters Per @jongyoul's review feedback, drop the Py4j bridge and custom Python wrappers (~1,270 lines) in favor of PySpark's native Spark Connect client. The Java interpreter forwards the connection URI via SPARK_REMOTE and Python calls SparkSession.builder.remote(...).getOrCreate() directly. Java/SQL and Python are now independent Spark Connect sessions on the same server; cross-language sharing flows through catalog tables.
Add spark-connect to the INTERPRETERS list of the interpreter-test-non-core job so the new module is built and its tests executed on every PR. The integration suites stay gated behind SPARK_CONNECT_TEST_REMOTE and are skipped when no Spark Connect server is available.
|
@dhama-shashank-meesho Hello, sorry for the late reply. I know we already had a spark-submit interpreter but I feel like we have several way to execute spark in Zeppelin. It's good itself but our codebase became conplicated. So I wonder if it's possible to merge all other type of spark interpreter into one way. I'm not 100% sure but spark-connect is the most recent way to execute spark qurey/script. Can you review the current way and merge them into one spark interpreter? WDYT? |
|
@jongyoul Agreed, there are too many spark paths right now. What this one is for: point Zeppelin at one shared long-running Spark cluster (Dataproc, or Spark on k8s) instead of every notebook starting its own Spark app.Today each interpreter holds its own driver and executors, so memory sits fragmented across many half-idle apps. With Connect there is no driver in Zeppelin at all, and the cluster manages memory centrally for everyone. On merging all of them into one, I'm for the goal but a few things block it now: no Scala interpreter here yet (only SQL and PySpark), no SparkContext in Connect so This PR only adds a new module and doesn't touch the existing interpreters — could we land it first and do the consolidation separately? |
2 participants
This PR introduces a new Spark Connect interpreter group for Apache Zeppelin, enabling notebooks to execute against an external Spark Connect server via the official SparkSession.remote() gRPC API (Spark 3.4+).
Unlike the classic Spark interpreter, no Spark driver or SparkContext runs inside the Zeppelin JVM/pod. The interpreter operates as a thin client, while all computation is executed remotely on the Spark cluster.
Key Features
Adds support for:
Remote Execution Model (gRPC)
Implements a remote-only execution model over Spark Connect gRPC, similar in deployment spirit to JDBC/Livy interpreters, but using Spark’s native Connect interface.
Session Management
Notebook-Level Fair Locking
Introduces NotebookLockManager to enforce FIFO paragraph execution ordering and prevent concurrent conflicts within the same note.
Optional Concurrent SQL Scheduling
Supports optional concurrent SQL execution via zeppelin.spark.concurrentSQL using a ParallelScheduler, while still enforcing notebook-level locking.
Streaming Result Mode
Adds zeppelin.spark.connect.streamResults to stream large result sets via iterator-based output.
PySpark Support (No Extra Spark Session)
Provides PySpark support using a Python subprocess + Py4j bridge, reusing the same Java Spark Connect session (avoids creating a separate Spark session in Python).
Connection String Handling
Connection URIs are built via SparkConnectUtils.buildConnectionString() and support token + SSL options. Sensitive values (e.g., token, user_id) are redacted in logs.
Cancellation Support
Stopping a paragraph triggers sparkSession.interruptAll(), cancelling active remote operations through Spark Connect.
Configuration Properties
Key configs include: