Apache Spark 3.3.0 might fail on Java 17 with the error message ‘Cannot Access Class sun.nio.ch.DirectBuffer’ due to changes in the module accessibility in Java. Let’s dive deeper into why this happens and how you can resolve it.
Background
With the introduction of the Java Platform Module System (JPMS) in Java 9, strict encapsulation of internal APIs was enforced. The `sun.nio.ch.DirectBuffer` class is part of the internal, non-public API, and its use can lead to issues with module access controls introduced in later versions of Java.
Apache Spark extensively uses low-level buffer management for performance reasons, and it looked up classes in the `sun.nio.ch` package, which might have started failing due to stricter module access controls in Java 17.
Understanding the Error
Java 17, being a long-term support (LTS) version, enforces module encapsulation more strictly. Hence, trying to access internal classes, like `sun.nio.ch.DirectBuffer`, directly would fail with access control errors.
Solution
To resolve this issue, you need to adjust the module access rules. Java’s `–add-exports` and `–add-opens` options can help make internal APIs accessible to Spark.
Steps to Resolve
- Open your Spark environment configuration file (often named `spark-env.sh` for Unix-based environments).
- Add the necessary JVM options to allow Spark to access the required internal classes.
Below is an example configuration to be added:
JAVA_OPTS="--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
Alternatively, if you are running Spark via a script or command line, you can directly include these options:
Example Command
spark-submit --conf "spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" \
--conf "spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" \
your_spark_application.py
This command configures both the driver and executor to open the `java.base/sun.nio.ch` module to all unnamed modules, which includes Spark’s runtime.
Conclusion
By ensuring proper module access controls using Java’s `–add-opens` and `–add-exports` options, you can run Apache Spark 3.3.0 on Java 17 without running into the `Cannot Access Class sun.nio.ch.DirectBuffer` error.