Java Gateway Process Exited Before Sending Its Port Number

This tutorial demonstrates the Java gateway process exited before sending the driver its port number error in Java.

Java Gateway Process Exited Before Sending the Driver Its Port Number

PySpark is used to write Spark applications using Python APIs and provides a shell for interactively analyzing the data in a distributed environment.

While running the PySpark application in PySpark Shell, spark-submit, or even from spyder the error Pyspark: Exception: Java gateway process exited before sending the driver its port number can occur.

If you want to run PySpark on your Windows, Mac, or Linux, you must install Java on your machine because PySpark is usually considered to work as a Python framework. The reason for this error to occur is that either you haven’t set the JAVA_HOME or PYSPARK_SUBMIT_ARGS.

Here are the solutions to solve the Pyspark: Exception: Java gateway process exited before sending the driver its port number error.

Install Open JDK or Oracle JDK

You must wonder why you need Java to run PySpark. It is because the Spark is basically written in Scala, but later the industry adopted its API, so the PySpark was released for Python using the Py4J, as Py4J is a Java Library that is why Java is required to run PySpark.

Py4J is integrated into PySpark, allowing Python to interface with JVM objects.

We can use the following command to install the OpenJDK.

# Installing Open JDK
sudo add-apt-repository ppa:openjdk-r/ppa
sudo apt-get update
sudo apt-get install openjdk-11-jdk

# Installing Oracle JDK
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

Once the JDK is installed, you need to set the JAVA_HOME path and follow the steps below.

The above commands are for the Linux environments for Windows. The installers can be downloaded from here:

1. OpenJDK
2. Oracle

To set the JAVA_HOME at Windows, follow the steps below.

Set PYSPARK_SUBMIT_ARGS

Most of the time, the error Pyspark: Exception: Java gateway process exited before sending the driver its port number occurs when the PYSPARK_SUBMIT_ARGS is not set. Use the following command to set the PYSPARK_SUBMIT_ARGS.

export PYSPARK_SUBMIT_ARGS="--master local[3] pyspark-shell"

The above command will set the PYSPARK_SUBMIT_ARGS using the master. Following both solutions will solve the problem.