Mind the Gap: Accessing Java Objects from Python and Vice Versa

As a software engineer, you often encounter scenarios where accessing functionalities implemented in a different programming language becomes essential. This need arises when your current programming language may not be the best choice for certain tasks, such as complex numerical computations, or when you must use specific libraries only available in another language.

For example, you may have a Web application written in Python using the Flask framework that must perform complex numerical computations on large datasets, for which Python is not suitable. Then, you might choose to implement the computationally intensive algorithms of the application in a language like C or C++, which offer better performance for such tasks. You can then leverage the interoperability between Python and C/C++ to access these algorithms.

This blog post delves into the exciting world of bidirectional communication between Python and Java. While many libraries facilitate interoperability between these languages, they often limit you to “one-way communication”, restricting you to accessing Java classes in Python. With Py4J, Python programs can dynamically interact with Java objects within a JVM and vice versa [1].

The following code snippet from Py4J depicts how one can connect to the JVM from Python and access Java classes as if they were Python objects.

from py4j.java_gateway import JavaGateway
gateway = JavaGateway()
random = gateway.jvm.java.util.Random()
number1 = random.nextInt(10)
number2 = random.nextInt(10)

The above code although simple, indicates how you can create random numbers in Python using Java Random class. Py4J has helpful documentation for different scenarios to help developers, including accessing Java collections (e.g., Map, List, Set, etc.) in Python, and accessing “custom” Python objects in Java and vice-versa.

To send custom Python objects to Java, you must implement a Java interface in Python as documented here. This example, however, does not show how a Python object with inheritance hierarchies can be sent to Java. The remaining part of the blog therefore provides a full-fledge, working example.

Required Libraries

First, install Py4J packages for Python and Java by following this documentation.

Java Interfaces and Classes

To exchange custom Python objects with Java, the Python classes should implement Java interfaces. This allows the JVM to call Python objects. Without this, you will get an error similar to the one indicated below:

Traceback (most recent call last): File "", line 1, in File "py4j/java_gateway.py", line 158, in call args_command = ''.join([get_command_part(arg) for arg in args]) File "py4j/java_gateway.py", line 68, in get_command_part command_part = REFERENCE_TYPE + parameter.get_object_id()AttributeError: 'list' object has no attribute 'get_object_id'

Below is the structure of the Java project with interfaces that Python must implement.

└ main

└── java

└─── com

└──── example

├───── Stack.java

├───── StackEntryPoint.java

└────── interfaces

├────── IDigestiveSystem.java

├────── IDog.java

├────── IMammal.java

├────── INervousSystem.java

├────── IRespiratorySystem.java

├────── IWhale.java

In our example, IDog, and IWhale inherit from IMammal, and IMammal has INervousSystem and IRespiratorySystem (implemented in the Python code). The code snippet below shows the IWhale.java interface with five methods that are later implemented in Python.

package com.example.interfaces;
import java.util.List;
public interface IWhale extends IMammal {
	void swim();
	void addChild(IWhale child);
	List<Object> getChildren();
	IMammal getFirstChild();
	String getSpecies();
}

Unlike the Stack.java class in Py4J documentation, we have overloaded various methods to handle different objects sent from Python. The code snippet is shown below:

package com.example;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import com.example.interfaces.IMammal;
import com.example.interfaces.IDog;
import com.example.interfaces.IWhale;
public class Stack {
	private final List<Object> internalList = new LinkedList<Object>();
	public void push(final int element) {
		System.out.println("Integer Overloaded method called......");
		this.internalList.add(element);
	}
	public void push(final String element) {
		System.out.println("String Overloaded method called......");
		this.internalList.add(element);
	}
	public void push(final Map<Object, Object> element) {
		System.out.println("Hashmap Overloaded method called......");
		this.internalList.add(element);
	}
	public void push(fianl List<Object> element) {
		System.out.println("List Overloaded method called......");
		this.internalList.add(element);
	}
	public void push(final IDog element) {
		System.out.println("IDog Overloaded method called......");
		element.bark();
		this.processMammal(element);
	}
	public void push(final IWhale element) {
		System.out.println("IWhale Overloaded method called......");
		element.swim();
		System.out.println(element.getSpecies());
		System.out.println(element.getFirstChild());
		System.out.println(element.getChildren());
		this.processMammal(element);
	}
	public void processMammal(final IMammal element){
		element.eat("bones");
		element.sleep();
		element.reproduce();
		this.internalList.add(element);
	}
	public void push(final IMammal element) {
		System.out.println("IMammal Overloaded method called......");
		element.eat("bones");
		element.sleep();
		element.reproduce();
		this.internalList.add(element);
	}
	public void push(final Object element) {
		this.internalList.add(element);
	}
	public void pushMultiple(final int number){
		for (int i = 0; i <= number; i++) {
			this.internalList.add(i);
		}
	}
	public Object pop() {
		return this.internalList.remove(0);
	}
	public List<Object> getInternalList() {
		return this.internalList.stream().collect(Collectors.toList());
	}
	public void pushAll(final List<Object> elements) {
		for (Object element : elements) {
			this.push(element);
		}
	}
}

Py4J knows which method to call based on the object that’s being passed. The StackEntryPoint.java class is similar to the one provided in Py4J documentation. The constructor of StackEntryPoint.java instantiates the Stack class and there is a getter that exposes the instantiated Stack object (for the sake of simplicity). The GatewayServer in StackEntryPoint.java by default runs on port 25333, however, it can be configured to run on a different port. When StackEntryPoint.java is run, it will be waiting for a connection from the Python program on port 25333 as indicated above.

The Python Classes

The structure of the Python project is indicated below:

├── __init__.py

├── digestive_system.py

├── dog.py

├── main.py

├── mammal.py

├── nervous_system.py

├── respiratory_system.py

└── whale.py

As with the Java interfaces, Dog and Whale extends Mammal and Mammal has a nervous_system and a respiratory_system. The implementation of mammal.py is shown in the code snippet below:

from hierarchy.nervous_system import NervousSystem
from hierarchy.respiratory_system import RespiratorySystem
from hierarchy.digestive_system import DigestiveSystem
class Mammal:
	def __init__(self, name, age, species):
		self.name = name
		self.age = age
		self.species = species
		self.nervous_system = NervousSystem()
		self.digestive_system = DigestiveSystem()
		self.respiratory_system = RespiratorySystem()
	def eat(self, food):
		print(f"{self.name} is eating {food}")
	def sleep(self):
		print(f"{self.name} is sleeping")
	def reproduce(self):
		print(f"{self.name} is reproducing")
	def toString(self):
		return f"Name: {self.name}\nAge: {self.age}\nSpecies: {self.species}" \
			f"\nNervous System: {self.nervous_system.toString()}" \
			f"\nDigestive System: {self.digestive_system.toString()}" \
			f"\nRespiratory System: {self.respiratory_system.toString()}"
	class Java:
		implements = ['com.example.interface.IMammal']

Class Mammal conforms to the Java interface as indicated with a Java class having the implements class member, which is a list of interfaces that class Mammal conforms to. In our case, class Mammal only conforms to the IMammal interface. This class Java definition is how you tell Py4J to relate Python classes with Java interfaces. (All IMammal interface methods, including toString(), are implemented in the Python class.)

The code snippet for whale.py is also shown below:

from hierarchy.mammal import Mammal
class Whale(Mammal):
	def __init__(self, name, age, gateway):
		super().__init__(name, age, "Whale")
		self.species = "Blue Whale"
		self.children = gateway.jvm.java.util.ArrayList()
		self.gateway = gateway
	def swim(self):
		print("Swimming gracefully")
	def addChild(self, child: 'Whale'):
		self.children.append(child)
	def getChildren(self):
		return self.children
	def getSpecies(self):
		return self.species
	def getFirstChild(self):
		return self.children[0]
	class Java:
		implements = ['com.example.interfaces.IWhale']

The attribute self.children is assigned a Java ArrayList with the statement self.children = gateway.jvm.java.util.ArrayList(). Using Python list type and not Java ArrayList results in an error when you try to call getChildren() from Stack.java in Java. The error is similar to the one shown above and indicates that Java could not interpret the object.

raise Py4JJavaError( py4j.protocol.Py4JJavaError: An error occurred while calling o0.push.: py4j.Py4J Exception: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last): File "../../../lib/python3.8/site-packages/py4j/java_gateway.py", line 2468, in _call_proxy get_command_part(return_value, self.pool) File "../../../lib/python3.8/site-packages/py4j/protocol.py", line 298, in get_command_part command_part = REFERENCE_TYPE + parameter._get_object_id() AttributeError: 'list' object has no attribute '_get_object_id'

(Again, the Python Whale class implements all the methods in the Java interface IWhale.)

The entry point of the Python program is in the main.py file with the code snippet below:

from hierarchy.dog import Dog
from hierarchy.whale import Whale
from py4j.java_gateway import JavaGateway, CallbackServerParameters, GatewayParameters
if __name__ == "__main__":
	gateway = JavaGateway( 	
			callback_server_parameters=CallbackServerParameters(),
	gateway_parameters=GatewayParameters(auto_convert=True))
	# Create an instance of Dog
	buddy = Dog("Buddy", 1)
	cuddy = Dog("Cuddy", 2)
	duddy = Dog("Duddy", 3)
	blue_whale = Whale('Blue', 15, gateway)
	first_child = Whale('Red', 2, gateway)
	second_child = Whale('Orange', 8, gateway)
	blue_whale.addChild(first_child)
	blue_whale.addChild(second_child)
	first_child.addChild(second_child)
	second_child.addChild(blue_whale)
	# Get a reference to the Java Stack
	stack = gateway.entry_point.getStack()
	# Push the objects onto the Java Stack
	stack.push(buddy)
	stack.push(cuddy)
	stack.push(duddy)
	stack.push("Hello World")
	stack.push(125)
	stack.push(blue_whale)
	stack.push(first_child)
	stack.push(second_child)
	print(stack.pop())
	stack.push({'name': 'Bob', 'age': 5})
	stack.push([1, 2, 3, 4])
	gateway.shutdown()

There are two important things to note in the instantiation of the JavaGateway class:

callback_server_parameters=CallbackServerParameters() is required to call Python methods in the Java code. For example, element.sleep() in the processMammal() method in Stack.java will result in an error without callback_server_parameters provided.
auto_convert=True in GatewayParameters is required to convert Python types like list and dictionary to the corresponding Java types. Without this, stack.push({'name': 'Bob', 'age': 5}) in main.py will result in an error.

The CallbackServerParameters() enables “two-way communication” for Java to call python methods. Without it, an error indicating that a connection could not be established to allow for Java to call Python methods occurs:

Traceback (most recent call last): File "main.py", line 28, in <module> stack.push(buddy) File "../../../lib/python3.8/site packages/py4j/java_gateway.py", line 1322, in __call__ return_value = get_return_value( File "../../../lib/python3.8/site-packages/py4j/protocol.py", line 326, in get_return_value raise Py4JJavaError( py4j.protocol.Py4JJavaError: An error occurred while calling o3.push. : java.net.ConnectException: Connection refused at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:579) at java.base/sun.nio.ch.Net.connect(Net.java:568)...

Also, an error similar to the one below will occur if auto_convert is not set to True. Py4J will not automatically map python types like a dictionary to a map in Java:

Traceback (most recent call last): File "main.py", line 37, in <module> stack.push({'name': 'Bob', 'age': 5}) File "../../.../lib/python3.8/site-packages/py4j/java_gateway.py", line 1314, in __call__ args_command, temp_args = self._build_args(*args) File "../../../lib/python3.8/site-packages/py4j/java_gateway.py", line 1283, in _build_args [get_command_part(arg, self.pool) for arg in new_args]) File "../../../lib/python3.8/site-packages/py4j/java_gateway.py", line 1283, in <listcomp> [get_command_part(arg, self.pool) for arg in new_args]) File "../../../lib/python3.8/site-packages/py4j/protocol.py", line 298, in get_command_part command_part = REFERENCE_TYPE + parameter._get_object_id() AttributeError: 'dict' object has no attribute '_get_object_id'

The Java class StackEntryPoint.java must be running before main.py is run. The main.py connects to the JavaGateway on port 25333 by default. You could configure both StackEntryPoint.java and main.py to use a different port other than 25333.

Although this example shows how communication can be initiated from the Python program, the reverse (where communication is initiated from the Java program) could be achieved with this documentation.

References

https://www.py4j.org/index.html