extras.kafka — Apache Kafka Queue

A YellowService for running an Apache Kafka Queue. Runs the Bitnami Kafka and Zookeeper docker images.

Note

Requires the kafka extra. For more information, see our installation guide.

Note

This service runs a kafka broker/zookeeper pair. For a lightweight kraft service, see the yellowbox-kraft library.

class extras.kafka.KafkaService(docker_client: DockerClient, tag_or_images: str | tuple[str, str] = 'latest', inner_port: int = 0, outer_port: int = 0, bitnami_debug=False, **kwargs)[source]

A service to run the kafka queue. Inherits from SingleEndpointService. Usable with RunMixin and AsyncRunMixin.

Parameters:

  • docker_client – The docker client to used to pull and create the Kafka containers.

  • tag_or_images – The images to use to run the Kafka queue. If a string, it is used as the tag for both the bitnami zookeeper and kafka broker images. If a tuple, it is used as the image name for the zookeeper and broker , respectively.

  • inner_port – The internal port to expose on the Kafka broker for the zookeeper. Default is 0, which will assign to a random unoccupied port.

  • outer_port – The external port to expose on the Kafka broker for consumers and producers. Default is 0, which will assign to a random unoccupied port.

  • bitnami_debug – If enabled, will run the zookeeper and kafka broker in debug mode (BITNAMI_DEBUG=true).

  • **kwargs – Additional keyword arguments passed to SingleEndpointService.

Has the following additional methods:

connection_port() → int[source]

Returns the outer port the Kafka broker is listening on.

consumer() → ContextManager[KafkaConsumer][source]

Creates a kafka consumer for the service and wraps it in a closing context manager.

Note

due to version incompatibility, this methd is only available for python 3.11 and lower

producer() → ContextManager[KafkaProducer][source]

Creates a kafka producer for the service and wraps it in a closing context manager.

Note

due to version incompatibility, this methd is only available for python 3.11 and lower