com.yahoo.bullet.pubsub

Class BufferingSubscriber

java.lang.Object

com.yahoo.bullet.pubsub.BufferingSubscriber

All Implemented Interfaces:

Subscriber, AutoCloseable

Direct Known Subclasses:

RESTSubscriber

public abstract class BufferingSubscriber
extends Object
implements Subscriber

This implements a Subscriber that provides a base subscriber that buffers a fixed number of messages read. See maxUncommittedMessages. It provides implementations of Subscriber.commit(String), Subscriber.fail(String) and Subscriber.receive() that honor the fixed number of messages to read. This class is intended to be used if your PubSub implementation does not care about (or cannot be) using commit and fail to reprocess messages from the PubSub and prefers to manage it in code.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	maxUncommittedMessages The maximum number of PubSubMessages we can have unacked at any time.
protected List<PubSubMessage>	receivedMessages A List of messages read.
protected Map<String,PubSubMessage>	uncommittedMessages A Map of messages that have not been committed so far.

Constructor Summary

Constructors

Constructor and Description
BufferingSubscriber(int maxUncommittedMessages) Creates an instance of this class with the given max for committed messages.

Method Summary

Modifier and Type	Method and Description
void	commit(String id) Commits allow clients to implement at least once, at most once or exactly once semantics when processing messages.
void	fail(String id) Marks the processing of the PubSubMessage with the given id as failed.
protected abstract List<PubSubMessage>	getMessages() Implement this method to read and return a List of PubSubMessage from your actual PubSub source.
protected boolean	haveMessages() Returns true if we already have messages to emit or if getMessages() returns non-null and non-empty List of messages.
PubSubMessage	receive() Gets a new PubSubMessage from the assigned partition/partitions (Here a partition is a unit of parallelism in the Pub/Sub queue, See PubSub).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.AutoCloseable

close

Field Detail

maxUncommittedMessages

protected final int maxUncommittedMessages

The maximum number of PubSubMessages we can have unacked at any time. Further calls to receive will return nothing.

receivedMessages

protected List<PubSubMessage> receivedMessages

A List of messages read. receive() emits from the head.

uncommittedMessages

protected Map<String,PubSubMessage> uncommittedMessages

A Map of messages that have not been committed so far.

Constructor Detail

BufferingSubscriber

public BufferingSubscriber(int maxUncommittedMessages)

Creates an instance of this class with the given max for committed messages.

Parameters:

maxUncommittedMessages - The maximum number of messages that this Subscriber will buffer.

Method Detail

receive

public PubSubMessage receive()
                      throws PubSubException

Description copied from interface: Subscriber

Gets a new PubSubMessage from the assigned partition/partitions (Here a partition is a unit of parallelism in the Pub/Sub queue, See PubSub).

Specified by:

receive in interface Subscriber

Returns:

the received PubSubMessage.

Throws:

PubSubException - when a receive fails.

commit

public void commit(String id)

Commits allow clients to implement at least once, at most once or exactly once semantics when processing messages. Common implementations might 1. Ack all received messages. 2. Commit current read offset to persistent/fault tolerant storage. Marks a message as fully processed. This message is forgotten and cannot be failed after. If we have equal or more than maxUncommittedMessages uncommited messages, further calls to receive() will return nulls till some messages are committed.

Specified by:

commit in interface Subscriber

Parameters:

id - The ID of the message to be marked as committed.

fail

public void fail(String id)

Marks the processing of the PubSubMessage with the given id as failed. Marks a message denoted by the id and sequence to have failed. This message is added for emission and will be emitted on the next receive().

Specified by:

fail in interface Subscriber

Parameters:

id - The ID of the PubSubMessage to mark as a processing failure.

haveMessages

protected boolean haveMessages()
                        throws PubSubException

Returns true if we already have messages to emit or if getMessages() returns non-null and non-empty List of messages. Otherwise, returns false.

Returns:

A boolean denoting whether we do have messages to emit.

Throws:

PubSubException - if there was an issue reading the messages.

getMessages

protected abstract List<PubSubMessage> getMessages()
                                            throws PubSubException

Implement this method to read and return a List of PubSubMessage from your actual PubSub source.

Returns:

A List of PubSubMessage if any were read. A null or an empty list if not.

Throws:

PubSubException - if there was an issue reading the messages.