com.yahoo.bullet.storage

Class MultiMemoryStorageManager<V extends Serializable>

java.lang.Object

com.yahoo.bullet.storage.StorageManager<V>

com.yahoo.bullet.storage.MultiMemoryStorageManager<V>

All Implemented Interfaces:

Serializable, AutoCloseable

public class MultiMemoryStorageManager<V extends Serializable>
extends StorageManager<V>
implements Serializable

A storage that stores everything in-memory and supports namespaces and partitions. It starts off with a fixed initial number of partitions for all namespaces. You may use repartition(String, int) to change it at runtime per namespace. Supported criteria:

1. MultiMemoryCountingCriteria that counts keys across namespaces

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected BulletConfig	config

Fields inherited from class com.yahoo.bullet.storage.StorageManager

DEFAULT_NAMESPACE

Constructor Summary

Constructors

Constructor and Description
MultiMemoryStorageManager(BulletConfig config) Constructor.

Method Summary

Modifier and Type	Method and Description
CompletableFuture<Boolean>	clear(String namespace) Clears the specified namespace.
CompletableFuture<Boolean>	clear(String namespace, int partition) Clears the specified partition under the given namespace.
CompletableFuture<Boolean>	clear(String namespace, Set<String> ids) Removes a given set of IDs from the storage under the given namespace.
void	close() .
protected V	convert(byte[] bytes) Converts a @byte[] to a type of the given object.
protected byte[]	convert(V object) Converts an object of the given type to a @byte[].
static <S> Map<String,byte[]>	fromObjectMap(Map<String,S> input, Function<S,byte[]> converter) Converts a Map of String and the given type to a Map of String to byte[].
CompletableFuture<V>	get(String namespace, String id) Retrieves data stored for a given String identifier in the given namespace as a Serializable object.
CompletableFuture<Map<String,V>>	getAll(String namespace) Retrieves all the IDs from the given namespace in the storage as a Map of IDs to their Serializable values.
CompletableFuture<Map<String,V>>	getAll(String namespace, Set<String> ids) Retrieves the values for the provided IDs from the given namespace in the storage as a Map of IDs to their Serializable values.
protected CompletableFuture<Map<String,byte[]>>	getAllRaw(String namespace) Retrieves all the IDs from the given namespace in the storage as byte[].
protected CompletableFuture<Map<String,byte[]>>	getAllRaw(String namespace, Set<String> ids) Retrieves the values for the provided IDs from the given namespace in the storage as byte[].
CompletableFuture<Map<String,String>>	getAllString(String namespace) Retrieves all the IDs from the given namespace in the storage as a Map of IDs to their String values.
CompletableFuture<Map<String,String>>	getAllString(String namespace, Set<String> ids) Retrieves the values for the provided IDs from the given namespace in the storage as a Map of IDs to their String values.
protected String	getDefaultNamespace() Gets the default namespace to store data under.
CompletableFuture<Map<String,V>>	getPartition(String namespace, int partition) Retrieves the IDs stored in the provided partition for the given namespace.
protected CompletableFuture<Map<String,byte[]>>	getPartitionRaw(String namespace, int partition) Retrieves the IDs stored in the provided partition for the given namespace.
CompletableFuture<Map<String,String>>	getPartitionString(String namespace, int partition) Retrieves the IDs stored in the provided partition for the given namespace.
protected CompletableFuture<byte[]>	getRaw(String namespace, String id) Retrieves a given ID from the given namespace in the storage.
CompletableFuture<String>	getString(String namespace, String id) Retrieves data stored for a given String identifier in the given namespace as a String.
int	numberOfPartitions(String namespace) Returns the number of partitions stored in this storage manager for the given namespace.
CompletableFuture<Boolean>	put(String namespace, String id, V value) Stores any Serializable object for a given String identifier in the given namespace.
CompletableFuture<Boolean>	putAll(String namespace, Map<String,V> data) Stores a map of IDs and values into the storage in the given namespace.
protected CompletableFuture<Boolean>	putAllRaw(String namespace, Map<String,byte[]> data) Stores a map of IDs and values into the storage in the given namespace.
CompletableFuture<Boolean>	putAllString(String namespace, Map<String,String> data) Stores a map of IDs and String values into the storage in the given namespace.
protected CompletableFuture<Boolean>	putRaw(String namespace, String id, byte[] value) Store a given ID and value for that ID into the given namespace in the storage.
CompletableFuture<Boolean>	putString(String namespace, String id, String value) Stores a String for a given String identifier in the given namespace.
CompletableFuture<V>	remove(String namespace, String id) Retrieves and removes data stored for a given String identifier as a Serializable object in the given namespace.
protected CompletableFuture<byte[]>	removeRaw(String namespace, String id) Removes a given ID from the given namespace in the storage.
CompletableFuture<String>	removeString(String namespace, String id) Retrieves and removes data stored for a given String identifier as a String in the given namespace.
CompletableFuture<Boolean>	repartition(String namespace, int newPartitionCount) Repartitions the data into the given new number of partitions for the given namespace.
static Map<String,byte[]>	toByteArrayMap(Map<String,String> input) Converts a Map of String to byte[] to a Map of String to String.
static byte[]	toBytes(String input) Converts a String to a byte[].
static <S> Map<String,S>	toObjectMap(Map<String,byte[]> input, Function<byte[],S> converter) Converts a Map of String to byte[] to a Map of String to the given type.
static String	toString(byte[] input) Converts a byte[] input to a String.
static Map<String,String>	toStringMap(Map<String,byte[]> input) Converts a Map of String to byte[] to a Map of String to String.
CompletableFuture<Boolean>	wipe() Removes all the IDs from the storage across all namespaces.

Methods inherited from class com.yahoo.bullet.storage.StorageManager

apply, clear, clear, clear, from, get, getAll, getAll, getAll, getAllString, getAllString, getPartition, getString, hash, numberOfPartitions, put, putAll, putAllString, putString, remove, removeString, repartition, retrieveAll

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

protected BulletConfig config

Constructor Detail

MultiMemoryStorageManager

public MultiMemoryStorageManager(BulletConfig config)

Constructor.

Parameters:

config - The BulletConfig to create this manager with.

Method Detail

putRaw

protected CompletableFuture<Boolean> putRaw(String namespace,
                                            String id,
                                            byte[] value)

Store a given ID and value for that ID into the given namespace in the storage.

Parameters:

namespace - The namespace to store the entry in.

id - The unique ID to represent this entry.

value - The value to store for this entry.

Returns:

A CompletableFuture that resolves to true if the storage was successful.

getRaw

protected CompletableFuture<byte[]> getRaw(String namespace,
                                           String id)

Retrieves a given ID from the given namespace in the storage.

Parameters:

namespace - The namespace to retrieve from.

id - The unique ID to retrieve from the storage.

Returns:

A CompletableFuture that resolves to the byte[] value for this ID or null if it does not exist.

getAllRaw

protected CompletableFuture<Map<String,byte[]>> getAllRaw(String namespace)

Retrieves all the IDs from the given namespace in the storage as byte[].

Parameters:

namespace - The namespace to retrieve from.

Returns:

A CompletableFuture that resolves to a Map of IDs to their stored values as byte[].

removeRaw

protected CompletableFuture<byte[]> removeRaw(String namespace,
                                              String id)

Removes a given ID from the given namespace in the storage.

Parameters:

namespace - The namespace of the ID.

id - The unique ID to remove from the storage.

Returns:

A CompletableFuture that resolves to the byte[] value for this ID or null if it does not exist.

getPartitionRaw

protected CompletableFuture<Map<String,byte[]>> getPartitionRaw(String namespace,
                                                                int partition)

Retrieves the IDs stored in the provided partition for the given namespace. By default, if the storage manager is unpartitioned, this works the same as getAllRaw(String)

Parameters:

namespace - The namespace to retrieve the IDs from.

partition - The partition number to return.

Returns:

A CompletableFuture that resolves to a Map of IDs to their stored values as byte arrays or null if no data is present.

wipe

public CompletableFuture<Boolean> wipe()

Removes all the IDs from the storage across all namespaces.

Returns:

A CompletableFuture that resolves to true if the wipe was successful.

clear

public CompletableFuture<Boolean> clear(String namespace)

Clears the specified namespace.

Parameters:

namespace - The namespace to clear.

Returns:

A CompletableFuture that resolves to true if the wipe was successful.

clear

public CompletableFuture<Boolean> clear(String namespace,
                                        Set<String> ids)

Removes a given set of IDs from the storage under the given namespace.

Parameters:

namespace - The namespace that has these IDs.

ids - The set of IDs to remove from the storage for the given namespace.

Returns:

A CompletableFuture that resolves to true if the wipe was successful and throws otherwise.

numberOfPartitions

public int numberOfPartitions(String namespace)

Returns the number of partitions stored in this storage manager for the given namespace. Partitions can be sharded in storage and can also be used as a smaller unit of processing to reduce memory requirements. Partitions are numbered with integers from 0 inclusive to numberOfPartitions(String) exclusive for the namespace. By default, if the storage manager is unpartitioned, this returns 0.

Parameters:

namespace - The namespace whose partitions are being asked for.

Returns:

The number of partitions in this storage manager.

clear

public CompletableFuture<Boolean> clear(String namespace,
                                        int partition)

Clears the specified partition under the given namespace.

Parameters:

namespace - The namespace for the partition.

partition - The partition to clear.

Returns:

A CompletableFuture that resolves to true if the wipe was successful.

repartition

public CompletableFuture<Boolean> repartition(String namespace,
                                              int newPartitionCount)

Repartitions the data into the given new number of partitions for the given namespace. If the storage manager is unpartitioned, this does nothing.

Parameters:

namespace - The namespace to repartition.

newPartitionCount - The new number of partitions to use.

Returns:

A CompletableFuture that resolves to true if the repartitioning was successful.

getDefaultNamespace

protected String getDefaultNamespace()

Description copied from class: StorageManager

Gets the default namespace to store data under.

Overrides:

getDefaultNamespace in class StorageManager<V extends Serializable>

Returns:

The default namespace.

putString

public CompletableFuture<Boolean> putString(String namespace,
                                            String id,
                                            String value)

Stores a String for a given String identifier in the given namespace.

Parameters:

namespace - The namespace to store this value under.

id - The ID to store this value under.

value - The object to store as the value.

Returns:

A CompletableFuture that resolves to true if the store succeeded.

getString

public CompletableFuture<String> getString(String namespace,
                                           String id)

Retrieves data stored for a given String identifier in the given namespace as a String.

Parameters:

namespace - The namespace of the data.

id - The ID of the data.

Returns:

A CompletableFuture that resolves to the data.

putAllString

public CompletableFuture<Boolean> putAllString(String namespace,
                                               Map<String,String> data)

Stores a map of IDs and String values into the storage in the given namespace.

Parameters:

namespace - The namespace to store the entries in.

data - The map of IDs and values to store.

Returns:

A CompletableFuture that resolves to true if the storage was completely successful.

getAllString

public CompletableFuture<Map<String,String>> getAllString(String namespace)

Retrieves all the IDs from the given namespace in the storage as a Map of IDs to their String values.

Parameters:

namespace - The namespace to retrieve from.

Returns:

A CompletableFuture that resolves to a Map of IDs to their String values.

getAllString

public CompletableFuture<Map<String,String>> getAllString(String namespace,
                                                          Set<String> ids)

Retrieves the values for the provided IDs from the given namespace in the storage as a Map of IDs to their String values.

Parameters:

namespace - The namespace of the IDs.

ids - The Set of IDs to retrieve.

Returns:

A CompletableFuture that resolves to a Map of IDs to their String values.

removeString

public CompletableFuture<String> removeString(String namespace,
                                              String id)

Retrieves and removes data stored for a given String identifier as a String in the given namespace.

Parameters:

namespace - The namespace of the data.

id - The ID of the data.

Returns:

A CompletableFuture that resolves to the data.

getPartitionString

public CompletableFuture<Map<String,String>> getPartitionString(String namespace,
                                                                int partition)

Retrieves the IDs stored in the provided partition for the given namespace. By default, if the storage manager is unpartitioned, this works the same as getAll(String)

Parameters:

namespace - The namespace to retrieve the IDs from.

partition - The partition number to return.

Returns:

A CompletableFuture that resolves to a Map of IDs to their stored values as Strings or null if no data is present.

toBytes

public static byte[] toBytes(String input)

Converts a String to a byte[].

Parameters:

input - The String input.

Returns:

The byte[] encoding of the input.

toString

public static String toString(byte[] input)

Converts a byte[] input to a String.

Parameters:

input - The byte[] input.

Returns:

The String decoded from the byte[].

toStringMap

public static Map<String,String> toStringMap(Map<String,byte[]> input)

Converts a Map of String to byte[] to a Map of String to String.

Parameters:

input - The String to byte[] map.

Returns:

The String to String converted map.

toByteArrayMap

public static Map<String,byte[]> toByteArrayMap(Map<String,String> input)

Converts a Map of String to byte[] to a Map of String to String.

Parameters:

input - The String to byte[] map.

Returns:

The String to String converted map.

close

public void close()

. By default, close does nothing. If the particular storage needs to release resources, this is the place to do it.

Specified by:

close in interface AutoCloseable

putAllRaw

protected CompletableFuture<Boolean> putAllRaw(String namespace,
                                               Map<String,byte[]> data)

Stores a map of IDs and values into the storage in the given namespace. By default, calls putRaw(String, String, byte[]) in parallel.

Parameters:

namespace - The namespace to store the entries in.

data - The map of IDs and values to store.

Returns:

A CompletableFuture that resolves to true if the storage was completely successful.

getAllRaw

protected CompletableFuture<Map<String,byte[]>> getAllRaw(String namespace,
                                                          Set<String> ids)

Retrieves the values for the provided IDs from the given namespace in the storage as byte[]. By default, calls getRaw(String, String) repeatedly on each of the given IDs in parallel.

Parameters:

namespace - The namespace of the IDs.

ids - The Set of IDs to retrieve.

Returns:

A CompletableFuture that resolves to a Map of IDs to their stored values as byte[].

put

public CompletableFuture<Boolean> put(String namespace,
                                      String id,
                                      V value)

Stores any Serializable object for a given String identifier in the given namespace.

Parameters:

namespace - The namespace to store this value under.

id - The ID to store this value under.

value - The object to store as the value.

Returns:

A CompletableFuture that resolves to true if the store succeeded.

get

public CompletableFuture<V> get(String namespace,
                                String id)

Retrieves data stored for a given String identifier in the given namespace as a Serializable object.

Parameters:

namespace - The namespace of the data.

id - The ID of the data.

Returns:

A CompletableFuture that resolves to the data.

putAll

public CompletableFuture<Boolean> putAll(String namespace,
                                         Map<String,V> data)

Stores a map of IDs and values into the storage in the given namespace.

Parameters:

namespace - The namespace to store the entries in.

data - The map of IDs and values to store.

Returns:

A CompletableFuture that resolves to true if the storage was completely successful.

getAll

public CompletableFuture<Map<String,V>> getAll(String namespace)

Retrieves all the IDs from the given namespace in the storage as a Map of IDs to their Serializable values.

Parameters:

namespace - The namespace to retrieve from.

Returns:

A CompletableFuture that resolves to a Map of IDs to their stored values.

getAll

public CompletableFuture<Map<String,V>> getAll(String namespace,
                                               Set<String> ids)

Retrieves the values for the provided IDs from the given namespace in the storage as a Map of IDs to their Serializable values.

Parameters:

namespace - The namespace of the IDs.

ids - The Set of IDs to retrieve.

Returns:

A CompletableFuture that resolves to a Map of IDs to their stored values.

remove

public CompletableFuture<V> remove(String namespace,
                                   String id)

Retrieves and removes data stored for a given String identifier as a Serializable object in the given namespace.

Parameters:

namespace - The namespace of the data.

id - The ID of the data.

Returns:

A CompletableFuture that resolves to the data.

getPartition

public CompletableFuture<Map<String,V>> getPartition(String namespace,
                                                     int partition)

Retrieves the IDs stored in the provided partition for the given namespace. By default, if the storage manager is unpartitioned, this works the same as getAll(String)

Parameters:

namespace - The namespace to retrieve the IDs from.

partition - The partition number to return.

Returns:

A CompletableFuture that resolves to a Map of IDs to their stored values as Serializable objects or null if no data is present.

convert

protected V convert(byte[] bytes)

Converts a @byte[] to a type of the given object. By default, uses SerializerDeserializer to deserialize using Java deserialization.

Parameters:

bytes - The byte[] to convert.

Returns:

The converted object or null if the input was null or the conversion was unable to be performed.

convert

protected byte[] convert(V object)

Converts an object of the given type to a @byte[]. By default, uses SerializerDeserializer to serialize using Java serialization.

Parameters:

object - The object to convert.

Returns:

The converted byte[] or null if the input was null or the conversion was unable to be performed.

fromObjectMap

public static <S> Map<String,byte[]> fromObjectMap(Map<String,S> input,
                                                   Function<S,byte[]> converter)

Converts a Map of String and the given type to a Map of String to byte[].

Type Parameters:

S - The type to convert from.

Parameters:

input - The String to the given type map.

converter - The Function that converts from the given type to byte[].

Returns:

The String to byte[] converted map.

toObjectMap

public static <S> Map<String,S> toObjectMap(Map<String,byte[]> input,
                                            Function<byte[],S> converter)

Converts a Map of String to byte[] to a Map of String to the given type.

Type Parameters:

S - The type to convert to.

Parameters:

input - The String to byte[] map.

converter - The Function that converts from byte[] to the given type.

Returns:

The String to the given type converted map.