Adding new data layer

All data layers have to inherit from DataLayer class. We recommend that you use tf.data while implementing your data layer.

You need to implement the following methods:

0. Static methods: get_required_params() and get_optional_params() to specify required and optional parameters correspondingly
1. __init__(self, params, num_workers=None, worker_id=None):
   • The params parameter should be a Python dictionary with options. Such as mini-batch shapes, padding, where to get the data from, etc.
   • If you are using tf.data, most of the ETL (extract-transform-load) logic should happen here.
2. gen_input_tensors(self):
   • This method should return a list of Tensors or Placeholders in which new data will be fed. With each call it should return new objects (for multi-GPU support). In case of tf.data, you can use output of tf.data.Iterator.get_next().
   • For example: TODO.
3. next_batch_feed_dict(self):
   • Most likely should return empty Python dictionary if you are using tf.data.
   • If you are using Placeholders in gen_input_tensors(self), this method should return corresponding feed_dictionary.
4. shuffle(self):
   • If you are not using tf.data this method should shuffle your dataset.
5. get_size_in_samples(self):
   • Should return the size of your dataset in samples.