python - How to make a generator callable?

Question

I'm trying to create a dataset from a CSV file with 784-bit long rows. Here's my code:

import tensorflow as tf

f = open("test.csv", "r")
csvreader = csv.reader(f)
gen = (row for row in csvreader)
ds = tf.data.Dataset()
ds.from_generator(gen, [tf.uint8]*28**2)

I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-22-4b244ea66c1d> in <module>()
     12 gen = (row for row in csvreader_pat_trn)
     13 ds = tf.data.Dataset()
---> 14 ds.from_generator(gen, [tf.uint8]*28**2)

~/Documents/Programming/ANN/labs/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py in from_generator(generator, output_types, output_shapes)
    317     """
    318     if not callable(generator):
--> 319       raise TypeError("`generator` must be callable.")
    320     if output_shapes is None:
    321       output_shapes = nest.map_structure(

TypeError: `generator` must be callable.

The docs said that I should have a generator passed to from_generator(), so that's what I did, gen is a generator. But now it's complaining that my generator isn't callable. How can I make the generator callable so I can get this to work?

EDIT: I'd like to add that I'm using python 3.6.4. Is this the reason for the error?

Answer 1

The generator argument (perhaps confusingly) should not actually be a generator, but a callable returning an iterable (for example, a generator function). Probably the easiest option here is to use a lambda. Also, a couple of errors: 1) tf.data.Dataset.from_generator is meant to be called as a class factory method, not from an instance 2) the function (like a few other in TensorFlow) is weirdly picky about parameters, and it wants you to give the sequence of dtypes and each data row as tuples (instead of the lists returned by the CSV reader), you can use for example map for that:

import csv
import tensorflow as tf

with open("test.csv", "r") as f:
    csvreader = csv.reader(f)
    ds = tf.data.Dataset.from_generator(lambda: map(tuple, csvreader),
                                        (tf.uint8,) * (28 ** 2))