metadata
library_name: setfit
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
metrics:
- accuracy
- precision
- recall
- f1
widget:
- text: >
<p>I'm having a problem serving my text classification model on
<code>Tensorflow 1.12</code>. I'm using
<code>tf.estimator.inputs.pandas_input_fn</code> to read in my data, and
<code>tf.estimator.DNNClassifier</code> to train/evaluate. I'd then like
to serve my model.
(Apologies in advance, it's tough to provide a full working example here,
but it's very much like the example TF provides at <a
href="https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier"
rel="nofollow
noreferrer">https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier</a>
)</p>
<p>I'm currently saving my model with ...</p>
<pre class="lang-py prettyprint-override"><code>...
estimator.export_savedmodel("./TEST_SERVING/",
self.serving_input_receiver_fn, strip_default_attrs=True)
...
def serving_input_receiver_fn(self):
"""An input receiver that expects a serialized tf.Example."""
feature_spec = {
'Headline': tf.VarLenFeature(dtype=tf.string),
'Description': tf.VarLenFeature(dtype=tf.string)
}
serialized_tf_example = tf.placeholder(
dtype=tf.string, shape=None, name='input_example_tensor')
receiver_tensors = {'examples': serialized_tf_example}
features = tf.parse_example(serialized_tf_example, feature_spec)
return tf.estimator.export.ServingInputReceiver(features, receiver_tensors)
</code></pre>
<p>This actually fails to run with the error:</p>
<pre class="lang-sh prettyprint-override"><code>TypeError: Failed to
convert object of type <class
'tensorflow.python.framework.sparse_tensor.SparseTensor'> to Tensor.
Contents: SparseTensor(indices=Tensor("ParseExample/ParseExample:0",
shape=(?, 2),
dtype=int64), values=Tensor("ParseExample/ParseExample:2", shape=(?,),
dtype=string), dense_shape=Tensor("ParseExample/ParseExample:4",
shape=(2,), dtype=int64)). Consider casting elements to a supported type.
</code></pre>
<p>I tried to save a second way doing:</p>
<pre class="lang-py prettyprint-override"><code>def
serving_input_receiver_fn(self):
"""Build the serving inputs."""
INPUT_COLUMNS = ["Headline","Description"]
inputs = {}
for feat in INPUT_COLUMNS:
inputs[feat] = tf.placeholder(shape=[None], dtype=tf.string, name=feat)
return tf.estimator.export.ServingInputReceiver(inputs, inputs)
</code></pre>
<p>This actually works, until I try testing it with the
<code>saved_model_cli</code>.
Some output for <code>saved_model_cli show --all --dir
TEST_SERVING/1553879255/</code>:</p>
<pre class="lang-sh prettyprint-override"><code>MetaGraphDef with tag-set:
'serve' contains the following SignatureDefs:
signature_def['predict']:
The given SavedModel SignatureDef contains the following input(s):
inputs['Description'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: Description:0
inputs['Headline'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: Headline:0
The given SavedModel SignatureDef contains the following output(s):
outputs['class_ids'] tensor_info:
dtype: DT_INT64
shape: (-1, 1)
name: dnn/head/predictions/ExpandDims:0
outputs['classes'] tensor_info:
dtype: DT_STRING
shape: (-1, 1)
name: dnn/head/predictions/str_classes:0
outputs['logits'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 3)
name: dnn/logits/BiasAdd:0
outputs['probabilities'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 3)
name: dnn/head/predictions/probabilities:0
Method name is: tensorflow/serving/predict
</code></pre>
<p>But now I can't seem to test it.</p>
<pre class="lang-sh prettyprint-override"><code>>>>
saved_model_cli run --dir TEST_SERVING/1553879255/ --tag_set serve
--signature_def predict --input_examples 'inputs=[{"Description":["What is
going on"],"Headline":["Help me"]}]'
Traceback (most recent call last):
...
File "/Users/Josh/miniconda3/envs/python36/lib/python3.6/site-packages/tensorflow/python/tools/saved_model_cli.py", line 489, in _create_example_string
feature_list)
TypeError: 'What is going on' has type str, but expected one of: bytes
</code></pre>
<p>Ok, lets turn it into a bytes object by changing to <code>b["What is
going on"]</code> and <code>b["Help me"]</code>...</p>
<pre class="lang-sh prettyprint-override"><code>ValueError: Type <class
'bytes'> for value b'What is going on' is not supported for
tf.train.Feature.
</code></pre>
<p>Any ideas/thoughts??
Thanks!</p>
- text: >
<p>In tensorflow <code>tf.keras.Model.compile</code>, you can pass a
<code>lambda y_true, y_pred: val</code> function as a metric (though, it
seems not documented), but I asked my self : "How does it aggregate it
over the batches" ?</p>
<p>I searched the documentation, but I've found nowhere how it is done
?</p>
<p>By the way, I don't even know if it is an undefined behavior to do so
and one should instead subclass the Metric class ? ( or at least provide
the required methods).</p>
<p>Also, is it pertinent to pass a loss as a metric (and in this case,
same question : how is it aggregated over the batches ? )</p>
- text: >
<p>I'm working on a project where I have trained a series of binary
classifiers with <strong>Keras</strong>, with <strong>Tensorflow</strong>
as the backend engine. The input data I have is a series of images, where
each binary classifier must make the prediction on the images, later I
save the predictions on a CSV file.</p>
<p>The problem I have is when I get the predictions from the first series
of binary classifiers there isn't any warning, but when the 5th or 6th
binary classifier calls the method <strong>predict</strong> on the input
data I get the following warning:</p>
<blockquote>
<p>WARNING:tensorflow:5 out of the last 5 calls to <function
Model.make_predict_function..predict_function at
0x2b280ff5c158> triggered tf.function retracing. Tracing is expensive
and the excessive number of tracings could be due to (1) creating
@tf.function repeatedly in a loop, (2) passing tensors with different
shapes, (3) passing Python objects instead of tensors. For (1), please
define your @tf.function outside of the loop. For (2), @tf.function
has experimental_relax_shapes=True option that relaxes argument shapes
that can avoid unnecessary retracing. For (3), please refer to
<a
href="https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args"
rel="noreferrer">https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args</a>
and <a href="https://www.tensorflow.org/api_docs/python/tf/function"
rel="noreferrer">https://www.tensorflow.org/api_docs/python/tf/function</a>
for more
details.</p>
</blockquote>
<p>To answer each point in the parenthesis, here are my answers:</p>
<ol>
<li>The <strong>predict</strong> method is called inside a for loop.</li>
<li>I don't pass tensors but a list of <strong>NumPy arrays</strong> of
gray scale images, all of them with the same size in width and height. The
only thing that can change is the batch size because the list can have
only 1 image or more than one.</li>
<li>As I wrote in point 2, I pass a list of NumPy arrays.</li>
</ol>
<p>I have debugged my program and found that this warning always happens
when the method predict is called. To summarize the code I have written is
the following:</p>
<pre><code>import cv2 as cv
import tensorflow as tf
from tensorflow.keras.models import load_model
binary_classifiers = [load_model(path) for path in path2models]
images = [
my_list = list()
for image in images:
image_reworked =
my_list.append(image_reworked)
predictions = [model.predict(x=my_list,verbose=0) for model in
binary_classifiers]
</code></pre>
<h3>What I have tried</h3>
<p>I have defined a function as tf.function and putted the code of the
predictions inside the tf.function like this</p>
<pre><code>@tf.function
def testing(models, faces):
return [model.predict(x=faces,verbose=0) for model in models]
</code></pre>
<p>But I ended up getting the following error:</p>
<blockquote>
<p>RuntimeError: Detected a call to <code>Model.predict</code> inside a
<code>tf.function</code>. Model.predict is a high-level endpoint that
manages
its own <code>tf.function</code>. Please move the call to
<code>Model.predict</code> outside
of all enclosing <code>tf.function</code>s. Note that you can call a
<code>Model</code>
directly on Tensors inside a <code>tf.function</code> like:
<code>model(x)</code>.</p>
</blockquote>
<p>So calling the method <code>predict</code> is basically already a
tf.function. So it's useless to define a tf.function when the warning I
get it's from that method.</p>
<p>I have also checked those other two questions:</p>
<ol>
<li><a
href="https://stackoverflow.com/questions/61647404/tensorflow-2-getting-warningtensorflow9-out-of-the-last-9-calls-to-function">Tensorflow
2: Getting "WARNING:tensorflow:9 out of the last 9 calls to
triggered tf.function retracing. Tracing is expensive"</a></li>
<li><a
href="https://stackoverflow.com/questions/65563185/loading-multiple-saved-tensorflow-keras-models-for-prediction">Loading
multiple saved tensorflow/keras models for prediction</a></li>
</ol>
<p>But neither of the two questions answers my question about how to avoid
this warning. Plus I have also checked the links in the warning message
but I couldn't solve my problem.</p>
<h3>What I want</h3>
<p>I simply want to avoid this warning. While I'm still getting the
predictions from the models I noticed that the python program takes way
too much time on doing predictions for a list of images.</p>
<h3>What I'm using</h3>
<ul>
<li>Python 3.6.13</li>
<li>Tensorflow 2.3.0</li>
</ul>
<h3>Solution</h3>
<p>After some tries to suppress the warning from the <code>predict</code>
method, I have checked the documentation of Tensorflow and in one of the
first tutorials on how to use Tensorflow it is explained that, by default,
Tensorflow is executed in eager mode, which is useful for testing and
debugging the network models. Since I have already tested my models many
times, it was only required to disable the eager mode by writing this
single python line of code:</p>
<p><code>tf.compat.v1.disable_eager_execution()</code></p>
<p>Now the warning doesn't show up anymore.</p>
- text: >
<p>Where one can find the github source code for
<code>tf.quantization.fake_quant_with_min_max_args</code>. Checking the <a
href="https://www.tensorflow.org/api_docs/python/tf/quantization/fake_quant_with_min_max_args"
rel="nofollow noreferrer">TF API documentation</a>, there is no link to
the github source file, and I could not find one on github.</p>
- text: >
<p>I'm trying to use <code>tf.Dataset</code> for a 3D image CNN where the
shape of the 3D image fed into it from the training set and the validation
set are different (training: (64, 64, 64), validation: (176, 176, 160)). I
didn't even know this was possible, but I'm recreating this network based
on a paper, and using the classic <code>feed_dict</code> method the
network indeed works. For performance reasons (and just to learn) I'm
trying to switch the network to use <code>tf.Dataset</code> instead.</p>
<p>I have two datasets and iterators built like the following:</p>
<pre class="lang-py prettyprint-override"><code>def _data_parser(dataset,
shape):
features = {"input": tf.FixedLenFeature((), tf.string),
"label": tf.FixedLenFeature((), tf.string)}
parsed_features = tf.parse_single_example(dataset, features)
image = tf.decode_raw(parsed_features["input"], tf.float32)
image = tf.reshape(image, shape + (1,))
label = tf.decode_raw(parsed_features["label"], tf.float32)
label = tf.reshape(label, shape + (1,))
return image, label
train_datasets = ["train.tfrecord"]
train_dataset = tf.data.TFRecordDataset(train_datasets)
train_dataset = train_dataset.map(lambda x: _data_parser(x, (64, 64, 64)))
train_dataset = train_dataset.batch(batch_size)
train_iterator = train_dataset.make_initializable_iterator()
val_datasets = ["validation.tfrecord"]
val_dataset = tf.data.TFRecordDataset(val_datasets)
val_dataset = val_dataset.map(lambda x: _data_parser(x, (176, 176, 160)))
val_dataset = val_dataset.batch(1)
val_iterator = val_dataset.make_initializable_iterator()
</code></pre>
<p><a
href="https://www.tensorflow.org/guide/datasets#creating_an_iterator"
rel="nofollow noreferrer">TensorFlow documentation</a> has examples
regarding switching between datasets using
<code>reinitializable_iterator</code> or <code>feedable_iterator</code>,
but they all switch between iterators of <strong>same</strong> output
shape, which is not the case here.</p>
<p>How should I switch between training set and validation set using
<code>tf.Dataset</code> and <code>tf.data.Iterator</code> in my case
then?</p>
pipeline_tag: text-classification
inference: true
base_model: sentence-transformers/all-MiniLM-L6-v2
model-index:
- name: SetFit with sentence-transformers/all-MiniLM-L6-v2
results:
- task:
type: text-classification
name: Text Classification
dataset:
name: Unknown
type: unknown
split: test
metrics:
- type: accuracy
value: 0.71
name: Accuracy
- type: precision
value: 0.7100840336134453
name: Precision
- type: recall
value: 0.71
name: Recall
- type: f1
value: 0.70997099709971
name: F1
The model has been trained using an efficient few-shot learning technique that involves:
Then you can load this model and run inference.
