TFOptimizer

TFOptimizer is used for optimizing a TensorFlow model with respect to its training variables on Spark/BigDL.

Remarks:

You need to install tensorflow==1.15.0 on your driver node.
Your operating system (OS) is required to be one of the following 64-bit systems: Ubuntu 16.04 or later and macOS 10.12.6 or later.
To run on other systems, you need to manually compile the TensorFlow source code. Instructions can be found here.

Create a TFOptimizer:

import tensorflow as tf
from zoo.tfpark import TFOptimizer
from bigdl.optim.optimizer import *
loss = ...
optimizer = TFOptimizer.from_loss(loss, Adam(1e-3))
optimizer.optimize(end_trigger=MaxEpoch(5))

For Keras model:

from zoo.tfpark import TFOptimizer
from bigdl.optim.optimizer import *
from tensorflow.keras.models import Model

model = Model(inputs=..., outputs=...)

model.compile(optimizer='rmsprop',
            loss='sparse_categorical_crossentropy',
            metrics=['accuracy'])

optimizer = TFOptimizer.from_keras(model, dataset)
optimizer.optimize(end_trigger=MaxEpoch(5))

Methods

from_loss (factory method)

Create a TFOptimizer from a TensorFlow loss tensor. The loss tensor must come from a TensorFlow graph that only takes TFDataset.tensors and the tensors in tensor_with_value as inputs.

from_loss(loss, optim_method, session=None, val_outputs=None,
                  val_labels=None, val_method=None,
                  clip_norm=None, clip_value=None, metrics=None,
                  tensor_with_value=None, **kwargs)

Arguments

loss: The loss tensor of the TensorFlow model, should be a scalar. The loss tensor must come from a TensorFlow graph that only takes TFDataset.tensors and the tensors in tensor_with_value as inputs.
optim_method: the optimization method to be used, such as bigdl.optim.optimizer.Adam
session: the current TensorFlow Session, if you want to used a pre-trained model, you should use the Session to load the pre-trained variables and pass it to TFOptimizer.
val_outputs: the validation output TensorFlow tensor to be used by val_methods
val_labels: the validation label TensorFlow tensor to be used by val_methods
val_method: the BigDL val_method(s) to be used.
clip_norm: float >= 0. Gradients will be clipped when their L2 norm exceeds this value.
clip_value: float >= 0. Gradients will be clipped when their absolute value exceeds this value.
metrics: a dictionary. The key should be a string representing the metric's name and the value should be the corresponding TensorFlow tensor, which should be a scalar.
tensor_with_value: a dictionary. The key is TensorFlow tensor, usually a placeholder, the value of the dictionary is a tuple of two elements. The first one of the tuple is the value to feed to the tensor in training phase and the second one is the value to feed to the tensor in validation phase.

from_keras (factory method)

Create a TFOptimizer from a tensorflow.keras model. The model must be compiled.

from_keras(keras_model, dataset, optim_method=None, **kwargs)

Arguments

keras_model: the tensorflow.keras model, which must be compiled.
dataset: a TFDataset
optim_method: the optimization method to be used, such as bigdl.optim.optimizer.Adam

set_train_summary

set_train_summary(summary)

Arguments

summary: The train summary to be set. A TrainSummary object contains information necessary for the optimizer to know how often the logs are recorded, where to store the logs and how to retrieve them, etc. For details, refer to the docs of TrainSummary.

set_val_summary

set_val_summary(summary)

Arguments

summary: The validation summary to be set. A ValidationSummary object contains information necessary for the optimizer to know how often the logs are recorded, where to store the logs and how to retrieve them, etc. For details, refer to the docs of ValidationSummary.

set_constant_gradient_clipping

set_constant_gradient_clipping(min_value, max_value)

Arguments

min_value: the minimum value to clip by
max_value: the maxmimum value to clip by

set_gradient_clipping_by_l2_norm

set_gradient_clipping_by_l2_norm(self, clip_norm)

Arguments

clip_norm: gradient L2-Norm threshold

optimize

optimize(self, end_trigger=None)

Arguments

end_trigger: BigDL's Trigger to indicate when to stop the training. If none, defaults to train for one epoch.