BigDL Estimator


Introduction

Analytics Zoo Orca BigDL Estimator provides a set APIs for running BigDL model on Spark in a distributed fashion.


Orca BigDL Estimator

Orca BigDL Estimator is an estimator to do BigDL training/evaluation/prediction on Spark in a distributed fashion.

It can support various data types, like XShards, Spark DataFrame, etc.

It supports BigDL backend in the unified APIs.

Create Estimator from BigDL Model

You can create Orca BigDL Estimator with BigDL model.

from zoo.orca.learn.bigdl import Estimator
Estimator.from_bigdl(*, model, loss=None, optimizer=None, feature_preprocessing=None,
                   label_preprocessing=None, model_dir=None)

Train BigDL model with orca BigDL Estimator

After an Estimator is created, you can call estimator API to train BigDL model:

fit(self, data, epochs, batch_size=32, feature_cols="features", label_cols="label",
    caching_sample=True, validation_data=None, validation_trigger=None,
    validation_metrics=None, checkpoint_trigger=None)

Inference with orca BigDL Estimator

After training or loading trained model, you can call estimator API to inference:

predict(self, data, batch_size=4, feature_cols="features", sample_preprocessing=None)

Evaluate model

After Training, you can call estimator API to evaluate BigDL model:

evaluate(self, data, batch_size=32, feature_cols=None, label_cols=None, validation_metrics=None)

Get model

You can get model using get_model(self)

Save model

You can save model using save(self, model_path) * model_path: (str) Path to model saved folder.

Load model

You can load saved model using

load(self, checkpoint, optimizer=None, loss=None, feature_preprocessing=None,
             label_preprocessing=None, model_dir=None, is_checkpoint=False):

Set TensorBoard & get Training and Validation Summary

During training and validation, Orca BigDL Estimator would save summary data to specified log_dir. This data can be visualized in TensorBoard, or you can use estimator's APIs to retrieve it. You can set the logdir and app name with such API:

set_tensorboard(log_dir, app_name)

This method sets summary information during the training process for visualization purposes. Saved summary can be viewed via TensorBoard. In order to take effect, it needs to be called before fit.

Training summary will be saved to 'log_dir/app_name/train' and validation summary (if any) will be saved to 'log_dir/app_name/validation'.

You can get Training summary with get_train_summary(self, tag=None) and Validation summary with get_validation_summary(self, tag=None).

Clear gradient clipping

You can clear gradient clipping parameters using clear_gradient_clipping(self). In this case, gradient clipping will not be applied. Note: In order to take effect, it needs to be called before fit.

Set constant gradient clipping

You can Set constant gradient clipping during the training process using set_constant_gradient_clipping(self, min, max). * min: The minimum value to clip by. * max: The maximum value to clip by. Note: In order to take effect, it needs to be called before fit.

Set clip gradient to a maximum L2-Norm

You can set clip gradient to a maximum L2-Norm during the training process using set_l2_norm_gradient_clipping(self, clip_norm). * clip_norm: Gradient L2-Norm threshold. Note: In order to take effect, it needs to be called before fit.