Advanced Activations


PReLU

Applies parametric ReLU, where parameter varies the slope of the negative part.

It follows: f(x) = max(0, x) + a * min(0, x)

Scala:

PReLU(nOutputPlane = 0, inputShape = null)

Python:

PReLU(nOutputPlane=0, input_shape=None)

Parameters:

Scala example:

import com.intel.analytics.zoo.pipeline.api.keras.layers.PReLU
import com.intel.analytics.zoo.pipeline.api.keras.models.Sequential
import com.intel.analytics.bigdl.utils.Shape
import com.intel.analytics.bigdl.tensor.Tensor

val model = Sequential[Float]()
model.add(PReLU[Float](inputShape = Shape(3)))
val input = Tensor[Float](2, 3).randn()
val output = model.forward(input)

Input is:

input: com.intel.analytics.bigdl.tensor.Tensor[Float] =
-0.9026888      -1.0402212      1.3878769
-0.17167428     0.08202032      1.2682742
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]

Output is:

output: com.intel.analytics.bigdl.nn.abstractnn.Activity =
-0.2256722      -0.2600553      1.3878769
-0.04291857     0.08202032      1.2682742
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]

Python example:

import numpy as np
from zoo.pipeline.api.keras.models import Sequential
from zoo.pipeline.api.keras.layers import PReLU

model = Sequential()
model.add(PReLU(input_shape=(3, )))
input = np.random.random([2, 3])
output = model.forward(input)

Input is:

[[0.61639702 0.08877075 0.93652509]
 [0.38800821 0.76286851 0.95777973]]

Output is

[[0.616397   0.08877075 0.9365251 ]
 [0.3880082  0.7628685  0.9577797 ]]

ELU

Exponential Linear Unit.

It follows: f(x) = alpha * (exp(x) - 1.) for x < 0, f(x) = x for x >= 0.

Scala:

ELU(alpha = 1.0, inputShape = null)

Python:

ELU(alpha=1.0, input_shape=None, name=None)

Parameters:

Scala example:

import com.intel.analytics.zoo.pipeline.api.keras.layers.ELU
import com.intel.analytics.zoo.pipeline.api.keras.models.Sequential
import com.intel.analytics.bigdl.utils.Shape
import com.intel.analytics.bigdl.tensor.Tensor

val model = Sequential[Float]()
model.add(ELU[Float](inputShape = Shape(3)))
val input = Tensor[Float](2, 3).randn()
val output = model.forward(input)

Input is:

input: com.intel.analytics.bigdl.tensor.Tensor[Float] =
-0.13405465 0.05160992  -1.4711418
1.5808829   -1.3145303  0.6709266
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]

Output is:

output: com.intel.analytics.bigdl.nn.abstractnn.Activity =
-0.1254577  0.05160992  -0.77033687
1.5808829   -0.73139954 0.6709266
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]

Python example:

import numpy as np
from zoo.pipeline.api.keras.layers import ELU
from zoo.pipeline.api.keras.models import Sequential

model = Sequential()
model.add(ELU(input_shape=(3, )))
input = np.random.random([2, 3])
output = model.forward(input)

Input is:

[[0.90404922 0.23530925 0.49711093]
 [0.43009161 0.22446032 0.90144771]]

Output is

[[0.9040492  0.23530924 0.49711093]
 [0.43009162 0.22446032 0.9014477 ]]

SReLU

S-shaped Rectified Linear Unit.

It follows: f(x) = t^r + a^r(x - t^r) for x >= t^r, f(x) = x for t^r > x > t^l, f(x) = t^l + a^l(x - t^l) for x <= t^l.

Scala:

SReLU(tLeftInit = "zero", aLeftInit = "glorot_uniform", tRightInit = "glorot_uniform", aRightInit = "one", sharedAxes = null, inputShape = null)

Python:

SReLU(t_left_init="zero", a_left_init="glorot_uniform", t_right_init="glorot_uniform", a_right_init="one", shared_axes=None, input_shape=None, name=None)

Parameters:

Scala example:

import com.intel.analytics.zoo.pipeline.api.keras.layers.SReLU
import com.intel.analytics.zoo.pipeline.api.keras.models.Sequential
import com.intel.analytics.bigdl.utils.Shape
import com.intel.analytics.bigdl.tensor.Tensor

val model = Sequential[Float]()
model.add(SReLU[Float](inputShape = Shape(2, 3)))
val input = Tensor[Float](2, 2, 3).randn()
val output = model.forward(input)

Input is:

input: com.intel.analytics.bigdl.tensor.Tensor[Float] =
(1,.,.) =
0.5599429   0.22811626  -0.027771426
-0.56582874 1.9261217   1.2686813

(2,.,.) =
0.7538568   0.8725621   0.19803657
0.49057     0.0537252   0.8684544

[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x2x3]

Output is:

output: com.intel.analytics.bigdl.nn.abstractnn.Activity =
(1,.,.) =
0.5599429   0.22811626  -0.009864618
0.07011698  1.9261217   1.2686813

(2,.,.) =
0.7538568   0.87256205  0.19803657
0.49057     0.0537252   0.8684544

[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x2x3]

Python example:

import numpy as np
from zoo.pipeline.api.keras.layers import SReLU
from zoo.pipeline.api.keras.models import Sequential

model = Sequential()
model.add(SReLU(input_shape=(2, 3)))
input = np.random.random([2, 2, 3])
output = model.forward(input)

Input is:

[[[0.42998132 0.47736492 0.9554154 ]
  [0.93264942 0.56851545 0.39508313]]

 [[0.5164102  0.22304862 0.44380779]
  [0.69137804 0.26413953 0.60638032]]]

Output is

[[[0.42998132 0.47736493 0.9554154 ]
  [0.93264943 0.5685154  0.39508313]]

 [[0.5164102  0.22304863 0.44380778]
  [0.69137806 0.26413953 0.60638034]]]

ThresholdedReLU

Thresholded Rectified Linear Unit.

It follows: f(x) = x for x > theta, f(x) = 0 otherwise.

Scala:

ThresholdedReLU(theta = 1.0, inputShape = null)

Python:

ThresholdedReLU(theta=1.0, input_shape=None, name=None)

Parameters:

Scala example:

import com.intel.analytics.zoo.pipeline.api.keras.layers.ThresholdedReLU
import com.intel.analytics.zoo.pipeline.api.keras.models.Sequential
import com.intel.analytics.bigdl.utils.Shape
import com.intel.analytics.bigdl.tensor.Tensor

val model = Sequential[Float]()
model.add(ThresholdedReLU[Float](inputShape = Shape(3)))
val input = Tensor[Float](2, 3).randn()
val output = model.forward(input)

Input is:

input: com.intel.analytics.bigdl.tensor.Tensor[Float] =
2.220999    1.2022058   -1.0015608
0.6532913   0.31831574  1.6747104
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]

Output is:

output: com.intel.analytics.bigdl.nn.abstractnn.Activity =
2.220999    1.2022058   0.0
0.0         0.0         1.6747104
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]

Python example:

import numpy as np
from zoo.pipeline.api.keras.layers import ThresholdedReLU
from zoo.pipeline.api.keras.models import Sequential

model = Sequential()
model.add(ThresholdedReLU(input_shape=(3, )))
input = np.random.random([2, 3])
output = model.forward(input)

Input is:

[[0.91854565 0.58317415 0.33089385]
 [0.82472184 0.70572913 0.32803604]]

Output is

[[0.0   0.0   0.0]
 [0.0   0.0   0.0]]