11. Activations

Neural network activation functions are the primary logic that dictates whether a given neuron should be activated or not, and by how much. Activation functions are best when they introduce some non-linearity to the neuron so that it can better model more complex behaviors.

In our case all activations are children of the Node abstraction to keep things consistent, and reduce the amount of redundant code.

Activations Status

Category	Name	Docs	Forward	Backward
activation	Argmax	Argmax
activation	Linear	Linear
activation	ReLU (Approx)	R_a(x)
activation	Sigmoid (Approx)	\sigma_a(x)
activation	Softmax	Softmax

Activation Functions

• 11.1. Argmax
  • 11.1.1. Argmax API
• 11.2. Linear
  • 11.2.1. Linear API
• 11.3. ReLU & Approximation
  • 11.3.1. ReLU \(R(x)\)
    • 11.3.1.1. \(R(x)\)
    • 11.3.1.2. \(\frac{dR(x)}{dx}\)
  • 11.3.2. ReLU-Approximation \(R_a(x)\)
    • 11.3.2.1. \(R_a(x)\)
    • 11.3.2.2. \(\frac{dR_a(x)}{dx}\)
  • 11.3.3. ReLU Approximate API
• 11.4. Sigmoid & Approximation
  • 11.4.1. Sigmoid \(\sigma(x)\)
    • 11.4.1.1. \(\sigma(x)\)
    • 11.4.1.2. \(\frac{d\sigma(x)}{dx}\)
  • 11.4.2. Sigmoid-Approximation \(\sigma_a(x)\)
    • 11.4.2.1. \(\sigma_a(x)\)
    • 11.4.2.2. \(\frac{d\sigma_a(x)}{dx}\)
  • 11.4.3. Sigmoid API
• 11.5. Softmax
  • 11.5.1. Example Architecture
    • 11.5.1.1. API