• Contents:

Intro to Neural Networks (NNs)

• Inspired by biology
  • Neurons (nodes) in cerebral cortex connected via synapses (axons)
  • When input signals activated enough, neuron triggers/fires to connected neurons
  • Billions of neurons, each with thousands of synapses → many layers → learning mind
  • Cortical columns → neurons in one column process info in parallel
    • GPUs work in a similar way
  • Biology Diagram
  • Deep NN diagram

DL Frameworks

• TensorFlow
  • Open-source, developed by Google
  • Keras: high-level NN API integrated into TensorFlow
    • easy-to-use interface, while TF handles all the computations under the hood
• MixNet (MXNet)
  • Backed by AWS & Apache MXNet

Types of Neural Networks

1. Feed-forward NN (FNN) → uni-directional flow (only forward)
   • 💡 "Vanilla” NN
     • a multilayer perceptron (MLP) is a name for a modern feedforward neural network consisting of fully connected neurons with nonlinear activation functions, organized in layers, notable for being able to distinguish data that is not linearly separable.[1]
     • Modern neural networks are trained using backpropagation[2][3][4][5][6] and are colloquially referred to as "vanilla" networks.
   • Convolutional NN (CNN) → sub-type of FNN, useful for spatial data (usually images)
     • e.g. Is there a STOP sign in this image?
   • Diagram
2. Recurrent NN (RNN) → flow can go backwards (loops)
   • Useful for sequences or time-series
     • e.g. Predict stock prices, understand words in a sentence, translation…
   • Diagram

Activation Functions

Ref: https://www.udemy.com/course/aws-certified-machine-learning-engineer-associate-mla-c01/learn/lecture/45285617

• 🔧 Math function inside the node of a NN
  • Given input (signals), defines the output of the node
  • Diagram

Graphs for the different activation functions

Impractical but theoretical functions

1. Linear activation function: literally propagates same input to next layer with a linear factor
   • Simple, doesn't “do” anything complex
   • Usually used at the output layer of a NN
2. Binary step function: ON/OFF
   • Can't handle multi-classification (just binary)
   • Vertical slopes don't work well with calculus (df(x)/x at x=0 is infinity) → very unstable