• Contents:

Intro to Neural Networks (NNs)

• Inspired by biology
  • Neurons (nodes) in cerebral cortex connected via synapses (axons)
  • When input signals activated enough, neuron triggers/fires to connected neurons
  • Billions of neurons, each with thousands of synapses → many layers → learning mind
  • Cortical columns → neurons in one column process info in parallel
    • GPUs work in a similar way
  • Biology Diagram
  • Deep NN diagram

DL Frameworks

• TensorFlow
  • Open-source, developed by Google
  • Keras: high-level NN API integrated into TensorFlow
    • easy-to-use interface, while TF handles all the computations under the hood
• PyTorch
  • Developed by Facebook
  • Pythonic interface with NumPy-like behavior
• MXNet (”MixNet”)
  • Backed by AWS & Apache MXNet

Types of Neural Networks

1. Feed-forward NN (FFNN or FNN) → uni-directional flow (only forward)
   • 💡 "Vanilla” NN
   • Some subtypes
     • Multilayer perceptron (MLP) → a modern type of FFNN with multiple layers
       • Fully connected → every neuron in one layer connects to every neuron in the next
       • Notable for being able to distinguish data that is not linearly separable
     • Convolutional NN (CNN)
       • Uses convolutional layers instead of fully connected layers to detect patterns
       • Excel at spatial data (usually images) → e.g. “Is there a STOP sign in this image?”
   • Diagram
2. Recurrent NN (RNN) → flow can go backwards (loops)
   • Useful for sequences or time-series
     • e.g. Predict stock prices, understand words in a sentence, translation…
   • Diagram

Activation Functions

Ref: https://www.udemy.com/course/aws-certified-machine-learning-engineer-associate-mla-c01/learn/lecture/45285617

• 🔧 Math function inside the node of a NN
  • Given input (signals), defines the output of the node
  • Diagram

Graphs for the different activation functions

Impractical but theoretical functions

1. Linear activation function: literally propagates same input to next layer with a linear factor
   • Simple, doesn't “do” anything complex
   • Usually used at the output layer of a NN