ML Inference - basic concept and types
- 🔧 Inference = trained ML model makes prediction (output) on new data
Real-Time Inferencing
- Computers have to make decisions quickly as data arrives
- Speed (low latency) is preferred over perfect accuracy → synchronous results
- Example: chatbots
- Diagram
Batch Inferencing
- Large amount of data that is analyzed all at once
- Speed of the results usually not a concern (high latency), but accuracy is → asynchronous results
- Often used for data analysis
- Diagram
Inferencing at the Edge
- Edge device: a device with generally less computing power, close to where data is generated, where internet connections can be limited
- Small Language Model (SLM) on an edge device
- Very low latency, low compute footprint
- Offline capability, local inference
- Example: SLM on a RaspberryPi