Grokking the AI System Design Interview
Vote

0% completed

​

Batch vs Online Inference

Know the Vocabulary

This lesson uses the terms below many times. Read them once now, and come back here when a later section needs one.

  • Inference: Running a trained model on an input to get a prediction.
  • Batch inference: Running the model ahead of time in a background job, and storing the results. A request then only looks up a stored answer.
  • Online inference: Running the model when a request arrives, while the user waits for the result.
  • Streaming inference: Running the model when an event happens, such as a photo upload

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content