labs/docs/01_inference_test/README.md

Lab 01: Inference Test & Prompting

Overview

This lab is designed to verify the cloud deployment and test basic prompting and text generation capabilities of the deployed model (e.g., Gemma-4-31B).

Setup

Ensure you have spun up the Modular Cloud MAX Engine or a RunPod inference endpoint (using scripts/01_setup_inference.sh and 02_start_server.sh).

Usage

Open 01_inference_test.ipynb in a Jupyter environment. The notebook contains examples of:

1. Connecting to the OpenAI-compatible API endpoint.
2. Constructing basic prompts.
3. Testing generation speed and context limits.

Goals

• Verify API connectivity.
• Understand basic prompting structures.
• Measure inference latency (tokens per second).