Streaming

LabsLLM provides streaming capabilities for real-time response processing from AI models. This feature is particularly useful for long-running requests or when you want to show progress to users as responses are generated.

WARNING

When using the Google provider, streaming cannot be used together with structured output due to API limitations.

Basic Streaming Usage

You can use streaming to process responses as they are generated:

$chat = LabsLLM::text()
    ->using(new Google('YOUR_API_KEY'))
    ->withSystemMessage('You are a history expert professor. You need to create a list of questions with 3 alternatives for each question, where one is correct and two are incorrect. The correct answer should be the first alternative.')
    ->executePromptStream('generate 3 questions');

foreach ($chat as $response) {
    echo $response->response;
}

Chat Streaming

$messages = MessagesBag::create([
    Message::user('Tell me about quantum computing.')
]);

$chat = LabsLLM::text()
    ->using(new OpenAI('YOUR_API_KEY', 'gpt-4o'))
    ->executeChatStream($messages);

foreach ($chat as $response) {
    echo $response->response;
}

How Streaming Works

Streaming in LabsLLM works by:

1. Using executePromptStream() instead of executePrompt() to enable streaming
2. Processing responses in chunks as they arrive from the model
3. Handling each chunk of the response separately

This is particularly useful for:

• Long-form content generation
• Real-time chat applications
• Progress indicators for users
• Processing large responses efficiently

Working with Function Calls in Streams

When using streaming with function calls, you can check for function calls in each chunk:

foreach ($chat as $response) {
    echo $response->response;
    
    if ($response->isCalledFunction('get_current_host')) {
        echo 'get_current_host_called:';
        echo PHP_EOL;
        echo json_encode($response->calledTools);
        echo PHP_EOL;
    }

    if ($response->isCalledFunction('send_user_email')) {
        echo 'send_user_email_called';
        echo PHP_EOL;
        echo json_encode($response->calledTools);
        echo PHP_EOL;
    }
}

Understanding Stream Responses

Each chunk in the stream contains a response object with:

// The current chunk of text from the model
echo $response->response;

// Information about any functions called in this chunk
if (!empty($response->calledTools)) {
    // Handle function calls
}

// Check if a specific function was called
if ($response->isCalledFunction('function_name')) {
    // Handle specific function call
}

For a detailed explanation of the response structure, see the API Response Object guide.

Best Practices

Error Handling

Always implement proper error handling for streams:

try {
    foreach ($chat as $response) {
        echo $response->response;
    }
} catch (Exception $e) {
    // Handle streaming errors
    echo "Error: " . $e->getMessage();
}

Timeout Management

Consider implementing timeouts for long-running streams:

$timeout = 30; // seconds
$startTime = time();

foreach ($chat as $response) {
    if (time() - $startTime > $timeout) {
        throw new Exception('Stream timeout exceeded');
    }
    echo $response->response;
}