Video Streaming

The video module provides both live video streaming from robot cameras and video playback of recorded files.

How Video Works: SDK Playground vs Standalone

Video streaming requires special handling because it involves WebRTC connections and displaying video in a browser - which behaves differently depending on where your code runs.

SDK Playground Architecture: In the SDK Playground, your code executes on the appliance (robot-side), but video needs to display in your browser. This creates an interesting challenge: how can robot-side code show video in the browser?

The system solves this with a remote video bridge:

When your appliance-side code calls oloClient.video.startVideo(), the appliance doesn't actually start video (it has no display)
Instead, it sends a command back to your browser via WebSocket
The browser receives this command and starts the actual WebRTC video stream
The video displays in the Playground's video panel

This is why videoElement is "automatically provided" in the Playground - it refers to the browser's video element, and the bridge handles connecting your robot-side video calls to the browser-side display.

Standalone Architecture: In standalone applications, your code runs in your own environment (browser, desktop app, etc.). Video calls execute directly where your code runs:

In a browser-based standalone app: WebRTC connects directly to the appliance and displays in your HTML video element
The videoElement parameter is the actual DOM element you provide

Practical Implications:

SDK Playground: Video "just works" - use videoElement as shown in the examples. The system handles the browser/appliance bridge automatically.
Standalone Browser Apps: Create an HTML <video> element and pass it to startVideo(). The WebRTC connection runs in your browser.
Standalone Python Apps: The OLO Python SDK includes full WebRTC video support. Install with pip install olo-client[video] to enable video streaming. Instead of a video element, you provide an on_frame callback that receives video frames as numpy arrays (BGR format). You can display these using OpenCV, tkinter, or any other Python GUI framework. See the Python GUI Test Client for a complete working example.

Live Video Streaming

`startVideo(topic, videoElement, options)`

Start live video streaming from a ROS camera topic.

Parameters:

topic (string) - Video topic name (e.g., '/camera/image_raw/compressed')
videoElement (HTMLVideoElement) - HTML video element to display the stream
options (object, optional) - Video options
- onConnectionStateChange (function) - Callback for connection state changes
- onError (function) - Callback for errors
- onProgress (function) - Callback for connection progress updates

Returns: Promise<string> - Session ID for this video stream

// In SDK Playground, videoElement is automatically provided
// In standalone apps, get the video element from your HTML: document.getElementById('video')
const session = await oloClient.video.startVideo('/camera/image_raw/compressed', videoElement, {
    onConnectionStateChange: (state) => {
        console.log('Video state:', state);
    },
    onError: (error) => {
        console.error('Video error:', error);
    },
    onProgress: (step) => {
        console.log('Video progress:', step);
    }
});
console.log('Video session started:', session);

`stopVideo(sessionId)`

Stop a live video stream.

Parameters:

sessionId (string) - Session ID returned from startVideo()

`stopAllVideos()`

Stop all active live video streams.

Video Playback

The video playback system allows you to play back previously recorded videos.

`startVideoPlayback(filename, videoElement, options)`

Start video playback of a recorded file.

Parameters:

filename (string) - Name of the recorded video file to play (from oloClient.videoRecording.getRecordedFiles())
videoElement (HTMLVideoElement) - HTML video element to display the playback
options (object, optional) - Playback options
- onConnectionStateChange (function) - Callback for connection state changes
- onError (function) - Callback for errors
- onProgress (function) - Callback for connection progress updates
- onPlaybackStateChange (function) - Callback for playback state changes (playing, paused, ended, ready)
- loop (boolean) - Whether to loop the video playback (default: false)
- playbackSpeed (number) - Playback speed multiplier (default: 1.0, supports 0.25x to 4x)

Returns: Promise<string> - Session ID for this video playback stream

// In SDK Playground, videoElement is automatically provided
// In standalone apps: const videoElement = document.getElementById('video');
const files = await oloClient.videoRecording.getRecordedFiles();
const latestFile = files[0]; // Most recent recording

const sessionId = await oloClient.video.startVideoPlayback(latestFile.filename, videoElement, {
    loop: true,
    playbackSpeed: 1.0,
    onConnectionStateChange: (state) => {
        console.log('Playback connection state:', state);
    },
    onPlaybackStateChange: (state) => {
        console.log('Playback state:', state);
        if (state === 'ready') {
            console.log('Video is ready to play');
        } else if (state === 'ended') {
            console.log('Video playback finished');
        }
    },
    onError: (error) => {
        console.error('Playback error:', error);
    }
});
console.log('Video playback started:', sessionId);

`controlVideoPlayback(sessionId, action, params)`

Control video playback with various actions.

Parameters:

sessionId (string) - Playback session ID returned from startVideoPlayback()
action (string) - Control action: 'play', 'pause', 'seek', 'speed', 'stop'
params (object, optional) - Action parameters
- For seek: { position: seconds } - Jump to specific time in video
- For speed: { speed: multiplier } - Change playback speed (0.25x to 4x)

Returns: void

// Pause playback
oloClient.video.controlVideoPlayback(sessionId, 'pause');

// Resume playback
oloClient.video.controlVideoPlayback(sessionId, 'play');

// Seek to 30 seconds
oloClient.video.controlVideoPlayback(sessionId, 'seek', { position: 30 });

// Change speed to 2x
oloClient.video.controlVideoPlayback(sessionId, 'speed', { speed: 2.0 });

// Slow motion (0.5x speed)
oloClient.video.controlVideoPlayback(sessionId, 'speed', { speed: 0.5 });

// Stop playback completely
oloClient.video.controlVideoPlayback(sessionId, 'stop');

`stopVideoPlayback(sessionId)`

Stop video playback and clean up resources.

Parameters:

sessionId (string) - Playback session ID returned from startVideoPlayback()

// Stop specific playback session
oloClient.video.stopVideoPlayback(sessionId);

Complete Video Playback Workflow

// In SDK Playground, videoElement is automatically provided
// In standalone apps: const videoElement = document.getElementById('video');
try {
    // Step 1: Get available recorded files
    const files = await oloClient.videoRecording.getRecordedFiles();
    
    if (files.length === 0) {
        console.log('No recorded files available. Record some videos first!');
    } else {
        console.log(`Found ${files.length} recorded files:`);
        files.slice(0, 5).forEach(file => {
            const date = new Date(file.created).toLocaleString();
            console.log(`  ${file.filename} (${file.sizeFormatted}) - ${date}`);
        });
        
        // Step 2: Select and play the most recent file
        const selectedFile = files[0]; // Most recent
        
        console.log(`Playing: ${selectedFile.filename}`);
        
        const sessionId = await oloClient.video.startVideoPlayback(selectedFile.filename, videoElement, {
            loop: false,
            playbackSpeed: 1.0,
            onConnectionStateChange: (state) => {
                console.log(`Connection state: ${state}`);
            },
            onPlaybackStateChange: (state) => {
                console.log(`Playback state: ${state}`);
                
                if (state === 'ready') {
                    console.log('✅ Video is ready to play');
                } else if (state === 'ended') {
                    console.log('🏁 Video playback finished');
                }
            },
            onError: (error) => {
                console.error('Playback error:', error);
            }
        });
        
        console.log(`Playback session started: ${sessionId}`);
        
        // Step 3: Demonstrate playback controls after a delay
        setTimeout(() => {
            console.log('🎮 Demonstrating playback controls...');
            
            // Pause for 2 seconds
            oloClient.video.controlVideoPlayback(sessionId, 'pause');
            console.log('⏸️ Paused playback');
            
            setTimeout(() => {
                // Resume at 2x speed
                oloClient.video.controlVideoPlayback(sessionId, 'play');
                oloClient.video.controlVideoPlayback(sessionId, 'speed', { speed: 2.0 });
                console.log('⏩ Resumed at 2x speed');
                
                setTimeout(() => {
                    // Seek to beginning and play at normal speed
                    oloClient.video.controlVideoPlayback(sessionId, 'seek', { position: 0 });
                    oloClient.video.controlVideoPlayback(sessionId, 'speed', { speed: 1.0 });
                    console.log('⏮️ Seeked to beginning, normal speed');
                }, 3000);
            }, 2000);
        }, 5000);
        
        // Step 4: Stop playback after demo
        setTimeout(() => {
            oloClient.video.stopVideoPlayback(sessionId);
            console.log('🛑 Stopped video playback');
        }, 15000);
    }
} catch (error) {
    console.error('Video playback demo failed:', error);
}

`detectVideoTopics()`

Auto-detect video topics.

Returns: Promise<Object> - Object with bestTopic and videoTopics array

const { bestTopic, videoTopics } = await oloClient.video.detectVideoTopics();
console.log('Best video topic:', bestTopic);
console.log('All video topics:', videoTopics);

`getActiveVideoSessions()`

Get list of active video session IDs.

Returns: Array - Array of active video session IDs

`getVideoSessionIdForElement(videoElement)`

Get the session ID for a specific video element (used for vision analysis).

Parameters:

videoElement (HTMLVideoElement) - Video element to get session ID for

Returns: string|null - Session ID or null if not found

// In SDK Playground, videoElement is automatically provided
// In standalone apps: const videoElement = document.getElementById('video');
const sessionId = oloClient.video.getVideoSessionIdForElement(videoElement);
console.log('Video session ID:', sessionId);

On This Page

Getting Started

API Reference

Standalone Client Development

Video Streaming

How Video Works: SDK Playground vs Standalone

Live Video Streaming

`startVideo(topic, videoElement, options)`

`stopVideo(sessionId)`

`stopAllVideos()`

Video Playback

`startVideoPlayback(filename, videoElement, options)`

`controlVideoPlayback(sessionId, action, params)`

`stopVideoPlayback(sessionId)`

Complete Video Playback Workflow

`detectVideoTopics()`

`getActiveVideoSessions()`

`getVideoSessionIdForElement(videoElement)`