Streaming from Microphones

Allowing your user speak to your app 🗣️

VideoKit makes it very easy to discover available audio input devices and stream the microphone audio. Below, we will explore doing so with and without code.

Using the Audio Manager Component

VideoKit provides the VideoKitAudioManager component to simplify working with audio devices in Unity scenes.

Adding the Component

The first step is to add a VideoKitAudioManager component in your scene.

[GIF here]

Creating an Audio Handler

Next, let's create an AudioHandler component which uses the audio manager to stream audio buffers from an AudioDevice:

using UnityEngine;
using VideoKit;

public class AudioHandler : MonoBehaviour {

    public VideoKitAudioManager audioManager;

    private void Start () {
        // Register a handler for audio buffers
        audioManager.OnAudioBuffer += OnAudioBuffer;
        // Start streaming
        audioManager.StartRunning();
    }

    private void OnAudioBuffer (AudioBuffer audioBuffer) {
        // Use the audio data
        Debug.Log($"Received {audioBuffer.sampleBuffer.Length} audio samples");
    }
}

Putting it Together

INCOMPLETE

Using the VideoKit API

Audio streaming is exposed with the AudioDevice class.

Discovering Audio Devices

Because audio devices access highly sensitive user data, the very first step is to request microphone permissions from the user. We provide the AudioDevice.CheckPermissions method for this:

// Request microphone permisisons from the user
PermissionStatus status = await AudioDevice.CheckPermissions(request: true);
// Check that the user granted permissions
if (status != PermissionStatus.Granted)
    return;

Once microphone permisisons have been granted, we can now discover available microphones using the AudioDevice.Discover method:

// Discover available audio devices
AudioDevice[] audioDevices = await AudioDevice.Discover();
// Use the first audio device
AudioDevice audioDevice = audioDevices[0];

Specifying the Audio Format

Audio devices allow for requesting a specific audio format, which comprises of the audio sample rate and channel count. We recommend specifying both of these before streaming audio:

// Set the audio sample rate
audioDevice.sampleRate = 16_000;
// Set the audio channel count
audioDevice.channelCount = 1

Streaming Audio Buffers

To begin streaming pixel buffers from the audio device, use the AudioDevice.StartRunning method:

// Start streaming audio buffers from the audio device
audioDevice.StartRunning(OnAudioBuffer);

You must provide a callback that will be invoked with audio buffers as they become available:

void OnAudioBuffer (AudioBuffer audioBuffer) {
    // ...
}