在本文中我們將研究以下主題

1. navigator.mediaDevices.getUserMedia 瀏覽器 Api
2. Google語(yǔ)音到文字 API

我們將從建立 React hook 開(kāi)始，它將執(zhí)行所有操作，例如 startRecording、stopRecording、建立 Audio Blob、錯(cuò)誤處理等。

在我們進(jìn)入正題之前，還有一些其他事情要注意

1. 最小分貝，高於該分貝我們將考慮將對(duì)話作為輸入，例如-35db（只是一個(gè)隨機(jī)數(shù)）
2. 表示使用者已停止輸入的暫停時(shí)間應(yīng)該是多長(zhǎng)，例如 2000 毫秒

const VOICE_MIN_DECIBELS = -35
const DELAY_BETWEEN_DIALOGUE = 2000

讓我們將我們的鉤子命名為useAudioInput.ts，我們將使用瀏覽器api，如navigator.mediaDevices.getUserMedia、MediaRecorder和AudioContext。 AudioContext 將幫助我們識(shí)別輸入音訊是否高於被視為輸入所需的最小分貝，因此我們將從以下變數(shù)和道具開(kāi)始

const defaultConfig = {
    audio: true
};

type Payload = Blob;

type Config = {
    audio: boolean;
    timeSlice?: number
    timeInMillisToStopRecording?: number
    onStop: () => void;
    onDataReceived: (payload: Payload) => void
};

export const useAudioInput = (config: Config = defaultConfig) => {
    const mediaChunks = useRef<Blob[]>([]);
    const [isRecording, setIsRecording] = useState(false);
    const mediaRecorder = useRef<MediaRecorder | null>(null);
    const [error, setError] = useState<Error| null>(null);
    let requestId: number;
    let timer: ReturnType<typeof setTimeout>;

    const createBlob = () => {
      const [chunk] = mediaChunks.current;
      const blobProperty = { type: chunk.type };
      return new Blob(mediaChunks.current, blobProperty)
    }
  ...
}

在上面的程式碼中，我們將使用 mediaChunks 作為變數(shù)來(lái)保存輸入 blob 和 mediaRecorder 來(lái)?yè)碛行?MediaRecorder 的實(shí)例，該實(shí)例將流作為來(lái)自 navigator.mediaDevices.getUserMedia 的輸入。接下來(lái)讓我們處理 getUserMedia 不可用的情況

...
useEffect(() => {
        if(!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
            const notAvailable = new Error('Your browser does not support Audio Input')
            setError(notAvailable)
        }

    },[]);
...

我們將開(kāi)始編寫鉤子的實(shí)際功能，其中包括各種函數(shù)，如 setupMediaRecorder、setupAudioContext、onRecordingStart、onRecordingActive、startRecording、stopRecording 等。

const onRecordingStart = () => mediaChunks.current = [];

const onRecordingActive = useCallback(({data}: BlobEvent) => {
        if(data) {
            mediaChunks.current.push(data);
            config?.onDataReceived?.(createBlob())
        }
    },[config]);

const startTimer = () => {
        timer = setTimeout(() => {
            stopRecording();
        }, config.timeInMillisToStopRecording)
    };

const setupMediaRecorder = ({stream}:{stream: MediaStream}) => {
        mediaRecorder.current = new MediaRecorder(stream)
        mediaRecorder.current.ondataavailable = onRecordingActive
        mediaRecorder.current.onstop = onRecordingStop
        mediaRecorder.current.onstart = onRecordingStart
        mediaRecorder.current.start(config.timeSlice)

    };

 const setupAudioContext = ({stream}:{stream: MediaStream}) => {
        const audioContext = new AudioContext();
        const audioStreamSource = audioContext.createMediaStreamSource(stream);
        const analyser = audioContext.createAnalyser();

        analyser.minDecibels = VOICE_MIN_DECIBELS;

        audioStreamSource.connect(analyser);
        const bufferLength = analyser.frequencyBinCount;
        const domainData = new Uint8Array(bufferLength)

        return {
            domainData,
            bufferLength,
            analyser
        }
    };

const startRecording = async () => {
        setIsRecording(true);

        await navigator.mediaDevices
            .getUserMedia({
                audio: config.audio
            })
            .then((stream) => {
                setupMediaRecorder({stream});
                if(config.timeSlice) {
                    const { domainData, analyser, bufferLength } = setupAudioContext({ stream });
                    startTimer()
                }
            })
            .catch(e => {
                setError(e);
                setIsRecording(false)
            })
    };



    const stopRecording = () => {
        mediaRecorder.current?.stop();

        clearTimeout(timer);
        window.cancelAnimationFrame(requestId);

        setIsRecording(false);
        onRecordingStop()
    };

    const createBlob = () => {
        const [chunk] = mediaChunks.current;
        const blobProperty = { type: chunk.type };
        return new Blob(mediaChunks.current, blobProperty)
    }

    const onRecordingStop = () => config?.onStop?.();

透過(guò)上面的程式碼我們已經(jīng)差不多完成了鉤子，唯一懸而未決的事情是識(shí)別用戶是否已經(jīng)停止說(shuō)話，如果2 沒(méi)有輸入，我們將使用DELAY_BETWEEN_DIALOGUE 作為我們要等待的時(shí)間秒後，我們將假設(shè)用戶已停止講話並將點(diǎn)擊語(yǔ)音轉(zhuǎn)文字端點(diǎn)。

...
const detectSound = ({ 
        recording,
        analyser,
        bufferLength,
        domainData
    }: {
        recording: boolean
        analyser: AnalyserNode
        bufferLength: number
        domainData: Uint8Array
    }) => {
        let lastDetectedTime = performance.now();
        let anySoundDetected = false;

        const compute = () => {
            if (!recording) {
                return;
            }

            const currentTime = performance.now();

            const timeBetweenTwoDialog =
                anySoundDetected === true && currentTime - lastDetectedTime > DELAY_BETWEEN_DIALOGUE;

            if (timeBetweenTwoDialog) {
                stopRecording();

                return;
            }

            analyser.getByteFrequencyData(domainData);

            for (let i = 0; i < bufferLength; i += 1) {
                if (domainData[i] > 0) {
                    anySoundDetected = true;
                    lastDetectedTime = performance.now();
                }
            }

            requestId = window.requestAnimationFrame(compute);
        };

        compute();

    }
...

const startRecording = async () => {
 ... 
  detectSound()
 ... 
}

在上面的程式碼中，我們使用 requestAnimationFrame 來(lái)偵測(cè)使用者音訊輸入，這樣我們就完成了鉤子，現(xiàn)在可以開(kāi)始在各個(gè)地方使用鉤子了。

例如

  const onDataReceived = async (data: BodyInit) => {
    const rawResponse = await fetch('https://backend-endpoint', {
      method: 'POST',
      body: data
    });
    const response = await rawResponse.json();

    setText(response)
  };

  const { isRecording, startRecording, error } = useAudioInput({
    audio: true,
    timeInMillisToStopRecording: 2000,
    timeSlice: 400,
    onDataReceived
  })

第二部分是連接一個(gè)節(jié)點(diǎn)伺服器，它可以與Google語(yǔ)音到文字 API 進(jìn)行通信，我已經(jīng)附上了我在創(chuàng)建節(jié)點(diǎn)方面時(shí)引用的文檔。
https://codelabs.developers.google.com/codelabs/cloud-speech-text-node。

// demo node server which connects with google speech to text api endpoint

const express = require('express');
const cors = require('cors');

const speech = require('@google-cloud/speech');

const client = new speech.SpeechClient();

async function convert(audioBlob) {
  const request = {
    config: {
      encoding: 'WEBM_OPUS', // Ensure this matches the format of the audio being sent
      sampleRateHertz: 48000, // This should match the sample rate of your recording
      languageCode: 'en-US'
    },
    audio: {
      content: audioBlob
    }
  };

  const [response] = await client.recognize(request);

  const transcription = response.results
    .map(result => result.alternatives[0].transcript)
    .join('\n');
  return transcription;
}

const app = express();

app.use(cors())
app.use(express.json());

app.post('/upload', express.raw({ type: '*/*' }), async (req, res) => {
    const audioBlob = req.body;

    const response = await convert(audioBlob);

    res.json(response);
});

app.listen(4000,'0.0.0.0', () => {
  console.log('Example app listening on port 4000!');
});

在本文中，我介紹瞭如何將音訊內(nèi)容或 blob 發(fā)送到 google 語(yǔ)音轉(zhuǎn)文字端點(diǎn)，我們還可以發(fā)送 blob uri 而不是內(nèi)容，唯一的變化是有效負(fù)載

// sending url as part of audio object to speech to text api 
...
audio: {url: audioUrl} or audio: {content: audioBlob}
...

與本文相關(guān)的程式碼存在於 Github 中。

以上是透過(guò) Google Speech to Text 進(jìn)行音訊轉(zhuǎn)文字輸入的詳細(xì)內(nèi)容。更多資訊請(qǐng)關(guān)注PHP中文網(wǎng)其他相關(guān)文章！