Thumbnail image

FasterWhisper STT Server Script

Running a FasterWhisper STT Server on a Raspberry Pi

Over the course of the last year, I’ve spent a considerable amount of time helping Neon and OVOS users customize their voice assistants. OVOS and Neon are both incredibly flexible platforms, which makes them powerful, but also complex. The biggest hurdle to getting a voice assistant running on a single small machine, such as Raspberry Pi, has been Speech-To-Text (STT). STT models are large and require significantly more computing power than other portions of a voice assistant.

Thankfully, in the last two years, OpenAI’s Whisper model has emerged as an extremely powerful and efficient STT model. Whisper is a lightweight, fast, and accurate STT model that can run on a Raspberry Pi 4 with ease. Whisper is the foundation of FasterWhisper, a community-driven project that provides pre-trained models of varying sizes. It’s multilingual but does offer some English-only models, which can improve performance and efficiency if you know you’re only going to be sending it data in English.

OVOS has several community-maintained STT servers that run FasterWhisper, but the ultimate goal of a private assistant is to be completely private and local as much as possible. Running everything on one device can be challenging, but OVOS and Neon make it easy to run your TTS and STT servers on a separate machine on your home network and then connect your assistant to it.

FasterWhisper STT Server Setup Script

For the first time, I’ve created a script to automate setting up a FasterWhisper STT server on a laptop or dedicated Raspberry Pi 4/5. This script is designed to be run on any Debian-based OS (Debian, Ubuntu, Raspberry Pi OS). Most of it should work for other Linux distros but it hasn’t been tested on them - you’ll need to adjust the part where python3.11-venv is installed. The script will install FasterWhisper, download a pre-trained model, and provide instructions to configure FasterWhisper to run as a service. It will also install a few dependencies and configure the Pi to run as a headless server. The script is available on GitHub Gists and I’m including a copy in this post as a backup.

#!/bin/bash

set -e

IP=$(hostname -I | cut -d' ' -f1)
cd ~ || echo "No home directory for this user, please install with a user that has a home directory." || exit 1

PIP=""
if command -v pip >/dev/null 2>&1; then
    PIP="pip"
fi
if command -v pip3 >/dev/null 2>&1; then
    PIP="pip3"
fi
if command -v python -m pip >/dev/null 2>&1; then
    PIP="python -m pip"
fi
if command -v python3 -m pip >/dev/null 2>&1; then
    PIP="python3 -m pip"
fi
if [ -z "$PIP" ]; then
    echo >&2 "FasterWhisper STT requires pip but it's not available. Please install it before running this script."
    exit 1
fi

echo "********************************************************************************"
echo "Making sure we can create a Python virtual environment..."
sudo apt install python3.11-venv

echo "********************************************************************************"
echo "Creating virtual environment at ~/STT_venv"
python3.11 -m venv STT_venv
source ~/STT_venv/bin/activate

echo "********************************************************************************"
echo "Installing STT requirements"
pip install --upgrade pip wheel
pip install --pre ovos-stt-http-server ovos-stt-plugin-fasterwhisper "ovos-utils>0.0.38"

echo "********************************************************************************"
echo "Configuring STT with default medium.en model. You can change this later by editing ~/.config/mycroft/mycroft.conf"
echo "Alternate model options can be found at https://github.com/OpenVoiceOS/ovos-stt-plugin-fasterwhisper/"
echo ""
mkdir -p ~/.config/mycroft
echo "********************************************************************************"
echo "Backing up any existing mycroft.conf to ~/.config/mycroft/mycroft.conf.bak"
echo ""
# If the file exists, make a backup copy
if [ -f ~/.config/mycroft/mycroft.conf ]; then
    cp ~/.config/mycroft/mycroft.conf ~/.config/mycroft/mycroft.conf.bak
fi

cat <<EOF > ~/.config/mycroft/mycroft.conf
  {
    "stt": {
      "module": "ovos-stt-plugin-fasterwhisper",
      "ovos-stt-plugin-fasterwhisper": {
        "model": "medium.en",
        "use_cuda": false
      }
    }
  }
EOF

echo "********************************************************************************"
echo "STT installed, you can now start the server with:"
echo "/home/$USER/STT_venv/bin/ovos-stt-server --engine ovos-stt-plugin-fasterwhisper --host $IP"
echo ""
# Give optional instructions for systemd service
echo "If you want to run this as a service, you can use the following systemd unit file:"
echo ""
cat <<EOF > ~/ovos-stt-server.service
[Unit]
Description=OVOS FasterWhisper STT Server
After=network.target

[Service]
Type=simple
User=$USER
ExecStart=/home/$USER/STT_venv/bin/ovos-stt-server --engine ovos-stt-plugin-fasterwhisper --host $IP
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
cat ~/ovos-stt-server.service

echo ""
echo "You can set this up as a long-running service with:"
echo "********************************************************************************"
echo "sudo cp ~/ovos-stt-server.service /etc/systemd/system/ovos-stt-server.service"
echo "sudo systemctl daemon-reload"
echo "sudo systemctl enable ovos-stt-server.service"
echo "sudo systemctl start ovos-stt-server.service"
echo "********************************************************************************"

Testing the server

Once your server is running, a quick test is in order. You can test the server by running the following commands:

wget https://github.com/OpenVoiceOS/ovos-stt-plugin-fasterwhisper/raw/dev/jfk.wav ~/jfk.wav
curl -X POST -H "Content-Type: audio/wav" --data-binary '@/home/$USER/jfk.wav' http://localhost:8080/stt

You should see the following output (as an example):

❯ curl -X POST -H "Content-Type: audio/wav" --data-binary '@/home/$USER/jfk.wav' http://localhost:8080/stt
"4 score and 7 years ago our fathers brought forth on this continent a new nation conceived in liberty and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war testing whether that nation or any nation so conceived and so dedicated can long endure. We are met on a great battlefield of that war. We have come to dedicate a portion of that field at a final resting place with us who here gave their lives that the nation might live. It is altogether fitting and proper that we should do this. But in a larger sense we cannot dedicate, we cannot consecrate, we cannot hollow this ground. The brave men, living and dead, who struggled here, have consecrated it far above our poor power to attract. The world will little note, nor long remember what we say here, but it can never forget what they did here. It is for us the living, rather to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us that from these on the dead we take increased diversion to that cause for which they gave the last full measure of diversion that we hear highly resolved that these dead shall not have died in vain that this nation, under God, shall have a new breath of freedom and the government of the people, by the people for the people, shall not perish from the earth."%

Connecting OVOS and Neon to the FasterWhisper STT Server

Once your server is running and you’ve confirmed your operating system firewall isn’t blocking inbound traffic on TCP 8080, you can connect your OVOS or Neon device to it. On OVOS, you can do this by editing ~/.config/mycroft/mycroft.conf and adding the following lines:

"stt": {
    "module": "ovos-stt-plugin-server",
    "ovos-stt-plugin-server": {
      "url": "http://<IP of your FasterWhisper STT server>:8080/stt"
    }
  }

For Neon (~/.config/neon/neon.yaml):

stt:
  module: ovos-stt-plugin-server
  ovos-stt-plugin-server:
    url: http://<IP of your FasterWhisper STT server>:8080/stt

You may also want to add public OVOS FasterWhisper servers as a fallback in case your server is down. You can do this by editing ~/.config/mycroft/mycroft.conf and using the following config:

"stt": {
    "module": "ovos-stt-plugin-server",
    "ovos-stt-plugin-server": {
      "url": [
        "http://<IP of your FasterWhisper STT server>:8080/stt",
        "https://fasterwhisper.ziggyai.online/",
        "https://stt.smartgic.io/fasterwhisper"
      ]
    }
  }

On Neon, you can do this by editing ~/.config/neon/neon.yaml and adding the following lines:

stt:
  module: ovos-stt-plugin-server
  ovos-stt-plugin-server:
    url:
      - http://<IP of your FasterWhisper STT server>:8080/stt
      - https://fasterwhisper.ziggyai.online/
      - https://stt.smartgic.io/fasterwhisper

Note that if you’re on a Mycroft Mark 2 devices running Neon, you will still have a fallback to STT on the device, which consumes a ton of memory and is very slow. You may want to consider disabling your fallback plugin, but adding public OVOS FasterWhisper servers in case yours is down. You can do this by editing ~/.config/neon/neon.yaml and using the following config:

stt:
  module: ovos-stt-plugin-server
  ovos-stt-plugin-server:
    url:
      - http://<IP of your FasterWhisper STT server>:8080/stt
      - https://fasterwhisper.ziggyai.online/
      - https://stt.smartgic.io/fasterwhisper
  fallback_module: ""

The url values are read in order, so if yours is first in the array, it will be used as long as it’s online.

Conclusion

I hope this script is helpful to some of you. If you have any questions or suggestions, please let me know in Matrix or via email. As always, I’m open to requests for posts and any help you might need getting your FasterWhisper STT server working.