Image by Author

Neural Networks for Real-Time Audio: Raspberry-Pi Guitar Pedal

This is the last of a five-part series on using neural networks for real-time audio.
For the previous article on Stateful LSTMs, click here.

Important Note: The software and hardware have been greatly improved since the time of writing this. These updates are listed at the end of this article and include a new video demo.

In this article we will go step-by-step to build a functional guitar pedal running neural nets in real-time on the Raspberry Pi.

We have now covered three different neural network models and implemented them in real-time guitar plugins with the JUCE framework. As a guitarist and engineer, the next logical step for me is to build a guitar pedal using what we have learned from the previous articles. Amazingly, all the software tools are available for free, and the hardware costs around $150 (less if you already have the required power and audio adapters handy).

The secret sauce for making this possible is Elk Audio OS. This is a Linux based open source operating system made specifically for low-latency audio processing on embedded devices. It allows you to run existing VST3 plugins on devices such as the Raspberry Pi.

The software I wrote is called NeuralPi, and is open source on Github. I included a model of the Ibanez TS9 Tubescreamer pedal, as well as a Fender Blues Jr. amplifier.

Here is a video demonstrating the completed project, using the NeuralPi VST3 plugin on the Raspberry Pi hardware:

Special thanks to my son (age 5) for filming the intro (Video by Author)

Project Setup

This project consists of four main components:

1. Raspberry Pi 4b (Mini computer running the plugin)
2. HiFi Berry ADC + DAC (Analog to Digital, Digital to Analog board that sits on top of the Raspberry Pi.
3. Elk Audio OS for Raspberry Pi4 (Low latency operating system specifically for audio)
4. Real-time VST3 plugin (NeuralPi) cross-compiled for Elk Audio OS (the plugin running the neural net engine with guitar amp/pedal models)

Full list of parts and prices:

Here is a list of everything I purchased for the project, all ordered from Amazon except for the HiFiBerry ADC +DAC , which I ordered from Chicago Electronics.

Raspberry Pi / HifiBerry setup (Image by Author)

1. Raspberry Pi 4b (Amazon), $50 (I’ve seen the Rpi4 on sale for $35)
2. Rpi4 Power Adapter (USB-C connector, 5.1V — 3.5A) $10
3. Micro SD card (minimum 8GB, I bought 32GB), $12
4. Rpi4 + HiFiBerry compatible enclosure, (you can spend as much or as little as you want here; I bought a customizable enclosure suitable for prototyping, but a different enclosure would be better for a finished product) $20
5. HDMI micro to Standard HDMI Adapter Cable (Rpi4 has 2 micro HDMI outputs) $8
6. HiFi Berry ADC + DAC, (There is a “pro” version for $65, but I stuck with the standard version) $50
7. Dual 1/4" Female to 1/8" Male Stereo Audio Adapter (for plugging the guitar into HifiBerry), $8
8. Stereo Male RCA to 1/4" Female Audio Adapter (I bought 1/8" for to use with my headphones, but 1/4" would be the typical guitar pedal output), $8

Total Parts Cost: $166 + Taxes and Shipping

Additional items I already had:

• Separate Computer Monitor with HDMI input (can use any screen with HDMI input) ($50 and up)
• Wired USB Keyboard ($10)
• Laptop running Ubuntu Linux (for building VST3 plugin and communicating with Rpi4 over WiFi (Refurbished, $250), You can also use a virtual machine running Linux
• 1/8" male to 1/4" female adapter $5 (for using an amp instead of headphones)

Note: The separate Linux Computer is not needed unless you are cross-compiling the plugin yourself. You will, however, need a separate computer to connect and upload the VST3 to the Raspberry Pi (either through WiFi or Ethernet)

Assembling the Hardware

The HifiBerry plugs into the top of the Raspberry Pi via the two rows of pin connectors. It came with spacers that secure the HifiBerry card on top of the Rpi4. I chose an enclosure for a generic prototyping setup, but the HiFiBerry website offers a nice looking enclosure for a simple guitar pedal project with no additional footswitches or knobs. I’ll probably end up building my own enclosure for a finalized product.

After the hardware is assembled you can connect all the necessary adapters. If you are familiar with using the Raspberry Pi 2, you’ll want to note a few key differences with the Raspberry Pi 4:

1. The HDMI out is Micro (instead of standard size)
2. The power cable is USB-C (instead of micro USB)
3. Wifi and Bluetooth are built into the Rpi4

Setting Up Elk Audio OS for Raspberry Pi 4 + HiFiBerry ADC + DAC

The first step is to flash the micro SD card with the Elk Audio OS for Raspberry Pi 4. To do this, we will use a tool called balenaEtcher, but there are other options for imaging SD cards.

Image the microSD with Elk Audio OS

1. Download the Elk Audio OS image for Rpi4 from Github (.bz2). Unzip the .wic file.
2. Download and install balenaEtcher.
3. Plug in your microSD card to computer (minimum 8GB) (USB adapter may be needed to connect the microSD to your computer)
4. Run balenaEtcher and follow the prompts to image the microSD with the download .wic file.
5. When imaging is complete, remove the microSD and plug into the microSD slot on the Raspberry Pi 4.

Boot up the Rpi4 with Elk Audio OS and perform initial setup

1. Connect the Rpi4 to an external monitor with HDMI, connect a USB keyboard, and connect power.
2. When the login prompt shows up, type “root” and hit enter to login as root (by default there is no password for root, but you can set one up if desired).
3. Configure the Elk Audio OS for use with HiFiBerry ADC + DAC. Type this into the command prompt:
   sudo elk_system_utils --set-audio-hat hifiberry-dac-plus-adc

And reboot the Rpi4 for the change to take effect:
sudo reboot

4. Configure for communicating over Wifi or Ethernet as detailed in the official Elk documentation.

5. You can now ssh (secure shell) to your Rpi4 from another computer over WiFi. From my Ubuntu Linux computer this is as simple as running the following from the terminal:

ssh mind@<ip_address_here>
# And enter the default password for "mind" user: "elk"
# Files can also be transferred to the Rpi4
scp -r YourFile root@<rpi-ip-address>:/target_path/

Note: Once you connect the Rpi4 to WiFi, you can obtain the IP Address assigned to it by typing “ip a” in the Elk OS terminal.

Selecting the Neural Net model for the Guitar Pedal

In the past three articles, we covered three neural net models and their real-time implementations. Each model has its own strengths and weaknesses. To select the most appropriate model for our pedal, a simple trade study was conducted. Each neural net model was given a point ranking from 1 to 3, with 3 being the best. Three categories were chosen: training speed, sound quality, and real-time performance.

Table 1: Model selection trade study (Image by Author)

Note: Each category was given an equal weighting, but realistically for using the Raspberry Pi, the Real-Time Performance would be the most important factor, followed by Sound Quality.

The Stateful LSTM model received the highest score for it’s real-time performance and sound quality. The Stateless LSTM has superior training speed, but for our application of creating a custom guitar pedal this is secondary to having a great sound. The WavNet has a great sound but its higher CPU usage may cause problems on the Raspberry Pi (although this is worth going back and testing). Also, WavNet’s difficulty in handling high gain guitar sounds makes it less appealing for use in a guitar pedal.

Cross-Compiling for Rpi4 with Elk Audio Toolchain and JUCE

If you are using NeuralPi.vst3 from Github, you can skip this section. These steps are for if you want to compile your own VST3 using JUCE, or modify an existing VST3 plugin to run on Elk OS. The Elk Audio OS runs “headless” plugins, which means there is no graphical interface. JUCE 6 has the headless build feature included, but you may need to further modify your plugin to meet the requirements detailed in the Build Plugins for Elk documentation.

On a Linux computer or Linux virtual machine (I used Ubuntu 18.04):

1. Download the Elk Audio SDK for Rpi4
2. Run the downloaded SDK (.sh file) to install. I used the default location: /opt/elk/1.0
3. Open your .jucer project using the Projucer application (JUCE 6, I used JUCE 6.08), and modify the project according to the Elk documentation (see “Inside Projucer:” section).
4. Create a Linux Makefile build target if it doesn’t already exist and save the Projucer project to generate the makefile directory and makefile.
5. Follow the steps as detailed in the Cross-Compiling JUCE Plugin documentation. Here are the exact steps I used:

After entering the make command, the plugin will begin to compile. If all goes smoothly, you will now have a .vst3 plugin compatible with Elk OS / Raspberry Pi 4.

Important: Ensure the architecture folder name in the compiled VST3 is “aarch64-linux”, for example, “PluginName.vst3/Contents/aarch64-linux”, otherwise the plugin will not run on Elk OS. Rename the folder to “aarch64-linux” if necessary.

Optimizing the plugin code using RTNeural

I immediately ran into problems when I first ran my plugin on the Raspberry Pi. It was running without any errors, but it made this horrible sound:

With some help from the Elk Audio forums, it was determined that my plugin was running too slowly. If I looked at the process diagnostics, it was using 99% of the CPU’s resources, and probably overrunning that amount. I had to find a way to optimize my code, or it simply would not be usable.

In a very serendipitous turn of events, the day after I came to the conclusion that my current plugin wouldn’t work on the Pi, the creator of RTNeural reached out to me, interested in implementing his code in one of my plugins. It turns out that he had done quite a bit of research on optimizing neural networks for audio processing, and created RTNeural as an inferencing engine for real-time audio.

It turned out to be fairly straight-forward to swap out my LSTM inference code for RTNeural. I loaded up the newly compiled plugin with RTNeural on the Pi, crossed my fingers, and looked at the process diagnostics. At first I couldn’t believe what I was seeing: it had dropped from 99% to 16%! I plugged in my guitar and headphones and it sounded just like the plugin on the laptop. I added some delay and reverb, and made this sample recording directly from the Raspberry Pi:

I won’t go into the details of why RTNeural was able to supercharge my plugin, but it has to do with utilizing SIMD (Single Instruction, Multiple Data) instructions. These are low-level operations that use process vectorization to run computations in parallel (similar to what graphics cards do). Modern processors have this capability for high performance computing on the CPU.

Running the Real-Time Plugin on Elk Audio OS

Elk uses a plugin host called “Sushi”. Sushi uses configuration files that set up audio routing, plugin settings, and midi settings. You can view the example configuration file for NeuralPi on Github. In order to move your plugin to the Raspberry Pi, I recommend using “ssh” (secure shell) over WiFi. I used a dedicated Ubuntu Linux laptop, but there are several ways you can accomplish this. I’ll go over the steps I used here.

1. Download the NeuralPi-elk.zip from Github and extract the .vst3 and configuration files.

Note: This article references the original release of NeuralPi. For the latest release with updated features, see the NeuralPi Releases page.

2. Move the .vst3 to the Rpi4.

Connect to the Pi using one of the methods described here. To connect through ssh over Wifi, I used the following commands on a separate Linux computer connected to the same Wifi network as the Pi.

# Secure copy the .vst3 to the Raspberry Pi
scp -r YourPlugin.vst3 root@<rpi-ip-address>:/home/mind/plugins/

# After you copy the vst3 to the RPi, you can ssh to verify it 
#  copied correctly
ssh root@<rpi_ip_address>

3. Create a .json config file for running the plugin. If you are running NeuralPi.vst3, you can use the one included in the Github release.

# Secure copy the .config to the Raspberry Pi
scp -r YourConfig.config    
       root@<rpi-ip-address>:/home/mind/config_files/

4. Connect the audio adapters, guitar input, and audio output device (speakers, headphones, amp).

5. Run the plugin using Sushi.

# Login to the "mind" user and run the plugin with sushi
#  (the default password is  "elk")
ssh mind@<rpi_ip_address>

sushi -r --multicore-processing=2 -c ~/config_files/config_neuralpi.json &

This will run the headless plugin in the background. You can view the process usage of the real-time plugin by typing the following:
watch -n 0.5 cat /proc/xenomai/sched/stat

Note: Using the standard Linux process tools like “top” will only show you the non-real time tasks.

To quit the plugin, type the following in the terminal:
pkill sushi

You can set up the Raspberry Pi to automatically run a configuration when it boots up. This is useful for operating the Pi like a typical audio effect device.

Real-Time Performance

At 16% process usage on the Pi, there’s plenty of room for adding other effects such as cab simulation (impulse response), reverb, delay, flange, or any number of less CPU intensive effects. The sky is the limit when it comes to the possibilities on the Raspberry Pi. One could create a phone app that controls the plugin over WiFi, add knobs and controls, or even dual boot another OS running a media center, web browser, or video game emulator.

Elk Audio OS is designed for low latency audio, so you don’t need to worry about other processes interfering with the plugin performance as you would on a normal laptop. The Raspberry Pi 4 processor has four cores, with two running real-time operations, and two for non-real time operations. This ensures that the audio performance stays constant.

Right now this Raspberry Pi guitar pedal is very bare-bones, but I plan on adding physical knobs and a nice looking enclosure to turn it into an actual guitar pedal. I’ll post a new article on the build process once that is completed.

Software Updates

UPDATE 1: Since the time of writing this article I’ve added the ability to control model selection, EQ/Gain/Volume over WiFi from a Windows or Mac computer using the NeuralPi plugin/app. Multiple amp and pedal models are available here. (Release v1.1) See the NeuralPi GitHub page for more details:

GuitarML/NeuralPi

UPDATE 2: A 3-D printed case is available for NeuralPi. The STL files are available here.

STL models (top) and printed case without cover (bottom) (Image by Author)

Update 3: Version 1.2 of the NeuralPi software now includes the ability to load Impulse Response files, as well as simple Delay and Reverb effects. Impulse response is a commonly used effect to model guitar cabinets or the reverb characteristics of a particular space. When combined with the LSTM model it provides a more realistic representation of a guitar amplifier.

Release NeuralPi 1.2.0 · GuitarML/NeuralPi

Update 4: Version 1.3 of NeuralPi adds the ability to load conditioned models (full range of a gain/drive knob) and swaps the default snapshot models for conditioned models of the TS-9, Blues Jr., and HT40 amplifier.

Release NeuralPi 1.3.0 · GuitarML/NeuralPi

Update 5 (2/9/2023): I built an actual guitar pedal called Proteus using an updated version of the NeuralPi software for more realistic models. The pedal includes a buffered input, volume control, and line/instrument level switch in a metal enclosure. You can view a video demo and hardware walkthrough here:

This concludes my five part series on using Neural Networks for real-time audio. I hope it sparked your curiosity on the possibilities of neural networks for music and maybe even convinced you to try it out for yourself.

Thank you for reading!

If you would like to continue reading about machine learning for guitar effects, check out my next article here:

Transfer Learning for Guitar Effects

Image by Author