In this article, we will study how to Create Music with Python From Images.

It would be an enormous understatement to suggest that music is essential to my existence. Since I’ve been playing the guitar for over 20 years, I’ve listened to music every day before starting my 12-year academic career as a physical chemist. I was about to embark on a musical career.

Music allows me to have fun and do beautiful things even if I cannot follow that career. For instance, it’d be nice to create software to generate music from photographs for this project. Some very creative efforts to develop such a thing have been made. The outcomes, nevertheless, are not wholly musical.

I’ll demonstrate my effort to create music from photographs for this post, and they sound great. This article will primarily cover the best outcomes and show some extraordinary software examples. You can access the complete code on my website or the GitHub repository if you want to view it. You may experiment with an app I created with Streamlight here.

Basic Concept

To do this, I will use the following technique and thinking process:

Pixels are what makeup images.

  • Pixels are made up of collections of integers that represent colors.
  • Color may be defined using color spaces like RGB, BGR, or HSV.
  • Sections of the color space might be created.
  • Lucid intervals are used to break down musical scales into notes.
  • Since sound is vibration, each note has a corresponding frequency.

As a result, a particular note in a musical scale might be assigned to each subdivision of a color space. This note would subsequently be linked with a frequency.

HSV Color Space use

Three values manage a color space called HSV. Brightness, brightness, and saturation are the three values.

According to the dictionary, HUE is “the degree to which a stimulus may be characterized. Similar to or distinct from stimuli classified as red, yellow, orange, blue, green, and violet.” Therefore, HUE is a representation of a color.

The term “colorfulness of a roundabout measured in percentage to its brightness” refers to saturation. In other words, saturation refers to how much white is incorporated into a hue.

Brightness is “perception caused by the luminance of a visual object.” In other words, saturation refers to how much black is incorporated into a hue.

Colors’ central hue values:

  • Orange 0-44, 
  • Green 76-150, 
  • Yellow 44-76
  • Blue 150-260,
  • Red 320-360 
  • Violet 260-320

I’ll work with the HSV color space as it is inherently split, and the ensuing frequency mapping will be easier to understand. Similar to how the other two tracks are separated from the hue channel, which is mainly in charge of color in this area, things are made simpler.

Here is an illustration of how color spaces for images and the code that creates them compare:

Making the Hue Channel Available

Since it is in HSV format, we can extract the hue (H) value from each pixel in our picture. The height and width of the image may be looped over many times to do this extraction.

The H value for each pixel is now included in an array. Therefore, I’ll put that result into a pandas data frame. Since each row in the data frame represents a pixel, information about each pixel will be found in each column. This data frame, which I’ll refer to as pixels df, is shown below:

Currently, the data frame comprises a single column named “hues,” where each row corresponds to a pixel from the picture I imported.

Hue to Frequency Conversion

My original plan for transforming a hue value into a frequency required mapping the H value to a specified range of frequencies. The mapping feature is shown as follows:

The function accepts as inputs the H value and an array of frequencies to map H. The frequencies are specified in the example using an array named scale freqs. The A harmonic minor scale represents the frequencies used in scale freqs.

Then, a set of H threshold values is defined as a set of thresholds. Using a lambda function, this array of thresholds may transform scale freqs’ value of H into a frequency.

NumPy Array to Playable Audio Conversion

I’ll turn the notes column into a NumPy array named frequencies now that I have an array of frequencies since I can use this to create a playable audio file later:] I can do this by using the scipy-built WAV file function, carefully utilizing the proper data type conversion.

The tune I created using the first 60 pixels of the picture below may be heard below (I could try using all 230,400 pixels that make up that image; nonetheless, that tune would eventually run for many hours).

Comprising Octaves

I decided to include the octave effect—making notes sound higher or lower—into my “song-making” process. An octave will be selected randomly from an array for a specific note.

Anthem of Octaves

Awesome! We are now experiencing some variation. We do, however, have a tonne of pixels. Why don’t we give it a go by selecting the frequencies from arbitrary pixels?

Creating Additional Scales

I’ve already demonstrated how the A-harmonic minor scale can create music from images. Even though this scale is incredible, having a more comprehensive range of tonic notes and other intervals outside those specified by the harmonic minor scale structure would be good. By using these scales, our algorithm can give the music it generates more character and diversity.

I initially required a means to procedurally create frequencies for the tonic note that we wanted to utilize to do this. Katie, He has a lovely post here that explores Python with music in an elegant way. To map piano notes to frequencies in my work, I modified one of her routines, as seen below:

The function above may be used to generate a dictionary that maps musical notes corresponding to the 88 keys of a typical piano to frequencies in units of Hz, as shown below. This dictionary will serve as the basis for my song/scale-generating routine.

Then, to index our notes, we must specify the scale intervals in terms of tones.

With this, we can now search the previous list of tones for the index of our scale. I’ll then reindex the list such that it begins with our chosen tonic. Thus I’ll need these scale intervals.

Following this, I may establish several scale arrays, where each element is associated with an index from the reindex array I created before.

I can now create a song using the outcomes of the earlier processes!

Awesome! Now that I have many factors, I can create tunes. I control the number of pixels, whether they are picked randomly, the scale, octave, key, Harmony, and length of each note.

For the next part, headphones are strongly advised if you’re reading this on a phone since specific lower frequencies don’t appear to sound well on regular audio.

Experiment with Pixel Art

Pixel art would be a natural medium for experimentation. The key of A Major was used to create this tune, with the third octave serving as the basis. It has a lovely sound. I get the impression that I’m strolling through a pleasant little town. This experiment works excellently in the opening sequence of an old-school video game.

Based on Marta Nael’s China Mountains, Pixel China Mountains was created by Matej Retro in January 2013.

Including Harmony Through 2D NumPy Arrays in Songs

I devised a procedure that permits us to add Harmony to our music. The user may choose the theme to use (for example, perfect fifth or minor 6th) and then use the techniques I previously showed to get the proper note interval. I’ll provide an example of how a song may be created from a picture, the Harmony that goes with it, and how a 2D NumPy array can combine the two into a single .wav file.

If I wish to write a 2D array into a.wav file, according to the documentation for scipy.io.wavfile.write, the 2D array must have dimensions in the form of (N samples and N channels). Observe the current structure of our collection (2 264600).

The sky is reflected in the lake of New Zealand’s Fiordland National Park. Mark Gray is the photo’s owner.

Utilizing Spotify’s Pedalboard Library to Enhance Our Songs

Great! But there’s so much more we can do! I’ll import the .wav files and edit them using Spotify’s Pedalboard plugin. You should check it out since it’s excellent. You may learn more here and here about the pedalboard library.

I’ll start by utilizing a pedalboard consisting of a Compressor, Gain, Chorus, Phaser, Reverb, and Ladder Filter to reprocess the previously demonstrated water song. The outcome is shown below:

The harmonized tune created from the nature image I previously showed will be reprocessed using a LadderFilter, Delay, Reverb, Chorus, PitchShift, and a Phaser as a final example. The results are shown below:

Mapping Other Musical Quantities with Librosa

Librosa is a stunning program that offers a wide range of operations on sound data. Another bundle that I strongly urge you to look into is this one. I used it, in this case, to quickly translate frequencies into “Notes” and “Midi Numbers.”

The ability to store our songs in the Musical Instrument Digital Interface (MIDI) file format would allow other interested artists or programmers to take the music and play with it. MIDI files can link to electronic musical instruments, computers, and other audio devices.

The functions I used to map the frequencies I produced using my technique to get the corresponding musical notes and midi numbers for our song are shown below.

MIDI Creation for Our Song

I can create a midi file now that I have a data frame with frequencies, notes, and midi digits! I could create sheet music for our song using this MIDI file:].

I’ll use the midiutil package to create a MIDI file. With the help of this package, we may create MIDI files from an array of MIDI numbers. Your file may be customized by adjusting the volume, tempos, and tracks. I’ll make a single-track midi file for now.

Conclusion

I demonstrated how to create music out of photos and how to export our tunes as .wav files for further manipulation. I explained how harmonies might be created using this technique and how odd, intricate, or rich balances could be made. I want to improve this process, but I’ll save it for later.

If you need help developing fresh song ideas, try putting a picture into my app to see if you receive any excellent suggestions. You always need to find out where an idea may strike!

This project was enjoyable for me, and I hope it will be fun for you to use and expand.

Please also read the fascinating articles.

J. Shaw

Joseph Shaw is a renowned expert with two decades of experience in health and fitness, food, technology, travel, and tourism in the UK. His multifaceted expertise and commitment to excellence have made him a highly respected professional in each field.

J. Shaw has 192 posts and counting. See all posts by J. Shaw

Comments are closed.