Project Reflection (YiDa)


GET1033 Final Project Reflection

In this project that I collaborated with Liang Hsi, we aimed to create a tool for guitar players and musicians. This includes a primitive tuning tool and a mutable chord library. The motivation for this was to incorporate technology into the music-making and learning process. Additionally, I saw this as an opportunity to practice the skills I have learnt during the course of this module- particularly the programming and coding aspects.
The choice of using OpenProcessing as the medium for our artefact was because of two main reasons:  1. The ability to conveniently host our sound samples online and 2. The availability of already built-in p5.js libraries.
            There were several challenges that I faced while creating this piece of work. Firstly, I underestimated the difficulty and mathematical complexity of creating a tuner that reacts to microphone input accurately. Secondly, abstracting the idea of chords and frets, that one associates with real physical guitar playing, into a computer code was not an immediately intuitive process.

Storing chord patterns in an Array data structure

            My approach to storing chords and shapes on the guitar was to identify them based on their frets. An important assumption was that this tool is intended for 6-string guitars in standard E-A-D-G-B-E tuning. The range of chords are limited to shapes that are within the first 8-9 frets on the guitar. Each chord formed on the guitar can be identified by the specific frets that one has to finger, as well as the strings that are to be strummed. Given that each fret can be identified by a fret number, my idea was to store these frets using an array data structure of size 6. Each element of the array from 0-5 represents the fret that is to be pressed on the respective string, start from the lowest-pitched string. Open strings are represented with the value 0, and strings that are not played are represented using a string element ‘x’. For example, [‘x’,0,2,2,2,0] means that the 2nd fret of the D, G, B strings are to be pressed, the A, and high E strings are to be left open, and the low E string is to be muted. Using this abstraction, Liang Hsi and I created a dictionary of chords based on their unique fretting patterns. The dictionary was treated as an object in JavaScript with the keys being the name of the chords, and the values being an array of size 6.

Visual representation of chords

The next step was to create a visual representation of this information. Liang Hsi was responsible for creating the main canvas of the project, which includes the lines that represented the strings as well as the fretboard of the virtual guitar. Black dots represent the frets to be depressed, unfilled dots represent open strings to be struck, and crosses represented strings that are to be muted. Initially, we used a rather brute-force method to generate these visuals by manually defining a function for each chord. In essence we used the “ellipse()” and “line()” primitive shapes available in the p5.js library to create these circles and crosses. Initially we tried to brute-force this process by defining a function for each chord. These functions included the ellipse() functions calls where the exact coordinates of the x-y positions (and size) of the dots and crosses on the canvas were specified. It did not take us long to realise the inefficiency of this process, and I created an abstraction for drawing these shapes.
Let us denote the array of size 6 that represents the fretting pattern of each chord as frets_array. Working under the assumption that the distance between each fret and each string on our virtual guitar is constant, I could come up with a mathematical relationship between the fret number and the x-y coordinates of its position on the canvas. Through the help of for- and while-loops, I defined the functions “make_x”, “draw_fret_on_string”, “make_chord” and “draw_chord”. The first two functions basically take each element of frets_array­, which is an integer value, and calculates the x-y positions of the frets and returns an ellipse or cross that will be drawn at that specific position. Each index of the array represents the corresponding string on the guitar in order. For example, the 0th index of the array represents the low E string, while the 4th index of the array represents the B string. Make_chord handles this information as it takes in the entire frets_array and extracts the elements of the array sequentially using a loop. The final piece of this abstraction is the draw_chord function which simply takes in the name of the chord (i.e. “E7” or “Am”) and retrieves the associated array. This method made the process of defining the library of chords much more intuitive and straightforward.

Audio playback of chords and sound sampling

Other than simply displaying chords on the canvas, there is also audio playback of the chords. This is achieved with the help of the p5.sound library. Initially I intended to use the oscillators and filters available in the p5.sound library to synthesize an acoustic guitar sound. After some research, I concluded that this was too complicated a process for me to achieve within this time frame, as the details involved to create a convincing guitar tone was simply too complex. As such I decided to implement the sound library using .mp3 samples. I manually recorded short audio samples of each string of an acoustic guitar using my Korg Krome keyboard at home. Using these 6 samples as a basis, I transposed the files on Cubase manually and created a range of samples that covered all the notes from E0 to E5 on the guitar. This brute-force method was preferred in the interest of time. Ideally, however, I should have executed the transposition on OpenProcessing itself using some kind of pitch-shift algorithm as that would have saved much file space. These sound samples are pre-loaded when the program runs using the preload() function. I chose to use standard international MIDI protocol to refer to each note – 60 refers to C4, 61 refers to C#4 and 59 refers to B3 etc. This make things very convenient as there is a clear relationship between the reference for each note. The sound samples are thus stored inside an object mySound_dict where the “keys” are the MIDI notes and the “values” are the corresponding samples. In a similar fashion to the abstraction in the previous paragraph, I defined a function “frets_to_midi_notes” to convert frets_array­ into an array that stores the respective MIDI notes that need to be sounded.
In order to simulate a guitar being strummed, I had to trigger each sample with an interval in between them, such that there is a sense of separation (just like how a real guitarist plays). This was achieved within my “playback_chord­” function that takes the MIDI array as input. Using the JavaScript built-in setTimeout function, I could separate each individual call of “mySound_dict[note].play()” by specifying a fixed time-delay of 150 milliseconds between each function call, thus achieving a pseudo-strumming effect.

Buttons and User Interaction

Finally, Liang Hsi and I created several buttons to allow for user-interaction with the program using the p5.dom library. The first is simply a search button and box where the user specifies the chord to retrieve. The ‘Strum again!’ button simply strums the chord that is currently being accessed, which defaults to ‘Cmaj7#11’ when the program initially runs. At the right-hand side of the UI, there are also 6 buttons corresponding to the open strings of the guitar that playback pitch-accurate audio of the respective open strings.
An important aspect of our program is the mutability of our library of chords. Given that a desired chord is not found within our pre-loaded library, the user can define a custom chord by specifying the name and its corresponding frets. When the ‘Create!’ button is pressed, the program takes in the frets as a value, and assigns the corresponding frets_array to the new chord inside the Chord_dict2 object. This way, the user can now lookup the new chord using the ‘Search!’ box and button. As a good measure to provide permanent loss of data in the Chord_dict2 object, I made it such that the program will first check whether a specific chord name has already been pre-declared. If it has indeed been pre-declared, the program will prevent the user from re-assigning a value to that name. 
Additionally, in place of our proposed Fast Fourier Transform (FFT) Tuner, Liang Hsi implemented 6 buttons on the right side of the interface that triggers samples of the opening strings for the user to tune by ear.

Difficulties and Challenges
            One of the biggest issues we faced was scoping. Due to the nature of JavaScript and its built-in setup(), draw() and preload() functions, where we chose to place our functions and objects in the code was very crucial. Instinctively, we created and call all our functions that created the ellipses and crosses within the draw() function’s block. This worked perfectly well until we implemented the sound library and the sound function calls mySound.play(). Because the draw() function repeatedly calls itself once the program has started, our sound samples were constantly called and playback when we placed our sound functions within the same block. This created an ugly wall of sound. I then moved the sound functions outside the draw() block into the global frame, thinking it will eliminate the problem of repeated sound function calls. However, a bigger problem arose – the sound library object declared in the preload() function cannot be accessed from the global frame. As a last resort, I moved all the sound functions and drawing functions into the setup() function’s block, completely removing the draw() function. This solved most of the problems until I realised that the dots and crosses drawn onto the canvas will accumulate. That is to say, if I were to use the Search button to call another chord while an existing chord is displayed on the canvas, the new chord’s visual representation will overlay onto the previous chord’s. In order to resolve this issue, I placed a recursive call of setup() within the draw_chord­ function to re-draw the entire canvas every time a new chord is looked-up.
            While the above measures have indeed resolved our issues, I am aware that they likely not the best ways to have written the program. Our canvas and background are relatively simple and basic, allowing the recursive calls of setup() to be rather smooth. This likely will not be the case had our canvas been much more elaborate. Furthermore, there was significant compression of the audio samples from .wav into .mp3 in order to not exceed the maximum file size capacity that OpenProcessing allows one to upload.
            Initially, we intended to implement a full-fledged FFT-based tuning algorithm that was able to analyse input audio via the microphone. Unfortunately, after much research and attempt, we were unable to implement this from scratch due to the significant mathematical complexity, and the need for external libraries that were difficult to import onto the OpenProcessing platform. Instead of resorting to copying large chunks of open-sourced code, we decided to dumb-down the tuning mechanism, and simply provide pitch-accurate playback of the open strings on a guitar for the user to tune manually, by ear.

Learning Points/Conclusion

            In conclusion, creating this digital artefact has allowed me to explore and practice the ideas that were taught in class. Our artefact is highly procedural, as there are many functions at each step to fully abstract the idea of representing and recreating the guitar chords on a digital medium. It is this specific set of idea that gives the user an intuitive experience in using our program. Our artefact however is limited in being spatial and encyclopaedic. Due to the fixed nature of the canvas, there is a limited number of chords that can be displayed within the fixed visual frame of the canvas. While there is a large library of chords that already exists in our code, it is certainly not exhaustive. Additionally, manually defining each chord is a highly tedious process. In terms of being participatory, our artefact gives the user some freedom in choosing and accessing the chords. Hands-on experience has taught me the importance of good programming skills and habits, such as proper indentation and debugging techniques.
            Our project makes use of the idea of objects and data structures that are available in computing to store information and ideas efficiently. Data structures like arrays and objects allow us to easily organize and manipulate information about chords as opposed to traditional method like print or physical charts. In terms of representation, we are essentially trying to transform the physical idea of sound and shapes into a digital one - bits and numbers to represent sounds and shapes, which can be manipulated by powerful digital systems. Furthermore, an important aspect of computers and computing is the ability for the system to update and change itself according to specific rules and inputs. This is achieved through the fact that our program and library of chords is mutable based on each user's specific requirements and inputs.
            Our current program sets a foundation for possible future improvements, such as higher quality of sounds, given enough space to host the audio files. Implementation of chord strumming can be extended to include more varied styles, such as fixed finger-picking patterns. Animations can also be included to visualize the idea of the individual strings being struck. Alternative tunings and string configurations can also be considered to provide for different styles and kinds of guitar players. Additionally, user interactivity can also be improved by allowing the user to directly construct chords on the fretboard itself, as opposed to manually specifying numbers that correspond to fret numbers. This can possibly make the program even more intuitive and easy to use.

(By Koh Yi Da A0187674L, for GET1033 AY18/19 Semester 1)

Comments

Popular posts from this blog

Geetar.io Video Tutorial

Welcome to Geetar.io !!