Tutorial for the vOICe
Objective
the objective of this tutorial is to allow a beginner to become familiar with the soundscapes. To a new user, the output of the vOICe is a meaningless jumble of sounds. Even though the user may know the soundscape mapping, it is extremely difficult to apply it when first hearing a soundscape. The attempt hear it is to walk the user through a series of commonly found objects so that the user can become familiar with what they sound like and consequently, get an idea of how to interpret the soundscapes.
Requirements
the best way to learn vision is to have an immersive setup with a laptop and a camera. However, at the first run, it would be good to start with static images. The disadvantage of an immersive setup on the first run is that there is too much data provided in the soundscape. So, it becomes very difficult for users to distinguish amongst the different objects in a soundscape.
caution
learning to see is like learning another language. Language teachers always advise the new learner to think in the language that is being learnt. For example, if one were to learn German, then it is best to think in German rather than say thinking in English and then translating into German. The same principle applies to the vOICe. Many users would be tempted to compare their sense of touch of a given object with the visual output of the vOICe. This is wrong. Vision and touch are totally different senses and initially, one should think in vision terms when looking at various objects.
System requirements
this tutorial will not go into the details of the setup required for using the vOICe. This information is readily available on the seeing with sound website. It is assumed for the purposes of this tutorial that the user has an adequate setup and the vOICe is up and running. Also, in sure that all settings of the vOICe are set to their defaults before continuing with this tutorial. Finally, please use a pair of headphones to listen to the soundscapes. It is possible to use speakers but you will not be able to make out the fine details required for a fruitful visual experience.
Your first soundscape
let us start by looking at your computer keyboard. The primary advantage of a keyboard is that it provides a uniform platform for the understanding of the soundscape. Before we actually look at the keyboard, let us go back a bit and understand the mapping of the soundscape.
1. A soundscape pans from left to right such that whatever is on your left is sounded in your left ear and whatever is on your right is sounded in the right ear. You can sense the curves of an object by the way the sound travels from the left side of your head to the right side of your head.
2. The pitch represents the height of an object. So, the higher the object, the higher the pitch of the sound.
3. The volume represents the brightness of an object such that the brighter the object, the louder the sound will be. You may wonder, what would happen if you were to look at a floodlight. Do not be concerned. The vOICe has been designed keeping safety in mind so you will not be deafened.
So now, we have taken a brief look at the mapping. Let us learn how to apply it by looking at the top view of a keyboard. If you have a camera connected to your computer, you would want to point the lengths such that it captures the top part of the keyboard.
Top view of a keyboard
A picture of the top view of a keyboard
Listened carefully to the soundscape. Notice how the sound moves as in pans
from left to right. You will notice the small bumps in the soundscape. Those
are the slightly higher pitched beeps that represent the keys on the keyboard.
For a closer look, let us zoom in to the keyboard and get a better idea of what
the keys look like.Close up of keyboard
(zoom factor 4)A picture of a
keyboard as it appears from a small distance. Hear how distinctive the keys
have become after we zoom in to the keyboard. Again, let us revised the
mapping. The soundscape is sounded from left to right, the pitch represents
the height of an object and the volume represents the brightness of an object.
To really get a feel for the curve of the keyboard, we can slow down the soundscape. This is what it will sound like. This time, as you listen to the soundscape, pay attention to how the sound moves inside your head.
Perspective
you could of course look at your keyboard from any side you choose as opposed to just looking at it from the top. You can try different viewing angles if you have a camera. For illustrative purposes, hear the two soundscapes below. You will be able to hear how the keys on the keyboard and the curvature of the keyboard change as you change the viewing angle.
An image of a keyboard that has been slanted to the left
an image of a keyboard that has been
slanted to the right
note the subtle change in the orientation of the keys. This is the only indication we have that the keyboard's orientation has changed.
The next thing we need to look at is the impact of distance. As your distance to an object increases, it fills a proportionately larger portion of your view. Therefore, you begin to hear greater detail of the object as you come close to it. Staying with the keyboard example, let us look at the first soundscape we've heard.
Compare the above soundscape with the close-up of the keyboard below.
In the above soundscape, you will find that a greater amount of detail of the keyboard is visible. Namely, the keys are more distinctive.
Conclusion
this section of the tutorial has covered certain basic concepts of using the vOICe. The next section will expand on these concepts and deal with looking at a few more commonly found objects.
A gentle introduction to the real world
let us now move into the real world. One of the basic things that people have to do is eat. A common implement found in such a context is a fork. If we reduce a fork to its constituents, we find that it has a handle on one end of which are a series of prongs. See if you can make out the prongs in the following soundscape.
A picture of a fork placed on a table.
Notice that the prongs will be sounded on the right as a set of closely spaced beeps. Remember we discussed orientation in the previous section? Now seen the image of the same fork but placed at a different angle. Namely, the handle of the fork is now pointing towards me and consequently the prongs are now pointing away from me.
Fork with the handle pointing towards me.
A picture of a fork with the handle of the fork pointing to words me.
The prongs will be more distinctive in this orientation.
You can try looking at any image in this way. For example, let us now try looking at a spoon. Try and find the bowl of the spoon.
A picture of a spoon with a large bowl.
The bowl of the spoon is on the right and well, sounds like a bowl.
Gauging distance
One of the things that people can do with great facility using vision is to gauge the distance to an object. What happens is that as you come closer and closer to an object, the object fills more of your view. You can experience this in the below movie. The primary object in this movie is the sideways view of my Buddy 5G microphone. The microphone has a base and a column. You will be able to make out the base at the beginning of the view and the upward sloping column. However, for ease of interpretation, I will first show you what a static image of the microphone looks like. This is again a sideways view.
Note:
in case you were wondering, why I had given a sideways view, there is no special reason except that there is a space constraint on my computer table. If I had tried the perspective exercise with the front view of the microphone, my face would have rammed into the mouthpiece of the microphone.
a static sideways view of my microphone
note:
did you hear the squishy sound at the right edge of the soundscape? This is the foam padding on the mouthpiece of the microphone.
a sideways profile of my microphone
me coming closer to the site of my microphone
Ewing colour
the vOICe at this point cannot let you experience colour. However, it can serve as a colour identifier. It is worth noting that the accurate identification of colour requires the camera angle and light level to be just right. the source of light needs to be constant and so does the camera angle. If possible, place the object for which you want to determine the colour on a scanner and then use the vOICe to determine the colour of the object.
As you view the image, the vOICe will speak the colour after sonifying every segment of that image that has that color.
Vision, a reality check what everyone must know about vision
in this section, I shall debunk certain myths about vision. Plus I shall also give you some interesting titbits that explain the mechanism of vision and common visual tasks. Do not worry if you do not understand them just now. As you progress with seeing with sound, these will become apparent to you. Also, you may get into a heated discussion with sighted people regarding these points. This is because most cited people do not experience them and lack the wherewithal to do so in the first place.
We don’t see with our eyes
while it is true that the I is used to convey images to the brain, the actual processing of the image happens in the brain. The brain is a massively complex inference engine and uses statistics to come up with shapes in response to visual input. This is why many a time, you hear the phrase “ I think I saw.” a classic example of this is seeing an approaching card while walking along a dark street. What people actually see are the headlights of the car. They do not see the full shape of the car. The question then arises how do they know that it is a car? They use several variables such as the space between the lights, the height at which the lights are moving and probably the speed at which the lights are coming towards them to determine that the approaching lights belong to a car.
A note on visual input
When you start using seeing with sound, you will realise that you get a lot of visual input. You do not need all of it to actively use vision. As you continue using the vOICe, your brain will learn how to filter this input.
Depth perception using one I
what most people do not consciously realise is that you need 2 eyes for depth perception. The way it works is that the brain gets slightly different views from both eyes. It compares these use and this comparison allows it to perceive depth. Given today's technology, the vOICe can only use a single camera. However, a feature has been implemented where you get slightly different views in either ear so, the theoretical possibility exists of being able to develop that perception. In fact, one of the users of the programme even without this feature had stated that she has been able to develop a significant degree of depth perception.
Recognizing faces
I have some good news and some bad news. The bad news is that there are special circuits in the brain that are used to recognize faces . The implication of this is that when you start using the vOICe, you will usually not be able to recognize faces. The good news is that there are a lot of other people like you. There is a condition called face blindness or Prosopagnosia. These people have developed coping strategies for these situations. For that matter, several accounts of poor memory for faces could actually be cases of face blindness. At the moment, using the vOICe, we don't know whether these special face recognition related circuits are activated.
Sources and attribution
this tutorial would not have been possible without the contributions of the members of the seeing with sound mailing list. In particular, the creator of the vOICe Dr Peter Meijer, has always been accessible to answer even the most basic of questions. For that matter, he is usually the first one to welcome newcomers to the vOICe. He also has the unique ability of translating complex technical concepts into layman language.
I would also like to thank all the other members of the seeing with sound mailing list for their respective contributions.
For more information on the vOICe, your first port of call should be the seeing with sound website. Its link is as follows.
linked to the seeing with sound website
Feel free to join the user group and ask questions. Also, Dr Meijer can be directly contacted. His contact information is given on the above website.
You are also most welcome to contact the author of this tutorial. He can be reached at the following e-mail address.