How different guitar woods sound? - Online Tests

[Samuele Carcagno]

But you can't produce evidence of it being or not being a confounder unless you test to control for everything but that (impossible I know). If it is present in an experiment to test another variable, you will never be able to say definitively that it does or does not influence your results. You have an anecdotal claim from Al that says it did (in that case) influence tone to a noticeable degree. This is where boosting your n will help but the water gets muddy that way too.

I agree, I'm just saying that variable won't necessarily have a large impact (perceptually). We may run some control experiments or collect more data on the physical characteristics of the guitars to assess this.

Also, before I take the tests, is there a concern with having participants know about the design?

no, that's OK, the test is blinded and I don't see how knowing the design could bias your responses.

I would also admit that I don't have as well developed ear for tone as you might like in your subjects; you say competent listeners how are you defining that (I very well may not be one). I'd hate to waste one of the participant slots the IRB approved <G>

when you start the survey you have to categorize yourself as an acoustic guitar player or not, if you're an acoustic guitar player you have to categorize yourself as an amateur, semi-professional, and professional. We expect that professional and semi-professional players should have a well developed ear for tone. Thanks for trying the tests.

[Rodger Knox]

While I doubt there will be any valid conclusions from this test, there are still good reasons to do it.
I've done enough experimental and testing work to know that when you finish a test, you have a much better idea of what works and what doesn't than you did at the beginning.
That information is at least part of what you need to know to revise the test so that it can produce valid conclusions.

It's the don't know squared problem. You start out not knowing what you don't know, so you don't know which questions to ask. The test may not give any answers, but may clarify the questions that need to be answered. Then at least, you know what you don't know, and can revise the test accordingly.

[Alan Carruth]

I'm glad that you're taking the criticism here in good part: it's meant to be constructive even if it's a bit discouraging. I'm also sorry that some folks on other fora have not been so helpful.

One of the rules of thumb in the musical acoustics business is that that there are things that are easy to measure and hard to hear, and things that are hard to hear and easy to measure. We're not very good at picking up fairly large changes in amplitude when the spectrum stays the same, but very sensitive to the addition or subtraction of harmonics near the threshold level of sensitivity. Doubling the power of a sound gives a 'just noticeable' difference in perceived loudness, but in some cases adding in a harmonic with 1% of the energy of the fundamental can make an audible difference in the tone. This is where those 'small' changes can trip you up.

I'm not much of a theorist, have terrible maths chops, and my hearing is lousy (otherwise I'd take your test). Most of the little I know has come from doing experiments. I tend to learn two main things when I do one:
1) it's a lot more complicated than I thought, and
2) I should have gotten more data.
I've NEVER done an experiment that didn't turn up something I didn't expect. I suspect you'll find the same.

At any rate, you'll probably set some broad limits for the next person that wants to do something similar, and that will be useful. The last thing I want to do is discourage anybody from doing real science: IMO even a flawed experiment is probably better than none at all. You're bound to get some useful data, if only because there's so little out there now.

[Trevor Gore]

Like many of the other respondents, I have so many problems with this experiment that it is difficult to know where to start.

I think you're asking the wrong question. I think it would be more useful to ask something along the lines of "How do we market alternative B&S woods so they become more acceptable to the guitar buying public?"

I come to this conclusion for a number of reasons.

1) Lots of builders make lots of different guitars which all sound very different and are all accepted by some proportion of the market. So there is no preferred sound on anything other than an individual basis.
2) I get to consult for some volume producers and so get to see their QC and returns records. Returns for sound quality reasons are close to zero, indicating that virtually any guitar can find a home
3) Many people, especially players, find it difficult to separate volume and tone, resulting in the infamous one-liner "give them loud and they'll hear tone"

Some further info on the construction of the guitars is available here:

http://www.fyldeguitars.com/news0914.html

the guitars were carefully built to be as similar as possible except for the back and sides wood. This doesn't mean that they're identical, but I think that residual differences are unlikely to be significant.

As similar as possible measured on what basis? Whilst Roger has experience in both engineering and acoustics there is no indication that the guitars were matched to acoustical criteria. There is no mention on his website that he does anything like that, which is, in fact, quite normal. Mark French did a study on Taylor guitars, 30 or so instruments, built to dimensional tolerances rather than acoustical tolerances and the T(1,1)1 resonances varied by 1.5 semitones and the T(1,1)2 resonances by 2 semitones, dues primarily to within-species variation of material properties which remains largely unmeasured and can vary by a factor of two. Guitars with resonances at either end of the spectrum would have sounded profoundly different irrespective of anything else. So if the low order resonances are not tightly controlled (there is no evidence to say they were) you are looking at differences much much deeper than just a change in B&S materials.

However, because within-species material property variations are large, it means also that there is a lot of overlap between the properties of different woods and so species substitution is possible. More details on that here.

That is why, without even trying to make guitars of different B&S woods sound the same, listeners are usually unable to identify the species of B&S wood the guitar is built from. The corollary of this is that if you have the intent (and the knowledge) you can make any wood sound pretty much as you want it to. But if you have two guitars that sound identical and one is of "traditional" wood and the other isn't, guess which one sells.

Hopefully, you'll find something constructive in what I've written and apologies if it comes across a little negative, but I think you're barking up the wrong tree.

[Simon Magennis]

Good luck with the test. I view it as a pilot to test methodology rather than a "scientific" experiment - it will be interesting to see what comes out of it. I hope you get lots of people to play the guitars. That is in some ways the most interesting aspect.

Regarding the methodology for the testing: I don't see why the dark room and goggles are necessary. Cover the back and sides with paper or thin, colored plastic. In a future experiment the simple expedient lacquering them all the same color would solve that from the beginning. I wonder if these guitars smell noticeably different?

I would definitely consider getting another set of results out of the experiment (pass it over to someone who needs a project in languages ?). It is very difficult to describe sound. While acoustic engineers may have some definitions for some of the words used: "loud", "bright" ,"rich" , "clear", "balanced", "warm" and "strong"; the rest of us may use some of these interchangeably or not at all. It should be quite easy to extract some information about how people use these words in the experiment. Someone in one of the language/linguistics departments would probably be able to use it for a publishable experiment or at least a minor thesis of some kind.

Looking forward to the results.

[Clay Schaeffer]

A possible addition to your experiment might be to do a "non blind" test, allowing the players to see and know which guitars are being played (or they are playing) and see how their scores compare to the "blind" playing tests. It might show how our perceptions are coloured by the perceived desirability of certain rare woods.

[Samuele Carcagno]

Hi Trevor, thanks for your comments, I had come across your POMA paper a while ago, and it's great to see people like you and Alan who have researched and published on the topic on this forum. As I said, criticism is welcome, I will actually be making a list of the issues that have been raised on the various fora, I think we'll be able to properly address several of those, those that we won't be able to address will limit (more or less severely) the conclusions of the study but will also help us design better experiments in the future.

However, because within-species material property variations are large, it means also that there is a lot of overlap between the properties of different woods and so species substitution is possible. More details on that here.

this is a very interesting point, for one thing it clearly means that the fact that a given guitar A is made of wood X does not necessarily make it sound better than another guitar made of wood Y. However, I'm a little unclear on further implications, depending on the actual distribution of the within and between species variations this may still mean that the best quality wood X will give a "better" sounding guitar than the best quality wood Y; another possibility is that a guitar made of wood X is more likely to sound "better" than a guitar made of wood Y.

[Samuele Carcagno]

Good luck with the test.

Regarding the methodology for the testing: I don't see why the dark room and goggles are necessary. Cover the back and sides with paper or thin, colored plastic. In a future experiment the simple expedient lacquering them all the same color would solve that from the beginning. I wonder if these guitars smell noticeably different?

Thanks! I agree. At a point we thought of having them lacquered all the same color, this would have simplified a bit the testing procedures. However, because the guitars are rented and will be sold at the end of the experiment this was not possible. The guitars don't really smell any different to me, but I've heard several people claiming to be able to distinguish guitar woods on the basis of their smell, to prevent this we've been using an air freshener in the testing room to mask any possible smell differences.

I would definitely consider getting another set of results out of the experiment (pass it over to someone who needs a project in languages ?). It is very difficult to describe sound. While acoustic engineers may have some definitions for some of the words used: "loud", "bright" ,"rich" , "clear", "balanced", "warm" and "strong"; the rest of us may use some of these interchangeably or not at all. It should be quite easy to extract some information about how people use these words in the experiment. Someone in one of the language/linguistics departments would probably be able to use it for a publishable experiment or at least a minor thesis of some kind.

we analysed hundreds of online acoustic guitar reviews and extracted the words that are most commonly used to describe the sound of an acoustic guitar. The descriptors that we're using in the laboratory tests come mostly from this linguistic analysis (a few descriptors were added by the guitar maker on the basis of his experience of what customers ask). For the online tests we're not using all of the descriptors we're using in the laboratory tests, the tests take already quite long as online tests and we wanted to keep their duration reasonably short. The choice of the online descriptors was mainly based on the results of a factor analysis on the results of the lab tests aimed to find how the different descriptors "cluster together". We'll give full details of the procedures once the study is finished.

[Samuele Carcagno]

A possible addition to your experiment might be to do a "non blind" test, allowing the players to see and know which guitars are being played (or they are playing) and see how their scores compare to the "blind" playing tests. It might show how our perceptions are coloured by the perceived desirability of certain rare woods.

Thanks, we're already doing that.

[Alan Carruth]

"we analysed hundreds of online acoustic guitar reviews and extracted the words that are most commonly used to describe the sound of an acoustic guitar. The descriptors that we're using in the laboratory tests come mostly from this linguistic analysis"

Some years ago there was a report on another list I'm on about the use of tone descriptors in violin tests. Sadly, I don't have a pointer to the research itself. As reported to me, the test involved recordings of the same musical passage on several violins, made in carefully controlled conditions by the same player. People were canvassed for possible tone descriptors, and the most common were chosen. In the test people listened to two recordings at random through headphones, and were asked to rate the sound on the basis of one of the tone descriptors: which is 'darker': A or B? There were several interesting results:
1) Everybody used the descriptors differently: one subject's 'dark' might be another's 'muddy'.
2) Each person used the descriptors consistently: if 'X' hears 'A' as darker than 'B', and B as darker than 'C', 'A' will always be darker than 'C'.
2) If some instrument had a tonal quality that a particular listener disliked, they would be unable to rate that instrument in terms of other tone qualities. If X dislikes 'nasal' sounding instruments, it would be hard for him to say whether a 'nasal' guitar was more or less 'even' than another.

SO: I have heard of other tests where the various descriptors were defined before hand, using sound examples other than the ones to be rated, possibly along with verbal descriptions. 'By 'even' we mean comparable perceived volume from note to note', followed by an example of an 'even' guitar and one that is not 'even'. I imagine this requires some training to get people to be consistent in their ratings.

Back when I was doing experiments on string forces on the bridge I came across an article on strings in one of the guitar magazines. At the beginning of the article one expert stated that Phosphor bronze is 'brighter' that 80/20, while later on another expert stated the opposite. It COULD have been a typo. of course, but I would not be at all surprised if it was an exact quote of the two experts, who simply use 'bright' to mean different things. This is, of course, the bane of every custom maker's existence: trying to figure out what a customer means by some term (what the hell is 'chocolaty' anyway?). Just another one of the little complications....

[Ryan Mazzocco]

I wonder if anything ever came of this
???

[Jason Rodgers]

There is an article in the latest AL called "The Leonardo Guitar Project" that sounds very similar.

[Gilbert Fredrickson]

Interesting experiment.