A: An instrument is much more than the sum of its parts. Why is a Stradivari violin worth millions and a modern Romanian factory built violin only worth a couple hundred dollars? They use essentially the same materials and are still largely hand made using the same measurements, contours, etc.
For a stringed instrument there are many complex factors that affect tone and playability. Because of the string tension (sometimes totaling a couple hundred pounds), the instrument has to be built very strong in the direction of stress, but can be made less strong in the perpendicular direction where the stress is less. This is why most string instruments use a type of fine grain spruce for the soundboard - it is very strong along the grain in the direction of stress, but is very light weight and flexible in the lateral direction. The balance between light/flexible and strong/rigid is key to making an effective instrument.
Factory made instruments are made to maximize profits for the manufacturer. They tend to use lesser quality woods that are not aged as long and are often somewhat thicker to stand up to machine processes, will often have inferior finishes that are too thick, and will not be precisely set up for optimum performance. Other factors that might be not be as well done include fret placement, leveling and finishing; and quality, smoothly operating tuners, whether mechanical tuners or violin pegs.
Some factory made instruments do a good job in most respects and a perfectly adequate instrument can be had for comparatively little money. But taken as a group, they are clearly inferior to instruments made by skilled craftspeople with superior materials.
There is also the intangible aura of brand names that is not without merit. There are hundreds of knockoffs of classic Gibson and Fender electric guitars, some of which actually play and sound very much like the real thing. So why should anyone buy a Gibson? The way a musician feels about his instrument makes a big difference in how he plays. If he knows he has the best tools available, he will play better. Many musicians use several different instruments, each with its own personality that comes out in the music.
Responding to your question about finishes, most modern finishes are sprayed on lacquers that are high gloss by default. Satin or flat finishes have chemical flatteners added to the lacquer. I don't think the level of gloss has much to do with sound. Oil or spirit based varnishes are not often used on mandolins and guitars because they are not as durable - they are more common on violins. What's more critical is the manner in which the lacquer is applied. If the finish is too thick it will tend to deaden the sound. It's another balancing act between durability and acoustic transparency.
A good luthier knows what he's doing and can predict the results. There's no (or very little) luck involved. But it takes many years to develop that level of expertise. I once knew a Celtic harp maker who would advertise new harp designs, but would not actually make it until he had an order. He knew his stuff well enough to predict with confidence exactly how it was going to feel and sound.
I think it also depends on your own aural acuity to a great extent. A couple of stories:
1. My father was a professional violist and he had a book and vinyl LP package that was published in the 60's, I think, called The Glory of Cremona. It featured famed violinist Ruggiero Ricci playing the same musical phrase on 15 different fabulous old Italian violins, including several Strads and Guarneris. The book featured beautiful full-color photos of each instrument. My father listened to this thing again and again. For the life of me, I couldn't tell the difference from one to the next and I consider myself to be pretty sensitive to aural nuances. But as a professional musician, my father had spent enough years to know what to listen for.
2. Several years ago, the Guild of American Luthiers held a convention at which a number of guitars were auditioned in a blind test (they may still do this at their conventions). Again, the same musician played the same music on all instruments, but he was sitting at the back of the room so the audience couldn't tell which instrument he was playing. The consensus was clearly that a cheap classical guitar purchased at Sears was junk. And a $15k guitar made by luthier Richard Bruné of Chicago was superior to the rest. All the other instruments, whether made by a beginning maker and selling in the $500-$700 range or by a master luthier in the $10,000+ range, were all pretty much the same. But the consensus was also that the higher end instruments definitely made the player feel like he was playing a superior instrument and that this was an important factor.
Brand names are not always a guarantee of quality, either. Stradivari did make a few not-so-good violins (though these have mostly long since perished). I'm sure Martin makes a bad guitar occasionally.
If I can't tell the difference between an expensive instrument and a cheap one by direct comparison, why should I buy the expensive one? Just because I know from objective evidence (or by popular acclaim) that it is superior in some ways, is that sufficient reason to spend the extra money? Maybe. Maybe not.
When it comes to mandolins, as it happens I am in the process of completing a sale of my own Martin mandolin. I've owned it for over 15 years and it is in mint condition. I paid very little for it and it's worth about three times that now. One reason it's in such good shape is that I've always been afraid to play it for fear of scratching it or something. It's much "too good" an instrument for my needs, since I don't really play much and then only for my own enjoyment. What I really need is an instrument that I won't worry about hauling around and banging on.