For my book, Career Change, I gave an early copy to people working in my department in the day job who I knew were dissatisfied with what they were doing. They came back with questions and suggestions for what to include as additional material. For The Healthy Writer, we asked medical doctors to read it as a sense check (even though my co-writer, Euan Lawson, is a doctor!)
Stephen, I’m not sure what you mean by “time wasted matching a style guide.” Any professional editor of trade books knows the Chicago Manual of Style well and as a matter of course automatically makes changes that are in line with it. What you said is akin to “They waste time correcting spelling errors.” They’re not wasting any time; they’re doing their job and it’s a basic and automatic part of that job.
Even if you wrote the book in your free time, this is my day job. The per-project cost many new authors would consider “manageable” would, if broken into an hourly rate, end up paying editors less than minimum wage. They don’t intend to pay what someone could earn making french fries; they just greatly underestimate how much time must be invested in editing.
You also said it’s not fair for self-publishing authors to apply those standards to editors when they don’t work with publishers themselves. It’s one thing for authors to have the right to publish their work and make it available for readers to judge its merits, and quite another for someone to ask those authors to invest in them for their expertise.
There’s a big difference between editors who point out your weaknesses and come up with solutions, versus those who give general advice that doesn’t go anywhere. One author who kept getting generic feedback from editors that amounted to, “It’s good, keep writing,” ended up with a 215,000-word tome. The weary man finally found an editor who was able to show him how to cut the manuscript in half and tighten the plot.
Regarding your last question, I know of no editor who gives a money-back guarantee for occasional errors that happen to slip through or for a different opinion by a later editor. I’m afraid such things are just the nature of the publishing beast. But by having a shorter portion edited first, you will get a decent feel for whether someone is the right editor for you, and will be able to be confident in his or her work on your entire book.