The problem is... everybody is a single-issue voter, whether they want to admit that or not.
How does the saying go? "Everything is always in the last place you look, because once you find it, you stop looking." I think the negative view of a "single-issue" voter stems from a problem of perception. We like to think of voting as a matter of selecting a candidate, when it is better modeled as a matter of eliminating competing candidates until we're left with a single option.
Face it - the chances that a political candidate will exactly match your views on all significant topics of political discourse is virtually nil. What you are left with, in that case, is an attempt to find the candidate that best matches your values. The logical and almost instinctive way to do this is to decide what issues matter most to you, and then eliminate candidates who don't match your views on that subject. If you have more than one candidate left, then you go on to the next most important issue. Lather, rinse, repeat until you are left with one person, the one who most closely represents your views.
This is where the "single-issue" voter comes from, and it's why we are all "single-issue" voters. When you follow this very natural selection process, you will always have a definite reason for rejecting a candidate. You'll find what you were looking for - a reason to reject - and after you find what you're looking for, you'll stop looking.
"She's opposed to gay marriage."
"He supported the war in Iraq."
"She supports legalizing drugs."
So if we're all single-issue voters, what does it mean then when someone explicitly hauls out that title as an accusation? Remember that the process of elimination is personal. You are taking the issues that are important to you, and deciding which candidates to eliminate based on your personal values regarding those issues. That should be enough to provide an accurate translation of the single-issue voter accusation, as follows:
"You're a single-issue voter"
"You don't share my values, therefore, you're wrong."
In other words, it's an accusation that clocks in with all the weight of a third-grader's taunt - if that. Yep, I'm a single-issue voter. So are you. So let's stop arguing like a couple of kids on a playground and at least try to act like adults, m'kay?