Jeff, I'm mulling over trying to introduce some additional games to the queue of my LGS because their offerings are pretty anemic, but I've only GMed for friends and curious people before. Is there any one piece of advice you have on how this might be a whole different animal, or not, other than "you can't choose who shows up to play so hunker down"?The biggest thing you can do for yourself is to make nice with the store owner and/or manager. Get a date set with them a couple weeks beforehand and ask them to help recruit. Briefly explain the kind of game you have in mind and ask them to steer good folks your way while discouraging the turkeys. I lucked out in that the guys at my store figured this out for me and took it upon themselves to make sure my games got off the ground, but you may not have that luxury. Don't hesitate to follow up with them a week out and a couple days before the game date, just to remind them that you could really use their help. Hopefully they are smart enough to realize that any regular gaming at their store will help them out.
But, yeah. You got to hunker down. If you're running an open game that means you have to be willing to run for whatever nerds, dorks, geeks, dweebs, spazzes, malcontents, weirdos and goofballs show up. That's part of the deal and you need to accept that up front. Some of your players may not be able to play in a non-open table situation because they don't have the social skills required. Sometimes that means your game will lag as you walk people through what you consider to be stuff they ought to have learned in kindergarten. Just try to be mellow and explain as kindly as possible that bogarting other peoples dice (or whatever) is against your 'table rules'. Be firm but don't make a federal case out of this stuff.
The one exception to this "be kind, speak softly" approach I try to maintain is when people are just plain mean. This doesn't happen often, but when someone is scoring points off of other players' emotional vulnerabilities you got to stop that shit cold. My last line of defense is "I run an open game, but I have one absolute requirement regarding who can play: I am the biggest douchebag at the table. Tone that shit down or find another game." That's a rare thing and you probably won't ever need to get that far. If you do, the situation may come down to the dillweed trying to go over your head to the store owner. (I know it sounds ridiculous that someone would try to force you to run a game for them, but I've seen it.) If the owner doesn't back you up, pack up your gear immediately and find a new store.