The record store you are thinking of is Used Kids. Years ago, that building caught fire and the basement Used Kids location had to move. (Interesting side note, Used Kids put their damaged records on the curb, and a friend of mine found a original copy of Die Kreuzen's "Cows and Beer" and it only had a bit of a smoke smell.)
Used Kids is still around, and still the best record store in town (IMO). It's on the 2nd floor of a building next to the old spot. Lost Weekend, Magnolia Thunderpussy (for new stuff) are a couple other record stores. (And there is an Urban Outfitters too, if you want to experience the world's biggest record store
As far as non-record shopping - Columbus has lots of great neighborhoods and lots of great restaurants, and, apparently, is the #4 shopping city in America (?). German Village, Short North, Clintonville, Olde Town East, Grandview are some of the different hoods with food and shopping and bars and whatnot. Downtown is ok, mostly for restaurants.
I say avoid OSU campus because it's awful, but that's me. (However, the Wexner Center for the Arts usually has something interesting going on).
Not sure exactly what you are looking for but, I live in Columbus (Worthington, excactly) and think it's a great city.
Here is a helpful website:
www.columbusunderground.comHopefully this is a little helpful. Let me know if you are looking for something specific.