Why Does No Have to be Personal?

Recently, I was asked out to dinner. And, it wasn't a casual invitation, but something with a more a "I want to go out with you" meaning. I declined and what has come of it has been interesting.

To set the scene, I was listening to a friend's band at a local bar and when I left I was approached by someone I had been introduced to that evening. She asked for my phone number. I was hesitant and made several different attempts to dodge the request with various excuses. She was persistent and I gave in. She also asked whether I prefered a phone call or a text (which I think is actually a good question). I said text. Point of decision number one: I should have had something ready to politely say no to the phone number request. I did not and let this interaction go to the next step - my bad.

Two days later, I got a text asking me to dinner. I responded with "thank you, but I can not go for various reasons". Yes, probably lame, but how do you say no to a dinner invitation without leading someone to believe there is still a chance and without making them feel bad?

I didn't hear back and thought everyone had moved on. A week later, I hear through the "grape vine" that several people (including the individual) had asked a friend of mine why I wouldn't go out with this person, that I was rude, what was wrong with me, etc. What had been a simple request that I declined became a discussion of "why" and also a discussion of the various reasons why people thought I had declined -none that I heard were correct. The truth is I just wasn't interested - nothing against the individual, nothing against me - just wasn't interested.

Why does no have to have some deeper meaning? Can't someone just not be interested? And, how do you say no to someone without making it personal?