I had a PhD interview quite some time ago for which I didn’t receive a response about the outcome. I initially didn’t worry about it and didn’t want to pester them. Thus, I simply thought that they took so long because they were waiting for references, still deciding, busy, etc. Yet, I wanted to know the outcome and started to think that it was a bit odd that I didn’t hear from them so I eventually contacted them and was told that they had offered the place to someone else.

Now is it childish, or am I right to be annoyed (that's a slight understatement) about the fact that they couldn’t even be bothered to tell me that I’ve been rejected? I always thought that it was the norm to let candidates know about the outcome post interview (especially when there were only a handful of people being interviewed). Well, that’s how it’s been like for me in the past when I was unsuccessful twice after a PhD interview. I know that I cannot do anything about this, but :-(

It is courteous to reply to unsuccessful candidates in both academia and the "real world", but in my experience this quite often doesn't happen, so I would get used to it. I've had a case in a job where I already worked and I applied for promotion, I didn't know I hadn't got an interview until I was asked if my office could be used for the candidates. I made a complaint and a few hours later a letter appeared on my desk informing me that unfortunately I hadn't been selected for interview. A similar thing happened in another job when a new appointee was announced by a memo on a noticeboard and I was still waiting to hear how I had got on at the interview. If companies don't even have the courtesy to contact their own employees then don't expect anything from other organisations.
I think you have the right to be annoyed though, but I take the attitude that I would not want to work with such rude people anyway!
Actually though I have a positive story from my own uni in that they took a while to decide between candidates for my PhD but the chief investigator kept e-mailing me to update me on why there was a delay, so there are still some thoughtful people around.

I don't think it's silly to be annoyed, I've had similar experiences myself recently and was a bit annoyed/upset that they couldn't be bothered to keep me informed. I think it's just the way these things work, and there's nothing we can do about it.

I was surprised though, because back when I was applying for graduate jobs (in the City), they either let you know the outcome of the interview before you left, or you got a call/email shortly afterwards (the same day or no more than a couple of days later). So to be left hanging on for weeks on end after PhD interviews was a bit strange.