Timefortea, I had to pull out of two conferences a few years ago, both were central to my research area. I did worry a bit about pulling out as I don't like messing people around, especially senior academics in my field. It doesn't seem to have made any difference in the long run though, as people involved in the conferences have continued to either be interested in my work or have offered help and advice since then.
If you're pregnant with twins, the organisers have absolutely no idea what stage your pregnancy is at, or whether you've had complications. They'd be pretty weird people to hold it against you for withdrawing from their event in those circumstances, even without the swine flu factor!