I went to one of these at the beginning of my studies, although in a hostel-type place, not a swanky hotel! I remember feeling exactly as you do but in truth, it was quite enjoyable. You probably will find that many of the people you meet at this residential weekend, you will meet at various conferences and/or classes throughout your studies. We had to give a short presentation of our still very raw proposals and get feedback from the various academics. Terrifying at the time but useful and given that feedback, constructive or otherwise, is a constant feature of doctoral work, it is good to get used to it from the get go.
It is also useful to hear other people's research, even if not in your field as I feel, it broadens your appreciation of the range of studies there is out there.
Try to enjoy it and don't do too much swimming on your own;-)