Well, it doesn't sound like you are worried about schools, so that makes it a bit easier. Williamstown is a lovely place to live, but expensive to buy in. Commute if you are working in Port Melbourne should be ok, but bear in mind that if you are driving the West Gate Bridge can be a nightmare in peak hours. There is a train from Williamstown to the city and then a tram to port Melbourne, but depending on where you need to get to, that total trip could take up to an hour.
As for other things, once you get here you will need to organise Medicare, tax file no., and bank accounts. The bank accounts you may be able to do from the uk, but the others I believe you actually need to be here for.
Williamstown does have lovely Cafes near the waterfront, and some beautiful walks. Are you looking to buy eventually or just rent? If looking to buy, your potential budget will dictate where you should look. You will need probably at least $1 million in Williamstown, but some surrounding areas could be slightly lower price wise, eg. Newport, Altona, Yarraville. I like the beaches on the other side of the bay, and it is where we live, but the commute to work would be much worse.