The weather in Victoria can be hugely variable - not just from day to day (bikini one day, balaclava the next) but almost microclimates within 100 miles of each other. Ballarat for example is going to be much hotter on hot days and much colder on cold days than Geelong and it is less than 100km away. The winters can be bitter - dont be misled, the wind whips up from the South Pole and there is nothing to get in the way! (Fond memories of Flinders St station in July

) The past few years have been a bit unrepresentative as there hasnt been as much rainfall as average and the dams supplying Melbourne are low and other water sources are drying up (Ballarat's lake is totally dry for example). Contrast that with Gippsland which has had a higher rainfall (and some flooding) and looks more like a green and pleasant land than do the western areas.
Skiing is a possibility but from Geelong it would take you about 4 hours to get to the snow.