Australia's largest energy retailer Origin has been fined more than $17 million for breaches of Victoria's energy rules affecting hundreds of thousands of customers.