Yeah, it's the relative size of the load. When I've towed loads that more than effectively double the weight of the HL, it drinks badly.
I towed my neighbours little tent trailer back from his vacation property and I didn't notice a real fuel economy penalty. Towing 4500lbs of reno waste in stop and go? Ugh. Terrible. Glug glug glug.