I don't have anything specifically against turbos, however if given the choice of a larger naturally aspirated engine or a smaller turbocharged engine, given similar power outputs I'll likely go for the NA engine. There are a couple of caveats to this: If I'm leasing a daily driver, or plan to get rid of it in 4 years, the turbo is fine. Or, conversely, if it's only a 'weekend' vehicle then a turbo would be acceptable too.
My problem with turbos is this: They add another level of complexity to an engine. It's just one more system that has the to potential to fail. What's more, from an engineering standpoint and turbos aside, a small-displacement over-stressed engine will never last as long as a larger displacement understressed engine. There's just no way it can. Eventually something will reach its breaking point.
Look at Chevy's small-block, for example. Relative to their size, they made little power. But they're unbelievably reliable units because of this. The same goes for the old 5.9L Cummins. Sure, they were turbocharged, but compared to todays diesels, it was relatively low pressure, and for the displacement, it didn't make half the torque and HP figures that todays diesels are. And an old 5.9 Cummins will last forever.