Jacob,
Having just bought my first minivan in 2003 (Odyssey EX), I took part in Car and Driver's 2004 minivan comparo and had a ball driving all of these vans at un-family speeds
The Ody stood way above the others on driving fun, and for me coming out of a manual Passat 1.8T, I needed the van to drive like a car. The 2004 Sienna's handling was nice with the stiffer awd setup, but way too soft in fwd form. We kept that van for nearly 10 years and bought a 2012 Odyssey EX which, as others said, hits the sweet spot on price vs value.
The Sienna SE was in my sights in 2012, but no way I could justify the extra 4k$ over the Ody EX. Plus, dw would have destroyed that low front airdam on the first grocery day, and would have blamed the van for being too low. I was not crazy about the 3rd row's plastic hinges taking up the rear well's floorspace
What you missed: the SE has a stiffer suspension and unique 19" wheels. That big cubby in the trunk's left wall is designed to store the 8th seat (or glorified armrest IMHO). The SE trims takes the LE up a notch and out of the rental class. However, at the Ody EX price level, the Sienna LE is quite drab inside.
Edmund's called the current Odyssey the S-Class of minivans, and I think they're right. I like my new van overall, but interior materials had higher quality in the '03.
Oh, and on that plywood thing, I had a ball at the Home Depot with the 2003. Picked-up two 4x8's, had left the 2nd row buckets at home but had the rear single-piece bench up with the 3 kids seated on it. Slid the sheets under the bench, closed the hatch, no issue with the rain.
Face on the macho guys with big 6' bed trucks: priceless.