The e-brake lever was sure close to the driver, but it seems to me that when the lever was released, it is pretty much flush with the rest of the center console / tube / column / whatever-you-call-it. The the e-brake was engage, it was poking out towards the driver every so slightly. I'm pretty thin myself, so my thigh was a good 3 inches or so away from the lever.
For bigger folks, their leg might butt right up against it. But it shouldn't be too much of a problem.
-Rick