I brewed 3-gallon all-grain batches on the stovetop for years, and then recently reduced my standard batch size to 2.5 gallons because I found with 3-gallon batches I wasn't letting myself brew as often as I'd like. 2.5 gallons is enough volume for me, and the small batch size does make the day go faster and easier. It would still be even faster and easier to do extract, but I'm just so rarely impressed at the quality and variety of extract beers that I can't help but stick with all-grain for maximum control over the recipe, especially considering fermentability. And yeah, it's so easy to squirt out 2.5 gallons of all-grain, why screw up a good thing.
Of course, other efficiencies can also be gained to save time, such as by cutting mash time down to 40 minutes, cutting boil time back to 45 to 60 minutes, brewing in a bag and thereby skipping a sparge, etc.
In any case, I definitely agree that each brewer needs to do whatever he needs to do to make him happy. If you've been making 5-gallon batches and following all the rules, but growing tired of it, or you're annoyed at certain parts of the process, ask yourself if there's something you could do different. The end result will most likely be the same -- you'll still be making good beer.