The more I think about it, a good middle ground may be to just partial mash. Even if I only got 50% efficiency, I could just do a brief 30 minute mash and be able to get some base malt in there as well.
I could top up with light DME. Wouldn't take anymore time than my extract method does and would probably yield a more flavorful wort.