I think this was the winner! Performance is much improved when I switch over to 3.3V (and repeatable that it returns to “bad” operation if I switch back to 5V). I am going to experiment a little more, but I am now pretty confident this will work.
Thanks to @DBMandrake for all your help on this!