My rule: plan mode when the change touches more than one file or when I can't describe the end result in one sentence. "Make the button blue" is neither.
Downside of always planning: you get used to skimming plans. A plan you don't read is just a slower prompt.