(granted, you did mention that you could shoehorn this but) I don’t think a fragment of any sentence needs to have the sentence make sense if you remove the fragment; we jam fragments that are required to understand a sentence into all sorts of locations of sentences, all the time.

The honest answer is that we don’t really have any hard rules about comma usage (as you point out, the sentence would work just as well without any commas), broadly, so people kind of just go vibes-based, most of the time.

I feel like “did so with poor grammar” very obviously doesn’t feel like a tack on to a sentence (like, starting with a verb wouldn’t make sense) so I’m inclined to disagree but I’m anal about comma placement so maybe the average person would.