A Course in Miracles says something to the effect that annoyance is actually thinly-veiled rage, that there are no levels of discontent. If this were empirically true in your experience it could be seen, at the very least, that this gnawing feeling is an opportunity to look at annoyance in a different way.
So if there is a “difficult” person in front of you (and another clue to the depth of dissatisfaction is that they are not in front of you presently, but continually popping up in your head as a source of annoyance), you can look right at the person, or the image, and see that the reaction is happening within you. The person, or their image, is just doing what it’s doing, but the physiological response is in you. This is where the conflict is going on, not out there. This is easy to see, if you look honestly.