Yes, it is fair.
- If it's a direct challenge, you don't need to accept it. Once you do, well, you chose to play.
- If it's your seek, you can set the rating range (custom settings). Again, you chose to play anybody.
- If it's a tournament, well, it's obvious that you cannot choose your opponents in the tournament.
And finally, it's a privilege to play higher rated opponent. These are the games that going to help you to improve.