As White

what's the most annoying opening or line to face

against nf3, e4, d4 or c4

for me it's the stonewall dutch against d4, it's just so hard to break

against e4, i just always loose against the french

As white nothing is annoying to play against.
However, less common replies of black cause white more trouble, as the black player has accumulated more experience with it, while white encounters it seldom if at all.
I guess Stonewall Dutch and French are less played than 1 d4 d5, 1 d4 Nf6, 1 e4 e5, 1 e4 c5 and this explains your difficulty.
If you alternate between 1 e4 and 1 d4, then that does not help you. Pick one and stick to it.

I find it diffucult to face the Berlin Defense in the Ruy Lopez, I usually play the Anti-Berlin with d3, but then I feel kind of stuck, because I'm not able to find an active plan there.

#1
Ok I know what to play..

Open Sicilian, too much theory, I am deciding between Alapin and Closed.

Struggled with french too, now I play exchange variation with Nf3 and c4 with ok results.

Against Stonewall: stick knight on e5 and launch queenside pawns, you should be faster than black on kingside.