That happened to my fridge light where it wouldn't always alight when I opened the door. The flicking the black thing sticking out of the light would turn it on. I don't remember more of what I did other that firmly flicking this switch up, down and any other direction thinking I'd dislodge whatever was making it flaky and by golly, it worked. I have not seen the problem since. I think it did this 3 or 4 years ago. I must add I don't use the RV very frequently, so maybe if I used it more often, I'd have to do something different.
I think in your case you might have to take the assembly apart to check it out. I have done this with other light switches and found dirt and residue, bad contacts, bad switches, bad sockets and such things. Don't forget to turn off the power.