I clean it off but not often. I use a variety of brushes, water with vinegar, plain water, and rags.
Brushing with water with vinegar takes some paint off. It is necessary on very rough rough-sawn boards. It may wear off or brush off on smooth surfaces but, after two or three years, I gave up...