I don't leave the side curtains on but I keep the middle section on and up all the time. I have been doing this for 7 years with the same setup and have never had a problem. That even includes some trips on the interstate at 65 to 70 mph.
My guess would be it all depends on what your setup is like in terms of attachments and aerodynamics. If it isn't connected very well or wind can't get over and around it as high speeds then you are going to have issues.
This is the boat with out the plastic and canvas on. they are snaps and zippers spaced about 8" apart. They go from the hardtop to the winshield and wrap around the sides to the back and down to the deck
I think I will travel with them off until
I at least try it on the lake all set up.