My thoughts on what might have gone wrong:
-for that depth or shallower, you probably want to be out there about 3 hours earlier
-if you can't get out earlier, drop to 35 ft and 6 colors
I would expect action closer to Wolfboro bay or Rattlesnake...never had much luck trolling in between like you did.