At Oscarsborg you can find small secluded beaches for a nice swim. Try out The Commandants beach at the north end of the island.