Thanks to Harry and Meghan, this town is the hottest new tourist destination