Horror stories about detainments at the border have also soured some from visiting during Trump’s second term