Alan Sugar was removed from Israel at the weekend amid growing concern for his safety.