My gut reaction (that others seem to share) is that the activity might be the problem. I always hate giving up my lunch during the workweek, because it’s my only alone time to unwind during the day.

Try a different activity (group activities can be less intimidating for sure, like “hey I’m meeting a few friends at a park Saturday if you feel like swinging by” because then there’s less pressure on them specifically). Maybe a walk or something on Sunday mornings, or ask him about his hobbies specifically? You can also just communicate via phone and still be friends. Maybe he’s still just working up to the point where he feels comfortable hanging out socially frequently.

Good luck!