Joe said he wasn't worried about his weight after Pun passed