Apology. Never have casual conversations with people. Its rude.
Also what Peter Shor said.
I would also add that if you ask the question, then usually the right thing is to offer this person to be a coauthor. If the person is decent and it was really casual they would thank you, ask you to put them in the ack, and decline to be a coauthor. If it was really minor and they still stay as coauthors, they would feel thankful to you. This might be beneficial later on - they might work hard on the paper and push it forward, you have this person as a coauthor (beneficial if they are well known), they would feel they "owe" you, etc.
The other possibility is that this person is a free rider and becomes a coauthor without doing anything and not having contributed anything. This is useful thing to know - you carefully avoid working/talking with this person. You gained useful information for little cost.
And this is all to avoid the worst case - you do not offer this person to be a coauthor and they become bitter and clinging to their guns and bibles (not necessarily in this order). They start spreading stories about you that you had stole their excellent paper. They shoot down your papers whenever they can (referee reports, PCs, etc). As your career collapse you become bitter and clinging to your gun - and you move to Texas. And all of this just because you did not want to offer them a coauthorship - sounds like a bad deal to me.
In short, there is enough glory for everybody - always mistake on the side of sharing glory... Publishing papers is not a zero sum game - there are long terms considerations that arise out of the fact that this a repeated game with side effects.