You know it is wrong. Did you inform @kyle-moore that by doing that you would be able to track all his past, present, and future transactions? How much money does that poor guy have?
Of course you did not. Otherwise, nobody would ever connect to any of those 20 or so servers maintained with the sole purpose of collecting uninformed people's xpubs.
If the xpubs collecting mafia were honest they would at least add a warning sign/text saying something like "By connecting to this server you accept that the maintainer can see all your finantial activity and your balance forever. Are you okay with that?"