So, its all done by the user client?
Yes. The client uses what's called "remote attestation" to verify its talking to, not just official servers, but official server code published on github. Read more about it here: https://signal.org/blog/building-faster-oram/
To put it simply, they're using the same technology that allows DRM protected videos to play back on your computer/phone/tablet, but against their own servers, to ensure its not a rogue host or code.
Your profile, like everything else on Signal, is also end-to-end encrypted. Your name and profile picture do get shared with whoever you chat with, groups or individuals. If you don't want your name and profile picture shared with randos, either don't set them or don't chat with randos.