Secure TCP communication between non-named machines?

Given a bunch of machines who are on the network and who shall talk to each other using TCP.

My demands are:

• Data must be encrypted.
• Data must not be sent to an unauthorized machine even if it is on the same network.
• Data must not be received from an unauthorized machine.
• Machines know only the IP addresses of the other machines.

For a better imagination, think of a chat network where only authorized clients are allowed to participate, and all communication shall be encrypted.

My initial idea was to use a TLS certificate:

• Each authorized machine has a copy of that certificate.
• Using the certificate the connection can be encrypted.
• Using the certificate on the server side it can be used to verify the server.
• Using the certificate on the client side it can be used to authenticate a client.
• As unauthorized machines do not have the certificate, they can neither decrypt, send nor receive data.

In theory, this should work out as expected. In practices, it doesn't.

One of the problems is that the machines only know each other's IP address, and that they need not be bound to a domain name. As certificates are not recommended for IP addresses (and Node.js won't even validate a certificate with an IP address as CN), and the IP addresses may even change over time, I started to think whether that's the correct approach.

Of course, I could start implementing all this by myself: An encryption layer, an authentication schema, ... but I don't run after that. If there's a more convenient way of solving this, I'd appreciate this.

So, basically it comes down to use an TLS certificate for encryption and authentication, not because of the binding to a domain name, but because a specific certificate is present.

How would I implement this and solve the above mentioned issues? Or - am I running in a completely wrong direction? How might I solve this in a better way?