Is it possible to configure SQL Azure DB to only accept connections from my Azure virtual private network? I would like to give access to it for all VMs in this private network as well as users connecting via Point-to-Site VPN client.
SQL Database is not part of your VNET (it's a separate service, so the individual IP addresses cannot be added to the firewall. While you can simply enable internal Azure endpoints to connect to your SQL Database connection, this would allow for any deployment within Azure to connect to your database.
If you have services running in one or more cloud services, you can open your database firewall for each individual service's VIP.
Everything you wanted to know about SQL Azure firewalls: Azure SQL Database Firewall