Properly implemented, we consider a known IP address to be an effective second factor. As far as it represents a physical location, an IP address is of the category Something You Are. Almost like a bio-metric.
Of course, it all depends on how well that IP address’ network is protected. If anyone can walk into your client’s waiting room, plug into an Ethernet port, and get a connection, so much for the second factor. And if anyone can connect to the VPN with nothing but a password, the IP address degrades down to Something You Know, and there goes your second factor.
The good news for you is that if either of those things happens, it constitutes a breach of your client’s systems and controls. Likewise, protecting passphrases is the responsibility of the client. So about the only way a malicious actor could exploit your authentication controls would be by first breaching a client system. I call that sitting pretty.
You have probably heard that is possible to spoof an IP address. True, it is not that hard to make incoming traffic appear to be originating from a false IP address, but it is very difficult to ex-filtrate data back to that spoofed IP address, especially with SSL on the pipe. For any realistic threat model, you can consider IP spoofing to be effectively impossible.
So go for it! It’s not as perfect as trusted device but I’d say it beats face recognition. I think you can call this true two-factor authentication, but it does make some assumptions about network security.