As per the title of this post, I want to get straight to the point.
If you are setting up VPC Peering with MongoDB, then wait 20 minutes for it to kick in, before you remove any IP Access list entries from the Network Access section and just have the peering CIDR range.
The story:
I’m the Technical Co-Founder of Drivible and started working on the SaaS platform for Car Dealerships a few years ago and have been the primary developer.
A while back we migrated off the previous MongoDB hosting by Mozzler (the company that originally started the project who I was working for) and we migrated to our own MongoDB Atlas account.
With that change we no longer had as precise Networking access restrictions. Mostly just a really complex username and password for the database access.
This had been the biggest security concern that I had for a long time. However my attempts to fix it didn’t work.
We use AWS’s Elastic Beanstalk. This means that Amazon Web Services will spin up a new machine and configure it for us fairly regularly. A machine just needs to burp (well more likely be overloaded for a couple of minutes) and it’ll be replaced.
Over a region with 3 data centers, there’s a bunch of possible IP addresses that could be and those IP address ranges can change whenever Amazon need.
The ideal option is to enable VPC peering.
In this case Virtual Private Cloud peering is where you enable a direct connection between the MongoDB Atlas servers (which as we selected were also running on AWS) and our AWS’s VPC.
The MongoDB Documentation goes into a fair bit of detail about setting up a VPC Peering connection. Including steps we didn’t have to do regarding DNS and Routing.
https://www.mongodb.com/docs/atlas/security-vpc-peering/
I tried to follow the instructions on two separate occasions and both times caused both our Staging and Production databases to become unreachable and our platform to fail as a result.
The 2nd time I did it I did even more research and tried a bunch of things, like ensuring we had Security Groups setup to enabled the CIDR range and more.
However what isn’t listed in their documentation is that once you’ve approved the Peering connection request in AWS on the MongoDB side there’s a check every 20mins to see if you accepted the peering connection. That cronjob needs to run before you can access the MongoDB side and for 2 way data flow.
That knowledge cost us hundreds of dollars to obtain as we got a MongoDB Consultant to help us.
Thankfully the Account Manager provided us with some MongoDB credits to partly offset the initial cost.
The process of even getting a consultant meant there was meetings with their sales team… I mean account manager(s).
Then multiple forms to fill out with plenty of questions.
Then the cost of the half day of consulting.
That half day is meant to cover 2 hours with an engineer on a video call and then they have another 2 hours to analyse your setup, give feedback and create a report.
So I got him to go through and on the Zoom call we setup VPC peering. Even he didn’t think about the 20 min time and after accepting the Peering connection and testing, we were investigating if our EC2 / Elastic Beanstalk setup was somehow not using our default Virtual Private Cloud. Turns out that was fine, it was.
We’ve only got one VPC. It’s a relatively simple setup.
But once it just randomly started working he went “Ohh yeah, it only checks every 20mins”.
Now, it might be that the cronjob is run just a couple of minutes after you accept the VPC Peering connection in AWS. In which case it’ll seem to all just work fine for you. But your best bet is to setup VPC Peering, accept the peering request in AWS… Then go do something else for 20mins.
When you come back, THEN you can remove the old IP address access.
Remember – After setting up VPC peering, wait 20mins before you make the IP address access more restrictive. This is usually the bit that causes an outage if you don’t have it correct.
You probably had 0.0.0.0/0 which enables ANYONE ON THE INTERNET to try accessing your database. Now you’ll have an Internal IP address range that only works if you are within your AWS network.
This doesn’t mean your database is safe. But it’s a LOT safer. No longer will just any script kiddie on the Internet be able to access your network.
A good option now is to setup an SSH jumphost in order to use MongoDB Compass (or SSH port forwarding) so you can access the database from wherever you are.
It looks like a t2.nano server only costs about $5/month and you just need it to have a consistent IP, or use DNS to point to whatever IP it is.
Setup a good SSH Keypair (probably a different one than what you use for deployments) and then use that. Or even better, create an SSH account for each user if you have multiple developers.
Let me know if this helped you.
Cheers!
