Categories
Geniuslink News

False Positives & Broken Communication Lines Cause DNS Hell

 

What else can 2020 throw at us? Turns out the year isn’t over yet and we’ve got a bit of a story for you…

In short, starting at approximately 22:30 pm PST on December 8th and gradually becoming more prominent over the next hour the geni.us domain was partially unavailable due to a DNS error. This broke a number of links using our service and thus clicks from these links wouldn’t resolve. The issue was fully resolved by 2:00 am on December 9th. 

Our sincerest apologies if this impacted you or your links during this time.

Ready for the details? It’s a bit of a crazy one…

Late in the evening of December 8th, the “geni.us” domain was put on “server hold” by Neustar, the company that administers the .US domain registry, essentially hijacking our control over it.

As I learned tonight, a “server hold” is what they do when they believe a domain isn’t being used in a good way.  For example:

…Following the discovery of a site being used in violation of the Acceptable Use Policy, Neustar works with relevant authorities within the structure of its policies to determine an appropriate course of action, which from a domain name perspective may include placing the domain on ‘serverHold’ (removing it from the DNS) or deleting the domain name.
Tackling the Opioid Crisis by Cutting off Online Sales in the usTLD

Why would they possibly do this? Great question and after multiple phone calls and many emails in the wee hours we got to the bottom of it…

First, it’s important to quickly note that Geniuslink is primarily used by Creators to recommend products to their audiences on Amazon and other reputable online stores (like 99.99% of our clients). While we do see an occasional bad actor try to use our service for malicious things we’ve got an ever-improving process to quickly catch them and ensure the links they created do no harm (eg. http://geni.us/eVKm1A).

Link no longer available

It turns out that Neustar placed a “server hold” on our domain because they got a report of our domain being used in a malicious manner. They claim that after reaching out to our registrar (the company we purchase our domains through, including geni.us), and asking them to alert us to this claim and fix it, then waiting 12 hours, they acted.

Turns out we never heard from our registrar on the issue. We are still working on getting to the bottom of why.

Simple enough? Nope, things get even weirder!

That claim of malicious activity, while Neustar didn’t say who it came from, lines up very closely with a claim of malicious activity we did get from another company called NetCraft (they monitor the shady areas of the internet for bad stuff, like phishing attempts, on behalf of major companies).

We periodically get “takedown” requests from NetCraft and they are super helpful for when stuff falls through the cracks in our monitoring activity (and thus help us continue to evolve our process).

BUT the most recent two requests from NetCraft, that correspond relatively closely with the timeline from Neustar, were, for the first time I can remember, false positives!  They were two perfectly legit links to books on Amazon. Exactly the thing we do for our many clients around the world every single month!

See for yourself: Incident report/link one and incident report/link two. 

Incident Report

It only took a quick note back to NetCraft to bring them to realize their mistake and resolve the situation!

Email

Unfortunately, this information didn’t appear to flow back to Neustar and thus created the cycle that led to tonight’s outage for the geni.us domain.

One has to scratch their head about how a false positive about a benign affiliate link to Amazon could cause such a negative chain of events but it is, after all, 2020.

Again our most sincere apologies for any impact this may have caused you and we sincerely appreciate all of your support.  Please reach out if you have any questions or want to discuss tonight’s issue or any issue at all.

Rest assured that while this was a total fluke we’ll still be using it as an opportunity to improve our processes to avoid any similar situations in the future.

Thanks for bearing with us.
-Jesse (CEO), Jesse (CTO) & the Geniuslink Team

Update: December 9th

We heard back from Neustar later this morning with some good news: The gist is that the geni.us domain has been correctly classified and they will never place us on “server hold” again!

Email