2021-03-10 - Large scale infrastructure issue – GO7 PSS Support

Dear Customers,

Due to an error in our infrastructure provider, a large scale DNS issue is causing websites, and systems not to be available.

Mar 10, 00:54 UTC: We are experiencing a large-scale DNS resolution issue, might affect entire system operations, Provider's Technical teams are currently investigating the issue

Mar 10, 01:48 UTC: a large-scale DNS resolution issue identified Google DNS and OpenDNS to have issues. Provider is still investigating this issue.

Mar 10, 02:14 UTC: We understand that remediation actions are on the way by the provider.

Mar 10, 02:27 UTC: A fix was applied, we are monitoring results.

Mar 10, 08:17 UTC: as some of our customers are still experiencing issues, please try to clear your DNS cache by resetting your router, or try to access the system via this link: https://clients.aerocrs.com/loginpage.asp

Mar 10, 09:10 UTC: Our provider notified us that some local ISP (internet service providers) will need to flush their DNS in order to expedite resolution of the issue, they are contacting multiple ISPs in order to take action, if you are still encountering the issue, please try to contact your local ISP and request them to flush their DNS cache.

Mar 11, 09:00 UTC: Our provider provided us with a RCA report, if required please ask support and we will send it to you, this marks the resolution of the issue - if you are still facing issues, please let us know.

Workaround - Set your computer's DNS settings to Google DNS 8.8.8.8 and 8.8.4.4

Technical information: Earlier today, an expired DNSSEC certificate key for the incapdns.net domain resulted in DNS resolution failures for those with DNS providers that rely on DNSSEC for validation, like Google and OpenDNS. As an immediate mitigative step, DNSSEC was disabled, which restored DNS to full operation. Though the issue has been mitigated, customer experience varies based on the expiration of DNS record time-to-live (TTL) values in resolver caches. Additionally, some ISPs may cache DNS records locally, longer than the TTL. This could potentially impact DNS propagation time.

We will provide additional details once updated.

If you are still unable to access the system, please contact our support.

Comments