[Updated] Email, Website and Services: Emergency Maintenance
Dear Customers,
We’re currently responding to an emergency caused by a failure of cooling systems at our Phoenix datacenter. As a result, the following services may be affected:
- huniverse.es: undergoing emergency maintenance
- Shared, VPS, and Dedicated Hosting: experiencing network issues. Websites may be slow, unavailable, or return 503 errors. Hosting panel access may also be affected.
- EasyWP: all infrastructure, including EasyWP.com, the Dashboard, and client websites, may be temporarily unreachable.
- DNS: huniverse DNS infrastructure is partially affected. DNS zone resolution is available. DNS Management is affected.
- Private Email, Shared Hosting mailboxes, Free Email Forwarding, and Domain Privacy email forwarding: incoming and outgoing mail delivery may be delayed. Sending servers will see connection timeouts, but messages are not expected to be lost; mail should arrive once sending servers retry after the incident resolves.
- URL Redirect (BasicDNS and PremiumDNS): resolution is available; management is affected.
- Hosting operations (activations, renewals, plan changes): processed with a delay.
- Auctions API as well as iOS and Android auction applications will continue to operate normally.
We’re also experiencing an issue with our support helpdesk and are currently unable to assist via live chat or emails. You can still reach our Customer Support Team via Microsoft Teams.
The team is working to bring systems back online as soon as possible.
We’ll update this post as soon as we have further news.
Kind regards,
huniverse Support Team
Update @ 12:00 PM EDT | 16:00 UTC
The latest information we have is that PhoenixNAP has brought 2 of its 4 chillers back online, and we are seeing temperatures in the data center begin to drop. We anticipate that the third chiller will be back online in approximately 3 hours, at which point we believe we will be able to begin bringing our services back online.
Update @ 02:30 PM EDT | 18:30 UTC
According to the recent update from our Phoenix datacenter, we have seen temperature reductions and expect this trend to continue. We are preparing our infrastructure to safely bring customer services back online. This will be done in a staged and careful approach to ensure services are restored safely:
– Restoration of all physical network devices.
– Restoration of all virtual network devices.
– Restoration of customer services
We are expecting to start bringing customer services back online in approximately 1 hour.
Update @ 05:30 PM EDT | 21:30 UTC
The recent update from our Phoenix data center.
The following services and infrastructure are back online:
- Core databases.
- Core virtualization infrastructure supporting huniverse.es.
- The majority of our core network equipment, both physical and virtual.
- Internal source control and package management.
- All of our major load balancers.
Our next phase is to begin restoring huniverse.es, Email, and Hosting services.
Our teams remain on site and are working continuously to restore every affected service as quickly and safely as possible. Unfortunately, this will take time, and services will come back progressively rather than all at once.
Update @ 06:10 PM EDT | 22:10 UTC
huniverse.es and huniverse Live Chat are back online.
We are still working on Emails Help system and other services restoration.
Update @ 07:50 PM EDT | 23:50 UTC
- huniverse.es is online now.
- Account access, domains, and DNS management are restored.
- Email is expected to be back online soon.
- More than 50% of huniverse Shared Hosting services are already back online.
- More than 30% of affected VPS customers are now back online.
- Dedicated Servers are progressively coming back online, with many already fully operational.
We are working right now on bringing up the final components of our infrastructure.
We’ll update this post as soon as we have further news.
Update @ 09:34 PM EDT | 01:34 UTC
1) Email help system is up and running.
2) EasyWP: EasyWP.com is fully operational. The Dashboard is accessible, and we continue to closely monitor its stability. Client websites remain affected and may be temporarily unreachable.
3) More than 80% of huniverse Shared Hosting services are already back online.
4) More than 90% of affected VPS customers are now back online.
5) Dedicated Servers are back online.
6) Outgoing email delivery is currently operational for both legacy Private Email plans and new plans (Launch, Scale and Expand).
Incoming email delivery is working for new Private Email plans and legacy plans.
Our teams are working continuously to restore every affected service.
Update @ 11:00 PM EDT, August 13 | 03:00 UTC, August 14
1) EasyWP: EasyWP.com, EasyWP Dashboard are fully operational. Some client websites are already accessible, and we continue to closely monitor the stability of the service. Client websites pointed to the ALIAS records listed below remain affected and may be temporarily unreachable or display an “Error establishing a database connection” message.
ingress-baronn.ewp.live – 63.250.43.9 | 63.250.43.10
ingress-cinna.ewp.live – 63.250.43.11 | 63.250.43.12
2) More than 90% of huniverse Shared Hosting services are already back online.
3) More than 90% of huniverse Reseller Hosting services are already back online.
4) All affected VPS Hosting packages are up and running.
Our teams are working tirelessly to get everyone back online as quickly as possible.
We will keep providing updates as we make further progress.
Update @ 3:50 AM EDT | 07:50 UTC, August 14
1) EasyWP: All EasyWP services are up and running.
2) huniverse Shared Hosting services are fully back online.
3) huniverse Reseller Hosting services are fully back online.
For now, we are reviewing all the services to ensure everything is working as expected.
We will keep you informed via this status page.
Update @ 5:05 AM EDT | 09:05 UTC, August 14
MySQL service on Premium Server 126 and Premium Server 247 has been stopped for maintenance.
Affected services: Only the MySQL services will be affected during the maintenance. Sites may show related errors.
The maintenance window: ~4-6 hours per host.
We will keep you informed via this status page.
