Rackspace hosted Exchange suffered a devastating interruption beginning December 2, 2022 and is still continuous as of 12:37 AM December fourth. At first referred to as connection and login issues, the guidance was eventually upgraded to announce that they were handling a security event.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be fixed.
Clients on Buy Twitter Verification reported that Rackspace was not responding to support e-mails.
This has been quite the day with #Rackspace. Every hosted exchange client has been down for 14 hours or two. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace consumer privately messaged me over social networks on Friday to relate their experience:
“All hosted Exchange customers down over the past 16 hours.
Not exactly sure how many business that is, but it’s considerable.
They’re serving a 554 long hold-up bounce so people emailing in aren’t familiar with the bounce for several hours.”
The official Rackspace status page provided a running upgrade of the interruption however the preliminary posts had no details aside from there was a failure and it was being investigated.
The first official upgrade was on December second at 2:49 AM:
“We are examining an issue that is affecting our Hosted Exchange environments. More information will be posted as they appear.”
Thirteen minutes later Rackspace started calling it a “connectivity issue.”
“We are investigating reports of connectivity problems to our Exchange environments.
Users may experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their e-mail client(s).”
By 6:36 AM the Rackspace updates explained the continuous issue as “connection and login concerns” then later that afternoon at 1:54 PM Rackspace revealed they were still in the “investigation stage” of the failure, still attempting to find out what failed.
And they were still calling it “connection and login concerns” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
Four hours later on Rackspace referred to the situation as a “significant failure”and started using their consumers free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround up until they comprehended the problem and could bring the system back online.
The main guidance mentioned:
“We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any additional issues while we continue work to bring back service. As we continue to work through the origin of the issue, we have an alternate service that will re-activate your ability to send and get emails.
At no charge to you, we will be providing you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 till additional notice.”
Rackspace Hosted Exchange Security Occurrence
It was not until almost 24 hours later at 1:57 AM on December 3rd that Rackspace officially announced that their hosted Exchange service was struggling with a security occurrence.
The statement even more revealed that the Rackspace technicians had actually powered down and disconnected the Exchange environment.
“After more analysis, we have actually figured out that this is a security incident.
The known impact is isolated to a portion of our Hosted Exchange platform. We are taking necessary actions to evaluate and secure our environments.”
Twelve hours later that afternoon they updated the status page with more details that their security group and outside professionals were still working on solving the failure.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has not released details of the security occasion.
A security event typically involves a vulnerability and there are two extreme vulnerabilities currently in the wile that were patched in November 2022.
These are the two most current vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to check out and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an attacker has the ability to run harmful code on a server.
An advisory released in October 2022 described the effect of the vulnerabilities:
“A verified remote opponent can perform SSRF attacks to escalate privileges and execute arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted versus Microsoft Exchange Mail box server, the aggressor can potentially access to other resources by means of lateral movement into Exchange and Active Directory site environments.”
The Rackspace interruption updates have actually not indicated what the specific issue was, just that it was a security occurrence.
The most existing status upgrade as of December 4th specified that the service is still down and customers are motivated to migrate to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make development in resolving the occurrence. The schedule of your service and security of your data is of high significance.
We have dedicated substantial internal resources and engaged first-rate external proficiency in our efforts to reduce unfavorable effects to consumers.”
It’s possible that the above noted vulnerabilities are related to the security incident affecting the Rackspace Hosted Exchange service.
There has been no announcement of whether customer details has been jeopardized. This event is still continuous.
Included image by Best SMM Panel/Orn Rin