Tendenci – The Open Source AMS Blog

A security update, and that’s what I get for being overly optimistic, but our initial scans found some issues. It is important to remember that a server that is on the Internet accepts inbound traffic on port 80 and 443, but it replies and can call out.

Our remediation plan called for building all new servers and porting the data, but if there is something that can call out once we open those ports then we are right back at ground zero. Possibly worse. And that is not acceptable.

I’ll let everyone know the minute we can let some testing begin. Maybe I’m being overly cautious at this point but given the situation, I believe it is warranted. Our current task is reconfiguring sites and we are having some challenges but are solving them one by one. (8.3 filenames get restored from backups for example and have to be removed again. That type of thing.) – Ed

Latest Update

Yesterday at 1pm CST Tendenci CEO Ed Schipul hosted a phone call to address questions from the Tendenci community, especially those affected by the recent network attack.

Points that were covered during the call:

• We understand how much this affects and recognize the important responsibility we have of protecting our community. We are sorry that the process of getting your website back up has taken much longer than anticipated.

• We are working to get the sites back up as soon as possible. Based on what we know today, we anticipate that it will be the end of this week and possibly into the start of next week before all sites are back up. It is in the best interests of your organization and your website users to make sure the sites pass a security scan before we bring them back online.

• Our focus is on doing everything in our power to get you back up and running, and we are putting measures in place to make sure this does not happen again.
Questions

What is worst case scenario for sites coming back up?

Knowing what we know today, we anticipate restoring websites by Friday, December 12th. That said, the worst case scenario would be that we experience additional attacks or other complications, in which case we could be further delayed until Monday, December 15th. We are doing our best to get sites back up securely this week.

Was this attack a sudden brute-force attack only performed over the past week, or was it a slow process over time with virus files on the server for an extended period?

Our investigation is still ongoing and we will share a full picture with the community on the nature of the attack when this is behind us. Once our tech team has gotten sites back up we will create a forensic report with the data we are recording on the servers. Right now they are focused on getting the sites up and in post action review can summarize what took place.

Do you foresee this outage affecting search engine rankings?

It is likely that there could be some impact on search engine performance. Most of the clients who have experienced the outage have a landing page up. This should have a neutral affect on your site’s search performance. But internal pages may be affected on search for the short-term.

How will you make this right for affected customers?

We will work hard to earn back your trust. This was a criminal attack that has created financial losses for everyone involved and we are heartbroken that many of our long-term customers have been impacted.

Our team continues to work round the clock to restore our system and those sites that have been affected by the recent DDoS attack.

At 1pm CST on Monday, December 8th, we will be hosting an conference call regarding the outage.

During this call, our CEO, Ed Schipul, will provide updates and address questions.

Please email any questions to communications@tendenci.com prior to the call to ensure we address as many as possible.

For dial in information, please submit a ticket to helpdesk.tendenci.com

US Toll-Free: 1-888-619-1583

Participant Passcode: 250006

The Tendenci team is continuing to work to restore our systems and get websites affected by this week’s DDoS attack back up. We are taking this issue very seriously and have mobilized all available resources to address it. We are sorry this was not accomplished for all affected sites by the end of the work week
as we had anticipated. Our team will continue to work through the weekend to resolve this.

For the websites we control, we are getting landing pages up. For those who have websites we do not control, we are sending records to update ip address, as we lack access to do so.

Please contact us through helpdesk.tendenci.com if any of the contact info on your landing page needs to be updated.

As we work through this, we are learning how we can improve our systems, our protocols, our response procedures and our communication to prevent a similar occurrence in the future. It has taken too long to restore our systems and get all affected websites back up. We are sorry. This is unacceptable and embarrassing. We promise to do everything we can to get all websites back up as quickly as possible.

How many sites were affected by the network attack?

Too many. One site is too many to begin with, and we are very sorry for the disruption this has caused in the organizations that we serve. In this case 10% to 15% of our clients who host with us were affected. Only our legacy Tendenci 4 software codebase on Microsoft technology is currently experiencing issues up to and including being offline. None of the newer Tendenci 5 software on Linux and Python is experiencing problems. With newer technology that we have in Linux and Python we can harden sites more and isolate them, therefore, we have no reason to believe that this situation will disrupt service for any of our clients on the Tendenci 5 software.

When will my site be back up?

We are very sorry that we have been unable to restore all Tendenci websites. Providing the Tendenci community with reliable service is the basis for our business and we have been unable to deliver. We are working hard to have at least some of the affected websites back up in a limited capacity today. We will continue to update those affected through posts to blog.tendenci.com, Facebook and Twitter, and via email.

What are my options?

Affected Tendenci are being provided with customized landing pages with alternate contact information, such as phone and email. When ready, we will restore these sites with limited functionality. With the realization that it could take several more days before all Tendenci sites are back up with limited functionality, and the high likelihood that it may be longer before all functionality is available, we understand if some organization choose to move off Tendenci.

What does limited functionality mean?

As an interim step on the pathway to returning all websites to full functionality, we will restore static websites. Temporarily, for a period of several days, Tendenci users will not have the ability to process transactions or make other updates to the website. We know this is not the type of capability you expect and we are sorry. We are mobilizing all available resources to get all websites back up.

Can I move my website to another server?

Unfortunately, the issue we are working to address cannot be solved by moving to another server. Based on our investigation into these attacks, the work we are doing to protect Tendenci websites going forward is essential. Moving to another server would not address the vulnerabilities that were revealed by these attacks.

Has my organization’s data been compromised?

We have no indication at this time that Tendenci user data has been accessed. Our investigation has found that these attacks were related to website traffic and not to hacking into information. That said, we are taking further steps to protect Tendenci user data.

Why wasn’t this problem prevented?

We work hard and invest aggressively in measures aimed at protecting our systems, but in this case it was clearly not enough. We are sorry. After we complete the restoration of our systems, we will examine the vulnerabilities in our systems and practices that left us open to these attacks in order to ensure something like this never happens again.

What are you doing to make sure it is not going to happen again?

One very basic thing we are doing is shutting down the ability for clients to FTP into their site. This created vulnerabilities in the system and can compromise other clients. Generally, the benefits of supporting functionality such as FTP as opposed to Secure FTP (sFTP) on the system that is based on Windows technology do not outweigh the risks to the Tendenci community that is still on Tendenci4.

What about clients that currently use the FTP function?

We can isolate that functionality with some inconvenience to clients on T4 yet still meet the business needs.

For example you can sign up for Amazon Web Services S3 services and we can dynamically include those files into your site. The business need can be met without the risk.

Why are the sites affected not back up yet?
We have put every single resource available, internal and external, including outside security consultants, our top programmers, deployed new scanning tools to see if they can see the patterns our current virus scanners and web application firewalls missed. We are working around the clock, and are doing everything we can to get your sites back up.

We are currently in the process of creating landing pages for the approximately 40 websites that were effected by the network outage.

As noted, previously we are mobilizing all available resources to address the situation as quickly as possible, but with new information it is unlikely that affected websites will be fully functional for several more days.

We are working to have at least some of the affected websites back today, in the meantime each individualized page will state that the network outage is due to your service provider.

We at Tendenci are sorry that the network outage that is affecting approximately 40 Tendenci websites remains unresolved. It has taken some time to understand the full scope of the issue and, unfortunately, getting all our systems back online has taken longer than anticipated. We understand that a website is
critically important to every organization, and we take our responsibility – and the trust of Tendenci users – very seriously. We are mobilizing all available resources to address the situation as quickly as possible, but it is unlikely that affected websites will be fully functional for several more days.

Here is what we know:

• On Wednesday, 11/26 at least one Tendenci server experienced a URL redirect attack. Spammers use this type of attack to send website traffic to other desired websites. In this case, traffic to certain Tendenci websites was redirected to a website selling handbags.

• The Tendenci team quickly responded, but this was followed by an additional distributed denial-of-service (DDoS) attack on our network. This same type of high bandwidth attack took down Xbox Live this week.

• The network outage is affecting approximately 40 Tendenci websites. Only websites on the legacy Tendenci 4 software.

• In our efforts to restore network service, we have uncovered a number of things that can be improved in our security systems and practices that left our network vulnerable to the DDoS attack. Addressing these vulnerabilities is a necessary first step in restoring the network and getting all Tendenci websites back up.

• We are working hard to have at least some of the affected websites back up in a limited capacity today, but it may be the end of the week or longer before all affected sites are restored.

We understand that having a website go down for days on end is unacceptable, and it breaks my heart to know that this week’s events may have shaken the community’s trust in Tendenci. We are working around the clock to restore our systems, and we will take what we learn during this process to improve security protocols for all Tendenci websites.