Update – Information Regarding the Network Outage

How many sites were affected by the network attack?

Too many. One site is too many to begin with, and we are very sorry for the disruption this has caused in the organizations that we serve. In this case 10% to 15% of our clients who host with us were affected. Only our legacy Tendenci 4 software codebase on Microsoft technology is currently experiencing issues up to and including being offline. None of the newer Tendenci 5 software on Linux and Python is experiencing problems. With newer technology that we have in Linux and Python we can harden sites more and isolate them, therefore, we have no reason to believe that this situation will disrupt service for any of our clients on the Tendenci 5 software.


When will my site be back up?

We are very sorry that we have been unable to restore all Tendenci websites. Providing the Tendenci community with reliable service is the basis for our business and we have been unable to deliver. We are working hard to have at least some of the affected websites back up in a limited capacity today. We will continue to update those affected through posts to blog.tendenci.com, Facebook and Twitter, and via email.


What are my options?

Affected Tendenci are being provided with customized landing pages with alternate contact information, such as phone and email. When ready, we will restore these sites with limited functionality. With the realization that it could take several more days before all Tendenci sites are back up with limited functionality, and the high likelihood that it may be longer before all functionality is available, we understand if some organization choose to move off Tendenci.


What does limited functionality mean?

As an interim step on the pathway to returning all websites to full functionality, we will restore static websites. Temporarily, for a period of several days, Tendenci users will not have the ability to process transactions or make other updates to the website. We know this is not the type of capability you expect and we are sorry. We are mobilizing all available resources to get all websites back up.


Can I move my website to another server?

Unfortunately, the issue we are working to address cannot be solved by moving to another server. Based on our investigation into these attacks, the work we are doing to protect Tendenci websites going forward is essential. Moving to another server would not address the vulnerabilities that were revealed by these attacks.


Has my organization’s data been compromised?

We have no indication at this time that Tendenci user data has been accessed. Our investigation has found that these attacks were related to website traffic and not to hacking into information. That said, we are taking further steps to protect Tendenci user data.


Why wasn’t this problem prevented?

We work hard and invest aggressively in measures aimed at protecting our systems, but in this case it was clearly not enough. We are sorry. After we complete the restoration of our systems, we will examine the vulnerabilities in our systems and practices that left us open to these attacks in order to ensure something like this never happens again.


What are you doing to make sure it is not going to happen again?

One very basic thing we are doing is shutting down the ability for clients to FTP into their site. This created vulnerabilities in the system and can compromise other clients. Generally, the benefits of supporting functionality such as FTP as opposed to Secure FTP (sFTP) on the system that is based on Windows technology do not outweigh the risks to the Tendenci community that is still on Tendenci4.


What about clients that currently use the FTP function?

We can isolate that functionality with some inconvenience to clients on T4 yet still meet the business needs.

For example you can sign up for Amazon Web Services S3 services and we can dynamically include those files into your site. The business need can be met without the risk.


Why are the sites affected not back up yet?
We have put every single resource available, internal and external, including outside security consultants, our top programmers, deployed new scanning tools to see if they can see the patterns our current virus scanners and web application firewalls missed. We are working around the clock, and are doing everything we can to get your sites back up.