Tendenci – The Open Source AMS Blog
Community Blog for Open Source AMS Makers
Happy Holidays from the Tendenci Team!
Our offices will be closed from 1pm on Wednesday, December 24th through 8:30am on Monday, December 29th.
In the meantime, leave us a message on our mainline at 281-497-6567 and we’ll get back to you as soon as possible next week or submit a ticket to helpdesk.tendenci.com.
We have been working to get all sites back up safely and securely that were affected by the recent DDoS attack on our T4 legacy platform.
As of this morning, we have 60% of sites that were affected back up online.
We have sent DNS information to a number of other clients whose sites will be restored once the DNS updates have been completed. We are still working on restoring a few websites. Sites that were image heavy have taken longer to restore, as many images were corrupted during the attack on the system and removed during the subsequent security scans.
We are continuing to bring sites back up throughout the day
Currently sites are experiencing limitations related to sending emails, updating via the wysiwyg editor, and exporting/importing data. Today we are making steps to bring email back up online to the sites. When we have an update for timeline on email we will be posting to the blog.
Thank you for your patience as we work to secure your site and your data. If you have questions about the status of your site please submit a ticket to helpdesk.tendenci.com or email us at support@tendenci.com.
We have started the process of bringing sites that were affected by the recent DDoS attack back online with limited functionality.
So far we have successfully restored 10% of the sites that were taken out by this attack. We will continue to bring sites online today.
Currently, sites that are being brought up have limitations in sending email, making updates through the wysiwig editor, and exports/imports. We are working on secure solutions for those functionalities and will have them working as soon as possible.
The Tendenci team has been individually contacting clients as their sites are ready for review.
We are still fighting through configuration errors on a few of the development sites and are contacting those clients to let them know we are working to fix these errors. Our team is working on a script to replace absolute links with relative links on sites which will correct linkage breaking issues. This should enable us to bring many more of the sites back up online.
If you have not yet heard from one of our team members and have questions on the status of your site, please submit a ticket at helpdesk.tendenci.com or email communications@tendenci.com.
To the clients on the shared T4 server. Today is the day we committed to having some form of access to you. We’ll have a more detailed and less technical communication update coming shortly.
While we remain optimistic, given our own internal scans of the sites when brought online, we are engaging yet another provider to enable a sixth possible solution by converting the databases to Postgresql.
You will need a Postgres database viewing tool such as the free cross platform PGAdmin utility from Postgres. http://www.pgadmin.org/screenshots/
Again – a less technical update will be posted by the communications team. As for the sites themselves, the remaining elements are technical and security based only. There is no point it opening the IP address if we know the server would be vulnerable to another attack and possibly risk exposing data. This is a team effort that is causing significant damage to our company, but it will NOT stop until you have your sites back.
A security update, and that’s what I get for being overly optimistic, but our initial scans found some issues. It is important to remember that a server that is on the Internet accepts inbound traffic on port 80 and 443, but it replies and can call out.
Our remediation plan called for building all new servers and porting the data, but if there is something that can call out once we open those ports then we are right back at ground zero. Possibly worse. And that is not acceptable.
I’ll let everyone know the minute we can let some testing begin. Maybe I’m being overly cautious at this point but given the situation, I believe it is warranted. Our current task is reconfiguring sites and we are having some challenges but are solving them one by one. (8.3 filenames get restored from backups for example and have to be removed again. That type of thing.) – Ed
We continue to push forward through the recent DDoS attacks that have impacted approximately 40 of our clients on the legacy T4 software.
We apologize this has taken longer to resolve than initially anticipated. We continue to work to bring affected sites back up Friday, however, it maybe the beginning of next week before all disrupted sites are back up.
Latest Update
Yesterday at 1pm CST Tendenci CEO Ed Schipul hosted a phone call to address questions from the Tendenci community, especially those affected by the recent network attack.
Points that were covered during the call:
• We understand how much this affects and recognize the important responsibility we have of protecting our community. We are sorry that the process of getting your website back up has taken much longer than anticipated.
• We are working to get the sites back up as soon as possible. Based on what we know today, we anticipate that it will be the end of this week and possibly into the start of next week before all sites are back up. It is in the best interests of your organization and your website users to make sure the sites pass a security scan before we bring them back online.
• Our focus is on doing everything in our power to get you back up and running, and we are putting measures in place to make sure this does not happen again.
Questions
What is worst case scenario for sites coming back up?
Knowing what we know today, we anticipate restoring websites by Friday, December 12th. That said, the worst case scenario would be that we experience additional attacks or other complications, in which case we could be further delayed until Monday, December 15th. We are doing our best to get sites back up securely this week.
Was this attack a sudden brute-force attack only performed over the past week, or was it a slow process over time with virus files on the server for an extended period?
Our investigation is still ongoing and we will share a full picture with the community on the nature of the attack when this is behind us. Once our tech team has gotten sites back up we will create a forensic report with the data we are recording on the servers. Right now they are focused on getting the sites up and in post action review can summarize what took place.
Do you foresee this outage affecting search engine rankings?
It is likely that there could be some impact on search engine performance. Most of the clients who have experienced the outage have a landing page up. This should have a neutral affect on your site’s search performance. But internal pages may be affected on search for the short-term.
How will you make this right for affected customers?
We will work hard to earn back your trust. This was a criminal attack that has created financial losses for everyone involved and we are heartbroken that many of our long-term customers have been impacted.
Our team continues to work round the clock to restore our system and those sites that have been affected by the recent DDoS attack.
At 1pm CST on Monday, December 8th, we will be hosting an conference call regarding the outage.
During this call, our CEO, Ed Schipul, will provide updates and address questions.
Please email any questions to communications@tendenci.com prior to the call to ensure we address as many as possible.
For dial in information, please submit a ticket to helpdesk.tendenci.com
US Toll-Free: 1-888-619-1583
Participant Passcode: 250006
The Tendenci team is continuing to work to restore our systems and get websites affected by this week’s DDoS attack back up. We are taking this issue very seriously and have mobilized all available resources to address it. We are sorry this was not accomplished for all affected sites by the end of the work week
as we had anticipated. Our team will continue to work through the weekend to resolve this.
For the websites we control, we are getting landing pages up. For those who have websites we do not control, we are sending records to update ip address, as we lack access to do so.
Please contact us through helpdesk.tendenci.com if any of the contact info on your landing page needs to be updated.
As we work through this, we are learning how we can improve our systems, our protocols, our response procedures and our communication to prevent a similar occurrence in the future. It has taken too long to restore our systems and get all affected websites back up. We are sorry. This is unacceptable and embarrassing. We promise to do everything we can to get all websites back up as quickly as possible.
How many sites were affected by the network attack?
Too many. One site is too many to begin with, and we are very sorry for the disruption this has caused in the organizations that we serve. In this case 10% to 15% of our clients who host with us were affected. Only our legacy Tendenci 4 software codebase on Microsoft technology is currently experiencing issues up to and including being offline. None of the newer Tendenci 5 software on Linux and Python is experiencing problems. With newer technology that we have in Linux and Python we can harden sites more and isolate them, therefore, we have no reason to believe that this situation will disrupt service for any of our clients on the Tendenci 5 software.
When will my site be back up?
We are very sorry that we have been unable to restore all Tendenci websites. Providing the Tendenci community with reliable service is the basis for our business and we have been unable to deliver. We are working hard to have at least some of the affected websites back up in a limited capacity today. We will continue to update those affected through posts to blog.tendenci.com, Facebook and Twitter, and via email.
What are my options?
Affected Tendenci are being provided with customized landing pages with alternate contact information, such as phone and email. When ready, we will restore these sites with limited functionality. With the realization that it could take several more days before all Tendenci sites are back up with limited functionality, and the high likelihood that it may be longer before all functionality is available, we understand if some organization choose to move off Tendenci.
What does limited functionality mean?
As an interim step on the pathway to returning all websites to full functionality, we will restore static websites. Temporarily, for a period of several days, Tendenci users will not have the ability to process transactions or make other updates to the website. We know this is not the type of capability you expect and we are sorry. We are mobilizing all available resources to get all websites back up.
Can I move my website to another server?
Unfortunately, the issue we are working to address cannot be solved by moving to another server. Based on our investigation into these attacks, the work we are doing to protect Tendenci websites going forward is essential. Moving to another server would not address the vulnerabilities that were revealed by these attacks.
Has my organization’s data been compromised?
We have no indication at this time that Tendenci user data has been accessed. Our investigation has found that these attacks were related to website traffic and not to hacking into information. That said, we are taking further steps to protect Tendenci user data.
Why wasn’t this problem prevented?
We work hard and invest aggressively in measures aimed at protecting our systems, but in this case it was clearly not enough. We are sorry. After we complete the restoration of our systems, we will examine the vulnerabilities in our systems and practices that left us open to these attacks in order to ensure something like this never happens again.
What are you doing to make sure it is not going to happen again?
One very basic thing we are doing is shutting down the ability for clients to FTP into their site. This created vulnerabilities in the system and can compromise other clients. Generally, the benefits of supporting functionality such as FTP as opposed to Secure FTP (sFTP) on the system that is based on Windows technology do not outweigh the risks to the Tendenci community that is still on Tendenci4.
What about clients that currently use the FTP function?
We can isolate that functionality with some inconvenience to clients on T4 yet still meet the business needs.
For example you can sign up for Amazon Web Services S3 services and we can dynamically include those files into your site. The business need can be met without the risk.
Why are the sites affected not back up yet?
We have put every single resource available, internal and external, including outside security consultants, our top programmers, deployed new scanning tools to see if they can see the patterns our current virus scanners and web application firewalls missed. We are working around the clock, and are doing everything we can to get your sites back up.