Author: Becky Leven

In migrating almost all of the remaining sites from Windows 2003 to Windows 2012 to ensure everyone is protected we ran into some issues implementing SSL on some of the sites.

Sites on T4 that were being migrated may experience connection outage anticipated through tomorrow afternoon while we implement SSL on these sites.

T4 sites that were already migrated last week experienced an outage of up to 2 hours today while the new firewall was being implemented. It was rolled back because of too many false positive blocking access to the sites. Those site have been brought back up with the rollback of the firewall.

In migrating almost all of the remaining sites from Windows 2003 to Windows 2012 to ensure everyone is protected we ran into some issues implementing SSL on some of the sites. Because it is a shared server we are evaluating if we should apply encryption at the firewall level for everyone or continue to on the current path.

Regardless this has become an urgent matter as similar to to the initial incident a large number of sites are offline and it is imperative that we get them back on line as soon as possible.

We had a conference call scheduled for 3pm today to address questions for clients who are part of this migration. T

Given our CEO’s role as part of the security and remediation team, we need to cancel today’s talk and will hopefully reschedule for Thursday as events play out. Clients who are affected have received an email.

If you have not an email from our team and had been part of todays conference call please email support@tendenci.com so we can make sure that we provide you with updated information for the rescheduled call.

We do not expect this outage to be more than 24 hours given our experience restoring the initial group of sites on the Windows platform.

The data migration for server maintenance that was started last night is still in progress and some legacy sites may still be experiencing an outage while the migration finishes.

Please email support@tendenci.com if you have questions specific to your site.

With the recent denial of service attacks on some of our T4 sites, we are taking measures to increase security on all T4 sites.

Sites that have not yet been moved to the secured server environment will be moved starting tonight and continuing through Tuesday. December 30th. T4 sites that were affected by the DDoS attack on our servers in past weeks have already been moved to this new server.

We expect full functionality restored although clients on our T4 legacy software will experience some limitations on their sites over the next few weeks. These moves are to ensure security for all of our legacy clients not yet using the Linux based Tendenci 5+ software.

These security updates do not apply to any clients on Tendenci 5 in the Linux environment.

We have reached out to clients on T4 who will be affected by this move through contact email addresses on file with our team.  If you have not received a message from the Tendenci Team and have questions about your site please email support@tendenci.com

Happy Holidays from the Tendenci Team!

Our offices will be closed from 1pm on Wednesday, December 24th through 8:30am on Monday, December 29th.

In the meantime,  leave us a message on our mainline at 281-497-6567 and we’ll get back to you as soon as possible next week or submit a ticket to helpdesk.tendenci.com.

We have been working to get all sites back up safely and securely that were affected by the recent DDoS attack on our T4 legacy platform.

As of this morning, we have 60% of sites that were affected back up online.

We have sent DNS information to a number of other clients whose sites will be restored once the DNS updates have been completed.  We are still working on restoring a few websites. Sites that were image heavy have taken longer to restore, as many images were corrupted during the attack on the system and removed during the subsequent security scans.

We are continuing to bring sites back up throughout the day

Currently sites are experiencing limitations related to sending emails, updating via the wysiwyg editor, and exporting/importing data. Today we are making steps to bring email back up online to the sites. When we have an update for timeline on email we will be posting to the blog.

Thank you for your patience as we work to secure your site and your data. If you have questions about the status of your site please submit a ticket to helpdesk.tendenci.com or email us at support@tendenci.com.

We have started the process of bringing sites that were affected by the recent DDoS attack back online with limited functionality.

So far we have successfully restored 10% of the sites that were taken out by this attack. We will continue to bring sites online today.

Currently, sites that are being brought up have limitations in sending email, making updates through the wysiwig editor, and exports/imports.  We are working on secure solutions for those functionalities and will have them working as soon as possible.

The Tendenci team has been individually contacting clients as their sites are ready for review.

We are still fighting through configuration errors on a few of the development sites and are contacting those clients to let them know we are working to fix these errors. Our team is working on a script to replace absolute links with relative links on sites which will correct linkage breaking issues. This should enable us to bring many more of the sites back up online.

If you have not yet heard from one of our team members and have questions on the status of your site, please submit a ticket at helpdesk.tendenci.com or email communications@tendenci.com.

Latest Update

Yesterday at 1pm CST Tendenci CEO Ed Schipul hosted a phone call to address questions from the Tendenci community, especially those affected by the recent network attack.

Points that were covered during the call:

• We understand how much this affects and recognize the important responsibility we have of protecting our community. We are sorry that the process of getting your website back up has taken much longer than anticipated.

• We are working to get the sites back up as soon as possible. Based on what we know today, we anticipate that it will be the end of this week and possibly into the start of next week before all sites are back up. It is in the best interests of your organization and your website users to make sure the sites pass a security scan before we bring them back online.

• Our focus is on doing everything in our power to get you back up and running, and we are putting measures in place to make sure this does not happen again.
Questions

What is worst case scenario for sites coming back up?

Knowing what we know today, we anticipate restoring websites by Friday, December 12th. That said, the worst case scenario would be that we experience additional attacks or other complications, in which case we could be further delayed until Monday, December 15th. We are doing our best to get sites back up securely this week.

Was this attack a sudden brute-force attack only performed over the past week, or was it a slow process over time with virus files on the server for an extended period?

Our investigation is still ongoing and we will share a full picture with the community on the nature of the attack when this is behind us. Once our tech team has gotten sites back up we will create a forensic report with the data we are recording on the servers. Right now they are focused on getting the sites up and in post action review can summarize what took place.

Do you foresee this outage affecting search engine rankings?

It is likely that there could be some impact on search engine performance. Most of the clients who have experienced the outage have a landing page up. This should have a neutral affect on your site’s search performance. But internal pages may be affected on search for the short-term.

How will you make this right for affected customers?

We will work hard to earn back your trust. This was a criminal attack that has created financial losses for everyone involved and we are heartbroken that many of our long-term customers have been impacted.

Our team continues to work round the clock to restore our system and those sites that have been affected by the recent DDoS attack.

At 1pm CST on Monday, December 8th, we will be hosting an conference call regarding the outage.

During this call, our CEO, Ed Schipul, will provide updates and address questions.

Please email any questions to communications@tendenci.com prior to the call to ensure we address as many as possible.

For dial in information, please submit a ticket to helpdesk.tendenci.com

US Toll-Free: 1-888-619-1583

Participant Passcode: 250006