In memory of all lives lost that day on 9/11, the families left behind, and the first responders’ sacrifices . We will never forget. United we stand.
The 500 errors on homepages for some of our T4 legacy clients reported at 5:32pm CT were resolved by 6pm this evening.
Our team is working on the root cause and will continue to investigate the incident.
We are aware that some T4 sites that had received the security updates last night are now experiencing 500 errors on their homepage.
This error was reported first at 5:32pm CT- We are aware of the issue and currently investigating.
Over night our programming engineering team put firewalls in place to provide increased security on the servers. This security would allow us to start reemploying some of the functionality on T4 that is currently disabled.
Some of the settings on the firewall that were put into place were employed too tightly. This has caused errors and outages on T4 sites.
The team is in the process of rolling back those changes at this time.
On January 2, 2015 at 3pm CT, our CEO, Ed Schipul, hosted an open conference call for T4 clients to inform them updates on outages that affected our T4 clients, the subsequent activities of our team, what we were doing to protect our clients’ data and bring the sites back online, and answer questions from the group regarding the attack on the T4 system.
A summary of that conference call is posted here.
Points covered during the call
- All functionality will be restored to Tendenci 4 once we are assured solutions are secure
- Our number one priority right now is getting up the few remaining sites that are still offline.
- Timeline for restoring all functionality to the system is dependent getting few remaining sites back up online
What was the nature of attacks affecting the T4 community?
In late November we had a Windows2003R2 server hosting Tendenci 4 (the classic ASP version – not the Linux based Tendenci 5) compromised as a result of an attack. This was a crime committed in which we have all suffered in the form of lost time, revenue, extreme frustration and anxiety over the Holidays . We are still not fully up to previous functionality on many sites, with a few sites still offline.
The server in question was behind Amazon’s firewalls, behind our own AWS firewall, and the ACL (Access Control List). The server was running Microsoft’s Windows Firewall, and per best practices we had run Microsofts IIS Hardening tool. Finally, in addition to all of that, we were running MacAfee’s Enterprise Virus and Malware real-time scanners.
Some clients experienced a brief outage. Unfortunately, some clients were down up to a month as the minute we saw a possible compromise we shut the server down. A first set of clients on our T4 servers was shut down and migrated starting in late November. A second set of our T4 client sites were shut down and migrated in late December when we suspected an infected file on the server on the second server that hosted our T4 sites.
We believe that the individual responsible for the attack was waiting to use websites on the server to relay web traffic to commercial websites during the holiday season. This type of activity is referred to as black hat SEO and can help sites gain in rankings on Google through damaging others.
Our first priority is protection of client data. At this point, most of the affected sites, though experiencing limitations in functionality, are back online with security in place. Some areas of vulnerability are still being addressed and are inaccessible. These include:
- WYSIWYG editor
- File uploads
- Newsletter send
- FTP access
- Photo Gallery
We are working on restoration of these features to insure security and stability.
The good news for our T4 clients is that you are now on a much higher security server running Windows 2012R2 behind a WAF with intensive logging.
The upgrade from Windows 2003, which was hardened using every best practice and running industry leading malware and virus detection, was necessary to ensure security of your databases.
We are building individual data portals for all clients initially to make it easier to extract your data.
We do realize it was sudden; however, if you cannot trust a server, there is no choice but to power it down immediately in the interest of protecting and preserving client data.
What can you do to assist?
(We will provide more details and instructions on implementing the following steps in subsequent blog posts)
- Claim your site in Google Webmaster Tools
- Claim your site in Google Analytics
- Sign up for an SMTP service. We recommend MailGun. The newsletter will be brought back up being routed through SMTP so you have greater access to your email
- Please make DNS entries if we have contacted you and requested you to do so
- Sign up for an S3 Bucket from Amazon
Q&A Session – Client questions answered as they were submitted
Why did you migrate clients who were still online to another server in late December?
Once the initial server was restored, it was decided that we needed to move quickly. We suspected this person had access to our other server. Cutting off the attacker’s revenue stream by securing the first server and stopping his redirects to commercial websites meant he might make moves to damage the server or our clients’ data in retribution. To protect clients on our second server, we moved their data onto the new server and converted sites to Windows 2012.
What was the point of Origin for the attack on the server?
We are not yet certain of the point of origin within the system. Confirming the point of origin will take additional forensics from our team. We have temporarily disabled features that are related to suspected entry points including image upload, FTP, and Cute FTP.
I understand that Tendenci 5 clients are not having any issues. Why not simply upgrade all Tendenci 4 clients to Tendenci 5?
Tendenci 5 is an open source product that was written in a different programming language (Python) for a different hosting environment (Linux). The conversion from one platform to another is close to building an entirely new web site. Converting all clients to T5 would take much longer than restoring and securing Tendenci 4 sites.
How long before I can download my database?
We are currently setting up separate database access for each client where you can download any data you need.
How long before the WYSIWYG editor is available?
WYSIWYG will be re-implemented once it has been stripped of vulnerabilities and will follow the restoration of any sites that are still offline.
In the short term to get the formatting that you would like on your pages there are several free online tools to help you convert text to HTML (https://www.google.com/search?q=wysiwig+editor&ie=utf-8&oe=utf-8#q=wysiwig+editor+online).
You can use these tools to cut and paste the formatted HTML into your Tendenci pages.
What about images?
You can include an image by pulling it from another source, for example dropbox through html on your page or by using an online wysiwyg editor and pasting into Tendenci.
When you link to an image you need to put the image URL in as your source
For example, you use dropbox to pull in an image
- Click on your image in drop box
- Left click on the image and click view original
- The URL of the original will be the URL you’ll want to pull into your wysiwig editor. (Typically will start with http://dl-web.drobox.com/get)
Your resulting html to be cut and pasted into your Tendenci site would look something like this:
<img alt=”” src=”http://dl-web.drobox.com/get….” style=”width: 100px; height: 75px;” />
Will you be bringing back all functionalities such as newsletters, exports, WYSIWYG editor?
Yes. Our first priority is to restore the websites for any client who is still offline.
We are working on testing and restoring functionality. Some of the modules will be configured differently when restored to eliminate vulnerabilities for all of our clients.
What should we do about newsletters in the short term?
You can still create/preview newsletters through the newsletter generator. Then copy the text into another program to send.
Here is what we recommend for the newsletter that needs to go out now.
- Generate your Newsletter.
- Preview the Newsletter.
- Copy the html structure for the Newsletter (You can do a view page source or download an application like site sucker http://www.sitesucker.us/home.html).
- Temporarily sign up to use a newsletter service:
Google gives a ton of options https://www.google.com/search?q=newsletter+serv…
- Paste the copied html code from the Preview into the email template provided – OR – Set up a regular HTML email and paste the code in from Preview.
Do you have an estimate as when we will be able to start updating our content?
You can update content now using the HTML editor. There are several free online tools to help you convert text to HTML that you can then cut and paste into your HTML editor until we get a new WYSIWYG editor installed.
What data was compromised for (our site)? What do we need to tell our website users? We pass transactions through to Authorize.net. Was that data compromised?
The good news is that we do not nor have ever stored credit card information on your website. We simply pass that directly to Authorize.net and other payment processors for processing and do not save it to the server. We know that the main purpose of the hack was to redirect websites for SEO. If users were redirected, they would know it because they would be looking at an entirely different site such as one that sold shoes.
Consider notifying your site users of the following:
- It is possible that their contact information was obtained by a hacker
- Let them know that because we encrypt passwords it is doubtful they have their passwords but we recommend everyone change their passwords regardless.
- Let them know Credit Cards were NOT obtained because they are not stored on the site at all. Those are strictly processed by your merchant provider on their site.
Can we get our content extracted and sent to us so we have a full copy of our data?
Yes. We are setting up these databases so that you can access and download whatever data you need. Short-term – we are going to replicate your data and place it into a Postgres database for individual access.
What is the timeline for Email /Export /Upload Data? Will these come back one at a time or all at once?
Our first priority is to restore the websites for any client who is still offline.
We will then begin restoring functionality and bring these features up as soon as they are secure. Our first priority now is providing an interface for exporting data so that administrators can implement alternate means to contact site users and members as we work to ensure functionality on the site are secure.
Thank you to all who participated in the conference call and contributed questions. It becomes clear quickly enough what the highest priority features are and will help us prioritize the items in our queue.
We do appreciate everyone’s patience and willingness to seek alternative methods for getting messages out to your association in ways that will not compromise your site or any other sites on your server.
Please feel free to post any additional questions to this blog and submit requests for assistance at helpdesk.tendenci.com.
And as always, thank you for being a Tendenci client.
In migrating almost all of the remaining sites from Windows 2003 to Windows 2012 to ensure everyone is protected we ran into some issues implementing SSL on some of the sites.
Sites on T4 that were being migrated may experience connection outage anticipated through tomorrow afternoon while we implement SSL on these sites.
T4 sites that were already migrated last week experienced an outage of up to 2 hours today while the new firewall was being implemented. It was rolled back because of too many false positive blocking access to the sites. Those site have been brought back up with the rollback of the firewall.
In migrating almost all of the remaining sites from Windows 2003 to Windows 2012 to ensure everyone is protected we ran into some issues implementing SSL on some of the sites. Because it is a shared server we are evaluating if we should apply encryption at the firewall level for everyone or continue to on the current path.
Regardless this has become an urgent matter as similar to to the initial incident a large number of sites are offline and it is imperative that we get them back on line as soon as possible.
We had a conference call scheduled for 3pm today to address questions for clients who are part of this migration. T
Given our CEO’s role as part of the security and remediation team, we need to cancel today’s talk and will hopefully reschedule for Thursday as events play out. Clients who are affected have received an email.
If you have not an email from our team and had been part of todays conference call please email email@example.com so we can make sure that we provide you with updated information for the rescheduled call.
We do not expect this outage to be more than 24 hours given our experience restoring the initial group of sites on the Windows platform.
The data migration for server maintenance that was started last night is still in progress and some legacy sites may still be experiencing an outage while the migration finishes.
Please email firstname.lastname@example.org if you have questions specific to your site.
With the recent denial of service attacks on some of our T4 sites, we are taking measures to increase security on all T4 sites.
Sites that have not yet been moved to the secured server environment will be moved starting tonight and continuing through Tuesday. December 30th. T4 sites that were affected by the DDoS attack on our servers in past weeks have already been moved to this new server.
We expect full functionality restored although clients on our T4 legacy software will experience some limitations on their sites over the next few weeks. These moves are to ensure security for all of our legacy clients not yet using the Linux based Tendenci 5+ software.
These security updates do not apply to any clients on Tendenci 5 in the Linux environment.
We have reached out to clients on T4 who will be affected by this move through contact email addresses on file with our team. If you have not received a message from the Tendenci Team and have questions about your site please email email@example.com
In Tendenci 4 and soon in Tendenci 5 you will have the option of a lower cost method of email tracking and having all emails come from your site or domain name. Two low cost options that we recommend are Mailgun, http://www.mailgun.com/ and Amazon Simple Email Services, http://aws.amazon.com/ses/.