On January 2, 2015 at 3pm CT, our CEO, Ed Schipul, hosted an open conference call for T4 clients to inform them updates on outages that affected our T4 clients, the subsequent activities of our team, what we were doing to protect our clients’ data and bring the sites back online, and answer questions from the group regarding the attack on the T4 system.
A summary of that conference call is posted here.
Points covered during the call
- All functionality will be restored to Tendenci 4 once we are assured solutions are secure
- Our number one priority right now is getting up the few remaining sites that are still offline.
- Timeline for restoring all functionality to the system is dependent getting few remaining sites back up online
What was the nature of attacks affecting the T4 community?
In late November we had a Windows2003R2 server hosting Tendenci 4 (the classic ASP version – not the Linux based Tendenci 5) compromised as a result of an attack. This was a crime committed in which we have all suffered in the form of lost time, revenue, extreme frustration and anxiety over the Holidays . We are still not fully up to previous functionality on many sites, with a few sites still offline.
The server in question was behind Amazon’s firewalls, behind our own AWS firewall, and the ACL (Access Control List). The server was running Microsoft’s Windows Firewall, and per best practices we had run Microsofts IIS Hardening tool. Finally, in addition to all of that, we were running MacAfee’s Enterprise Virus and Malware real-time scanners.
Some clients experienced a brief outage. Unfortunately, some clients were down up to a month as the minute we saw a possible compromise we shut the server down. A first set of clients on our T4 servers was shut down and migrated starting in late November. A second set of our T4 client sites were shut down and migrated in late December when we suspected an infected file on the server on the second server that hosted our T4 sites.
We believe that the individual responsible for the attack was waiting to use websites on the server to relay web traffic to commercial websites during the holiday season. This type of activity is referred to as black hat SEO and can help sites gain in rankings on Google through damaging others.
Our first priority is protection of client data. At this point, most of the affected sites, though experiencing limitations in functionality, are back online with security in place. Some areas of vulnerability are still being addressed and are inaccessible. These include:
- WYSIWYG editor
- File uploads
- Newsletter send
- FTP access
- Photo Gallery
We are working on restoration of these features to insure security and stability.
The good news for our T4 clients is that you are now on a much higher security server running Windows 2012R2 behind a WAF with intensive logging.
The upgrade from Windows 2003, which was hardened using every best practice and running industry leading malware and virus detection, was necessary to ensure security of your databases.
We are building individual data portals for all clients initially to make it easier to extract your data.
We do realize it was sudden; however, if you cannot trust a server, there is no choice but to power it down immediately in the interest of protecting and preserving client data.
What can you do to assist?
(We will provide more details and instructions on implementing the following steps in subsequent blog posts)
- Claim your site in Google Webmaster Tools
- Claim your site in Google Analytics
- Sign up for an SMTP service. We recommend MailGun. The newsletter will be brought back up being routed through SMTP so you have greater access to your email
- Please make DNS entries if we have contacted you and requested you to do so
- Sign up for an S3 Bucket from Amazon
Q&A Session – Client questions answered as they were submitted
Why did you migrate clients who were still online to another server in late December?
Once the initial server was restored, it was decided that we needed to move quickly. We suspected this person had access to our other server. Cutting off the attacker’s revenue stream by securing the first server and stopping his redirects to commercial websites meant he might make moves to damage the server or our clients’ data in retribution. To protect clients on our second server, we moved their data onto the new server and converted sites to Windows 2012.
What was the point of Origin for the attack on the server?
We are not yet certain of the point of origin within the system. Confirming the point of origin will take additional forensics from our team. We have temporarily disabled features that are related to suspected entry points including image upload, FTP, and Cute FTP.
I understand that Tendenci 5 clients are not having any issues. Why not simply upgrade all Tendenci 4 clients to Tendenci 5?
Tendenci 5 is an open source product that was written in a different programming language (Python) for a different hosting environment (Linux). The conversion from one platform to another is close to building an entirely new web site. Converting all clients to T5 would take much longer than restoring and securing Tendenci 4 sites.
How long before I can download my database?
We are currently setting up separate database access for each client where you can download any data you need.
How long before the WYSIWYG editor is available?
WYSIWYG will be re-implemented once it has been stripped of vulnerabilities and will follow the restoration of any sites that are still offline.
In the short term to get the formatting that you would like on your pages there are several free online tools to help you convert text to HTML (https://www.google.com/search?q=wysiwig+editor&ie=utf-8&oe=utf-8#q=wysiwig+editor+online).
You can use these tools to cut and paste the formatted HTML into your Tendenci pages.
What about images?
You can include an image by pulling it from another source, for example dropbox through html on your page or by using an online wysiwyg editor and pasting into Tendenci.
When you link to an image you need to put the image URL in as your source
For example, you use dropbox to pull in an image
- Click on your image in drop box
- Left click on the image and click view original
- The URL of the original will be the URL you’ll want to pull into your wysiwig editor. (Typically will start with http://dl-web.drobox.com/get)
Your resulting html to be cut and pasted into your Tendenci site would look something like this:
<img alt=”” src=”http://dl-web.drobox.com/get….” style=”width: 100px; height: 75px;” />
Will you be bringing back all functionalities such as newsletters, exports, WYSIWYG editor?
Yes. Our first priority is to restore the websites for any client who is still offline.
We are working on testing and restoring functionality. Some of the modules will be configured differently when restored to eliminate vulnerabilities for all of our clients.
What should we do about newsletters in the short term?
You can still create/preview newsletters through the newsletter generator. Then copy the text into another program to send.
Here is what we recommend for the newsletter that needs to go out now.
- Generate your Newsletter.
- Preview the Newsletter.
- Copy the html structure for the Newsletter (You can do a view page source or download an application like site sucker http://www.sitesucker.us/home.html).
- Temporarily sign up to use a newsletter service:
Google gives a ton of options https://www.google.com/search?q=newsletter+serv… - Paste the copied html code from the Preview into the email template provided – OR – Set up a regular HTML email and paste the code in from Preview.
Do you have an estimate as when we will be able to start updating our content?
You can update content now using the HTML editor. There are several free online tools to help you convert text to HTML that you can then cut and paste into your HTML editor until we get a new WYSIWYG editor installed.
What data was compromised for (our site)? What do we need to tell our website users? We pass transactions through to Authorize.net. Was that data compromised?
The good news is that we do not nor have ever stored credit card information on your website. We simply pass that directly to Authorize.net and other payment processors for processing and do not save it to the server. We know that the main purpose of the hack was to redirect websites for SEO. If users were redirected, they would know it because they would be looking at an entirely different site such as one that sold shoes.
Consider notifying your site users of the following:
- It is possible that their contact information was obtained by a hacker
- Let them know that because we encrypt passwords it is doubtful they have their passwords but we recommend everyone change their passwords regardless.
- Let them know Credit Cards were NOT obtained because they are not stored on the site at all. Those are strictly processed by your merchant provider on their site.
Can we get our content extracted and sent to us so we have a full copy of our data?
Yes. We are setting up these databases so that you can access and download whatever data you need. Short-term – we are going to replicate your data and place it into a Postgres database for individual access.
What is the timeline for Email /Export /Upload Data? Will these come back one at a time or all at once?
Our first priority is to restore the websites for any client who is still offline.
We will then begin restoring functionality and bring these features up as soon as they are secure. Our first priority now is providing an interface for exporting data so that administrators can implement alternate means to contact site users and members as we work to ensure functionality on the site are secure.
_____________________________________________________________________________
Thank you to all who participated in the conference call and contributed questions. It becomes clear quickly enough what the highest priority features are and will help us prioritize the items in our queue.
We do appreciate everyone’s patience and willingness to seek alternative methods for getting messages out to your association in ways that will not compromise your site or any other sites on your server.
Please feel free to post any additional questions to this blog and submit requests for assistance at helpdesk.tendenci.com.
And as always, thank you for being a Tendenci client.