EOL Policy for Tendenci 4 (T4) Software

EOL Policy for Tendenci 4 (T4) Software

Tendenci 6 Interface for Upgraded clients
Tendenci 6 Nav Bar

[EDIT FOR EMPHASIS] April 21, 2015 is End of Life for T4. The Windows servers on the Tendenci hosted network will be shut down and be offline permanently. [END-EDIT] 

Tendenci has always been at the forefront of technology with regard to meeting the needs of associations and nonprofits. When we released Version 5.0 of Tendenci in 2012, the software took a major leap forward by going completely open source, allowing for outside contributions from the development community on software enhancements and bringing a level of transparency and complete control into the hands of all Tendenci users. We are excited about what the future holds for Tendenci as we have Version 6.0 currently in beta and a roadmap for Version 7.0 already underway.

With the focus on the future and what we can achieve with the new technology available, we have made the business decision to formally establish an End-of-Life (EOL) policy. In particular the EOL for Version 4.0 of the Tendenci on the Microsoft software platform is now set for April 21, 2015.

Tendenci was revolutionary when released in 2001. By the time we released Tendenci 4.0 in 2004, it ran seamlessly on the technology that existed at the time. As Microsoft phases out support for its older technology, Tendenci must also adapt to the newer technology options that will provide the best environment for stability and growth.

For those clients still running on the Tendenci 4.0 software, there are two paths for moving forward.

  1. Upgrade to the latest version of Tendenci. T6 is mobile ready using Bootstrap, brings back the newsletter functionality using a client provided smtp relay like Mailgun, and can accommodate a host of pre-built bootstrap 3 templates. Tendenci is open source and runs on Linux, an open source server environment. This means complete freedom for you with regard to customization and hosting. We can migrate you to the new platform, including transfer of existing data and implementation of a mobile-ready theme for $7500.
  2. You can stay on T4 and self host. If you would like to stay on the older technology, we can provide limited assistance to your IT team or an outside vendor with the move to your internal server environment. We will provide a single instance of Tendenci 4.0 for your use (not for resale) and you will need to establish your own security, monitoring, database server, DNS, mail servers, firewall and a VPC (recommended). We estimate the cost of the move at $5000 for the web server portion and this is variable based on the exact hosting environment to be configured.

The last two months have underscored the need to migrate away from the outdated server environment and jump headfirst into the new era of open source hosting options. We all must adapt as we receive new information. I stated previously that we intended to restore the functionality fully of Windows 2003 on Windows 2012 R2 if it could be done securely. After further research, it is clear to me that while you can definitely secure a Windows environment, it can only be done securely on dedicated servers or dedicated virtual machines isolating each client. We cannot bring full functionality back to you securely in a shared hosting environment using classic ASP. On a dedicated server, you can have security parameters that are set by you, for you. There are a number of IT firms that can assist with this and we will extend a single use license in perpetuity if this is the route you choose in the short term.

Effective immediately, there will be no additional changes to the T4 software or hosting environment so that we can focus on the release of T6 and ensuring the migration for our clients is a seamless transition.

The Microsoft sites will be taken offline permanently in 90 days.

For clients wishing to migrate to Tendenci 6.0 on our hosted servers, we will begin migrations on February 16, 2015. We expect the migration to take 30 days and are requiring full payment up front. To achieve this, there will be a need for some compromises on layouts initially, but being upgraded to a responsive design is long overdue and we can continue to work on layouts once we get everything secured and you can edit your sites easily again.

For clients wishing to self host or move to another platform, we will provide a one-time export of your data within the next 90 days. We will be accepting requests for exports starting February 2, 2015. There will be no charge for this export and it will be limited to a one-time event. If this needs to be expedited, we can refer you to an outside trusted contractor although they will charge a fee.

There will be a conference call on Friday, January 23 at 11:00AM CST (details to be emailed separately) to answer any questions about the most recent server issues and to discuss the best course of action for your organization.

We appreciate the support of all of our clients as we have fought to protect and restore your sites during this time. We can all agree that despite our best efforts, the only course of action at this point is to adapt to the changing environment and look forward to what the newer technologies have to offer. Tendenci is a great product and successfully serves websites throughout the world. We look forward to a continued relationship with our clients in the open source world of dynamic software.

[UPDATE: Another option – Generate a Static Sites. You can simply pull the site down in static format using a one line Unix command or a $5 program on the Mac. Then edit it in a product like Dreamweaver. FTP the content to any number of hosting providers. So you CAN download and transfer your site right now to fulfill any obligations. As posted previously there is also simply linking from Dropbox or AWS if that is more convenient. Neither are as convenient as Tendenci, but will keep the sites secure.]

[Update: For developers you can use this script to download. Please be nice to the servers. And scan your files! Several clients had malware on their PC and then uploaded it to the server. All responsibility is on YOU to be sure any files pulled down. This is one of the reasons we are moving away from this older technology. Virus scanners won’t catch it all. IT IS A MANUAL PROCESS TO CLEAN IF FOUND. You must review it carefully by hand. Code snippet below

wget --limit-rate=400k --no-clobber --convert-links --restrict-file-names=windows --random-wait -r -p -E -e robots=off -U mozilla URLHERE

Did I mention scan your files!?

Thanks]

Continued Configuration Changes on Windows Legacy Servers

Update: We will be doing a planned reboot of the Windows servers late this afternoon Wednesday January 21, 2014 to begin the process of restoring two of the remaining clients that are still offline.

Scope: This update applies to Tendenci 4 clients on Windows only. It specifically does NOT apply to Tendenci 5 or Tendenci 6 clients on Linux.

To give you an idea of the scope and velocity of hack attacks that continue, these are attempted crimes mind you, I’ve attached a 15 second video taken several days ago of actual attacks on one of our servers INSIDE the allowed ports.

15 seconds of network attacks

A further update on the 404 errors that the legacy Tendenci 4 clients have been experiencing intermittently. We have been measuring everything possible and tweaking the configuration settings as we see patterns in the logs. Each day generates over 1GB in security alerts across the data centers. All of these are either known attacks, or zero day attempts.

This is what we are fighting and it is relentless. The fact remains that we have protected the legacy sites by moving them from Windows 2003R2 IIS 6 to Windows 2012R2 IIS 8. But to make ASP classic run in IIS 8 we are running the servers in “compatibility mode” which is not an ideal configuration for any technology. And “secure” does not mean “functional” if your sites locked down to the point of not meeting functional requirements.

We have taken a step back and concluded that a technology platform started in 2001 is not up for the cyberwars of 2015. We will have a further update posted later today on possible paths forward for Tendenci 4 clients.

~ Ed

T4 Reported Issue Resolved

The 500 errors on homepages  for some of our T4 legacy clients reported at 5:32pm CT were resolved by 6pm this evening.

Our team is working on the root cause and will continue to investigate the incident.

Update – Tendenci 4 Site Errors Post WAF Installation

Over night our programming engineering team put firewalls in place to provide increased security on the servers. This security would allow us to start reemploying some of the functionality on T4 that is currently disabled.

Some of the settings on the firewall that were put into place were employed too tightly. This has caused errors and outages on T4 sites.

The team is in the process of rolling back those changes at this time.

 

Tendenci 4 Microsoft Clients Update

To our Tendenci 4 clients experiencing difficulties, you are ABSOLUTELY STILL MY TOP PRIORITY and the top priority of the entire team.

Huge progress has been made by the team this week and with the help of you, our clients with DNS entries and flexibility and understanding. The good news is that at this point most of you are back on line.

The Tendenci 4 functionality is slowly being recreated on the latest version of Windows Server 2012 R2. In the short term, given I constantly troll the helpdesk, I know y’all are frustrated by the lack of full functionality.

Yet I need you to hang on just a bit longer as this process MUST BE DONE SECURELY. I simply can’t and won’t compromise on that. You don’t rush through open heart surgery and Tendenci, as y’all know, is quite a bit larger than other products because the challenges we address, sites with sometimes 100k users, are much more complex than shopping carts or photos sharing sites.

Still heartbreaking to me is that I am profoundly aware we have a few remaining very important clients to bring back online. And that is a task with multiple people actively working on restoring them, even if they are leaving (and who can blame them) but regardless we will get a stable version for them.

The Good News – The vast majority of Tendenci 4 sites are back online as I type this. Yes you are faced with limited functionality, but have patience as we have to rewrite a lot of code to make the jump to Windows 2012 R2 and most of us have been on the Linux side for a while now.  We are seeing your functionality being incrementally restored daily. ETA is probably early next week to get to 75% functionality.

25% of the functionality will only return if we can find a way to securely implement it for all of you such that each client is isolated. Thus the functionality we plan to restore is only within the limits of new security.

What are the known issues for Tendenci 4 clients (the .asp clients)?

Current limitations – all of which are in place to protect you.

  1. Four sites still off line. Top priority. Period. They know who they are and with each I have personally been in contact.
  2. Limited functionality. Everyone else on the Microsoft version of Tendenci who is back up is still facing limited functionality. We are aware of this. No need to submit a ticket. It is coming back as fast as we can do it SECURELY. If we can’t return functionality securely it will not return at all but that is hopefully not going to be the case as I think we can find a work around for all of it. Specifically items that we know are not working and can’t be turned on just yet are posted in a series of posts right after this one. But in brief we are aware of and working on the following.
    1. Notifications – these will be back by early next week at the latest. Like “forgot my password” and “payment submitted” (just not newsletters.)
    2. Newsletters – Not enabled. You will each need to sign up with a third party email relay service. It could even be your own Amazon Simple Email Service account. This is a required change for all clients to sign up with an SMTP relay provider like Mailgun. Newsletter Generator will return; however, Newsletter Send is NOT coming back on the shared mail server. You MUST sign up for a newsletter provider that supports smtp authentication and clean your email lists. This you can start now.
    3. Uploads – these will come back slowly, limited, restricted and only in non executable areas. You will not be able to upload asp files, js files or any form of executable file going forward. This is a permanent change, but really it is a return to how it was designed and at some point we diverged from fundamentals.
    4. FTP – FTP is not coming back to T4 going forward. Never. But before you scream, web sites are not FTP portals and full FTP is no longer feasible. It shouldn’t have been allowed in the first place except to restricted folders and that got lost over the years by our team despite being documented internally. The Internet has changed, we have to change with it. And fortunately there are so many options for you on this. For example on T5 you can FTP into one folder named media. Or use Amazon S3 for static files. So it will be OK. From dedicated servers to S3 buckets to dropbox to gdrive links – you will have lots of options.
    5. WYSIWYG – we will be implementing a stripped down version of one (1) of the two current ftp editors that are in T4. Think minimalistic like wordpress, but you can still jump over to another html editor and use code view to paste tables and such back in for richer formatting if you prefer. Neither of the rich text editors you are used to will be coming back in the same format for security reasons. But you have work arounds.
    6. WYSIWYG uploads – read only files, no java script, no flash. But you can reference those from an external data store (see FTP permanent discontinuation above.)

Next steps. Today yet another firewall that is already in place will have more of its functionality turned on. It is already handling all of the traffic and has quietly been keeping track of things to find patterns that we need to allow (whitelist) so that our other security rules don’t get carried away. Thus it will be brought online slowly.

The new firewall is another layer of security typically called a WAF (web application firewall). While it’s true that we already have a WAF that was running, it was one that reported instead of dynamically taking action to block an attack. Furthermore it was designed like a virus scanner to look for known issues, not the unknown. The new WAF analyzes the traffic passing in-between the firewalls instead of just protocols and ports so it is much more advanced. And if it doesn’t like something, it jumps into action and blocks it.

Remember iRobot? Ya, kind of like that. So we unfortunately WILL experience some false positives. Yet he’s had enough “training” and is ready to be turned loose so us humans can get mad at him and we can fully educate him on what is legitimate traffic and what is not. Studying logs is one thing, but he’s got to get into the wild and test the real world. We ask for your patience on this. Again, it is to protect YOU!

Moving carefully forward…

Sincerely,

Ed Schipul, CEO, Tendenci

Server Reboots Today Jan 14, 2014 for Security Patches

First – it is Wednesday and Microsoft pushes out patches on Tuesday evenings. So in an overabundance of caution we will be rebooting the Tendenci 4 Microsoft Servers between 4 and 4:30 PM today (10 minutes from now or sooner as I type this.)

Update on Tendenci 5 sites

To our clients on the Open Source Tendenci 5, and the brave clients volunteering to beta test with us on Open Source Tendenci 6 (which I haven’t even had a chance to blog about yet) – all of y’all are still online, have had zero downtime and remain rock solid. Linux and Django and Containers are definitely proving how much stronger they can make Tendenci. This is done by design and made possible by virtue of the flexibility and low cost associated open source in the cloud. It is achieved through isolation, portability and flexibility. I hope you are not frustrated by our team being laser focused on helping our long time clients who experienced outages. I apologize for the slower response time. I know you are missing reports and other items that were there in T4; they will return to being my focus once all of our data centers are fully back online regardless of technology.

Further I am aware of the fact this has thrown numerous projects wildly behind on their timelines and disrupted you as well. All things considered, if your site was offline, you would demand the same from us – to focus on bringing everyone back up.

Ethically, we (Tendenci) must stay the course and get these sites functional. Even now I feel guilty taking the time to write this instead of working on the technical details. I also know people need to know we have a plan (we do) and there is an end in site (there is) and that it will be a success (it will be). And that we have learned from it (we have).

To our Tendenci4 legacy clients on the Microsoft platform, you are and have been MY TOP PRIORITY and the top priority of the entire team. We knew the Internet had changed, just perhaps not how much it had changed in the category of zero day types of threats. See next post.

 

data portals will be rolling out next week for T4 clients

FRIDAY UPDATE FROM THE CEO

To the T4 (Microsoft legacy sites) clients who are still running on limited functionality or no functionality for a few sites still. There is frustration and anger and I hear you loud and clear. We continue to work around the clock and reach out to trusted resources to help us in the rebuild. It just isn’t easy to take a web site up from Windows 2003 to Windows 2012 and reconfigure everything by hand to try to be sure the code it clean. Still, we have learned a lot so that we will be more prepared in the future and I’m extra committed to the migration to the open source Linux version. But what about RIGHT NOW?!

First – data portals are being configured with the sites that have been fully offline going up first.

https://github.com/epantry/django-sql-explorer
Django-SQL-Explorer attached to Replicated Databases

 

You will be notified through the helpdesk via tickets as soon as we have yours up. We may get a few up as soon as this weekend, and then the speed will pick up as we can clone it and modify the authentication information for each client. Thank you for using https://helpdesk.tendenci.com as it has been the only way I personally could jump in and help with tickets and track progress. I know the phone is more personal, but when the bullets are flying overhead it is efficiency we need, and I think we can all agree that it wasn’t efficient enough and things are still going too slow despite automation simply because of the volume.

There are a few other obvious items that we are still working through.

  1. Email notifications. With the changed IP addresses we are seeing some clients delivery rates drop significantly and need to update your DNS to send from an email address at your organization. This requires a site setting update on your site and your DNS provider to make DKIM and SPF record entries for email delivery. It’s tedious but has to be done. Spammers have made things complicated.
    Workaround – the system does record most notices as they are sent for administrators at /en/emails/search.asp on your site.
  2. File uploads – the new web application firewall is much tighter than before, and I know we have had numerous requests to re-enable things like Word Docs and Excel files, but both of those document types support macros in vbscript and are executables. Until we can put them in a read-only bucket for now the only solution is to convert documents to eliminate all spaces and use lowercase and make them PDFs. Why? Because URL encoding can be used to trick people and spaces aren’t as secure.
  3. Creating new pages and image edits. – Again this requires writing to the file system and we need to isolate every site further before this can be turned back on.
  4. Broken images and missing files – not all, but most of those, had embedded code in the images. Unfortunately this also strongly suggests that for the clients experiencing this the most, there is probably a virus on your home or work network and we strongly encourage you scan and analyze your computers. You can use Trend Micro’s HouseCall for a free virus scan.
  5. SITES THAT ARE STILL DOWN – we have NOT forgotten about you. This remains my top priority for the team and is being done either by a different group of people (I’m leading the charge on the few sites still offline personally) or it takes precedence over the items listed above.

To our Tendenci 5 clients, and the sales contact forms, and clients used to a higher level of service who are feeling, and sometimes are, being ignored by our team. It’s not that we don’t care, it’s simply the result of clients who are victims of the hack attack and they have to be our priority.

And lastly, as difficult as this time has been for all of us, because it was a crime and crimes are not victimless, I appreciate the patience of some, I understand the anger and frustration of others, but please know that we will get through this. Even the clients who left, we’re still going to restore your data so you can get it.

I’m hugely grateful to our team for handling the front lines so the technical people like me could focus on solutions instead of discussing them, which ultimately is what everyone wants. This whole thing saddens me and I can’t apologize enough, while at the same time it infuriates me that it happened in the first place.

 

Summary – Conference Call with the CEO Ed Schipul

On January 2, 2015 at 3pm CT, our CEO, Ed Schipul, hosted an open conference call for T4 clients to inform them updates on outages that affected our T4 clients, the subsequent activities of our team, what we were doing to protect our clients’ data and bring the sites back online, and answer questions from the group regarding the attack on the T4 system.

A summary of that conference call is posted here.

Points covered during the call

  • All functionality will be restored to Tendenci 4 once we are assured solutions are secure
  • Our number one priority right now is getting up the few remaining sites that are still offline.
  • Timeline for restoring all functionality to the system is dependent getting few remaining sites back up online

What was the nature of attacks affecting the T4 community?

In late November we had a Windows2003R2 server hosting Tendenci 4 (the classic ASP version – not the Linux based Tendenci 5) compromised as a result of an attack. This was a crime committed in which we have all suffered in the form of lost time, revenue, extreme frustration and anxiety over the Holidays . We are still not fully up to previous functionality on many sites, with a few sites still offline.

The server in question was behind Amazon’s firewalls, behind our own AWS firewall, and the ACL (Access Control List). The server was running Microsoft’s Windows Firewall, and per best practices we had run Microsofts IIS Hardening tool.  Finally, in addition to all of that,  we were running MacAfee’s Enterprise Virus and Malware real-time scanners.

Some clients experienced a brief outage. Unfortunately, some clients were down up to a month as the minute we saw a possible compromise we shut the server down. A first set of clients on our T4 servers was shut down and migrated starting in late November. A second set of our T4 client sites were shut down and migrated in late December when we suspected an infected file on the server on the second server that hosted our T4 sites.

We believe that the individual responsible for the attack was waiting to use websites on the server to relay web traffic to commercial websites during the holiday season. This type of activity is referred to as black hat SEO and can help sites gain in rankings on Google through damaging others.

Our first priority is protection of client data. At this point, most of the affected sites, though experiencing limitations in functionality, are back online with security in place. Some areas of vulnerability are still being addressed and are inaccessible. These include:

  • WYSIWYG editor
  • File uploads
  • Newsletter send
  • FTP access
  • Photo Gallery

We are working on restoration of these features to insure security and stability.

The good news for our T4 clients is that you are now on a much higher security server running Windows 2012R2 behind a WAF with intensive logging.

The upgrade from Windows 2003, which was hardened using every best practice and running industry leading malware and virus detection, was necessary to ensure security of your databases.

We are building individual data portals for all clients initially to make it easier to extract your data.
We do realize it was sudden; however, if you cannot trust a server, there is no choice but to power it down immediately in the interest of protecting and preserving client data.

 

What can you do to assist?

(We will provide more details and instructions on implementing the following steps in subsequent blog posts)

  • Claim your site in Google Webmaster Tools
  • Claim your site in Google Analytics
  • Sign up for an SMTP service. We recommend MailGun. The newsletter will be brought back up being routed through SMTP so you have greater access to your email
  • Please make DNS entries if we have contacted you and requested you to do so
  • Sign up for an S3 Bucket from Amazon

 

Q&A Session – Client questions answered as they were submitted

 

Why did you migrate clients who were still online to another server in late December?

Once the initial server was restored, it was decided that we needed to move quickly. We suspected this person had access to our other server. Cutting off the attacker’s revenue stream by securing the first server and stopping his redirects to commercial websites meant he might make moves to damage the server or our clients’ data in retribution. To protect clients on our second server, we moved their data onto the new server and converted sites to Windows 2012.

 

What was the point of Origin for the attack on the server?

We are not yet certain of the point of origin within the system. Confirming the point of origin will take additional forensics from our team. We have temporarily disabled features that are related to suspected entry points including image upload, FTP, and Cute FTP.

 

I understand that Tendenci 5 clients are not having any issues. Why not simply upgrade all Tendenci 4 clients to Tendenci 5?

Tendenci 5 is an open source product that was written in a different programming language (Python) for a different hosting environment (Linux). The conversion from one platform to another is close to building an entirely new web site. Converting all clients to T5 would take much longer than restoring and securing Tendenci 4 sites.

 

How long before I can download my database?

We are currently setting up separate database access for each client where you can download any data you need.

 

How long before the WYSIWYG editor is available?

WYSIWYG will be re-implemented once it has been stripped of vulnerabilities and will follow the restoration of any sites that are still offline.

 

In the short term to get the formatting that you would like on your pages there are several free online tools to help you convert text to HTML (https://www.google.com/search?q=wysiwig+editor&ie=utf-8&oe=utf-8#q=wysiwig+editor+online).

You can use these tools to cut and paste the formatted HTML into your Tendenci pages.

 

What about images?

You can include an image by pulling it from another source, for example dropbox through html on your page or by using an online wysiwyg editor and pasting into Tendenci.

When you link to an image you need to put the image URL in as your source

For example, you use dropbox to pull in an image

  • Click on your image in drop box
  • Left click on the image and click view original
  • The URL of the original will be the URL you’ll want to pull into your wysiwig editor. (Typically will start with http://dl-web.drobox.com/get)

Your resulting html to be cut and pasted into your Tendenci site would look something like this:

<img alt=”” src=”http://dl-web.drobox.com/get….” style=”width: 100px; height: 75px;” />

 

Will you be bringing back all functionalities such as newsletters, exports, WYSIWYG editor?

Yes. Our first priority is to restore the websites for any client who is still offline.

We are working on testing and restoring functionality. Some of the modules will be configured differently when restored to eliminate vulnerabilities for all of our clients.

 

What should we do about newsletters in the short term?

You can still create/preview newsletters through the newsletter generator. Then copy the text into another program to send.
Here is what we recommend for the newsletter that needs to go out now.

  • Generate your Newsletter.
  • Preview the Newsletter.
  • Copy the html structure for the Newsletter (You can do a view page source or download an application like site sucker http://www.sitesucker.us/home.html).
  • Temporarily sign up to use a newsletter service:
    Google gives a ton of options https://www.google.com/search?q=newsletter+serv…
  • Paste the copied html code from the Preview into the email template provided – OR – Set up a regular HTML email and paste the code in from Preview.

 

 

Do you have an estimate as when we will be able to start updating our content?

You can update content now using the HTML editor. There are several free online tools to help you convert text to HTML that you can then cut and paste into your HTML editor until we get a new WYSIWYG editor installed.

 

What data was compromised for (our site)? What do we need to tell our website users? We pass transactions through to Authorize.net. Was that data compromised?

The good news is that we do not nor have ever stored credit card information on your website. We simply pass that directly to Authorize.net and other payment processors for processing and do not save it to the server. We know that the main purpose of the hack was to redirect websites for SEO. If users were redirected, they would know it because they would be looking at an entirely different site such as one that sold shoes.

 

Consider notifying your site users of the following:

  • It is possible that their contact information was obtained by a hacker
  • Let them know that because we encrypt passwords it is doubtful they have their passwords but we recommend everyone change their passwords regardless.
  • Let them know Credit Cards were NOT obtained because they are not stored on the site at all. Those are strictly processed by your merchant provider on their site.

 

Can we get our content extracted and sent to us so we have a full copy of our data?

Yes. We are setting up these databases so that you can access and download whatever data you need. Short-term – we are going to replicate your data and place it into a Postgres database for individual access.

 

What is the timeline for Email /Export /Upload Data? Will these come back one at a time or all at once?

Our first priority is to restore the websites for any client who is still offline.

We will then begin restoring functionality and bring these features up as soon as they are secure. Our first priority now is providing an interface for exporting data so that administrators can implement alternate means to contact site users and members as we work to ensure functionality on the site are secure.

_____________________________________________________________________________

 

Thank you to all who participated in the conference call and contributed questions. It becomes clear quickly enough what the highest priority features are and will help us prioritize the items in our queue.

 

We do appreciate everyone’s patience and willingness to seek alternative methods for getting messages out to your association in ways that will not compromise your site or any other sites on your server.

 

Please feel free to post any additional questions to this blog and submit requests for assistance at helpdesk.tendenci.com.

 

And as always, thank you for being a Tendenci client.