Ed Schipul

2015-01-14

Tendenci 4 Microsoft Clients Update

To our Tendenci 4 clients experiencing difficulties, you are ABSOLUTELY STILL MY TOP PRIORITY and the top priority of the entire team.

Huge progress has been made by the team this week and with the help of you, our clients with DNS entries and flexibility and understanding. The good news is that at this point most of you are back on line.

The Tendenci 4 functionality is slowly being recreated on the latest version of Windows Server 2012 R2. In the short term, given I constantly troll the helpdesk, I know y’all are frustrated by the lack of full functionality.

Yet I need you to hang on just a bit longer as this process MUST BE DONE SECURELY. I simply can’t and won’t compromise on that. You don’t rush through open heart surgery and Tendenci, as y’all know, is quite a bit larger than other products because the challenges we address, sites with sometimes 100k users, are much more complex than shopping carts or photos sharing sites.

Still heartbreaking to me is that I am profoundly aware we have a few remaining very important clients to bring back online. And that is a task with multiple people actively working on restoring them, even if they are leaving (and who can blame them) but regardless we will get a stable version for them.

The Good News – The vast majority of Tendenci 4 sites are back online as I type this. Yes you are faced with limited functionality, but have patience as we have to rewrite a lot of code to make the jump to Windows 2012 R2 and most of us have been on the Linux side for a while now. We are seeing your functionality being incrementally restored daily. ETA is probably early next week to get to 75% functionality.

25% of the functionality will only return if we can find a way to securely implement it for all of you such that each client is isolated. Thus the functionality we plan to restore is only within the limits of new security.

What are the known issues for Tendenci 4 clients (the .asp clients)?

Current limitations – all of which are in place to protect you.

Four sites still off line. Top priority. Period. They know who they are and with each I have personally been in contact.
Limited functionality. Everyone else on the Microsoft version of Tendenci who is back up is still facing limited functionality. We are aware of this. No need to submit a ticket. It is coming back as fast as we can do it SECURELY. If we can’t return functionality securely it will not return at all but that is hopefully not going to be the case as I think we can find a work around for all of it. Specifically items that we know are not working and can’t be turned on just yet are posted in a series of posts right after this one. But in brief we are aware of and working on the following.
1. Notifications – these will be back by early next week at the latest. Like “forgot my password” and “payment submitted” (just not newsletters.)
2. Newsletters – Not enabled. You will each need to sign up with a third party email relay service. It could even be your own Amazon Simple Email Service account. This is a required change for all clients to sign up with an SMTP relay provider like Mailgun. Newsletter Generator will return; however, Newsletter Send is NOT coming back on the shared mail server. You MUST sign up for a newsletter provider that supports smtp authentication and clean your email lists. This you can start now.
3. Uploads – these will come back slowly, limited, restricted and only in non executable areas. You will not be able to upload asp files, js files or any form of executable file going forward. This is a permanent change, but really it is a return to how it was designed and at some point we diverged from fundamentals.
4. FTP – FTP is not coming back to T4 going forward. Never. But before you scream, web sites are not FTP portals and full FTP is no longer feasible. It shouldn’t have been allowed in the first place except to restricted folders and that got lost over the years by our team despite being documented internally. The Internet has changed, we have to change with it. And fortunately there are so many options for you on this. For example on T5 you can FTP into one folder named media. Or use Amazon S3 for static files. So it will be OK. From dedicated servers to S3 buckets to dropbox to gdrive links – you will have lots of options.
5. WYSIWYG – we will be implementing a stripped down version of one (1) of the two current ftp editors that are in T4. Think minimalistic like wordpress, but you can still jump over to another html editor and use code view to paste tables and such back in for richer formatting if you prefer. Neither of the rich text editors you are used to will be coming back in the same format for security reasons. But you have work arounds.
6. WYSIWYG uploads – read only files, no java script, no flash. But you can reference those from an external data store (see FTP permanent discontinuation above.)

Next steps. Today yet another firewall that is already in place will have more of its functionality turned on. It is already handling all of the traffic and has quietly been keeping track of things to find patterns that we need to allow (whitelist) so that our other security rules don’t get carried away. Thus it will be brought online slowly.

The new firewall is another layer of security typically called a WAF (web application firewall). While it’s true that we already have a WAF that was running, it was one that reported instead of dynamically taking action to block an attack. Furthermore it was designed like a virus scanner to look for known issues, not the unknown. The new WAF analyzes the traffic passing in-between the firewalls instead of just protocols and ports so it is much more advanced. And if it doesn’t like something, it jumps into action and blocks it.

Remember iRobot? Ya, kind of like that. So we unfortunately WILL experience some false positives. Yet he’s had enough “training” and is ready to be turned loose so us humans can get mad at him and we can fully educate him on what is legitimate traffic and what is not. Studying logs is one thing, but he’s got to get into the wild and test the real world. We ask for your patience on this. Again, it is to protect YOU!

Moving carefully forward…

Sincerely,

Ed Schipul, CEO, Tendenci

2015-01-14

Server Reboots Today Jan 14, 2014 for Security Patches

First – it is Wednesday and Microsoft pushes out patches on Tuesday evenings. So in an overabundance of caution we will be rebooting the Tendenci 4 Microsoft Servers between 4 and 4:30 PM today (10 minutes from now or sooner as I type this.)

2015-01-14

Update on Tendenci 5 sites

To our clients on the Open Source Tendenci 5, and the brave clients volunteering to beta test with us on Open Source Tendenci 6 (which I haven’t even had a chance to blog about yet) – all of y’all are still online, have had zero downtime and remain rock solid. Linux and Django and Containers are definitely proving how much stronger they can make Tendenci. This is done by design and made possible by virtue of the flexibility and low cost associated open source in the cloud. It is achieved through isolation, portability and flexibility. I hope you are not frustrated by our team being laser focused on helping our long time clients who experienced outages. I apologize for the slower response time. I know you are missing reports and other items that were there in T4; they will return to being my focus once all of our data centers are fully back online regardless of technology.

Further I am aware of the fact this has thrown numerous projects wildly behind on their timelines and disrupted you as well. All things considered, if your site was offline, you would demand the same from us – to focus on bringing everyone back up.

Ethically, we (Tendenci) must stay the course and get these sites functional. Even now I feel guilty taking the time to write this instead of working on the technical details. I also know people need to know we have a plan (we do) and there is an end in site (there is) and that it will be a success (it will be). And that we have learned from it (we have).

To our Tendenci4 legacy clients on the Microsoft platform, you are and have been MY TOP PRIORITY and the top priority of the entire team. We knew the Internet had changed, just perhaps not how much it had changed in the category of zero day types of threats. See next post.

2015-01-092015-01-09

data portals will be rolling out next week for T4 clients

FRIDAY UPDATE FROM THE CEO

To the T4 (Microsoft legacy sites) clients who are still running on limited functionality or no functionality for a few sites still. There is frustration and anger and I hear you loud and clear. We continue to work around the clock and reach out to trusted resources to help us in the rebuild. It just isn’t easy to take a web site up from Windows 2003 to Windows 2012 and reconfigure everything by hand to try to be sure the code it clean. Still, we have learned a lot so that we will be more prepared in the future and I’m extra committed to the migration to the open source Linux version. But what about RIGHT NOW?!

First – data portals are being configured with the sites that have been fully offline going up first.

https://github.com/epantry/django-sql-explorer — Django-SQL-Explorer attached to Replicated Databases

You will be notified through the helpdesk via tickets as soon as we have yours up. We may get a few up as soon as this weekend, and then the speed will pick up as we can clone it and modify the authentication information for each client. Thank you for using https://helpdesk.tendenci.com as it has been the only way I personally could jump in and help with tickets and track progress. I know the phone is more personal, but when the bullets are flying overhead it is efficiency we need, and I think we can all agree that it wasn’t efficient enough and things are still going too slow despite automation simply because of the volume.

There are a few other obvious items that we are still working through.

Email notifications. With the changed IP addresses we are seeing some clients delivery rates drop significantly and need to update your DNS to send from an email address at your organization. This requires a site setting update on your site and your DNS provider to make DKIM and SPF record entries for email delivery. It’s tedious but has to be done. Spammers have made things complicated.
Workaround – the system does record most notices as they are sent for administrators at /en/emails/search.asp on your site.
File uploads – the new web application firewall is much tighter than before, and I know we have had numerous requests to re-enable things like Word Docs and Excel files, but both of those document types support macros in vbscript and are executables. Until we can put them in a read-only bucket for now the only solution is to convert documents to eliminate all spaces and use lowercase and make them PDFs. Why? Because URL encoding can be used to trick people and spaces aren’t as secure.
Creating new pages and image edits. – Again this requires writing to the file system and we need to isolate every site further before this can be turned back on.
Broken images and missing files – not all, but most of those, had embedded code in the images. Unfortunately this also strongly suggests that for the clients experiencing this the most, there is probably a virus on your home or work network and we strongly encourage you scan and analyze your computers. You can use Trend Micro’s HouseCall for a free virus scan.
SITES THAT ARE STILL DOWN – we have NOT forgotten about you. This remains my top priority for the team and is being done either by a different group of people (I’m leading the charge on the few sites still offline personally) or it takes precedence over the items listed above.

To our Tendenci 5 clients, and the sales contact forms, and clients used to a higher level of service who are feeling, and sometimes are, being ignored by our team. It’s not that we don’t care, it’s simply the result of clients who are victims of the hack attack and they have to be our priority.

And lastly, as difficult as this time has been for all of us, because it was a crime and crimes are not victimless, I appreciate the patience of some, I understand the anger and frustration of others, but please know that we will get through this. Even the clients who left, we’re still going to restore your data so you can get it.

I’m hugely grateful to our team for handling the front lines so the technical people like me could focus on solutions instead of discussing them, which ultimately is what everyone wants. This whole thing saddens me and I can’t apologize enough, while at the same time it infuriates me that it happened in the first place.

2015-01-062015-01-06

Tendenci 4 Status Update January 6 2015

Another quick update on the status of the network outages. Tuesday Jan 6 2015 – we are still focused on a few long-standing clients experiencing outages or reduced functionality who are still on Tendenci 4, the powerful but legacy version of Tendenci build on the Microsoft platform.

Yesterday was another 12+ hour day for most of the team. They are working hard, but I do insist they sleep some as typos and DNS entries don’t work well together.

If we didn’t communicate directly, rest assured we are working hard to get everyone restored. ~ Ed Schipul, CEO, Tendenci

PS – Please continue to utilize the helpdesk at https://helpdesk.tendenci.com/ for the fastest response. It is the only thing all of us are checking. And luckily the volume is dropping as the sites are being restored. Specifically if it is important enter it directly on the help web site as opposed to starting with an inbound email as then any follow up email communication will continue to be attached to that thread. I thank you for helping us help you.

2014-12-15

updates pending

To the clients on the shared T4 server. Today is the day we committed to having some form of access to you. We’ll have a more detailed and less technical communication update coming shortly.

While we remain optimistic, given our own internal scans of the sites when brought online, we are engaging yet another provider to enable a sixth possible solution by converting the databases to Postgresql.

You will need a Postgres database viewing tool such as the free cross platform PGAdmin utility from Postgres. http://www.pgadmin.org/screenshots/

Again – a less technical update will be posted by the communications team. As for the sites themselves, the remaining elements are technical and security based only. There is no point it opening the IP address if we know the server would be vulnerable to another attack and possibly risk exposing data. This is a team effort that is causing significant damage to our company, but it will NOT stop until you have your sites back.

2014-12-12

initial scans found issues, continuing lock down

A security update, and that’s what I get for being overly optimistic, but our initial scans found some issues. It is important to remember that a server that is on the Internet accepts inbound traffic on port 80 and 443, but it replies and can call out.

Our remediation plan called for building all new servers and porting the data, but if there is something that can call out once we open those ports then we are right back at ground zero. Possibly worse. And that is not acceptable.

I’ll let everyone know the minute we can let some testing begin. Maybe I’m being overly cautious at this point but given the situation, I believe it is warranted. Our current task is reconfiguring sites and we are having some challenges but are solving them one by one. (8.3 filenames get restored from backups for example and have to be removed again. That type of thing.) – Ed

2014-12-11

10% client outage resolution by Monday

Great news. Sites are up and running in a jailed IP block while we scan and test. If all goes well everyone will be back online soon. If it fails the security tests, then to be frank we won’t allow it to be opened up and thus we need your help.

Geeky stuff: This is a quick update from Ed given my communications team is out. The current status of the rebuilding of sites for the portion of our clients who have been offline for a significant period of time is that our new servers in a new higher security “IP Jail” is running well as of this morning.

We are and will continue to scan and work to remediate any compromised files. The original operating systems have been formatted/replaced and all legacy Windows T4 clients that were on Windows 2003R2 are being jumped from IIS 6 to IIS 8.5 on Windows 2012R2 so you will be on the most secure Microsoft Platform ever.

(Note – No Tendenci 5 clients had any issues and I apologize to y’all for the lack of responsiveness on day to day issues as our team addressed the issues for our other clients.)

For our T4 clients coming back online in the new environment, yes, there will be issues as we change IP addresses and email relays and the like but our timeline of Monday is still on track, hopefully sooner. And perhaps a few strong clients will volunteer not to be online first, but to be a volunteer to go through a third party security audit of their site on behalf of everyone who has been a victim of this unfortunate crime. It is like a stress test that attacks a site in a silo to be sure when opened to the public it works as designed. I believe this is an important step to get third party validation before bringing everyone back online for the sake of safety and security.

As a CEO it is my job to foresee and prevent these occurrences, and in this case I missed the mark by a long shot. The Monday deadline will only be possible with some assistance from the community testing a few sites off of the public network for functionality as we work out the transition of over 50 sites to an entirely new cloud based security system that may be (OK, it is….) locked down quite tight. Yet it is better to lock and release, than to risk having to protect our clients by shutting down a server again.

And as I have said I apologize again. This is a crime. We are documenting it for the authorities as best we can. But that isn’t the point. The point is we work with caused based and people trying to change the world for the better. That is what Tendenci IS! And we let you down. Help us fix it because it isn’t us and you, it is just “us”.

Sincerely,

Ed Schipul
CEO, Tendenci, Inc.

2014-12-05

Update: Work Continues to Restore Systems and Websites

The Tendenci team is continuing to work to restore our systems and get websites affected by this week’s DDoS attack back up. We are taking this issue very seriously and have mobilized all available resources to address it. We are sorry this was not accomplished for all affected sites by the end of the work week
as we had anticipated. Our team will continue to work through the weekend to resolve this.

For the websites we control, we are getting landing pages up. For those who have websites we do not control, we are sending records to update ip address, as we lack access to do so.

Please contact us through helpdesk.tendenci.com if any of the contact info on your landing page needs to be updated.

As we work through this, we are learning how we can improve our systems, our protocols, our response procedures and our communication to prevent a similar occurrence in the future. It has taken too long to restore our systems and get all affected websites back up. We are sorry. This is unacceptable and embarrassing. We promise to do everything we can to get all websites back up as quickly as possible.

2014-12-052014-12-05

Update – Information Regarding the Network Outage

How many sites were affected by the network attack?

Too many. One site is too many to begin with, and we are very sorry for the disruption this has caused in the organizations that we serve. In this case 10% to 15% of our clients who host with us were affected. Only our legacy Tendenci 4 software codebase on Microsoft technology is currently experiencing issues up to and including being offline. None of the newer Tendenci 5 software on Linux and Python is experiencing problems. With newer technology that we have in Linux and Python we can harden sites more and isolate them, therefore, we have no reason to believe that this situation will disrupt service for any of our clients on the Tendenci 5 software.

When will my site be back up?

We are very sorry that we have been unable to restore all Tendenci websites. Providing the Tendenci community with reliable service is the basis for our business and we have been unable to deliver. We are working hard to have at least some of the affected websites back up in a limited capacity today. We will continue to update those affected through posts to blog.tendenci.com, Facebook and Twitter, and via email.

What are my options?

Affected Tendenci are being provided with customized landing pages with alternate contact information, such as phone and email. When ready, we will restore these sites with limited functionality. With the realization that it could take several more days before all Tendenci sites are back up with limited functionality, and the high likelihood that it may be longer before all functionality is available, we understand if some organization choose to move off Tendenci.

What does limited functionality mean?

As an interim step on the pathway to returning all websites to full functionality, we will restore static websites. Temporarily, for a period of several days, Tendenci users will not have the ability to process transactions or make other updates to the website. We know this is not the type of capability you expect and we are sorry. We are mobilizing all available resources to get all websites back up.

Can I move my website to another server?

Unfortunately, the issue we are working to address cannot be solved by moving to another server. Based on our investigation into these attacks, the work we are doing to protect Tendenci websites going forward is essential. Moving to another server would not address the vulnerabilities that were revealed by these attacks.

Has my organization’s data been compromised?

We have no indication at this time that Tendenci user data has been accessed. Our investigation has found that these attacks were related to website traffic and not to hacking into information. That said, we are taking further steps to protect Tendenci user data.

Why wasn’t this problem prevented?

We work hard and invest aggressively in measures aimed at protecting our systems, but in this case it was clearly not enough. We are sorry. After we complete the restoration of our systems, we will examine the vulnerabilities in our systems and practices that left us open to these attacks in order to ensure something like this never happens again.

What are you doing to make sure it is not going to happen again?

One very basic thing we are doing is shutting down the ability for clients to FTP into their site. This created vulnerabilities in the system and can compromise other clients. Generally, the benefits of supporting functionality such as FTP as opposed to Secure FTP (sFTP) on the system that is based on Windows technology do not outweigh the risks to the Tendenci community that is still on Tendenci4.

What about clients that currently use the FTP function?

We can isolate that functionality with some inconvenience to clients on T4 yet still meet the business needs.

For example you can sign up for Amazon Web Services S3 services and we can dynamically include those files into your site. The business need can be met without the risk.

Why are the sites affected not back up yet?
We have put every single resource available, internal and external, including outside security consultants, our top programmers, deployed new scanning tools to see if they can see the patterns our current virus scanners and web application firewalls missed. We are working around the clock, and are doing everything we can to get your sites back up.