Amazon Email Relay Service Outage Resolved Feb 28, 2017

To our clients and end users hosted on the AWS (Amazon Web Services) Cloud – the email outage and partial S3 (storage) outage have been resolved per the Amazon status notification site https://status.aws.amazon.com/

AWS SES Status Update

Visit the AWS status blog at https://status.aws.amazon.com/ and the AWS subreddit at https://www.reddit.com/r/aws/ . AWS’s email outage is described in detail in the reddit post at https://www.reddit.com/r/aws/comments/5wphqj/amazon_s3_and_amazon_ses_are_down/

And as always please keep an eye on this blog for Tendenci updates.

Edit: for more coverage visit: http://www.cbsnews.com/news/amazon-web-services-cloud-outage-internet-crashes/

Rolling out Docker Swarm for Clients Seeking Higher Availability

Tendenci is rolling out Docker Swarm in our cloud data centers for clients seeking higher availability. What is docker swarm? It looks like this:

Docker Swarm

For more on docker-swarm, I recommend this slideshare for an overview.

This is just part of our efforts at Tendenci to avoid downtime from the inevitable security updates. More on this to follow as we continue to review automatic security updates causing unintended consequences at times.

Cloud Server Upgrades in Progress

Clouds from a trip a long time ago...
Clouds from a trip a long time ago…

For the few night owls out there, you may have noticed your sites going offline for 5 to 10 minutes at some point in the middle of the night. Well, some good news! We are in the process of upgrading the Tendenci cloud servers to further increase response time to serve you better.

If you notice anything unusual please contact us at https://helpdesk.tendenci.com

Follow along on this blog, the Tendenci forums or on github. We’d love your input as we set the milestones for Tendenci 8 even while we are still working on Tendenci 7.1.x.  Your voice matters, you are the ones who we are listening to. And it is your input that sets the roadmap for Tendenci.

We appreciate you. And we believe you will appreciate the performance upgrades as we finish the night-shift updates throughout the week and wrapping up next weekend.

Thank You!

Tendenci 4 Microsoft Clients Update

To our Tendenci 4 clients experiencing difficulties, you are ABSOLUTELY STILL MY TOP PRIORITY and the top priority of the entire team.

Huge progress has been made by the team this week and with the help of you, our clients with DNS entries and flexibility and understanding. The good news is that at this point most of you are back on line.

The Tendenci 4 functionality is slowly being recreated on the latest version of Windows Server 2012 R2. In the short term, given I constantly troll the helpdesk, I know y’all are frustrated by the lack of full functionality.

Yet I need you to hang on just a bit longer as this process MUST BE DONE SECURELY. I simply can’t and won’t compromise on that. You don’t rush through open heart surgery and Tendenci, as y’all know, is quite a bit larger than other products because the challenges we address, sites with sometimes 100k users, are much more complex than shopping carts or photos sharing sites.

Still heartbreaking to me is that I am profoundly aware we have a few remaining very important clients to bring back online. And that is a task with multiple people actively working on restoring them, even if they are leaving (and who can blame them) but regardless we will get a stable version for them.

The Good News – The vast majority of Tendenci 4 sites are back online as I type this. Yes you are faced with limited functionality, but have patience as we have to rewrite a lot of code to make the jump to Windows 2012 R2 and most of us have been on the Linux side for a while now.  We are seeing your functionality being incrementally restored daily. ETA is probably early next week to get to 75% functionality.

25% of the functionality will only return if we can find a way to securely implement it for all of you such that each client is isolated. Thus the functionality we plan to restore is only within the limits of new security.

What are the known issues for Tendenci 4 clients (the .asp clients)?

Current limitations – all of which are in place to protect you.

  1. Four sites still off line. Top priority. Period. They know who they are and with each I have personally been in contact.
  2. Limited functionality. Everyone else on the Microsoft version of Tendenci who is back up is still facing limited functionality. We are aware of this. No need to submit a ticket. It is coming back as fast as we can do it SECURELY. If we can’t return functionality securely it will not return at all but that is hopefully not going to be the case as I think we can find a work around for all of it. Specifically items that we know are not working and can’t be turned on just yet are posted in a series of posts right after this one. But in brief we are aware of and working on the following.
    1. Notifications – these will be back by early next week at the latest. Like “forgot my password” and “payment submitted” (just not newsletters.)
    2. Newsletters – Not enabled. You will each need to sign up with a third party email relay service. It could even be your own Amazon Simple Email Service account. This is a required change for all clients to sign up with an SMTP relay provider like Mailgun. Newsletter Generator will return; however, Newsletter Send is NOT coming back on the shared mail server. You MUST sign up for a newsletter provider that supports smtp authentication and clean your email lists. This you can start now.
    3. Uploads – these will come back slowly, limited, restricted and only in non executable areas. You will not be able to upload asp files, js files or any form of executable file going forward. This is a permanent change, but really it is a return to how it was designed and at some point we diverged from fundamentals.
    4. FTP – FTP is not coming back to T4 going forward. Never. But before you scream, web sites are not FTP portals and full FTP is no longer feasible. It shouldn’t have been allowed in the first place except to restricted folders and that got lost over the years by our team despite being documented internally. The Internet has changed, we have to change with it. And fortunately there are so many options for you on this. For example on T5 you can FTP into one folder named media. Or use Amazon S3 for static files. So it will be OK. From dedicated servers to S3 buckets to dropbox to gdrive links – you will have lots of options.
    5. WYSIWYG – we will be implementing a stripped down version of one (1) of the two current ftp editors that are in T4. Think minimalistic like wordpress, but you can still jump over to another html editor and use code view to paste tables and such back in for richer formatting if you prefer. Neither of the rich text editors you are used to will be coming back in the same format for security reasons. But you have work arounds.
    6. WYSIWYG uploads – read only files, no java script, no flash. But you can reference those from an external data store (see FTP permanent discontinuation above.)

Next steps. Today yet another firewall that is already in place will have more of its functionality turned on. It is already handling all of the traffic and has quietly been keeping track of things to find patterns that we need to allow (whitelist) so that our other security rules don’t get carried away. Thus it will be brought online slowly.

The new firewall is another layer of security typically called a WAF (web application firewall). While it’s true that we already have a WAF that was running, it was one that reported instead of dynamically taking action to block an attack. Furthermore it was designed like a virus scanner to look for known issues, not the unknown. The new WAF analyzes the traffic passing in-between the firewalls instead of just protocols and ports so it is much more advanced. And if it doesn’t like something, it jumps into action and blocks it.

Remember iRobot? Ya, kind of like that. So we unfortunately WILL experience some false positives. Yet he’s had enough “training” and is ready to be turned loose so us humans can get mad at him and we can fully educate him on what is legitimate traffic and what is not. Studying logs is one thing, but he’s got to get into the wild and test the real world. We ask for your patience on this. Again, it is to protect YOU!

Moving carefully forward…

Sincerely,

Ed Schipul, CEO, Tendenci

Server Reboots Today Jan 14, 2014 for Security Patches

First – it is Wednesday and Microsoft pushes out patches on Tuesday evenings. So in an overabundance of caution we will be rebooting the Tendenci 4 Microsoft Servers between 4 and 4:30 PM today (10 minutes from now or sooner as I type this.)

Update on Tendenci 5 sites

To our clients on the Open Source Tendenci 5, and the brave clients volunteering to beta test with us on Open Source Tendenci 6 (which I haven’t even had a chance to blog about yet) – all of y’all are still online, have had zero downtime and remain rock solid. Linux and Django and Containers are definitely proving how much stronger they can make Tendenci. This is done by design and made possible by virtue of the flexibility and low cost associated open source in the cloud. It is achieved through isolation, portability and flexibility. I hope you are not frustrated by our team being laser focused on helping our long time clients who experienced outages. I apologize for the slower response time. I know you are missing reports and other items that were there in T4; they will return to being my focus once all of our data centers are fully back online regardless of technology.

Further I am aware of the fact this has thrown numerous projects wildly behind on their timelines and disrupted you as well. All things considered, if your site was offline, you would demand the same from us – to focus on bringing everyone back up.

Ethically, we (Tendenci) must stay the course and get these sites functional. Even now I feel guilty taking the time to write this instead of working on the technical details. I also know people need to know we have a plan (we do) and there is an end in site (there is) and that it will be a success (it will be). And that we have learned from it (we have).

To our Tendenci4 legacy clients on the Microsoft platform, you are and have been MY TOP PRIORITY and the top priority of the entire team. We knew the Internet had changed, just perhaps not how much it had changed in the category of zero day types of threats. See next post.

 

Tendenci 4 Status Update January 6 2015

Another quick update on the status of the network outages. Tuesday Jan 6 2015 – we are still focused on a few long-standing clients experiencing outages or reduced functionality who are still on Tendenci 4, the powerful but legacy version of Tendenci build on the Microsoft platform.

Yesterday was another 12+ hour day for most of the team. They are working hard, but I do insist they sleep some as typos and DNS entries don’t work well together.

If we didn’t communicate directly, rest assured we are working hard to get everyone restored. ~ Ed Schipul, CEO, Tendenci

PS – Please continue to utilize the helpdesk at https://helpdesk.tendenci.com/ for the fastest response. It is the only thing all of us are checking. And luckily the volume is dropping as the sites are being restored. Specifically if it is important enter it directly on the help web site as opposed to starting with an inbound email as then any follow up email communication will continue to be attached to that thread. I thank you for helping us help you.

Heads Up – Some Tendenci Sites Are Having Content Display Issues

UPDATE: All Tendenci Sites are Back to Normal – Content is Displaying Correctly

As of 12:45 PM Central, Tendenci sites are displaying all current content once again. If you have any questions, please contact our Support team at support@tendenci.com or call 281-497-6567 ext 411.

Don’t Worry – Your Site Will Return to Normal Before Lunchtime

Newer Tendenci websites are hosted with Amazon Web Services and last night, Amazon updated their servers.  As a result, our index servers did not successfully come back online last night.

What This Means to You

Your website may not show current information and in some cases, you may not see recently added content when you log on to your website this morning.

No Data is Missing

Your data is safe and sound in the database, it may not be displaying correctly on your website.

We are restarting the production index servers and your website is re-indexing as I write this.

Our estimated timeline for all Tendenci websites to be back to normal is approximately 2 hours.

If you have any questions or concerns about something – please contact our Support team at support@tendenci.com or call 281-497-6567 ext 411.

Update – back online! Some sites experiencing outages – we’re working on it!

Thanks for your patience to all who had sites affected by our technical difficulties yesterday.  All sites are back up and running today, we apologize for the inconvenience.

Some of our Tendenci sites may be offline this morning as our IT team works on some hardware issues.

We will update you as soon as possible and want you to know that we are working diligently.

Need some help? Contact our Support Team by emailing support AT schipul DOT com or by phone at (281) 497.6567, ext. 411.

Thank you, as always, for your patience and your business!