“Constantly seek criticism. … A well thought out critique of whatever you are doing is as valuable as gold.”- Elon Musk
“If you do the simple math, if somebody else is working 50 hours and you are working 100, you’ll get twice as much done in the course of a year as the other company.”
“Just work like hell. You’ve gotta put in 80, 100 hour work weeks.” – Elon Musk
“Starting a business. Number 1 is having a high pain threshold.” – Elon Musk
“You are always going to buy the trusted brand unless there is a BIG difference.” – Elon Musk
“Constantly seek criticism. … A well thought out critique of whatever you are doing is as valuable as gold.” – Elon Musk
“Usually your friends know what is wrong. They don’t want to tell you because they don’t want to hurt you. … Usually your friends are right. …. You should take the approach that as an entrepreneur you are wrong. You want to be less wrong.” – Elon Musk
We’ve had a lot of crucial conversations lately about decisions that we made between 2006 and 2010. Yup, really. We are explaining now about how we are possibly too far ahead of the curve and why if you give it a bit of time, it will make you look like a rock-star.
MobileGeddon being a great example of how our early adopters are benefiting the absolute most!
We started using Python, the programming language named after Monty Python, in 2004 if not earlier. We first tested Pinax in 2008 if not earlier under J who was running our programming team.
We used Python extensively in our old environment to move files and push out content to our sites. Tasks that are now done by Puppet and Chef and Docker-Compose. We rolled our own using Python on Windows.
So for the curious, that explains why we have this huge depth of knowledge on Python programming dating back to when nobody heard of it. We’ve had to train numerous graduates of Tech, UofH, Aggies, Rice, Penn State, etc, what Python even was!
But that is all ancient history. Why? Because Python is now number 1!
It’s hard to predict the future. We started out writing our own compete web framework in ASP. We were too early in 2001. PHP soon arrived and, being basically identical but open source, the outcome was PHP won. It should have, and did, win. We were too early. But with timing there is also a bit of luck.
I’ll do another post about GIS and mapping and why our move to strictly Postgres with GIS enabled is working out so well. Another post. And I’ll edit this one with links soon. Just needed to get the content out.
Bootstrap3. – Because we know that we take gambles on technology and they have an impact. On you. And that is serious business. We take it seriously as evidenced by our decision to shut down Windows in line with Microsoft’s EOL policies. These are hard choices. Crucial Conversations. We’re the messenger.
And we CARE about YOU. Our clients. The future is bright. We picked our technology future amazingly well. Too well, so now perhaps our problems is more one of resources. And we’ll work through that.
Thank you. If I can leave you with one thought – it is this. THANK YOU! For those who stuck with us, WOW, um… our position for search and the future is crazy good. Open Source means freedom. Results mean donations and sales. Software means sustainable business models.
We appreciate you. Yes closure for some was hard. We wish you the best. We appreciated your time with us while it lasted longer than a Honda. As some depart and some charge forward, I’m especially excited about those who chose to charge forward.
We, you and us, we didn’t “guess” right. We did our homework and validation came ironically on April 21, the same day mobilegeddon hit and our Tendenci 6 clients jumped up in the search rankings. Luck? Hard work? I don’t know.
What do I know? I know how to serve. I serve y’all.
Seriously, we love you, but off you go. Go check out the source code at https://github.com/tendenci/ or something. None of this blog post applies to those of you in the 95% majority of our clients.
This isn’t to say we don’t think you look great today. You do! And we’re looking forward to being able to focus on YOU more after tomorrow.
Upgrading Clients – YEA!!!
A HUGE THANK YOU to all of our former T4 clients who have chosen to stay with us and upgrade to Tendenci 6.
As we’ve communicated to each of you individually, there may be a period where we have a holder page up for a bit and unlike a usual long term project we will be going live and incrementally flushing out the site. It’s a lot of sites. We are going to focus on functionality first, SEO next and then bring more uniqueness of design per the scope of each of your migration projects.
We’re excited for both of the groups above that we will be focused on one technology stack and can accelerate Tendenci’s growth.
Microsoft Servers Offline Tomorrow April 21 at 12:00 CST
For our former clients who were running on Tendenci 4 and are choosing to leave, we’re sorry to see you go. I get it. I hated the idea of giving up my blackberry because “it just worked”. Yet now I can’t imagine not having a smart phone. Software is emotional. It just is.
[IMPORTANT! If you are pointed at our DNS Servers or Email Servers make sure your new provider makes these updates!]
I believe we have communicated with everyone in person at this point, and the remaining group is small.
This is just a courtesy reminder that you may want to have your new provider make their DNS entries tonight to avoid any downtime.Don’t forget they will need to set up email relays and transfer dns servers so be nice to your new vendor and give them another reminder please as all of us want to see you succeed.
A website is an ecosystem of databases and content and media and email and relays. Be sure your new provider, if you chose to leave, is on top of it.
Please know that we have appreciated your business and wish you the best in the future. You are always welcome back, or even consider using Tendenci Open Source with another company. That’s the point – Freedom!
Tendenci has always been at the forefront of technology with regard to meeting the needs of associations and nonprofits. When we released Version 5.0 of Tendenci in 2012, the software took a major leap forward by going completely open source, allowing for outside contributions from the development community on software enhancements and bringing a level of transparency and complete control into the hands of all Tendenci users. We are excited about what the future holds for Tendenci as we have Version 6.0 currently in beta and a roadmap for Version 7.0 already underway.
Tendenci was revolutionary when released in 2001. By the time we released Tendenci 4.0 in 2004, it ran seamlessly on the technology that existed at the time. As Microsoft phases out support for its older technology, Tendenci must also adapt to the newer technology options that will provide the best environment for stability and growth.
For those clients still running on the Tendenci 4.0 software, there are two paths for moving forward.
Upgrade to the latest version of Tendenci. T6 is mobile ready using Bootstrap, brings back the newsletter functionality using a client provided smtp relay like Mailgun, and can accommodate a host of pre-built bootstrap 3 templates. Tendenci is open source and runs on Linux, an open source server environment. This means complete freedom for you with regard to customization and hosting. We can migrate you to the new platform, including transfer of existing data and implementation of a mobile-ready theme for $7500.
You can stay on T4 and self host. If you would like to stay on the older technology, we can provide limited assistance to your IT team or an outside vendor with the move to your internal server environment. We will provide a single instance of Tendenci 4.0 for your use (not for resale) and you will need to establish your own security, monitoring, database server, DNS, mail servers, firewall and a VPC (recommended). We estimate the cost of the move at $5000 for the web server portion and this is variable based on the exact hosting environment to be configured.
The last two months have underscored the need to migrate away from the outdated server environment and jump headfirst into the new era of open source hosting options. We all must adapt as we receive new information. I stated previously that we intended to restore the functionality fully of Windows 2003 on Windows 2012 R2 if it could be done securely. After further research, it is clear to me that while you can definitely secure a Windows environment, it can only be done securely on dedicated servers or dedicated virtual machines isolating each client. We cannot bring full functionality back to you securely in a shared hosting environment using classic ASP. On a dedicated server, you can have security parameters that are set by you, for you. There are a number of IT firms that can assist with this and we will extend a single use license in perpetuity if this is the route you choose in the short term.
Effective immediately, there will be no additional changes to the T4 software or hosting environment so that we can focus on the release of T6 and ensuring the migration for our clients is a seamless transition.
The Microsoft sites will be taken offline permanently in 90 days.
For clients wishing to migrate to Tendenci 6.0 on our hosted servers, we will begin migrations on February 16, 2015. We expect the migration to take 30 days and are requiring full payment up front. To achieve this, there will be a need for some compromises on layouts initially, but being upgraded to a responsive design is long overdue and we can continue to work on layouts once we get everything secured and you can edit your sites easily again.
For clients wishing to self host or move to another platform, we will provide a one-time export of your data within the next 90 days. We will be accepting requests for exports starting February 2, 2015. There will be no charge for this export and it will be limited to a one-time event. If this needs to be expedited, we can refer you to an outside trusted contractor although they will charge a fee.
There will be a conference call on Friday, January 23 at 11:00AM CST (details to be emailed separately) to answer any questions about the most recent server issues and to discuss the best course of action for your organization.
We appreciate the support of all of our clients as we have fought to protect and restore your sites during this time. We can all agree that despite our best efforts, the only course of action at this point is to adapt to the changing environment and look forward to what the newer technologies have to offer. Tendenci is a great product and successfully serves websites throughout the world. We look forward to a continued relationship with our clients in the open source world of dynamic software.
[UPDATE: Another option – Generate a Static Sites. You can simply pull the site down in static format using a one line Unix command or a $5 program on the Mac. Then edit it in a product like Dreamweaver. FTP the content to any number of hosting providers. So you CAN download and transfer your site right now to fulfill any obligations. As posted previously there is also simply linking from Dropbox or AWS if that is more convenient. Neither are as convenient as Tendenci, but will keep the sites secure.]
[Update: For developers you can use this script to download. Please be nice to the servers. And scan your files! Several clients had malware on their PC and then uploaded it to the server. All responsibility is on YOU to be sure any files pulled down. This is one of the reasons we are moving away from this older technology. Virus scanners won’t catch it all. IT IS A MANUAL PROCESS TO CLEAN IF FOUND. You must review it carefully by hand. Code snippet below
Update: We will be doing a planned reboot of the Windows servers late this afternoon Wednesday January 21, 2014 to begin the process of restoring two of the remaining clients that are still offline.
Scope: This update applies to Tendenci 4 clients on Windows only. It specifically does NOT apply to Tendenci 5 or Tendenci 6 clients on Linux.
To give you an idea of the scope and velocity of hack attacks that continue, these are attempted crimes mind you, I’ve attached a 15 second video taken several days ago of actual attacks on one of our servers INSIDE the allowed ports.
A further update on the 404 errors that the legacy Tendenci 4 clients have been experiencing intermittently. We have been measuring everything possible and tweaking the configuration settings as we see patterns in the logs. Each day generates over 1GB in security alerts across the data centers. All of these are either known attacks, or zero day attempts.
This is what we are fighting and it is relentless. The fact remains that we have protected the legacy sites by moving them from Windows 2003R2 IIS 6 to Windows 2012R2 IIS 8. But to make ASP classic run in IIS 8 we are running the servers in “compatibility mode” which is not an ideal configuration for any technology. And “secure” does not mean “functional” if your sites locked down to the point of not meeting functional requirements.
We have taken a step back and concluded that a technology platform started in 2001 is not up for the cyberwars of 2015. We will have a further update posted later today on possible paths forward for Tendenci 4 clients.
To our Tendenci 4 clients experiencing difficulties, you are ABSOLUTELY STILL MY TOP PRIORITY and the top priority of the entire team.
Huge progress has been made by the team this week and with the help of you, our clients with DNS entries and flexibility and understanding. The good news is that at this point most of you are back on line.
The Tendenci 4 functionality is slowly being recreated on the latest version of Windows Server 2012 R2. In the short term, given I constantly troll the helpdesk, I know y’all are frustrated by the lack of full functionality.
Yet I need you to hang on just a bit longer as this process MUST BE DONE SECURELY. I simply can’t and won’t compromise on that. You don’t rush through open heart surgery and Tendenci, as y’all know, is quite a bit larger than other products because the challenges we address, sites with sometimes 100k users, are much more complex than shopping carts or photos sharing sites.
Still heartbreaking to me is that I am profoundly aware we have a few remaining very important clients to bring back online. And that is a task with multiple people actively working on restoring them, even if they are leaving (and who can blame them) but regardless we will get a stable version for them.
The Good News – The vast majority of Tendenci 4 sites are back online as I type this. Yes you are faced with limited functionality, but have patience as we have to rewrite a lot of code to make the jump to Windows 2012 R2 and most of us have been on the Linux side for a while now. We are seeing your functionality being incrementally restored daily. ETA is probably early next week to get to 75% functionality.
25% of the functionality will only return if we can find a way to securely implement it for all of you such that each client is isolated. Thus the functionality we plan to restore is only within the limits of new security.
What are the known issues for Tendenci 4 clients (the .asp clients)?
Current limitations – all of which are in place to protect you.
Four sites still off line. Top priority. Period. They know who they are and with each I have personally been in contact.
Limited functionality. Everyone else on the Microsoft version of Tendenci who is back up is still facing limited functionality. We are aware of this. No need to submit a ticket. It is coming back as fast as we can do it SECURELY. If we can’t return functionality securely it will not return at all but that is hopefully not going to be the case as I think we can find a work around for all of it. Specifically items that we know are not working and can’t be turned on just yet are posted in a series of posts right after this one. But in brief we are aware of and working on the following.
Notifications – these will be back by early next week at the latest. Like “forgot my password” and “payment submitted” (just not newsletters.)
Newsletters – Not enabled. You will each need to sign up with a third party email relay service. It could even be your own Amazon Simple Email Service account. This is a required change for all clients to sign up with an SMTP relay provider like Mailgun. Newsletter Generator will return; however, Newsletter Send is NOT coming back on the shared mail server. You MUST sign up for a newsletter provider that supports smtp authentication and clean your email lists. This you can start now.
Uploads – these will come back slowly, limited, restricted and only in non executable areas. You will not be able to upload asp files, js files or any form of executable file going forward. This is a permanent change, but really it is a return to how it was designed and at some point we diverged from fundamentals.
FTP – FTP is not coming back to T4 going forward. Never. But before you scream, web sites are not FTP portals and full FTP is no longer feasible. It shouldn’t have been allowed in the first place except to restricted folders and that got lost over the years by our team despite being documented internally. The Internet has changed, we have to change with it. And fortunately there are so many options for you on this. For example on T5 you can FTP into one folder named media. Or use Amazon S3 for static files. So it will be OK. From dedicated servers to S3 buckets to dropbox to gdrive links – you will have lots of options.
WYSIWYG – we will be implementing a stripped down version of one (1) of the two current ftp editors that are in T4. Think minimalistic like wordpress, but you can still jump over to another html editor and use code view to paste tables and such back in for richer formatting if you prefer. Neither of the rich text editors you are used to will be coming back in the same format for security reasons. But you have work arounds.
WYSIWYG uploads – read only files, no java script, no flash. But you can reference those from an external data store (see FTP permanent discontinuation above.)
Next steps. Today yet another firewall that is already in place will have more of its functionality turned on. It is already handling all of the traffic and has quietly been keeping track of things to find patterns that we need to allow (whitelist) so that our other security rules don’t get carried away. Thus it will be brought online slowly.
The new firewall is another layer of security typically called a WAF (web application firewall). While it’s true that we already have a WAF that was running, it was one that reported instead of dynamically taking action to block an attack. Furthermore it was designed like a virus scanner to look for known issues, not the unknown. The new WAF analyzes the traffic passing in-between the firewalls instead of just protocols and ports so it is much more advanced. And if it doesn’t like something, it jumps into action and blocks it.
Remember iRobot? Ya, kind of like that. So we unfortunately WILL experience some false positives. Yet he’s had enough “training” and is ready to be turned loose so us humans can get mad at him and we can fully educate him on what is legitimate traffic and what is not. Studying logs is one thing, but he’s got to get into the wild and test the real world. We ask for your patience on this. Again, it is to protect YOU!
First – it is Wednesday and Microsoft pushes out patches on Tuesday evenings. So in an overabundance of caution we will be rebooting the Tendenci 4 Microsoft Servers between 4 and 4:30 PM today (10 minutes from now or sooner as I type this.)
To our clients on the Open Source Tendenci 5, and the brave clients volunteering to beta test with us on Open Source Tendenci 6 (which I haven’t even had a chance to blog about yet) – all of y’all are still online, have had zero downtime and remain rock solid. Linux and Django and Containers are definitely proving how much stronger they can make Tendenci. This is done by design and made possible by virtue of the flexibility and low cost associated open source in the cloud. It is achieved through isolation, portability and flexibility. I hope you are not frustrated by our team being laser focused on helping our long time clients who experienced outages. I apologize for the slower response time. I know you are missing reports and other items that were there in T4; they will return to being my focus once all of our data centers are fully back online regardless of technology.
Further I am aware of the fact this has thrown numerous projects wildly behind on their timelines and disrupted you as well. All things considered, if your site was offline, you would demand the same from us – to focus on bringing everyone back up.
Ethically, we (Tendenci) must stay the course and get these sites functional. Even now I feel guilty taking the time to write this instead of working on the technical details. I also know people need to know we have a plan (we do) and there is an end in site (there is) and that it will be a success (it will be). And that we have learned from it (we have).
To our Tendenci4 legacy clients on the Microsoft platform, you are and have been MY TOP PRIORITY and the top priority of the entire team. We knew the Internet had changed, just perhaps not how much it had changed in the category of zero day types of threats. See next post.
Another quick update on the status of the network outages. Tuesday Jan 6 2015 – we are still focused on a few long-standing clients experiencing outages or reduced functionality who are still on Tendenci 4, the powerful but legacy version of Tendenci build on the Microsoft platform.
Yesterday was another 12+ hour day for most of the team. They are working hard, but I do insist they sleep some as typos and DNS entries don’t work well together.
If we didn’t communicate directly, rest assured we are working hard to get everyone restored. ~ Ed Schipul, CEO, Tendenci
Great news. Sites are up and running in a jailed IP block while we scan and test. If all goes well everyone will be back online soon. If it fails the security tests, then to be frank we won’t allow it to be opened up and thus we need your help.
Geeky stuff: This is a quick update from Ed given my communications team is out. The current status of the rebuilding of sites for the portion of our clients who have been offline for a significant period of time is that our new servers in a new higher security “IP Jail” is running well as of this morning.
We are and will continue to scan and work to remediate any compromised files. The original operating systems have been formatted/replaced and all legacy Windows T4 clients that were on Windows 2003R2 are being jumped from IIS 6 to IIS 8.5 on Windows 2012R2 so you will be on the most secure Microsoft Platform ever.
(Note – No Tendenci 5 clients had any issues and I apologize to y’all for the lack of responsiveness on day to day issues as our team addressed the issues for our other clients.)
For our T4 clients coming back online in the new environment, yes, there will be issues as we change IP addresses and email relays and the like but our timeline of Monday is still on track, hopefully sooner. And perhaps a few strong clients will volunteer not to be online first, but to be a volunteer to go through a third party security audit of their site on behalf of everyone who has been a victim of this unfortunate crime. It is like a stress test that attacks a site in a silo to be sure when opened to the public it works as designed. I believe this is an important step to get third party validation before bringing everyone back online for the sake of safety and security.
As a CEO it is my job to foresee and prevent these occurrences, and in this case I missed the mark by a long shot. The Monday deadline will only be possible with some assistance from the community testing a few sites off of the public network for functionality as we work out the transition of over 50 sites to an entirely new cloud based security system that may be (OK, it is….) locked down quite tight. Yet it is better to lock and release, than to risk having to protect our clients by shutting down a server again.
And as I have said I apologize again. This is a crime. We are documenting it for the authorities as best we can. But that isn’t the point. The point is we work with caused based and people trying to change the world for the better. That is what Tendenci IS! And we let you down. Help us fix it because it isn’t us and you, it is just “us”.