We at Tendenci are sorry that the network outage that is affecting approximately 40 Tendenci websites remains unresolved. It has taken some time to understand the full scope of the issue and, unfortunately, getting all our systems back online has taken longer than anticipated. We understand that a website is
critically important to every organization, and we take our responsibility – and the trust of Tendenci users – very seriously. We are mobilizing all available resources to address the situation as quickly as possible, but it is unlikely that affected websites will be fully functional for several more days.
Here is what we know:
On Wednesday, 11/26 at least one Tendenci server experienced a URL redirect attack. Spammers use this type of attack to send website traffic to other desired websites. In this case, traffic to certain Tendenci websites was redirected to a website selling handbags.
The Tendenci team quickly responded, but this was followed by an additional distributed denial-of-service (DDoS) attack on our network. This same type of high bandwidth attack took down Xbox Live this week.
The network outage is affecting approximately 40 Tendenci websites. Only websites on the legacy Tendenci 4 software.
In our efforts to restore network service, we have uncovered a number of things that can be improved in our security systems and practices that left our network vulnerable to the DDoS attack. Addressing these vulnerabilities is a necessary first step in restoring the network and getting all Tendenci websites back up.
We are working hard to have at least some of the affected websites back up in a limited capacity today, but it may be the end of the week or longer before all affected sites are restored.
We understand that having a website go down for days on end is unacceptable, and it breaks my heart to know that this week’s events may have shaken the community’s trust in Tendenci. We are working around the clock to restore our systems, and we will take what we learn during this process to improve security protocols for all Tendenci websites.
At Tendenci we are aware of network outages at several data centers. They are being worked on. We are aware of it. We are working on it. Please be patient.
The image below is the world we live in and we are defending our network against it. Black Friday and Cyber Monday aren’t holidays around here – they are battle zones unfortunately.
We understand how critical your infrastructure is to you and take it very seriously. This is the world we live in now. (read more after the jump)
IP Viking Network Attacks Map
All of our technical people are working on bringing the clients on Tendenci 4 back online and not taking phone calls or emails as the issues are known. Some require moving large amounts of data which takes time. To not make backups prior to moving servers isn’t acceptable either.
The solution is, unfortunately, we will need to further bulk up load balancers and the already double and triple firewall, WAF (web application firewalls), virus scanners.
Tendenci 4 used basic Dreamweaver DWT files so designers could see everything they were doing. Then we used very strongly typed and very exact html comments along with Python to chop up the template to integrate it with the code.
The benefits were huge in that designers could design, programmers could program. Life was good. Well, almost, there was the whole 2002 use of tables because Netscape and IE were fighting and munged up generated HTMl inside of the script blocks rendered out. Very non MVC but it worked. And it allowed for amazing designer freedom.
Django – Tendenci has been a team effort since 2001 and the big jump came in 2009 when we started the rewrite in fully open source code (old news, I won’t bore you with it, but you can google it.). Django, a web framework written in Python, replaces the custom written framework we built in Tendenci 4 from scratch (hint – avoid writing frameworks from scratch. It makes no sense anymore. Time changes things.)
The problem? Django templates are out of date, they use too much magic (how many times does a designer add a code block and forget the templatetag? Or more common, remove a code block and leave the templatetag out of forgetfulness or fear.
And the worst part – you CAN’T SEE IT unless it is rendered. No more Dreamweaver or any other wysiwyg tools. We take visual people and thrown them into a text based world. For comparison, this would be like coding in Python in Photoshop. It doesn’t make sense.
I wish this post had a happy ending. It is a happy goal. A worthy goal. But the state of affairs with Django and Templates remains stagnant at best as explained here:
The new Tendenci helpdesk system has been a real eye opener for us. I fear we have let you down, actually I know we have, and I also know the blame resides squarely on my shoulders as CEO. Nobody else. If you have great people (and we do, and we’ve lost some of them now) and the results aren’t great to stellar then it is a SYSTEMS PROBLEM. And that is a CEO’s job. I can dissect it down to particular leverage points, staffing levels, transparency, lack of integrated systems, a naive belief in accepting suggestions from everyone without stress testing them, etc, but ultimately it falls on me.
So let’s be clear about who is at fault for our turn-over of late. Me. There are many things in hindsight I should have done to prevent it. It has cost me personally, professionally and on every other level. As a CEO you don’t get that luxury. Because it’s not about me, it’s about you. And isn’t that the real issue? It’s how it has impacted everyone else. We all count on Tendenci to just work!
I’m familiar with failure. It hurts more when it is your friends. I won’t sugarcoat that. Nor will I give up as I have an obligation to our clients and rebuilding is the only option, up to and including a few critical conversations to ensure a strong future for all of us. We need you, our clients and open source contributors, more than you need us now that we are open source and I fully get that.
LEARN FROM MY MISTAKES – NO PER USER PRICING
We liked zendesk, but as I have blogged in other places, I still view the per-named-user business model as a failed business model that is inherently unfair to clients and part time employees. The economics incentives of named-user-licensing is to create devastating silos of information to save 70 dollars a month for someone to just check in.
It is particularly harmful for remote workers who need to be able to see more information, not less, to keep up with those in centralized teams. The per-named-user-license model creates an economic incentive to do the opposite. “Does this contractor really need to use the toll road? Nah, let’s just assign them work and they don’t need to know the backstory or inside jokes of the office.” – see? Fundamentally evil.
Why did it take me so long to figure that out? #duh
I’ll grant you that some companies are starting to find a hybrid pricing model.
Chinese Year of the Horse
Hipchat has a great offering and, then, sure we upgrade for the 2 or 3 dollars per person to get the call functionality. That is reasonable. But we love hipchat internally.
Salesforce? Not reasonable. Long term contracts, integrators that don’t work and nobody held accountable. They aren’t so much “no software” as “software that isn’t as evil as Oracle.” – not a big differentiator IMHO.
Given a choice I prefer to work in the environment we program in – Python/Django/Postgres/Ubuntu and hence we went with Django-Helpdesk to organize and provide transparency and accountability. Drop in Django-Model-Reports and I can actually see the level of support my clients are receiving. And while clearly my responsibility, results are results and they completely unacceptable.
The data and timelines were just hidden. No more. You can’t manage what you can’t see and I can see it now. I will need some help from clients to copy their support email account to keep from reverting to silos again.
SIMPLIFY SIMPLIFY SIMPLIFY
For the helpdesk and ongoing projects in the shop, with candor, it may take a month or longer to dig out and create a refined simplified simplified simplified system. But we’ve done it before and we’ll do it again.
Thank to our clients for your patience. I frustrated a lot of clients and lost some great clients and employees as a result of not having systems in place. I apologize.
IT’S ABOUT CUSTOMER SATISFACTION
Yet as I have said many times –
“Clients don’t want great customer service, they want great customer satisfaction. They want the software to work so they don’t have to call in and get a quick response. They want it to work so they don’t have to call or email in the first place.”
– we need better systems. This is one of them.
We will be announcing partnerships to fill in and take care of holes made apparent by the extreme transparency of the new helpdesk system.
DIFFERENTIATION
We will start with differentiation. Clients on paid support contracts should and will go straight to the top of the queue. Our old systems didn’t have the ability to filter and prioritize. Now we can. I will announce these partnerships, some with former Schipul/Tendenci employees who already are familiar with our clients and strict security systems.
We will do our best to forge ahead, rebuild the client contact and communication portions of Tendenci the company. Our technical team is still charging forward with long term goals rest assured. Like the return of the newsletter generator.
I thank you for your business, your contributions to the software, for being a part of the global tendenci team, and for not giving up. I won’t either. It’s not me or you, it’s us. #peace
All – we are migrating entire network including our dev servers and email relays this weekend, November 7,8,9 2014. That will be follow up by some structural reorganizations within the company as well to align our structure more closely to that of our evolving client base. Focusing externally instead of internally and improving prioritization.
You deserve details. I don’t have all of them yet. I committed to being more open about our communication and I’m sticking by that commitment even if it means I have to post something like “we are changing and stuff” because we are.
You’ll know more the minute I have made sure clients and employees are taken care of in the transitions to the best of my abilities.
DNS updates ahoy! If you see something unusual visit https://helpdesk.tendenci.com and submit a ticket or post on the forum. We love our clients. And we’ve been in business since 1997. For the backbone-crew of the company, you know, we’ve got a couple of years on us, and forgive us if we don’t remember every DNS entry for the last 17 years.
I am working to clean up years of cruft between the schipul and the tendenci aliases across all of our sites. It took me a while to figure out that the majority of sub-domain entries (like the “www” part of your domain) had long since been moved to production sites. So, forgive me, but I kinda went nuclear and just cleared out three or four hundred sets of entries from ten years ago.
HOWEVER, it is possible that you may have had a graphic or image using one of these old subdomains from 2005. Keep an eye out for that.
If this should cause a problem with your site it’s a great opportunity to clean things up. Which is why I did it on a Saturday night so it will replicate by Sunday and I’ll personally be monitoring the queue over the weekend (this is Ed typing.)
These changes literally go back to 2005 so my bet is nobody on your team will remember. But they CAN fix it with the template editor by fixing any old out of date links. You’ll score better in search engines as an added bonus and you don’t even need any help from us.
Or if you have the budget to prepay for mods, we can help. Of course we’d rather focus on building out the Tendenci platform. For free assistance visit https://community.tendenci.com but otherwise billable.
I thought I should give y’all a heads up about us clearing out the cobwebs from Halloween and encourage you to keep an eye out for anything that looks different. If you see it, please fix it. We can help, but that is billable obviously ( I wish my employees worked for free but they currently don’t. Nor do they donate money to the company. It’s a conundrum.)
We wouldn’t be cause related software developers or working at cause related companies that aspire to the #openeverythingOSCON type of ethos to begin with.
First – it is to serve.
To achieve our communication goals we are using some amazing open source software based on django and postgres just like tendenci of course – The Misago Open Source Discussion Forums is used to power the new https://community.tendenci.com site to provide a “place” for that dialog to take happen. Developers are welcome to continue posting issues on github of course, clients who prefer to have us (or you!) manage their tendenci hosting will submit billable tendenci support requests, but there is something different about a forum that is toned down and not quite as public. I can’t quite explain why.
As the CEO of Tendenci, the company and the software – I apologize for my lack of communication to you. I apologize to you -our long-time clients and our our new clients. I apologize to the developers who have deployed Tendenci on your own servers with little or no documentation and not posting public developer training.
I hope you forgive me (OK, I flipped this one around from “I forgive you” to be reflective of our situation at Tendenci.)
Your voice matters to me very much and quite frankly I haven’t kept up communication. Instead I have let the media, public, private and social – dictate the dialog. I know better.
I can start the dialog by dispelling a few rumors right away with some clarity to questions I have (actually) been asked.
No I’m not moving to LA. I just fly a lot. And…. it wasn’t LA anyway….
Yes I love San Francisco and Houston both. (not sure about LA #heh)
Yes I still love Houston more. And yes the Bay Area can also be awesome.
Yes, there will be more changes at Tendenci the company and with the pending release of Tendenci 6.
No I won’t do any more phone interviews with reporters – only email or recorded so I don’t get selectively quoted.
No we aren’t going out of business. Been at it 17 years. Still here. Still serving my clients. Now expanding Tendenci to build a global legacy that is better than the proprietary vendors like Blackbaud while we still make a profit and grow. What else does anyone recommend I do that helps my clients the most? (Feel free to comment. But make no mistake, the tipping point is near. Open Source will win. Why should that be different for non-profits and association management software needs?)
Yes we are restructuring and that includes costs that go along with it. We’ve had some ups and downs and yes we are downsizing our offices in Houston and being inclusive of more remote team members.
True rumor – NO, Tendenci 5 does NOT have all of the functionality of Tendenci 4. Nor the other way around. They are different. It’s like going from a PC to a Mac. It’s different. Don’t convert if it isn’t for you yet.
And number 10….. yes, apparently people still actually read the paper. Didn’t realize I was newsworthy. Not sure if that is good or not. I just want to build Tendenci to “Connect and Organize the World’s People. Do Good.” Sorry if I messed up the PR part by having my head down laser focused on Tendenci.
I deleted the rest of this blog post to keep it short. Consider it my own test to myself to keep the dialog going. And I am committed to keeping open communication. Links and systems solutions in the next few posts.
Thank you,
Ed
PS. Tendenci is all about YOU! I get that. I deeply respect that. #candid #honest #servant
The best part about attending client events, well there are several really, but they include things like:
Attending a client event – It’s a chance to say thank you to our client! With over 300+ that we host it gets harder and harder to say thank you as I don’t know many of you personally. A while back we added up a count of users logging in different Tendenci sites just in our data centers (we have three). Just the totals of course with no identifying data, and it was over 1.2 million souls. It makes me proud and also nervous when we tweak the Tendenci user interface!
Attending a client event – I listen and YOU tell us how to improve what is both yours and our product! Yes, Tendenci is fully open source and you can download it from github at https://github.com/tendenci. So there may not be a direct financial benefit. Yet that is truly OK as Tendenci is clearly not just about the money. And a lot of y’all do host with us. We appreciate both the developers who use and tell us how to improve and the clients we host and manage everything for them.
Speaking of feedback – be sure to tell us how to improve the software on our new community discussion forum at http://community.tendenci.com. We talk to a lot of developers on github so the forum is more for…. you know… humans who use the software day to day. What are your needs? What would you like to see in Tendenci?
I’ll do another blog post about the Tendenci community forums soon. (This also means that we now have forums integrated with Tendenci. We found another great Django based open source project named Misago which works great with Tendenci given we also use Django. Next is the single-sign-on server process….
One of the things that excites us the most at Tendenci is seeing other people really accelerate open source across all sectors. Not surprisingly we see community building as essential in both academics and in business. Because business… can be just a bit too “business like” when we like to work hard but also play hard. Another thing I love to do is talk to clients. And being open source, and as a company with no commissions but that takes care of our people by doing the right thing, it’s fun sometimes to highlight great open source projects that aren’t directly related to Tendenci (but OK, all of these do integrate with Tendenci as they are all based on Django as well.)
Open Source Django projects that Integrate with Tendenci
How about a community discussion board with karma points, community building, banning of bad guys, rewards for the helpful. It includes a playful board you could say personifies the balance between making a gamified application and what some refer to as gamification. I don’t want a gold coin, but I do like the respect of my peers. Misago is just that. And being Django it integrates with Tendenci which is also Django.
Our implementation of EdX for internal use isn’t quite as pretty but it still shows that we love education. EdX is great to train employees, members of your association, your volunteers and much more. The screenshots below are our actual EdX demo site as well as the one we use internally.
EdX Open Learning Screenshot on Tendenci
Course writing in Django EdX. Oh, and you can import courses from other people as well! Sharing is one of the values of Open Source! But what about course material? How about CNX Openstax! Write your textbook online, let people download it or print on demand. Most CNX textbooks are even free! And of course CNX is also written in Django and Python like Tendenci which makes it part of our big integrated solution that you can deploy OPEN SOURCE. You stay in control.
Is this a trend? Yup. We aren’t the only ones who have noticed. You can read about the growth of MOOC’s or massive open online courses in several tech and education articles and quickly moving into the mainstream.
The growth of MOOCs and online courses
How do you track all of those students if you are a giant school? Well, how about Django-SIS? Yup, manage an entire school, even alumni with django SIS (School Information Systems).
School Information Systems – School Management Software
