To our clients on the Open Source Tendenci 5, and the brave clients volunteering to beta test with us on Open Source Tendenci 6 (which I haven’t even had a chance to blog about yet) – all of y’all are still online, have had zero downtime and remain rock solid. Linux and Django and Containers are definitely proving how much stronger they can make Tendenci. This is done by design and made possible by virtue of the flexibility and low cost associated open source in the cloud. It is achieved through isolation, portability and flexibility. I hope you are not frustrated by our team being laser focused on helping our long time clients who experienced outages. I apologize for the slower response time. I know you are missing reports and other items that were there in T4; they will return to being my focus once all of our data centers are fully back online regardless of technology.
Further I am aware of the fact this has thrown numerous projects wildly behind on their timelines and disrupted you as well. All things considered, if your site was offline, you would demand the same from us – to focus on bringing everyone back up.
Ethically, we (Tendenci) must stay the course and get these sites functional. Even now I feel guilty taking the time to write this instead of working on the technical details. I also know people need to know we have a plan (we do) and there is an end in site (there is) and that it will be a success (it will be). And that we have learned from it (we have).
To our Tendenci4 legacy clients on the Microsoft platform, you are and have been MY TOP PRIORITY and the top priority of the entire team. We knew the Internet had changed, just perhaps not how much it had changed in the category of zero day types of threats. See next post.
Another quick update on the status of the network outages. Tuesday Jan 6 2015 – we are still focused on a few long-standing clients experiencing outages or reduced functionality who are still on Tendenci 4, the powerful but legacy version of Tendenci build on the Microsoft platform.
Yesterday was another 12+ hour day for most of the team. They are working hard, but I do insist they sleep some as typos and DNS entries don’t work well together.
If we didn’t communicate directly, rest assured we are working hard to get everyone restored. ~ Ed Schipul, CEO, Tendenci
PS – Please continue to utilize the helpdesk at https://helpdesk.tendenci.com/ for the fastest response. It is the only thing all of us are checking. And luckily the volume is dropping as the sites are being restored. Specifically if it is important enter it directly on the help web site as opposed to starting with an inbound email as then any follow up email communication will continue to be attached to that thread. I thank you for helping us help you.
Great news. Sites are up and running in a jailed IP block while we scan and test. If all goes well everyone will be back online soon. If it fails the security tests, then to be frank we won’t allow it to be opened up and thus we need your help.
Geeky stuff: This is a quick update from Ed given my communications team is out. The current status of the rebuilding of sites for the portion of our clients who have been offline for a significant period of time is that our new servers in a new higher security “IP Jail” is running well as of this morning.
We are and will continue to scan and work to remediate any compromised files. The original operating systems have been formatted/replaced and all legacy Windows T4 clients that were on Windows 2003R2 are being jumped from IIS 6 to IIS 8.5 on Windows 2012R2 so you will be on the most secure Microsoft Platform ever.
(Note – No Tendenci 5 clients had any issues and I apologize to y’all for the lack of responsiveness on day to day issues as our team addressed the issues for our other clients.)
For our T4 clients coming back online in the new environment, yes, there will be issues as we change IP addresses and email relays and the like but our timeline of Monday is still on track, hopefully sooner. And perhaps a few strong clients will volunteer not to be online first, but to be a volunteer to go through a third party security audit of their site on behalf of everyone who has been a victim of this unfortunate crime. It is like a stress test that attacks a site in a silo to be sure when opened to the public it works as designed. I believe this is an important step to get third party validation before bringing everyone back online for the sake of safety and security.
As a CEO it is my job to foresee and prevent these occurrences, and in this case I missed the mark by a long shot. The Monday deadline will only be possible with some assistance from the community testing a few sites off of the public network for functionality as we work out the transition of over 50 sites to an entirely new cloud based security system that may be (OK, it is….) locked down quite tight. Yet it is better to lock and release, than to risk having to protect our clients by shutting down a server again.
And as I have said I apologize again. This is a crime. We are documenting it for the authorities as best we can. But that isn’t the point. The point is we work with caused based and people trying to change the world for the better. That is what Tendenci IS! And we let you down. Help us fix it because it isn’t us and you, it is just “us”.
At Tendenci we are aware of network outages at several data centers. They are being worked on. We are aware of it. We are working on it. Please be patient.
The image below is the world we live in and we are defending our network against it. Black Friday and Cyber Monday aren’t holidays around here – they are battle zones unfortunately.
We understand how critical your infrastructure is to you and take it very seriously. This is the world we live in now. (read more after the jump)
IP Viking Network Attacks Map
All of our technical people are working on bringing the clients on Tendenci 4 back online and not taking phone calls or emails as the issues are known. Some require moving large amounts of data which takes time. To not make backups prior to moving servers isn’t acceptable either.
The solution is, unfortunately, we will need to further bulk up load balancers and the already double and triple firewall, WAF (web application firewalls), virus scanners.
The new Tendenci helpdesk system has been a real eye opener for us. I fear we have let you down, actually I know we have, and I also know the blame resides squarely on my shoulders as CEO. Nobody else. If you have great people (and we do, and we’ve lost some of them now) and the results aren’t great to stellar then it is a SYSTEMS PROBLEM. And that is a CEO’s job. I can dissect it down to particular leverage points, staffing levels, transparency, lack of integrated systems, a naive belief in accepting suggestions from everyone without stress testing them, etc, but ultimately it falls on me.
So let’s be clear about who is at fault for our turn-over of late. Me. There are many things in hindsight I should have done to prevent it. It has cost me personally, professionally and on every other level. As a CEO you don’t get that luxury. Because it’s not about me, it’s about you. And isn’t that the real issue? It’s how it has impacted everyone else. We all count on Tendenci to just work!
I’m familiar with failure. It hurts more when it is your friends. I won’t sugarcoat that. Nor will I give up as I have an obligation to our clients and rebuilding is the only option, up to and including a few critical conversations to ensure a strong future for all of us. We need you, our clients and open source contributors, more than you need us now that we are open source and I fully get that.
LEARN FROM MY MISTAKES – NO PER USER PRICING
We liked zendesk, but as I have blogged in other places, I still view the per-named-user business model as a failed business model that is inherently unfair to clients and part time employees. The economics incentives of named-user-licensing is to create devastating silos of information to save 70 dollars a month for someone to just check in.
It is particularly harmful for remote workers who need to be able to see more information, not less, to keep up with those in centralized teams. The per-named-user-license model creates an economic incentive to do the opposite. “Does this contractor really need to use the toll road? Nah, let’s just assign them work and they don’t need to know the backstory or inside jokes of the office.” – see? Fundamentally evil.
Why did it take me so long to figure that out? #duh
I’ll grant you that some companies are starting to find a hybrid pricing model.
Chinese Year of the Horse
Hipchat has a great offering and, then, sure we upgrade for the 2 or 3 dollars per person to get the call functionality. That is reasonable. But we love hipchat internally.
Salesforce? Not reasonable. Long term contracts, integrators that don’t work and nobody held accountable. They aren’t so much “no software” as “software that isn’t as evil as Oracle.” – not a big differentiator IMHO.
Given a choice I prefer to work in the environment we program in – Python/Django/Postgres/Ubuntu and hence we went with Django-Helpdesk to organize and provide transparency and accountability. Drop in Django-Model-Reports and I can actually see the level of support my clients are receiving. And while clearly my responsibility, results are results and they completely unacceptable.
The data and timelines were just hidden. No more. You can’t manage what you can’t see and I can see it now. I will need some help from clients to copy their support email account to keep from reverting to silos again.
SIMPLIFY SIMPLIFY SIMPLIFY
For the helpdesk and ongoing projects in the shop, with candor, it may take a month or longer to dig out and create a refined simplified simplified simplified system. But we’ve done it before and we’ll do it again.
Thank to our clients for your patience. I frustrated a lot of clients and lost some great clients and employees as a result of not having systems in place. I apologize.
IT’S ABOUT CUSTOMER SATISFACTION
Yet as I have said many times –
“Clients don’t want great customer service, they want great customer satisfaction. They want the software to work so they don’t have to call in and get a quick response. They want it to work so they don’t have to call or email in the first place.”
– we need better systems. This is one of them.
We will be announcing partnerships to fill in and take care of holes made apparent by the extreme transparency of the new helpdesk system.
DIFFERENTIATION
We will start with differentiation. Clients on paid support contracts should and will go straight to the top of the queue. Our old systems didn’t have the ability to filter and prioritize. Now we can. I will announce these partnerships, some with former Schipul/Tendenci employees who already are familiar with our clients and strict security systems.
We will do our best to forge ahead, rebuild the client contact and communication portions of Tendenci the company. Our technical team is still charging forward with long term goals rest assured. Like the return of the newsletter generator.
I thank you for your business, your contributions to the software, for being a part of the global tendenci team, and for not giving up. I won’t either. It’s not me or you, it’s us. #peace
All – we are migrating entire network including our dev servers and email relays this weekend, November 7,8,9 2014. That will be follow up by some structural reorganizations within the company as well to align our structure more closely to that of our evolving client base. Focusing externally instead of internally and improving prioritization.
You deserve details. I don’t have all of them yet. I committed to being more open about our communication and I’m sticking by that commitment even if it means I have to post something like “we are changing and stuff” because we are.
You’ll know more the minute I have made sure clients and employees are taken care of in the transitions to the best of my abilities.
DNS updates ahoy! If you see something unusual visit https://helpdesk.tendenci.com and submit a ticket or post on the forum. We love our clients. And we’ve been in business since 1997. For the backbone-crew of the company, you know, we’ve got a couple of years on us, and forgive us if we don’t remember every DNS entry for the last 17 years.
I am working to clean up years of cruft between the schipul and the tendenci aliases across all of our sites. It took me a while to figure out that the majority of sub-domain entries (like the “www” part of your domain) had long since been moved to production sites. So, forgive me, but I kinda went nuclear and just cleared out three or four hundred sets of entries from ten years ago.
HOWEVER, it is possible that you may have had a graphic or image using one of these old subdomains from 2005. Keep an eye out for that.
If this should cause a problem with your site it’s a great opportunity to clean things up. Which is why I did it on a Saturday night so it will replicate by Sunday and I’ll personally be monitoring the queue over the weekend (this is Ed typing.)
These changes literally go back to 2005 so my bet is nobody on your team will remember. But they CAN fix it with the template editor by fixing any old out of date links. You’ll score better in search engines as an added bonus and you don’t even need any help from us.
Or if you have the budget to prepay for mods, we can help. Of course we’d rather focus on building out the Tendenci platform. For free assistance visit https://community.tendenci.com but otherwise billable.
I thought I should give y’all a heads up about us clearing out the cobwebs from Halloween and encourage you to keep an eye out for anything that looks different. If you see it, please fix it. We can help, but that is billable obviously ( I wish my employees worked for free but they currently don’t. Nor do they donate money to the company. It’s a conundrum.)
We wouldn’t be cause related software developers or working at cause related companies that aspire to the #openeverythingOSCON type of ethos to begin with.
First – it is to serve.
To achieve our communication goals we are using some amazing open source software based on django and postgres just like tendenci of course – The Misago Open Source Discussion Forums is used to power the new https://community.tendenci.com site to provide a “place” for that dialog to take happen. Developers are welcome to continue posting issues on github of course, clients who prefer to have us (or you!) manage their tendenci hosting will submit billable tendenci support requests, but there is something different about a forum that is toned down and not quite as public. I can’t quite explain why.
As the CEO of Tendenci, the company and the software – I apologize for my lack of communication to you. I apologize to you -our long-time clients and our our new clients. I apologize to the developers who have deployed Tendenci on your own servers with little or no documentation and not posting public developer training.
I hope you forgive me (OK, I flipped this one around from “I forgive you” to be reflective of our situation at Tendenci.)
Your voice matters to me very much and quite frankly I haven’t kept up communication. Instead I have let the media, public, private and social – dictate the dialog. I know better.
I can start the dialog by dispelling a few rumors right away with some clarity to questions I have (actually) been asked.
No I’m not moving to LA. I just fly a lot. And…. it wasn’t LA anyway….
Yes I love San Francisco and Houston both. (not sure about LA #heh)
Yes I still love Houston more. And yes the Bay Area can also be awesome.
Yes, there will be more changes at Tendenci the company and with the pending release of Tendenci 6.
No I won’t do any more phone interviews with reporters – only email or recorded so I don’t get selectively quoted.
No we aren’t going out of business. Been at it 17 years. Still here. Still serving my clients. Now expanding Tendenci to build a global legacy that is better than the proprietary vendors like Blackbaud while we still make a profit and grow. What else does anyone recommend I do that helps my clients the most? (Feel free to comment. But make no mistake, the tipping point is near. Open Source will win. Why should that be different for non-profits and association management software needs?)
Yes we are restructuring and that includes costs that go along with it. We’ve had some ups and downs and yes we are downsizing our offices in Houston and being inclusive of more remote team members.
True rumor – NO, Tendenci 5 does NOT have all of the functionality of Tendenci 4. Nor the other way around. They are different. It’s like going from a PC to a Mac. It’s different. Don’t convert if it isn’t for you yet.
And number 10….. yes, apparently people still actually read the paper. Didn’t realize I was newsworthy. Not sure if that is good or not. I just want to build Tendenci to “Connect and Organize the World’s People. Do Good.” Sorry if I messed up the PR part by having my head down laser focused on Tendenci.
I deleted the rest of this blog post to keep it short. Consider it my own test to myself to keep the dialog going. And I am committed to keeping open communication. Links and systems solutions in the next few posts.
Thank you,
Ed
PS. Tendenci is all about YOU! I get that. I deeply respect that. #candid #honest #servant
