Our program engineer team will continue to work on the site through the evening.
We will be working 24/7 on this until it is resolved.
We will continue to post updates to the blog as we have more detailed information available.
Update – Running Back Ups Prior to a Restore
We still working to resolve the network outages that some of our T4 clients are experiencing.
We are in the process of restoring several large data stores prior to attempting to bring them online.
So far we have not run into any unexpected obstacles in restoring data bases and are therefore are still expecting the servers to be back up and functioning to full capacity within the 24-48 hour timeline previously posted.
We will continue to post updates to the blog.
Network Outage Update
We are aware of the continued outages on some of our data servers and apologize to those who are experiencing those outages
The servers experienced a traffic overage due to high traffic over the holiday weekend. We are in the process of migrating sites to a higher capacity server and anticipate all servers to be functioning at full capacity in the next 24-48 hours. We appreciate your patience.
We will continue to keep the blog.tendenci.com updated as we have further information available.
Thank you again for your patience while we migrate.
We have our team focused on this issue and are working to have all servers back up to full capacity as soon as possible.
network outages at several data centers
At Tendenci we are aware of network outages at several data centers. They are being worked on. We are aware of it. We are working on it. Please be patient.
The image below is the world we live in and we are defending our network against it. Black Friday and Cyber Monday aren’t holidays around here – they are battle zones unfortunately.
We understand how critical your infrastructure is to you and take it very seriously. This is the world we live in now. (read more after the jump)
All of our technical people are working on bringing the clients on Tendenci 4 back online and not taking phone calls or emails as the issues are known. Some require moving large amounts of data which takes time. To not make backups prior to moving servers isn’t acceptable either.
The solution is, unfortunately, we will need to further bulk up load balancers and the already double and triple firewall, WAF (web application firewalls), virus scanners.
Step 1 is to get everything back up.
Tendenci Templating – we can Improve
Tendenci 4 used basic Dreamweaver DWT files so designers could see everything they were doing. Then we used very strongly typed and very exact html comments along with Python to chop up the template to integrate it with the code.
The benefits were huge in that designers could design, programmers could program. Life was good. Well, almost, there was the whole 2002 use of tables because Netscape and IE were fighting and munged up generated HTMl inside of the script blocks rendered out. Very non MVC but it worked. And it allowed for amazing designer freedom.
Django – Tendenci has been a team effort since 2001 and the big jump came in 2009 when we started the rewrite in fully open source code (old news, I won’t bore you with it, but you can google it.). Django, a web framework written in Python, replaces the custom written framework we built in Tendenci 4 from scratch (hint – avoid writing frameworks from scratch. It makes no sense anymore. Time changes things.)
The problem? Django templates are out of date, they use too much magic (how many times does a designer add a code block and forget the templatetag? Or more common, remove a code block and leave the templatetag out of forgetfulness or fear.
And the worst part – you CAN’T SEE IT unless it is rendered. No more Dreamweaver or any other wysiwyg tools. We take visual people and thrown them into a text based world. For comparison, this would be like coding in Python in Photoshop. It doesn’t make sense.
I wish this post had a happy ending. It is a happy goal. A worthy goal. But the state of affairs with Django and Templates remains stagnant at best as explained here:
https://speakerdeck.com/mitsuhiko/lets-talk-about-templates
We can improve and the future will be brighter when we can just drop in a bootstrap-theme.css file and change our sites as easily as css zengarden.
Newsletter Generator Coming Soon!
We are excited to be in the testing phase of our Tendenci Newsletter Generator on alpha sites!
This feature will be available for our general community when individual sites are upgraded to Tendenci 6 from Tendenci 5.2. But just to give you some eye candy, and to save Executive Directors HOURS of work building newsletters, here are some screen shots of what is coming.
Which results in a newsletter similar to the quick test below. Forgive the graphics – but you get the concept of aggregating the content so you can use it in the email provider of your choice with a simple copy paste with fully qualified URLs for the images so they work via email.
There is a slight catch that only sites using the new navigation will be able to use it as the old navigation (blue bar) causes issues with mobile and we are trying to make these frameworks coexist as we have settled on Bootstrap3 for the core, but a front end designer can override this.
In other words, there are a few strings attached but it truly saves our webmasters hours and hours of work.
Next up is to allow you to put in an account for mailgun or SES if you don’t need full reporting from higher end tools, and many of us don’t given we can derive similar information on open rates from mailgun and use google analytics for AB testing etc….
Have general questions about Tendenci? Check out our community forum! https://community.tendenci.com/
New Helpdesk System in Full Swing
TRANSPARENCY
The new Tendenci helpdesk system has been a real eye opener for us. I fear we have let you down, actually I know we have, and I also know the blame resides squarely on my shoulders as CEO. Nobody else. If you have great people (and we do, and we’ve lost some of them now) and the results aren’t great to stellar then it is a SYSTEMS PROBLEM. And that is a CEO’s job. I can dissect it down to particular leverage points, staffing levels, transparency, lack of integrated systems, a naive belief in accepting suggestions from everyone without stress testing them, etc, but ultimately it falls on me.
So let’s be clear about who is at fault for our turn-over of late. Me. There are many things in hindsight I should have done to prevent it. It has cost me personally, professionally and on every other level. As a CEO you don’t get that luxury. Because it’s not about me, it’s about you. And isn’t that the real issue? It’s how it has impacted everyone else. We all count on Tendenci to just work!
I’m familiar with failure. It hurts more when it is your friends. I won’t sugarcoat that. Nor will I give up as I have an obligation to our clients and rebuilding is the only option, up to and including a few critical conversations to ensure a strong future for all of us. We need you, our clients and open source contributors, more than you need us now that we are open source and I fully get that.
LEARN FROM MY MISTAKES – NO PER USER PRICING
We liked zendesk, but as I have blogged in other places, I still view the per-named-user business model as a failed business model that is inherently unfair to clients and part time employees. The economics incentives of named-user-licensing is to create devastating silos of information to save 70 dollars a month for someone to just check in.
It is particularly harmful for remote workers who need to be able to see more information, not less, to keep up with those in centralized teams. The per-named-user-license model creates an economic incentive to do the opposite. “Does this contractor really need to use the toll road? Nah, let’s just assign them work and they don’t need to know the backstory or inside jokes of the office.” – see? Fundamentally evil.
Why did it take me so long to figure that out? #duh
I’ll grant you that some companies are starting to find a hybrid pricing model.
Hipchat has a great offering and, then, sure we upgrade for the 2 or 3 dollars per person to get the call functionality. That is reasonable. But we love hipchat internally.
Salesforce? Not reasonable. Long term contracts, integrators that don’t work and nobody held accountable. They aren’t so much “no software” as “software that isn’t as evil as Oracle.” – not a big differentiator IMHO.
Given a choice I prefer to work in the environment we program in – Python/Django/Postgres/Ubuntu and hence we went with Django-Helpdesk to organize and provide transparency and accountability. Drop in Django-Model-Reports and I can actually see the level of support my clients are receiving. And while clearly my responsibility, results are results and they completely unacceptable.
The data and timelines were just hidden. No more. You can’t manage what you can’t see and I can see it now. I will need some help from clients to copy their support email account to keep from reverting to silos again.
SIMPLIFY SIMPLIFY SIMPLIFY
For the helpdesk and ongoing projects in the shop, with candor, it may take a month or longer to dig out and create a refined simplified simplified simplified system. But we’ve done it before and we’ll do it again.
Thank to our clients for your patience. I frustrated a lot of clients and lost some great clients and employees as a result of not having systems in place. I apologize.
IT’S ABOUT CUSTOMER SATISFACTION
Yet as I have said many times –
“Clients don’t want great customer service, they want great customer satisfaction. They want the software to work so they don’t have to call in and get a quick response. They want it to work so they don’t have to call or email in the first place.”
– we need better systems. This is one of them.
We will be announcing partnerships to fill in and take care of holes made apparent by the extreme transparency of the new helpdesk system.
DIFFERENTIATION
We will start with differentiation. Clients on paid support contracts should and will go straight to the top of the queue. Our old systems didn’t have the ability to filter and prioritize. Now we can. I will announce these partnerships, some with former Schipul/Tendenci employees who already are familiar with our clients and strict security systems.
We will do our best to forge ahead, rebuild the client contact and communication portions of Tendenci the company. Our technical team is still charging forward with long term goals rest assured. Like the return of the newsletter generator.
I thank you for your business, your contributions to the software, for being a part of the global tendenci team, and for not giving up. I won’t either. It’s not me or you, it’s us. #peace
Dev Server & Network Migration this Weekend November 7,8,9 2014.
All – we are migrating entire network including our dev servers and email relays this weekend, November 7,8,9 2014. That will be follow up by some structural reorganizations within the company as well to align our structure more closely to that of our evolving client base. Focusing externally instead of internally and improving prioritization.
You deserve details. I don’t have all of them yet. I committed to being more open about our communication and I’m sticking by that commitment even if it means I have to post something like “we are changing and stuff” because we are.
You’ll know more the minute I have made sure clients and employees are taken care of in the transitions to the best of my abilities.
#technologyandstuff
DNS Cobwebs – Tendenci.com and Schipul.com
DNS updates ahoy! If you see something unusual visit https://helpdesk.tendenci.com and submit a ticket or post on the forum. We love our clients. And we’ve been in business since 1997. For the backbone-crew of the company, you know, we’ve got a couple of years on us, and forgive us if we don’t remember every DNS entry for the last 17 years.
I am working to clean up years of cruft between the schipul and the tendenci aliases across all of our sites. It took me a while to figure out that the majority of sub-domain entries (like the “www” part of your domain) had long since been moved to production sites. So, forgive me, but I kinda went nuclear and just cleared out three or four hundred sets of entries from ten years ago.
HOWEVER, it is possible that you may have had a graphic or image using one of these old subdomains from 2005. Keep an eye out for that.
If this should cause a problem with your site it’s a great opportunity to clean things up. Which is why I did it on a Saturday night so it will replicate by Sunday and I’ll personally be monitoring the queue over the weekend (this is Ed typing.)
These changes literally go back to 2005 so my bet is nobody on your team will remember. But they CAN fix it with the template editor by fixing any old out of date links. You’ll score better in search engines as an added bonus and you don’t even need any help from us.
Or if you have the budget to prepay for mods, we can help. Of course we’d rather focus on building out the Tendenci platform. For free assistance visit https://community.tendenci.com but otherwise billable.
I thought I should give y’all a heads up about us clearing out the cobwebs from Halloween and encourage you to keep an eye out for anything that looks different. If you see it, please fix it. We can help, but that is billable obviously ( I wish my employees worked for free but they currently don’t. Nor do they donate money to the company. It’s a conundrum.)
Communications Needs a Town Square
Communication needs a town square. Step 1 was to implement a forum independent from our site. Encrypted (of course) and endorsed, and with it’s own unique karma.
Talk to us. Tell us what we can do better. Group listen – a challenge to all of us is to listen as a group. To truly listen.
And the software needs direction. Our job is to listen to and foster the community of Tendenci users across the globe while respecting that ultimately we, they, you, them, us, are all independent individuals and organizations, cultures and people, nonprofits and companies, we speak different languages, the only thing in common with the global tendenci community is our diversity. Just as the only constant is change.
We wouldn’t be cause related software developers or working at cause related companies that aspire to the #openeverything OSCON type of ethos to begin with.
First – it is to serve.
To achieve our communication goals we are using some amazing open source software based on django and postgres just like tendenci of course – The Misago Open Source Discussion Forums is used to power the new https://community.tendenci.com site to provide a “place” for that dialog to take happen. Developers are welcome to continue posting issues on github of course, clients who prefer to have us (or you!) manage their tendenci hosting will submit billable tendenci support requests, but there is something different about a forum that is toned down and not quite as public. I can’t quite explain why.
Join us?