Developers and programmers are frequently (ok, almost always) asked to accomplish the impossible yesterday. So this post is for the Tendenci developers and anyone else who uses docker containers, cgroups, jailed name spaces or similar.
Situation: You have a server that is spiking when it previously did not.
Let’s just assume you already have something like OSSEC and the ElasticSearch Stack (ELK Stack) installed and are using a WAF/IDS/IPS endpoint. You are on top of your game. You see the errors from writing to the file system in dockers using the overlayfs file system (please no aufs, just don’t.) How to diagnose it:
“htop” is very good at showing you the issue. It (htop) is also frequently replaced by malware so double check yourself with “ctop” which most variants of common malware omit. Regardless, in this case, we can clearly see we have a stuck process. Enter “ctop” (open source like Tendenci at https://ctop.sh/ and on github at https://github.com/bcicen/ctop .
Running ctop you can quickly identify the container that is using the resources and then enter that container for further trouble shooting. “ctop” look like this:
The solution to a container over utilizing its resources is up to you and your developers. ctop is however a great way to zero in on at least which container is the problem.
In our case, a quick stop/start of the container removed the load and allowed us to do more debugging to figure out the cause. Tendenci is a mature and large codebase for association management (AMS Software) so it’s an iterative process to zero in on issues. And it can be done with the right tools.
This is what one of the Tendenci Cloud docker servers looked like after debugging and killing the process causing the problem. “Yes” of course there is no replacement for “grep”. But with containers the debugging is a new art even for experienced programmers.
The following graphs show what a Distributed Denial of Service (DDoS) attack on an association looks like. The names, rates and volume of the association have been blurred for security reasons. We are thankful to AWS for their own defenses in front of ours, which help us mitigate these issues.
Note: The graphic above, is filtered for a 24 hour span for one client. The infrastructure is in place, and highly redundant, so we can monitor and keep our clients safe. For clients in the US or hosted in other countries (we have multiple Tendenci clouds as needed.)
Note 2: Make no mistake – If a bad-actor has the budget – they can and will purchase enough bots to take a site down. This is well documented. Even our resources at AWS are limited in what they can handle. Budget (yes BUDGET) accordingly.
The team at Tendenci has been working to make improvements to our T7 software. The batch of updates noted in this release are mostly centered on: the Tendenci Nav, Reports, and Newsletters. You can look for these changes on your T7 site.
New top menu (for both admin and logged in users)
Newsletters format update and clone feature
Reports format update (including invoices, memberships, ..)
Events views – Added sub menu for month view, week view, day view
Separated join approval and renewal approval for membership notices
Wysiwyg editor – Enabled the image title input field in the image dialog. Added class dropdown to the tinymce link dialog box
Updated the directories categories to make it easy manage
Added drag-drop functionality to the testimonials
Added memberships overview report
Added a link on Profile page to view past events.
More minor changes
(Security) Disabled GZipMiddleware to prevent BREACH attacks
(Security) Prevent fraudulent simultaneous reuse of PayPal transactions
Resolved the issue regarding manage.py hangs when caching is enabled. Re-enabled the cache for site settings.
Resolved the subprocess venv issue.
Fixed exports for directories, jobs, resumes, pages.
Fixed “Most Viewed Files” report.
The Tendenci Nav has been reorganized to help you find what you’re looking for. We hope you’ll take the time that review your new Tendenci Nav and get familiar with some of these changes. Here’s a screenshot of what it should look like:
*For all of these updates, some sites will display slightly different fonts and design features based on the site theme.
Updates in the Reports module includes consolidation of all reports into their own Tendenci Nav item.
This release also includes an update to the Invoices module, making invoices faster to browse visually in a table-based format. To view invoices in the new navigation on your site, click: Reports > 5. Financial > Invoicing.
The Newsletters module works almost exactly the same as before. We have made a few features a little bit easier to use. For example, if you’d like to re-assign the newsletter to a different group, you can now do so even after you’ve generated the Newsletter in the first step. We’ve also made some layout improvements that take advantage of Bootstrap’s framework.
The “clone” feature has also been added to this module. With clone, you can send a test e-mail, then clone it and send it to your target audience rather than copying and pasting the contents from the first send.
To our clients. The above graph is a filtered subset of what is a *typical* day of network alerts. As the media has stated, the issue is quite real.
We greatly appreciate you and it is important to us that you remain safe. To further advance that objective in the current geopolitical environment, all hosted Tendenci sites will be encrypted going forward per our CEO.
Why? Because security. The Internet has changed and we must adapt.
Adapt? Remember when that Steve Jobs guy invented the iPhone and suddenly sites that were awesome the week before… well… they weren’t as awesome the next day? The. Next. Day. Technology is like that.
(This is a cross post from our CEOs personal blog. Note that Tendenci sites do NOT use Apache and the vulnerabilities in Equifax’s implementation of Apache Struts do NOT impact your Tendenci site. Still be aware that nothing is is 100% secure so stay vigilant and be prepared friends!)
From the second article on the Equifax breach linked above, this portion really galls me:
… not only are none of the last names tied to your Social Security number, but there’s no way to tell if you were really impacted.
It’s clear Equifax’s goal isn’t to protect the consumer or bring them vital information. It’s to get you to sign up for its revenue-generating product TrustID.
Earlier it was revealed executives had sold stock in the company before going public with the leak. We also found TrustID’s Terms of Service to be disturbing. The wording is such that anyone signing up for the product is barred from suing the company after.
The following phrase alone, if true, combined with Equifax literally trying to monetize their security errors, is what gives capitalism a bad name:
The wording is such that anyone signing up for the product is barred from suing the company after.
I have to believe the Equifax PR team is working for PharmaBro or Putin trying to make them look good in comparison.
Note: Equifax has changed the indemnification, but only under duress imho. Furthermore 30 days free credit monitoring by the company that released your data and then you will have to pay monthly still seems wrong. But to be fair, here is their update:
(Editor: well ya, duh!?)
(Editor: but did you fire the person who did it in the first place?)
I get it. Nothing is secure. If the NSAs hacking tools get stolen and OPM loses all of the data on security clearance checks on our own people, then truly nothing is safe. I get it.
What I do not understand is a company as large as Equifax not being prepared for something like this. That Equifax did not announce it promptly. That Equifax executives sold stock before announcing it. That Equifax then attempted to indemnify themselves. That Equifax is using the crisis to sell a monitoring service that you have to pay for after 30 days. A service to monitor YOUR data that THEY lost control of!
This boggles the mind of a PR Professional.
The Internet was not built for e-commerce – it was built for knowledge sharing in a “walled garden”. Therefore keeping sites secure is not possible. Any security professional will tell you best practice is to white-list good guys (selective inclusion) as opposed to trying to find every attack and block it. Therefore the difficulty at a high level is primarily in identifying and blocking bad actors.
I hate to say it folks, but we are playing whack-a-mole with your identity and money. It will always be an uphill battle to maintain security on the Internet and you will never ever be 100% safe.
Python, the language used to program Tendenci – The Open Source AMS, continues it’s meteoric rise in the world of developers. And where the developers go is where the rest of us go. Thus Python’s rise matters. And it benefits every Tendenci user, self hosted or hosted with our small company (same software either way).
They have numerous charts to back up the data, but these two in particular paint a telling picture.
From Stack Overflow – the current tag questions viewed:
The above graphs should give you confidence in your choice of using Tendenci as your AMS as the developers are not only there, but growing. Given Tendenci is fully open source (this is different from “free trial” AMS systems which are NOT actually FOSS (Free and Open Source Software). Wikipedia describes the difference as:
(FOSS means) anyone is freely licensed to use, copy, study, and change the software in any way, and the source code is openly shared so that people are encouraged to voluntarily improve the design of the software. This is in contrast to proprietary software, where the software is under restrictive copyright and the source code is usually hidden from the users.
The good news is with the growth of Python, it only make sense that developers will look at and many will join in to help the community improve the software as they join associations themselves.
We’ve written about why we chose Python over PHP to develop Tendenci open source several times. Correctly choosing the open source stack gives us, and everyone in the community, confidence to see the trends predicted correctly. It wasn’t rocket science – we just listened to our team, we listened to younger developers, and most importantly we listened to our clients on what the future was/is going to be.
Why are associations unwilling to accept apps that meet only minimal requirements? Um… because they started as Guilds and go back to Medieval times. From Britannica on Guilds and Trade Associations:
Guild, also spelled gild , an association of craftsmen or merchants formed for mutual aid and protection and for the furtherance of their professional interests. Guilds flourished in Europe between the 11th and 16th centuries and formed an important part of the economic and social fabric in that era.
and Britannica goes on….
… associations are known to have existed in ancient Rome, however, where they were called collegia. These craft guilds seem to have emerged in the later years of the Roman Republic. They were sanctioned by the central government and were subject to the authority of the magistrates.
This is a huge topic of course. Just know that Tendenci is the ONLY top ranked AMS system that is truly FOSS. Unlimited admins, users, contacts – you can self host or if hosted with us we only charge for processing power. Got 1M users and contacts and 50 admins? No problem. And the growth of Python assures your continued freedom from vendor lock-in no matter what.
Hurricane Harvey Update from Tendenci – Monday Sept 4, 2017
A large part of our small team lives in or near Houston TX where Hurricane Harvey hit us with more rain than any storm in US history.
For those near us, we all know it’s not over yet. The roads are still flooded. Power is going on and off. And we are the *lucky* ones as so many lost their homes and over 60 people lost their lives. It’s important to keep in perspective.
We have been pushing updates *as urgently needed only* on twitter and facebook regarding Hurricane Harvey. The quiet messaging is what I was taught in Crisis Communications – don’t muddy the waters.
Tendenci’s servers and our diverse multinational team maintained and all is well. I do have some minor requests from us to the Tendenci community:
The current policy of prioritizing client requests by the impact in the disaster area remains in place. Please be patient if you have a normal request that doesn’t get addressed as quickly as usual. I expect this to last another week. However don’t forget tendenci is open source and another developer can use the template interface to pretty much fix anything you need if we aren’t available immediately.
Position 1 also obviously applies to the tragedy unfolding in India, Nepal, and Pakistan as well, although I’m not sure if we have any clients impacted in those areas. I do know Tendenci has numerous open source users in Nepal and our thoughts and support are with you.
If a particular hosted client is in a difficult situation regarding this disaster please contact us.
For those of you not impacted by Hurricane Harvey – please consider a donation to the charity of your choice that you believe will benefit those suffering the most.
For more information on Hurricane Harvey in Houston please check: