Non profits don’t have money to waste. Therefore we aligned our product to major industry supported technology.
Our technology stack as of 2018 is:
Django Web Framework
Python Programming Language
Postgres Database with GIS
For more on The Open Source AMS integration via API visit our AMS API Helpfile or read up on everything Tendenci Works With. Or if you aren’t into open source, there are definitely alternatives to Tendenci.
If you do pick an alternative, we suggest you consider Security FIRST and go from there.
Developers and programmers are frequently (ok, almost always) asked to accomplish the impossible yesterday. So this post is for the Tendenci developers and anyone else who uses docker containers, cgroups, jailed name spaces or similar.
Situation: You have a server that is spiking when it previously did not.
Let’s just assume you already have something like OSSEC and the ElasticSearch Stack (ELK Stack) installed and are using a WAF/IDS/IPS endpoint. You are on top of your game. You see the errors from writing to the file system in dockers using the overlayfs file system (please no aufs, just don’t.) How to diagnose it:
“htop” is very good at showing you the issue. It (htop) is also frequently replaced by malware so double check yourself with “ctop” which most variants of common malware omit. Regardless, in this case, we can clearly see we have a stuck process. Enter “ctop” (open source like Tendenci at https://ctop.sh/ and on github at https://github.com/bcicen/ctop .
Running ctop you can quickly identify the container that is using the resources and then enter that container for further trouble shooting. “ctop” look like this:
The solution to a container over utilizing its resources is up to you and your developers. ctop is however a great way to zero in on at least which container is the problem.
In our case, a quick stop/start of the container removed the load and allowed us to do more debugging to figure out the cause. Tendenci is a mature and large codebase for association management (AMS Software) so it’s an iterative process to zero in on issues. And it can be done with the right tools.
This is what one of the Tendenci Cloud docker servers looked like after debugging and killing the process causing the problem. “Yes” of course there is no replacement for “grep”. But with containers the debugging is a new art even for experienced programmers.
The AUFS file system, part of what gives us C-Groups, now called containers, now called Dockers, etc, but it is the onion-style file system that gives Dockers (we’re gonna just settle on calling them dockers) their magical powers.
This can lead to some very unexpected results, for example deleting a file in container “X” will appear to delete it. However let’s presume the previous base box “A” had the file and you want to make an new image and container from “A”. You might presume that file “abc” was deleted from all of the layers. But with AUFS that isn’t how works. You either keep layering up (meaning build your new site as a container from an image of the latest container you were working on.
This layering is a critically important concept to fully understand if you are working with dockers and the aufs file system. Rather than take my amateur explanation of it, I’ll refer you to the full docs on and let you go from there. Just *please* don’t overlook file system layers in AUFs when trouble shooting issues with containers.