ctop – measure container cpu utilization like htop

Developers and programmers are frequently (ok, almost always) asked to accomplish the impossible yesterday. So this post is for the Tendenci developers and anyone else who uses docker containers, cgroups, jailed name spaces or similar.

Situation: You have a server that is spiking when it previously did not.

Let’s just assume you already have something like OSSEC and the ElasticSearch Stack  (ELK Stack) installed and are using a WAF/IDS/IPS endpoint. You are on top of your game. You see the errors from writing to the file system in dockers using the overlayfs file system (please no aufs, just don’t.) How to diagnose it:

“htop” is very good at showing you the issue. It (htop) is also frequently replaced by malware so double check yourself with “ctop” which most variants of common malware omit. Regardless, in this case, we can clearly see we have a stuck process. Enter “ctop” (open source like Tendenci at https://ctop.sh/ and on github at https://github.com/bcicen/ctop .

Running ctop you can quickly identify the container that is using the resources and then enter that container for further trouble shooting. “ctop” look like this:

The solution to a container over utilizing its resources is up to you and your developers. ctop is however a great way to zero in on at least which container is the problem.

In our case, a quick stop/start of the container removed the load and allowed us to do more debugging to figure out the cause. Tendenci is a mature and large codebase for association management (AMS Software) so it’s an iterative process to zero in on issues. And it can be done with the right tools.

Happy Container Utilization

This is what one of the Tendenci Cloud docker servers looked like after  debugging and killing the process causing the problem. “Yes” of course there is no replacement for “grep”. But with containers the debugging is a new art even for experienced programmers.

Hopefully this is helpful for all of the open source self-hosted Tendenci – the Open Source AMS self install developers using an AMS with 75+ languages out there.

And if you are a Python/Django developer – fork Tendenci open ams on github!

#peace

knowledge of how to combine is the mother of all other forms of knowledge

If you run an Association, volunteer, join in, help, learn and participate – well – at Tendenci we think you are kind of a BIG DEAL! It’s easy at times to lose sight of the bigger picture when you are on the board of directors and planning the details for a fundraiser.  Please remember – we need you, we appreciate you, and YOUR CAUSE IS WORTH IT.

Alexis de ToquevilleI get asked why Tendenci is Open Source. My reply is to point to the role of associations in society.  The role of associations, your association management system as well, are both too important to survive the conflict of interest with purely commercial solutions. To clarify why this is so important to me, and I believe you, I can only quote those far more educated and eloquent then myself.

Alexis du Tocqueville viewed civil society as the third leg of the stool that allows democracies to function.

Americans of all ages, all stations of life, and all types of disposition are forever forming associations… In democratic countries knowledge of how to combine is the mother of all other forms of knowledge; on its progress depends that of all the others.

and further

Americans combine to give fêtes, found seminaries, build churches, distribute books, and send missionaries to the antipodes. Hospitals, prisons, and schools take shape in that way. Finally, if they want to proclaim a truth or propagate some feeling by the encouragement of a great example, they form an association.

In every case, at the head of any new undertaking, where in France you would find the government or in England some territorial magnate, in the United States you are sure to find an association…. I have often admired the extreme skill they show in proposing a common object for the exertions of very many and in inducing them voluntarily to pursue it.

Alexis du Tocqueville, Democracy in America (source)

Back in 1995 Senator Bill Bradley wrote “Democracy’s Third Leg.” and he described it in a similar manner.

CIVIL society is the place where Americans make their home, sustain their marriages, raise their families, hang out with their friends, meet their neighbors, educate their children, worship their God.

It lies apart from the realms of the market and the government, and possesses a different ethic.

and

Civil society, on the other hand, is the sphere of our most basic humanity — the personal, everyday realm that is governed by values such as responsibility, trust, fraternity, solidarity, and love.

…. There must also be a healthy, robust civic sector — a space in which the bonds of community can flourish. Government and the market are similar to two legs on a three-legged stool. Without the third leg of civil society, the stool is not stable and cannot provide support for a vital America.

Maya Angelou wrote one of my favorite poems which I believe relates. It is “A Brave and Startling Truth.”

Maya Angelou is of course a giant not just of our time, but of all time. She speaks of greatness in the form of unity and love. That is what Civil Society does. Associations, churches, clubs, political movements … all of these things are simply too important to our planet to NOT be open source. And we will come to it. YOU and your AMS software are too important to be locked in or cut off if a proprietary vendor chooses.

Quoting Senator Bill Bradley’s piece again, he states:

The language of the marketplace says, ”Get as much as you can for yourself.” The language of government says, ”Legislate for others what is good for them.” But the language of community, family, and citizenship at its core is about receiving undeserved gifts.

Building the Tendenci AMS community Open Source – giving you control – is how I handle the brutal truth that “we must confess that we are the possible. we are the miraculous.”

#peace