Data Lakes for Smarter Associations

Open Source is Winning the Internet. Join us to be the BEST regardless of culture or country.

What is a “data lake”?

Data lake – it’s a big, partially structured and differently structured set of data from your association that you can use to answer interesting questions. Think of it like throwing everything in your pantry into one magical cauldron and it comes back with answers. Of course be careful what you ask, as SECURITY IS ALWAYS JOB 1! But it’s cool.

This is how AWS visualizes it:

Source: https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/

Having been personally involved in politics, analytics, electrical engineering, programming, robotics, public relations and media strategy not to mention the whole entrepreneurship thing, I must to start with a warning; do NOT go overboard.

Luckily Tendenci as your AMS makes this a whole lot easier with all of the ways to import and export data (or your whole database.) And pretty much every report has a structured export as well, from custom forms, donations, event registrations, memberships, you name it. You have FULL ACCESS to YOUR DATA in both flat files and in the actual database schema. For example:

I like this quote from wipro:

If the data in a lake is not well curated, then it may turn into a data swamp, flooding an organization with information.

https://www.wipro.com/en-US/analytics/five-best-practices-to-keep-your-data-lake-healthy/

You can see more screen shots of the built in reporting within Tendenci. Yet let’s be realistic, you are integrating data from many sources and a Data Lake including but not limited to your data on Tendenci might be just the thing. Contact us for more, because we LOVE DATA!

And we’ve got over 20+ years of experience to back it up.

Why not make 2020 the banner year to grow your membership and donations? Because that’s what we THRIVE ON! YOUR SUCCESS!

ctop – measure container cpu utilization like htop

Developers and programmers are frequently (ok, almost always) asked to accomplish the impossible yesterday. So this post is for the Tendenci developers and anyone else who uses docker containers, cgroups, jailed name spaces or similar.

Situation: You have a server that is spiking when it previously did not.

Let’s just assume you already have something like OSSEC and the ElasticSearch Stack  (ELK Stack) installed and are using a WAF/IDS/IPS endpoint. You are on top of your game. You see the errors from writing to the file system in dockers using the overlayfs file system (please no aufs, just don’t.) How to diagnose it:

“htop” is very good at showing you the issue. It (htop) is also frequently replaced by malware so double check yourself with “ctop” which most variants of common malware omit. Regardless, in this case, we can clearly see we have a stuck process. Enter “ctop” (open source like Tendenci at https://ctop.sh/ and on github at https://github.com/bcicen/ctop .

Running ctop you can quickly identify the container that is using the resources and then enter that container for further trouble shooting. “ctop” look like this:

The solution to a container over utilizing its resources is up to you and your developers. ctop is however a great way to zero in on at least which container is the problem.

In our case, a quick stop/start of the container removed the load and allowed us to do more debugging to figure out the cause. Tendenci is a mature and large codebase for association management (AMS Software) so it’s an iterative process to zero in on issues. And it can be done with the right tools.

Happy Container Utilization

This is what one of the Tendenci Cloud docker servers looked like after  debugging and killing the process causing the problem. “Yes” of course there is no replacement for “grep”. But with containers the debugging is a new art even for experienced programmers.

Hopefully this is helpful for all of the open source self-hosted Tendenci – the Open Source AMS self install developers using an AMS with 75+ languages out there.

And if you are a Python/Django developer – fork Tendenci open ams on github!

#peace

Amazon Email Relay Service Outage Resolved Feb 28, 2017

To our clients and end users hosted on the AWS (Amazon Web Services) Cloud – the email outage and partial S3 (storage) outage have been resolved per the Amazon status notification site https://status.aws.amazon.com/

AWS SES Status Update

Visit the AWS status blog at https://status.aws.amazon.com/ and the AWS subreddit at https://www.reddit.com/r/aws/ . AWS’s email outage is described in detail in the reddit post at https://www.reddit.com/r/aws/comments/5wphqj/amazon_s3_and_amazon_ses_are_down/

And as always please keep an eye on this blog for Tendenci updates.

Edit: for more coverage visit: http://www.cbsnews.com/news/amazon-web-services-cloud-outage-internet-crashes/