FRIDAY UPDATE FROM THE CEO
To the T4 (Microsoft legacy sites) clients who are still running on limited functionality or no functionality for a few sites still. There is frustration and anger and I hear you loud and clear. We continue to work around the clock and reach out to trusted resources to help us in the rebuild. It just isn’t easy to take a web site up from Windows 2003 to Windows 2012 and reconfigure everything by hand to try to be sure the code it clean. Still, we have learned a lot so that we will be more prepared in the future and I’m extra committed to the migration to the open source Linux version. But what about RIGHT NOW?!
First – data portals are being configured with the sites that have been fully offline going up first.
You will be notified through the helpdesk via tickets as soon as we have yours up. We may get a few up as soon as this weekend, and then the speed will pick up as we can clone it and modify the authentication information for each client. Thank you for using https://helpdesk.tendenci.com as it has been the only way I personally could jump in and help with tickets and track progress. I know the phone is more personal, but when the bullets are flying overhead it is efficiency we need, and I think we can all agree that it wasn’t efficient enough and things are still going too slow despite automation simply because of the volume.
There are a few other obvious items that we are still working through.
- Email notifications. With the changed IP addresses we are seeing some clients delivery rates drop significantly and need to update your DNS to send from an email address at your organization. This requires a site setting update on your site and your DNS provider to make DKIM and SPF record entries for email delivery. It’s tedious but has to be done. Spammers have made things complicated.
Workaround – the system does record most notices as they are sent for administrators at /en/emails/search.asp on your site.
- File uploads – the new web application firewall is much tighter than before, and I know we have had numerous requests to re-enable things like Word Docs and Excel files, but both of those document types support macros in vbscript and are executables. Until we can put them in a read-only bucket for now the only solution is to convert documents to eliminate all spaces and use lowercase and make them PDFs. Why? Because URL encoding can be used to trick people and spaces aren’t as secure.
- Creating new pages and image edits. – Again this requires writing to the file system and we need to isolate every site further before this can be turned back on.
- Broken images and missing files – not all, but most of those, had embedded code in the images. Unfortunately this also strongly suggests that for the clients experiencing this the most, there is probably a virus on your home or work network and we strongly encourage you scan and analyze your computers. You can use Trend Micro’s HouseCall for a free virus scan.
- SITES THAT ARE STILL DOWN – we have NOT forgotten about you. This remains my top priority for the team and is being done either by a different group of people (I’m leading the charge on the few sites still offline personally) or it takes precedence over the items listed above.
To our Tendenci 5 clients, and the sales contact forms, and clients used to a higher level of service who are feeling, and sometimes are, being ignored by our team. It’s not that we don’t care, it’s simply the result of clients who are victims of the hack attack and they have to be our priority.
And lastly, as difficult as this time has been for all of us, because it was a crime and crimes are not victimless, I appreciate the patience of some, I understand the anger and frustration of others, but please know that we will get through this. Even the clients who left, we’re still going to restore your data so you can get it.
I’m hugely grateful to our team for handling the front lines so the technical people like me could focus on solutions instead of discussing them, which ultimately is what everyone wants. This whole thing saddens me and I can’t apologize enough, while at the same time it infuriates me that it happened in the first place.