DoOO V2 and rhfw
Don’t let the fact that I have a computer science degree or know (barely) a few languages fool you. I was never great at the lower-level or more theoretical parts of CompSci necessary for writing anything beyond scripts that manipulate text and store data (mostly text) in dictionaries and arrays and dictionaries of arrays (and sometimes arrays of dictionaries), which means that I am not very good at developing real software.
What even is a pumping lemma?
Hell, in my Software Engineering class at Uni I was mostly responsible for setting up and maintaining the environment the rest of my team actually did the development work in.
I did do alright in my Systems Administration class after all.
Still, that’s never stopped me from trying to make real projects, like my text adventure game engine. Than again, all that’s really doing is text manipulation and storing data in multi-dimensional arrays and dictionaries.
Is there really that much more to programming?
Anyways, I actually wrote some real software. Two softwares in fact.
Domain of One’s Own - Version 2 (DoOO V2)
Just as “Newsboyz” was the flagship show (gfydl), “Domain of One’s Own” is Reclaim Hosting’s flagship product. Its basically a white-label shared hosting product for universities that they can use to provide their students, faculty, and staff with web hosting. For the longest time (before I worked at Reclaim and up until fairly recently) it used WordPress as the frontend, WHMCS as the middleware, and WHM/cPanel as the backend. It was complex, confusing, tedious to set up, and a major source of/contributor to our tech debt. And while vague ideas on how to fix it floated around for years, very little progress was made since – for all of its flaws – the WP/WHMCS-based DoOO worked well enough. And if something mostly works it’s often better to simply leave it be.
An idea that had some steam a while ago for a fix was the creation of a unified API to tie together everything in the DoOO stack (and beyond). From WHMCS to cPanel and even Reclaim Cloud (our instance of the Virtuozzo Application Platform). All of it seamlessly integrated in a way that a user would have no idea they were all separate systems. But that approach fizzed out due to a unified API being just a behemoth of a project. So we kept deploying DoOO instances the way they were: WordPress on WHMCS on cPanel.
One of the first projects (read: things I did because I didn’t like how we were doing them before and now I was in a position where I could unilaterally make the decision to change them) I undertook as Reclaim’s SysAdmin was automating some of the deployment of DoOO’s WP/WHMCS/cPanel jenga tower stack to help standardize the setup a bit and generally make things less tedious. However, that script was clunky and I spent as much time tweaking the script to work for the next DoOO setup it as I did actually setting up DoOO instances. I thought I had a breakthrough about two or three years ago when I realized I could use a backup of a WordPress install with all the themes and plugins and customizations as a “template”, but reality hit me rather quickly when I remembered that WordPress was only one part of DoOO. There was still WHMCS and all of the other components to think about. So I left the script and the template stuff in a half-broken mess that required even greater manual intervention when deploying a DoOO instance than was previously required.
I repeatedly thought about just taking all of the essentially functionality of DoOO and putting it into a single WordPress plugin. It was far smaller in scope than the unified API was supposed to be, so it seemed far easier, but I still knew next to nothing about WP plugin development. So I left it alone. Something that maybe I’d get to in the vague future. But then in early 2025 I had the same feeling that I had when I decided to start leaning Italian a couple of years ago. Or when I decided to finally go vegan several years ago (in order to remain vegan I am obligated to tell everyone that I am vegan whenever possible). Or when I decided to start playing bass over a decade and a half ago. Something inside my head clicked and I learned again (for the millionth time) that I can’t start doing something unless I start doing something. Why waste time talking about starting if I can just start today?
So in February 2025 I started. I read some WordPress plugin development docs and bashed out a PHP-like psudocode skeleton.
And by April 2025, I finally let everyone know what I was doing after I had something that looked like it’d probably work if it was rewritten in actual PHP. Rewriting it would be a task, but now this was in motion as something of a real project I could actually expend the effort to rewrite it. After a few (more than a couple but less than several) weeks I finally had something that WordPress could load as an actual plugin and talk to WHM/cPanel without halting and catching fire (most of the time). There were several days where the only windows I had up on my work machine were my IDE, a terminal full of git commands (and sometimes vim for quick changes to small files), and browser tabs upon browser tabs of WordPress and PHP docs. But by the end of May, I had something functional with almost everything we used WHMCS for in a DoOO instance fully re-implemented inside a WordPress plugin.
The period between the end of the spring semester and the start of the fall semester (which I think normal people call summer) is a relatively quiet time for Reclaim, and that quiet gave me the chance to do quite a bit more work on the growing codebase of DoOO V2. Older code was cleaned up, new features were added, and I removed a bunch of //TODOs from all over the files.
Given I was writing a plugin for a web application that can quickly become an open sore on the internet if a poorly written plugin is installed, security was never far from the front of my mind. But no matter how much I ensure that all user input is filtered and validated or how many checks on who is calling a function and from where that I write, I admit only know so much when it comes to cybersecurity (no matter how much posturing on topics like digital privacy I engage in). So, in order to get an actual professional involved in the security-of-it-all, I pulled Noah (who is currently a heretic but at the time was Reclaim’s Security Admin) into my insanity and had him look over the plugin’s code with me for anything obvious that would get us pwned the second it was enabled on a real, publicly accessible, WordPress site. After one or two manual checks where we went through everything by hand, line-by-line, he introduced me to PHPStan, which not only sped up the auditing process but saved him from a fate worse than death: reading any more of the PHP that I wrote.
At least I didn’t write any of this in Perl and make him read that.
By late summer/early fall 2025, DoOO V2 was both feature complete and well audited (internally). But it looked awful; my frontend web design skills have not progressed far beyond the dozen-or-so lines of HTML they had us write during a “computer literacy” lesson in elementary school. I attempted to polish things up with some CSS, and while I was able to get the buttons looking nice (using WordPress’ stylesheet, not my own hand-written code), we still had one major issue: the hell of vertical infinity. Accounts and servers and settings and just about everything else that the plugin pulls from the database are printed out to HTML tables, but all of the requested data is returned. Not a sub-set. All of it. On our development server this wasn’t a big deal since we only had a handful of accounts, but if we were to put this on a real DoOO instance with hundreds or thousands of accounts, we’d at the very least need some kind of pagination to prevent the list from being unmanageable. And I had no idea how to implement that in a clean way that wouldn’t mean total breakage or total rewrite-age. But then Taylor brought salvation in the form of the Datatables javascript library, which allowed us to do client-side pagination (and added a bunch of additional features like search, CSV download, and so on).
After Datatables made the whole thing finally look presentable we had the plugin’s code reviewed by third parties, human and machine (anyone who has interacted with me in any capacity should know that I have no love for “AI” but using an LLM to parse a large amount of human-written text – such as code – and identify patterns – such as bugs or other potential issues – is something that I can stomach), and spun up few live betas to get outside input on how we put everything together. Given how small Reclaim Hosting is, almost everyone was at least involved in the early testing of DoOO, so these things were essential to find and fix bugs we would have never found (like how we were escaping PHP-generated Javascript output with esc_html() instead of esc_js() like we should have been and are doing now) and implement features we would have never considered (like caching queries in WP Transients to really speed things up). And while we’ve fixed plenty of bugs/made plenty of improvements/dedicated quite a bit of time to its development, I have no delusions that everything is resolved or that this is the most perfect WordPress plugin ever written. We’ll always be finding things to fix or make better. That’s just how software development works; it’s a continuous process. And I’m excited for this process to continue to make DoOO safer, simpler, and more stable.
rh-firewall (rhfw)
The standardization of Reclaim Hosting’s security stack has been a whirlwind of evaluating and re-evaluating different firewalls and AV solutions and other components. It has been so hectic and confusing that every attempt I made to write in in any kind of linear order that made sense just left me more confused. And I was there for the whole thing! So I’ll jump to the end: I wrote a simple IP-blocking firewall called rh-firewall (or rhfw for short), and we open sourced it under the BSD 3-Clause license since other folks may find some use in it.
Networking has never been my strength, and everyone else on the team seems to be scared of it as well. But writing this tool forced me to dive in to the basics of iptables, nftables, and ipset. Not deep enough to where I’m comfortable setting up routing rules or anything mind you (I still break networking on my homelab quite often), but enough to be confident that this piece of software does what it says on the label: blocks IPs. It’s nothing fancy though. Just some ugly bash scripts that make it easier for someone like me who has no idea what they’re doing (I am made of stupid) to work with real firewall software without breaking networking and taking down several production servers.
“The stupider it looks, the more important it probably is.”
– J.R. “Bob” Dobbs
The catalyst for us finally pushing this out into the open was the discontinuation of CSF, which we were using pretty heavily for both its functionality as a real firewall frontend that blocks/limits access to ports and it’s ability to block IPs on demand. Since rhfw only blocks IPs, it’s still only been a partial replacement; real firewall frontends like ufw or firewalld (or even something like DigitalOcean’s Cloud Firewall) are still necessary to handle most real firewall tasks. A hammer shouldn’t be the only tool in your toolbox, but you should still have one for when there are nails. And believe me, sometimes, there is nails.
Ideally, once I become more comfortable with the language, I’d like to come back and rewrite rhfw in Perl. Why Perl? Because I’m a masochist it’d allow me to handle certain tasks more gracefully than I can in bash while maintaining a more-or-less equivalent level of portability (show me a Linux web server that doesn’t have Perl installed). But maybe I’ll change my mind down the line and end up rewriting it in Ruby or Python in order to protect my sanity. Or I’ll finally snap and rewrite it in PHP like a madman. Only time will tell.