EN Tekst
Why is zjk.pl the way it is? — 30 answers after 25 years of work. An "extraordinary" FAQ :)
1. Why do I even maintain my own servers?
Because
I needed, and still need, to have a working server. Not some testing
environment, not a home laboratory – no homelab. Of course, it
can occasionally serve that purpose too, but only as long as it doesn't
interfere with its primary function – which is acting as a
production server.
Let's
remember – 25 years ago, external hosting options were scarce.
But I wasn't interested in them anyway, because they imposed their own
parameters and pre-defined functionalities. For someone with a strong
sense of freedom and autonomy, that was unacceptable. Today, even
though they are easily accessible – they remain completely
unattractive to me.
2. Is it worth it financially and time-wise?
- Financially:
Yes. The costs are just electricity, maintaining the servers' health,
and rare hardware upgrades. It comes out cheaper than renting a
dedicated server.
- Time-wise:
Of course not. It requires a lot of systematic work, and sometimes an
immediate response for repairs and servicing at the least convenient
moment.
- Satisfaction?
I do it for myself and for the websites I support – the people
who use them. Although rarely anyone remembers the caretaker of the
entire system.
- Space:
It takes up very little room – just a larger desk. If it weren't
for the massive UPS and its batteries, it probably wouldn't take more
than 0.5 m².
From
the very beginning, it was a conscious choice: FreeBSD. Since this was
meant to be a production system, the choices on the market were very
limited. Of course, OpenBSD could have been considered as well. Linux
was never even an option. FreeBSD means stability, predictability,
great documentation support, and the completeness of the entire
operating system.
4. Why not virtualization and one large server?
The
scale of services and running programs is so large that a single
physical computer wouldn't be able to handle them safely. It could have
any number of cores, but memory, disks, and peripherals will always
remain critical. Above all, however – it creates a huge
vulnerability to failure, a classicSPOF (Single Point of Failure).
5. Why MooseFS and SeaweedFS?
There
was a time when I had several disks with NFS or Samba for my file
servers. And that setup worked quite well. However, each disk became an
individual SPOF – requiring separate backups and additional
archives. MFS simplified this radically. Of course, it still requires a
backup and an archive, but it becomes a single point of management. And
sharing and mounting common resources is just so convenient! :) No more
file synchronization issues between multiple web servers.
We
have to understand the core philosophy behind its creation. It's not
just a zettabyte file system, but a complete ecosystem of tools for
managing data stored with it. It’s a completely different level
of understanding and operating the entire IT architecture. From a
simple snapshot to disk encryption, the whole thing elevates resource
management to another dimension. But don't believe the myth that it is
100% fail-proof ;)
Placing
all connectivity security on an internal gateway computer, while
possible, is difficult and risky. Moving these services to a dedicated
appliance brought me great relief. It acts as an excellent isolation
layer, a true "firewall." The system is specialized and secure, built
on FreeBSD. I equally recommend its near-twin, pfSense. Additionally,
clustering two machines in Master-Slave mode using CARP provides
incredible convenience – the internet is always up, even when I'm
performing maintenance or upgrading the main router.
You
could view this as an unnecessary luxury that we might never fully
utilize. However, extra Mbps always come in handy. Furthermore, two IP
addresses from different subnets provide great security for web access,
and especially for mail (dual mail servers and independent MX records).
Finally – the plan is to take over external DNS hosting, which
strictly requires two independent IPs from different subnets.
9. Haven't you gotten bored of it after 25 years?
Not
in the slightest. Which doesn't mean I dedicate myself to it entirely
– quite the contrary. My daily duty is a quick check on the
server status, but I don't sit and work on them non-stop. I allow
myself longer periods of rest, only to return with fresh energy and new
ideas.
I
don't use it, which doesn't mean I don't know it. For a while, a
single, spare machine even ran it (or still runs it if needed –
right now it's a Raspberry Pi), but I turn to it very reluctantly.
Early attempts at upgrading Linux almost always ended in system
failure, and trying to "undo" those changes caused an even bigger mess.
And configuration? Every release seems to bring a different concept.
When entering Linux, I feel a sense of chaos (perhaps my fault, due to
a lack of deep knowledge of the system) and I return to FreeBSD with a
peace of mind. Not just because I know it, but because I know it won't
let me down.
11. Why don't I use the cloud?
The
cloud is no longer "mine." It's "somewhere out there," and I don't even
know exactly where. What's more, the cloud provider can restrict my
access to it, or share my data with someone else. I have no ultimate
control over who has access to it and under what conditions – or
simply no guarantee that its confidentiality will be protected. I never
entrust anything important to the cloud. Although I'm sure an exception
can be found, as it relates to our smartphones – nowadays, it's
hard to completely defend yourself against the cloud.
In
the massive mess of Linux libraries, Kubernetes simply had to be born
– and that is its great advantage in that ecosystem. On the other
hand, FreeBSD has Nomad withpot, though one could also mention CBSD – both solutions rely on native FreeBSD containers (jails
). This is a highly interesting way to manage applications across
multiple computers, and I don't rule out using it here. Nomad generally
has solid FreeBSD support. Unfortunately, I'm afraid that the overhead
of such heavy layers on my very lightweight servers wouldn't bring much
benefit, but rather cause unnecessary problems.
In the FreeBSD world, there is a project called Sylve utilizing bhyve
virtual machines – I will definitely try it out someday. A few
years ago, I planned to introduce CBSD and possibly bhyve; today, the
more interesting Sylve and the aforementioned Nomad are available.
Sylve will most likely be first. Long-term testing will decide how
useful it proves to be within the zjk.pl infrastructure. I don't reject
virtualization as a concept – I am simply more interested in
solutions that are fully native to my system.
14. Why so many small servers instead of one large one?
This is the ultimate minimization of SPOF
(Single Point of Failure). You can check how this looks in detail under
the "tests" section of the website. In any case, losing one small
server practically does not affect the continuity of the entire server
room. So, it works similarly to advanced clusters, but implemented
simply – at the physical hardware level.
15. Why host my own email?
Again,
it’s a matter of independence, though it is a very difficult and
demanding battle. Today’s giant corporate email systems have a
tendency to baselessly reject messages coming from small, private
server rooms. For it to work, you have to prioritize quality,
authentication, and security in the highest possible standard.
It’s a tough challenge, but also a source of great satisfaction
when everything runs smoothly.
16. Why still stick to classic HTML?
Web
development technologies are probably the most rapidly changing trend
in the world – it's impossible to count all the emerging
frameworks and layers designed to create a "modern" look. There's
nothing wrong with that, but I simply don't want to waste time chasing
these temporary trends. HTML is simple, stable, and incredibly fast. My
main goal is not visual gimmicks (although I do pay plenty of attention
to aesthetics), but content. It might go a bit against the grain of the
modern internet, but in my opinion, it is the substantive content that
builds a website's true value.
17. Why is documentation so important?
With
a system this complex and distributed, I would have lost my way long
ago without proper documentation. Graphical windows and configuration
panels won't help much here – critical matters must be precisely
written down. After a decade or two, the human mind cannot remember
every design decision, configuration quirk, and service dependency.
Thus, documentation becomes the external memory of the entire system.
18. Why don't I call it a homelab?
Because
it is a fully functional, production system. It provides diverse,
real-world resources with absolute priority given to continuous,
stable, and reliable operation over many years. As long as it doesn't
conflict with this fundamental requirement, there is, of course, a safe
space for research, testing, and experimentation.
19. Why does everything still run from home?
Because
you really don't need much to run your own server room these days, and
ongoing hardware miniaturization only makes it easier. While the
general consensus is that professional services require massive
commercial data centers (colocation), modern, energy-efficient hardware
allows for incredible performance and stability right at home –
without paying exorbitant subscription fees to external companies.
Although a portion of users will move to the "cloud," there might be a
growing number of people who simply don't care for it – and
zjk.pl stands as an answer to the question of how to have something
like this on your own terms.
20. What will happen in the next 25 years?
I
don't know. When I started back in 2002, I never imagined the system
would still be running a quarter of a century later. If health, time,
and technology permit, perhaps in another 25 years, someone will be
reading the answer to this very same question on servers that succeed
the current zjk.pl. There's a chance I'll still be alive then, but you
can easily verify that by visiting zjk.pl – if the website is
still up, then I'm probably still alive too ;)
21. How many times did it all break down?
The
"vacuuming day" described on this website causes the server room to be
shut down for almost a whole day every year. But actual major outages?
There have been only a few. I don't remember any of them lasting for
more than two days (to be honest, I don't even remember one lasting a
full day) – but everything is still ahead of me ;) In any case, a
partial degradation of functionality might have lasted longer than a
day (because a replacement disk or power supply had to arrive by mail),
but as a whole, the server room has a lot of built-in redundancy and
handles various unforeseen situations just fine.
22. What does a typical day for the administrator look like?
Am
I constantly sitting at the console? No – though when I'm
deploying a specific project, it does happen. Normally, however, it
takes just a few minutes: checking the logs and looking over service
emails. The server room usually runs completely on its own, defends
itself against external threats from the web, and is capable of
automatically compensating for any losses caused by hardware failures.
23. How much electricity does it consume?
Around
350-400 VA, reaching 450 VA under heavier loads. This translates to
roughly 300 PLN per month. It's not a negligible amount, but a
reasonable hardware configuration policy yields significant savings
here.
24. How much data does the server room store?
Terabytes,
but not hundreds of them – rather around 30-40 TB in total. That
amounts to roughly a few million files. The website itself (depending
on how you count it) takes up about 30-80 GB and consists of 400,000 to
850,000 files.
25. What was the hardest part?
Securing high-quality internet links to the outside world. To this day, that remains probably the most challenging hurdle.
26. Which decisions would I make differently today?
There
isn't a single, pivotal decision. Changes are introduced gradually and
thoughtfully; if a modification yields unsatisfactory results, you just
have to back out of it quickly, and that's it. That is why today I have
no reason to regret anything. The server room simply must work, so you
must follow the path that best sustains its core functionality.
27. Which solutions turned out to be a dead end?
All
the large, bloated, and heavy applications and systems. Anything that
cannot be deployed easily, boots up slowly, and runs with a delay will
always generate trouble at some stage. I won't name anything specific
though – let everyone experience it for themselves.
28. What has been working continuously to this day, despite 25 years passing?
FreeBSD, DNS, SMTP, HTML, Apache – they have outlived dozens of tech trends and remain completely reliable.
29. Which elements changed most frequently?
The content of the websites themselves, and certificate renewals.
30. What are the biggest myths about running your own server room?
- Myth 1: You need a separate, dedicated room.
- Myth 2: You need a fortune to get started.
- Myth 3: You need a 10 Gb/s symmetrical connection.
- Myth 4: Everything must be virtualized.
- Myth 5: It's impossible to do without the cloud.
31. What does having your own server truly give you?
I
have come to the conclusion that the ultimate answer is not: "I have my
own server," but rather: "I don't have to ask anyone for permission":
- For my own email,
- For my own DNS,
- For my own WWW,
- For my own data,
- For my own experiments.
Tekst EN
