Black Raven Dragoons

[Zancarius]

The last few evenings may have presented you with a few hiccups. ZFS on Linux (ZoL) doesn't readily release, in my experience at least, memory allocated to the ARC (Adaptive Replacement Cache--think how ext4 and similar use the kernel's page cache to cache commonly accessed files only on steroids). So, the server was eating up all but the last 1GiB of RAM (default as per ZFS) and began swapping a small sum of sleeping processes to disk. This isn't supposed to happen according to the documentation I've read: ZFS is supposed to surrender its cache when it detects increasing memory pressure from active applications. Perhaps it does (for some definition of "active"), but I've yet to see it in action.

As a solution, I've temporarily reduced the ARC's max to 1GiB so I have a bit more room to play with one or more Minecraft instances. This isn't ideal, and not even really recommended (2GiB is probably the absolute minimum for this workload, and that's pushing it) since this operates as a file server that I want to keep within a certain power consumption limit. I need to explore some additional options, but I'm both out of time and a bit passed the budget I want to put toward the older hardware I have in this server. Plus, it's probably in need of a main board replacement, and I'd like to eventually replace it with a rack mount case.

While I may explore the possibility of bumping the RAM from its current 8GiB to 16GiB, I'm reminding myself that the more I spend on it now means the longer I'll have to wait before I eventually relent, save up the extra money, and replace it with better hardware. Except right now the problem is two-fold: 1) The server market for low TDP CPUs is pretty abysmal, and only Intel has 25W rated CPUs worth anything; but 2) being as they're Atom-based chipsets, they don't have very good loading characteristics if you're to believe some of the more-anecdotal-but-not-quite-purely-synthetic benchmarks. For my use case, it's probably fine, but I'll have to see how they perform when serving up a game of Minecraft (if they perform at all). I actually have an Atom board somewhere that's of about 2009 vintage or so, and it might be useful to use a chipset I know to be relatively underwhelming so I can gauge its relative performance. I just have to remember what I did with it. Bummer.

Also, I migrated the server's network settings to systemd-networkd for a couple of reasons. First, netctl is relatively slow and generally blocks network-dependent services (PostgreSQL, MySQL, etc) until it's done loading. Second, netctl may eventually be deprecated on Arch once systemd-networkd reaches better feature parity (it's basically there, IMO). Finally, it's more aware of interface dependencies based on configuration. If you have an IPv6 tunnel (like I do), systemd-networkd is smart enough to know that your external interface needs to be active before the tunnel can be brought up. Likewise, if you have a bridge setup (like I do for containers), it'll configure that before bringing up other interfaces that depend on it. netctl has dependency configurations, but I've never managed to get them working well enough with a complex setup. It works fine on a desktop with one NIC, but as soon as you have more than one NIC, a tunnel, and other related configs, netctl quickly becomes unwieldy and the dependency nightmare grows substantially. This doesn't mean anything to you, but if you tried connecting recently and it was down, I was probably in a cycle of reboot-break-fix-reboot or restart networking-break-fix-restart networking much of the evening yesterday and into part of the day today.

Pending any unusually annoying problems, I don't expect anything else to change. Minecraft should be available for when you guys want to jump on, and if there's enough interest, it might be possible to talk me into restarting our longest running incantation of it with the 1.8 beta world (the one that lived most of the way through release). I just can't decide if I want to keep it at the last Minecraft version it was running or whether I should just let it upgrade the world files...

Anyway, hope this explains some of the downtime recently. This server is presently doubling as my GitLab instance for work-related nonsense, so while I try to do what I can to keep Minecraft running, game hosting isn't exactly one of the top priorities for this hardware.

[Zancarius]

Apparently ZFS is not without its teething problems.

I started an rsync backup process on Tuesday night only to find it hung about a half hour later. Kernel I/O was pegged and rsync had all but frozen due to CPU starvation, and Minecraft eventually exited shortly thereafter for the same reason. I suspect the problem is related to zfsonlinux/zfs#4319 and more specifically explained by this comment by one of the lead developers. Disabling dynamic thread creation has returned ZFS to a functional state.

Minecraft appears to be completely stable now and has been through yesterday after applying this workaround.

I will probably be taking the server down again for maintenance sometime in the next week or two to apply some updates, but I'll make a new post when that's coming down the turnpike.

[Zancarius]

There has been an ongoing issue with the server related to deadlock conditions in ZFS causing Minecraft to panic believing has crashed. While I don't imagine this is causing any data loss with the server itself, I am going to be taking steps to generate some nightly backups and/or snapshots of the server. The issue is most probably related to this issue and causes arc_reclaim and related kernel threads to enter a deadlock condition. During this time, system load averages spike, the system becomes almost unresponsive and sluggish, and Minecraft generally complains:

Code: Select all

[13:16:00] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 2921ms behind, skipping 58 tick(s)
[13:16:46] [Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 3443ms behind, skipping 68 tick(s)
[13:18:38] [Server Watchdog/FATAL]: A single server tick took 60.00 seconds (should be max 0.05)


I'm not sure if the server save state completes successfully during this time. From my limited testing, it appears as though it does, but being as this is the first crash I've seen that occurred while someone was connected (sorry Matt), there's a small possibility that about a minute's worth of work may have been lost. The user data has a timestamp indicating a save point just a few seconds before the crash completed, so I suspect the save ticks still function.

The cause of the ZFS deadlock is currently unknown but may be related to a data backup process that ran yesterday and likely triggered sometime in the early hours of the morning. I will be implementing some additional checks tonight to make sure that if this does happen it doesn't remain in this state for long and sends me an alert.

I apologize for the inconvenience this may be causing. I switched this machine to ZFS because it's my file server that I use for data storage here at home and also functions as a remote backup for a few other odds and ends. It isn't intended as a game server, but I'm more than happy to run Minecraft on it for us. Unfortunately, this does mean there are going to be some times during which the Minecraft server may be inoperable, either due to maintenance or due to unforeseen circumstances.

I'm hoping the ZFS on Linux project pushes some of these patches into their stable branch soon in the hopes this will be resolved. I'm reluctant to run their master branch on live data, at least for now, because generating server backups takes so long due to interruptions from deadlocks triggered by rsync.

[Zancarius]

I've migrated the server to a Spigot instance running several plugins we were once accustomed to operating on the Goon instance, including WorldEdit and WorldGuard. I'll be exploring permissions options for granting some world control (water filling/leveling) to all users. We may revisit group membership allowing (or not) the ability to interact with the world until such time as a moderator is able to upgrade a new user's group. At this point in time, the whitelisting mechanism has been disabled and all are welcome.

Backup generation has been moved to Git.