this post was submitted on 04 Jun 2025
351 points (100.0% liked)

Fediverse

34444 readers
1923 users here now

A community to talk about the Fediverse and all it's related services using ActivityPub (Mastodon, Lemmy, KBin, etc).

If you wanted to get help with moderating your own community then head over to !moderators@lemmy.world!

Rules

Learn more at these websites: Join The Fediverse Wiki, Fediverse.info, Wikipedia Page, The Federation Info (Stats), FediDB (Stats), Sub Rehab (Reddit Migration)

founded 2 years ago
MODERATORS
ruud@lemmy.world
TragicNotCute@lemmy.world
Xylinna@lemmy.world
MrCenny@lemmy.world
automodbeta@lemmy.world
woelkchen@lemmy.world
351
Slrpnk.net outage (resolved) (feddit.org)
submitted 1 week ago* (last edited 2 days ago) by Kris@feddit.org to c/fediverse@lemmy.world
133 comments fedilink hide all child comments
 

Edit: unexpectatly we are back online. Will share how we found a solution later. There might be still some federation issues though.

Alt account of poVoq@slrpnk.net here.

Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.

I am still trying a few things, but I am not very optimistic that I can get access.

The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.

As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable πŸ˜“

I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing πŸ˜₯

Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.

top 50 comments
sorted by: hot top controversial new old
[–] Coolkat@jlai.lu 3 points 3 days ago (1 children)

I have heard of the migration to Piefed, but will it remove the (very cool) aesthetics of the lemmy ? I loved it :'(

[–] Kris@feddit.org 4 points 3 days ago (1 children)

I think we will try to come up with a theme for Piefed with similar colors.

[–] Coolkat@jlai.lu 1 points 2 days ago

yayyy :D

[–] signaleleven@lemmy.sdf.org 3 points 3 days ago

Found this post after a couple of days of hopeful reloading of the page! Took another couple of days to log in an old alt just to say I'm glad you communicated this, and let us know how we can help to make it more resilient (if at all necessary).

To the kind soul that was helping me with my greenhouse infested ants (in the unlikely event they are reading here - but iirc they were not from slrpnk.net but just passing by) I'm glad we spoke, it seems like they are harmlessly using my panels for hatching, and then moving on! Related, who has a nice gardening/greenhouse community to suggest? :)

[–] silence7@lemmy.world 5 points 6 days ago (1 children)

Thanks for letting us know what is going on.

[–] Kris@feddit.org 5 points 5 days ago (1 children)

Sorry that it took so long to get the word out. I guess we relied a bit too much on the xmpp server for internal communication.

[–] silence7@lemmy.world 6 points 5 days ago (1 children)

Even the big operations break in unexpected ways. Facebook accidentally broke the authentication for their door locks when they took the site down a few years back and had to fly people to a datacenter and break down a door to get to servers and restart things.

[–] Kris@feddit.org 3 points 2 days ago

We managed to resolve the issue and I have many ideas how to avoid this in the future πŸ€“

[–] wit@lemmy.world 1 points 5 days ago* (last edited 5 days ago) (1 children)

I am a slrpnk user. Oh well, shit happens. What makes me a bit sad was that a few months (?) ago something similar happened. Slrpnk.net was down for aa day or two? Don't remember the exact amount of time. What I do remember is asking if there was intention of adding a second admin and making the point that it would be good to create that sort of redundancy. I was ignored.

Mid of july is basically 1 and a half months from now. Hopefully a new solarpunk instance emerges, one that is safer from such scenarios.

Enjoy your vacation and don't stress too much! Shit happens.

[–] Kris@feddit.org 6 points 5 days ago (1 children)

I am genuinely confused. What makes you think you were ignored? We do have additional admins, the problem is physical access to the servers to replace hardware that seems to have broken.

We plan to add additional safeguards to such scenarios, but in the end it is a trade-off between reliability and costs. Running a Lemmy instance of our size on a fully managed rented server would easily triple the monthly costs. This would be beyond what I am willing to cover out of my own private pocket, thus making the instance dependent on donations. And that IMHO makes it more likely to be shut down in the future as donations are very variable.

I am not on vacations, but on a work deployment abroad, which further limits the time I can spend on this. Not that it would have made much of a difference, but at least the domain redirection and placeholder text was only possible to do on the weekend.

[–] wit@lemmy.world 0 points 4 days ago* (last edited 4 days ago) (2 children)

Well, I got no reply and I didn't see anyone else in the "mods:" bit in the bottom of the sidebar on slrpnk.net. I understand that having more admins wouldn't help in this specific case, but it would help in others.

Regardless, having a public lemmy instance only ran by 1 person is, in my opinion, a bad practice. And please know that I am not personally attacking you. I am just stating my thoughts. This time you were out on a work deployment. Next time you could be sick. The next time maybe a family member had an emergency and you had to help. And the next time maybe your house caught fire or whatever.

Resilience and redundancy is extremely important in public lemmy instances. And SolarPunk is, in my opinion, all about community. There should be more (trustworthy) people with the same level of access as you do.

Regarding the price and all that, maybe that is an interesting discussion to have with the slrpnk.net community? And it would also be good to let slrpnk.net users know how the instance is being ran, what hardware, where, etc, so users can be an educated decision on whether they want to see an instance or not? Maybe providing that info on the sidebar would be nice?

But again, shit happens. Hopefully this serves as a learning event. If not, so much shit is going through the world, losing a lemmy instance is not something about which one should lose sleep.

EDIT: I may be missremembering about the whole "something similar happened a few months ago and my question about new admins was ignored"... I think that might have been on mander.xyz and not slrpnk.net. I really can't fully remember, but I think it was mander.xyz after-all. Sorry! The rest of the post stands though!

[–] PirateFrog@lemmy.dbzer0.com 4 points 4 days ago

I see in your edit that you confused us for a different instance, but I want to mention regardless that Slrpnk.net has 3 active admins, it's not just poVoq (Kris).

[–] Kris@feddit.org 4 points 4 days ago (1 children)

Regarding the price and all that, maybe that is an interesting discussion to have with the slrpnk.net community? And it would also be good to let slrpnk.net users know how the instance is being ran, what hardware, where, etc, so users can be an educated decision on whether they want to see an instance or not? Maybe providing that info on the sidebar would be nice?

We have been very transparent about that and documented that in the wiki which is linked in the sidebar. But it seems few people ever read the Wiki and having it integrated like it is with Piefed would probably help.

I agree that more admins with remote access would be good to have, and we have been working towards adding more (there are already more than one, who are not directly involved with slrpnk, but rather the mother project f-hub.org), but physical access will always be limited by the location (Azores) unless we move to an entirely different data-center (which in turn would make it hard for me to access the physical servers).

[–] wit@lemmy.world 1 points 4 days ago (1 children)

We have been very transparent about that and documented that in the wiki which is linked in the sidebar. But it seems few people ever read the Wiki and having it integrated like it is with Piefed would probably help.

Oh ya. I knew about that, and I am fine with that. Though I think having it behind a link is bad UI. People will not read it haha.. It should be presented on account creation, in my opinion. Just 1 sentence "This instance is ran on private hardware and may suffer some down time here and there" would suffice.

[–] Kris@feddit.org 3 points 4 days ago

Sure, that seems reasonable. We will add that when the instance is back online.

If I forget, please remind me. For sure we will not just ignore you 😊

[–] tofu@lemmy.nocturnal.garden 96 points 1 week ago (3 children)

Solarpunks can have a little downtime, as a treat

[–] evasive_chimpanzee@lemmy.world 62 points 1 week ago (1 children)

Seriously, I think a big part of solarpunk ethos is combating the notion that everything has to always be available 24/7. Society pays a lot to deliver every convenience like fruit out of season from the other side of the world.

[–] stabby_cicada@lemmy.blahaj.zone 27 points 1 week ago

And on the punk side of things, anybody familiar with volunteer community action, cooperative events, whatever, knows that sometimes a person doing shit for the community fails to do the shit they were supposed to do because some other shit happened in their life. It happens. We just move on to the backup shit-doer-person or find a workaround for the lack of shit.

And this is the cool thing about a distributed network like Lemmy. If a giant centralized database like FB goes down, its users can't do anything. If our favorite Lemmy node goes down, we can log into other nodes until our home base is back up.

load more comments (2 replies)
[–] ZonenRanslite@feddit.org 46 points 1 week ago (2 children)

Work trip and server problems. Where have I seen this before?

[–] Kris@feddit.org 34 points 1 week ago (1 children)

Yeah, but at least I have not disappeared yet πŸ˜…

[–] ZonenRanslite@feddit.org 15 points 1 week ago (3 children)

I see you understand the allusion.

load more comments (3 replies)
load more comments (1 replies)
[–] Sergio@lemmy.world 40 points 1 week ago

Sorry to hear that! My main account is on slrpnk.net. But don't let it stress you out, "solarpunk" says "punk" which implies "DIY" which means this sort of thing happens. Much better to have an instance run by an admin who truly cares, rather than some soulless corporation.

[–] Initiateofthevoid@lemmy.dbzer0.com 37 points 1 week ago (1 children)

Try not to stress too much about it while you're away! If nothing can be done, nothing can be done. Solarpunks need to be resilient to things like this anyway. Community over convenience.

[–] Kris@feddit.org 20 points 1 week ago

Thanks, I am trying, but it really is quite annoying to lose access like that.

[–] PirateFrog@lemmy.dbzer0.com 35 points 1 week ago (3 children)

Alt of ProdigalFrog@slrpnk.net chiming in, thanks for giving us the sitrep, poVoq!

This is an unusually unlucky set of circumstances, but for the Solarpunks who find their way here, let's try to make the best of it. If you don't already have an alternative account somewhere, you may want to create one to use until we're back up and running. lemmy.dbzer0.com, sopuli.xyz, or lemmy.cafe are some solid choices to consider. After that, you may want to favorite poVoq's post here so you can check back on any news.

Even without our home base, we can still share the solarpunky stuff we find and create around the threadiverse (though as Tofu mentioned, maybe this could be a time to take a breather to be refreshed when we come back πŸ˜„).

Besides, if the past is any indication, time will fly by and we'll be back before you know it!

[–] Kris@feddit.org 15 points 1 week ago (6 children)

Well, on the plus side lots of lessons learned and I think I might move at least the xmpp server to an external vps to have a backup communication channel.

load more comments (6 replies)
load more comments (2 replies)
[–] grysbok@lemmy.sdf.org 34 points 1 week ago (21 children)

This is making me want to create an alt on slrpnk.net , since you're so chill and communicative about issues.

load more comments (21 replies)
[–] sabreW4K3@lazysoci.al 28 points 1 week ago

Thanks for letting us know

[–] Blaze@lemmy.dbzer0.com 21 points 1 week ago

Thank you for letting us know, good luck!

[–] stabby_cicada@lemmy.blahaj.zone 21 points 1 week ago

Shit happens. Thanks for the update πŸ™‚

[–] ocean@lemmy.selfhostcat.com 21 points 1 week ago (1 children)

I this it’s a law that servers run 100% perfect until the literal day one leaves town with zero way to return home. One of the many reasons I got all my services off of unraid.

Very cool to learn you’re running your own machines. Do you go into detail about this anywhere?

[–] Kris@feddit.org 15 points 1 week ago* (last edited 1 week ago)

We have a small write up about the hardware on our wiki, but it is also down right now.

I think we will share a post-mortem write up of the actual improvements we will do to avoid this in the future.

One thing I will definitly do is to add a KVM remote management console to one of our server boards and move the main firewall into a VM with hardware passthrough of the NICs (this was anyways planned for a 10gbit network upgrade for the second half of 2025). This way I should be able to reboot and even reinstall the main ingress point remotely, so that only the fiber gateway remains as a failure point that requires physical access.

[–] gandalf_der_12te@discuss.tchncs.de 21 points 1 week ago (2 children)

This is actually great timing, i'd say. Just now most of the northern hemisphere is starting into summer. At least i deeply feel it on a personal level that summer is arriving. I take this as a gift from the universe that directs me towards touching grass, and going swimming. Enjoy your summer, you all! Connect with your local community, if you can.

[–] DragonTypeWyvern@midwest.social 17 points 1 week ago

thirty minutes later

Okay I touched grass and now I'm bored and itchy

[–] Kris@feddit.org 15 points 1 week ago

Glad that at least someone sees a positive side of it, because I sure don't... well at least it gives me a bunch of ideas how to avoid such situations in the future 🀷

[–] Sergio@lemmy.world 20 points 1 week ago (4 children)

Hey fellow slrpnk.net refugees... if you're looking to recreate your list of subscribed communities, here's a good place to start:

click here for a list of communities that are NOT politics, tech, or meme -related.

Most are currently active (except for the ones with a * which were less active last I checked) Sometimes politics, tech or memes sneak in but they're not the focus.

GENERAL DISCUSSION / QUESTIONS

ART / PHOTOS

ANIMALS

COMICS / GRAPHIC NOVELS

ENTERTAINMENT

GENRES / STYLES

HISTORY

INFORMATION / KNOWLEDGE

OTHER

FEDIVERSE

FINDING NEW/GOOD COMMUNITIES ON LEMMY

click here for a list of (mostly non-political) meme communities

MEMES, SOCIAL MEDIA REPOSTS, AND HUMOR

Most of these are currently active. (except for the ones with a * which were less active last I checked). Sometimes these include politics but that's not the sole focus.

(luckily I was posting this to newcomers whenever I had the chance...)

load more comments (4 replies)
[–] qaz@lemmy.world 19 points 1 week ago

Thanks for the update, good luck with your trip.

[–] FrostyTrichs@crazypeople.online 17 points 1 week ago

Ouch. Best of luck getting it back up and running ASAP.

load more comments
view more: next β€Ί