Does lemmy have any communities dedicated to archiving/hoarding data?
Is there a context to this or just random thought?
this post was submitted on 17 Oct 2025
1081 points (99.0% liked)
linuxmemes
27710 readers
1687 users here now
Hint: :q!
Sister communities:
Community rules (click to expand)
1. Follow the site-wide rules
- Instance-wide TOS: https://legal.lemmy.world/tos/
- Lemmy code of conduct: https://join-lemmy.org/docs/code_of_conduct.html
2. Be civil
- Understand the difference between a joke and an insult.
- Do not harrass or attack users for any reason. This includes using blanket terms, like "every user of thing".
- Don't get baited into back-and-forth insults. We are not animals.
- Leave remarks of "peasantry" to the PCMR community. If you dislike an OS/service/application, attack the thing you dislike, not the individuals who use it. Some people may not have a choice.
- Bigotry will not be tolerated.
3. Post Linux-related content
- Including Unix and BSD.
- Non-Linux content is acceptable as long as it makes a reference to Linux. For example, the poorly made mockery of
sudoin Windows. - No porn, no politics, no trolling or ragebaiting.
4. No recent reposts
- Everybody uses Arch btw, can't quit Vim, <loves/tolerates/hates> systemd, and wants to interject for a moment. You can stop now.
5. π¬π§ Language/ΡΠ·ΡΠΊ/Sprache
- This is primarily an English-speaking community. π¬π§π¦πΊπΊπΈ
- Comments written in other languages are allowed.
- The substance of a post should be comprehensible for people who only speak English.
- Titles and post bodies written in other languages will be allowed, but only as long as the above rule is observed.
6. (NEW!) Regarding public figures
We all have our opinions, and certain public figures can be divisive. Keep in mind that this is a community for memes and light-hearted fun, not for airing grievances or leveling accusations. - Keep discussions polite and free of disparagement.
- We are never in possession of all of the facts. Defamatory comments will not be tolerated.
- Discussions that get too heated will be locked and offending comments removed. Β
Please report posts and comments that break these rules!
Important: never execute code or follow advice that you don't understand or can't verify, especially here. The word of the day is credibility. This is a meme community -- even the most helpful comments might just be shitposts that can damage your system. Be aware, be smart, don't remove France.
founded 2 years ago
MODERATORS
You can ignore politics, but politics will not ignore you.
Years ago I bought a physical encyclopedia. I remember having one as a kid and using it for school reports. Also just looking through it can be cool. Learning about something you never knew existed is just a unique experience and doing it through a physical book just deepens the whole experience.
I also learned the practice of printing a physical encyclopedia is going out of fashion. I think there is only one company the still prints a yearly encyclopedia and it's not Encyclopedia Britannica of all things. Might have change since I bought my copy but go give some physical media some love if you can.
I bought a 14tb drive just for backups of all my other drives... and I got a shitload more space.
I thought the whole point of torrenting was to decentralise distribution. I use torrents to get my distros.
In my own little bubble, I thought that's how most people got their distro.
What happens when they just cut the underwater cables? Torrent over carrier pigeon for a linux distro would take ages
Sneakernet to the rescue. Some of you are too young to know about walking around with boxes full of disks.
@Maroon I thought torrent technology to be a godsend for package managers.
Why none of them use it?
I mean, damn.
Neither are that bad honestly. I have jigdo scripts I run with every point release of Debian and have a copy of English Wikipedia on a Kiwix mirror I also host. Wikipedia is a tad over 100 GB. The source, arm64 and amd64 complete repos (DVD images) for Debian Trixie, including the network installer and a couple live boot images, are 353 GB.
Kiwix has copies of a LOT of stuff, including Wikipedia on their website. You can view their zim files with a desktop application or host your own web version. Their website is: https://kiwix.org/
If you want (or if Wikipedia is censored for you) you can also look at my mirror to see what a web hosted version looks like: https://kiwix.marcusadams.me/
Note: I use Anubis to help block scrapers. You should have no issues as a human other than you may see a little anime girl for a second on first load, but every once and a while Brave has a disagreement with her and a page won't load correctly. I've only seen it in Brave, and only rarely, but I've seen it once or twice so thought I'd mention it.
I rarely get bounced by Anubis, but oddly enough it has happened to me a couple times in FF, I suspect itβs the fingerprinting resistance settings that cause this to happen? Hasnβt happened in a while though
Welcome to datahoarders.
We've been here for decades.
Also follow 3-2-1 people. 3 Backups, 2 storage mediums, 1 offsite.
"backups"? Pray tell, fine sir and or madam, what is that?
You know there's only two kind of people, those who do backups and those that haven't lost a hard drive/data before. Also: raid is no backup
FWIW :
fabien@debian2080ti:/media/fabien/slowdisk$ ls -lhS offline_prep/
total 341G
-rw-r--r-- 1 fabien fabien 103G Jul 6 2024 wikipedia_en_all_maxi_2024-01.zim
-rw-r--r-- 1 fabien fabien 81G Apr 22 2023 gutenberg_mul_all_2023-04.zim
-rw-r--r-- 1 fabien fabien 75G Jul 7 2024 stackoverflow.com_en_all_2023-11.zim
-rw-r--r-- 1 fabien fabien 74G Mar 10 2024 planet-240304.osm.pbf
-rw-r--r-- 1 fabien fabien 3.8G Oct 18 06:55 debian-13.1.0-amd64-DVD-1.iso
-rw-r--r-- 1 fabien fabien 2.6G May 7 2023 ifixit_en_all_2023-04.zim
-rw-r--r-- 1 fabien fabien 1.6G May 7 2023 developer.mozilla.org_en_all_2023-02.zim
-rw-r--r-- 1 fabien fabien 931M May 7 2023 diy.stackexchange.com_en_all_2023-03.zim
-rw-r--r-- 1 fabien fabien 808M Jun 5 2023 wikivoyage_en_all_maxi_2023-05.zim
-rw-r--r-- 1 fabien fabien 296M Apr 30 2023 raspberrypi.stackexchange.com_en_all_2022-11.zim
-rw-r--r-- 1 fabien fabien 131M May 7 2023 rapsberry_pi_docs_2023-01.zim
-rw-r--r-- 1 fabien fabien 100M May 7 2023 100r-off-the-grid_en_2022-06.zim
-rw-r--r-- 1 fabien fabien 61M May 7 2023 quantumcomputing.stackexchange.com_en_all_2022-11.zim
-rw-r--r-- 1 fabien fabien 45M May 7 2023 computergraphics.stackexchange.com_en_all_2022-11.zim
-rw-r--r-- 1 fabien fabien 37M May 7 2023 wordnet_en_all_2023-04.zim
-rw-r--r-- 1 fabien fabien 23M Jul 17 2023 kiwix-tools_linux-armv6-3.5.0-1.tar.gz
-rw-r--r-- 1 fabien fabien 16M Oct 6 21:32 be-stib-gtfs.zip
-rw-r--r-- 1 fabien fabien 3.8M Oct 6 21:32 be-sncb-gtfs.zip
-rw-r--r-- 1 fabien fabien 2.3M May 7 2023 termux_en_all_maxi_2022-12.zim
-rw-r--r-- 1 fabien fabien 1.9M May 7 2023 kiwix-firefox_3.8.0.xpi
but if you want the easier version just get Kiwix on whatever device in front of you right now (yes, even mobile phone assuming you have the space) then get whatever content you need.
If need a bit of help I recorded TechSovereignty at home, episode 11 - Offline Wikipedia, Kiwix and checksums with a friend just 3 weeks ago.
I also wrote randomly update https://fabien.benetou.fr/Content/Vademecum and coded https://git.benetou.fr/utopiah/offline-octopus but tbh KDE-Connect is much better now.
The point though is having such a repository takes minutes. If you don't have the space, buy a 512Go microSD for 50EUR then put that on, stuff it in a drawer then move on. If you want to every 3 months or whenever you feel like it, updated it.
TL;DR: takes longer to write such a meme than actually do it.
Watch out for flash data corruption. Lots of cheap flash (USB sticks, SD cards, SSDs) lose data after just a few years of offline storage. Something something quantum tunnel bullshit, iirc.
So either look for media that guarantee long cold storage retention (lots of businesses need to keep shit for 10 years for tax reasons), or occasionally plug it in and let do the housekeeping.
Thanks but even though it's on a plugged HDD I don't even care for any of that data. What I mean is that none of that data is sensitive. It might be useful, potentially, but it's not unique. What I mean is that if somehow my .zim file for Wikipedia was corrupted I could download it again from https://library.kiwix.org/#lang=eng&category=wikipedia or elsewhere in ~30min (just checked).
What I'm trying to highlight here is more the process than the actual outcome.
TL;DR: yes, if one is actually serious about just getting and storing, they should verify periodically if the data is indeed fine. What I do want to highlight though is to first know how to do it at all. Anyway, you are right that for a proper solution on the long run one must understand how (cold) storage actually works. My heuristic is that it's like can food (which I don't use much), it might last a while, but not forever.
I thought the point of backing stuff up was to have things in case just downloading it again isn't a viable option?
It's more that flash NAND uses a small electric charge to keep the NAND gates in the correct configuration. Over time, that charge dissipates. If you power the storage device every once in a while, you minimize these chances.
Here's a video explaining why it happens to Wii U's after being powered off for a while. https://youtu.be/JHME4zLs6Qs
load more comments
(2 replies)
I still have a copy of wikipedia from 2021 somewhere on my NAS.
Might store it on an external HDD. I got plenty.
I heart there is Wikipedia on ipfs. Is that a good solution for Linux packages too?
Last year I bought a hard copy of my favorite webcomic in case the website goes down.
load more comments
(4 replies)
I kind of want that hackermans diy pc that runs on 18650 cells
we need all repos to be stored offline, and documentations to troubleshoot.
the 1st i have no idea how much space we will need. Most linux packages are prerry light, no? But there is A LOT of them...
the 2nd is easy. Heard someone say the entire of wikipedia is 200GB, should be doable. Dont forget the technical wikis too: Debian, Gentoo, Arch.
Can't remember who it was (b3ta? popbitch? penny-arcade?), but I recently saw a comment by someone who's been running a website since the turn of the millennium, and they said that fully 99% of the links they posted two decades ago were no longer valid.
To really put that into perspective, you have to remember that for most sites to get linked to from a popular site like that, meant that it was usually something of value that would have had a lot of work put into it, and that people found interesting or useful.
load more comments
(1 replies)
For wikipedia you'll want to use Kiwix. A full backup of wikipedia is only like 100GB, and I think that includes pictures too.
120GB not including Wikimedia π
Also, I wish they included OSM maps, not just the wiki.
load more comments
(8 replies)
On prem NAS or SAN⦠you should have built it 3 years ago.
why? what happened 3y ago?
view more: next βΊ