Federation and Fandom

[impertinence]

Fandom is, once again, in flux. Anyone who's been around for more than a few years knows the signs have been building: deleted accounts and new terms of service, followed by talk of moving elsewhere, fracturing communities over multiple platforms, sometimes breaking communities entirely. It's a common pattern a lot of us have come to see as an inevitability, just something that happens every five or ten years. But of course there is a cost, archival as well as communal. That chatfic you loved gets deleted. The old homepage of your favorite vidder goes down. Entire fandoms' discussions disappear forever.

We've all been there at least once, Or we're about to be there, because again: it's happening with Tumblr. People won't all leave at once. The demise of LJ was measured in years, not days. But eventually, even if it's still up, it won't be what it was. Content will slowly degrade - image hosting, external links, even the posts themselves deleted due to inactivity or changes in management. Link rot, in other words.

So. It's happening. What can be done about it?

There have been a lot of really good discussions so far. Consensus so far appears to be some combination of the following: the new platform must allow photo/text/video sharing as well as conversations; the new platform must allow NSFW content and have robust filtering; the new platform must be resistant to the kind of profit-driven vicissitudinous behavior most tech companies embrace.

I agree with all of this. I also agree with the problem statement, which I've seen repeated again and again, that fandom should be more distributed - that no single (especially not corporate-owned) social network should be the point of failure.

So. Let's talk about server federation.

The most well-known federated technology right now is probably Mastadon. In simple terms, Mastadon is a Twitter-like social network. However, one huge difference between it and Twitter is federation.

What is federation? Well, once upon a time, the internet was much more distributed. One of the backbones of that distribution was mailing lists - which were email-based. Imagine a world where you can only send emails to other Gmail users! That's basically how Twitter functions right now. Email, fortunately, is protocol-based; any technology that uses the protocol can send and receive emails. What matters for communication is the protocol, not the website or brand or company or framework. Federated social networks, wikis, and media hosting aims to bring that pattern back.

Under the hood, Mastadon implements protocols like ActivityPub to exchange information. These protocols are standardized so that different Mastadon servers can replicate data, and so that different platforms can communicate and exchange data. What this means in practical terms is that the following scenarios are possible:

- On Mastadon.one, A makes a post. On Mastadon.two, B enters "UserA@Mastadon.one" in the search bar. B then follows A, and can see user A's posts, even though they are on totally different servers run by completely different people.
- On Mastadon.one, A follows UserB@Hubzilla.two. Hubzilla is a totally different, Tumblr-esque platform. But because Hubzilla also implements the ActivityPub protocol, A can see B's posts in A's Mastadon feed.
- A is on Mastadon.one. On Mastadon.one, NSFW content is required to be flagged. B is on Mastadon.two. There are no content controls on Mastadon.two. Despite very different priorities and values, A can still follow B and vice versa, while remaining within their own communities.

This sort of setup is also fairly future-proofed: what matters is the protocol. You don't have to worry about someone's hobby framework going under, for example. Hubzilla takes this idea one step further and implements what they call "nomadic identity". The basic idea is that you can log into different Hubzilla servers with the same credentials, and each account is equivalent, so even if your original account dies, your identity persists on all other servers.

Federation would solve a lot of fandom's current problems. For starters, there is a huge variety of roll-your-own federated services out there. There's Peertube, which is focused on video hosting; Pixelfed, focused on image hosting; Mastadon and other microblogging frameworks; Friendica and Osana, which are focused on social networking; Hubzilla, which is focused on all-purpose social networking, file storage, and wiki management; and Nextcloud, which is focused on file storage. This network of frameworks is incredibly powerful, and some of them are very easy to set up. I've run a Nextcloud instance for several years. It has my calendar, my files, a chat platform, an audio streamer, a Trello-esque project management app, and an e-reader. Hubzilla can be used as a wiki, a forum, a personal blog, a Reddit-esque community, or some unholy combination of all of the above. All of these platforms can communicate within separate instances of the same framework (Nextcloud to Nextcloud, or Friendica to Friendica), and many can communicate between frameworks too.

So: there are frameworks to do basically every fannish activity you could imagine, and a few of them even encompass Tumblr's functionality. But also, because these frameworks were built by privacy-focused nerds, there are often a LOT of privacy and filtering controls. The idea behind federated communities is partially that even groups of decent people aren't always going to get along, and a group of 100k+ people is likely to be incredibly hard to manage and no longer a true community. Federation gives you the ability to exist in online communities without being completely isolated from one another. Mastadon takes this idea so far that it shows health as green/yellow/red, with larger instances being less healthy.

Federation allows for data replication, so that data's more resistant to takedowns. All of the above frameworks are open source, which means that fandom-specific extensions can be built without having to build the whole framework. Fandom has tons of technologically capable people who'd be capable of installing instances, and tons of data science people who could help ensure they don't disappear.

I would strongly suggest everyone look at federated frameworks, maybe play with getting them set up if you already have a server. This would be a brilliant and relatively quick way to get people onto independent platforms. Corporations' websites are often nicer, because they pay people to make them look shiny and new. Those salaries come from you, come from us. Ad revenue, subscription, angel investment, it doesn't matter; eventually, some C-suite douche will look at all our wonderful weirdness and ask why they can't pull in influencers instead. We know we need to get out ahead of that garbage; that's why AO3 exists. My proposal is we build on that by joining the ranks of people who know decentralization is the answer to passionate, fractious communities staying together. It's time for fandom to join the fediverse.

Comments

[cesperanza]

THIS THIS THIS THIS THIS THIS. Also: THIS. And in conclusion: THIS.

[impertinence]

:D

[norsellie]

OMFG YES.

[impertinence]

let's gooooooo :D

[ithiliana]

Here from a link in cesperanza's journal. I'm fascinated by this whole new (to me!) technology and possibilities of fandom (though right now, it's zooming way over my head)!

[impertinence]

Awesome! Let me know if you have any specific questions or if you'd like to join a demo instance. I think the best way to think of it is like email - yahoo's mastadon, gmail's hubzilla, but they can still talk to each other! but admittedly it's definitely a more niche thing right now.

[vaysh]

I'm here as well from Speranza's journal. I am equal parts intrigued and overwhelmed by the concept of federations. Can you explain, to a layperson, what "instances" are? Thank you!

[ithiliana]

Am with you in the "intrigued and overwhelmed" corner!

[impertinence]

Absolutely! So I'm going to talk about Hubzilla, but everything I say should hold true for the other technologies (Mastadon, Nextcloud, Peertube, etc). If I'm talking about something Hubzilla specific I'll mention it.

An instance is simply an installation of the framework on someone's server - a website, basically. I have a Hubzilla instance up right now, and what that means is I've logged onto my server and installed Hubzilla-the-framework to be a live website others can visit and use. The actual files the site needs live there, as well as user data.

Unlike a website like LJ or Twitter or Tumblr, though, these frameworks implement a variety of protocols to talk to each other. So there's basically a set of rules for what info one instance - one server - sends to another when doing common things like sending an update to someone's feed, liking/favoriting a post, following a user, and so on. (Hubzilla uses ActivityPub to communicate with Mastadon, its main protocol is Zot, there are others - but if you want to do some googling, ActivityPub is a good place to start.) Because these protocols are implemented by the frameworks, it's possible for, say, a Hubzilla instance - a single Hubzilla application running on a single person's server - to talk to both other Hubzilla instances, and other instances of Mastadon, Pleroma, and so on. Just as you might send email to someone's Outlook inbox via Gmail, so you can follow and reply to people on Mastadon from Hubzilla.

Federation itself also means there are levels of trust between servers. So for Nextcloud, for example - Nextcloud is a federated file server, and what this means is that I can share files with a "trusted" server, the same way I'd be able to if the users on that server were on my server. And it is again about protocols, rules of information exchange that allow things to run smoothly even when the actual users are on totally separate machines.

[raine]

I'm having serious flashbacks to the days of BBSes and private BBSes now - but yes, this has to be the best way forward, given the tech we have now.

Honestly, I don't think it matters which highway we use as long as we'll all in the same mall together to see the movie.

[impertinence]

Exactly! And one nice thing about federation is that you CAN actually choose different tech with wildly different user experiences, and still talk to each other. Which, like, imagine the dw vs tumblr thing if they were federated. (Impossible at the time, but not impossible anymore!)

One area where I think a central authority would be very valuable is in coordination & information backup - I've seen people talk about AO3 for social networks, and if someone started that kind of thing up, using an existing federated framework, then the focus could be on facilitating other fans starting small servers and backing up identities and data centrally. Which would rule! But that's kind of a whole separate convo.

[peoriapeoriawhereart]

So, I'm understanding this as a capital (servers) and labor collaboration, where it's not who has the most toys wins, because having too many users in the same instance gives you the sad plant red condition.

From Arpanet to a network of Walters.

[impertinence]

That's a good way of putting it! It's very much a return to internet-as-NETWORK, not internet-as-monolith.

[havocthecat]

I would strongly suggest everyone look at federated frameworks, maybe play with getting them set up if you already have a server.

What would you suggest for people who can't program worth a damn?

[impertinence]

If you want to play with Mastadon, Hubzilla, or others, there are public instances you could join. Otherwise, I'd say join post_tumblr_fandom and/or the discord, as eventually there might be calls for testing. There's not a ton of programming needed - mostly customization/extension (where some programming is needed but not from-scratch stuff), installation (sysadmin/administration skills), and testing (anyone!). So it really depends on your comfort level, what you'd even want to do, if this kind of system sounds good to you, etc.

[lurkmuch]

Thank you so much explaining all this, it's fascinating! (I'm so tempted to play around with both of them if only I didn't have holiday-related crap to do.)

One question, if I may? (And I bet the answer is "it depends".) How much hardware do you need to run an instance? Or maybe more accurately...how much could I host with say, a Raspberry Pi? Or my 3 year old Macbook?

[impertinence]

Haha the answer IS "it depends" :D And it varies by app. Pleroma is a twitter-like one and says it can be run on a Raspberry Pi - I installed it on the cheapest Linode.com instance and it worked fine. For something more highly powered like Nextcloud (file hosting like Dropbox) or Hubzilla (wiki/chat/microblogging/blogging/web pages) you'd probably want a lower-range server - like one of the cheapest Linode.com instances I mentioned. So, yeah, it depends. But all of these aren't p2p tech, they're just regular php websites that implement specific communication protocols. So 1. you could set them up on a VPS or shared hosting, and 2. how much power you need will largely depend on how much traffic the site gets and what your storage needs are*.

* and one last digression - a lot of the platforms work with external storage, like ftp/s3/etc. So linode.com's pricing for example is mostly based on memory & traffic, not hard drive space, as they're assuming site runners will be using external storage.

[tei]

Here via yourlibrarian! This is super interesting, and I had no idea about any of this stuff before reading it, but it certainly sounds like a better bet than handing over fandom to any one website.

I have a bunch of questions, feel free to answer only some (or none) of them!

1. If I'm understanding correctly, it sounds like basically anyone can st up an instance, and have as many or as few people as they like on it. If that's the case, I'm assuming that there are a lot of instances that get abandoned or shut down when the people running them no longer want to keep doing it? When that happens, what are the options for users of that instance? Is it easy to just move your data over to another one that'll have you? I guess this will probably come up lots even in cases where an instance isn't actually shutting down... if I started using one of the open-to-anyone instances on Mastodon right now, and then wanted to move my existing data to a fandom-specific instance later, would that be easy?

2. I'm also curious if you have opinions on how instances should be organized, if fandom in general did start moving to a federated system. If an instance is just a bunch of data all stored in one place, it seems to me that tolerance to various levels of adult content would be the logical defining factor between instances-- is that how you see it? Or would there be an advantage to fandom- or ship-specific instances?

3. Is it possible for sites that already exit-- like DW and the AO3- to "turn themselves into" federated frameworks? Or is the difference between federated and non-federated social media fundamental enough that they're "stuck" even if fandom moves on to federated frameworks and comes to expect that functionality from our tools?

4. What do you (or anyone else here who is knowledgeable about this!) think needs to happen to get from where we are to fandom in general using primarily federated social media? From your description of hubzilla, it sounds like it's ugly and difficult to use but essentially workable as it stands as a replacement for tumblr-- is that so?

Thank you so much for this post!

[impertinence]

Hello! Thank you for reading my word vomit - I'm happy to answer as many questions as I can. :D

1. You are understanding correctly. As to the data issue, it depends on the platform. I installed Hubzilla specifically because of the "nomadic identity" thing - there's a description of what happens if one hub goes offline here. Now, that post elides the possibility of data loss. As far as I can tell, Hubzilla makes moving your identity easy, but moving your data more difficult. Mastadon doesn't have identity portability - but it does have cross-server communication for posts, likes, etc.

So basically - moving data isn't solved. You can talk between instances, but you can't necessarily move everything over. But having said that, your actual data is going to be stored either in the server's filesystem, or in external storage the sysadmin has configured. So my personal preference would be to try to set up at least a couple instances purely for redundancy, with database backups as needed, to guard against data loss. That's a trickier proposition but it's not impossible.

I will add, federation does define protocols for data getting in and out, so it's possible we'll see more advanced API-based data migration in the future.

2. I think it depends on the fandom! Adult content vs none seems like an obvious delineation, but I've definitely been in fandoms where the ships were either super separate or would've preferred to exist separately. And while federation isn't silo'd, obviously you could set up your own server where the rules were you can only call Spike "that motherfucker" and Angel "the one" if you really wanted. :D So - it depends and I have no super strong opinion, aside from generally agreeing that small-ish instances are better. Most of my opinions are on the infrastructure side - I think the one place centralization would be useful would be in helping with backups, DR (disaster recovery) plans, etc.

3. I definitely think it's possible. Federation really just refers to implementing a protocol such as ActivityPub, and as far as I'm aware there are no hard and fast limits about what kind of platform can implement those protocols. Having said that, I don't know that I'd consider it likely, simply because AO3 and DW are both pretty mature systems and weren't really built with federation in mind. But - never say never! It could certainly be a feature people requested.

4. Yes. I'd call it feature complete in terms of what tumblr can do vs what Hubzilla can do - but it's uglier and harder to use out of the box. So ideally we'd get a Hubzilla theme like I described, admin tools like I described, and - stretch goal - some kind of infrastructure for broadly decentralized data backup, both for fanworks (audio/image/vids) and for people's social media data. That is a serious stretch goal and doesn't really touch Tumblr competition; to be equal to Tumblr I mostly think it's a question of streamlining the install and creating a nicer interface. (But the data backup and disaster recovery stuff is still super important, imo.)

[morgandawn]

Yes, I tested out a PeerTube instance for fanvids. QueerTube (obviously we'd need to run our own).
https://queertube.org/video-channels/meedee_channel/videos

My question: since many of us have band-width caps (ex I am limited to 1TB up and down a month), what type of bandwidth load would someone need if they are considering hosting a server?

[impertinence]

I was looking at getting Peertube set up this weekend! Will for sure link you if I do.

Regarding bandwidth - it very much depends on content and storage backend. I have no idea what p2p would look like as far as home bandwidth, my guess would be you'd have to limit streaming quite a bit. For a VPS host like Linode, you get 4TB transfer for $20/month, and that would cover quite a lot of I/O even for audio or video. Linode, on top of that, tends to assume you're using external storage like s3 or wasabi - hence the lower ssd storage (80gb for the $20/month, much less for the $5/$10 tiers). For contrast, I paid about $3 to s3 over 2 months of testing file transers of 200mb files. And on top of that, peertube claims to limit your bandwidth compared to straight streaming, as you know - but I'm not sure if they support external storage for the server's original copies of the videos, which would make a larger difference in price and how much bandwidth the non-storage server consumes.

Personally, while I think p2p is longterm the way to go, I'd support federated nextcloud instances with hubzilla as the video frontend if home bandwidth is a concern. The reason for that is you can then have nextcloud as the server actually serving the video - and so that bandwidth is offloaded to a VPS - but since nextcloud is a file sync service you have a local data backup too in case of TOS/server dying/etc.

[cassini]

[here via fortutious tumblr reblog

fascinating stuff, not practical to me right now but surely very useful to the future of fandom beyond users like me. i'm sure i'll keep my eye on this post and its info ]

[impertinence]

yes! right now I'm just trying to raise awareness that this tech exists. :)

[ineptshieldmaid]

I barely understand this but I like the cut of its gib.

[impertinence]

:D More explanations (hopefully with video!) are coming!

[hushpiper]

Having skimmed and having no firsthand experience with any federated software, it sounds to me like people are really not grasping how great the forward-compatibility is. Some kind of shiny new software came up and all your friends are moving to it? No problem! You can stay in touch with them without ever having to move your own account or data, so long as both platforms support ActivityPub. Less like switching from Tumblr to Twitter, more like switching web browsers.

I'm still curious about a lot of the specifics (how does it handle deduplication of posts over platforms? how easy is it to move your data in case of things going south with one instance/platform? do you have to choose a "home" platform/community, and how do you do that? I'm sure I'll think of others). But the fact that as of January, ActivityPub is a W3C recommended standard, is very encouraging.

(Also, I have long thought that fandom really needs to cozy up to the old school anti-authoritarian privacy-focused parts of the open source community, since they share so many of the same needs and ideals, so I'd love to see that happen.)

[impertinence]

I definitely think it's kind of - a set of new or less well known concepts that people are understandably wary of! But yeah, ActivityPub really seems like it has a ton of momentum and the ability to use different platforms with different strengths & weaknesses is REALLY nice.

As far as your questions - I guess we'll find out! :D

(And yes - we are 1000000% in agreement, there.)

[lovetincture]

This gave me a lot of new things to chew on. Thank you for the brilliant post!

[impertinence]

thank you so much for reading it (I'm late D:)

Fascinating post.

[jesse_the_k]

Thanks for breaking it down enough for a non-coder to comprehend.

I've created jessethek@fandom.stopthatimp.net because I could.

(I also created another account just so I could delete it. Now to figure out how to do that.)

Re: Fascinating post.

[impertinence]

Thank you for reading! Feel free to experiment at will. :D