Tag: pleroma

By Posted on Posted in Tech 2 Comments Tagged with , , , ,

I’ve been running my own node on the fediverse for a little while now: pleroma.gidikroon.eu. That is mainly a hobby, but partly to learn from it as well. I had not done sysadmin before or set up my own server. This WordPress blog is one of those one-click affairs. I also believe that part of the promise of the fediverse is to be able to self-host your social media presence while staying in contact with others on other servers; I wanted to experience how that would work out.

If a server is only for personal use, you don’t need much resources. Some people install the software on a PC at home that they leave turned on. Others use a Raspberry Pi, a very cheap single board computer that you can get for under €50,-. Cheap cloud providers like Digital Ocean and Hetzner are popular too. There are also services that can run your fediverse server for you, like masto.host and spacebear.

I wanted to try using Amazon Web Services (AWS). The by far easiest setup as well as the cheapest (within AWS) is using Amazon Lightsail. The smallest server option is only $3,50 per month with everything included. If you calculate doing the same thing on Amazon EC2 (having a t3a.nano instance running full time with block storage, network traffic and an ip address) it is more expensive and you have to setup everything yourself. Lightsail (and Digital Ocean and Hetzner and others) are really easy and cheap for people who don’t want to spend a lot of effort in self-hosting something.

On this smallest server option I have run at the same time:

  • My Pleroma server (an ActivityPub server)
  • The PostgreSQL database
  • Another development Pleroma server (to test stuff)
  • A relay that receives and sends posts between fediverse servers

The Pleroma server was also subscribed to one of the busiest relays, so posts came in at a rate of about three per second, around the clock. I had however stopped the second Pleroma server a while ago.

So what are some of the things I learned so far? The main thing relates to cloud servers being ‘burstable’. This means that they can only run at full speed for short periods of time (up to around half an hour), after which the speed gets reduced to 5% of full capacity. Which is perfect for personal servers that only get visited occasionally. Not so much for a server that gets constant traffic 24/7.

Amazon is clear that you don’t get full capacity. In fact these are recent metrics for my server where they clearly indicate a very low sustainable zone:

At the end of the graph you can also see that after my server has been above the sustainable zone for over a couple of hours, it has now been put into what I call cpu jail: the server will not do more than 5% of its full capacity. The problem is that other servers, relays, etc keep talking to it and when their connections fail, they will just retry later. The load been placed on the server will not let up and so this cpu jail situation will not resolve itself.

(BTW you can see that my approach to sysadmin is very low-effort, since I’m continuing to type this blog post while the above graph is a real current picture of the server that’s locked up. It needs to be a hobby after all and I’ll look into it when I feel like it.)

In the graph you can also see that early in the morning the server had just crashed after a similar period in cpu jail.

What to do about it?

For now I have stopped the relay server and unsubscribed from the other relay. This should mean that (after a while) the only social media posts coming in will be of people I actually follow. This takes a while to take effect however, as most servers are still retrying their earlier failed deliveries.

I have also looked into the configuration of the software. By default it is configured for medium sized servers that have dedicated resources. These have multiple cpu’s that work in parallel. The software uses that by queuing background jobs and having several worker threads in parallel dealing with the queued jobs. The standard configuration has about five different queues with up to 25 parallel workers each. That is not suitable for a cheap burstable cloud server.

Remember that I get only 5% of one cpu. Not 25 full time cpu’s…

So I changed the configuration to have only one worker per queue. That is still too much, but I can’t configure it lower. I have also reduced the amount of parallel connections to the database (which was 10), for the same reasons.

The database. That’s the next thing.

First the easy thing: with the cheapest server you get 20GB of storage, which is more than enough to keep your database on. However, since I subscribed to a relay and kept the posts for three months I essentially had everything published on the whole fediverse over the last few months in my database, which was about 15GB. It turns out that PostgreSQL always needs to have space for double the size of the database for its management operations. So I attached an extra disk within Lightsail for all data, so it would also be easier to move between instances. Costs some extra, but makes stuff easier.

A while ago I decided to upgrade the database from version 11 to 12. Seemed easy enough, since there is a command in the Debian operating system that does everything for you:

pg_upgradecluster 11 main

Doing this it will automatically create a new version 12 cluster and use the ‘dump’ method (also used for backups) to stream data out of the old cluster straight into the new cluster. Sounds fine.

But the ‘burstable’ concept also applies to reading and writing data on a disk. Again, Amazon is clear about this and says that you can’t really load more than 10GB of data into a database on the smallest server options.

So after this had been reading/writing with about 100MB/s for a while, the speed dropped to about 100kB/s. I left it running for over 24 hours, but in the end the process just crashed. The ‘pg_upgradecluster’ nicely rolled back all changes and everything was operational again, but still on version 11.

Reading up on things I found that I should have added an extra option:

pg_upgradecluster --method upgrade 11 main

With this extra option, it would not use the ‘dump’ method, but an ‘upgrade’ method to change the data from the version 11 format to the version 12 format. With this, the process was done, successfully, in five minutes, rather than unsuccessfully in 24 hours.

I realize however that if I ever need to restore the database from a backup, which necessarily uses the ‘dump’ method, I will run into the same problems.

It seems that theoretically you can run the Pleroma software on the cheapest cloud server option, but there are limits to what it can do. I am glad however that I used Pleroma and not Mastodon, since it requires much more resources to run and is not really suitable for self-hosting.

The learning continues.

By Posted on Posted in Tech No Comments Tagged with , , , , ,

I was interested to find out how the different pieces of software that make up the fediverse connect together so I tried to investigate. I got confused about all the different cases and exceptions, so I wanted to write it down in a post. I’ll keep updating this post when I learn more.

Specs

The relevant specs are ActivityPub, ActivityStreams (AS2) and the AS2 Vocabulary.

ActivityStreams and its Vocabulary define the elements that make up posts, timelines, followers and likes, the kind of stuff we know from social media. ActivityPub defines how these messages can be exchanged between servers. This creates a decentralised web of servers and services where users can follow and message each other. This is the fediverse.

Imagine being able to use your Twitter account to follow someone’s Instagram account and reply with praise about their photos. And imagine your sister setting up her own Twitter server and giving you an account on that. The fediverse is like that. Except of course Twitter and Instagram don’t support that, they want to keep you within their systems because they need your clicks to make money. Instead most software in the fediverse is open source software.

Mastodon

Mastodon (see https://joinmastodon.org/) is a Twitter like service where you post short messages, follow other people’s messages and reply to them.

The posts are modelled as objects with type Note. The text is in the attribute content (and/or contentMap?) and is assumed to be plain text or html. The summary attribute has a special meaning: it is not shown as a short excerpt of the post followed by a ‘read more’ link, but as a content warning or trigger warning, followed by a link to read anyway; the result is the same.

If the content contains html, it is completely stripped and presented as plain text. Except hyperlinks, which remain functional even though there’s no way in the Mastodon UI itself to create hyperlinks.

If a post of type Article comes in, it is converted to a Note and treated as such.

When someone posts a message, it is sent to other servers as a Create activity of the Note object. If someone likes a post, it is sent as a Like activity. If someone shares a message (reposts, reblogs, retweets, boosts) it is sent as an Announce activity. Like and Announce can be undone by sending the Undo activity. The Delete activity of posts is supported. The Update activity for posts is not.

Media (photo’s, video’s, audio files) are accepted as content of the attachments attribute of the Note object. Such an Image object contains a Link object. Mastodon will retrieve the media from the remote server and store it internally. This media will be kept. Mastodon allows either four images, or one video or one audio file.

Direct media objects (e.g. a Create activity with an Image object) are not supported.

Following someone is done by sending a Follow activity to the server of the person you want to follow, with the Actor object as the object of the activity. That server responds with an Accept or Reject activity (with the Follow activity as object) either immediately, or after the followed user manually accepted/rejected the request. You stop following someone by sending an Undo activity with the original Follow as object. The other server can stop the follow relationship as well by sending a Reject (with the Follow as object) even after first accepting it.

Note that because Mastodon doesn’t store the original activities, it will not include the original Follow activity in any Reject or Undo, but create a fake new one. In that Mastodon is not spec compliant.

If a user blocks another user, Mastodon sends a Block activity (with the user’s Actor as object) to inform the other server (and possibly that user) that they are blocked and will they please stop reading these messages.

If a post contains text that looks like a web address, the UI tries to fetch and show a preview of that page when showing the post. This preview is not part of the post as created and sent to other servers.

(to check: polls)

Glitch-soc

Glitch-soc (see https://glitch-soc.github.io/docs/) is a fork of Mastodon and it does basically the same thing, but a little more. E.g. glitch-soc is able to receive rich text (as html in the content attribute) and show it to the users.

Pleroma

Pleroma (see https://pleroma.social/) is also a Twitter like service. It interacts with other servers in much the same way as Mastodon, with some exceptions.

Pleroma tries to be more spec compliant. It stores ActivityStream objects internally, so if the spec says to include the original activity, e.g. in an Undo, it does so.

Pleroma accepts rich text as html content and only strips dangerous tags. It also provides users with a UI to enter rich text in several formats like html, markdown, bbcode, but they are all converted to html. Users can also enter hyperlinks.

It accepts Article posts without the need to convert them.

Media are also modelled the same as Mastodon does (as Image, Video, etc, objects in the attachment attribute which has Link objects pointing to the original), and retrieves the content from the remote server. These remote media contents are however only cached temporarily and refetched if needed.

Direct media objects (e.g. a Create activity with an Image object) are not supported.

To check: is audio supported?

Pleroma also doesn’t send Block activities to the affected servers or users. Blocking is handled internally by making sure the blocking user doesn’t see posts or notifications by the blocked user.

If a post contains text that looks like a web address, the UI tries to fetch and show a preview of that page when showing the post. This preview is not part of the post as created and sent to other servers.

(to check: polls)

Pleroma is also the only implementation I know of that supports the Client to Server part of the ActivityPub spec.

Friendica

Friendica (see https://friendi.ca/) can do everything. It tries to connect to all other services and be the spider in the web. At its heart it is a Facebook alternative, where by default accounts are set to need a mutual connection (so a manual Accept to a Follow request), though a user can set their account to soapbox-style so Follow requests are accepted automatically.

It also has concepts like Events, Photo Albums, etc, though I don’t know how these are communicated over ActivityPub, if at all.

Friendica seems to accept any ActivityPub message, even though it can not always display it right. E.g. a direct image post (a Create activity with an Image as object) is accepted, but the image doesn’t show.

It tries to distinguish between flavours of ActivityPub, e.g. it knows when it’s talking to a Pleroma server or a Mastodon server.

Friendica also connects over the OStatus protocol, with Diaspora, and integates with Twitter, Tumblr, WordPress, etc.

The post editor allows posts to be created in bbcode. If a photo is inserted, the editor creates a post with a hyperlink to the original and a smaller preview as attachment. This is how the post is sent to other servers. This differs from the behaviour of Mastodon and Pleroma, which just attach (a reference to) the image and leave stuff like previews to the UI of the system the receiving user is using.

Friendica also supports groups in the form of Forums, that can be addressed by using “!” instead of “@” in a mention. The forums can be located on a different server than the user’s server, but the forum server will still be able to authenticate the user as a member of the forum because Friendica supports OpenWebAuth.

Pixelfed

Pixelfed (see https://pixelfed.org/) is a photo sharing application like Instagram.

It supports communication with Mastodon, Pleroma, etc over ActivityPub. If a user makes a post of a photo (or multiple photos) a Create activity with a Note object is sent to that user’s followers. The Note object contains the caption in the content (or contentMap) attribute. The attachments attribute contains a list of Image objects, one for each photo. Each contains a Link object that refers to the photo on the originating server. (to do: check) Similar for a video post.

This is also the way Pixelfed expects to receive posts (and the way Mastodon sends them). If a Note object is received as part of a Create activity, its attachments are checked for video or image content. If it is just a text post, or has just an audio attachment, it is not received. If it is a direct media post (a Create activity with a Image object) it is not received as there is currently no other software that sends such messages reliably.

If the object is of type Article it is also not received, even if it has media attachments.

PeerTube

PeerTube (see https://joinpeertube.org/) is a video sharing platform like Youtube. Users can have multiple channels in which they can post videos. Within the platform users can subscribe to (follow) specific channels of other users, they can’t follow the user’s account itself.

PeerTube accounts and channels can be followed over ActivityPub from Mastodon, Pleroma, etc. Which gives the weird situation that PeerTube accounts (rather than channels) can be followed by non-PeerTube accounts only.

Non-PeerTube accounts can comment on (reply to) video’s by normal Note messages. Like messages are also supported, as are Dislike messages. The latter seem only the available to Peertube accounts within the platform.

If a user posts a video, after it has been processed by the server, a Create activity is sent with a Video object. This is different from the other ActivityPub software which sends media as attachments to Note or Article objects.

Other PeerTube servers will only receive a Create activity with a Video object if it conforms to strict criteria. E.g. the content attribute which contains the caption needs to be encoded in markdown (not html). Also the url needs to be a webtorrent link.

If someone viewed a video (or the majority of it) a View activity is sent to indicate the video has been seen. The server hosting the video can update the view count.

WordPress

A WordPress blog can be attached to the fediverse by installing the ActivityPub plugin. It can also cooperate with other IndieWeb sites using the relevant IndieWeb plugins. These plugins are compatible.

There’s another plugin that attaches WordPress to the fediverse: Pterotype. I don’t have experience with that one. It is not fully compatible with the Indieweb plugins.

When a WordPress post or page is published, a Create activity is sent to the blog’s followers on the fediverse. The object in the Create activity is either a Note or an Article or any other type like Image, Video, depending on user settings. It is best to keep this setting at Note for compatibility with e.g. Pixelfed.

Delete is not yet supported as an activity to send. Update neither (to do: check).

The WordPress ActivityPub plugin supports received Follow activities (will automatically send an Accept) and replies in the form of Notes.

Funkwhale

Funkwhale is an audio sharing platform like Soundcloud. I have no experience with it. I believe it is using a part of the ActivityStreams vocabulary which is not supported by the other software.

SocialHome

Will soon support ActivityPub. Besides that I think it supports the Diaspora protocol (to do: check).

Plume

To do

Writefreely

To do

Misskey

To do

Osada

Supports both ActivityPub and Zot as protocols.

Hubzilla

Only supports ActivityPub if the admin has installed it as plugin. The demonstration instance for new users does not have it installed.

See https://zotlabs.org/page/hubzilla/hubzilla-project.

Hubzilla (as a Zot implementation I think, to do: check) supports nomadic identities. This means that a user account is not linked to a single server (Hub), but can be used on multiple hubs. This solves the problem of servers going out of service and users moving home.

Zap

See https://zotlabs.com/zap/. Supports Zot as protocol, not ActivityPub. Is migrating to a new version of Zot which will make it incompatible with Osada and other Zot implementations.

Diaspora

Does their own thing and uses only its own protocol. I’ve been told it connects with Friendica and Hubzilla.

Relays

In Mastodon and several other services, messages are only delivered to a server if there is someone on that server following that author. Even if the message is a publically visible message. This means that the network timeline of public messages can remain quite empty especially for new or small servers.

Mastodon has the ability for one server to follow another server. This is not commonly done. Peertube does have the same ability, though here it is much more common as I understand.

Pleroma has relay functionality built in in the software, where the administrator can subscribe to a relay. This relay is separate software (Litepub Relay, see https://git.pleroma.social/pleroma/relay) which is provided by Pleroma and which creates Announce (reblogs) activities for each Create activity it receives. Mastodon can follow such a relay as well.

I think Friendica also has such a separate relay software. (to do: check)

Diaspora seems to have separate relay software where servers (pods in their terminology) can subscribe to hashtags instead of the whole feed. To do: check.

Profile directories

Mastodon has an optional, opt-in, profile directory with the scope of the local server. There is no global directory.

Friendica has a Site Directory (for the local server) as well as a Global Directory as a separate but decentralised application. Both are opt-in only. See https://dir.friendica.social/.