Lemmy: Federated Alternative to Reddit in Rust(github.com)
github.com
Lemmy: Federated Alternative to Reddit in Rust
https://github.com/dessalines/lemmy
107 comments
One suggestion: if you want the federated crowd to jump on board, then choose another font provider than Google, which uses their font links for tracking.
Thanks for pointing this out. I guess this is something I had assumed Google does by serving the fonts for free, but I never really bothered to look into it. Going to remove all Google Fonts references from my projects now.
Google claims they don't track with fonts: https://developers.google.com/fonts/faq#what_does_using_the_...
Whether you believe them or not is up to you of course.
Whether you believe them or not is up to you of course.
Where does it claims that it does not track?
From your link:
Google Fonts logs records of the CSS and the font file requests, and access to this data is kept secure. Aggregate usage numbers track how popular font families are, and are published on our analytics page. We use data from Google’s web crawler to detect which websites use Google fonts. This data is published and accessible in the Google Fonts BigQuery database. To learn more about the information Google collects and how it is used and secured, see Google's Privacy Policy.
From your link:
Google Fonts logs records of the CSS and the font file requests, and access to this data is kept secure. Aggregate usage numbers track how popular font families are, and are published on our analytics page. We use data from Google’s web crawler to detect which websites use Google fonts. This data is published and accessible in the Google Fonts BigQuery database. To learn more about the information Google collects and how it is used and secured, see Google's Privacy Policy.
Just before the part you quoted has this:
"Use of Google Fonts is unauthenticated. No cookies are sent by website visitors to the Google Fonts API. Requests to the Google Fonts API are made to resource-specific domains, such as fonts.googleapis.com or fonts.gstatic.com, so that your requests for fonts are separate from and do not contain any credentials you send to google.com while using other Google services that are authenticated, such as Gmail."
That's basically saying Font download is not being associated with any individual. And the part you quoted says they keep the aggregate stats, which is reasonable, especially given that the aggregated stats are available publicly.
"Use of Google Fonts is unauthenticated. No cookies are sent by website visitors to the Google Fonts API. Requests to the Google Fonts API are made to resource-specific domains, such as fonts.googleapis.com or fonts.gstatic.com, so that your requests for fonts are separate from and do not contain any credentials you send to google.com while using other Google services that are authenticated, such as Gmail."
That's basically saying Font download is not being associated with any individual. And the part you quoted says they keep the aggregate stats, which is reasonable, especially given that the aggregated stats are available publicly.
Thanks for the clarification. However I'm not sure that means they don't track. They specify some of the things that they do with the data, but the last sentence "To learn more about the information Google collects and how it is used and secured, see Google's Privacy Policy" means that they could also use any piece of data to improve their products.
Keep in mind you can download the fonts and host locally. You don't have to let Google host.
Would one run into licensing issues with this? I'm honestly not sure if their usage is predicated on being served from Google, nor knowledgeable enough about the law to even guess.
Interesting question. I never thought of this and I don't have the answer.
Source?
Serious question: from a user perspective, isn't spam going to be insanely bad in the Fediverse? How do you stop spam without a central authority?
As someone running a federated server (mastodon); you get more problems with local spam than with foreign spam.
Most instances will quickly shut down any spam, so it doesn't really fill the federated timeline that much. Local accounts are something you as an administrator need to act on, so they are the annoying ones.
In my experience, instances that allow spam are rare and the get quickly blacklisted by everyone from showing up in the timelines (there is luckily a neat middleground between blocking the instance entirely and not doing that; allow local users to subscribe to that instance, which is my default action for all instances that aren't hosting illegal material for me).
Most instances will quickly shut down any spam, so it doesn't really fill the federated timeline that much. Local accounts are something you as an administrator need to act on, so they are the annoying ones.
In my experience, instances that allow spam are rare and the get quickly blacklisted by everyone from showing up in the timelines (there is luckily a neat middleground between blocking the instance entirely and not doing that; allow local users to subscribe to that instance, which is my default action for all instances that aren't hosting illegal material for me).
It doesn't look like that might scale at all. In other words, it works because Mastodon is a very small, niche community.
Good thing is that the Mastodon instance owners use shared block lists for servers that are currently used to ban free speech instances; that infrastructure could also be used to create a collectively managed list of instances that are full of spam.
Good thing is that the Mastodon instance owners use shared block lists for servers that are currently used to ban free speech instances; that infrastructure could also be used to create a collectively managed list of instances that are full of spam.
Atm shared block lists are still manual, but I'm hoping they'll become automated in the future...
[deleted]
Similar to an email service. You rely on the node operators to manage it.
So delegate this all to a few large blacklists (spamhaus)? And block all residential users by default?
Generally not, proactive blocking or silencing is reserved for instances that host illicit content like child porn or hate speech. Inter-moderator communication happens between your instance admin/democracy (or whatever structure your instance uses) and other instances.
Traditional tools like muting a user, silencing an instance (if they're spamming up the Federated Timeline on other instances, like Humblr.social does) and blocking (if the mods are total shit) can still be used. Small instances are the bulk of the content, so you aren't about to block them. Large instances like Mastodon.social are generally blocked tho (if your admin/democracy cares about a vibrant Fediverse).
Traditional tools like muting a user, silencing an instance (if they're spamming up the Federated Timeline on other instances, like Humblr.social does) and blocking (if the mods are total shit) can still be used. Small instances are the bulk of the content, so you aren't about to block them. Large instances like Mastodon.social are generally blocked tho (if your admin/democracy cares about a vibrant Fediverse).
> Large instances like Mastodon.social are generally blocked tho (if your admin/democracy cares about a vibrant Fediverse).
Ah, so you achieve vibrancy by .. blocking a large number of the users?
Ah, so you achieve vibrancy by .. blocking a large number of the users?
Honestly, I think this is the wrong way of looking at things. The Fediverse isn't really like a single social network that's shattered amongst many instances ... it's more like a bunch of individual, separate hangouts, in the vein of classic forums and IRC chatrooms and fandom websites, which have the ability to talk to each other and share stuff if they want.
It's not so much about "blocking" mastodon.social as not choosing to link up with it - because your instance is a specific community, and actual human social relations thrive when they can choose who they're hanging out with. So you choose to talk to other smaller, more diverse communities that you can actually get to know a good chunk of the people in them, instead of just drinking straight from the firehose that is a big instance.
Federating with another instance is basically two groups of people deciding to hang out together, effectively merging. Federating with mastodon.social means, effectively, merging your community with mastodon.social ... which means submerging it beneath the mass of mastodon.social's much greater activity.
A vibrant Mastodon isn't a more-obtuse recreation of the Twitter experience. Twitter is already Twitter. It's about fostering all the things that Twitter isn't - personal and Dunbar-scaled and locally moderated and diverse.
It's not so much about "blocking" mastodon.social as not choosing to link up with it - because your instance is a specific community, and actual human social relations thrive when they can choose who they're hanging out with. So you choose to talk to other smaller, more diverse communities that you can actually get to know a good chunk of the people in them, instead of just drinking straight from the firehose that is a big instance.
Federating with another instance is basically two groups of people deciding to hang out together, effectively merging. Federating with mastodon.social means, effectively, merging your community with mastodon.social ... which means submerging it beneath the mass of mastodon.social's much greater activity.
A vibrant Mastodon isn't a more-obtuse recreation of the Twitter experience. Twitter is already Twitter. It's about fostering all the things that Twitter isn't - personal and Dunbar-scaled and locally moderated and diverse.
I think it's sad that we cannot figure out a way to have LARGE federated communities that remain functional.
I think this is inherently a sociological problem than a technological one. When there are enough people in an organically grown group, the community will become unsustainable and then 1) it disintegrates into multiple groups until it becomes stable again or 2) it transforms into a hierarchial, authoritative society where a small group of people makes most of the decisions.
Also, I think Mastodon’s design deliberately steers instances away from becoming large ones; it rather tries to be a nurturer for various small communities and subcultures. It doesn’t view size as a good thing, and that’s totally okay. Nowadays, I think we need less monoculture and more diversity/creativity in our society.
Also, I think Mastodon’s design deliberately steers instances away from becoming large ones; it rather tries to be a nurturer for various small communities and subcultures. It doesn’t view size as a good thing, and that’s totally okay. Nowadays, I think we need less monoculture and more diversity/creativity in our society.
This sounds awesome. How do I find these smaller and more diverse communities?
[deleted]
That sounds like it only works because spammers aren't really trying yet.
It'll probably end up "solved" in a similar way to email span being "solved" - if global spam starts to become a problem, you'll have big blocklists of spammy instances and general suspicion towards federating with any new instance that doesn't look sufficiently trustworthy.
Malicious actors definitely try to spam and data mine the fediverse, but their accounts get disabled by the local instance admin, and when they try evading that by spinning up a few instances on their own domains to hide their spam or follow bots behind, those instances generally get blocked by instances that see said spam toots or spam followers.
I don't think mastodon.social is generally blocked, atleast not the instances I federate with block it.
Might be your particular corner of the fediverse, I prefer to have as much reach as possible.
Might be your particular corner of the fediverse, I prefer to have as much reach as possible.
Yeah, and look how well it works in the email world.
There can also be spam servers that publish blacklists like with emails
Might be offset by way fewer ads, influencers and platform-driven notifications. I find that those things are the main sources of spam on say Twitter or Instragram.
Federated bayesian filters or local machine learning ?
You could have a personal (stored on device) Bayesian filter that learns from your preferences, and a shared one to block spam spotted by your contacts
As other people suggested, I don't think it should be at the node operator level, for freedom of speech concerns
You could have a personal (stored on device) Bayesian filter that learns from your preferences, and a shared one to block spam spotted by your contacts
As other people suggested, I don't think it should be at the node operator level, for freedom of speech concerns
Don't know why you are being downvoted. Thunderbird was marketed as having a bayesian filter a decade or more ago.
Looks like it uses ActivityPub - https://github.com/dessalines/lemmy/blob/master/docs/API.md
Nice.
Nice.
Mastodon has been responsible for helping drive a lot of growth in the federated app space. PeerTube for videos, and perhaps a dozen other services (see https://blog.joinmastodon.org/2018/06/why-activitypub-is-the... for some of them).
I also recently found the early but promising Plume project for operating a federated blog ( https://joinplu.me/ ).
Icing on the cake? Quite a few of the federated projects are written in Rust.
I also recently found the early but promising Plume project for operating a federated blog ( https://joinplu.me/ ).
Icing on the cake? Quite a few of the federated projects are written in Rust.
Federated blog?
Is that any different from a regular website?
Is that any different from a regular website?
I would assume it is just a regular website with some backend for federated discovery.
I don't read many blogs but I would assume having tags or a "more like this" that search across the fediverse could be useful?
Are those things that people have in blogs?
I don't read many blogs but I would assume having tags or a "more like this" that search across the fediverse could be useful?
Are those things that people have in blogs?
Also the whole commenting/liking thing
It doesn't look like there's any federation right now, the test site (https://dev.lemmy.ml/) says "Federation into the ActivityPub network is on the roadmap."
I have also reddit/HN clone running on https://esourcenews.com with code on github (written in python - Flask https://github.com/matoous/newz). Although this one is not federated I added tools such as automatic RSS feed import.
Oh. Activity pub is complex and difficult to implement. The test suite is down.
Leaving it to last makes me worry that it won’t happen.
Leaving it to last makes me worry that it won’t happen.
I'm not convinced by ActivityPub. A lot of people seem to think that it's cool but the documentation makes it look like it's just a schema for social media applications. It might be an obvious move to try to standardize social media platforms because they all involve roughly the same kinds of user activities. Interoperability is cool, I agree. But the ActivityPub documentation has a whiff of the old XML/"semantic web" hype about it. It's verbose, unnecessarily object-oriented, and it doesn't seem to advance any kind of new or interesting vision for the architecture of these interconnected systems.
I get that somebody probably needed to do this work of standardization. As a project it seems to lack imagination, though. There is nothing to get excited about as far as I can see. It's just a consolidation of very familiar social media functions.
I get that somebody probably needed to do this work of standardization. As a project it seems to lack imagination, though. There is nothing to get excited about as far as I can see. It's just a consolidation of very familiar social media functions.
Check out https://gitlab.com/mbajur/prismo/.
Or https://github.com/mariusor/littr.go - which is my project.
Not federated yet either unfortunately. But I think one of the few fediverse projects that dogfoods ActivityPub by using the client to server protocol to communicate between frontend and api.
Not federated yet either unfortunately. But I think one of the few fediverse projects that dogfoods ActivityPub by using the client to server protocol to communicate between frontend and api.
sharing some others in case it helps
- https://github.com/mariusor/littr.go
- https://gitlab.com/tuxether/anancus
- https://github.com/mariusor/littr.go
- https://gitlab.com/tuxether/anancus
Another, decentralized, project https://getaether.net/
Yup, creator of Aether here. Small difference, this is federated, Aether is decentralised. Far as I understand this is more like Mastodon where some people host servers that talk to each other. Aether is more like a decentralised Usenet, every user is a server.
Can you explain the 6 month thing to me? If I understand correctly, posts/comments older than 6 months get deleted, unless you decide to save it locally? Even if someone saves a post older than 6 months locally, others cannot see it, right?
That does not seem like a feature to me, and completely breaks geteather imo, sadly. Most of the good content on reddit is older than 6 months. Imagine if reddit decided to delete posts older than 6 months, jesus, every 6 months there would be a flood of the very same questions/posts/comments...
That does not seem like a feature to me, and completely breaks geteather imo, sadly. Most of the good content on reddit is older than 6 months. Imagine if reddit decided to delete posts older than 6 months, jesus, every 6 months there would be a flood of the very same questions/posts/comments...
It's in keeping with the project's philosophy of being ephemeral.
There are other tools for recording information for posterity.
There are other tools for recording information for posterity.
Excellent answer.
[deleted]
Love the processes in this aether project with the voting for mods and such. Wish there was an easy way to clone it and create a subspace aether system that was not attached to the others. I have some user groups that would use this kind of thing - but they would quickly try to game the mod voting thing in many ways, wreck all kinds of havoc trying to get things blocked / banned and ruined - and so we'd need a function for a high level set of super mods and a way to prevent the coming toxic soup from running over into the main project.
Took a look at the open source page on there and there is a bunch of stuff, it's impressive, but too complex for me to try to piece together. Perhaps something like riot matrix is more like we need, and to find some way to incorporate the mod voting kind of system as some kind of addin to a self hosted instance of matrix kind of thing.
This is actually something that I'm working on, if not the thing I'm working on. :)
Where the hell is the source-code? Can't find it anywhere linked on the website..
It's here: https://github.com/nehbit/aether
The quality of this seems high, from the frontend on the dev instance. All actions are effectively instantaneous.
Good work!
Good work!
Very nice! I am curious how this would compare to Discourse feature and usability wise.
It would also nice to have a NNTP gateway to these webforums. VBulletin had one.
It would also nice to have a NNTP gateway to these webforums. VBulletin had one.
off topic but ~20 years ago Lemmy was a beautiful,native vi clone for Windows. I wish the author had kept it going or made it open source. Vim has such an ugly GUI.
[deleted]
I think a project like Diaspora failed because they chose to write it in Ruby on Rails. Ruby is nice from a purist point of view. But you'd want to build something like this on top of a language which is popular, accessible, has a low learning curve and can be easily deployed anywhere.
Nice project!
(I'm also doing a Fediverse project -- MeowCat https://github.com/cabalamat/meowcat2)
(I'm also doing a Fediverse project -- MeowCat https://github.com/cabalamat/meowcat2)
Nice!
What I haven't found in the docs is how federation would work.
What I haven't found in the docs is how federation would work.
Via ActivityPub — see https://github.com/dessalines/lemmy/blob/master/docs/API.md.
For more general info on ActivityPub itself, see https://activitypub.rocks.
For more general info on ActivityPub itself, see https://activitypub.rocks.
Is there a rust implementation of mastodon? The dependencies for the current implementations are too brittle IMHO.
Perhaps rustodon: https://github.com/rustodon/rustodon
There's a fairly popular project written in Elixir called Pleroma. It's not Rust, but it's a lot faster and lighter than Ruby.
There's honk https://flak.tedunangst.com/post/honk-preview (not rust, but go)
Wow, this is awesome.
It works really well and looks great.
Very nice work!
Very nice work!
Your site is blank without JS...
Nice! I hope this gets traction.
[deleted]
[deleted]
I don't understand what this is. At first, I thought it was a scraper for links which is the only reason I go to reddit at all. Such a thing would help stop me from getting sucked into the 80% of reddit which is nothing but a smelly pile of poo.
So what does this do?
So what does this do?
This is a reimplementation of a reddit-like site. But open source, and with ambitions to work in a federated (as opposed to centralised) way.
You could use this code to host your own Reddit (or you will be able to once they project is more developed).
You could use this code to host your own Reddit (or you will be able to once they project is more developed).
The code quality is amazing. Good job.
Given the response to this, i would be interested to hear why you thought the code quality was amazing. We seem to have two people with different ideas about code quality, and perhaps we would all learn something by investigating that difference.
I was mainly commenting on the frontend as I'm not well-versed in Rust, my dissenter seems to have a problem with the backend and the organization of the project, which seems unfair since it's in it's infancy, you can always restructure a project and reorganize code. I was commenting on the code, not the files.
As for the frontend, it's all typed, and very minimal. Very rarely do we see frontend code that does just what it needs to do and nothing else. The fact that they achieved a "React-like" (Inferno) environment and managed to keep the frontend under 80KB is a feat.
That's not to say it's perfect, but it's not finished. I was just very surprised to be able to jump right into the code and see what was going on without wading through boilerplate.
As for the frontend, it's all typed, and very minimal. Very rarely do we see frontend code that does just what it needs to do and nothing else. The fact that they achieved a "React-like" (Inferno) environment and managed to keep the frontend under 80KB is a feat.
That's not to say it's perfect, but it's not finished. I was just very surprised to be able to jump right into the code and see what was going on without wading through boilerplate.
I'm not sure why my compliments are being downvoted. Do ya'll hate the code that much?
Forget it Jake, it's Hacker News.
On the contrary, I thought the code was terrible quality. If someone wrote this code at my company they would be fired, or more likely not hired in the first place.
That's a terrible, sad and rude comment that does not belong in HN. Instead of just criticize you could offer a constructive review. I hope English is not your native language and it has a less aggressive tone in the translation.
What makes you say so? Can you give examples?
Well, for a start... having a single file called "server.rs" which is 2256 lines long is already disqualifying. This file, code structure would be incredibly poor for developer efficiency. It just grows and grows and does all the things while having a vague sounding name like "server.rs" Why is server.rs doing "EditSite." What does the name EditSite mean anyway?
Poor code reuse, same bits of code copy and pasted all over instead of being refactored out into their own methods. Methods which are doing too much different crap. The single responsibility principle is broken everywhere.
Imagine trying to write unit tests for "Perform for EditPost", this method is doing far too much and would be a pain in the ass to fully test. If you're making a method called "EditPost" it should only be doing the actual post editing, not making connections, or user validation or checking for a ban. These should happen in a method which only handles connections, or a method which is called "ValidateUser" which is properly re-used.
Generally speaking, the structure as a whole isn't very readable, poor names, methods and files which are doing too much. The project wouldn't scale well with a bigger team or if it grew in terms of codebase size.
Poor code reuse, same bits of code copy and pasted all over instead of being refactored out into their own methods. Methods which are doing too much different crap. The single responsibility principle is broken everywhere.
Imagine trying to write unit tests for "Perform for EditPost", this method is doing far too much and would be a pain in the ass to fully test. If you're making a method called "EditPost" it should only be doing the actual post editing, not making connections, or user validation or checking for a ban. These should happen in a method which only handles connections, or a method which is called "ValidateUser" which is properly re-used.
Generally speaking, the structure as a whole isn't very readable, poor names, methods and files which are doing too much. The project wouldn't scale well with a bigger team or if it grew in terms of codebase size.
What is the max lines in a single file for a developer "to get fired" at your company?
The company has a shell script running in Cron. If it finds a source file with more than 1000 lines it deletes the author from the payroll.
Sadly this cron has fired the shell script author, and no one dares to touch it.
No, it is written in APL for extra conciseness. When they wrote it in shell it deleted the authors from the payroll.
The author is entire team.
I do have to say, 2k lines for a single file is scary sounding to me. It would probably make me consider splitting it up.
BUT, sometimes things to just logically belong together and having one file to find them is just easier.
It's a rediculous metric to use.
BUT, sometimes things to just logically belong together and having one file to find them is just easier.
It's a rediculous metric to use.
Depends a lot on tooling as well. If you have tools that give you effective jump-to-definition, find-usages, etc, then the file structure sort of fades away, because you jump from fragment to fragment without really worrying about what files they are in.
Ever had the pleasure of opening a 20k+ line file in IntelliJ? (Not 2k, but 20k!). That happened one time, it wasn't a fun experience.
I consider IJ and most jetbrains products to be top-notch but that was a crippling experience. And I can't really say I blame them :)
EDIT: I agree with you by the way* just saying that even good tools can fail when the sizes get crazy!
I consider IJ and most jetbrains products to be top-notch but that was a crippling experience. And I can't really say I blame them :)
EDIT: I agree with you by the way* just saying that even good tools can fail when the sizes get crazy!
Small note here: it's obviously not it's an "absolute" rule that X lines of codes = fired. But for me, it's just a major code smell when people are writing very large single files with generalized names like "server.rs" or "processor.java."
It tends to mean that someone isn't following the single responsibility principle and the code is harder to understand and less maintainable. Obviously these aren't hard and fast rules but are combined with many other code smells and metrics.
For example, I really really dislike long methods with generic names. Usually this means there are too many things happening in a single method. The method name, it's input and output parameters should give the reader a fairly exact idea of what's happening in the method.
So if I wrote a method with this signature: public List<bool> ProcessData(List<string> input)
If you tried to predict what this might contain in the body, you would have zero idea.
Now this method:
public List<bool> ValidateVipCodes(List<string> input)
Just seeing this signature, you could easily imagine what the body would look like without much thought at all.
Great code should be structured in such a way that someone can view the project, ONLY LOOKING at file names and method names and have a good idea about what is going on and how it might work.
It tends to mean that someone isn't following the single responsibility principle and the code is harder to understand and less maintainable. Obviously these aren't hard and fast rules but are combined with many other code smells and metrics.
For example, I really really dislike long methods with generic names. Usually this means there are too many things happening in a single method. The method name, it's input and output parameters should give the reader a fairly exact idea of what's happening in the method.
So if I wrote a method with this signature: public List<bool> ProcessData(List<string> input)
If you tried to predict what this might contain in the body, you would have zero idea.
Now this method:
public List<bool> ValidateVipCodes(List<string> input)
Just seeing this signature, you could easily imagine what the body would look like without much thought at all.
Great code should be structured in such a way that someone can view the project, ONLY LOOKING at file names and method names and have a good idea about what is going on and how it might work.
It's a ridiculous metric.
573
There are a few opportunities for code generation here.
The server.rs looks like an implementation of RPC over websockets. So a refactoring might be to model the interface using something like https://grpc.io/ or any other IDL then code generate Rust server stubs and Typescript client stubs.
Cargo has a mechanism for this. https://doc.rust-lang.org/cargo/reference/build-scripts.html
The other area where there is some boiler plate is the Diesel ORM. Diesel does generate some code ie. schema.rs but you are left to write the structs that implement these as well as Form structs for passing data to Diesel.
The server.rs looks like an implementation of RPC over websockets. So a refactoring might be to model the interface using something like https://grpc.io/ or any other IDL then code generate Rust server stubs and Typescript client stubs.
Cargo has a mechanism for this. https://doc.rust-lang.org/cargo/reference/build-scripts.html
The other area where there is some boiler plate is the Diesel ORM. Diesel does generate some code ie. schema.rs but you are left to write the structs that implement these as well as Form structs for passing data to Diesel.
> 2256 lines long is already disqualifying
I'd love to hear a justification for this idea.
I'd love to hear a justification for this idea.