CSS changes that increased Reddit account creation by 200K yearly users(donotlick.com)
donotlick.com
CSS changes that increased Reddit account creation by 200K yearly users
http://www.donotlick.com/2015/01/05/8-increase-in-reddit-account-registrations/
58 comments
In fact, from my reading the rate of signups was increasing before hand and continued increasing afterwards, implying the design made no difference. That's what I can see from the trendline anyway, and the rest of the graph is too uneven to draw any conclusions.
They are wrong to draw the trendline the way they did. Since what they are claiming is not steady growth (then the trendline would make sense) but a shift in behaviour they should instead have drawn one line for average before the change, and one line for average after the change.
It is quite obvious that something has changed. Before the change there are five days above 7.5%, and after there are more than 10 days above 7.5%. The valleys are much higher too.
It is quite obvious that something has changed. Before the change there are five days above 7.5%, and after there are more than 10 days above 7.5%. The valleys are much higher too.
> It is quite obvious that something has changed.
Yup, but that's not an 8% increase in AR's. That's maybe an 8% increase in the rate of increase in AR's. That could be attributable to a LOT of things though (there isn't even enough context to control for network & seasonal effects, let alone whatever increased attention there might have been to sign ups after the rollout).
Ironically, probably the best way to isolate the impact to the launch is to narrow the time window and resolution to the minutes before and after the redesign launched.
Yup, but that's not an 8% increase in AR's. That's maybe an 8% increase in the rate of increase in AR's. That could be attributable to a LOT of things though (there isn't even enough context to control for network & seasonal effects, let alone whatever increased attention there might have been to sign ups after the rollout).
Ironically, probably the best way to isolate the impact to the launch is to narrow the time window and resolution to the minutes before and after the redesign launched.
More days above 7.5% would be true even if the underlying trendline was going up anyway.
They really should plot the moving average and include a longer timeseries - plotting the trendline before and after would be trying to prove what they are trying to show, rather than trying to work out whether what they are trying to show (our fancy new design wasn't a complete waste of time) is actually true.
They really should plot the moving average and include a longer timeseries - plotting the trendline before and after would be trying to prove what they are trying to show, rather than trying to work out whether what they are trying to show (our fancy new design wasn't a complete waste of time) is actually true.
Yes, exactly. The graph is full of spikes and I do not see any indication that there was any real change present from the CSS change.
This is why A/B testing is important.
Actually, it' why A/B testing is rife with false signaling. ;-)
The ugly truth is, the reason for Reddit's popularity growth in long term is /r/gonewild and its children. It's so integral part of Reddit, not even the femi-nazi-caterpillartrack-crazies dare to even attempt SJW'ing it.
/r/gonewild is not even in the first 50 subreddits with most subscribers. What makes you think it's the reason for Reddit's popularity growth?
They removed the CAPTCHA, I'm not surprised.
Quote from a previous article (so I wouldn't attribute all 8% to bots):
We’re experimentally removing CAPTCHA and using other methods for spambot detection.
We know it’s annoying for humans, and are starting to worry the bots are actually enjoying it.
Plus, our visual CAPTCHA was failing on accessibility for non-visual redditors.Sorry, to be clear, I wasn't attributing the lack of CAPTCHA to bots. Rather the improved usability and increased human sign-up rate to the removal of a painful CAPTCHA.
What other method for spambots could be used?
In addition to plenty of stuff I cant think of, you could see:
1) High percentage posts flagged as 'reported'
2) High percentage posts flagged as 'spam'
3) High volume of posts containing URLs
4) Posts containing identified spam messages (like how an email spam filter may work)
5) Multiple accounts from similar/same IPs
Almost all spam in a subreddit I mod for is from accounts < week old. (I believe there may already be a limit on how much you can post after registration though)
1) High percentage posts flagged as 'reported'
2) High percentage posts flagged as 'spam'
3) High volume of posts containing URLs
4) Posts containing identified spam messages (like how an email spam filter may work)
5) Multiple accounts from similar/same IPs
Almost all spam in a subreddit I mod for is from accounts < week old. (I believe there may already be a limit on how much you can post after registration though)
I mod a subreddit with >100k subscribers. We use AutoModerator and have it set to remove anything posted by an account less than a day old.
It catches some false positives from people who sign up just to post something, but it's also drastically decreased the spam. Most spambots are still at the "sign up and then blast shotgun-loads of posts before the algorithms catch you" level of behavior.
It catches some false positives from people who sign up just to post something, but it's also drastically decreased the spam. Most spambots are still at the "sign up and then blast shotgun-loads of posts before the algorithms catch you" level of behavior.
Do you review the removals and undo those for legitimate users? If not, wow. Talk about being hostile towards new users.
Most people never notice. The ones who do get an explanation of what happened when they ask about it in modmail.
It can be rough on a brand-new account that wanted to jump into a discussion right away, but at the same time the quantity of spam hitting your subreddit once you pass the 100k mark is just ludicrous, and this is by far the most effective way to stop it.
It can be rough on a brand-new account that wanted to jump into a discussion right away, but at the same time the quantity of spam hitting your subreddit once you pass the 100k mark is just ludicrous, and this is by far the most effective way to stop it.
In terms of community building, that might not be the worst thing in the world, especially if the community objective is quality over quantity.
IIRC they probably mean to detect them through how they operate on the website, rather than how they register (tracking navigation, how they upvote/downvote, and what content they post).
Simply testing for javascript still works pretty well for drive-by bots. For ones targeting Reddit specifically the captcha probably wasn't doing much anyway.
Adding a simple honeypot would probably cut spam in half.
Yeah, I hope internally they projected how many people were dying on the CAPTCHA, how many were believed to be bots, and then looked at the results through those filters.
You might be onto something here.
Sorry but this chart:
http://www.donotlick.com/wp-content/uploads/2015/01/newdesig...
Doesn't seem convincing evidence to me
Doesn't seem convincing evidence to me
I agree. While it does show an average increase of ~7.5% after the line, it also showed that before the line! It's hard to tell that anything happened.
And yet it is still impossible to use on a mobile device, due to a crappy custom dialog implementation.
It's refreshing to see what is to my eyes clearly a cleaner and more modern design actually performing better than what looks like a kludgy FrontPage design. Too often these analyses purport to show the opposite.
The before-and-after image shows a much less cluttered sign-up form
http://www.donotlick.com/wp-content/uploads/2014/11/cheat.pn...
However, the Nielsen Norman Group (NNG) recommend against using placeholder text in form fields. They list 7 main reasons. This is taken from their website
1. Disappearing placeholder text strains users' short-term memory
2. Without labels, users cannot check their work before submitting a form
3. When error messages occur, people don’t know how to fix the problem
4. Placeholder text that disappears when the cursor is placed in a form field is irritating for users navigating with the keyboard
5. Fields with stuff in them are less noticeable.
6. Users may mistake a placeholder for data that was automatically filled in
7. Occasionally users have to delete placeholder text manually
Obviously, not all of these will apply to the Reddit sign-up form (or to your own website). It's worth reading the full NNG article to get a different but valid perspective based on their research:
http://www.nngroup.com/articles/form-design-placeholders/
http://www.donotlick.com/wp-content/uploads/2014/11/cheat.pn...
However, the Nielsen Norman Group (NNG) recommend against using placeholder text in form fields. They list 7 main reasons. This is taken from their website
1. Disappearing placeholder text strains users' short-term memory
2. Without labels, users cannot check their work before submitting a form
3. When error messages occur, people don’t know how to fix the problem
4. Placeholder text that disappears when the cursor is placed in a form field is irritating for users navigating with the keyboard
5. Fields with stuff in them are less noticeable.
6. Users may mistake a placeholder for data that was automatically filled in
7. Occasionally users have to delete placeholder text manually
Obviously, not all of these will apply to the Reddit sign-up form (or to your own website). It's worth reading the full NNG article to get a different but valid perspective based on their research:
http://www.nngroup.com/articles/form-design-placeholders/
Submission title is a bit clickbaity and does not match the more modest actual article title, which is "8% Increase in reddit Account Registrations".
I wish I had a screenshot of the original version. It was absolutely awful. It seemed like the code wasn't touched in at least 5 years.
Nice job with the redesigned version, but in all seriousness... I wonder what took so long? Every few months a company should take a spin as a new user and find ways to refine the new-user process.
Use this as a reminder to take another look at your registration sign-up process in 2015.
Nice job with the redesigned version, but in all seriousness... I wonder what took so long? Every few months a company should take a spin as a new user and find ways to refine the new-user process.
Use this as a reminder to take another look at your registration sign-up process in 2015.
There is a screenshot shown in the linked previous article: http://www.donotlick.com/2014/11/17/redesign-of-reddits-logi...
Direct link: http://www.donotlick.com/wp-content/uploads/2014/11/cheat.pn...
Direct link: http://www.donotlick.com/wp-content/uploads/2014/11/cheat.pn...
There's a link to the original article about the redesign included in this article. It has a comparison of screenshots.
Here you go:
http://www.donotlick.com/2014/11/17/redesign-of-reddits-logi...
Here you go:
http://www.donotlick.com/2014/11/17/redesign-of-reddits-logi...
This is a typically poor example of ignoring the complexities inherent in analysing new user acquisition.
Suppose you implement a change that causes the registration form to crash for some subset of people? I am sorry to say for a while I was responsible for causing such issues at a very large service ;) Depending where the crash occurs and where you are counting signups you may actually get a massive boost in your metrics, as the victims retry with every attempt met by failure.
Suppose you correct an issue making it easier for people to login? Then lower registrations - the key here is to make sure you have a good metric to look at (e.g. accounts created that are interacting well with the site after as long a window as possible).
Suppose you implement a change that causes the registration form to crash for some subset of people? I am sorry to say for a while I was responsible for causing such issues at a very large service ;) Depending where the crash occurs and where you are counting signups you may actually get a massive boost in your metrics, as the victims retry with every attempt met by failure.
Suppose you correct an issue making it easier for people to login? Then lower registrations - the key here is to make sure you have a good metric to look at (e.g. accounts created that are interacting well with the site after as long a window as possible).
The article is about an 8% increase in account registrations not hits.
Exactly.
An account registration is basically like a hit metric.
The creation of an account does not imply that it will be used or the person who created it has not created many more either accidentally or for example because they could not log in.
An account registration is basically like a hit metric.
The creation of an account does not imply that it will be used or the person who created it has not created many more either accidentally or for example because they could not log in.
Just imagine what would happen if they applied better design to the rest of the site !
The subs have a lot of control over what is presented. I think they often look worse, but that is all a design choice.
Design is a choice? Radical thinking.
A bit more data would be nice to corroborate the claim. I understand some of the criticism in the comments. :)
If you are looking for some ridiculous correlation claims, check out the spurious correlation.
Here's an example,
The number of people who drowned by falling into a swimming pool correlates with the number of films Nicolas Cage appeared in.
http://www.tylervigen.com/view_correlation?id=359
:)
If you are looking for some ridiculous correlation claims, check out the spurious correlation.
Here's an example,
The number of people who drowned by falling into a swimming pool correlates with the number of films Nicolas Cage appeared in.
http://www.tylervigen.com/view_correlation?id=359
:)
IMO the design of Reddit's website could use an overhaul. I can't even snap it to one side of my screen without totally screwing up the front page and forget about using the site on mobile. The sidebars contain way too much unnecessary information as well. I do love me some reddit though.
Shame that they didn't do some A/B testing to more accurately quantify the effect.
They probably have the same answer they have for the "Why can't I search comments" question: they can't afford it.
any way to check which ones are real people and which ones are bots creating accounts for some reason ? maybe their activity posting vs subreddit popularity... I guess you may be surprised.
We ran some analyses and determined the rise is not due to an increase in spambots. I don't know the specifics of how that was determined, sorry.
That chart is a textbook example of "we drew a trendline through noisy data and drew completely unwarranted conclusions as a result".
Yeah, that what a properly modern designed account form given today standards can do this days.
I'd say they're correlated for sure.
My sarcasm detector is having trouble. Was your comment sarcasm? Was the original article and accompanying graph sarcasm? Is it sarcasm to say that unix time is incrementing at the same rate in 2015 as it did in 2014, and therefore it being 2015 is responsible for the increase in unix time?
No. As many of the other comments have here have said in more words, correlation != causation.
though, doesn't this suggest Unix time will not increment at the same rate throughout the year.
http://hpiers.obspm.fr/iers/bul/bulc/bulletinc.dat
I am just wondering how this rise is attributed to the CSS change? It may have had some effect, but I'm confused how this single graph says that it is because of a CSS change?