Twitter meets TensorFlow(blog.twitter.com)
blog.twitter.com
Twitter meets TensorFlow
https://blog.twitter.com/engineering/en_us/topics/insights/2018/twittertensorflow.html
72 comments
ML is like a genie in that it is very good at optimizing exactly what you tell it to, and so you need to think long and hard about what you're telling it to optimize, because it might turn out to be bad proxy for what you really wanted.
This is a super insightful analogy! Thank you.
+1
ML requires you know exactly what you want. And not what you need...
If you're not exact, disaster is imminent. Uber accident comes to mind.
Yes, rolling stones reference too.
ML requires you know exactly what you want. And not what you need...
If you're not exact, disaster is imminent. Uber accident comes to mind.
Yes, rolling stones reference too.
Sure, glad you liked it!
My timeline would be so much healthier if Twitter put me in charge of who shows up in it.
After Google killed Reader I was forced to follow all the people who used to blog to keep up with their professional work. But especially in the last 3 years Twitter feeds have all come with a large side order of political hot takes that I have zero interest in sitting through.
To counteract this I have an extensive word blocklist and turn off retweets on many people so my Twitter can remain about the work of interesting people in my industry and not just an open sewer of politics.
Twitters insistence to show me posts that people I follow liked or engaged highly with bypasses all this.
After Google killed Reader I was forced to follow all the people who used to blog to keep up with their professional work. But especially in the last 3 years Twitter feeds have all come with a large side order of political hot takes that I have zero interest in sitting through.
To counteract this I have an extensive word blocklist and turn off retweets on many people so my Twitter can remain about the work of interesting people in my industry and not just an open sewer of politics.
Twitters insistence to show me posts that people I follow liked or engaged highly with bypasses all this.
Same here. Recently discovered the RSS reader Cappuccino (thanks HN) and so far it seems to fix the problem. It even syncs unread posts between Mac and iPad.
If Twitter ever moves to a fully algorithmic timeline like Instagram (has annoying done), it will shrivel up and die. The linear timeline of Twitter is what makes Twitter....Twitter. I've noticed a lot lately that more and more I see tweets out of order because of the algorithms. I believe you can turn that off, but sometimes those are useful...but more often they are spam in the moment, especially if a big story has broken. I feel like it used to only occur when you first opened up Twitter, you'd see old highlights...now it feels like it's intertwined more...it's very annoying. Twitter is, by far, the most powerful and useful social media platform out there for information...I sure hope they aren't going to strangle it to death.
I second what you noticed. I now see myself with a WTF expression on my face when I am trying to figure out what happened with the linear timeline and all that junk I pretty much don't care about inserted in what I am interested to read. Feels like random ad inserts. Not good.
Unfortunately you can't turn off the algorithm which treats other people's favourites as if they were retweets, which makes me think twice about using the favourite feature at all. I know I've had stuff put into my feed which the person doing the favouriting would never have retweeted.
I wrote an extension/user script to filter those - and retweets - out into their own lists on the front page [1], as I just want a chronological timeline of what people are saying now, which mustn't be "engaging" enough for Twitter's purposes.
[1] https://github.com/insin/manage-twitter-engagement#readme
I wrote an extension/user script to filter those - and retweets - out into their own lists on the front page [1], as I just want a chronological timeline of what people are saying now, which mustn't be "engaging" enough for Twitter's purposes.
[1] https://github.com/insin/manage-twitter-engagement#readme
I thought I was about to read some satire based on that quote.
Turns out it's not.
Turns out it's not.
I suppose the same applies to many other ML products? Like Netflix's recommendations, Amazon's suggestions and almost all the ads you see these days?
I agree that the problem exists, but its not just Twitter, and this is an unfortunate side-effect of recommendations in general: even if you do count-based recommendations, you are going to have a bit of echo chamber.
I agree that the problem exists, but its not just Twitter, and this is an unfortunate side-effect of recommendations in general: even if you do count-based recommendations, you are going to have a bit of echo chamber.
Those two examples are fine applications of ML. The problem is when you're dealing with sequential, timestamped content that SHOULD be displayed in the order it was posted. I think companies should very judiciously apply ML in these cases, and only possibly for 'top stories' or huge events that are occurring. However it appears ML is being applied across the board, using multiple signals and throwing the chronological order of the content out of whack, which is very frustrating.
Empowering users with Machine Learning, i.e. offering simple powerful ML based filtering tools for the Twitter user.
More importantly allow opt out and configuration of stuffing feeds with politics and liked by those I follow.
But the idea is not to empower users but masticate users to provide more 'value' to Advertisers by selling them 21stC version of 'hits' - the false & forced engagement metric.
This all stems from the mentality that Twitter are serving the money not the user.
It is short-termism.
Hopefully crap-stuffing users will allowing the resurgence of a diverse market that necessitates open standards unlike our current hyper monopolies and their data silos.
More importantly allow opt out and configuration of stuffing feeds with politics and liked by those I follow.
But the idea is not to empower users but masticate users to provide more 'value' to Advertisers by selling them 21stC version of 'hits' - the false & forced engagement metric.
This all stems from the mentality that Twitter are serving the money not the user.
It is short-termism.
Hopefully crap-stuffing users will allowing the resurgence of a diverse market that necessitates open standards unlike our current hyper monopolies and their data silos.
I’m not even sort of interested in what Twitter’s heavy-handed definition of “healthy conversations” are.
I dare say they’re acting well outside of their wheelhouse. Their job is to provide a platform for discussion, not whatever that mess of corpspeak I just read is.
I dare say they’re acting well outside of their wheelhouse. Their job is to provide a platform for discussion, not whatever that mess of corpspeak I just read is.
Those 3000 employees need to justify their jobs.
What better way than program a robot to decide what 'healthy conversations' are.
Dorsey's capitulation to the complaint army over his recent chicken sandwich says it all.
I do see the hate mobs calming down ( or being ignored ), eventually, but when that happens we know Dorsey and a lot of these CEOs have little or no backbone.
What better way than program a robot to decide what 'healthy conversations' are.
Dorsey's capitulation to the complaint army over his recent chicken sandwich says it all.
I do see the hate mobs calming down ( or being ignored ), eventually, but when that happens we know Dorsey and a lot of these CEOs have little or no backbone.
Their job is make money. The sense of entitlement people have about social media is just wild.
[deleted]
Do you remember hiw bad YouTube comments have been? And it's significantly better lately?
That happened due to ML.
Not everyone is out there to use ML for nefarious purpose. And there are multiple use cases for ML.
That happened due to ML.
Not everyone is out there to use ML for nefarious purpose. And there are multiple use cases for ML.
How are yt comments any better now?
They are now mostly real people organically being morons, as opposed to mostly bots spamming other channels and a bunch of stick figure chain mail.
It's not perfect, but it's way better. The spam is mostly gone. Inane comments are lower in priority.
In addition to ML a lot of people are reporting offensive comments. If you care about your account then you should perhaps think twice.
oh wow you are right. I actually participate in youtube comments now, i used to completely ignore them. Thats a beautiful example of ML.
At least for music videos, comments are still pretty shitty. It feels like more than half of them are one of the following patterns:
- "Like if you're watching in June 2018?" [2k upvotes]
- "Most underrated artist of our time" [5k upvotes]
- "Wow, mad respect for <artist>" (written by verified account by other artist trying to promote their own content)
- "Like if you're watching in June 2018?" [2k upvotes]
- "Most underrated artist of our time" [5k upvotes]
- "Wow, mad respect for <artist>" (written by verified account by other artist trying to promote their own content)
Why are those bad comments? The first one is a bit ridiculous, but the others are fine. They're comments. They're not discussion points. YouTube isn't a forum or a social network. There's nothing compelling people to leave intelligent or even constructive points. The comment section is for people to note what they think. That's it.
When people say these sorts of things are bad I think it's because their expectations and understanding of a comments section are wrong.
When people say these sorts of things are bad I think it's because their expectations and understanding of a comments section are wrong.
It's just super repetitive, and actually good and original comments get buried because they don't match the hivemind.
Of course, it's a question of taste what comments should be more visible. If the community actually prefers reading the same stuff over and over again, then the current state is probably justified, and it's just a moronic community.
Of course, it's a question of taste what comments should be more visible. If the community actually prefers reading the same stuff over and over again, then the current state is probably justified, and it's just a moronic community.
I wonder, what comments are you expecting under pop music videos? Deep discussion about chord progressions or lyrics themes?
Depends what music you listen to. If you listen to smaller artists on videos that have under 1 million or 100k views you will see many more high effort comments talking about the band/band members or about their experience with the music.
e.g. "I got a chance to see these guys live in October, and [insert band member here] is actually a really cool guy. Talked to him afterwards and shared [so and so interaction]"
e.g. "This song was actually inspired by [such and such random factoid]"
e.g. "This song reminds me of my mom"
e.g. "I got a chance to see these guys live in October, and [insert band member here] is actually a really cool guy. Talked to him afterwards and shared [so and so interaction]"
e.g. "This song was actually inspired by [such and such random factoid]"
e.g. "This song reminds me of my mom"
28 people hate don't appreciate art!!1 [video has 28 dislikes]
[video has 28 dislikes]
7 people are morons!!!
9 people are sad they have no taste in music111
What is wrong with the 13 downvotes?
WHHHHHHHHYYYYYY don't 16 people have any taste?
Well it's obvious that 18 people don't have any taste.
21 downvotes can suck it!
22 people are morons!!!
WTF, 24 downvotes? There should be negative downvotes!
25 people are morons!!!
Well I see Hitler and his 26 closest friends have seen this video!
27 people are morons!!!
28 people hate don't appreciate art!!1>Do you remember hiw bad YouTube comments have been?
Yes
>And it's significantly better lately?
It's gotten worse and a few youtubers I follow have simply disabled it in recent time (less than 1 month ago was the most recent one)
Normal conversation on youtube is barely possible since the algorithm randomly decides to not notify me of responses anymore or removes entire comments from my view that other people reference and I'm left wondering what is going on.
The only good thing about youtube comments is that I can remove them easily with a plugin.
Yes
>And it's significantly better lately?
It's gotten worse and a few youtubers I follow have simply disabled it in recent time (less than 1 month ago was the most recent one)
Normal conversation on youtube is barely possible since the algorithm randomly decides to not notify me of responses anymore or removes entire comments from my view that other people reference and I'm left wondering what is going on.
The only good thing about youtube comments is that I can remove them easily with a plugin.
> Normal conversation on youtube is barely possible
It's also interface. I've been notified of replies to comments, click the notification, and I'm not taken to the reply. This happens to me on mobile enough that I've basically just given up trying to engage for most things on youtube.
It's also interface. I've been notified of replies to comments, click the notification, and I'm not taken to the reply. This happens to me on mobile enough that I've basically just given up trying to engage for most things on youtube.
Happens on the desktop too. It's impossible to respond or follow up to anything on there. You'll get notified of unrelated comments, you're not able to respond to a specific person, everything is just in one parent thread. I've since blocked comments from even displaying.
"Significantly better" is a stretch.
The comments got much better after Youtube added user ratings. Hardly "ML". I'm guessing they added ML later, but I haven't noticed any difference, to be honest...
And by "much better" I mean they went from amoeba level to early multi-cellular organism :)
The comments got much better after Youtube added user ratings. Hardly "ML". I'm guessing they added ML later, but I haven't noticed any difference, to be honest...
And by "much better" I mean they went from amoeba level to early multi-cellular organism :)
They are likely using those user ratings to build a model of what kind of comments bad users make.
Most of the bang for their buck was when they used a plain old sorting of those ratings. That already hid a lot of the sludge. The ML stuff is just a minor optimization compared to that.
Everything is just my opinion, of course, they're probably following a bunch of metrics I don't have access to and which could prove that the ML effort was worth it.
Everything is just my opinion, of course, they're probably following a bunch of metrics I don't have access to and which could prove that the ML effort was worth it.
This is hardly unique to twitter. You don't this is happening at Facebook, Instagram, Youtube, etc...
Just to play Devil's advocate:
One that wants to manipulate your mind
Like your post here? Or my present reply to it? And so on...
one that echochambers your discovery
Which is good, because nobody has enough time in the day to process every conceivable byte of information they might stumble across...
one that censors arbitrarily
OK, because while everyone may have the right to speak freely, not all speech needs to be heard by everyone...
One that wants to manipulate your mind
Like your post here? Or my present reply to it? And so on...
one that echochambers your discovery
Which is good, because nobody has enough time in the day to process every conceivable byte of information they might stumble across...
one that censors arbitrarily
OK, because while everyone may have the right to speak freely, not all speech needs to be heard by everyone...
The idea is I have a timeline and am able to choose what's on it. Good, bad, ugly. It's up to me. If I don't like what someone's saying I unfollow, mute or block. Easy.
With Twitter, FB, and modern social media, they let their algorithm decide what I should see. They think they know what people want and ultimately what's best for me. I disagree with this so I opt-out of their services. Sadly. Because there are ( or were ) a lot of interesting and novel voices on there.
Also, I suppose at a certain point those 3XXX employees need to justify their existence.
And yes it's their prerogative to do whatever they want with their product. And yes I know private companies are not bound by the 1st Amendment.
With Twitter, FB, and modern social media, they let their algorithm decide what I should see. They think they know what people want and ultimately what's best for me. I disagree with this so I opt-out of their services. Sadly. Because there are ( or were ) a lot of interesting and novel voices on there.
Also, I suppose at a certain point those 3XXX employees need to justify their existence.
And yes it's their prerogative to do whatever they want with their product. And yes I know private companies are not bound by the 1st Amendment.
> Easy.
If you're spectating, maybe. Completely falls apart when you try to engage in dialogue on Twitter, which is what both Twitter, Inc. and I want.
Say I follow someone because I like what they say. And I want to discuss what they're saying, or to explore existing discussions. I'm going into this ready to have a thoughtful debate or two.
So I open the replies, and they're what I consider to be garbage: flamewars, reaction GIFs, propaganda, straight-up bad-faith trolling, and worse. But maybe there's some good in here somewhere?
Assume I'm a typical human with emotions and stuff. Do I:
A) Immediately close the thread! It's not worth it, and I have the ability and willpower to bow out.
B) Scroll through the replies, muting and blocking people. Maybe if I do this enough I'll have a nicer experience on the next thread I open. Wow there sure are a lot of these folks!
C) Take the bait. Someone is wrong on the internet.
You know already, but I'll tell you anyway. I, typical human, tend to rush headlong into option C), which wastes my time, makes me unhappy, and gives me a negative perception of the quality of discussions on Twitter. Twitter, Inc. doesn't want that either, but they've picked these replies to show me. Clearly they don't know what kind of thing I think is garbage. So what can they do?
It's not easy.
If you're spectating, maybe. Completely falls apart when you try to engage in dialogue on Twitter, which is what both Twitter, Inc. and I want.
Say I follow someone because I like what they say. And I want to discuss what they're saying, or to explore existing discussions. I'm going into this ready to have a thoughtful debate or two.
So I open the replies, and they're what I consider to be garbage: flamewars, reaction GIFs, propaganda, straight-up bad-faith trolling, and worse. But maybe there's some good in here somewhere?
Assume I'm a typical human with emotions and stuff. Do I:
A) Immediately close the thread! It's not worth it, and I have the ability and willpower to bow out.
B) Scroll through the replies, muting and blocking people. Maybe if I do this enough I'll have a nicer experience on the next thread I open. Wow there sure are a lot of these folks!
C) Take the bait. Someone is wrong on the internet.
You know already, but I'll tell you anyway. I, typical human, tend to rush headlong into option C), which wastes my time, makes me unhappy, and gives me a negative perception of the quality of discussions on Twitter. Twitter, Inc. doesn't want that either, but they've picked these replies to show me. Clearly they don't know what kind of thing I think is garbage. So what can they do?
It's not easy.
Replies with the most upvotes are at the top. That generally filters out the BS. There is a date posted versus upvote count balance that needs to be reached, but ML is not required.
And this all assumes you're trying to have conversation around someone with many many followers. If you're in a small circle, none of your argument applies.
Further posts by the original poster are at the top too.
Yes, you may have to filter through some posts on your own. If you plan on having a conversation with Obama, good luck. And for good reason.
But I don't want Twitter Inc filtering for me. Essentially upvoting and downvoting comments based on their algorithm. Sorry.
At least give users the choice to use their ML algorithm or not. I'd even pay to have it turned off or to tweak how I see my timeline.
So I don't follow.
And this all assumes you're trying to have conversation around someone with many many followers. If you're in a small circle, none of your argument applies.
Further posts by the original poster are at the top too.
Yes, you may have to filter through some posts on your own. If you plan on having a conversation with Obama, good luck. And for good reason.
But I don't want Twitter Inc filtering for me. Essentially upvoting and downvoting comments based on their algorithm. Sorry.
At least give users the choice to use their ML algorithm or not. I'd even pay to have it turned off or to tweak how I see my timeline.
So I don't follow.
If you're in a small circle, none of your argument applies.
Try being a woman or Jewish and have a conversation on Twitter about - say - gaming.
Try being a woman or Jewish and have a conversation on Twitter about - say - gaming.
[deleted]
>> One that wants to manipulate your mind
> Like your post here? Or my present reply to it? And so on...
Except that Twitter thrives on controversy. There's controversy on HN as well, but escalations lead to flagging of content, not promotion of it.
>> one that echochambers your discovery
> Which is good, because nobody has enough time in the day to process every conceivable byte of information they might stumble across...
Strawman. This statement isn't about quantity, it's about content. Focusing on information that reinforces one's one view, instead of engaging in critical thought, is a dumbing-down.
>> one that censors arbitrarily
> OK, because while everyone may have the right to speak freely, not all speech needs to be heard by everyone...
Strawman. This isn't about what gets filtered, but who filters and how this information gets filtered.
> Like your post here? Or my present reply to it? And so on...
Except that Twitter thrives on controversy. There's controversy on HN as well, but escalations lead to flagging of content, not promotion of it.
>> one that echochambers your discovery
> Which is good, because nobody has enough time in the day to process every conceivable byte of information they might stumble across...
Strawman. This statement isn't about quantity, it's about content. Focusing on information that reinforces one's one view, instead of engaging in critical thought, is a dumbing-down.
>> one that censors arbitrarily
> OK, because while everyone may have the right to speak freely, not all speech needs to be heard by everyone...
Strawman. This isn't about what gets filtered, but who filters and how this information gets filtered.
Not enabling echochambering doesn't mean you get overwhelmed.
If the algorithms only ever selects things you already like, how would you discover things you might not know you like? Or change your mind about a subject because you only ever see tweets that agree with your worldview?
A good timeline would not only show you speech, ideas or content you agree to or like but also stuff you don't like and don't agree with.
If the algorithms only ever selects things you already like, how would you discover things you might not know you like? Or change your mind about a subject because you only ever see tweets that agree with your worldview?
A good timeline would not only show you speech, ideas or content you agree to or like but also stuff you don't like and don't agree with.
I always enjoy browsing the Twitter Engineering blog because of 1) the granular detail that they give 2) the concise, organized format that the information is presented, and 3) because I love their creativity. However, the first sentence of this article "Machine learning enables Twitter to drive engagement, surface content most relevant to our users, and promote healthier conversations. As part of its purpose of advancing AI for Twitter in an ethical way, Twitter Cortex is the core team responsible for facilitating machine learning endeavors within the company" is a complete turn off. Keep it technical. Leave the Marketing/PR out of it.
- Regular blog.twitter.com/engineering reader
- Regular blog.twitter.com/engineering reader
This is very confusing and meandering. It gives flow charts and lists of steps that don’t map to my experience building deep learning models at scale, and spends a strange amount of time passive aggressively dismissing Lua Torch and extolling virtues of TensorFlow that aren’t very important.
As with all of these purported pipelining systems, I’m skeptical and happy to let a bunch of other people deal with the headches of making it adequately general for a few years before I’ll even start caring about grokking it for my use cases.
In the meantime, creating build tooling, data pretreatment tooling and deployment tooling is pretty valuable for me to understand business considerations and make sure all my modeling & experimentation aren’t just time wasting ivory tower projects, particularly in terms of customizing performance characteristics on a situation-to-situation basis, free to design the deployed system without a constraint to a particular serving architecture.
It also makes me very disinterested in applying to work for the Cortex team, because even though the article is talking about DeepBird v2 as a means to free ML engineers to do more research, it seems pretty obvious that there’s a huge surface area of maintenance and feature management for this platform. Your job is probably going to be less about research, which is scarce work that people compete over anyway.
Possibly attractive for people who just like deep C++ platform building, which is an internal drive not often found in people wanting to solve business problems with ML models.
As with all of these purported pipelining systems, I’m skeptical and happy to let a bunch of other people deal with the headches of making it adequately general for a few years before I’ll even start caring about grokking it for my use cases.
In the meantime, creating build tooling, data pretreatment tooling and deployment tooling is pretty valuable for me to understand business considerations and make sure all my modeling & experimentation aren’t just time wasting ivory tower projects, particularly in terms of customizing performance characteristics on a situation-to-situation basis, free to design the deployed system without a constraint to a particular serving architecture.
It also makes me very disinterested in applying to work for the Cortex team, because even though the article is talking about DeepBird v2 as a means to free ML engineers to do more research, it seems pretty obvious that there’s a huge surface area of maintenance and feature management for this platform. Your job is probably going to be less about research, which is scarce work that people compete over anyway.
Possibly attractive for people who just like deep C++ platform building, which is an internal drive not often found in people wanting to solve business problems with ML models.
> Possibly attractive for people who just like deep C++ platform building, which is an internal drive not often found in people wanting to solve business problems with ML models.
And the world has plenty of people who are not interested in trying to solve business problems with ML models, but are rather interested in the engineering side of ML. I am one of those people. My current work (at cortex) involves improving the time taken to train the types of models not usually described in research papers.
If anyone is interested in machine learning but wants to build high performance training / serving platform, message me over twitter (@pavan_ky), Cortex is hiring.
And the world has plenty of people who are not interested in trying to solve business problems with ML models, but are rather interested in the engineering side of ML. I am one of those people. My current work (at cortex) involves improving the time taken to train the types of models not usually described in research papers.
If anyone is interested in machine learning but wants to build high performance training / serving platform, message me over twitter (@pavan_ky), Cortex is hiring.
Personally I am very interested in both how models can help solve business problems and how to make effectively engineered tools for machine learning.
In my work I spend a lot of time on new deep learning architectures or experimenting with modifications or fine-tuning or ensembling.
I write a lot of container and Makefile tooling to ensure experiments are reproducible and results have identifiers that map back to the full set of data, software and parameters.
I also write a lot of backend server software to wrap trained models in a web application, mostly in Python, and do a lot of work with Cython after profiling to target only those spots of the code that reveal actual performance bottlenecks in terms connected directly to a specific business problem’s latency or throughput requirements — as in, not taking the huge premature optimization step of assuming a whole system needs to be written in C++, and instead using profiling and case-by-case diagnostics to know when to write something as an optimized C extension module callable from Python for very specific and localized sections of code.
My experience has been that there is such a lack of transparency about how deployment will work, how performance will work, etc., when using cookie cutter pipeline approaches, like sklearn Pipelines, TensorFlow serving, Fargate, etc. You’ll always need to break some assumption of the pipeline, layer in new diagnostics, debug latency issues, etc., on a case-by-case basis.
99% of the time, ease of specifying a new model or articulating an experiment is not hard, requires little dev work, and only represents about 10% of the actual work needed to explore a model’s appropriateness for a given problem at hand.
The rest requires very specialized control and visibility to basically perform application-specific surgery on the pipeline, customizing and tailoring many aspects, from how multi-region deployment should look to how optimized the web service code should be to whether to use asynchronous workers or a queuing service to stage and process requests, to optimizing preprocessing treatments, to instrumenting some extra New Relic metric tracking that the pipeline isn’t extensible enough to just specify in some config, and so on.
What’s been most important is that the deep learning engineers on the team, who are researchers, are also excellent system engineers at all those topics too and display a high degree of curiosity towards them, and absolutely do not look at it like “boring work” that distracts from the experimentation they would rather do. Their value add is not driven by spending more time experimenting — that’s virtually never the case. Their value add is in both knowing the details of the deep learning models intimately while also knowing the deep details of implementation, optimization, deployment, and diagnostics.
In that sense, tying model development to a cookie cutter pipeline framework, whether Spark or sklearn or something custom in-house, is something I believe strongly is an anti-pattern.
In my work I spend a lot of time on new deep learning architectures or experimenting with modifications or fine-tuning or ensembling.
I write a lot of container and Makefile tooling to ensure experiments are reproducible and results have identifiers that map back to the full set of data, software and parameters.
I also write a lot of backend server software to wrap trained models in a web application, mostly in Python, and do a lot of work with Cython after profiling to target only those spots of the code that reveal actual performance bottlenecks in terms connected directly to a specific business problem’s latency or throughput requirements — as in, not taking the huge premature optimization step of assuming a whole system needs to be written in C++, and instead using profiling and case-by-case diagnostics to know when to write something as an optimized C extension module callable from Python for very specific and localized sections of code.
My experience has been that there is such a lack of transparency about how deployment will work, how performance will work, etc., when using cookie cutter pipeline approaches, like sklearn Pipelines, TensorFlow serving, Fargate, etc. You’ll always need to break some assumption of the pipeline, layer in new diagnostics, debug latency issues, etc., on a case-by-case basis.
99% of the time, ease of specifying a new model or articulating an experiment is not hard, requires little dev work, and only represents about 10% of the actual work needed to explore a model’s appropriateness for a given problem at hand.
The rest requires very specialized control and visibility to basically perform application-specific surgery on the pipeline, customizing and tailoring many aspects, from how multi-region deployment should look to how optimized the web service code should be to whether to use asynchronous workers or a queuing service to stage and process requests, to optimizing preprocessing treatments, to instrumenting some extra New Relic metric tracking that the pipeline isn’t extensible enough to just specify in some config, and so on.
What’s been most important is that the deep learning engineers on the team, who are researchers, are also excellent system engineers at all those topics too and display a high degree of curiosity towards them, and absolutely do not look at it like “boring work” that distracts from the experimentation they would rather do. Their value add is not driven by spending more time experimenting — that’s virtually never the case. Their value add is in both knowing the details of the deep learning models intimately while also knowing the deep details of implementation, optimization, deployment, and diagnostics.
In that sense, tying model development to a cookie cutter pipeline framework, whether Spark or sklearn or something custom in-house, is something I believe strongly is an anti-pattern.
I’m developing the build tooling for C/C++ in pants (the twitter build tool) to support this work, see [0]. Twitter open sources basically all of its infrastructure, we don’t really like throwing open source over the wall, and we maintain several high quality open source libraries that don’t require you to develop specific expertise in platform building (e.g. scalding [1], one of my faves)!
[0] https://github.com/pantsbuild/pants/pull/5815 [1] https://github.com/twitter/scalding
[0] https://github.com/pantsbuild/pants/pull/5815 [1] https://github.com/twitter/scalding
Maybe they can use ML on the user feedback to figure out what features people actually would like on the platform, instead of trying to figure out relevancy, which no company has ever successfully done. /s
Is there a recent in-depth article somewhere about Twitter's internals? It must be frustrating working on features very few people want to use.
I still can't edit a tweet I just posted, any video looks absolutely horrible for the first 10 seconds, the timeline is a mess, third-party developers that make interesting/much better clients are getting stomped on, and harassment is running mostly unchecked on the platform. Yet I read interesting development articles on the engineering blog. Wasted talent?
I really like the Twitter engineering blog articles, but it seems like it's just an HR tool.
Is there a recent in-depth article somewhere about Twitter's internals? It must be frustrating working on features very few people want to use.
I still can't edit a tweet I just posted, any video looks absolutely horrible for the first 10 seconds, the timeline is a mess, third-party developers that make interesting/much better clients are getting stomped on, and harassment is running mostly unchecked on the platform. Yet I read interesting development articles on the engineering blog. Wasted talent?
I really like the Twitter engineering blog articles, but it seems like it's just an HR tool.
This blog post describes twitter's move from Lua Torch to Tensorflow. I am surprised to see it so highly ranked on the front page because there's very little content here. Basically, they describe the sorts of data structures they use and list a couple of advantages of Tensorflow vs. the out-of-date Lua Torch framework.
It's probably on the front page because a lot of people are deciding between PyTorch or Tensorflow for their company's production ML pipeline, and this is an article about making exactly that decision. Even if they didn't present any actual analysis, just the sentence "Twitter's ML team, which had previously used LuaTorch, recently evaluated PyTorch and Tensorflow and ultimately decided to standardize on Tensorflow" would be of interest to a lot of people.
This post says all you need to know about Twitter.
"Machine learning enables Twitter to drive engagement...".
The very first thing they mention is engagement. They don't care about quality, or what users want, or to foster communication. They care about one thing and one thing only: engagement. More clicks, more likes, more moving around the interface.
"Machine learning enables Twitter to drive engagement...".
The very first thing they mention is engagement. They don't care about quality, or what users want, or to foster communication. They care about one thing and one thing only: engagement. More clicks, more likes, more moving around the interface.
> given the migration of the Torch community from Lua to Python via PyTorch, and subsequent waning support for Lua Torch, Cortex began evaluating alternatives to Lua Torch
I'm a bit out of the loop, has PyTorch really become so much more popular than Lua Torch, in the short time span since it launched? And is it true that most of the original Lua Torch community switched to PyTorch?
I'm a bit out of the loop, has PyTorch really become so much more popular than Lua Torch, in the short time span since it launched? And is it true that most of the original Lua Torch community switched to PyTorch?
PyTorch is catching up to TensorFlow, but still has a far way to go especially compared with the user experience of Keras with TensorFlow.
In PyTorch you have to do quite a bit more work specifying gradients, zeroing the gradients between update steps, specifically describing the operations of the backward pass in a custom layer, even in cases when it could just be inferred by the computation graph library as in TensorFlow.
I’ve heard anecdata that PyTorch performance can be better, but I think it’s still a case-by-case topic, and both libraries are still changing rapidly.
I attribute the switch from Lua Torch to PyTorch as just an obvious effect based on how bad of a language Lua is for the type of interactive and experimental programming needed for this type of work. Lua is a good language for some applications, but not this — and the value of spending effort to build the primary Torch frontend in Python became obvious quickly.
In PyTorch you have to do quite a bit more work specifying gradients, zeroing the gradients between update steps, specifically describing the operations of the backward pass in a custom layer, even in cases when it could just be inferred by the computation graph library as in TensorFlow.
I’ve heard anecdata that PyTorch performance can be better, but I think it’s still a case-by-case topic, and both libraries are still changing rapidly.
I attribute the switch from Lua Torch to PyTorch as just an obvious effect based on how bad of a language Lua is for the type of interactive and experimental programming needed for this type of work. Lua is a good language for some applications, but not this — and the value of spending effort to build the primary Torch frontend in Python became obvious quickly.
Off topic, but the article mentions "deep learning at scale" which triggered me. Is this use of the term "at scale" something new(ish)? As far as I know "at scale" means something like "in the appropriate amount". Here it seems to be shorthand for "implemented in a scalable manner". This use seems all over the place now. Is there a native English speaker who can comment on that?
I am not a native English speaker, but quite familiar with the use of "at (large) scale" in similar context.
It means that one has deployed it/is using the tool with a non-trivial (could be anywhere from few 100s to few 1000s of machines) amount of CPU and/or data. In the context of serving, "large scale" could also mean the number of queries/second hitting the serving layer.
It means that one has deployed it/is using the tool with a non-trivial (could be anywhere from few 100s to few 1000s of machines) amount of CPU and/or data. In the context of serving, "large scale" could also mean the number of queries/second hitting the serving layer.
[deleted]
Brought my following down to 9 accounts last week and enjoying the flow again. I have always used Twitter passively, only re-tweeting 3-4 interesting stories a day. Hoping TensorFlow for healthy conversations does not spoil this sort of 24/7 breaking news from trusted sources, now. What I fear the most, is having my flow made of one tweet from sources I choose & one tweet TensorFlow suggests to me.
> I have always used Twitter passively, only re-tweeting 3-4 interesting stories a day
That seems fairly active to me. In almost 10 years I did my first re-tweet last year.
That seems fairly active to me. In almost 10 years I did my first re-tweet last year.
You’re right, of course, ehehe. Just avoiding to express personal opinions, so that no flamewars or endless arguments waste your day, is not passive enough if you still sign up once and sign in daily. Therefore, the next step is deactivating my account for good while following trusted sources without logging in at all? Twitter cookies would still track my activity, though.
What could possibly go wrong?
Did twitter drop Scala? Wonder how the story is with ML and Scala
No,they have not. Especially on the machine learning teams
Twitter has become a tragedy.
PR sucks. It looks like PR.
One that wants to manipulate your mind, one that echochambers your discovery, one that censors arbitrarily.