At Blackrock, machines are rising over managers to pick stocks(nytimes.com)
nytimes.com
At Blackrock, machines are rising over managers to pick stocks
https://www.nytimes.com/2017/03/28/business/dealbook/blackrock-actively-managed-funds-computer-models.html
70 comments
There's been a lot of interest lately in the hypothesis that increased passive investing might cause non-competition as mutual funds hold shares in every company in a sector. It's now gotten a couple studies from both academia and even the Federal Reserve, and so far the evidence is that there is nothing to worry about yet. Check out Matt Levine's daily column and the "Should index funds be illegal?" section, he's been following these developments.
That doesn't necessarily mean it won't be a problem in the future. Right now the big index funds like Vanguard and BlackRock own something like 10% of public shares. If they reach, say, 50%, we might have to revisit this, but for now, it seems to be a non-issue.
That doesn't necessarily mean it won't be a problem in the future. Right now the big index funds like Vanguard and BlackRock own something like 10% of public shares. If they reach, say, 50%, we might have to revisit this, but for now, it seems to be a non-issue.
Hi, I am curious, do you do lot's of finance type research, would like to chat if possible to learn more.
P.S. Also really like Matt Levine's postings.
Following a market cap index investing - the most popular method of passive investing - means buying a set of stock weighted by market cap and then mostly doing nothing. It's hardly the machines taking over. Algorithm investment is probably not passive investment.
There are actually two trends:
1) Move to ETFs: Clients have been moving away from high-fee hedge funds into passive funds (ETFs) where trading is generally simpler.
2) Automation: There used to be a lot of analysts working on reports to monitor fund performance, tracking error, trading costs, risk exposures and so on. They are being replaced by software.
Exactly, how can algorithm investing be truly passive investing?
There's a lot of confusion in the article.
There's a lot of confusion in the article.
Disclosure: I work for a company which does this stuff. I'm not in the financial programming side.
There are several things you can do.
For example, say you wanted to hold the S&P500, but as individual stocks rather than paying someone at a mutual fund or ETF to hold it for you. This might be beneficial in a number of ways, but it has the serious drawback of having to manage 500 entries. You might already own a dozen or two of those entries already, or you have overlapping purchases in other investments.
You could tell a smart portfolio management program to use the S&P500 list as a target, and tell it about all the rest of your current holdings. Then it could analyze your current portfolio each day and make recommendations about how to get closer to your target, while avoiding wash sales and duplicate purchases. If you trusted that program, you could feed the output to your brokerage and have the trades executed automatically. Or you could look at the output and make decisions about whether you want to make the changes. Either way, you are no longer doing analysis yourself, and you're edging closer to your target, at which point you would be passive.
On the other hand, suppose you wanted an approximation of the S&P500 -- a target that would give you most of the same exposure and opportunity, but had a reduced number of individual stocks. You could run simulations on subsets of the S&P500 until you got a group that performed sufficiently similarly to the whole thing - 300? 250? 100? 50? as you reduce the set size you reduce the fidelity of the model - and then buy and hold those.
You can do a lot with an algorithm, and if your strategy doesn't need to operate in realtime, it will certainly look passive compared to HFT bots. A human might not want to rebalance more than annually or quarterly, because it can be a lot of work -- but a robot has no problems doing all the calculations daily and looking for a sufficient reward to present to you.
There are several things you can do.
For example, say you wanted to hold the S&P500, but as individual stocks rather than paying someone at a mutual fund or ETF to hold it for you. This might be beneficial in a number of ways, but it has the serious drawback of having to manage 500 entries. You might already own a dozen or two of those entries already, or you have overlapping purchases in other investments.
You could tell a smart portfolio management program to use the S&P500 list as a target, and tell it about all the rest of your current holdings. Then it could analyze your current portfolio each day and make recommendations about how to get closer to your target, while avoiding wash sales and duplicate purchases. If you trusted that program, you could feed the output to your brokerage and have the trades executed automatically. Or you could look at the output and make decisions about whether you want to make the changes. Either way, you are no longer doing analysis yourself, and you're edging closer to your target, at which point you would be passive.
On the other hand, suppose you wanted an approximation of the S&P500 -- a target that would give you most of the same exposure and opportunity, but had a reduced number of individual stocks. You could run simulations on subsets of the S&P500 until you got a group that performed sufficiently similarly to the whole thing - 300? 250? 100? 50? as you reduce the set size you reduce the fidelity of the model - and then buy and hold those.
You can do a lot with an algorithm, and if your strategy doesn't need to operate in realtime, it will certainly look passive compared to HFT bots. A human might not want to rebalance more than annually or quarterly, because it can be a lot of work -- but a robot has no problems doing all the calculations daily and looking for a sufficient reward to present to you.
Thanks for the explanation, there are 2 things I don't quite understand though.
What is the advantage of building a replica of the S&P500 with individual stocks rather than having it done with an index fund? Isn't the second option much cheaper?
Secondly, isn't finding a subset of the S&P500 and trying to replicate it still an active strategy?
What is the advantage of building a replica of the S&P500 with individual stocks rather than having it done with an index fund? Isn't the second option much cheaper?
Secondly, isn't finding a subset of the S&P500 and trying to replicate it still an active strategy?
Most investors are just like you and me. We have a basic portfolio which we needed managed cheaply.
But... huge amounts of assets are held by people/companies that may have odd holdings (ultra concentrated positions) or odd needs (liability matching). Those people/companies may need algorithms to help them move their portfolio to an optimal portfolio with minimum cost.
But... huge amounts of assets are held by people/companies that may have odd holdings (ultra concentrated positions) or odd needs (liability matching). Those people/companies may need algorithms to help them move their portfolio to an optimal portfolio with minimum cost.
You can actually arb the SP500 components against the ETFs like SPY in a process that creates and redeems the ETF. This is how the ETF is kept in line.
I expect that there are tax advantages to holding individual stocks rather than an index fund.
Is finding a subset an active strategy? Depends. Do you do it every day, or do you figure out your subset and then buy and hold for fifteen years?
Is it passive when you rebalance against your target quarterly?
I think we can all agree that it's not passive when you are doing HFT, and anything which involves picking new stocks daily or weekly is active -- but having an algorithm do the rebalancing against your existing target daily and executing when a threshold is met? You're not making new picks, just readjusting against what you've already picked.
Is finding a subset an active strategy? Depends. Do you do it every day, or do you figure out your subset and then buy and hold for fifteen years?
Is it passive when you rebalance against your target quarterly?
I think we can all agree that it's not passive when you are doing HFT, and anything which involves picking new stocks daily or weekly is active -- but having an algorithm do the rebalancing against your existing target daily and executing when a threshold is met? You're not making new picks, just readjusting against what you've already picked.
One tax advantage to approximating an index with individual stock positions is "tax loss harvesting": sell one stock at a loss and buy an equivalent. e.g., sell Coke and buy Pepsi. An advisor told me that you can add ~1% to your after tax returns. You need sufficiently large positions such that rebalancing transaction fees are negligible.
> I expect that there are tax advantages to holding individual stocks rather than an index fund.
None that I am aware of. Mutual funds can force capital gains realization, but ETFs do not.[0]
IMO there's no reason to roll your own index fund when you can just buy an ETF with a very low expense rate. If you have enough money that rolling your own (costs a fair amount to manage) is cheaper than the public fee, you might be eligible for a special shareclass rate anyway.
[0]: http://www.investopedia.com/articles/investing/090215/compar...
None that I am aware of. Mutual funds can force capital gains realization, but ETFs do not.[0]
IMO there's no reason to roll your own index fund when you can just buy an ETF with a very low expense rate. If you have enough money that rolling your own (costs a fair amount to manage) is cheaper than the public fee, you might be eligible for a special shareclass rate anyway.
[0]: http://www.investopedia.com/articles/investing/090215/compar...
Here's one: as a US citizen living in Canada, I'm unable to invest in ETFs in Canada (e.g. Vanguard Canada's S&P500) without suffering punitive taxes and onerous reporting to the IRS. If it's an active investment, then it's fine, it's taxed as normal.
Unless you're playing with a million dollar, the fees for rebalancing are more expensive than any returns you might make. Especially if you have to trade hundreds of funds separately.
Don't rebalance quarterly either. Index funds are long term, meaning years.
Don't rebalance quarterly either. Index funds are long term, meaning years.
Rebalancing ETFs is free or included in the price of a roboadvisor.
>With all these people passively investing, are we going to see reduced competition within industries...
I read an article a while back, wish I could find it. It was about how CEO compensation is often tied to absolute performance as opposed to performance relative to the whole industry and how passive management actually prefers this so I'd say yes.
>How good is price setting if there are less and less actively managed funds? Are there new inefficiencies that are created by all this passive investing?
Well Shiller PE is getting up there. I think this is super interesting because in our next correction or perhaps even now, as we move to an increasing rate environment, we will get to see some analytical minds actually beating the market. It's gonna be interesting.
I read an article a while back, wish I could find it. It was about how CEO compensation is often tied to absolute performance as opposed to performance relative to the whole industry and how passive management actually prefers this so I'd say yes.
>How good is price setting if there are less and less actively managed funds? Are there new inefficiencies that are created by all this passive investing?
Well Shiller PE is getting up there. I think this is super interesting because in our next correction or perhaps even now, as we move to an increasing rate environment, we will get to see some analytical minds actually beating the market. It's gonna be interesting.
Social aspect of this are also interesting. We use the stock ticker as a reward and punishment for the companies involved, so introducing a new product that sells in unexpectedly large quantities generally leads to the stock price going up, while any bad news, e.g. pollution, poisoning, product recall, illegal behavior by the exec generally lead to a sell-off.
In an index-centric world if neither of the events cause the company to leave the index, I wonder if we'll still see such price adjustments.
In an index-centric world if neither of the events cause the company to leave the index, I wonder if we'll still see such price adjustments.
The really interesting social aspect, to me, is that we have a huge number of people in the US that are concentrated in the financial sector around active management. What happens to all those people? Are we about to see a huge downsizing of the financial industry?
Is the number huge, though? Retail stock-picking is done through mutual funds and active ETFs, which are typically owned by a larger outfit like Blackrock or Franklin Templeton, so the active stock picking is done by a select few managers of the fund. If that goes away, Blackrock or FT can still sell indexes and variety of bond funds.
On the hedge fund site extreme churn is fairly typical and considered part of the job. Well-known pickers (known mainly due to their windfall successes of the past and survivor bias) like Stephen Cohen, Carl Icahn and John Paulson are motivated less by the need to live paycheck to paycheck and more by the gambler's high.
On the hedge fund site extreme churn is fairly typical and considered part of the job. Well-known pickers (known mainly due to their windfall successes of the past and survivor bias) like Stephen Cohen, Carl Icahn and John Paulson are motivated less by the need to live paycheck to paycheck and more by the gambler's high.
Computer stock pickers look at balance sheets, p/l statements, etc. if anything the markets will become more rational and less prone to bubbles.
Computer stock pickers generally don't view balance sheets, P/L statements and other fundamentals data in isolation - they match them up temporally to historical market data and try to model what the effect of future financials releases will be on the price of the relevant security. If markets are historically irrational w.r.t. fundamentals, due to human falliability, or whatever else, our machines seem destined to repeat our mistakes.
Is a mistake still a mistake when it is a profitable action?
If fitting to human irrationality increases generalization performance, then it does not matter if the "machines seem destined to repeat our mistakes", it is still a useful signal. If fitting to human irrationality decreases generalization performance, your algorithm is overfit to noise (and you have bigger fish to fry than human irrationality).
Overfitting to noise is perfectly avoidable, not pre-destined when part of your data is noisy (noisy data is the rule not the exception).
If fitting to human irrationality increases generalization performance, then it does not matter if the "machines seem destined to repeat our mistakes", it is still a useful signal. If fitting to human irrationality decreases generalization performance, your algorithm is overfit to noise (and you have bigger fish to fry than human irrationality).
Overfitting to noise is perfectly avoidable, not pre-destined when part of your data is noisy (noisy data is the rule not the exception).
Why? If the robot sees a historical parallel to a popped bubble---the robot might anticipate the popping short-sell now?
Algorithms look at the very same things humans look at. They will try to time the market and speculate as much as now or even more.
It's not a case of exclusive or. Gather a list of a 100 or so human stock pickers. Train a model to predict the effect of following their suggestions (buy, hold, sell) to both identify which experts are most right, and which particular experts do well for particular kinds of stocks. This model will do better than random guessing, and quite likely, better than any individual stock picker.
Algorithms that predict the past accurately but are no better than random at the future are a dime a dozen, and I don't see how your suggested algorithm is different.
Then look at backtesting. The evaluation data set is out of time, meaning the better than random performance is on unseen future data. The algorithm was implemented, not merely a suggestion.
You can still leak information from your backtested time series by choosing WHICH algorithm to use out of a large pool of algorithms. You'll get regression to the mean because you optimized for a noisy signal (backtesting performance) of future earnings.
It is possible to leak information, but then you are doing it wrong. Don't use only a single out of time test set to do parameter or model selection, keep an out of time holdout set.
But really, this is the bare basic of forecasting. It is somewhat annoying to have to regurgitate all of this: Like non-leaking forecasting is impossible somehow. It would be a better discussion if everyone just assumes proper forecasting practices. Instead people seem to assume I have no clue what I am doing, discarding my technique, because I did not mention removing duplicates, scaling, proper validation techniques, ... and a 100 other things, which are of no importance to the technique itself.
But really, this is the bare basic of forecasting. It is somewhat annoying to have to regurgitate all of this: Like non-leaking forecasting is impossible somehow. It would be a better discussion if everyone just assumes proper forecasting practices. Instead people seem to assume I have no clue what I am doing, discarding my technique, because I did not mention removing duplicates, scaling, proper validation techniques, ... and a 100 other things, which are of no importance to the technique itself.
Corollary: having a time machine that can go a few minutes back in time is worth much less than one that can go a few minutes forward in time.
This implies the time machine can go only one way. Taking information from the future back in time is worth more than taking information from the current to the future. We do the latter all the time, we do not need a time machine for this, just patience.
So you're basically saying the model should identify which human stock pickers to follow for each sector?
If it's so easy why not to identify directly which stocks to invest in?
If you can predict which investors will perform well, then just try to predict which stocks will perform well directly.
If it's so easy why not to identify directly which stocks to invest in?
If you can predict which investors will perform well, then just try to predict which stocks will perform well directly.
> If it's so easy why not to identify directly which stocks to invest in?
It is not easy. Both to set-up, and the problem itself (you won't get a very high accuracy, but you will get much better than random guessing).
> If you can predict which investors will perform well, then just try to predict which stocks will perform well directly.
This won't work, because you don't have access to all the information that the stock pickers have access to, just their advice, and some features about the stock / the company the stock pickers work for.
It is not easy. Both to set-up, and the problem itself (you won't get a very high accuracy, but you will get much better than random guessing).
> If you can predict which investors will perform well, then just try to predict which stocks will perform well directly.
This won't work, because you don't have access to all the information that the stock pickers have access to, just their advice, and some features about the stock / the company the stock pickers work for.
It's not so much machines getting better. It's investors switching to low cost passive funds rather than giving 2% a year to pay for the managers next vacation home.
Exactly. These algorithms are not "picking" stocks so much as tracking stocks.
This is a story about investor behavior.
This is a story about investor behavior.
Reminds me of the story of the manager of CALPERS (California's public workers pension fund). Can't find the article but it focused on how hard work it is for a moneymanager to do nothing. CALPERS is showing good return for it's inactivity.
He's in Nevada actually
https://www.fnlondon.com/articles/nevada-pension-fund-manage...
https://www.fnlondon.com/articles/nevada-pension-fund-manage...
Hah! That's why I couldn't find it! Could have sworn it was about CALPERS. Thanks! It was in some other publication thou (if my memory isn't Swiss Cheese).
Yeah there were a few articles about it. The first one I found when googling just now was WSJ but that's fully pay-walled at this point
Interesting to note that BlackRock's Aladdin software platform, powering much of this stuff, is being largely written / re-written in Julia.
https://juliacomputing.com/case-studies/blackrock.html
https://juliacomputing.com/case-studies/blackrock.html
Does Julia have good substitutes for Numpy / Scipy / sk-learn / Pandas / Matplotlib / Keras / Tensorflow / Theano yet either natively (for Numpy) or from libraries (for others)? Ease of analysis being almost at the level of matlab is why I stick with python. I would love to be able to switch to something that doesn't have such a heavy handed approach at the top.
If I understand Julia correctly, "Numpy" is built into the core language - i.e. the basic Julia array class is fast like Numpy arrays and does what they do.
As for equivalents to Pandas, Matplotlib etc., Julia has thin wrappers around the Python versions of these so you can use them just like in Python with the same performance.
As for equivalents to Pandas, Matplotlib etc., Julia has thin wrappers around the Python versions of these so you can use them just like in Python with the same performance.
Julia can simply use python libs
> I would love to be able to switch to something that doesn't have such a heavy handed approach at the top.
Meaning?
Meaning?
[deleted]
Not surprising.
Just take any function you write and write code_native( your_function, () )
Knowing that the JIT is actually executing means a lot to me. For investment banks this means lower cost (quants can write code that goes into production) and lower latencies.
Just take any function you write and write code_native( your_function, () )
Knowing that the JIT is actually executing means a lot to me. For investment banks this means lower cost (quants can write code that goes into production) and lower latencies.
Any idea what it was written in before?
[deleted]
C++ and Java - including some Scala I believe.
A lot of Perl before Java came along.
Java primarily.
This is for mutual funds, which are ultimately intended to operate under given parameters -- balancing a cocktail of risk, reward, timeframe etc. Seems natural that you'd eventually move to algorithms for this, particularly as information asymmetry drops or is negated by other factors.
This is really now becoming a form arbitrage between the mega-funds on a sizable, but discrete, chunk of the trading market. Probably the same chunk that's already reserved for market-makers who can (usually) succeed through volume and brute force alone.
I think this is a good thing - arbitrage is a terrific leveller for the rest of us. It opens up the real market for us small fry.
This is really now becoming a form arbitrage between the mega-funds on a sizable, but discrete, chunk of the trading market. Probably the same chunk that's already reserved for market-makers who can (usually) succeed through volume and brute force alone.
I think this is a good thing - arbitrage is a terrific leveller for the rest of us. It opens up the real market for us small fry.
Not to be skeptical, but Blackrock was never known as a stock picker's shop, it's a fixed income shop. The equities side has largely been a function of their indexed ETF biz.
If Fidelity or TR Price were going all in machine based stock picking, then this would be news.
If Fidelity or TR Price were going all in machine based stock picking, then this would be news.
Reminds me of this art project by Michael Marcovici where they 'taught' rats to trade. Sadly no longer online. https://web-beta.archive.org/web/20140220072141/http://www.r...
That was bizzare but interesting. Looking for more on it now. Thanks for sharing.
Hopefully, we can eventually cut down on fees with those mutual funds too.
You don't have to worry so much about algorithm engaging in insider trading or some other securities fraud. The auditors just have to look at the algorithm's data sources to verify the fund stays compliant with securities regulations.
Fewer costs (hopefully) mean fewer fees
You don't have to worry so much about algorithm engaging in insider trading or some other securities fraud. The auditors just have to look at the algorithm's data sources to verify the fund stays compliant with securities regulations.
Fewer costs (hopefully) mean fewer fees
BlackRock isn't a whole lot more advanced than many other asset managers. So sure, Wall St. uses algorithms to trade, but that's not really news...
Separately, BlackRock bought FutureAdvisor so that it could use machines to do the opposite of picking stocks; i.e. rebalance assets using index funds. A very different proposition.
Separately, BlackRock bought FutureAdvisor so that it could use machines to do the opposite of picking stocks; i.e. rebalance assets using index funds. A very different proposition.
It will be interesting to see what happens when machines and algorithms really start coming for jobs on that level. I wonder if far down the line, will there be one sole human at the top of all of this automation, protected from its outcomes?
So... wait, this is what investment banks have been doing for literally over a decade, is this news? For legal accountability someone has to sign off on the orders but that's about it. The computers have been picking stocks for years.
exactly what i was thinking. does not seem remotely new.
Does the machine just run a random generator to pick the stock?
There's a back room with monkeys throwing darts somewhere :-)
[deleted]
I wonder how much of this trend is helped by the stock market going mostly up and sideways in the past years. In that case passive investing is very good, you need to minimize fees. Timing the exit before the bubble bursts again and re-entering at the right time will be a big part of the long-term performance and I expect many humans to outdo the robotic overlords who will just stay invested.
Hasn't the Aladdin system been doing this for a long, long time? Or was it actually semi-automated?
Aladdin simply provided portfolio managers with data. it did not pick stocks. the onus was on the portfolio managers to pick the best stocks given that data.
Thank you. I wssn't clear on it.
The real questions for me with this transition though are:
-With all these people passively investing, are we going to see reduced competition within industries because you don't have active, non-diversified, significant, institutional shareholders to drive aggressive competition against other industry participants? In fact, we may get the opposite where institutional shareholders don't want to see aggressive competitive moves because it is profit destroying for them on both sides (aggressor spends money to return less than the defender loses in profit--lose, lose for investor in both companies win, win for the consumers of this industry).
-How good is price setting if there are less and less actively managed funds? Are there new inefficiencies that are created by all this passive investing?
-Where aren't the machines taking over? Obviously, something like replicating an index is a great exercise for automation and programming. But truly maximizing return? Who is successfully, consistently beating the benchmarks with active strategies automated or not?