Analyzing Cryptocurrency Markets Using Python(blog.patricktriest.com)
blog.patricktriest.com
Analyzing Cryptocurrency Markets Using Python
https://blog.patricktriest.com/analyzing-cryptocurrencies-python/?utm_source=hackernews
55 comments
Thanks for the tips! I'll update the tutorial code with that info in the next day or so. Until then, I've just added a note warning about the bias, and recommending the use of the ".pct_change()" method.
I think it helps to "step out" with correlation/regression to understand why something may be spurious or why you get the high correlation values that you do. In some cases, there is no logical connection between variables which leads to spurious regression. However, in this case, if you "step out", the reason for the correlation is pretty obvious.
There has been a lot of money pouring into crypto recently because most people are speculating on the space as a whole. Bitcoin's price is too high for smaller investors to make a significant amount of money on, but when a big Bitcoin move makes the news, those investors want a piece. They then pour money into the smaller coins, hoping to get a larger return on their investment.
All of this is to say that I think this is the opposite of spurious correlation. However, that doesn't make the correlation meaningful in any way. When ETH or BTC jumps and makes the news, the other coins tend to follow because the whole space is speculative right now.
There has been a lot of money pouring into crypto recently because most people are speculating on the space as a whole. Bitcoin's price is too high for smaller investors to make a significant amount of money on, but when a big Bitcoin move makes the news, those investors want a piece. They then pour money into the smaller coins, hoping to get a larger return on their investment.
All of this is to say that I think this is the opposite of spurious correlation. However, that doesn't make the correlation meaningful in any way. When ETH or BTC jumps and makes the news, the other coins tend to follow because the whole space is speculative right now.
what does "step out" mean?
(As a quant) when calculating correlations of returns, we use log changes.
Rather than smoothing, have you tried looking at the correlations of the residuals?
This would be more appropriately titled "downloading, cleaning, and plotting cryptocurrency price data using pandas and plotly in a Jupyter notebook."
TL DR the analysis is just a couple of time series correlation coefficient heatmaps.
That being said this is a great tutorial for people just starting out with data handling and analysis in pandas.
TL DR the analysis is just a couple of time series correlation coefficient heatmaps.
That being said this is a great tutorial for people just starting out with data handling and analysis in pandas.
It's great how readily available financial data is with cryptocurrencies! As a complement to this post, I've been recently working with Jupyter notebooks to analyze high-frequency trading activity in Bitcoin markets [1]. I've been listening to the GDAX socket since end July, so I have almost a month's worth of tick data (~ 7GB+ gzip compressed JSON data, it surely explodes to ~100 GB after extracting). If somebody is interested to carry out further analysis I can give you a link to download it.
[1] https://nbviewer.jupyter.org/github/ghgr/HFT_Bitcoin/blob/ma...
[1] https://nbviewer.jupyter.org/github/ghgr/HFT_Bitcoin/blob/ma...
What are you using to listen to GDAX sockets? I've been able to successfully connect using their node library but when trying to connect directly without using their library I'm unable to connect.
I use the Python library WebSocketClient. Here is the code:
https://github.com/ghgr/HFT_Bitcoin/blob/master/download_boo...
Thanks for the code. I've been trying with the Elixir library websockex [1] but it has been giving me nothing but troubles. I just gave up and poll every X minutes now.
[1] https://github.com/Azolo/websockex
[1] https://github.com/Azolo/websockex
very interesting. do you have a contact email somwhere?
sure! I just updated my profile
(Shameless self-promotion)
You can watch a couple dozen the Bitcoin markets trade in real-time all on one chart with my site: https://bitcoin.clarkmoody.com/tickers/
It gets very interesting when the price really starts to move, since all the markets tend to move in lockstep. The response time reveals how active the trading bots are, making sure to reduce arbitrage opporunities.
You can watch a couple dozen the Bitcoin markets trade in real-time all on one chart with my site: https://bitcoin.clarkmoody.com/tickers/
It gets very interesting when the price really starts to move, since all the markets tend to move in lockstep. The response time reveals how active the trading bots are, making sure to reduce arbitrage opporunities.
I'm surprised they didn't mentioned Machine Learning to further analyze the cryptocurrencies. Especially, that a few years ago an experiment shown, that if we would trade algorithmically on S&P500 via Machine Learning, traders could earn 8.5% returns comparing to the 5.6% by a random tactic. Here is a nice explanation: https://sigmoidal.io/machine-learning-for-trading/
I've started dabbling with R (started with Python and just got fed up with it and switched to R) and ML to analyze bitcoin from GDAX over the past few days. Coming from a non-stats background (I did time-series analysis in undergrad ala Econometrics but that was over a decade ago) I've had issues trying to create meaningful models. With packages like Caret[1] it is easy to slap code together and throw factors at the algorithms to see what sticks. The problem is that the models that are generated typically suck (I think the best RMSE I was able to get was around 12 and that was with linear regression model)
ML can definitely help trading by using things like Ensemble Learning[2] but I would argue that for the novice it isn't going to add much to a trading strategy than doing other analysis on their own would do.
[1] https://cran.r-project.org/web/packages/caret/index.html
[2] https://en.wikipedia.org/wiki/Ensemble_learning
ML can definitely help trading by using things like Ensemble Learning[2] but I would argue that for the novice it isn't going to add much to a trading strategy than doing other analysis on their own would do.
[1] https://cran.r-project.org/web/packages/caret/index.html
[2] https://en.wikipedia.org/wiki/Ensemble_learning
[deleted]
I wanted to do something like that, eventually I did it on the elasticsearch.
Then I wanted to share with the society and added grafana.
The final result is real-time analytics of trading on main exchanges and for major pairs: https://cointradeanalysis.com
The final result is real-time analytics of trading on main exchanges and for major pairs: https://cointradeanalysis.com
Wow, that is very cool.
Some feedback/ideas:
1) things like xrp_usd need more decimal places for price (Max price 0.16, Min price 0.15, you really need at least 4 decimal places.)
2) What you call "Amount" should really be called "Volume".. "Count orders" should be "Count Trades" or perhaps just "Trades".
3) The buy/sell graph: i would place the buy/sell bars beside each other (instead of having sell underneath pointing down). This would make it much easier to see which is higher, buy or sell.
Some feedback/ideas:
1) things like xrp_usd need more decimal places for price (Max price 0.16, Min price 0.15, you really need at least 4 decimal places.)
2) What you call "Amount" should really be called "Volume".. "Count orders" should be "Count Trades" or perhaps just "Trades".
3) The buy/sell graph: i would place the buy/sell bars beside each other (instead of having sell underneath pointing down). This would make it much easier to see which is higher, buy or sell.
1) I'll try to fix it.
2) Corrected, thank you very much.
3) The first option was like you describe, but something I did not like about it - I'll think about it :)
2) Corrected, thank you very much.
3) The first option was like you describe, but something I did not like about it - I'll think about it :)
Your site breaks the back button. Please don’t do that. Otherwise, it’s well done.
Really... I did not notice this before...
Yep. Not sure if it’s just mobile, but after visiting and scrolling, it was impossible to swipe to leave, and just kept reloading the page. Forever.
Interesting site. It would be an interesting exercise to try and arbitrage the price differences of BTC on different exchanges.
[deleted]
Cool! Bookmarked.
Don't think I've ever really seen the full $BTCUSD chart since inception like that. More than slight resemblance to Nasdaq-100 chart circa late 1999-early 2000. Just saying, exercise caution out there ;)
Next step: prediction! As an active research subject, crypto-currencies may be the ideal candidate for using deep learning to forecast non-stationary time series data.
Theory and Algorithms for Forecasting Non-Stationary Time Series
http://www.cs.nyu.edu/~mohri/talks/NIPSTutorial2016.pdf
Next step: prediction! As an active research subject, crypto-currencies may be the ideal candidate for using deep learning to forecast non-stationary time series data.
Theory and Algorithms for Forecasting Non-Stationary Time Series
http://www.cs.nyu.edu/~mohri/talks/NIPSTutorial2016.pdf
I'm a big fan of Stellar. It looks like the only cryptocurrency with an actual use case that works today and minimises costs for companies[1][2] building on top of it. Everything else looks like mostly hype driven. I would say that this is one of the reasons it trades so different from everything else, except Ripple.
As many developers hang out here I want to point out that the Stellar Foundation has a rolling competition for developers: https://www.stellar.org/lumens/build/. You can see here all the projects submitted to the last round: https://galactictalk.org/t/sbc2017april
[1]: https://tempo.eu.com/en [2]: http://chippercash.com/
As many developers hang out here I want to point out that the Stellar Foundation has a rolling competition for developers: https://www.stellar.org/lumens/build/. You can see here all the projects submitted to the last round: https://galactictalk.org/t/sbc2017april
[1]: https://tempo.eu.com/en [2]: http://chippercash.com/
Could you help me to put your opinion in the right context by answering a few questions?
- Do you own Stellar Lumens?
- Are you involved in any Stellar related projects?
- Do you own Stellar Lumens?
- Are you involved in any Stellar related projects?
I won Lumens in the previous build challenges, run a community forum and are building a company that uses the Stellar network.
I'm not trying to give investment advice in my post. You should definitely not be buying Lumens if you don't need them. Most of them are going to be given away: https://www.stellar.org/about/mandate/
I'm not trying to give investment advice in my post. You should definitely not be buying Lumens if you don't need them. Most of them are going to be given away: https://www.stellar.org/about/mandate/
Thanks, I appreciate your openness.
Is that the thing Stripe (or someone) launched years ago? The one where they were giving 50 coins free per subscriber?
Yes, Stripe invested into Stellar. https://stripe.com/blog/stellar
Semi-related question:
I got some of those ICO coins back then - i registered with username/password back then
Still have username/password but i am unsure where to actually login? Anyone familiar w/ Stellar knows how to get to the account?
I got some of those ICO coins back then - i registered with username/password back then
Still have username/password but i am unsure where to actually login? Anyone familiar w/ Stellar knows how to get to the account?
I just went through the same process – had some coins from the ICO a couple of years ago.
If you head to https://launch.stellar.org/#/login you can log in. I got a message prompting me to contact Support, which I've done and now awaiting a reply.
If you head to https://launch.stellar.org/#/login you can log in. I got a message prompting me to contact Support, which I've done and now awaiting a reply.
Same here, it asked me for a confirmation code over email, but I received nothing, so I'm waiting for Support.
Is it possible to get data on economic activity in cryptocoins? Which is to say, not just the coin trading, but goods and services being paid for with coin tokens? Every now and then you see a post to the effect of 'so-and-so is taking payment in Bitcoin.' Does anyone actually do that? Or are the coins just hoarded for their investment value?
At the moment, there aren't a ton of real-world use cases aside, but the future will be very different!
Check out Open Bazaar for an entire market place that's facilitated via cryptocurrencies.
Check out Open Bazaar for an entire market place that's facilitated via cryptocurrencies.
What's a good resource to learn the technical statistical background for an analysis like this? It's beyond the basics of calculating P values but it also builds on the basics by analyzing a large space of correlation coefficients.
Would this be considered data science? It doesn't dive into machine learning, or anything advanced computer science wise.
Would this be considered data science? It doesn't dive into machine learning, or anything advanced computer science wise.
Layering, Spoofing and Momentum Ignition are not "HFT Strategies." They're illegal market manipulation techniques, most often used by manual traders. You don't need to be ultra fast to bully prices around or enter large non bonafide orders. I've never heard of a legitimate proprietary trading firm (i.e. one that pays a salary and hires highly qualified people, not a boiler room operation) intentionally using these techniques. Most are in the arbitrage, market making or statarb business.
It would be much more interesting to see the author dive into which events lead to fast price changes. How quickly are price shocks on other markets are reflected on GDAX? How are changes in outright contracts like BTC/USD and ETH/USD reflected in ETH/BTC? Do USD denominated pairs drive price discovery more or less than EUR denominated ones, and does this change when US based traders are asleep? Lots of stuff to look at here.
It would be much more interesting to see the author dive into which events lead to fast price changes. How quickly are price shocks on other markets are reflected on GDAX? How are changes in outright contracts like BTC/USD and ETH/USD reflected in ETH/BTC? Do USD denominated pairs drive price discovery more or less than EUR denominated ones, and does this change when US based traders are asleep? Lots of stuff to look at here.
If you're looking for malicious actors in the wild, crypto trading is where you want to be. There's all sorts of things that are ridiculously illegal on proper security exchanges that are de rigeur in cryptos. c.f. Mt. Gox admitting to painting the tape in 2015, a Bitfinex officer with access to the database admitting to trading on his exchange in 2015. A bot set up to detect that sort of thing would have plenty of data to work with.
Directly from Wikipedia [1]:
> In July 2013, it was reported that Panther Energy Trading LLC was ordered to pay $4.5 million to U.S. and U.K. regulators on charges that the firm's high-frequency trading activities manipulated commodity markets. Panther's computer algorithms placed and quickly canceled bids and offers in futures contracts including oil, metals, interest rates and foreign currencies, the U.S. Commodity Futures Trading Commission said.[109] In October 2014, Panther's sole owner Michael Coscia was charged with six counts of commodities fraud and six counts of "spoofing". The indictment stated that Coscia devised a high-frequency trading strategy to create a false impression of the available liquidity in the market, "and to fraudulently induce other market participants to react to the deceptive market information he created".[110]
I guess it's HFT if it's executed fast enough.
[1] https://en.wikipedia.org/wiki/High-frequency_trading#Strateg...
> In July 2013, it was reported that Panther Energy Trading LLC was ordered to pay $4.5 million to U.S. and U.K. regulators on charges that the firm's high-frequency trading activities manipulated commodity markets. Panther's computer algorithms placed and quickly canceled bids and offers in futures contracts including oil, metals, interest rates and foreign currencies, the U.S. Commodity Futures Trading Commission said.[109] In October 2014, Panther's sole owner Michael Coscia was charged with six counts of commodities fraud and six counts of "spoofing". The indictment stated that Coscia devised a high-frequency trading strategy to create a false impression of the available liquidity in the market, "and to fraudulently induce other market participants to react to the deceptive market information he created".[110]
I guess it's HFT if it's executed fast enough.
[1] https://en.wikipedia.org/wiki/High-frequency_trading#Strateg...
Panther was a one-man shop that did nothing but market manipulation. Yes, they used automation but for every Panther, there are 1000 manual traders being fined by CME for spoofing every year. Automation doesn't imply state of the art HFT. The Flash Crash spoofer was using a Trading Technologies system that any small trader can rent with a few scripts loaded in.
Look at real players with hundreds of employees. None of them make their money from market manipulation. It's all from low-latency arbitrage. If you're fast, you can make way, way more money doing legitimate trading than market manipulation.
Look at real players with hundreds of employees. None of them make their money from market manipulation. It's all from low-latency arbitrage. If you're fast, you can make way, way more money doing legitimate trading than market manipulation.
Nice and everything, but you can't make money with this, or can you?
What's the point of correlation ? If I put 2 buys either I will have double profit or double loss. If I put 1 buy 1 sell I get zero.
If two assets are correlated and one of them moves up but not the other, you expect the return of both over a longer time period to converge, you can make a statistical arbitrage by selling the one that moved up and buying the other c.f. https://en.wikipedia.org/wiki/Pairs_trade
If you are looking into diversification. You probably don't want to buy 2 1/1 correlated stocks or assets, since basically buying one of them will do the thing.
> What's the point of correlation?
Knowledge.
> If I put 2 buys either I will have double profit or double loss. If I put 1 buy 1 sell I get zero.
Yea, so it pays to know if something is correlated to avoid this. It also pays to know if something is correlated and their spread gets out of whack short term allowing you to profit from the expectation they'll return to being correlated.
Knowledge.
> If I put 2 buys either I will have double profit or double loss. If I put 1 buy 1 sell I get zero.
Yea, so it pays to know if something is correlated to avoid this. It also pays to know if something is correlated and their spread gets out of whack short term allowing you to profit from the expectation they'll return to being correlated.
Move your time scales way down and it becomes pretty useful.
Anyone having trouble with the link?
[deleted]
- it is not really pertinent to compute the correlation between prices. This takes the currencies trend into account since prices, contrary to e.g. "returns", are non stationary. This will lead to a biased higher correlation. Just do a ".pct_change()" before the correlation.
- also averaging the price between exchanges is a bit naive. It hides arbitrage opportunities and does not reflect the underlying traded volume. A VWAP would at least provide a better approximation.