rlh2 · HackerTrans

People also love to hate R but data.table is light years better than pandas in my view

3 yıl öncediscuss

I did not, mostly because the parsers are custom to how I want things displayed. Here is a simplified example for Chase, though:

  BEGIN{
      FS = ",";
  }
  {
      if(FNR > 1){
          date = gensub(/([0-9]{2})\/([0-9]{2})\/([0-9]{4})/, "\\3-\\1-\\2", 1, $1);
          print date, $3
          print "    liabilities:chase"
          if($5 != "Payment"){
              print "    expenses:" $4, "  $", $6 \* -1.0
          }else{
              ## This exists because I have an offset in the account
              ## I use to pay the card with. If the amount in my residual
              ## account != 0, I know there is something wrong
              print "    liabilities:chase:residual  $", -$6
          }
      }
  }

Suppose this script is called 'parse.awk', you would then run: awk -f parse.awk myfile.csv

I've been doing this for a couple years now and so far I haven't had file formats change on me. Most of the complexity I have experienced so far is when the regex to parse line-items into the appropriate ledger account is non-trivial. It started off as a way for me to learn awk and double entry bookkeeping and somehow turned into something useful.

rlh2

3 yıl öncediscuss

I tried this out and switched to ledger-cli instead. All of the features listed above are possible and the the flexibility is incredible. The hardest part is data entry. For accounts that provide API access I wrote curl/jq/awk parsers and for accounts that you can only get via download, I wrote short csv parsers in awk. I have ~ 15 accounts to keep track of and various loans/securities and I use ledger-cli to keep track of cost basis, interest vs principal on loans and my mortgage, all expenses, etc.

Took some time to write the transaction parsers, but updating everything takes about 5 min/month now and I have a complete financial picture without sending any data to something like mint.

rlh2

3 yıl öncediscuss

The obsession with cpu speed almost always confuses me in these topics. Time it takes to program is way more important, and that’s where a terse language like R shines. The base/most common functions are almost always executing C anyway. It’s kind of like lisp in that it’s easy to write slow code, but who cares if it’s “fast enough”? Also, it’s almost always easy to speed up if necessary at the R level and R’s C API is also easy to use for for numeric computing/optimization which is exposed at the C level if you want to use it.

rlh2

4 yıl öncediscuss

I have been using R for almost 20 years now. I work on a medium-sized quant team at a large asset manager and we run several $BN off R - we mostly trade equities and vanilla derivatives. Our models are primarily statistical/econometric-based. In aggregate, we probably have about a hundred scheduled jobs associated with a variety of models and on the order of 15 shiny applications to facilitate implementation. We have an internal CRAN-like repo and everything we produce is packaged/versioned with gitlab CI/CD. We have RStudio Server at my firm and half my team uses that for development, the other half, including myself, uses emacs/ess. All of us use RConnect for scheduling & application hosting - it has it's quirks, but it's excellent in a constrained IT environment.

I often chuckle when people complain about R in production and how it isn't a good general purpose programming language, my experience has been the polar opposite. You can write bad code in any language, and R is no exception, but R allows you to write so much less code and R-core is truly exceptional at backwards compatibility. Our approach to R is basically:

- Don't have a lot of dependencies, and when you do have dependencies, make sure they themselves don't have a lot of dependencies. While we do use shiny as mentioned above, our core models are very dependency light and shiny is just a basic front end.

- data.table (which was designed by quants) is a zero-dependency package that is by far the best tabular data manipulation package that has ever been created since the dawn of time. We generally work on an EC2 instance running linux with a ton of memory. In the < .01% of cases where a dataset doesn't fit in memory (e.g. tick data), we do initial parsing with awk if file based or SQL if DB based and then work in R.

- Check/coerce argument types and lengths on function input to catch and avoid all the quirky edge cases that drive people nuts - it's so easy!

- I hate OOP and I love that R doesn't encourage it. Mutable state, especially for non-software engineers, is the devil. Don't get me wrong, OOP has its place, but the fact that R encourages functional programming is one of the best things about it. The slight inefficiency this produces is almost never a problem.

- R is not slow at all when used correctly. Additionally, the C API is a joy to use when necessary.

- Stick to the base types: vectors, matrices, lists, environments and data.tables (only exception). The fact that you can name, and then use names to index all of the above is stunningly powerful. The only "objects" we really create are lightweight extensions of lists with an S3 print method.

- We have an internal version of renv/packrat that creates a plain text "dependency file" for projects and we pin package versions in docker containers. RConnect doesn't use docker right now, but they do have a versioning system that works quite well in my experience.

I definitely wouldn't want to build something like a company website in R, but I also wouldn't want to build that in C either. R definitely has it's place a server-side language even outside it's assumed domain of statistics.

Haters gonna hate, but joke is on them.

comments