solaxun · HackerTrans

2 years agodiscuss

On the latency of the first request - How is the CFG cached?

Is it done at the API Key + schema level? Meaning that for a given API key, the latency penalty for a new schema is only paid one time, regardless of how far apart requests are? Or is cached with less duration, e.g. each session, conversation thread, etc?

solaxun

3 years agodiscuss

I've read that SFT is good for "leveraging existing knowledge" gained during initial pretraining, and helpful in changing the way that the model responds, but not useful for teaching it new knowledge. In your experience is that true?

For example, changing the way in which it responds could be:

  - debate me
  - brainstorm
  - be sarcastic

Which also seems like something that could be accomplished with a system prompt or few shot examples, so I'm not sure when SFT is the more appropriate approach or what the tradeoffs are.

Alternatively, gaining new knowledge would be training it on a dataset of e.g. sports trivia to make it highly effective at answering those types of questions.

P.S. nice username... Irving Fisher would approve.

solaxun

3 years agodiscuss

fine tune the chat or base model?

solaxun

3 years agodiscuss

Tough to answer since I've only read the two intro chapters currently available. I'm still in the early stages myself with with Deep Learning but have years of experience with programming and lisp, so I'm coming in with a background that might bias my views.

If you have never used lisp, you need to be patient with the notation and resist the urge to balk at what is unfamiliar. I remember grumbling quite a bit when I first went through TLS, but that phase was over pretty quickly and within days I had no trouble following the code.

Are there better ways to introduce deep learning and neural networks? Maybe.. but I like "little" books and there is no better way to learn something than by building it yourself. For that reason alone I'd recommend the book sight unseen (but with the knowledge of prior little books). I do think choosing what is likely an unfamiliar language for most may be somewhat of an impediment for an already challenging topic, but scheme is a simple language and allows the author to focus on the ML concepts.

I'm highly confident you'll learn a lot, if you put in the work. For DL as a separate topic, the best resources I've found are:

  - 3 blue 1 brown Neural Net playlist
  - Karpathy's "The spelled-out intro to neural networks and backpropagation"

solaxun

3 years agodiscuss

Books in the little series are all about "learning by doing", and teach using the Socratic method (question/answer). Each concept is introduced with small problems that build upon each other. You are forced to get your hands dirty from the beginning, passive learning is not really an option. For most people, I think this helps facilitate a much deeper understanding than just reading a text.

I always had trouble with recursive functions when I was new to programming, and many recommended working through "the little schemer" to solve that problem. It was a tough read for me, but the investment was well worth it and it did for me what it said on the tin. I didn't have nearly as much trouble with recursion after that book, but an unfortunate side effect was developing an affinity for lisps which I haven't yet shaken.

solaxun

3 years agodiscuss

Not sure on relative popularity. If you haven't already, you should try it! Here's why I like it:

  - Very consistent, easy to keep the whole language in your head
  - Stable, core rarely changes ("accretive only" mindset).
  - REPL
  - Focus on a handful of data structures which will solve 95% of the problems you face, rather than a deluge of complicated abstractions.
  - Excellent concurrency primitives
  - Access to Java libraries if a Clojure equivalent doesn't exist
  - Macros

The only blemish for me was horrifically bad error messages.. other than that it's near perfect.

solaxun

3 years agodiscuss

Nothing little about a 450 page book, but still... I have to buy it.

solaxun

4 years agodiscuss

I think the point is that given typical demand, the amount of supply would have been sufficient to not cause meteoric price increases. Demand exploded due to incompetent monetary policy driving a speculative mania, and that demand was what gave the appearance of a supply constraint. Until recently housing starts were at a 10 year high, and they are still on the higher-end of the past decade: https://fred.stlouisfed.org/graph/?g=Rzky

solaxun

4 years agodiscuss

I'm not sure I understand the point being made here. Are you simply saying that higher rates means a higher monthly payment, and that the amount that prices have fallen is still not enough to offset the increase in the mortgage payment? That's fairly obvious.... for now.

The old "it's a supply constraint" argument will not age will IMO. FWIW, the fed agrees:https://www.federalreserve.gov/econres/feds/volatility-in-ho...

solaxun

4 years agodiscuss

15 years ago was a different animal because the money largely stayed in the banking sector, as excess reserves. This time, a substantial portion of it was deposited directly into consumer's accounts (stimulus checks) - and that money makes it's way into the real economy, pushing prices up.

solaxun

4 years agodiscuss

Out of curiosity (not much experience with Actor systems), do these systems solve a different problem than having disparate processes communicate by sticking a distributed message queue between them? From my naive point of view it seems like alot of the scaling and fault tolerance concerns nowadays can be solved in any language, by having a queue be the interface between two services.

Same question applies to Erlang, which I realize is not exactly an actor system as per the below comment, and has a much more sophisticated error recovery story with supervisors, but the general question holds.

solaxun

4 years agodiscuss

I always found it strange that I really like salmon sushi/sashimi, but don't really care for cooked salmon. For some reason cooked salmon tastes too... heavy. I realize that's probably the fat content, but for whatever reason it doesn't bother me raw.

solaxun

4 years agodiscuss

The one they built, apparently.

solaxun

4 years agodiscuss

So I guess that means handling SWIFT MT / ISO 20022 messages is a big part of the custom stack?

solaxun

4 years agodiscuss

That's an interesting question... for a lot of fintechs that "wrap" banks in a unified API, if something goes wrong on the app side the bank is not going to be liable. This is a bit different since they are actually the bank themselves.

solaxun

4 years agodiscuss

I understand your concerns, but the alternative (at least in the U.S.) of credit agencies are absolutely abysmal. Despite the privacy concerns it's difficult for me to imagine more data not improving the current state.

For 10 years I had to fight one of the 3 major U.S. credit bureaus to get them to remove a bankruptcy that was showing on my report because a relative with the same name happened to have a bankruptcy in their past. Think about how incredibly stupid that is - I was 18 at the time, so according to this report I started a business at 12 and filed Chapter 11 at 16. I had a different social security number, address, and DOB that should have made things painfully obvious, yet still the burden of proof was on me to show why this report was false. Oh, and you can only challenge it by snail-mail (at least back then). If you don't get a response, there is no number to call, no address to visit, and practically speaking no way to challenge the agency short of hiring a lawyer.

Not to mention the obvious problem of credit bureas relying on length of history as a primary determinant of credit-worthiness. I don't like debt so I didn't get a credit card until 28, and did so then only because I knew I needed to build history if I ever wanted a house. If you are a responsible steward of your finances and choose to avoid debt in lieu of paying with savings, then that lack of history may preclude you from getting a loan in the future. You could have a $2mm net worth but that credit history will still be a problem with traditional lenders.

It's worse for immigrants - the U.S. credit agencies often don't interface with their home country's agencies. Imagine a responsible 60 year old adult with a pristine record in their homeland, viewed as a risky borrower by the bureas here due to lack of history. Another problem is not having a large enough number of credit accounts. I don't need more than one card, so I just use one card, but that's a negative on my report because the agencies wan't to see multiple "revolving accounts" (credit cards) holding a balance. You have to... wait for it... have more debt to show that you're good at managaging debt.

I could go on at length about how shitty the 3 main bureas are, and if this comes off as ranty it's because, if it's not obvious, I absolutely despise them and would love nothing more than to see them relegated to the graveyeard in short order.

solaxun

4 years agodiscuss

I think the idea is that once you've been counting long enough, you know the reduction that you need to make intuitively. Of course weight will come back if you revert to old habits, but I don't think counting calories forever is necessary.

If you're hungry between meals, eat low calorie density foods like broccoli and drink water. I realize that's not exactly as fun as eating doritos, but it's how sustainable change happens. Fad diets never work because they are by definition temporary.

solaxun

4 years agodiscuss

This is the only correct answer.

I've put on, and lost, 25 lbs probably 8+ times in my life (intentionally). I weigh 180 at my heavier end so that's a meaningful percentage for me. Depending how disciplined you are able/want to be, the easiest and most consistent way is to count calories.

If you don't want to be that strict, then just cut either your portions by 1/4 or remove a meal/snack that you estimate to be 250-500 calories. For me I just cut breakfast since I'm not really a breakfast person. Some people like to call this "intermittent fasting" and will give you an entire lecture about it's benefits. I call it "skipping breakfast".

Assuming you've cut 250-500 calories per day, weigh yourself every morning before any food/water. You should be losing 1-2 lbs per week, but it can fluctate so I would really look at 4-8 lbs per month as a reasonable target. If you are not losing that weight, you didn't cut enough calories, remove another 250/500 and try again.

On top of this do weight training and cardio, the type doesn't matter.

Good luck!

solaxun

5 years agodiscuss

This is my main qualm with testing as a means of validating understanding. If you truly understand a topic, you will probably pass a test on it. However passing a test most definitely does not guarantee understanding.

I remember very, very little from the CFA exams (passed L3 about 10 yrs ago), because it tested surface level knowledge of an incredibly broad array of subjects. Far too broad to allow a deep-dive in any one thing. At the time, I could fit a bunch of formulas in my head and regurgitate them on command, but it was like filling up a leaky bucket. I had to load that sucker up and run into the exam center before it poured out.

IMO testing well is it's own skill, and has little to do with understanding a topic. Understanding is when you've forgotten all the stuff you've memorized, but you can work your way back to a solution from first principles. I think the only way to get to that point is genuine interest and sustained study.

comments