You will need a combination of the two. The filename should contain a compound business identifier, essentially a tuple of appropriate facts, as well as a (truncated) hash of the content.
And yes, the reason is auditability but also bare necessity.
Imagine vendor A publishes fresh files every day, but you don't exactly know when. And suppose that the vendor can (and does) republish some of them at any time, with or without notice. You need to watch those files for changes and update your own sources accordingly.
For the collection and dedup, you can construct a tuple of meta-information that will give you a good idea if the file is the same (name, time last modified, file size, etc). Then only if you're still unsure do you download the file and hash it, compare with what you already have, and make a decision. (What if the file name changed slightly and now you must re-categorize the data in the file, or what if a single row was added or removed, etc).
It's also useful when auditing (tamper-proof) or when re-running or debugging calculations that used the data in question. If a user used the data available at 9AM, which was unfortunately incomplete and led to issues, having a full data lineage really helps. You can trivially re-run the computation with the information available at that time to see what happened. Or you can mark some results as stale, etc.
There are different ways to do this and one is keeping track of your content-adressed files in a relational table and give each row a unique identifier, so that you can tell that this particular row in your AAPL OHLCV data comes from this particular file which was fetched at this particular time etc, etc. Pair that ID with a timestamp and when doing computations you can query for "the latest" AAPL data while storing the exact file ID (time is fuzzy, content hash is not). Look into temporal tables and the like.
Nice. The book contains a bunch of good information that could already be found elsewhere but collecting it is quite practical. I highly suggest to read Kleppmann's Designing Data-Intensive Applications. The first edition was very good, a second one came out recently.
I was CTO of a FinTech where I built the whole software stack from scratch: the lessons in the book are mostly correct. I say mostly, because as always, there is a lot of "it depends" to take into consideration for your particular project. For example, I chose to not use event-sourcing to avoid the whole state computation issue. A standard append-only audit trail can do the job.
You can't guarantee exactly-once delivery but you can construct effectively-once processing, and that is what you really want.
Store every request and response : absolutely, and not only when consuming APIs, but when collecting any information from the outside world (and, if you can, also log every intermediate transformation step within your perimeter). Content-adressed buckets + a relational table are great for this.
The text also does not mention anything about data lineage. What happens if a vendor updates some data mid-day that you absolutely need to be aware of? You need to be able to account for that, while also re-playing computations that used the old values and get the same result. It's not a particularly hard problem to solve, but it takes some thought.
OK, sure the concepts might be harder than setting up an environment. But setting things up is, by experience, a very real, very annoying and time-consuming blocker. We have students come in with tablets and it "just works"!
Are these first-year undergrad courses? We can't assume students know anything about Docker or can even install packages on their systems. Some of them really have no idea at all, so pointing them to codeBoot saves a lot of time. Docker would definitely work for more experienced students though!
BTW, here's another quite nice feature of codeBoot: easy sharing! The following link will bring you to a suspended execution of 'print("hello, world!")' that you can then single-step.
Yes, definitely the case. Just imagine a class with 100 students, each with a system that has its own quirks (OS, browser, and a mix of versions, etc). We built codeBoot (https://codeboot.org) specifically for this reason!
Hey, this is super interesting! We have been working on a quite similar system over the past few years at Université de Montréal. We develop and use codeBoot (https://codeboot.org) to teach introductory programming courses there. We also wrote an executable book for teaching a Cegep course (think last year of high school in the US). You can find it here (https://420-sn1.codeboot.org/) but mind you, the English version is not completed past chapter 1. The book was written in French.
Our tech stack is different but the choices we made are quite similar. Multi-tiered platform, markdown for authoring, executable exercises, teacher platform to produce, give, receive and grade homework/exams, etc. A distinguishing characteristic is that codeBoot's Python interpreter (pyinterp) allows single-stepping through the code. That's quite useful for teaching and studying.
We have a few exciting features coming up and we're working on a proper landing page and clean English translation for the book. If anybody is interested to learn more, reply here or contact me (email in my profile). I'd love to connect with educators, students or hackers alike!
Willing to relocate: No unless very strong commitment.
Technologies: Python, JavaScript, Scheme, PostgreSQL, ClickHouse, compilers, interpreters, foreign function interfaces, FastAPI, Flask, Celery, RabbitMQ, Redis, Pydantic, SQLModel, technical strategy, team management (6+ devs + interns), SDLC best practices, etc
Résumé/CV: Sent by email
Email: See profile!
I have over a decade of experience working in various roles and industries, from healthcare to education to finance. My last role was CTO of a FinTech that I brought from zero to one, architecting and implementing all of the tech stack from compliance to distributed locks while managing a team of 6 developers.
I am looking for interesting problems and a good team. I strongly believe that good, thoughtful work pays dividends when things break or need to move fast. Boring, proven tech before shiny new thing.
I would say about 80% of the code was in "production". Obvious hiccups were fixed after a few weeks of running. The platform was collecting, transforming, distributing data as designed without any major issues. The financial code was heavily tested by a PhD Wall Stree quant. The less tested parts were the front-end and the swarm itself under load.
And to clarify: I didn't "forget" how to code. I froze for a moment, after relying on a tool to write for me for most of a year. More like rusting.
My first task at my last job was removing access to an employee being let go. I had just gone through onboarding so I knew every (documented) service we needed to handle. We live tested it on my own accounts, measured the time before I noticed, and then proceeded to successfully go through the checklist.
Except not everything was properly documented, and it turned out the employee had given admin rights on some resources to a contractor which proceeded to wreak havoc on their behalf (the 'rm -rf' kind). Eh!
I don't fully agree with the quote. You can't really outsource thinking nor understanding. You can outsource the generation of streams of tokens that may or may not be appropriate for what you're looking for. But you absolutely have to know what you're looking for, or have a very solid intuition of what it should look like and behave, otherwise you're just digging your own grave.
The skill is in making the LLMs reliably generate useful and pertinent streams of tokens. That takes work, reading the output, intuition, experience, rigor, real commitment to doing good work, not fall prey to being lazy, etc.
I have just stepped down from a CTO job where I built a FinTech's stack from the ground up. I leveraged a Claude Max plan for about 8 months and I can say with absolute certainty I would not have been as productive without it. I have barely written any code by hand during that time, but I did read almost every single line of code produced. My role was much more that of an "editor", as another article posted here mentioned. There is no doubt in my mind that you can be very highly productive with AI, it's just not the magical silver bullet some people market it as. I have a lot of notes describing the process that I'm considering publishing.
Just yesterday I was interviewing for a very interesting job and I completely flunked the coding question in an unacceptable way for my level of experience. The question was easy, I just couldn't get past some syntactic issues. For 8 months, Claude wrote all of my Python classes and Pydantic types. Now I had to write a dataclass, and because I always just resorted to standard classes before the advent of LLMs, I stumbled. And froze. And panicked. And that was it. Of course you could say I should have just scrapped the dataclass and written it as a simple class. The point is I felt very, very stupid. LLMs suddenly felt like a huge disadvantage.
All this to say I disagree with LLMs "rotting" my brain. Quite the opposite, I know that it's possible to use LLMs to be efficient and correct. It's more the actual mechanical act of writing that gets rusty.
Willing to relocate: No unless very strong client commitment.
Technologies: Python, JavaScript, Scheme, PostgreSQL, ClickHouse, compilers, interpreters, foreign function interfaces, FastAPI, Flask, Celery, RabbitMQ, Redis, Pydantic, SQLModel, technical strategy, team management (6+ devs + interns), SDLC best practices, etc
Résumé/CV: Sent by email
Email: See profile!
I have about 14 years of experience working in various roles and industries, from healthcare to education to finance. My last role was CTO of a FinTech that I brought from zero to one, architecting and implementing all of the tech stack from compliance down to distributed locking while managing a team of 6 developers.
I am looking for interesting problems and a good team. I strongly believe that good, thoughtful work pays dividends when things break or need to move fast. Boring, proven tech before shiny new thing. Sleeping at night is priceless.
And yes, the reason is auditability but also bare necessity.
Imagine vendor A publishes fresh files every day, but you don't exactly know when. And suppose that the vendor can (and does) republish some of them at any time, with or without notice. You need to watch those files for changes and update your own sources accordingly.
For the collection and dedup, you can construct a tuple of meta-information that will give you a good idea if the file is the same (name, time last modified, file size, etc). Then only if you're still unsure do you download the file and hash it, compare with what you already have, and make a decision. (What if the file name changed slightly and now you must re-categorize the data in the file, or what if a single row was added or removed, etc).
It's also useful when auditing (tamper-proof) or when re-running or debugging calculations that used the data in question. If a user used the data available at 9AM, which was unfortunately incomplete and led to issues, having a full data lineage really helps. You can trivially re-run the computation with the information available at that time to see what happened. Or you can mark some results as stale, etc.
There are different ways to do this and one is keeping track of your content-adressed files in a relational table and give each row a unique identifier, so that you can tell that this particular row in your AAPL OHLCV data comes from this particular file which was fetched at this particular time etc, etc. Pair that ID with a timestamp and when doing computations you can query for "the latest" AAPL data while storing the exact file ID (time is fuzzy, content hash is not). Look into temporal tables and the like.