Commenting just to go against the other two answers. I think it's fine to not remember things, no matter the apparent simplicity.
Quite surprised by others finding this as a... Surprise? I get there is people who never experience this, but they also not know anyone personally to whom this would happen?
What does data frames mean in this context? I'm used to them in spark or pandas but does this relate to something in how duckDB operates or is it something else?
Without any context of culture or country, just trying to be helpful: in my limited (<20 total interviews) experience, I would think about budget issues.
Meaning, what you ask for (or how expensive you are perceived, if you have that strong resumee) for the industry you apply, may be too different and leading to limited access.
Sometimes I feel junior people have it easier (I felt like I did, personally) since the expense in salary is pretty limited compared to either other roles or more senior people
It's been mentioned but I want to add that the original idea of the post (mid size VPS hosting apache spark) might be missing that spark is ideal for distributed and resilient work (if a node fails the framework is able to avoid losing that work).
If you don't need this features, specially the distributed one, going tall (single instance with high capacity, replicate when necessary) or going simpler (multiple servers but without spark coordinating the work) could be good options depending on your/the team's knowledge
Recently I have been thinking about this, because I feel I have managed to become way more organized than I ever thought it was possible.
What is working for me right now is noting everything in a calendar so I cannot forget it or as TODO in a somewhat heavy personalized Obsidian configuration.
A few years ago (5-6 aprox) I started copying my older co-workers habits to see myself improve. Physical notebooks were soon discarded because I never remember where I wrote down things.
I used a TODO plugin in sublime which worked for several months, until I felt I needed screenshots so I moved to OneNote. After a while I became frustrated with not being able to customize it enough, so I started trying out different things. I saw a coworker using Obsidian, watched a couple long YouTube videos to learn how to customize, and I'm never going back.
My team this week told me they are impressed with how much info I write down and it was a very proud moment for me!
PyIceberg is nice but we had to drop it because it's behind Java API and it's unclear when it will match up, so depending on which features are needed I'd look it up
As someone doing this transition, I would love some references that help me... Train myself I guess? Other than by doing and analyzing myself, which is my current situation
I have realized I can give so many tips and reference so many great content online to learn math, programming, engineering... But find myself missing anything about managing
Since I feel the same, and my habits are really similar to yours (I'm forcing myself to read more articles compared to when I used reddit), I want to comment something I felt lacked in the article analysis.
For me, what's missing is intent. Most of the time I feel I could be doing better, I stop and think: wait, this is my resting time. Or, wait, this is my hobby. I feel even though I fall in one end of the balance, many times this articles fall in the other.
If I tried to deep dive and study as much as is suggested, I'd have no energy for other things that I value more right now. Reading author articles it seems to me others have different energy or their effort is focused differently, but I'm always cautious because burning out it's not something I want to ever repeat
Quite surprised by others finding this as a... Surprise? I get there is people who never experience this, but they also not know anyone personally to whom this would happen?