Commenting just to go against the other two answers. I think it's fine to not remember things, no matter the apparent simplicity.
Quite surprised by others finding this as a... Surprise? I get there is people who never experience this, but they also not know anyone personally to whom this would happen?
What does data frames mean in this context? I'm used to them in spark or pandas but does this relate to something in how duckDB operates or is it something else?
Without any context of culture or country, just trying to be helpful: in my limited (<20 total interviews) experience, I would think about budget issues.
Meaning, what you ask for (or how expensive you are perceived, if you have that strong resumee) for the industry you apply, may be too different and leading to limited access.
Sometimes I feel junior people have it easier (I felt like I did, personally) since the expense in salary is pretty limited compared to either other roles or more senior people
It's been mentioned but I want to add that the original idea of the post (mid size VPS hosting apache spark) might be missing that spark is ideal for distributed and resilient work (if a node fails the framework is able to avoid losing that work).
If you don't need this features, specially the distributed one, going tall (single instance with high capacity, replicate when necessary) or going simpler (multiple servers but without spark coordinating the work) could be good options depending on your/the team's knowledge
Recently I have been thinking about this, because I feel I have managed to become way more organized than I ever thought it was possible.
What is working for me right now is noting everything in a calendar so I cannot forget it or as TODO in a somewhat heavy personalized Obsidian configuration.
A few years ago (5-6 aprox) I started copying my older co-workers habits to see myself improve. Physical notebooks were soon discarded because I never remember where I wrote down things.
I used a TODO plugin in sublime which worked for several months, until I felt I needed screenshots so I moved to OneNote. After a while I became frustrated with not being able to customize it enough, so I started trying out different things. I saw a coworker using Obsidian, watched a couple long YouTube videos to learn how to customize, and I'm never going back.
My team this week told me they are impressed with how much info I write down and it was a very proud moment for me!
PyIceberg is nice but we had to drop it because it's behind Java API and it's unclear when it will match up, so depending on which features are needed I'd look it up
Quite surprised by others finding this as a... Surprise? I get there is people who never experience this, but they also not know anyone personally to whom this would happen?