Our architect wrote some cute Datadog wrappers that I implement in every pipe I roll out (and is that instrumental in diagnosing bugs). He also programmed in a call to keys in a store that times out from too many requests and hangs the whole function the wrapper is decorated on, that took me a while to diagnose pinpoint
>>>are on call if one of your models breaks and have built the system where you "just drop a container"?
Our motto is ‘you wrote it, you fix it!’ If the container pipe works, and you dropped a bomb in it that doesn’t work, why should the DE have to pickup the DS’s garbage?
>>>If you are, then you are a data engineer too,
:D
>>>otherwise you're standing on the shoulders of your data eng team and they make it look easy for you. Go buy them some donuts first time you go back to the office.
I actually support a few data scientists in the manner you described above where they generate some metrics notebooks or container and I have to diagnose their crap in a DE capacity (recent problem involved optimizing their pyspark code to be less memory intensive)
As a DS, I collect and clean my
own data (sometimes literally as they’re coming off the upload line, if I’m not building the upload pipeline in question too), serving the raw data AS WELL AS metrics/models/algorithms Generated by notebooks/containers from raw data in from hive via Spark queries.
Mmm, I disagree. I do a fair amount of mucking around in notebooks/containers, but I also drop them (when they’re refined enough) into a pipeline where I (not another person dubbed a DE) can evaluate the integrity of the results on an ongoing basis. Though I do defer pipeline design dilemmas to our architect I do some of the implementation / testing work myself, I firmly feel that for a data scientist to be effective in their role they must also be part Data engineer (and visa versa). It’s hard to compartmentalizations those duties and get results efficiently.
Our architect wrote some cute Datadog wrappers that I implement in every pipe I roll out (and is that instrumental in diagnosing bugs). He also programmed in a call to keys in a store that times out from too many requests and hangs the whole function the wrapper is decorated on, that took me a while to diagnose pinpoint
>>>are on call if one of your models breaks and have built the system where you "just drop a container"?
Our motto is ‘you wrote it, you fix it!’ If the container pipe works, and you dropped a bomb in it that doesn’t work, why should the DE have to pickup the DS’s garbage?
>>>If you are, then you are a data engineer too,
:D
>>>otherwise you're standing on the shoulders of your data eng team and they make it look easy for you. Go buy them some donuts first time you go back to the office.
I actually support a few data scientists in the manner you described above where they generate some metrics notebooks or container and I have to diagnose their crap in a DE capacity (recent problem involved optimizing their pyspark code to be less memory intensive)