I imagine this has it's roots in the demographic and population shift the city has seen starting in the 50s. Philly's population in 1990 was 75% of what it was in the 50s. I'm not an expert in this area, but I'm sure there was overhead in maintaining infrastructure, paying pensions, etc as the population shrunk.
At this point, maybe the federal government should just bail out city school districts in this situation. Why should an underfunded school district be paying a chunk of its budget on debts?
for this specific example, I think the shared library is not the correct approach. Queues work for simple fifo behavior, but in this case they also need fairness (and perhaps other logic, like rate limiting per client, different priority for certain clients etc)
For example, "Customer-A is clogging up the work queue and starving other customers out". The solution to this could look something like linux's completely fair scheduler, where every client is allocated some amount of messages per interval of time. This means messages need to be processed in a different order then they are enqueued, and queues are not good at reordering messages.
I would suggest implementing the queue abstraction as a rest or grpc service, backed by a database like postgres, which holds message payloads and everything related to message state (in progress, retry after times, etc). Now we can implement all the necessary scheduling logic within our queue service.
- we don’t need any kind of backwards compatibility, we just update everything.
if you don't care about backwards compatibility, then you can stay on v1 forever. Have you considered a monorepo? That would simplify updating packages and give you the behavior you want.
- For the client to update, it’s not a simple path change in go.mod
if a package moves from v1 to v2, there are breaking changes in either api or behavior. I think this implies more than a simple change to go.mod. This also allows importing both versions of a package if necessary.
I worked on a project which required some medium scale web scraping (less than 100 million pages), and went with node primarily because of puppeteer.
The system had a couple dozen worker processes doing the scraping, and one coordinator which maintained a queue of pages which needed to be scraped. There was some logic to balance requests between sites, so we weren't making more than a request/s to any in particular. The coordinator just had a rest api endpoint, which the workers would hit to get their next job and to return w/e data.
Each worker process was ran on a separate aws instance, believe it was a t2 with unlimited cpu enabled. These are only a few dollars a day, and it was necessary to have as many ip addresses as possible (at least 5% of the sites we were scraping had some preventative measures in place, but they all seemed to be ip based)
I agree this can be awkward, especially if you let these constructs propagate through your codebase and database. However, if a string or int can be null, then all strings and ints are essentially pointers, so you've just introduced this construct everywhere.
A couple things I have tried:
- hope default values align with your business logic, eg an empty string isn't a valid name and 0 isn't a valid age.
- for partial updates, populate the existing values before unmarshalling, then unmarshal on top. Missing fields in the json won't overwrite the existing values
- unmarshall into a map[string]interface{}, which gives you the semantics you want.
i would suggest bias your implementation against false negatives. They can always come back and update it if it's wrong, and their url could just as easily be "valid" but incorrect, eg any typo in a domain name.
if it's really important, you could try making a request to the url and see if it loads, but that still doesn't validate its the url they intended to input.
might be cool to load the url with puppeteer and capture a screenshot of the page. if they can't recognize their own website, it's on them.
I think this article misses two important points on siloed qa teams
1. Qa doesn’t know what can be covered by unit or integration tests
2. Since they treat our code like a black box, they may create permutations of tests which which cover the same functionality
Maybe this is part of the draw of having a qa team. Feature coverage rather than code coverage. The downside is this can create a huge number of expensive to run manual tests which may be hitting the same code paths in functionally identical ways.
The tooling for automating manual tests of web apps is almost there: puppeteer, recording user inputs and network calls, replaying everything and diffing screenshots.
Since qa tests are tied to features and not code, There’s also the problem of having to run all qa tests even if you’re releasing minor code changes. My build tools are smart enough to return cached results for unit tests whose dependencies didn’t change, but there’s no equivalent for qa tests.
cool, so this is like warby parker's app for renewing prescriptions?
that really saved my ass recently, when my glasses broke and I found my prescription was expired. Apparently, they let you use an expired prescription up to 18 months, and mine had expired like 20 months ago.
Their app worked, but was pretty painful. Required holding my phone 14 feet from my monitor- had to move my desktop to a different room to get the app to approve.
how significant is 1 if you are using cloud sql proxy? My understanding is that the proxy tunnels traffic over an encrypted connection, so there is no benefit to adding an extra layer.
would have been useful for something we ran into at a previous job. For legacy reasons large csv files had been zipped instead of gzipped (or some other streamable format), and were causing some process in our data pipeline to consume too much memory.
I wound up coming up with a quick solution that appeared to work. Taking advantage of being able to get s3 objects at arbitrary ranges, I implemented seek and read on top.
did you use any kind of message framing? Got bit by this at a previous job where we needed to change the message format to improve compression. Wound up figuring something out, but would have been easier if we had reserved a byte for versioning.
The best use case I've had for partitioning tables is when there is a lot of churn that would otherwise result in significant vacuuming. For example, a table holding the past 7 days of data would require deleting 1/7 of its rows every day.
Swapping out a child table and dropping the old one is free in that sense, but does require an exclusive lock which can cause issues if you have long running queries.
interesting, how did the spend breakdown between cloud run and firebase?
did you have any limit to how many req/s you made to an individual site? It seems this would be difficult to implement with this architecture.
how did you deal with following links in circles/ avoiding scraping the same page multiple times?
I had built something similiar at a previous job, recursively scraping ecommerce sites. The first thing I noticed was some of the sites we were scaping couldn't handle more than a couple requests a second (in particular as we scaped uncached pages by sites running php). Other sites were quick to ip ban.
I kept things simple, a few dozen micro instances on aws (think they were like $3 a day) running puppeteer. A single server acting as a controller, keeping a per site queue and allowing us to set per site request limits if necessary. All the state of which links were already seen just kept in memory. Of course everything was also persisted to a db, and if the controller process needed to be restarted, it could restore the queue/ seen state and resume.