I think some domain-specific considerations include:
1. You need a really big in-memory data set that you touch ~all of several times for each request, so you really want to e.g. memory-map it and make sure it actually fits in memory on the machine.
2. If using a GPU, you have to make sure the GPU is hooked up to the serving process. You probably want your processes to be heavier-weight than they otherwise would be.
3. You might want to batch requests from several users for processing in the same stream of commands to the GPU. So you need to collect the right number of requests before processing any of them, without making any requests wait too long. You might need to sort these out by what inference parameters they want to override, and send them to different servers, because they might need to be batched accordingly.
4. You might want to stream the output more or less character by character. Possibly to several users, from one live run on a GPU, after having batched up enough requests to justify a run.
5. Content moderation when you are sending data to the user before you have even seen all of it yourself is an unsolved problem.
When PayPal goes to make the withdrawal, the bank doesn't have to let them and give the account a negative balance. Usually you can turn off overdraft and withdrawals will bounce if the funds aren't there.
This doesn't stop PayPal from reversing a previous inbound transfer they decide shouldn't have happened, but it ought to stop PayPal unilaterally deciding they would like some of your money.
Not all proprietary software currently insists that you get into a TPM-mediated Dom/sub relationship with its developers. Its currently possible, and might be ethically necessary, not to buy those ones, and instead to buy the other ones.
But it's also probably important to pursue a political avenue as well. The government should absolutely not be using this stuff, and shouldn't be advising citizens to do it to access government services. We could even pass a law requiring purchased hardware and software to meet a fiduciary standard towards its users.
1. You need a really big in-memory data set that you touch ~all of several times for each request, so you really want to e.g. memory-map it and make sure it actually fits in memory on the machine.
2. If using a GPU, you have to make sure the GPU is hooked up to the serving process. You probably want your processes to be heavier-weight than they otherwise would be.
3. You might want to batch requests from several users for processing in the same stream of commands to the GPU. So you need to collect the right number of requests before processing any of them, without making any requests wait too long. You might need to sort these out by what inference parameters they want to override, and send them to different servers, because they might need to be batched accordingly.
4. You might want to stream the output more or less character by character. Possibly to several users, from one live run on a GPU, after having batched up enough requests to justify a run.
5. Content moderation when you are sending data to the user before you have even seen all of it yourself is an unsolved problem.