I started doing this with 10k reports, the different formatting between different years was too much of a pain in the ass. They started embedding Excel documents in what I think is base64 into text files? I don't know. In the 90s the tables were in plain text.
I once hacked together AI to try and predict if cost of Bitcoin will go up or down based only on time and history of price. The program worked, but I remember it didn't predict very well. Maybe tinkering and reworking it would lead to something, but the combining the AI with the exchange APIs is daunting.
I use the Google daydream headset mostly for YouTube, I would kill for a browser so I can plug my keyboard in and browse the internet on a gigantic screen.
I tried to get bulk data from EDGAR a while back. Turns out bulk data acquisition has been turfed to a third party, which charges for downloads. This is supposed to be federal free data, I was so pissed off. I am still pissed off.
I made fedcrawl.com, the entire backend runs on Lua/torch. This involves scraping federal contract sites, cleaning text, and ultimately indexing everything on elasticsearch. It's really just a few Lua scripts running as services via systemd and passing messages with redis. I love it for the speed and how simple the code looks.
Not really, it's pretty unorthodox to look for genetic sequences as evidence of infection. Typically you look for antibodies, as the virus may not be circulating, or is simply not where you are sampling.
Folding is dynamic over the lifetime of the RNA, things like SHAPE are nice but only provide a snapshot. It is still leagues better than simply going by the old school thermodynamics based folding algorithms. A neat way to check out this 'folding space' is to generate all possible structures (possible even for long sequences if you don't include psuedoknots), then do alignments, or integrate experimental data like SHAPE.
Alfresco is an open source Enterprise content management software which has frontend UI called Alfresco Share. The installation is simple using their all inclusive executable which bundles the server, database, and Solr for searching any and all data. It also has this concept of business workflows which you say is a pain point for you. Each document can have workflows assigned to it at which point it has a life of it's own, passing from user to user, emailing itself, tagging itself, etc. There is a learning curve on the dev side, especially if you aren't Java savvy, but once you get the hang of it it's very powerful. I've personally written an API in Lua for syncing documents. I assign bot accounts from the Share UI which are given permission to crawl and sync directories to bot accounts on the server (origin server or external server), and each bot has a job to do on each document it gets, for which it writes and dispatches a service with systemd, then uploads the processed documents back to alfresco. And this is probably 1% of what Alfresco is actually capable of (and my implemention reeks of hackiness, because I hate Java). A team of devs is bound to get more use out of it.
I once wrote a thing that would take text files uploaded to the alfresco ecm, containing run options or data input for some automation script sitting on a server that's part of the corporate subnet, and depending on how the text file was tagged or which folder it was in, it would create service and timer unit files for systemd. The files would sync to the server via alfresco's atom API, and any output generated would be uploaded back into alfresco. This was neat because you could make alfresco user accounts for these bots, thereby turning them on or off for any project by simply adding the user and further modify which directories should be crawled for tasks to run by assigning read permissions to the bot account all via the client side alfresco share interface. Server images containing the bot code could be provisioned in say AWS, and integrated into an alfresco cluster. It made it much simpler for allowing non-technical users to run scheduled web crawlers, pipe them into document formatters, for web publication, or attach to email out the results hot to a given email list all from the nice alfresco share interface.
edge-cases. I feel like edge cases balloon exponentially relative to data size, where it feels as if at some point you might as well just screw the automation and handle each entry individually. Who cares, it's never going to be 'complete' anyway.