How I Hacked My University's Registration System with Python and Twilio(twilio.com)
twilio.com
How I Hacked My University's Registration System with Python and Twilio
https://www.twilio.com/blog/2017/06/hacked-my-universitys-registration-system-python-twilio.html
162 comments
Greg from Twilio here. Congrats on your immigration! Glad we could play a small part in that journey, but credit goes to you for your courage and creativity. Would love to hear more of your story. Would you mind dropping me a note at [email protected]?
Sounds like an amazing case study. Just looked/stalked your profile and ended up watching your talk on "Developers, entrepreneurs and depression"[1]. Really inspiring and awesome!
[1] https://vimeo.com/78419167
[1] https://vimeo.com/78419167
Welcome to Canada! Also: it's not all like Saskatchewan, most of it is really nice. ;)
That's along the lines of what I do when I am waiting for a dedicated machine from OVH, since they list all of their clearance machines on a special page and you have to keep checking them. So I just used my script with Twilio to send me SMS when it's available to buy. Done that all three times I needed a box. I would not call it "hacking" though.
I think hacking in a sense of having a problem and piecing together a solution is the context they were going for (much like hackernews) vs traditional escalation type hacking.
Sure, but the term for this is "problem solving" and calling it "hacking" is disingenuous. I am not a big fan of the crowds that co-opt terminology for sort of superficial (e.g. marketing) reasons. I'm a bit hypocritical about it, because I think rms has historically done extremely well with this tactic, pushing ideals that I agree with. But there are other groups which do this that grinds on my soul.
Warning you now - that strategy on Kimsufi boxes at least is extremely popular. If you're willing to pay a little more check out their "SoYouStart" line - they seemed more readily available.
[deleted]
I did something similar for a job offer on LinkedIn. I had to go out with friends so I polled the site every minute (not as gentlemanly as you) looking for the apply button. I was going to use Twilio for I had just signed up for a trial, but the internet was unreliable and I ended up using Gammu with a 3G dongle and a local SIM card to send an SMS.
I was surprised to see the number of applicants increase without the button having ever become available (no SMS received and I had logging to confirm). It was fun, nonetheless.
It's made me want to make something like this to diff arbitrary things and send notifications to people who're watching them, but I didn't sit down to do it, and sitting down to do it is all that matters.
I was surprised to see the number of applicants increase without the button having ever become available (no SMS received and I had logging to confirm). It was fun, nonetheless.
It's made me want to make something like this to diff arbitrary things and send notifications to people who're watching them, but I didn't sit down to do it, and sitting down to do it is all that matters.
I got tickets to a certain-to-sell-out concert doing the same thing. My friends were shocked I got them A-class seats. I'm teaching their kids to know that they will have more control over their world than they might imagine.
This works until the technique gets popularized, then it won't anymore.
Did you manage to keep your compensation around the same? Or did you have to give up on it somewhat?
Canada does seem more attractive than US. However, there are definitely a lot more jobs in technology, and a lot more software engineers here in the US.
Canada does seem more attractive than US. However, there are definitely a lot more jobs in technology, and a lot more software engineers here in the US.
The distill browser add-on will work for this too, except for the phone call. SMS is a paid feature but email and popup notifications are free.
And maybe also the fact that the computer and browser has to be open all the time, right?
I believe OP did this by running a cron or so on an always-on server.
I believe OP did this by running a cron or so on an always-on server.
Nice story, and great that Twilio worked out for you - but this really is the sort of thing that email alerts (or IM alerts, or IRC alerts) should also work fine for.
I would argue that the fact that you find SMS/phone calls a more urgent alert is a fault with your phone/communication setup - and twilio is an interesting hack around that.
But there really should be an easy way to make just as much ruckus from a simple email, based on topic/sender filtering... (hm, maybe there's a nice side project for an app in there...).
I would argue that the fact that you find SMS/phone calls a more urgent alert is a fault with your phone/communication setup - and twilio is an interesting hack around that.
But there really should be an easy way to make just as much ruckus from a simple email, based on topic/sender filtering... (hm, maybe there's a nice side project for an app in there...).
The post you replied to barely even mentioned Twilio. It made no comparison to email or IM. It never said those wouldn't have worked. You seem to imply that choosing to use Twilio at all indicates a fault in one's communication setup...because they should use email instead?
The funny thing is, you go on to acknowledge that a phone call is psychologically more urgent than an email, and ponder how email might be extended to cover that use case...or maybe, the person was not at fault and was actually using the right tool for the job!
The funny thing is, you go on to acknowledge that a phone call is psychologically more urgent than an email, and ponder how email might be extended to cover that use case...or maybe, the person was not at fault and was actually using the right tool for the job!
If you treat emails with the same urgency you give phone calls, I would argue there's a fault in your phone/communication set up. For most people I know, email can wait an hour or two, or even overnight. Phone calls require you to choose to ignore them.
Maybe in your world that works.
Until the FCC develops redevelops any sort of interest in dealing with scammers, my phone is basically useless for most of the world to speak to me.
If you're trying to reach me and are not on my list of approved callers, you can email me, or decide you didn't actually want to talk to me. (No, I don't use FB Messenger or related spyware, either.)
On the other hand, I apparently pay much more attention to email than you - I generally respond within about five minutes, assuming there's some reason.
Until the FCC develops redevelops any sort of interest in dealing with scammers, my phone is basically useless for most of the world to speak to me.
If you're trying to reach me and are not on my list of approved callers, you can email me, or decide you didn't actually want to talk to me. (No, I don't use FB Messenger or related spyware, either.)
On the other hand, I apparently pay much more attention to email than you - I generally respond within about five minutes, assuming there's some reason.
One of the points of email is that it is asynchronous communication. You can even respond in batches like RMS does. While everyone's email workflow is different, it would be a mistake to assume that email was ever designed to be used as an instant-messaging-style communication tool.
IRC exists for a reason.
IRC exists for a reason.
Yeah, this isn't really a major Twilio win or a hack - it's a basic scraping technique.
I found a house for myself and a friend in a similar way, reverse engineer mobile app from MLS, scrape all listings once a day to get new stuff as it hits the market and you can now be far more selective than the crappy search on the MLS site allows.
I found a house for myself and a friend in a similar way, reverse engineer mobile app from MLS, scrape all listings once a day to get new stuff as it hits the market and you can now be far more selective than the crappy search on the MLS site allows.
Yeah you can sends texts to your phone just like sending an email.
How to do that?
Put the phone number in front of a domain for each carrier, a list is here.
https://github.com/WilliamFCipriano/FreeSMS/blob/master/data...
https://github.com/WilliamFCipriano/FreeSMS/blob/master/data...
There are several international lists too, here's a couple:
https://martinfitzpatrick.name/list-of-email-to-sms-gateways...
https://www.opentextingonline.com/emailtotext.aspx
https://martinfitzpatrick.name/list-of-email-to-sms-gateways...
https://www.opentextingonline.com/emailtotext.aspx
Do they work? For Portugal it still lists Telecel - which hasn't existed since '99.
My list is accurate as of a year or so ago, or at least they don't bounce back when you send an email to them.
[deleted]
Thanks, this hack (if it's a hack) looks interesting.
It works, but know that it is nowhere as reliable as an SMS sent through a carrier's gateway.
I am using Telegram for such notifications. I made a simple Telegram bot, it does not do anything itself, I just use it as a sender of my messages to my phone. And I have a particular web address, which collects requests from any of my scripts and it just resends it (it is a simple python script) through the bot to me.
You can check Twilio's website for information.
[deleted]
Which MLS mobile app are you referring to?
SMTP is not an instantaneous protocol. Email prioritizes reliability over speed when it comes to delivery. An email alert could very well be delayed to the point that it becomes useless for something which is time sensitive.
Given the synchronous nature of a phone call, it seems like the perfect medium for this kind of alert. Similarly, SMS piggybacks off of the same network and therefore inherits some of the same properties.
Given the synchronous nature of a phone call, it seems like the perfect medium for this kind of alert. Similarly, SMS piggybacks off of the same network and therefore inherits some of the same properties.
I tried web alerts, but the emails were not instant, plus no phonecalls. Most the services I found didn't look reliable enough as I had no window for mistakes for my immigration. So I rolled my own.
> I would argue that the fact that you find SMS/phone calls a more urgent alert is a fault with your phone/communication setup
Doesn't the asynchronous nature of an IM make it less urgent than a synchronous phone call?
Doesn't the asynchronous nature of an IM make it less urgent than a synchronous phone call?
The industry I'm in is all SMEs - they regularly complain about people messaging them [ie SMS, aka "text"] out of work hours. Like "a customer texted me at 9pm about something how inconsiderate". They don't seem to understand it's absolutely considerate and that the idea is they respond when they next want to make themselves available. Mind you some people will get annoyed if they're not texted back straight away; why they don't then phone I'm not sure.
Yeah I've heard the same sentiment. I think people feel rude phoning because they feel rude ignoring calls, so if they call they're making the other party feel responsible for answering. This is silly of course.
> but this really is the sort of thing that email alerts (or IM alerts, or IRC alerts) should also work fine for.
Except when you are asleep.
And when it comes to immigration, you are happy to be woken up in the middle of the night by a phone call when it means the difference between failure and success.
Except when you are asleep.
And when it comes to immigration, you are happy to be woken up in the middle of the night by a phone call when it means the difference between failure and success.
Receiving phone calls is free, a data plan isn't :)
I use Pushover for (personal) things like this.
[deleted]
May I ask what PR stream you applied under that "opens randomly"?
After years of hard work I successfully got in via Provincial Nomination.
After years of hard work I successfully got in via Provincial Nomination.
We're always glad to have more hackers in Canada, welcome! (If you've got your PR this was probably a while ago, but still)
Thank you. I got my PR some months back. My spouse can work now. We are very happy than being in the US where she was not authorized to work. We enjoy the real freedom here.
[deleted]
_sbrk(4)
I did this for one of my classes as well. But since it was just a one off, I just put this bash script in a cron job on a free-tier AWS ec2. It just did a quick and dirty parse of the html, which for my university wasn't behind a login wall.
$seats="$(curl $URL | sed \"140qd\" | sed -nE \"s/<TD CLASS=\\\"$CLASS\\\">|<\\/TD>//gp\")"
if [ $seats -gt 0 ]; then
echo "Go register for class $URL" | msmtp -a "default" $EMAIL
fiYep. This is what I did. Also, I just used the email address for my phone's SMS. I'm not sure why you need Twilio for this, seems like over kill to me.
It's an ad for Twilio, of course Twilio is needed for that.
Did the same thing for an apartment hunt several years ago (one particular realtor had all the best stuff going up on a personal site) and also just notified via email. This is done the way that it is by their content marketing department.
The email-to-sms gateway for AT&T was very unreliable for me.
If you were able to do that prior to taking CS101, you might not need CS101.
If you want to get that piece of paper that some companies require...
At my school, they would waive requirements if you take a higher level course in the same track. For example, if I just took CS102 that requires CS101, I wouldn't have to go back to satisfy the CS101 requirement.
Obviously everywhere is different.
Obviously everywhere is different.
How... are you taking CS102 that requires CS101... without taking CS101? Are the requirements not actually required?
IDK about the parent's school, but my at university (RPI), when I was there, students could simply opt to skip "CS101", and move to "CS102". I think it was also encouraged (but again, not required) to have taken one of the AP CS tests if you did so.
I think you could also opt out of "CS102" if you desired. It was on the student to determine how much they knew. The school just understood that a number of CS students had been doing this for some time on their own, and had mastered some of the basics.
I skipped the "CS101" course, which was mostly intro to programming, control flow, etc. I decided against skipping the "CS102" course; the first half wasn't that exciting (I knew how linked lists, vectors, hash tables worked) but trees got me (and some of the more exotic stuff at the end of the course), so I'm glad I took it.
I think you could also opt out of "CS102" if you desired. It was on the student to determine how much they knew. The school just understood that a number of CS students had been doing this for some time on their own, and had mastered some of the basics.
I skipped the "CS101" course, which was mostly intro to programming, control flow, etc. I decided against skipping the "CS102" course; the first half wasn't that exciting (I knew how linked lists, vectors, hash tables worked) but trees got me (and some of the more exotic stuff at the end of the course), so I'm glad I took it.
Interesting. At least when I was at my UC 10 years ago we weren't able to skip any classes. Maybe they have wisened up and made them optional or have a system in place to prove them unnecessary. It makes some sense to make sure everyone has the same level of fundamentals though.
Course prerequisite are more like suggestions. I came in knowing how to code pretty well, so I didn't want to take Java for new programmers.
Weird. At my (US) university there was no wiggle room. Everyone had to take the intro courses regardless of previous ability/accomplishments. Maybe that has changed in the past 10 years though?
and also immigration..
If you're using Python, please don't reinvent the wheel and write your own scraper. Use Scrapy (https://scrapy.org/). It's one of the most tested and powerful scrapers out there.
For a use case like in the article would it make more sense to use beautifulsoup or is there an advantage to scrapy?
I did a similar thing my freshman year of college, and by junior year I created an API based around the class registration system and an iPhone/Android app for it that sends push notifications to students when the class they want opens up (shameless plug: http://www.eaglescribe.com/).
This past semester, the active users was between 1/3-1/2 of the undergrad students in the school. As you can imagine, popular classes had dozens of people "subscribed" to receive notifications when it opened, so it became a race to sign up once the push went out. On the plus side, this gave us a treasure trove of data on the most popular courses, and we've been in communication with the school to see if they would be interested in this data.
This past semester, the active users was between 1/3-1/2 of the undergrad students in the school. As you can imagine, popular classes had dozens of people "subscribed" to receive notifications when it opened, so it became a race to sign up once the push went out. On the plus side, this gave us a treasure trove of data on the most popular courses, and we've been in communication with the school to see if they would be interested in this data.
Twilio is great and all but an SMS or other notification doesn't actually guarantee you a class that's available to all students.
A class was sniped from me one time while I was still on the confirmation screen. So unlike OP I fully automated the process.
EDIT: A notification & response is good for solving a captcha if they have one, if you don't want to outsource that to mechanical turk (or trust their timing / accuracy).
A class was sniped from me one time while I was still on the confirmation screen. So unlike OP I fully automated the process.
EDIT: A notification & response is good for solving a captcha if they have one, if you don't want to outsource that to mechanical turk (or trust their timing / accuracy).
This is an interesting proof of concept. I'm impressed at how simple the SMS signup code is.
For many universities though, it won't be this simple. Many hide the open seats behind a login, which means you need to be a student there to see them. Many also use student information systems that are a real pain to scrape like this. Once you get to the point where interacting with the student information system is done via AJAX requests mediated by a mess of JavaScript, it starts to push the limits of what you can do with this technique.
When it gets to this point you either have to carefully dissect everything on the Network tab of your developer tools, or use a real browser. I've used PhantomJS (basically a headless WebKit) and Selenium WebDriver with great success on sites that are not amenable to scraping. The neat thing is that you really only need to use that for the interaction to get to where the information is. Once you've navigated there you can just have it dump the rendered page as HTML and parse it using the same techniques shown here.
For many universities though, it won't be this simple. Many hide the open seats behind a login, which means you need to be a student there to see them. Many also use student information systems that are a real pain to scrape like this. Once you get to the point where interacting with the student information system is done via AJAX requests mediated by a mess of JavaScript, it starts to push the limits of what you can do with this technique.
When it gets to this point you either have to carefully dissect everything on the Network tab of your developer tools, or use a real browser. I've used PhantomJS (basically a headless WebKit) and Selenium WebDriver with great success on sites that are not amenable to scraping. The neat thing is that you really only need to use that for the interaction to get to where the information is. Once you've navigated there you can just have it dump the rendered page as HTML and parse it using the same techniques shown here.
Author here--you're totally right. When we did this for our actual registration site, we spent hours inspecting requests in a proxy[0] to get to ~194 lines of PHP.
0: We used https://www.charlesproxy.com/
0: We used https://www.charlesproxy.com/
Yeah, that's more what I'd expect to see with a standard student information system. I can only imagine the amount of work it took - the last time I encountered a site like that that I wanted to scrape I gave up (this was before I discovered PhantomJS).
It's really too bad that universities don't foster creativity like this.
It's really too bad that universities don't foster creativity like this.
Wondering why you didn't also automate the process of actually signing up for the seat? Getting notified that a seat is open is helpful but automatically taking the open seat sounds a lot more helpful.
Though at that point you'd have a lot more luck with selenium or some other web driver.
Though at that point you'd have a lot more luck with selenium or some other web driver.
We thought about doing that but were scared of the repercussions from the school. We felt like it was much easier to argue for notification than for automatic sign-up. Luckily, we didn't get in any trouble, so we didn't have to argue it anyway.
I wondered that too until I realized it was the Twilio blog, so they would need the use case to fit their particular service.
Sometimes universities have weird rules about automating the actual sign up process versus automating being notified.
Would you rather fail to register for an open seat or fail to find an open seat? Maybe I'm biased because I made the same choice as an undergrad, I stopped short of registering for the class automatically.
Yeah, automating that next step is making the failure mode much worse.
I was able to get auto-registration working with mechanize a few quarters ago (similar setup to OP); I got into two classes that way! I was just using SMS/emails from UW's provided notification system to trigger the script, though, so I don't think I was actually any faster than other students-just less stressed about watching for notifications.
Need to be careful about this one. A student at my alma mater was suspended for doing the same thing: https://www.reddit.com/r/ucf/comments/xo5ye/ucf_student_pena...
You have to love the education system...
You have to love the education system...
In your example, he was charging for the service, which was against the terms of use of the web site he was scraping.
So in general, do be sure to not violate a site's terms, but I don't see what that point has to do with your final comment...
So in general, do be sure to not violate a site's terms, but I don't see what that point has to do with your final comment...
I did a similar thing (free mobile app with notification + backend to scrap), gotten a cease and desist letter and threatening not to let me graduate.
This blew up at all the Florida unis when it happened. I wonder how much it leaked out to the rest of the Internet.
I tried building a similar service at my university when I was in undergrad, and it led to lots of administrative conflicts, because people using my service were thought to have an unfair advantage in the registration process.
Burp is probably the better tool for the job in most cases since you're usually just looking to repeat a request until you get the response you're looking for.
If you just want to automate registration to ensure you get a seat in a course you should be looking into your school's network topology to minimize latency.
If you just want to automate registration to ensure you get a seat in a course you should be looking into your school's network topology to minimize latency.
I'm dating myself, but when I went, it was a standard phone AVR, with touch tone menus. I used a war dialer to get into all my classes. Ahh, the good ole days.
Which wardialer?
Cool, I'm dating myself too. We should double-date
For the curious, if you just want to one-off this for yourself, you probably don't need twilio to do it. Most carriers have a special domain that if you send email with the recipient's phone number, it turns into a text. I've got Verizon and its as simple as this from the command line:
echo 'Text message text' | mail -s 'Subject will display in braces in the message' -a 'From: [email protected]' '[email protected]'
This ended up working faster than Twilio for me. (Less than 10 seconds from enter key to text on phone).
echo 'Text message text' | mail -s 'Subject will display in braces in the message' -a 'From: [email protected]' '[email protected]'
This ended up working faster than Twilio for me. (Less than 10 seconds from enter key to text on phone).
Yep, they're called SMS gateways. I don't think there's a single canonical source for them, unfortunately, but I keep this file bookmarked for when I need to look it up: https://github.com/cubiclesoft/email_sms_mms_gateways/blob/m...
Is the SMS free? If not, who pays for it?
Well, it depends on your phone plan. If you have unlimited texting, then yeah its "free" (and you pay as a part of your monthly phone bill), if you pay per text, you pay whatever the rate for an SMS message is.
Your number is now on hacker news!
Just saw that. Yowzer... damn clumsy of me! (Thankfully a published work number but still!)
Some time ago Berkeley switched from Telebears to a new registration system developed by PeopleSoft/Oracle. Needless to say, it's garbage. I wrote a couple scripts to pull class data from the API in an effort to help me get into our heavily oversubscribed CS courses by searching for open sections, but the system seems to be designed to make using it as painful and worthless as possible. Luckily this semester the EECS department has put dummy sections in for all classes so students who are able to register for a lecture don't get permanently stuck on the waiting list because of a full section.
College administrators who are considering Oracle for your registration needs, please consider anyone else instead, for everyone's sake.
College administrators who are considering Oracle for your registration needs, please consider anyone else instead, for everyone's sake.
I remember a classmate of mine doing the exact same 'hack' and distributed it to the whole school (Well, it was posted on school's major Facebook group with 10k-ish members) but it didn't have the SMS bit (It notified you via E-Mail, AFAIK).
I don't know about you, but for me receiving an email and receiving an SMS are identical notifications.
This is pretty cool. I did something similar with adwords scripts a few years ago.
I was doing facebook advertising a few years ago and the ads featured products that would occasionally sell out, but there wasn't a way to monitor them so if something sold out, the ad would go to an empty product page. I rigged up a system that would take a list of URLs, look for "sold out" on the content of the page and then send me a text if it found a sold out product. I used Adwords scripts because it was able to run jobs hourly and all I really knew at the time was JS. It was a fun little project.
I used to work in University IT and registration time is the worst. The systems never handle the load well and everyone freaks out. I think your work might anger the sysadmins b/c of the increased load.
The sad thing is that with a little bit of effort on the part of University IT, something like this could be built to put a lot less load on the system.
Using this example, couldn't someone (University IT) write an SQL script to dump out the current seat count/capacity for every "current" class and throw it on a web server somewhere? Assuming that the underlying database schema isn't completely insane, that would probably be about a half-day of effort. The student writing the scraper could then point it at the output of this script.
With a half-day of effort, you've reduced the load on the application, you've provided a service that the students probably want, and best of all you've got someone else to do most of the hard work.
Using this example, couldn't someone (University IT) write an SQL script to dump out the current seat count/capacity for every "current" class and throw it on a web server somewhere? Assuming that the underlying database schema isn't completely insane, that would probably be about a half-day of effort. The student writing the scraper could then point it at the output of this script.
With a half-day of effort, you've reduced the load on the application, you've provided a service that the students probably want, and best of all you've got someone else to do most of the hard work.
At my University, where I both worked and studied, we had a second semester senior post his automatic registration script to the school's Facebook group. This sent up alarm bells and I sent emails to warn the appropriate people, but it was the night before registration. The load immediately and totally crushed the system, and registration had to be rescheduled. They switched around some tags on the page (the script did very fragile webscraping), and that was enough that there were no more issues. The student had to fight an academic misconduct trial for it as well. A painful situation all the way around.
Couldn't you just let anybody sign up for $LONG_ENOUGH_PERIOD and then randomize who gets in the classes? It sounds like a much better solution in general.
My university switched to that for some (mostly optional) classes. This resulted in a lower attendance rates because as a student you end up registering for pretty much everything instead of one or two classes you really want. There was a system in place to fill the empty seats with students still in the waiting queue by the second session but most students already got into other classes and just didn't remove themselves from the queues. So the students got an email that a seat was free for them which they ignored and because you can only miss two sessions before failing the classes, the seats stayed mostly empty.
Okay, so what about this:
Students rank their course preferences in an order and submit it. For each course, free seats are first allocated to the students who ranked it as their first choice, then what is left to the students who ranked it as a second choice, and so on until the round n and there are less seats available than people who ranked the course as their nth choice: the remaining seats are allocated by a lot to them. (For a very popular course, they would have to start with the random selection right away on the round n=1).
However, instead of enrolling the lucky students right away, everyone gets notified about which classes they have been tentatively admitted, and they must confirm their attendance before they can take their seat. If they don't confirm in a timely manner, they lose the opportunity and the free seats are propagated to students left in the waiting list until someone accepts (again using random lot).
The one problem I can think of: the process (especially the propagation of rejected offers in the waiting list) may be too long-winded to be practical. So maybe skip the confirmation part: there's a set maximum number of courses you can take, M, and slots are filled until you're enrolled to maximum of M courses.
But in any case, if the overbooking is enough of a problem so that you are looking at methods like these, the real problem is that have too many students compared to the teaching resources. All other attempts can only mitigate the symptoms of the root cause.
So maybe staff should choose limit the amount of students eligible to take some very popular courses by an entrance exam, or imposing certain mandatory minimal GPA requirement in the prerequisite courses. This option should be especially considered if the attrition rate is high (there's a problem of students taking much-coveted seats in the class but who fail to show up or otherwise don't put in enough effort to pass). Usually that would have been done on the admission level already, though.
Students rank their course preferences in an order and submit it. For each course, free seats are first allocated to the students who ranked it as their first choice, then what is left to the students who ranked it as a second choice, and so on until the round n and there are less seats available than people who ranked the course as their nth choice: the remaining seats are allocated by a lot to them. (For a very popular course, they would have to start with the random selection right away on the round n=1).
However, instead of enrolling the lucky students right away, everyone gets notified about which classes they have been tentatively admitted, and they must confirm their attendance before they can take their seat. If they don't confirm in a timely manner, they lose the opportunity and the free seats are propagated to students left in the waiting list until someone accepts (again using random lot).
The one problem I can think of: the process (especially the propagation of rejected offers in the waiting list) may be too long-winded to be practical. So maybe skip the confirmation part: there's a set maximum number of courses you can take, M, and slots are filled until you're enrolled to maximum of M courses.
But in any case, if the overbooking is enough of a problem so that you are looking at methods like these, the real problem is that have too many students compared to the teaching resources. All other attempts can only mitigate the symptoms of the root cause.
So maybe staff should choose limit the amount of students eligible to take some very popular courses by an entrance exam, or imposing certain mandatory minimal GPA requirement in the prerequisite courses. This option should be especially considered if the attrition rate is high (there's a problem of students taking much-coveted seats in the class but who fail to show up or otherwise don't put in enough effort to pass). Usually that would have been done on the admission level already, though.
You might be interested in bidding systems for enrolling in courses. The National University of Singapore has one http://www.nus.edu.sg/registrar/events/module-enrolment.html
"All students are given an equal amount of points per semester to bid for modules... The allocation of modules is based on the lowest successful bid points against the last available quota for the module at the end of each bidding round. If supply (module quota) exceeds demand (number of bidders) for a module for any bidding round, the lowest successful bid will be 1 bid point. If there is a tie in the lowest successful bid points, the outcome will be based on first-come-first-served. Unsuccessful bidders will be fully refunded. Any unused bid points after each round will be carried over to the next bidding round or to the next semester at the end of the registration exercise."
"All students are given an equal amount of points per semester to bid for modules... The allocation of modules is based on the lowest successful bid points against the last available quota for the module at the end of each bidding round. If supply (module quota) exceeds demand (number of bidders) for a module for any bidding round, the lowest successful bid will be 1 bid point. If there is a tie in the lowest successful bid points, the outcome will be based on first-come-first-served. Unsuccessful bidders will be fully refunded. Any unused bid points after each round will be carried over to the next bidding round or to the next semester at the end of the registration exercise."
That won't work very well. You register for multiple classes at once, not just one. Which class(es) you register for are dependent on when the other classes you registered for are scheduled. It's hard to get a schedule with classes that don't overlap if you are uncertain if you'd get a seat in a couple classes.
An even better solution would be to add a wait list, as mentioned by the OP.
I built a similar service when I was in school at UNC. Unfortunately, the IT dept blocked my server IP and asked me to shut down the service after it got press (and lots of usage.)
There exists something like this for Rutgers University. Anyone can submit a course to be tracked along with email, and once a course pops up as open, an email is sent. Text messages are not used since they are not free and this project is maintained by students, not university. But it can be easily modified and self-hosted since the source is on Github:
https://github.com/v/sniper
I've done a similar thing with Southwest Airline's check-in. They let people get seated based on the order in which they check in for their flight. So right before the 24 hour period began I would run a script that tried to check me in repeatedly until it succeeded. Usually got really good seats.
EDIT: to those asking for the source, it was just a bash script that I have probably lost. The GitHub linked in this subthread looks a lot better than what I had.
EDIT: to those asking for the source, it was just a bash script that I have probably lost. The GitHub linked in this subthread looks a lot better than what I had.
Greg from Twilio here. I love Southwest[1] and have wanted to do something like this forever. Would you be up for sharing the source? Can drop me a note at [email protected].
[1]: http://baugues.com/southwest
[1]: http://baugues.com/southwest
I used this last year for ~ 10 flights.
https://github.com/aortbals/southwest-checkin
This is great. And RUBY!! Thank you.
[deleted]
I would love to check this out as well. [email protected]
I did the same thing at UCSB 10 years ago to constantly poll for seats opening up in a class.
This was back when GOLD was a primitive web interface that was clearly screen-scraping an IBM AS/400 terminal app. I used python and twill--put values into forms (no JavaScript and the form IDs never changed), login, search for the class, and scrape.
Same at UCSC, I even considered trying to monetize the script given what an atrocious system it was.
Odd that this is still an issue 20 years after the first time I heard about this.
This reminds me of the story of beartracks, and how it was shut down by the University of Alberta to make everyone register by phone .. and eventually was brought back by the university.
Tried to find a link I had read about the story, but can't seem to find it.
This reminds me of the story of beartracks, and how it was shut down by the University of Alberta to make everyone register by phone .. and eventually was brought back by the university.
Tried to find a link I had read about the story, but can't seem to find it.
https://github.com/ChrisCinelli/scraperNotifier - This can check if regex exist or do not exist. It send an email or a message on Slack using a hook.
I used successfully with Eventbrite for hackthons sign up.
I used successfully with Eventbrite for hackthons sign up.
I've written a similar thing for the Dell Outlet: https://outletmeknow.redfern.me/. SMS notification isn't actually working at the moment as I've run out of Twilio credit, but Email should be fine.
I remember I made this same thing when I was in undergrad b/c my uni didn't have waitlists. So people could waitlist on my app. Their priority in the waitlist is determined by a market.
I remember getting SMS's in the middle of the night alerting me that a seat opened up!
I remember getting SMS's in the middle of the night alerting me that a seat opened up!
"Their priority in the waitlist is determined by a market."
I'm very surprised registration systems aren't (generally) some kind of phased auction. There's got be some research, experiments somewhere exploring different strategies.
I worked on student facing software in higher ed. Admins were obsessed with fairness. Our registration systems had all sorts of arcane rules. Which were hard to explain, validate, troubleshoot. And probably neither fair or effective.
Some kind of auction, where blocks of seats were made available to different populations, progressively over time, would greatly simplify the implementation and understanding.
For (a greatly simplified) example, GRADUATION REQUIREMENT ABC has 200 seats. 50 seats are released every hour. First hour, seniors have first shot. Second hour, opens up for juniors and program enrollees. Third hour, sophomores. Fourth hour, freshmen.
Of course, there are other factors, like multiple sections (same class offered at different times).
--
We also talked about predicting demand (capacity planning). A novel solution there might moot the entire registration stampede. Perhaps a "buying club" type solution, where students state which classes they'll "buy" and roughly when. Then registrars form up sections to satisify the largest number of students. This could reduce the twin connundrums of waitlists and over capacity.
Anywho. It's an interesting problem, ripe for innovative optimization, matchmaking, auction/market solutions.
I'm very surprised registration systems aren't (generally) some kind of phased auction. There's got be some research, experiments somewhere exploring different strategies.
I worked on student facing software in higher ed. Admins were obsessed with fairness. Our registration systems had all sorts of arcane rules. Which were hard to explain, validate, troubleshoot. And probably neither fair or effective.
Some kind of auction, where blocks of seats were made available to different populations, progressively over time, would greatly simplify the implementation and understanding.
For (a greatly simplified) example, GRADUATION REQUIREMENT ABC has 200 seats. 50 seats are released every hour. First hour, seniors have first shot. Second hour, opens up for juniors and program enrollees. Third hour, sophomores. Fourth hour, freshmen.
Of course, there are other factors, like multiple sections (same class offered at different times).
--
We also talked about predicting demand (capacity planning). A novel solution there might moot the entire registration stampede. Perhaps a "buying club" type solution, where students state which classes they'll "buy" and roughly when. Then registrars form up sections to satisify the largest number of students. This could reduce the twin connundrums of waitlists and over capacity.
Anywho. It's an interesting problem, ripe for innovative optimization, matchmaking, auction/market solutions.
I'm very surprised prereigstration systems exist at all for college level lecture courses. Personally, I prefer the system where anyone can sign up for (almost) any class and then the university hires/reassigns teaching assistants as needed:
http://www.thecrimson.com/admissions/article/2013/9/10/cours...
http://www.thecrimson.com/admissions/article/2013/9/10/cours...
[deleted]
I did a similar thing like this back in university, I made it a mobile app with notification and passed around. It got popular and I received cease and desist from my university shortly afterwards, now the system has a captcha on every page haha.
Is there a generic page scraper library or toolset? I can see a ton of demand here and it seems like something you could use on your day to day. Things like this actually seem like an outlier, I'd imagine airfare would be a huge one.
The nearest thing I can think of is Scrapy (https://scrapy.org/).
For airfare, I think there's already a few players in the "this flight got cheaper" space (Kayak and Hopper come to mind), but they don't have data on some airlines (because of the airlines' ToS IIRC).
For airfare, I think there's already a few players in the "this flight got cheaper" space (Kayak and Hopper come to mind), but they don't have data on some airlines (because of the airlines' ToS IIRC).
Someone started charging for this service near the end of my time at University of Washington. I think the university implemented their own notification system so that class registration didn't become pay to win.
Why not just send yourself an email? Much simpler than using Twilio. Your smartphone will alert you either way.
for sending yourself updates when a bug is encountered or a condition is met, isn't it better to use something like PushBullet instead of normal text messages?
A lot of things here smells like pure advertisement. hmm...
Can we stop calling screen scraping scripts a hack?
Cool! But I'd use a Telegram bot for free. :-)
Now just figure out a way to sell the whole solution back to the university, and you'll have paid for your tuition fees.
Sounds like the quickest way to get a call from the university's IT department…
Probably the word "hacked" made it click baity, atleast in my opinion. Automation, yes. Not sure there was a vulnerability involved.
It's also a bit risky, as some University administrators may not understand that he isn't actually being malicious. Automated scraping can be considered a violation of some IT policies though, so OP should be careful about something like this.
I'm in higher-ed infosys and we see a number of automated registration sniping hacks in our monitoring / metrics. We usually don't run it up the chain unless it's malicious or impacting the service for other students. Otherwise we just investigate it a little bit and share it on Slack with the group if it's clever. The only time we've ever reached out to a student is to ask him to stop sharing it with others (and fix it) because it was performing some really aggressive polling.
Some other schools in our system are really reactionary, though, and consider any automation a ToS violation and will freak out.
e: And if you know the URL pattern / platform of your Uni's registration system, there's probably already a couple of examples on github of a registration bot.
Some other schools in our system are really reactionary, though, and consider any automation a ToS violation and will freak out.
e: And if you know the URL pattern / platform of your Uni's registration system, there's probably already a couple of examples on github of a registration bot.
I've seen where it could also be seen as a violation of the academic honor code. Georgia Tech has some serious warnings about trying this sort of stuff.
A 'neat hack' is the original use of the word hack. Hacking does not at all require the involvement of vulnerabilities.
Agreed, but it's just automation at play here. Clever, and will save time though, just thought that the headline was a little misleading, considering a registration system was involved.
Eh, I haven't read the article but I'm pretty sure hacking is doing something outside its intended purpose.
Agreed, but it's just automation at play here. Clever, and will save time though, just thought that the headline was a little misleading, considering a registration system was involved.
Apologies for the mislead! When I titled this, I imagined talking a lot more about stepping through the registration system in something like mitmproxy to find the exact requests to send, which feels a lot closer to hacking/reverse engineering. I ended up not really talking about that, but neglected to change the title.
I'd be worried if it was actually about "hacking", because the result is usually very severe. Universities are still quite backwards went it comes to security unfortunately.
[deleted]
The immigration website of Saskatchewan province opens up randomly to apply for immigration. I missed it many times because there is no indication other than the "Apply" button being enabled and a small text in their homepage which says "Applications are now 'open' ". They will close the application intake when they have reached X number of applicants. So timing is very important.
So I hacked up a script which diffs their home page every 10 minutes for "open" regex. When there is an "open" keyword in the diff, the python script calls twilio API to make a phone call to me along with an SMS.
So this script was running in AWS for many weeks and one day I got the call. Logged in to Saskatchewan's immigration homepage and applied. Now I am in Canada as a permanent resident. Thanks to Twilio.
edit: add H1B to make clear which type of immigration is broken IMO.