Automate the Boring Stuff with Python (2015)(automatetheboringstuff.com)
automatetheboringstuff.com
Automate the Boring Stuff with Python (2015)
https://automatetheboringstuff.com/2e/
73 comments
I had the same exact experience, however no one really cared when I explained my spreadsheet automation (I’m partially to blame there), so I quit my road to nowhere finance job to move to Australia and also change careers with a boot camp. 1000% changed my life for the best.
As someone in a bootcamp right now, I'm really happy to hear this! Im hell-bent on starting a new career as a dev
The most valuable thing you will bring to the world of tech is first hand knowledge of genuine business problems that can be solved.
1000x this.
Figure out how to put your industry knowledge to work and you'll reap massive rewards. At most early-stage startups, programming & industry knowledge is generally split between two or more people, which slows things down tremendously.
If you're both of those people, you can build something that you know people will want, and likely faster, because you were the customer, and you will know exactly what will—and won't—matter. If that's not a startup superpower, I don't know what is.
Figure out how to put your industry knowledge to work and you'll reap massive rewards. At most early-stage startups, programming & industry knowledge is generally split between two or more people, which slows things down tremendously.
If you're both of those people, you can build something that you know people will want, and likely faster, because you were the customer, and you will know exactly what will—and won't—matter. If that's not a startup superpower, I don't know what is.
I expected this story to end with you selling your tools to other accountants or businesses. Seems like an easy business opportunity. Did you have any thoughts about that?
I think building a superior product would be trivial and profitable for a reasonably funded, small team of developers. Most of the accounting profession uses tools that are from the 80s and 90s, so the user experience is horrible for staff accountants.
The main problem is that selling to accounting firms is difficult because procurement usually goes through partners. At the firms I worked for, our procurement process required buy-in from a majority of the partners and the product would require serious security features that are beyond my current abilities, so those two facts deterred me from pursuing that route.
The real future is integrating audit software with financial bookkeeping. This is one area where I think blockchain/smart contracts actually could be utilized to ensure audit quality. Auditors essentially look at a small portion of the transactions made by a company during a year, so software could monitor it in parallel. You’d still need an audit team to review the findings and ensure the software was working properly, but the cost would be a fraction of what it takes to audit a company by hand today.
Thank you for asking — it’s been a while since I’ve been able to geek out like this! One day I’d like to build those tools, but in the meantime I’m working on a personal finance product. It’s almost done, so hopefully you’ll see it when I submit it to HN to get dragged across the coals.
The main problem is that selling to accounting firms is difficult because procurement usually goes through partners. At the firms I worked for, our procurement process required buy-in from a majority of the partners and the product would require serious security features that are beyond my current abilities, so those two facts deterred me from pursuing that route.
The real future is integrating audit software with financial bookkeeping. This is one area where I think blockchain/smart contracts actually could be utilized to ensure audit quality. Auditors essentially look at a small portion of the transactions made by a company during a year, so software could monitor it in parallel. You’d still need an audit team to review the findings and ensure the software was working properly, but the cost would be a fraction of what it takes to audit a company by hand today.
Thank you for asking — it’s been a while since I’ve been able to geek out like this! One day I’d like to build those tools, but in the meantime I’m working on a personal finance product. It’s almost done, so hopefully you’ll see it when I submit it to HN to get dragged across the coals.
I’m working on financials-related software that has increasing involvement with our internal compliance/audit team.
I totally agree with your premise, although in practice I’ve found that the phrase “review the findings and ensure the software was working properly” snuggles in a lot of complexity. It implies that working properly is well-defined and formally verifiable, when it usually isn’t.
I’ve had very frank meetings where I’ve explained that, ultimately, log statements can be totally fictional, and we can technically pay out amounts that diverge from the amounts in a report. The amount of trust you have in the developer and your dependencies are the weakest links in the chain, and, if you’re honest in explaining to that level of detail, most compliance minded people come away uneasy. But this is how every company operates?
There’s definitely a need for verifiable money transfer infrastructure, but I think most companies that roll their own payment infrastructure would rather pretend that the problem is a boogeyman.
I totally agree with your premise, although in practice I’ve found that the phrase “review the findings and ensure the software was working properly” snuggles in a lot of complexity. It implies that working properly is well-defined and formally verifiable, when it usually isn’t.
I’ve had very frank meetings where I’ve explained that, ultimately, log statements can be totally fictional, and we can technically pay out amounts that diverge from the amounts in a report. The amount of trust you have in the developer and your dependencies are the weakest links in the chain, and, if you’re honest in explaining to that level of detail, most compliance minded people come away uneasy. But this is how every company operates?
There’s definitely a need for verifiable money transfer infrastructure, but I think most companies that roll their own payment infrastructure would rather pretend that the problem is a boogeyman.
Yup! I got started with this book and automated a bunch of things in one of my internships about 3 years ago. I was in a network administration focused program in university. I decided to automate a ton of the repetitive tasks that people on my team had to do (creating config files with specific line endings for finicky cisco devices, creating an inventory of all our network connected devices, parsing and comparing large spreadsheets, etc).
Looking back, my code was absolutely atrocious and I was on a team of 1 (just me) but this book and that 4 month co-op is essentially what got me to transition from applying to network admin roles after graduation to applying and getting a platform engineering focused role and focus on software development. It's what I've been doing ever since!
Looking back, my code was absolutely atrocious and I was on a team of 1 (just me) but this book and that 4 month co-op is essentially what got me to transition from applying to network admin roles after graduation to applying and getting a platform engineering focused role and focus on software development. It's what I've been doing ever since!
Wow! That’s awesome! I’m happy for you, congrats!
My story is mostly similar to yours (though not related to this book), appreciations for writing it out.
Did you abandon public accounting entirely or are you now in a mixed role?
Thanks for sharing, I‘m going to try doing the same.
Incredible story
I started reading this book this week in an effort to open up my job prospects down the line. I'm a "technical" Industrial Engineer often serving as a liason between IT and the business. I can program in VBA and do anything you can name in Excel, but I want to move to a more programming oriented role down the line.
I want to be able to do statistical analysis, queries, and just "be better" at breaking down a problem into easily solvable steps.
I'm using this book as a "learn the basics" stepping stone, and Project Euler as a list of solvable problems which will let me practice solving problems. After that I'll move into some personal projects I have in mind.
Does anyone have additional book or resource recommendations beyond that? I know packages like NumPy and/or SciPy will be useful down the line.
* Codecademy feels overpriced at $40/month or $250/year (no refunds for using a partial year).
* Google Python class was free but about 10 years old. With the Python 2->3 debacle, I was unsure if fundamentals had changed and hesitant to sink my time into it.
* Lynda runs $25 or $40/month but would offer python as well as many other resources.
* Udemy - has a number of courses including one by Al, the author of this book. Looked like it'd be a good value, but I'm not sure how much more value the videos add than the book.
I want to be able to do statistical analysis, queries, and just "be better" at breaking down a problem into easily solvable steps.
I'm using this book as a "learn the basics" stepping stone, and Project Euler as a list of solvable problems which will let me practice solving problems. After that I'll move into some personal projects I have in mind.
Does anyone have additional book or resource recommendations beyond that? I know packages like NumPy and/or SciPy will be useful down the line.
* Codecademy feels overpriced at $40/month or $250/year (no refunds for using a partial year).
* Google Python class was free but about 10 years old. With the Python 2->3 debacle, I was unsure if fundamentals had changed and hesitant to sink my time into it.
* Lynda runs $25 or $40/month but would offer python as well as many other resources.
* Udemy - has a number of courses including one by Al, the author of this book. Looked like it'd be a good value, but I'm not sure how much more value the videos add than the book.
> "be better" at breaking down a problem into easily solvable steps
i highly recommend the courses how to code: simple data and how to code: complex data to supplement your learning to address the quoted goal. don’t be put off by the teaching language or the simplicity at the start. it gets more complicated as you go. the courses teach systematic program design and are based upon the book how to design programs.
everybody and their dog knows python. you will not separate yourself by just knowing python. so in the long term, you need other skills beyond “just programming”. and much of the industry is moving to functional-first multiparadigm langauges and more and more concurrent/asynchronous programming. python is a terrible choice for both of those and there is no clear evolution for python in those directions. it is a very outdated language in my opinion, its usage and amount of libraries available for it aside.
by all means, learn python, since there are a lot of resources and jobs that need it, but i highly recommend also learning a functional language. my recommendations would be f# (a .net language) and elixir. f# is a very ripe language for automation, probably even more so than python.
there are also python courses by MIT on edX you should check out. (actually, they are more computer science courses that use python, but there is an advanced one for using python for scientific applications.)
i highly recommend the courses how to code: simple data and how to code: complex data to supplement your learning to address the quoted goal. don’t be put off by the teaching language or the simplicity at the start. it gets more complicated as you go. the courses teach systematic program design and are based upon the book how to design programs.
everybody and their dog knows python. you will not separate yourself by just knowing python. so in the long term, you need other skills beyond “just programming”. and much of the industry is moving to functional-first multiparadigm langauges and more and more concurrent/asynchronous programming. python is a terrible choice for both of those and there is no clear evolution for python in those directions. it is a very outdated language in my opinion, its usage and amount of libraries available for it aside.
by all means, learn python, since there are a lot of resources and jobs that need it, but i highly recommend also learning a functional language. my recommendations would be f# (a .net language) and elixir. f# is a very ripe language for automation, probably even more so than python.
there are also python courses by MIT on edX you should check out. (actually, they are more computer science courses that use python, but there is an advanced one for using python for scientific applications.)
You might check with your public library. Ours offers free access to Lynda.com from your home by redirecting through the public library's websitt.
My old library provided this, but unfortunately it's not available in my new city.
You can't still use it with your "old library"?
It's been a while, but I used to have "library cards" from libraries in all my nearby cities.
It's been a while, but I used to have "library cards" from libraries in all my nearby cities.
The Ames, Iowa city library expires cards on an annual basis, likely because so many students cycle through.
Thinking about it now, I think my Lynda access was through Iowa State.
Thinking about it now, I think my Lynda access was through Iowa State.
I really liked “Problem Solving With Algorithms and Data Structures Using Python”.
https://runestone.academy/runestone/books/published/pythonds...
https://runestone.academy/runestone/books/published/pythonds...
[deleted]
To offer a different perspective- my favorite way of learning to code has been to find coding puzzles (like codewars.com) and try to tackle those. I ended up learning a lot about control structures, data structures, and logic through there.
Programming is inherently creative and few books that I've read have room for creativity. I find it better to discover information than get it from a curriculum.
Programming is inherently creative and few books that I've read have room for creativity. I find it better to discover information than get it from a curriculum.
I think exercism.io is great
check out https://freecodecamp.org
Got hardcopy of this book for Xmas from my 3 year old (yea OK the wife!) about a 1/3 into it.
Have programmed c++ about a decade ago.. Nice to get back into it I have to say, there is an instant gratification from solving something in code, perhaps is the binary nature of something working or not, it's very black and white versus dealing with management issues, which is more like my day job, where your constantly in some grey area or dealing with noise, emails, endless directionless meetings.
Anyway highly recommend this book for non programmers, MU is also a very nice and clean IDE but prefer VS Code.
Have programmed c++ about a decade ago.. Nice to get back into it I have to say, there is an instant gratification from solving something in code, perhaps is the binary nature of something working or not, it's very black and white versus dealing with management issues, which is more like my day job, where your constantly in some grey area or dealing with noise, emails, endless directionless meetings.
Anyway highly recommend this book for non programmers, MU is also a very nice and clean IDE but prefer VS Code.
I love this book! I bought the physical copy years ago and it has frequently been a go-to book of mine. So many tech books (not all, of course) either do a great job of theory but are of little immediate, practical value or they are "cookbook" style and give you lots of useful tidbits but fail to help you grasp the big picture. While there are valid reasons for both of these approaches, of course, this gem is one of the few that are able to do both in one text. You learn a new computer language in the first part, and as soon as you have a foundation in the basics, you immediately jump into practical applications that can benefit your life almost immediately. Very well done, IMHO.
Aww I wrote the Python docx module years ago when I first got into programming as a sysadmin (I was doing lots of xml munging so knew lxml pretty well at the time). Nice to see it included in a guide like this.
Thank you for writing it. I used it for the first time last week and it made my job so much easier than the other solutions I was looking at.
Thanks for crafting this! While i've only used this module for learning/playing around (and not yet for production needs), I'm very appreciative and thankful for people like you!!!
The problem with automating 'every day stuff' is that processes aren't formalized in most cases. So there isn't an interface to implement against. There are no clear or reliable sources of data. Management needs to understand the issue, and if it doesn't there isn't much you can do as a single individual. So I more and more often tend to ask myself, if it is really feasible to automate task X.
A key question is "will this task benefit from formalization". If the answer is Yes, one can begin to structure it and gain benefits quite quickly. Clear and relatively quick gains, and helping out the people that do these tasks today, is key to buy-in. Then once a structure is in place, one can consider automating it.
I was all gung-ho about automating some things at a job I had but soon found out I couldn't for legal reasons. Even a script to change formats of videos would be seen as tampering by certain legal folks. Sometimes you don't know or can't see the consequences of things you're no privy to.
Honest question -
I'm very familiar with Ruby and for my automation tasks I tend to use it almost exclusively. Of these, is there anything that -
1. Ruby can't do
2. Or is more complex / complicated to do in Ruby
I'm very familiar with Ruby and for my automation tasks I tend to use it almost exclusively. Of these, is there anything that -
1. Ruby can't do
2. Or is more complex / complicated to do in Ruby
I'd say the languages are largely equivalent. Roughly the same level of abstraction, roughly the same technical trade-offs. Highly productive, lots of people enjoy them, not very strict.
What Python does have specifically for automation in my eyes is that it is often the default language for most Raspberry Pi-related hardware libraries. So if you want to build home automation on accessible and easy hardware, the Pi is very welcoming to Python just in terms of making things easy, providing libraries.
It is also heavily supported for machine learning and computer vision which is related. Again, this is generally just a matter of library support.
I don't know the Ruby ecosystem well enough to say if it has equivalents to all the Python scraping, fuzzy matching and such libraries out there. But I expect it does. So that bit is probably a wash.
Hope that gives a perspective.
What Python does have specifically for automation in my eyes is that it is often the default language for most Raspberry Pi-related hardware libraries. So if you want to build home automation on accessible and easy hardware, the Pi is very welcoming to Python just in terms of making things easy, providing libraries.
It is also heavily supported for machine learning and computer vision which is related. Again, this is generally just a matter of library support.
I don't know the Ruby ecosystem well enough to say if it has equivalents to all the Python scraping, fuzzy matching and such libraries out there. But I expect it does. So that bit is probably a wash.
Hope that gives a perspective.
Yes, I know about Python's extensive ML libraries.
Didn't know about it being the default language for Raspberry Pi libraries.
Thanks. I've been planning to look at both learning some ML and doing some small Raspberry Pi projects. That's good motivation for diving into Python!
Didn't know about it being the default language for Raspberry Pi libraries.
Thanks. I've been planning to look at both learning some ML and doing some small Raspberry Pi projects. That's good motivation for diving into Python!
1. no
2. no, not really - however there may be libraries that python has that ruby doesn't have
Side note: Ruby has more syntax to grok, which makes it harder to read (for me). On the other hand Python does have its own idiosyncrasies and subtle behaviors. However, Ruby also has some features (like multiline inline functions) that one may easily miss in Python, but they haven't been worth switching over for me.
Side note: Ruby has more syntax to grok, which makes it harder to read (for me). On the other hand Python does have its own idiosyncrasies and subtle behaviors. However, Ruby also has some features (like multiline inline functions) that one may easily miss in Python, but they haven't been worth switching over for me.
Python has multiline inline functions--perhaps you mean lambda?
my bad: meant multiline lambdas.
Honestly, I've done a lot of professional work in both languages, and there is pretty much nothing you can't do with ruby that you can in python. There is no intrinsic difference in power of expressiveness between the languages. Although ruby is lagging a little behind with the asyncio syntactic sugar stuff. You may reach a point where there simply isn't any client written for the service you are trying to automate in either language. And python got more love in that aspect what with all the being the number one preferred scripting language on planet earth, but in case your target of automation provides a simple rest or websocket protocol you could easily do without a dedicated client library for it.
Thanks. I really don't have any idea about the asyncio syntatic sugar that you mention. Can you point me to some books / articles?
Oh async/await is just python's and javascript's way of executing functions concurrently within the same process. asyncio emerges as a rewrite of a standard library to support this idiom. Browsers that execute javascript ES6 and node v7 onwards have this feature. Instead of passing callbacks to functions and have them executed concurrently, you create a task or future (in python 3.6+) or a Promise (in javascript) and await that. That is very useful because it untangles the callback-inside-callback structure.
There's a comparison between node and python's async/await syntaxes here https://medium.com/@interfacer/intro-to-async-concurrency-in...
For ruby I found this article which shows the gist of what we are trying to accomplish with async/await (we want to gather a bunch of promises and await all their results and have them run concurrently in an event loop so that when one is waiting for IO to complete, the other one takes over and runs) https://www.codeotaku.com/journal/2018-06/asynchronous-ruby/...
There's a comparison between node and python's async/await syntaxes here https://medium.com/@interfacer/intro-to-async-concurrency-in...
For ruby I found this article which shows the gist of what we are trying to accomplish with async/await (we want to gather a bunch of promises and await all their results and have them run concurrently in an event loop so that when one is waiting for IO to complete, the other one takes over and runs) https://www.codeotaku.com/journal/2018-06/asynchronous-ruby/...
I understand async/await as I'm conversant with NodeJs. When you mentioned asyncio, I assumed it be something specific in the Python library.
In case of Ruby, I've found these libraries (ruby gems) to be very useful.
1. Sidekiq - https://github.com/mperham/sidekiq
2. RocketJob - https://github.com/rocketjob/rocketjob
What I'm guessing you're saying is that this kind of behavior is available natively in Python, whereas in Ruby, it is achieved by some roundabout fashion.
In case of Ruby, I've found these libraries (ruby gems) to be very useful.
1. Sidekiq - https://github.com/mperham/sidekiq
2. RocketJob - https://github.com/rocketjob/rocketjob
What I'm guessing you're saying is that this kind of behavior is available natively in Python, whereas in Ruby, it is achieved by some roundabout fashion.
These gems are not really related to the asyncio stuff in python. (They're async job runners though... overloaded names can be annoying)
async/await works at a function level in the same process. Asyncio in python is pretty much the async/await from node/js.
Sidekiq alternative in python would be closer to http://www.celeryproject.org/
async/await works at a function level in the same process. Asyncio in python is pretty much the async/await from node/js.
Sidekiq alternative in python would be closer to http://www.celeryproject.org/
Yes, async functions are natively available in python3.6+ and node7+, and requires a gem in ruby. Ruby can also use fibers and eventloop to do concurrency using just the standard library, but it is missing the async/await syntactic sugar.
> [Python] being the number one preferred scripting language on planet earth
Is it?
I'd imagine JavaScript usage/execution is probably way way higher.
Is it?
I'd imagine JavaScript usage/execution is probably way way higher.
Yes python is more preferred when choosing a scripting language and javascript is more executed in browsers. And the two do not contradict.
I think you're both correct. For sysadmin definition of scripting, python would be most popular. For the 'fast to write' definition of scripting then sure, JavaScript.
But people don't use javascript to write scripts. Javascript is mostly used in writing web services.
I mostly write javascript now for my day job... But python is nicer for scripts... The asynchronous nature of node makes it a pain for me
Async/await largely solves that though.
To an extent...
If you're using something that supports promises and not callbacks, which not all of the things you want to use to write an actual script do.
You can easily transform a callback into a promise via simple glue-functions:
let result = await CallbackPromise.from(some.callbackConventionFunction, param);
Python is significantly more popular than Ruby. It doesn't mean that it's a better programming language, but a library for X or an example of how to do X are significantly more likely to exist for Python than for Ruby, and there are other popularity advantages like more implementations.
There's micropython for the esp8266... not sure if ruby has been ported
If you like ruby, checkout crystal.
Honestly, variations of this question has been asked too many times to be taken as an “honest question”. See also: Rails developer hijacking Django thread.
It’s weird lots of Ruby developers feel the need to shout “don’t forget ruby!” when nice things about Python are posted. Just take your pick, who cares.
Btw, I know, assume good faith, yada yada.
It’s weird lots of Ruby developers feel the need to shout “don’t forget ruby!” when nice things about Python are posted. Just take your pick, who cares.
Btw, I know, assume good faith, yada yada.
I researched Ruby, but the syntax idiosyncrasies of the language made it appear to be created by an amateur. Like weird language constructs to do something, that makes you go, "huh! What kind of random junk is this?"
Some people obviously swear by it, and thus created Ruby on Rails.
Python does have some weird features as well, but I try to steer away from it, and keep my code simple and boring. My primary concern is how efficient the code is executed as it gets translated into machine code.
The cost/benefit ratio of switching to Ruby never quite seemed worth it. So I continue to stick with Python, for applications that are not time execution sensitive. For all others, there's Java, then C++, then C itself.
Some people obviously swear by it, and thus created Ruby on Rails.
Python does have some weird features as well, but I try to steer away from it, and keep my code simple and boring. My primary concern is how efficient the code is executed as it gets translated into machine code.
The cost/benefit ratio of switching to Ruby never quite seemed worth it. So I continue to stick with Python, for applications that are not time execution sensitive. For all others, there's Java, then C++, then C itself.
you say “created by an amateur” like it’s an insult.
some of the best things we have are created by total amateurs. thats how this works.
i’m not 100% on the life histories of Matz and Guido but wasnt Guido kind of also an amateur too?
some of the best things we have are created by total amateurs. thats how this works.
i’m not 100% on the life histories of Matz and Guido but wasnt Guido kind of also an amateur too?
They were both involved academically with language design.
"...Python, for applications that are not time execution sensitive. For all others, there's Java, then C++, then C itself."
That is assuming you know how to work with data structures and algorithms the right way. A O(n^n) implementation runs slow in C as well.
That is assuming you know how to work with data structures and algorithms the right way. A O(n^n) implementation runs slow in C as well.
This should not be dated 2015 because this is the second edition that just came out today.
This is the second edition, which is new, so it shouldn't be (2015).
I’m right there with ya calderarrow. I’m a middle aged physician with no prior programming experience, but I picked this book up on a whim a few years ago and it changed my life. I started automating little thing a here and there and now I see opportunities everywhere that a “real developer” could capitalize on. For as expensive as EMR’s are, you will NEVER meet a doctor who loves theirs, they just hate some less than others. Patient portals are even worse. This industry is ripe for disruption.
I am biased because the author is a friendly acquaintance, but this is a great book (and Al is a great person.)
Discussed at the time:
https://news.ycombinator.com/item?id=9415891
I had no prior experience with programming, and at the time I only wanted to learn how to write enough code to help me with my job. This book was perfect for that goal.
It took me a couple of weeks to get through it all and write the program I needed to write, but when it worked, I was amazed. I remember going to grab coffee to celebrate and on my walk I started thinking about other job tasks I could automate. Most of public accounting deals with comparing PDF reports and Excel data, so I genuinely believed I could write programs to automate the majority of tasks at my job.
I started to learn on my own with some common online resources. I would get so excited to come home from work so I could dive further into my studies, and the more I learned, the more opportunities I saw: what if there was a way for a computer to perform all the analysis and statistical processing for an audit? What if a program could monitor financial transactions and automatically complete taxes for clients?
Flash forward a few months and I went all-in on software development. Quit my job as a public accountant, finished a boot camp, and started working as a developer.
It was the best decision of my life and this book was the catalyst. I’m so grateful that Al wrote this, and I highly recommend it for people whose jobs have a lot of boring stuff —- especially public accountants :).