A Go “clone” of the great and famous Requests library(github.com)
github.com
A Go “clone” of the great and famous Requests library
https://github.com/levigross/grequests
76 comments
I personally welcome those kind of ports and hate blind insistent on "idiomatic" code.
Idiomatic Java, Perl and other languages were shit for a long time, they only improved when some brave soulds fed up with it and explorer non-idiomatic solutions (like the Play framework). And some things I see in "idiomatic" Go make my hair crawl...
Idiomatic Java, Perl and other languages were shit for a long time, they only improved when some brave soulds fed up with it and explorer non-idiomatic solutions (like the Play framework). And some things I see in "idiomatic" Go make my hair crawl...
I'm going to disagree with you there.
Idiomatic code is so called, because people familiar with the language can quickly read it. It makes reading code a much lighter task.
Frameworks are a different thing. Frameworks build around its host language, but normally are meant to use in a specific way. In its own idiomatic way.
Idiomatic code is so called, because people familiar with the language can quickly read it. It makes reading code a much lighter task.
Frameworks are a different thing. Frameworks build around its host language, but normally are meant to use in a specific way. In its own idiomatic way.
>Idiomatic code is so called, because people familiar with the language can quickly read it. It makes reading code a much lighter task.
In the case of Java circa 2000-2006 it needlessly messed up code, making it more difficult and slow to read EVEN for people familiar with the language.
Following conventions doesn't make code better to read automatically just because people are already familiar with them -- those conventions should be good in the first place for that to happen.
Node has the callback hell "idiom" many are familiar with. And yet if somebody rewrote the code with async, await, even the people "familiar" with callbacks would be able to read it faster and understand it better.
In the case of Java circa 2000-2006 it needlessly messed up code, making it more difficult and slow to read EVEN for people familiar with the language.
Following conventions doesn't make code better to read automatically just because people are already familiar with them -- those conventions should be good in the first place for that to happen.
Node has the callback hell "idiom" many are familiar with. And yet if somebody rewrote the code with async, await, even the people "familiar" with callbacks would be able to read it faster and understand it better.
I don't think this is universal as it hasn't applied where I work. Without understanding how promises work they are no easier and it doesn't help that many things exists that do not use the standard .then() and .catch().
Async/await is not the same as bare promises though with then() and catch.
What did your hair do when you saw idiomatic perl?
It stood up and whispered to my ear to move to Python, which I did, circa 1997.
When you realised that Python had extremely similar problems to Perl but hadn't even managed things like Unicode support in 2015, what did you do?
I immediately noticed this as well. This API has flags for "asynchronous" requests, but Go style is to provide a blocking API, and let the caller decide to use a goroutine if they need concurrency.
Constructing a channel (as it does though) is just pragmatic though and definitely not any less go-like.
You get a channel - you can read from it until it's closed which is always safe. Go-routines are cheap so you don't care about the background details. For simplifying HTTP, this is great.
You get a channel - you can read from it until it's closed which is always safe. Go-routines are cheap so you don't care about the background details. For simplifying HTTP, this is great.
Originally the API was completely asynchronous (every "request" function returned a channel).
I got a lot of feedback that I shouldn't do that, so I moved the asynchronous requests to specific functions e.g GetAsync and made the standard API synchronous.
I got a lot of feedback that I shouldn't do that, so I moved the asynchronous requests to specific functions e.g GetAsync and made the standard API synchronous.
You should go further, removing the async methods altogether. In Go, making sync APIs async is trivial and well-understood.
The first project I published in Go almost 3 years ago, was a similar port of the python logger. While today I find this "I want to log like I do in python" approach a bit silly, IMO the Go standard logging package is still lacking, and I'm still using my python-ish port, not because it's similar to python, but because it has things like formatting, context printing and most importantly logging levels.
What's your take on Logrus? It seems like it would cover your major concerns, and it's widely recommended.
https://github.com/Sirupsen/logrus
https://github.com/Sirupsen/logrus
it's nice, very similar to my API sans the structured messages thing.
I am not trying to do things in the "Python way". I am trying to bring some of the same flexibility I get when using the Python requests library to Go.
There's nothing wrong with porting something to another language so your project Stack is all in 1 language. Nuts to having a million languages and cryptic dependencies. More singular platforms are easier to maintain for sysadmins and developers alike.
If the porting language isn't as expressive as the original source -- oh well thats a price i'm willing to pay.
If the porting language isn't as expressive as the original source -- oh well thats a price i'm willing to pay.
Very poorly named given Requests has a branch using Gevent that's already called grequests. https://github.com/kennethreitz/grequests
I didn't even notice that part, very true. I think it should probably be gorequests. Sounds better than grequests anyways.
I agree, gorequests is the obvious choice, and changing the name before it becomes too popular would be a good idea.
sounds like a great FPS title from the mid-90's. GoreQuests, Rated M for Mature.
There are so many packages with "go" in the name. Why not be a little more creative and avoid that? I don't put the word Go in any of my package names and it's been fun.
Gore Quests? Are you sure? :-)
Yeah, I definitely just reread that post and realized what it said. Maybe go-requests or something...
I am not very good at naming things and figured to just slap a 'g' on requests => grequests.
I figured that I would write the code and decide on a name later...
I figured that I would write the code and decide on a name later...
Whenever I'm reading through Go code and come across something that uses this sort of abstraction library, I usually just close it and move on. The stdlib is really nice and I find code that stays as "vanilla" Go as possible to be the easiest to read.
It's not the first time I've heard this sentiment around Go. I find it a fascinating difference in mindset to for example Node.js where in my experience even the smallest application will pull in dozens of dependencies and there's a package for absolutely everything imaginable. Because of the way dependencies are hierarchically scoped, there's no version conflicts in Node and a project may end up with 3 different Requests modules with only disk space as the cost.
What do you consider to be the downsides to liberally using libraries that reduce the amount of code by wrapping the standard lib or that build on top of other libraries? Personally, and I may be wrong as someone still new to Go, it seems to me that the single monolithic repository model that Google uses permeates the whole community and has made adding an external dependency a Big Deal and something you have to carefully consider instead of something you instinctively do.
What do you consider to be the downsides to liberally using libraries that reduce the amount of code by wrapping the standard lib or that build on top of other libraries? Personally, and I may be wrong as someone still new to Go, it seems to me that the single monolithic repository model that Google uses permeates the whole community and has made adding an external dependency a Big Deal and something you have to carefully consider instead of something you instinctively do.
Simply put: dependencies are code, and code comes at a maintenance cost. Thin abstractions, such as a this library, can be replaced by a function or two, which is a lower maintenance overhead because there is less code, you wrote the code, and all the code is being used.
IMHO, dependencies should be pulled in for things that would take you weeks to do correctly. Avoid pulling in a dependency for something you can do yourself in 5 minutes. So I would pull in an http library if it actually did the mechanics of http better than the one in the stdlib, but not one that just wrapped it up for me a bit.
For me, that's not a Go thing, that's just an I'm Older Now thing.
Also, snarkily, dependency management is a mess in Go right now; none of us have any dependencies because we can't figure out how to do it. :P That is definitely a hole right now (though it's starting to get plugged in 1.5 with the /vendor directory) and is definitely exacerbated by the core team not being able to use any solution they could provide for the community.
IMHO, dependencies should be pulled in for things that would take you weeks to do correctly. Avoid pulling in a dependency for something you can do yourself in 5 minutes. So I would pull in an http library if it actually did the mechanics of http better than the one in the stdlib, but not one that just wrapped it up for me a bit.
For me, that's not a Go thing, that's just an I'm Older Now thing.
Also, snarkily, dependency management is a mess in Go right now; none of us have any dependencies because we can't figure out how to do it. :P That is definitely a hole right now (though it's starting to get plugged in 1.5 with the /vendor directory) and is definitely exacerbated by the core team not being able to use any solution they could provide for the community.
I agree with you.
I wrote this library because I didn't want to rewrite those functions every time I started on a new project. I put it online because I figured that I would be useful to others as well.
I wrote this library because I didn't want to rewrite those functions every time I started on a new project. I put it online because I figured that I would be useful to others as well.
btw for the record I don't think there's something wrong with making small packages like this, just that I'm more likely to copy over a function or pattern that I like into my codebase, rather than bring in a dependency.
I know... But as you said (and I agreed) :) dependencies add complexity...
> What do you consider to be the downsides to liberally using libraries that reduce the amount of code by wrapping the standard lib or that build on top of other libraries? Personally, and I may be wrong as someone still new to Go, it seems to me that the single monolithic repository model that Google uses permeates the whole community and has made adding an external dependency a Big Deal and something you have to carefully consider instead of something you instinctively do.
Because there's no equivalent of npm in Golang, culturally speaking.
Because there's no equivalent of npm in Golang, culturally speaking.
forgive my ignorance, is the original library http://python-requests.org ?
If true, I don't think it's ignorance. Not everyone is a Pythonista.
Exactly how famous is this library?
Exactly how famous is this library?
The Python standard library is overall very solid. However, it has a few darker corners - and, in the past, HTTP handling was not as easy as it could be. In the age of APIs, the requests library filled this void with a compellingly simple solution and became one of the most popular libraries (with millions of downloads every month [1]).
[1] https://pypi.python.org/pypi/requests
[1] https://pypi.python.org/pypi/requests
Extremely popular - almost every HTTP client library I use has switched to requests over the last couple years.
As a proxy, check out the download counts on this tracker using data from the PyPI, where the vast majority of Python packages are installed from:
https://python3wos.appspot.com/
Looking at the top few packages: * simplejson is the upstream for the json module in stdlib which people install since it gets performance updates first * requests * six: the most common library used to bridge the Python 2/3 transition * virtualenv: near-ubiquitous development tool (it allows you to maintain a separate Python environment for each project to avoid cross-talk, and is also used by popular testing tools like tox which run your tests under a variety of Python versions) * distribute: a few years back, the stdlib setuptools module was forked for a major overhaul. That's since been merged back in but many packages still reference it, particularly in older releases * boto: AWS client library, used by the official awscli tool * pip: Python package installer, now bundled with Python but updates & older versions of Python use the PyPI version
As a proxy, check out the download counts on this tracker using data from the PyPI, where the vast majority of Python packages are installed from:
https://python3wos.appspot.com/
Looking at the top few packages: * simplejson is the upstream for the json module in stdlib which people install since it gets performance updates first * requests * six: the most common library used to bridge the Python 2/3 transition * virtualenv: near-ubiquitous development tool (it allows you to maintain a separate Python environment for each project to avoid cross-talk, and is also used by popular testing tools like tox which run your tests under a variety of Python versions) * distribute: a few years back, the stdlib setuptools module was forked for a major overhaul. That's since been merged back in but many packages still reference it, particularly in older releases * boto: AWS client library, used by the official awscli tool * pip: Python package installer, now bundled with Python but updates & older versions of Python use the PyPI version
About as famous as they come in the python community. It is the perfect example of a library done right. I've yet to meet a developer who have used it that dislike it.
It handles all of the low level stuff for you, such as sessions, cookies, gzip, form encoding POSTs, composing URLs with arguments, thread safety etc. etc. without you even having to be aware of them, all in a terse and more readable format. It does it without taking the ability away to control those if you really want to, not that you're likely to need to anyway.
Take a look at the two examples here, one using the standard library for http, urllib2, and the other using requests: https://gist.github.com/kennethreitz/973705
I want requests to exist in pretty much every language. The API is clean and simple and perfectly expressive.
It handles all of the low level stuff for you, such as sessions, cookies, gzip, form encoding POSTs, composing URLs with arguments, thread safety etc. etc. without you even having to be aware of them, all in a terse and more readable format. It does it without taking the ability away to control those if you really want to, not that you're likely to need to anyway.
Take a look at the two examples here, one using the standard library for http, urllib2, and the other using requests: https://gist.github.com/kennethreitz/973705
I want requests to exist in pretty much every language. The API is clean and simple and perfectly expressive.
Kenneth Reitz needs to do something with datetime in Python.
> Exactly how famous is this library?
It is the de-facto standard for anything that has to interact with a API, or just HTTP in general. The old way to do these sort of things was the urllib2 module (part of the Python standard library). urllib2 has some design flaws and can be a pain to work with.
It is the de-facto standard for anything that has to interact with a API, or just HTTP in general. The old way to do these sort of things was the urllib2 module (part of the Python standard library). urllib2 has some design flaws and can be a pain to work with.
Not only is it famous for actually performing HTTP requests, but it is widely considered a seminal example of excellent API design.
You'll see quite a few Python libraries that aspire to emulate this quality of API design, often adopting Requests' tagline: "XXXX for humans"
You'll see quite a few Python libraries that aspire to emulate this quality of API design, often adopting Requests' tagline: "XXXX for humans"
In the Python world it's as famous as Joda or Hibernate would be for Javaists...
[deleted]
I'm guessing this is based off of https://github.com/request/request though I'm not 100%.
It's not, and I think it's older than Node.js.
[deleted]
[deleted]
yes
What's a use case in Go for an HTTP request library interface that returns a new channel for each request? There's no good way to select on a slice of channels, is there?
No good way, no.
I consider this a Go anti-pattern. Without a good reason, do not try to provide "asynchronousness" in a library. Go natively supports goroutines and channels, and it's considered baseline skill in the language to be able to fire something off in a goroutine (it's literally a two-character keyword) and receive something on a channel, if you want to. It would be not only adequate but preferable to implement this library as a fully synchronous request system and expect the user of the library to implement what asynchronousness they may require. Your hardwiring of exactly how the "asynchronousness" works may conflict with my own needs.
All the "asynchronousness" you need to provide is already hard-wired into the Go runtime itself; what certain language communities have trained you to think is "synchronous" code already isn't. You don't have to super-duper-extra make it even more asynchronous.
I won't quite call exposing channels in a library API a code smell, it's a little too useful for that, but it's still something where you ought to pause for a moment and really think about what it means, especially if you created it rather than receiving it.
Oh, and let me be clear: Levigross, I'm seriously suggesting that you change this library wholesale to be fully synchronous, and the fact that this will be an API change is a feature, not a bug. You'll find it also simplifies the API significantly, also a feature.
I consider this a Go anti-pattern. Without a good reason, do not try to provide "asynchronousness" in a library. Go natively supports goroutines and channels, and it's considered baseline skill in the language to be able to fire something off in a goroutine (it's literally a two-character keyword) and receive something on a channel, if you want to. It would be not only adequate but preferable to implement this library as a fully synchronous request system and expect the user of the library to implement what asynchronousness they may require. Your hardwiring of exactly how the "asynchronousness" works may conflict with my own needs.
All the "asynchronousness" you need to provide is already hard-wired into the Go runtime itself; what certain language communities have trained you to think is "synchronous" code already isn't. You don't have to super-duper-extra make it even more asynchronous.
I won't quite call exposing channels in a library API a code smell, it's a little too useful for that, but it's still something where you ought to pause for a moment and really think about what it means, especially if you created it rather than receiving it.
Oh, and let me be clear: Levigross, I'm seriously suggesting that you change this library wholesale to be fully synchronous, and the fact that this will be an API change is a feature, not a bug. You'll find it also simplifies the API significantly, also a feature.
Jerf,
Thank you for the suggestion. I plan on adding some functionality to make asynchronous APIs friendlier to use (like a `Do` or `Apply` function etc...). But if you don't choose to use the asynchronous APIs everything still functions the same (in fact the asynchronous APIs use the synchronous functions and slap channels on them)
Thank you for the suggestion. I plan on adding some functionality to make asynchronous APIs friendlier to use (like a `Do` or `Apply` function etc...). But if you don't choose to use the asynchronous APIs everything still functions the same (in fact the asynchronous APIs use the synchronous functions and slap channels on them)
No, seriously, just remove your "asynchronous APIs". You don't need them. They aren't adding anything. Just remove them. Go already does all that.
Oh, what the heck, I'll name names. This is part of why I've been so pissy at the Node community for the past few years. Their definition of synchronous is wrong, and it's really become quite popular. Synchronous does not mean "blocks the whole process". That is a weakness of Node, not a universal programming truth. From the Node perspective, all Go code is always and automatically asynchronous. You don't have to add it. It already is. You're not adding functionality by trying to "nicely" provide an asynchronous API, you're simple redundantly adding what is already there, and what you're adding is significantly less flexible than what Go already provides.
(Now with more correct definitions of the terms, it is meaningful to say that Go code is synchronous within a goroutine. But Go already provides abundant tools for dealing with that within its runtime. We can already compose these tools together trivially to do whatever async patterns we want.)
Now, I suppose, standard disclaimer, this is your library, do as you like. I'm not actually personally passionate about this like I'm yelling at my monitor or anything. I'm trying to help you save time and effort. It's up to you what you decide to do.
Oh, what the heck, I'll name names. This is part of why I've been so pissy at the Node community for the past few years. Their definition of synchronous is wrong, and it's really become quite popular. Synchronous does not mean "blocks the whole process". That is a weakness of Node, not a universal programming truth. From the Node perspective, all Go code is always and automatically asynchronous. You don't have to add it. It already is. You're not adding functionality by trying to "nicely" provide an asynchronous API, you're simple redundantly adding what is already there, and what you're adding is significantly less flexible than what Go already provides.
(Now with more correct definitions of the terms, it is meaningful to say that Go code is synchronous within a goroutine. But Go already provides abundant tools for dealing with that within its runtime. We can already compose these tools together trivially to do whatever async patterns we want.)
Now, I suppose, standard disclaimer, this is your library, do as you like. I'm not actually personally passionate about this like I'm yelling at my monitor or anything. I'm trying to help you save time and effort. It's up to you what you decide to do.
I definitely support this sentiment w/r/t the single threaded mindset of node and their async all the things approach. The only question with Go is if we guarantee sync libs, then where is the good async example code? I agree libraries should be synchronous, but does it really hurt to not have some best practices "how to do it async" in the docs? or should it be... in the code?
Kind of a meta question, but an interesting one.
Kind of a meta question, but an interesting one.
The "good asynchronous" code in Golang is straight-line code that would be called "synchronous" in Node.
Part of the problem is that even as an attempt at a "non-synchronous" interface, this seems clumsy. It demand-creates new channels for requests. That doesn't seem right.
Part of the problem is that even as an attempt at a "non-synchronous" interface, this seems clumsy. It demand-creates new channels for requests. That doesn't seem right.
Whats wrong with "demand-creating" channels? The Go stdlib does it within their network library https://golang.org/src/net/singleflight.go#L67
Because you can't select on an arbitrary collection of channels.
It might make sense for a library to return a new channel for something you do once, or once per goroutine.
It doesn't make sense to use returned channels as a basis for async dispatch.
It might make sense for a library to return a new channel for something you do once, or once per goroutine.
It doesn't make sense to use returned channels as a basis for async dispatch.
Jerf,
1. Thank you for the feedback
As you have already seen, I submitted my library to /r/golang and have gotten a lot of feedback – from which I modified the constructs (originally the functions returned channels).
I want to write something that is useful to as many people as possible (all while not alienating anyone) and therefore try not to force users (like I originally did) to use the "asynchronous APIs".
I didn't expect this to end up on HN and was going to start a discussion on golang-nuts on the pros and cons of this construct. Based on that, I was going to remove or keep the APIs.
1. Thank you for the feedback
As you have already seen, I submitted my library to /r/golang and have gotten a lot of feedback – from which I modified the constructs (originally the functions returned channels).
I want to write something that is useful to as many people as possible (all while not alienating anyone) and therefore try not to force users (like I originally did) to use the "asynchronous APIs".
I didn't expect this to end up on HN and was going to start a discussion on golang-nuts on the pros and cons of this construct. Based on that, I was going to remove or keep the APIs.
+1 for removal, YAGNI.
> all Go code is always and automatically asynchronous
wat?
wat?
I think that was an overstatement, but in the sense we're talking about with an HTTP request library, his overall point is true: Golang's runtime yields to other coroutines on I/O system calls. There usually† isn't much point in designing your own I/O scheduling system in Golang; you're almost always better off just forking off a zillion coroutines and letting them merge their results back across a single channel.
† At least in 1.1, I couldn't write a fast port-scanner without doing my own scheduling, but that's a corner case.
† At least in 1.1, I couldn't write a fast port-scanner without doing my own scheduling, but that's a corner case.
Not without involving the reflect package. So no good way.
The alternatives:
Using http://golang.org/pkg/reflect/#Select to build what you need;
Or building something specific to your use case without the above, which seems likely to be... goofy and unmanageable. Creating objects that create new channels to do `select`s over some fixed number of cases and composing those.
Do you have any experience using `reflect.Select`? While not exactly ergonomic, it seemed a... pretty okay tradeoff, inconvenience for power, and at the very least more power for approximately the same amount of pain as the rest of the `reflect` package.
Using http://golang.org/pkg/reflect/#Select to build what you need;
Or building something specific to your use case without the above, which seems likely to be... goofy and unmanageable. Creating objects that create new channels to do `select`s over some fixed number of cases and composing those.
Do you have any experience using `reflect.Select`? While not exactly ergonomic, it seemed a... pretty okay tradeoff, inconvenience for power, and at the very least more power for approximately the same amount of pain as the rest of the `reflect` package.
One problem with using "reflect" this way, apart from the fact that it's (a) counter to idiom and (b) takes your code in a direction of being counter to idiom in general, is that "reflect" is pretty slow.
You can do it, but since the only reason to do it is to avoid spawning goroutines, I'm not sure why you'd bother.
You can do it, but since the only reason to do it is to avoid spawning goroutines, I'm not sure why you'd bother.
As much as I love to see new things written in Go I am not a big fan of directly porting libraries written in different languages. It's not even that they're not idiomatic - I can live with that just fine. But the real power of Go comes from interfaces - two great examples being `io.Reader` and `io.Writer`. Once you can structure your library code so that it reasonably implements these standard interfaces you can do complex stuff in a trivial way. Say I want to get a file via HTTP, encrypt it while getting a hash of plaintext and upload it to say Google Cloud. I can pretty trivially do it by putting one `io.Writer` (Google Cloud) in another (encryption) and another (hashing, via `io.MultiWriter`) and then perform the whole thing using a simple `io.Copy`. This sort of expressive power can be achieved iff all libraries you're using adhere to the same philosophy - which is not the case with ports.
I agree with this sentiment and therefore my Response method is also an `io.ReadCloser` (exposing the http.Response.Body io.ReadCloser that Go has to offer).
After reading https://github.com/mozillazg/request, I found yours looks a little bit better.
Thanks – I wrote this library because I like using requests when writing in Python and wanted something similar when writing in Go.
Until go fixes it's issues with IPv6 (noticeably: it won't work on IPv6 out-of-the-box), I can't really respect it or consider it "production ready" for anything network related.
No, I'm not trolling. #8453 got closed, and immediately, #11081 was opened since it re-broke this.
Original issue reported by myself: https://github.com/golang/go/issues/8124
No, I'm not trolling. #8453 got closed, and immediately, #11081 was opened since it re-broke this.
Original issue reported by myself: https://github.com/golang/go/issues/8124
Go is great, its open source. I wish Hacker News open sourced their code so that we knew what they are doing. Scared of having folks game the system? Why not have the community make it the best algorithm possible. Why keep pushing YC all the time? Do they not have enough already? Painters and hackers! ha!
It's not an abstract community, much less one with some FOSS mission statement or anything, it's a YC combinator site.
Go prefers that you be more explicit and use more function declarations, even if it means repetition in the library code. When developers try to get around this, you see awkward constructs like the pointer-to-a-config-struct used here.