Google’s revamped Cloud Source Repositories in beta(cloud.google.com)
cloud.google.com
Google’s revamped Cloud Source Repositories in beta
https://cloud.google.com/blog/products/application-development/introducing-new-cloud-source-repositories
55 comments
Very nice. Reading that makes me miss Google’s internal dev environment. I may not much use anymore Google Search and gmail, but Google’s paid products like GCP (and Google Play Music, Movies, etc.) are awesome. Also, amazing permanent free tier levels. I need to use github for the repos for my books because they are already linked, but for my personal private repos I think I may switch.
What do you use instead of Google Search and Gmail?
Could you share why as a former Google dev you no longer use Google search and gmail? I'm curious why you mentioned that in response to a post about cloud source repositories. Thanks.
I'm not OP, but I'm also a former Googler. I switched to DDG and Firefox after I left because I think it's important that alternatives exist. Google may be a better actor in monopoly situations than some other companies, but they're still a company trying to make profit for shareholders.
(As a bonus, I find ddg+Firefox snappier than Google+Chrome)
(As a bonus, I find ddg+Firefox snappier than Google+Chrome)
> Google may be a better actor in monopoly situations than some other companies
That made me think, so thank you and yes I agree.
That made me think, so thank you and yes I agree.
I'm a former Googler and I mostly don't use Gmail for the mundane reason that its filtering isn't as powerful as what I do with dovecot and sieve.
I may transition from my mail server after I've deployed automation for this built atop GCP and the Gmail API.
Even then I'll still make plenty of use of mutt for better supporting OpenPGP, bottom-posting, and true threading - all of these things are still useful in Debian, and I am a Debian developer. But I'll be more easily able to mix usage of Gmail and mutt, without some hacks I currently use to achieve that.
I may transition from my mail server after I've deployed automation for this built atop GCP and the Gmail API.
Even then I'll still make plenty of use of mutt for better supporting OpenPGP, bottom-posting, and true threading - all of these things are still useful in Debian, and I am a Debian developer. But I'll be more easily able to mix usage of Gmail and mutt, without some hacks I currently use to achieve that.
I am interested in this aswell :)
> Permanent free tier levels
Well, permanent until they kill the entire service 5 years from now.
Well, permanent until they kill the entire service 5 years from now.
This looks interesting but impractical for me to mirror hundreds of repos here just to get better search.
Has anyone played with this or similar tools that can provide search by indexing at source. It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc.
I would like a good code search solution especially because I have projects across lots of different repo servers but want developers to be able to find code regardless of repo home. A colleague indexed it with solr and that was ok, but had limited semantic ability and definitely nothing like what Google describes where your recent search history and activity is displayed to each user.
Has anyone played with this or similar tools that can provide search by indexing at source. It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc.
I would like a good code search solution especially because I have projects across lots of different repo servers but want developers to be able to find code regardless of repo home. A colleague indexed it with solr and that was ok, but had limited semantic ability and definitely nothing like what Google describes where your recent search history and activity is displayed to each user.
> It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc.
GitHub excludes a few patterns in robots.txt that means a lot of the public data is never seen by crawlers that respect robots.txt, for example:
They probably want people landing mainly on
- user profiles
- repo roots
- individual issues
- individual pull requests
And I am glad they do, because when I search on Google, the above mentioned kinds of results are exactly what I am looking for, not individual source files etc.
GitHub excludes a few patterns in robots.txt that means a lot of the public data is never seen by crawlers that respect robots.txt, for example:
Disallow: /*/*/tree/*
Disallow: /*/*/blob/*
Disallow: /*/*/commits/*/*
Disallow: /*/*/commits/*?author
Disallow: /*/*/commits/*?path
Disallow: /*/*/branches
Disallow: /*/*/tags
They probably want people landing mainly on
- user profiles
- repo roots
- individual issues
- individual pull requests
And I am glad they do, because when I search on Google, the above mentioned kinds of results are exactly what I am looking for, not individual source files etc.
Author of the blog post here.
Thanks for taking a look. I'm glad you find Cloud Source Repositories interesting. We'd love to Cloud Source Repositories more practical to use. Can you explain what is impractical for you? Is the initial process of setting up multiple mirrors? Or the routine of having to go to another site for search?
(Disclaimer: I work at Google and am the PM on this product)
Thanks for taking a look. I'm glad you find Cloud Source Repositories interesting. We'd love to Cloud Source Repositories more practical to use. Can you explain what is impractical for you? Is the initial process of setting up multiple mirrors? Or the routine of having to go to another site for search?
(Disclaimer: I work at Google and am the PM on this product)
Thanks for responding. I work in an organization (happy to PM) with a few Github orgs and about 300 repos. Dozens of project teams work in them and use various GitHub features.
I have no easy insight or overview into code across these repos without scripting. The impractical part is in cloning mirroring all of these repos to get insight from your new function because user behavior is baked into all the workflow around the Github repos.
The go to another site isn’t so bad as this is a new function and not available in Github, so programmers will likely check it out when they need.
But manually forming everything and syncing. It would be neat if I can import an org in one fell swoop.
That being said, I do have scripting forking everything over on my list of things to do when I have cycles on the weekend.
I have no easy insight or overview into code across these repos without scripting. The impractical part is in cloning mirroring all of these repos to get insight from your new function because user behavior is baked into all the workflow around the Github repos.
The go to another site isn’t so bad as this is a new function and not available in Github, so programmers will likely check it out when they need.
But manually forming everything and syncing. It would be neat if I can import an org in one fell swoop.
That being said, I do have scripting forking everything over on my list of things to do when I have cycles on the weekend.
For those that haven't used Google's code search before, I believe this will be very similar to Chromium's code search[0] (which is based on Google's internal one).
[0] https://cs.chromium.org/
[0] https://cs.chromium.org/
Similar to OpenGrok, I'm assuming?
Why would I want to host code on Google's Cloud when they shut down Google Code? What's to stop them from shutting down Google Cloud in a few years?
I believe the difference is this is part of GCP and their terms for GA products are that they will support for at least one year on deprecated versions / products and not just instantly remove something.
Google code is back? They really do indicate they can host your repo in the post!
GCP has been able to host your repo since before the turndown of Google Code. In fact, the original GCP Source Repo team and the Google Code team were the same team (though the current product has a different backend and team).
Yep, I used to work on the Google code specific features but there was a lot of collaboration between the devs.
One nice thing is how trivial it is to mirror a repository from Bitbucket or GitHub. I wish it was as easy to mirror a repository from Bitbucket to GitHub or GitHub to Bitbucket.
It would be nice if it was so easy to mirror bitbucket server, or gitlab, etc.
> For Java, JavaScript, Go, C++, Python, TypeScript and Proto files, you’ll see result suggestions indicating whether the match is an entity such as a class, method, enum, or field.
Apparently they forgot about Flutter users.
Apparently they forgot about Flutter users.
[deleted]
Why would someone move from bitbucket?
Author of the blog post here.
We've generally heard that developers move from Bitbucket to Cloud Source Repositories because they want to use other Google Cloud services and it's simpler for them to manage their source in the same place where they debug, build and deploy that code. Developers often mention that they appreciate the unified identity, IAM permissions, and integrations that Cloud Source Repositories has with Cloud Shell, Cloud Build, Cloud Debugger and others Google Cloud services.
However, developers don't need to move from Bitbucket to take advantage of the utility provided by Cloud Source Repositories. You can mirror any number of your repositories into Cloud Source Repositories and take advantage of the code browser, code search, and various integrations with Google Cloud Platform.
(Disclaimer: I work at Google and am the PM on this product)
We've generally heard that developers move from Bitbucket to Cloud Source Repositories because they want to use other Google Cloud services and it's simpler for them to manage their source in the same place where they debug, build and deploy that code. Developers often mention that they appreciate the unified identity, IAM permissions, and integrations that Cloud Source Repositories has with Cloud Shell, Cloud Build, Cloud Debugger and others Google Cloud services.
However, developers don't need to move from Bitbucket to take advantage of the utility provided by Cloud Source Repositories. You can mirror any number of your repositories into Cloud Source Repositories and take advantage of the code browser, code search, and various integrations with Google Cloud Platform.
(Disclaimer: I work at Google and am the PM on this product)
FWIW, it'd also be great to have mirroring with Gitlab (especially since I think Google Cloud has a deal for users with them?).
They should list expected months until closure on the product page so people can plan accordingly. I kid, but not really.
how many gcp products has google abandoned in total?
I know we deprecated the prediction API with a year's warning while bringing out Cloud ML Engine. And Cloud SQL changed a bit when the second generation came in and supported MySQL, PostgreSQL and more.
Can't think of others off the top of my head, so that is two if we want to consider major upgrades "abandoned." Perception often doesn't match objective fact.
(Disclosure: I work at GCP)
Can't think of others off the top of my head, so that is two if we want to consider major upgrades "abandoned." Perception often doesn't match objective fact.
(Disclosure: I work at GCP)
Nice. Is the code search the same as what Google has internally? I do miss that.
(Though honestly, the value from the internal code search was that everything was in there. So if you got an error message from a service you were a client of, you could just search for the error message in their code and see what it actually meant. "Aha, this 'optional' field that's marked 'deprecated' isn't actually optional or deprecated," most of the time ;)
(Though honestly, the value from the internal code search was that everything was in there. So if you got an error message from a service you were a client of, you could just search for the error message in their code and see what it actually meant. "Aha, this 'optional' field that's marked 'deprecated' isn't actually optional or deprecated," most of the time ;)
Author of the blog post here.
The code search functionality is the same as what Google engineers use internally. Cloud Source Repositories uses the same indexing and retrieval technologies on the same type of infrastructure.
As you mention, the more code you have, the more benefit you get from having fast search tools across your entire code base that can perform complex semantic and regular expression queries. Even with smaller codebases, I find it the fastest way to find the code I need.
(Disclaimer: I work at Google and am the PM on this product)
The code search functionality is the same as what Google engineers use internally. Cloud Source Repositories uses the same indexing and retrieval technologies on the same type of infrastructure.
As you mention, the more code you have, the more benefit you get from having fast search tools across your entire code base that can perform complex semantic and regular expression queries. Even with smaller codebases, I find it the fastest way to find the code I need.
(Disclaimer: I work at Google and am the PM on this product)
You might also want to check out https://kythe.io and https://github.com/google/kythe.
It’s great that they opened Kythe but as someone who deployed it from scratch I say it’s like ordering a DIY aircraft carrier kit. Many steps before you get something useful.
Having glanced at it a bit I don’t see how this can be based on internal code search / kythe. Kythe works by building your codes. That’s how it can find callers and specializations and whatnot. This doesn’t seem to.
"It looks for matches by file names, classes, functions and other symbols, and matches the context and namespace of the symbols"
Correct. Looks like it just parses and indexes the source.
Correct. Looks like it just parses and indexes the source.
Quickly glanced through the blog post and the docs but didn’t find any mention of Git LFS. Does Cloud Source Repositories support mirroring LFS content?
Author of the blog post here.
Cloud Source Repositories currently supports very large repositories. The same backend scales to the needs of the Android open source project, which regularly checks in massive binary files. There are many APKs and VM images in the Git repositories we host. However, Cloud Source Repositories does not support LFS or the mirroring of LFS content.
LFS is not deeply integrated into Git, creating usability problems. Because the content is not part of the object graph of the repository, you have to decide in advance to use Git LFS. You don't get the benefit in existing repositories with large files. You also can't back out --- once you're using Git LFS on a medium-sized file, you can't change your mind and instruct Git to send it inline in fetches, without rewriting history, which breaks existing clients that have cloned the repository.
For the same reason, history mining commands like "git blame" and "git log -S" don't have access to the object.
In addition, it complicates migrating to another host. Usually in Git, you can take out your content by running "git clone --mirror one-url and then "git -C directory.git push --mirror another-url". With Git LFS, this copies over the pointer files but not the underlying large file content and you must remember to take extra steps to instruct the new host about where the blobs are stored.
Google's Cloud Source Repositories team believes that Git itself needs to deal better with large files. The first step of this work within the Git project has been partial clone, which is supported in Cloud Source Repositories and in the public Git 2.17 release (for best results, please use Git 2.19, released September, 2018). If you run "git clone --filter=blob:limit=512M <url>", files larger than 512M will be omitted from the initial clone and fetched automatically on demand when needed (for example during checkout operations). See https://crbug.com/git/2 for more details about this feature. We are continuing to work with the community on adding other features related to large file support into Git.
(Disclaimer: I work at Google and am the PM on this product)
Cloud Source Repositories currently supports very large repositories. The same backend scales to the needs of the Android open source project, which regularly checks in massive binary files. There are many APKs and VM images in the Git repositories we host. However, Cloud Source Repositories does not support LFS or the mirroring of LFS content.
LFS is not deeply integrated into Git, creating usability problems. Because the content is not part of the object graph of the repository, you have to decide in advance to use Git LFS. You don't get the benefit in existing repositories with large files. You also can't back out --- once you're using Git LFS on a medium-sized file, you can't change your mind and instruct Git to send it inline in fetches, without rewriting history, which breaks existing clients that have cloned the repository.
For the same reason, history mining commands like "git blame" and "git log -S" don't have access to the object.
In addition, it complicates migrating to another host. Usually in Git, you can take out your content by running "git clone --mirror one-url and then "git -C directory.git push --mirror another-url". With Git LFS, this copies over the pointer files but not the underlying large file content and you must remember to take extra steps to instruct the new host about where the blobs are stored.
Google's Cloud Source Repositories team believes that Git itself needs to deal better with large files. The first step of this work within the Git project has been partial clone, which is supported in Cloud Source Repositories and in the public Git 2.17 release (for best results, please use Git 2.19, released September, 2018). If you run "git clone --filter=blob:limit=512M <url>", files larger than 512M will be omitted from the initial clone and fetched automatically on demand when needed (for example during checkout operations). See https://crbug.com/git/2 for more details about this feature. We are continuing to work with the community on adding other features related to large file support into Git.
(Disclaimer: I work at Google and am the PM on this product)
Sounds great but at this point I've been burned by Google EOLing products that I use too many times.
I was going to make this same joke. It's hard to put faith into a product when Google keeps killing them off. Wave and Google Code are two top of mind products that I missed when they were gone.
Conversely, I think it's important for Google engineers to be able to try new things and publish products without having to worry about supporting them forever. Is there some sort of happy medium? Like a Google labs of products that are in various states of whimsical testing?
Cloud source looks nice, and for GCP projects I'm sure it will have a nice tie in, but for now I'm 100% more likely to push code to Gitlab until it makes sense to do otherwise.
Conversely, I think it's important for Google engineers to be able to try new things and publish products without having to worry about supporting them forever. Is there some sort of happy medium? Like a Google labs of products that are in various states of whimsical testing?
Cloud source looks nice, and for GCP projects I'm sure it will have a nice tie in, but for now I'm 100% more likely to push code to Gitlab until it makes sense to do otherwise.
I know that is the meme, but are there any examples of Google dropping enterprise (i.e. paid) offerings? I can think of a few free services that they have dropped, but that's a different story than dropping a commercial service that you're selling to enterprise users.
I don't know. A quick perusal of the graveyard[1] shows what I would consider mostly free services.
[1] https://en.wikipedia.org/wiki/List_of_Google_products#Discon...
[1] https://en.wikipedia.org/wiki/List_of_Google_products#Discon...
Maps wasn't dropped, but the pricing shifts are similar in impact.
https://news.ycombinator.com/item?id=16980166
https://news.ycombinator.com/item?id=16980166
> Like a Google labs of products that are in various states of whimsical testing?
They actually had precisely that, but then they cancelled it.
> Google described Google Labs as "a playground where our more adventurous users can play around with prototypes of some of our wild and crazy ideas and offer feedback directly to the engineers who developed them."
https://en.wikipedia.org/wiki/Google_Labs
They actually had precisely that, but then they cancelled it.
> Google described Google Labs as "a playground where our more adventurous users can play around with prototypes of some of our wild and crazy ideas and offer feedback directly to the engineers who developed them."
https://en.wikipedia.org/wiki/Google_Labs
Were people really that affected by code? They EOL'ed with quite a lenient time schedule, and provided migration tools.
This is pretty cool. What can we expect in terms of additional language support?
Author of the blog post here.
I'm glad you like it. Which languages would you like to see support added for?
Note that search works today across all languages but the semantic understanding of source which enhances search is limited to Java, JavaScript, Go, C++, Python, TypeScript and Proto files.
(Disclaimer: I work at Google and am the PM on this product)
I'm glad you like it. Which languages would you like to see support added for?
Note that search works today across all languages but the semantic understanding of source which enhances search is limited to Java, JavaScript, Go, C++, Python, TypeScript and Proto files.
(Disclaimer: I work at Google and am the PM on this product)
Thanks for the post Russell. Our SaaS application is running on Rails / Typpescript frontend. We're patiently waiting for App Engine to add Standard support for Ruby before we go all in on GCP. I would be most excited to see Cloud Source support Ruby. Hope your team will consider it.
In priority order, I would suggest:
1. Swift
2. Obj-C
3. Bash
4. SQL
5. Ruby
6. C#
Clojure would be a nice feature
Being able to search code so efficiently across the whole company’s code base is a huge productivity gain. But I found people who hasn’t worked in Google or other companies that have this capability generally don’t believe it’s a big deal.
Well, you need to have a fairly large codebase with a fair amount of in-house infrastructure and interdependencies before you have both the need to find code that you're not working on and have a dense enough source tree where search makes more sense than just learning the organization and navigating to where you need to go.
Honestly, it's not even the monorepo and tooling. Simply the ability to navigate our code IDE-style in the browser, and move back and forth between different revisions, makes a huge difference.
I've been mucking around with some code in javac recently and was trying to show someone something I'd found, and browsing the openjdk mercurial repo was just so painful compared to hunting around google3.
I've been mucking around with some code in javac recently and was trying to show someone something I'd found, and browsing the openjdk mercurial repo was just so painful compared to hunting around google3.