GitHub code search is generally available(github.blog)
github.blog
GitHub code search is generally available
https://github.blog/2023-05-08-github-code-search-is-generally-available/
155 comments
> more people aren't talking about how Microsoft has a near monopoly on the developer ecosystem.
But do they? In my day job, outside of the occasional use of Visual Studio and developing on a Windows machine, I use no Microsoft products for development.
> credit to Microsoft for rehabbing their reputation with developers
With a fair number of younger developers, but certainly not all. Most devs I know don't think of Microsoft any more kindly now than in the past.
But do they? In my day job, outside of the occasional use of Visual Studio and developing on a Windows machine, I use no Microsoft products for development.
> credit to Microsoft for rehabbing their reputation with developers
With a fair number of younger developers, but certainly not all. Most devs I know don't think of Microsoft any more kindly now than in the past.
Your particular individual experience is not representative of the overall reach that Microsoft has, which is what OP is pointing out.
GitHub is the defacto "point" of software these days, with most devs jumping to that before anything else.
VSCode is the highest ranked editor from the StackOverflow 2023 survey, with (IIRC) something akin to 70%.
Azure is the icing on the cake, because now you have an entire generation of developers building on GitHub, from VSCode, and deploying onto Microsoft infrastructure.
GitHub is the defacto "point" of software these days, with most devs jumping to that before anything else.
VSCode is the highest ranked editor from the StackOverflow 2023 survey, with (IIRC) something akin to 70%.
Azure is the icing on the cake, because now you have an entire generation of developers building on GitHub, from VSCode, and deploying onto Microsoft infrastructure.
2023 isn't live yet, but 74% of respondents used VSCode in 2022[0].
That said, the numbers are hard to compare because they allow selecting multiple choices and they split the JetBrains products up, while VSCode is considered as one tool.
If you add together all the JetBrains products they reach 94%. I expect there's a ton of overlap that brings that number back down, but it's enough to make me suspect that VSCode's lead isn't as high as the numbers in the survey make it look.
[0] https://survey.stackoverflow.co/2022/#technology-most-popula...
That said, the numbers are hard to compare because they allow selecting multiple choices and they split the JetBrains products up, while VSCode is considered as one tool.
If you add together all the JetBrains products they reach 94%. I expect there's a ton of overlap that brings that number back down, but it's enough to make me suspect that VSCode's lead isn't as high as the numbers in the survey make it look.
[0] https://survey.stackoverflow.co/2022/#technology-most-popula...
you can do one better: imagine those developers writing typescript frontends for asp.net backends!
One which is using npm for package management and the other using nuget. And guess who owns both?
I typo'd 2023, so thanks for that correction!
> But do they? In my day job, outside of the occasional use of Visual Studio and developing on a Windows machine, I use no Microsoft products for development.
According to your comment history, you are using JavaScript or used it in the past at least, which usually means you've used npm, which is owned by Microsoft.
And since lots of "young" developers use JavaScript and TypeScript, most of them are also interacting with npm.
Microsoft has captured maybe a larger part of the developer market than you realize. Not by being amazing or with "Microsoft <3 FOSS", but by buying up lots of the market.
According to your comment history, you are using JavaScript or used it in the past at least, which usually means you've used npm, which is owned by Microsoft.
And since lots of "young" developers use JavaScript and TypeScript, most of them are also interacting with npm.
Microsoft has captured maybe a larger part of the developer market than you realize. Not by being amazing or with "Microsoft <3 FOSS", but by buying up lots of the market.
I don't think they have one today, but it is looking like they could soon have one. Especially since I believe recent reports show that Azure is starting out pace AWS in adoption? So you have .NET, Visual Studio (Code), Azure, Github, OpenAI, Windows, and I am pretty sure more I am forgetting about. I think the big one that wasn't initially mentioned was Azure.
All of that adds up to having a very significant, perhaps majority, of the market locked up. But it's also all centering around a particular sort of product and product development. Microsoft might be able to lock up that segment, but I don't think they're in a position to monopolize the larger software development space in the near or medium future.
Don't forget TypeScript and npm as well which basically covers 99% of the JavaScript ecosystem if not more.
Then Dependabot for large swatches of more developers outside of the earlier mentioned ecosystems. LinkedIn for everyone's career.
Then Dependabot for large swatches of more developers outside of the earlier mentioned ecosystems. LinkedIn for everyone's career.
"monopoly on developer ecosystem"
GitHub? fair
OpenAI? how is this a part of dev. ecosystem?
Vs Code? wtf? there's a lot of other IDEs/editors and many would argue that they are better
>embrace, extend, extinguish strategy with WSL
They are EEEing their product - Windows?
GitHub? fair
OpenAI? how is this a part of dev. ecosystem?
Vs Code? wtf? there's a lot of other IDEs/editors and many would argue that they are better
>embrace, extend, extinguish strategy with WSL
They are EEEing their product - Windows?
> Vs Code? wtf? there's a lot of other IDEs/editors and many would argue that they are better
VS Code is hugely dominant in terms of IDEs used by developers per the StackOverflow survey. It commands more than double the percentage of the next most used… and the second place is Visual Studio.
https://survey.stackoverflow.co/2022/#most-popular-technolog...
VS Code is hugely dominant in terms of IDEs used by developers per the StackOverflow survey. It commands more than double the percentage of the next most used… and the second place is Visual Studio.
https://survey.stackoverflow.co/2022/#most-popular-technolog...
Those numbers are a bit misleading because they split up the JetBrains products, making JetBrains look like a smaller market than it actually is. If you add up all their tools, they hit 94% to VS Code's 74%. Obviously there are a lot of people using multiple JetBrains tools and that bloats that number, but the huge difference demonstrates how flawed SO's numbers are for gauging market share.
Also, I suspect that most people who mark a JetBrains IDE also mark a secondary editor that's lighter weight, but it's hard to tell how that all breaks down with the way they present the data. I spend about 90% of my work day in WebStorm and occasionally move over to VSCode for loading a large directory that I don't want to index. When I filled out the survey, due to they way they worded it, I marked both tools.
Also, I suspect that most people who mark a JetBrains IDE also mark a secondary editor that's lighter weight, but it's hard to tell how that all breaks down with the way they present the data. I spend about 90% of my work day in WebStorm and occasionally move over to VSCode for loading a large directory that I don't want to index. When I filled out the survey, due to they way they worded it, I marked both tools.
Add Comment
lolinder 11 days ago | parent | context | on: GitHub code search is generally available
Those numbers are a bit misleading because they split up the JetBrains products, making JetBrains look like a smaller market than it actually is. If you add up all their tools, they hit 94% to VS Code's 74%. Obviously there are a lot of people using multiple JetBrains tools and that bloats that number, but the huge difference demonstrates how flawed SO's numbers are for gauging market share. Also, I suspect that most people who mark a JetBrains IDE also mark a secondary editor that's lighter weight, but it's hard to tell how that all breaks down with the way they present the data. I spend about 90% of my work day in WebStorm and occasionally move over to VSCode for loading a large directory that I don't want to index. When I filled out the survey, due to they way they worded it, I marked both tools.
If you haven't already, would you mind reading about HN's approach to comments and site guidelines?
Add Comment lolinder 11 days ago | parent | context | on: GitHub code search is generally available
Those numbers are a bit misleading because they split up the JetBrains products, making JetBrains look like a smaller market than it actually is. If you add up all their tools, they hit 94% to VS Code's 74%. Obviously there are a lot of people using multiple JetBrains tools and that bloats that number, but the huge difference demonstrates how flawed SO's numbers are for gauging market share. Also, I suspect that most people who mark a JetBrains IDE also mark a secondary editor that's lighter weight, but it's hard to tell how that all breaks down with the way they present the data. I spend about 90% of my work day in WebStorm and occasionally move over to VSCode for loading a large directory that I don't want to index. When I filled out the survey, due to they way they worded it, I marked both tools.
If you haven't already, would you mind reading about HN's approach to comments and site guidelines?
Those numbers are a bit misleading because they split up the JetBrains products, making JetBrains look like a smaller market than it actually is. If you add up all their tools, they hit 94% to VS Code's 74%. Obviously there are a lot of people using multiple JetBrains tools and that bloats that number, but the huge difference demonstrates how flawed SO's numbers are for gauging market share. Also, I suspect that most people who mark a JetBrains IDE also mark a secondary editor that's lighter weight, but it's hard to tell how that all breaks down with the way they present the data. I spend about 90% of my work day in WebStorm and occasionally move over to VSCode for loading a large directory that I don't want to index. When I filled out the survey, due to they way they worded it, I marked both tools.
If you haven't already, would you mind reading about HN's approach to comments and site guidelines?
Add Comment lolinder 11 days ago | parent | context | on: GitHub code search is generally available
Those numbers are a bit misleading because they split up the JetBrains products, making JetBrains look like a smaller market than it actually is. If you add up all their tools, they hit 94% to VS Code's 74%. Obviously there are a lot of people using multiple JetBrains tools and that bloats that number, but the huge difference demonstrates how flawed SO's numbers are for gauging market share. Also, I suspect that most people who mark a JetBrains IDE also mark a secondary editor that's lighter weight, but it's hard to tell how that all breaks down with the way they present the data. I spend about 90% of my work day in WebStorm and occasionally move over to VSCode for loading a large directory that I don't want to index. When I filled out the survey, due to they way they worded it, I marked both tools.
If you haven't already, would you mind reading about HN's approach to comments and site guidelines?
VSCode is just the lowest common denominator IDE that works decently with most languages and platforms, I use it a lot but would not think twice about switching to something else (as I have many times in my career). I don’t think it really has any “moat”
> > embrace, extend, extinguish strategy with WSL
> They are EEEing their product - Windows?
No, Linux obviously.
First they like and integrate Linux into their own products. Azure, WSL and others.
Then, they provide extensions that are closed-source on top of those.
With the goal to extinguish the original project so they have more control over the direction.
> They are EEEing their product - Windows?
No, Linux obviously.
First they like and integrate Linux into their own products. Azure, WSL and others.
Then, they provide extensions that are closed-source on top of those.
With the goal to extinguish the original project so they have more control over the direction.
WSL has been out for at least five years now and it’s still just a fancy kernel adapter without feature parity. I might begin to worry once things like containers actually work properly. Until then, I don’t think Microsoft has fully embraced Linux much less extended it.
I did just now learn that systemd was finally added to WSL. Originally that was never going to be added and back in the days of WSL 1 I remember the WSL writing blog posts about that being ridiculous. Who knows, proper container support might be added soon.
I did just now learn that systemd was finally added to WSL. Originally that was never going to be added and back in the days of WSL 1 I remember the WSL writing blog posts about that being ridiculous. Who knows, proper container support might be added soon.
Containers do work in WSL2, although you hit Hyper-V bugs if you spin up very many of them. WSL2 is the basis of Docker Desktop and similar apps.
Microsoft has also already added proprietary extensions to WSL2, like the integration with the VSCode remote development plugin and DirectX passthrough for machine learning and maybe other things (there's a DirectX driver on Linux which is usable only on WSL guests).
Microsoft has also already added proprietary extensions to WSL2, like the integration with the VSCode remote development plugin and DirectX passthrough for machine learning and maybe other things (there's a DirectX driver on Linux which is usable only on WSL guests).
We're at the embrace stage right now, that's why you see it getting integrated.
And if you run WSL, you usually end up with a special kernel, it's not just the upstream one without changes.
Both my Arch and Ubuntu WSL is on kernel `Linux desktop 5.15.90.1-microsoft-standard-WSL2` rather the ones the distributions ship with by default if installed normally. This is the one they ship via Windows Update which you end up using on WSL: https://github.com/microsoft/WSL2-Linux-Kernel
Watch them slowly make it different than the upstream one, without contributing patches upstream.
And if you run WSL, you usually end up with a special kernel, it's not just the upstream one without changes.
Both my Arch and Ubuntu WSL is on kernel `Linux desktop 5.15.90.1-microsoft-standard-WSL2` rather the ones the distributions ship with by default if installed normally. This is the one they ship via Windows Update which you end up using on WSL: https://github.com/microsoft/WSL2-Linux-Kernel
Watch them slowly make it different than the upstream one, without contributing patches upstream.
If it was that easy to kill Linux it would have happened already. Chad Linux, open source free software, withstanding the full might of the multi billion dollar juggernaut Microsoft.
They're still doing their best to kill Linux on the desktop.
Notice how, as much as .net core is cross-platform, the desktop UI side of MAUI is absolutely not, and left 'to the community'.
They do not want to enable desktop developers under Linux, as they know that a decent environment would likely remove any need for Windows in that space.
Notice how, as much as .net core is cross-platform, the desktop UI side of MAUI is absolutely not, and left 'to the community'.
They do not want to enable desktop developers under Linux, as they know that a decent environment would likely remove any need for Windows in that space.
I mean google uses VSCode internally as their officially supported IDE… i’d say they’re doing pretty well.
No it's based on theia
Because of products created before VSCode
What do you mean?
They are doing well because of products and market share they held before using vscode. So it's unrelated success.
I meant, VSCode is pretty successful if it can get Google to switch over to using it.
except windows itself is not loved by most developers, microsoft will take over the developer world when it replaces its windows with linux fully(instead of WSL2, which is nice but not great)
Can't see Linux replacing windows, their most profitable products (Windows and MS Office) are both based on closed ecosystem
Isn't Azure a much higher percentage of profit than Windows or Office for MS at this point?
They very carefully don't actually tell us in their quarterly report. Microsoft is not new to the stock market, wall street, or making things seem different than they are, so O365 revenue isn't actually broken out from Azure revenue, so we just have to guess, based on our biases, which component is doing better vs GCP, and buy stock based on that analysis.
I don't know the numbers by heart. But if it isn't a higher percentage of profit right now.It certainly takes the cake for largest growth percentage with it eclipsing everything else soon (if it hasn't already).
I don’t know. I’m happily using ms office on a mac and in the browser.
Part of the reason Windows isn't loved by developers is also hardware. So a switch to Linux won't fix this, unless they made the switch when Apple was releasing those horrific keyboards!
No one is talking about it because developer tools is literally the core business Microsoft is in and was founded as such, 48-years ago.
Yes, they have expanded in many ways - but developer tools/languages/IDEs/libraries is their bread & butter.
The famous “Developers Developers Developers” by Steve Ballmer.
https://youtu.be/Vhh_GeBPOhs
Yes, they have expanded in many ways - but developer tools/languages/IDEs/libraries is their bread & butter.
The famous “Developers Developers Developers” by Steve Ballmer.
https://youtu.be/Vhh_GeBPOhs
Now it's helping developers to feed the AI for free.
I think there was a lot of discussion on this when Microsoft took over GitHub but as time goes people kind of accepted the reality.
i think it extends further than that, since they have: vscode, github, linkedin, npm, typescript, chatgpt. for many, this is almost the entire developer ecosystem.
at a high level they pretend to embrace open source but many of the best features of vscode are closed source, such as remote editing and various language servers (pylance, etc.) the lsp saga is particularly unfriendly, since they pushed it as an open standard, tons of people contributed and adopted it, and then they closed the source to their most valuable language servers, making them only compatible with their product (vscode).
there are countless similar examples. the way i see microsoft and the way they want to be perceived are entirely different.
at a high level they pretend to embrace open source but many of the best features of vscode are closed source, such as remote editing and various language servers (pylance, etc.) the lsp saga is particularly unfriendly, since they pushed it as an open standard, tons of people contributed and adopted it, and then they closed the source to their most valuable language servers, making them only compatible with their product (vscode).
there are countless similar examples. the way i see microsoft and the way they want to be perceived are entirely different.
I guess Steve Ballmer was right all along...
About developers?
developers, developers, developers, developers
https://www.youtube.com/watch?v=XxbJw8PrIkc
https://www.youtube.com/watch?v=XxbJw8PrIkc
I would expect them to purchase StackOverflow too. I guess it's not that essential, seems a low cost acquisition.
Why would they purchase StackOverflow ? They are partnered with the StackOverflow killer: ChatGPT.
If StackOverflow dies, surely the developer-relevant quality of training data will suffer?
After all, the capabilities of ChatGPT are basically proportional to how well a topic is represented in the training data, which is largely the internet.
After all, the capabilities of ChatGPT are basically proportional to how well a topic is represented in the training data, which is largely the internet.
If you kill your concurrent, you don't need to make your product better.
But it will keep learning from it's users.
Most of the use I get out of GPT is letting it summarize stackoverflow posts for me.
It can often respond your Google query you used to find the SO page.
Don't forget Teams, which seems to be killing every other Chat/Video platform in the enterprise sector.
Github -> gitlab, vs code -> jetbrains. why use wsl just go to linux. No one forces you to use their products they’re just a better developer experience imo aside from windows. Plenty of competition in the space. The question is does better equal monopoly?
How is WSL an example of EEE? Other than adding a few things like the ability to navigate the Windows filesystem from WSL, they have really done nothing in the way of extending or extinguishing.
If anything is classic EEE of what you listed it’s GitHub, given it’s basically a series of extensions on an open standard. Its Linux EEE program is Azure where Microsoft sells you a Linux server running in its datacenter in fancy ways that lock you in.
I feel Microsoft’s monopoly around development is not that strong and it’s not terribly hard to completely avoid them.
If anything is classic EEE of what you listed it’s GitHub, given it’s basically a series of extensions on an open standard. Its Linux EEE program is Azure where Microsoft sells you a Linux server running in its datacenter in fancy ways that lock you in.
I feel Microsoft’s monopoly around development is not that strong and it’s not terribly hard to completely avoid them.
I had the same realisation. Also remember they’ve extended JavaScript with Typescript, and built LSP. It does indeed feel like they are eating the whole of software.
Aren't those FOSS?
Meanwhile CMA: you are leading cloud gaming and hence can't acquire Activision.
Count me in the list of dev teams where that is not remotely true. I use exactly 0 microsoft projects at my software engineering job.
gitea
... is nice but needs the social network effects. Maybe adding some federation and stars / comments as a protocol (not just a program) could help. Maybe it exists and lacks coherence/ publicity.
One of the best things that could be done for open source is to break the monopoly on those damn stars.
Open source would be a lot healthier if social proof were portable across platforms.
Open source would be a lot healthier if social proof were portable across platforms.
...who has ever cared about stars?
„Our product has 5000 stars on GitHub, therefore give us money, it’s a business opportunity”. Seen plenty of those on LinkedIn.
"Stars" do provide a nice signal about whether something sees wide use or not. I typically check out most dependencies I add to my projects, but I will give a lot more attention to a project with 9 stars than a dependency with 9,000 stars.
And it also gives a good indication what common de-facto standards are within an ecosystem.
And it also gives a good indication what common de-facto standards are within an ecosystem.
good for small projects, I guess the real competitor is gitlab
Now they just need StackOverflow.
They already run a .NET stack....
They already run a .NET stack....
I have been using since the beta, truly the most impressive product released in the last 5 years (along with chatgpt).
The amount of indexed code, the quickness and the precision of this search is simply stunning.
Absolutely. GitHub Code Search is by far the most valuable online development tool I have used the past year. It is so much more useful than Copilot or any of the AI LLMs in my experience.
With Code Search, I have:
* Rewritten a CMake build system, which would have been practically impossible without access to real-world examples because of how poorly designed and documented it is;
* Validated machine-generated translations by looking up language strings from projects that used human translators;
* Tracked down bugs in unfamiliar codebases, using symbol-based navigation, without the downtime of fetching a bunch of files and waiting for a language server to process them locally;
* Reviewed how projects were using the APIs of a library I work on to determine whether high-maintenance features were actually used, and whether tricky features were being used correctly or needed redesigning to reduce programmer error
Kudos to the team at GitHub. Genuinely stellar work.
With Code Search, I have:
* Rewritten a CMake build system, which would have been practically impossible without access to real-world examples because of how poorly designed and documented it is;
* Validated machine-generated translations by looking up language strings from projects that used human translators;
* Tracked down bugs in unfamiliar codebases, using symbol-based navigation, without the downtime of fetching a bunch of files and waiting for a language server to process them locally;
* Reviewed how projects were using the APIs of a library I work on to determine whether high-maintenance features were actually used, and whether tricky features were being used correctly or needed redesigning to reduce programmer error
Kudos to the team at GitHub. Genuinely stellar work.
That's really interesting, do you have any examples handy of search queries you used?
For CMake stuff I would search like `<function name> path:/CMakeLists\.txt$|cmake/`.
For translations I would search a phrase using `<phrase> lang:"Gettext Catalog" path:/<language iso code>/`.
For reviewing library consumers I would just search for `<my library name> (<function name> OR <other function name> OR <third function name>) NOT path:<my library name>/**`.
There may be better queries for some of these, but I really didn’t have to do any magic or weird tricks. Everything more or less just works.
For translations I would search a phrase using `<phrase> lang:"Gettext Catalog" path:/<language iso code>/`.
For reviewing library consumers I would just search for `<my library name> (<function name> OR <other function name> OR <third function name>) NOT path:<my library name>/**`.
There may be better queries for some of these, but I really didn’t have to do any magic or weird tricks. Everything more or less just works.
I'm not the one you asked to, but here my usages:
Searching how an API is used.
Searching how people configured something.
Searching accidental uses of an attribute:
https://github.com/dotnet/csharplang/discussions/5657#discus...
Searching how an API is used.
Searching how people configured something.
Searching accidental uses of an attribute:
https://github.com/dotnet/csharplang/discussions/5657#discus...
It's amazing for searching examples of rarely used / non-public APIs
Is the difference that now the basic search functionality actually works?
The previous standard GitHub search I found to be remarkably bad. I would be looking at some small public repo, search for an exact string match I know exists in the code, scope the search to that repo only, and still see zero results. Even copy-pasting a line of code from a file in the repo often resulted in zero matches.
The previous standard GitHub search I found to be remarkably bad. I would be looking at some small public repo, search for an exact string match I know exists in the code, scope the search to that repo only, and still see zero results. Even copy-pasting a line of code from a file in the repo often resulted in zero matches.
I don't get it too. Maybe one needs to enable something. The search is still useless. Locally I use git-grep.
You might want to consider replacing git-grep with ripgrep.
It seems you still need to enable the feature. Click your user icon in the top right -> choose "Feature preview" in the dropdown -> enable the "New Code Search and Code View" feature.
I'm pretty sure sourcegraph.com has been doing all that (search all GH repos, exact search in quotes, case-sensitive search, regex search, limit files using filename regex) for longer than 5 years.
Could you elaborate more on why this is a significant feature?
I can see how it would be handy to search a codebase online, especially one you don't have cloned locally, but for my own codebases, I can search the entire thing just fine in VS Code with ctrl-shft-f.
I can see how it would be handy to search a codebase online, especially one you don't have cloned locally, but for my own codebases, I can search the entire thing just fine in VS Code with ctrl-shft-f.
It's not only about searching your own repository, it allows you to search through every single public repository on GitHub. I personally use it a lot to learn more obscure APIs which are badly documented or which I'm just not used to, simply search for the method I'm trying to use and find infinite examples of real world usage, along with the code license right next to it.
It's also great if you drank the GitHub kool-aid as you can do a single search and find related code snippets, issues, pull requests and discussions that could possibly help. I'm personally not to big into the ecosystem, in fact I'm considering moving to Fossil so I can have everything inside the repo, but for those who are it's a great feature.
It's also great if you drank the GitHub kool-aid as you can do a single search and find related code snippets, issues, pull requests and discussions that could possibly help. I'm personally not to big into the ecosystem, in fact I'm considering moving to Fossil so I can have everything inside the repo, but for those who are it's a great feature.
Ever wanted to use an API and found the documentation to be lacking?
GitHub code search pretty much solves that. For any API you can find an example of someone else using it.
I've been using it for this for a year now and I wouldn't want to live without it.
GitHub code search pretty much solves that. For any API you can find an example of someone else using it.
I've been using it for this for a year now and I wouldn't want to live without it.
> For any API you can find an example of someone else using it.
Amazing. This was the light bulb moment for me. I work at [big tech co] and we have an internal code search tool, and anytime I need to use a new API I pull it up to find examples of how it's used.
Now I can do this for the entire world of OSS, amazing.
Amazing. This was the light bulb moment for me. I work at [big tech co] and we have an internal code search tool, and anytime I need to use a new API I pull it up to find examples of how it's used.
Now I can do this for the entire world of OSS, amazing.
I think others have pretty much summarised it already:
Searching for non-documented or badly documented API, find implementation of an algorithm or specific pattern, find how people are using a niche tool, etc.
I have even used it to find my own API in the wild to look for potential breaking changes and improvements to do.
Searching for non-documented or badly documented API, find implementation of an algorithm or specific pattern, find how people are using a niche tool, etc.
I have even used it to find my own API in the wild to look for potential breaking changes and improvements to do.
For me it's been very useful at finding codebases with work done on extremely specific niche things that would've been near impossible to find otherwise (e.g. tools for obscure protocols hidden away on obtusely named repos)
Yes, it comes pretty close. Well done!
I'm Colin from GitHub's code search team, happy to answer any questions.
For more info on how we built this, you can check out our technical blog post from a few months ago https://github.blog/2023-02-06-the-technology-behind-githubs...
For more info on how we built this, you can check out our technical blog post from a few months ago https://github.blog/2023-02-06-the-technology-behind-githubs...
How does it compare to Sourcegraph? What is the main differentiator?
It would be really awesome if code search could one day consume LSIF for precise results in its index similar to source graph. The symbol search is good now, but approximate. Having more precise code search by allowing devs to upload LSIF data in their CI pipelines would allow for precise symbol search (go to definition / find usages actually being accurate) and remove irrelevant result.
Great point. Yes, we initially focused on zero-config approximate code navigation. But we do intend to support build-based code navigation in the future, since the approximate code navigation experience can be pretty poor for some languages (e.g. C/C++).
Great stuff, is there any update on searches which include code in branches? I often manually find interesting work done in development branches of cloned repos of the one I'm focused on but which never sees the light of day and not found in search. I imagine having the network also part of the search would be a good facet.
I don't see anything in https://docs.github.com/en/search-github/github-code-search/... :/
I've been using it for a few weeks, and it's a great improvement, congrats on the launch.
(Pleeeease make the right click context menu just be the normal default menu though, I (apparently) "Right-click > Back" a lot when using a mouse rather than a trackpad and I keep getting that goofy Emoji dialog.)
(Pleeeease make the right click context menu just be the normal default menu though, I (apparently) "Right-click > Back" a lot when using a mouse rather than a trackpad and I keep getting that goofy Emoji dialog.)
Are you planning to release code search as open source? I can’t find a link to the source code anywhere.
Loving the new Code Search! Might be super specific, but is there any syntax for searching attributes in HTML elements. For example if a React Component called <Button ...some-props color="red" /> what's the best way to find all the buttons that are red?
Hmm, you can construct a regular expression, something like:
https://github.com/search?q=lang%3Atsx+%2F%3CButton%5B%5E%5C...
lang:tsx /<Button[^\\].\* color="red"/
https://github.com/search?q=lang%3Atsx+%2F%3CButton%5B%5E%5C...
You can do code blocks with 4 consecutive spaces. Backticks are not supported unfortunately
Ooh nice, this looks great! Thanks!!
I imagine a future in which this is integrated into vscode so I can go from an error message in the terminal to a search through my code + third-party modules that my code is importing
You can already do that by including your module folder (eg. node_modules, site-packages etc) in your project tree
Any plans to add support for Vue SFC syntax highlighting?
Edit: correction it looks like that's been fixed since last week, nm
Edit: correction it looks like that's been fixed since last week, nm
I generally like the new code search, but I've got one big gripe: there's no way to sort code results by any kind of proxy for recency.
The old code search had the ability to sort by indexed date. This wasn't perfect, but it was something.
I like keeping up with who's using my code and whether they're leaving comments or commit chains that outline trouble they're having with it. Sometimes old code pops up in the recently-indexed sort, but if I regularly search and look at the top page, I can see most new uses.
Without it, code search is basically useless for this purpose :/
The old code search had the ability to sort by indexed date. This wasn't perfect, but it was something.
I like keeping up with who's using my code and whether they're leaving comments or commit chains that outline trouble they're having with it. Sometimes old code pops up in the recently-indexed sort, but if I regularly search and look at the top page, I can see most new uses.
Without it, code search is basically useless for this purpose :/
(I work on code search.) Yeah, sorry about that. We've heard this feedback a lot. There's two reasons why we haven't implemented this. First, content is shared between repositories which makes this harder than before, when it wasn't. Second, we rebuild the index weekly or even more frequently, so the proxy of "when was this added" that was used doesn't work any more. What we would like to use is "when was this blob added to this branch" but that's extremely expensive to retrieve from Git because Git trees don't record it.
git blame is expensive, especially at scale, on big repos, but the thing to understand is the exact hash something was commmitted at isn't important for time-based indexing. What's wanted is when, ± a few days, something was committed at, which makes for a much cheaper query. (How merges are dealt with might also be material)
Barring that though, the equivalent of a post-commit git hook that updates the DB with 'when this blob was added to this branch' and then run a backfill-enough job.
The easy answer, though, would seem to be keep a copy of last week's index, and run the query twice and figure out a way to efficiently compare results to figure out if something is this week's but not last weeks index.
Also of note, "when was this blob added to this branch" isn't even actually the same as git blame, which means that if a file was touched that matched the search but the latest change to the file doesn't affect the matching line of code, it'd show up as recent, which is not what the user wants.
Barring that though, the equivalent of a post-commit git hook that updates the DB with 'when this blob was added to this branch' and then run a backfill-enough job.
The easy answer, though, would seem to be keep a copy of last week's index, and run the query twice and figure out a way to efficiently compare results to figure out if something is this week's but not last weeks index.
Also of note, "when was this blob added to this branch" isn't even actually the same as git blame, which means that if a file was touched that matched the search but the latest change to the file doesn't affect the matching line of code, it'd show up as recent, which is not what the user wants.
Does this mean it will not be implemented?
We want to do it right if we do implement it, but I can't promise anything concretely. It's not trivial, unfortunately.
Sorting the code search results by date has often been my only guide in a breadth-first search, to find out how to use an API in a recent state of the ecosystem, that I am working with.
I am working with .NET and .NET has seen lots and lots of changes in the past years. I often have to see the usage of a library in the current form of the .NET ecosystem, instead of say, an outdated .NET Framework project from 5 years ago.
Is there any equivalent for doing such a search in the new Code Search?
I am working with .NET and .NET has seen lots and lots of changes in the past years. I often have to see the usage of a library in the current form of the .NET ecosystem, instead of say, an outdated .NET Framework project from 5 years ago.
Is there any equivalent for doing such a search in the new Code Search?
maybe lower standards would help - what does doing it not-quite-right-but-in-one-week look like? can mitigate by setting expectations accordingly
The feature isn't working well yet on C and C++. If I recall correctly it's based on Tree-Sitter[1] parsing, and there are still too many bugs in corresponding grammars - tree-sitter-c[2] and tree-sitter-cpp[3]. Hopefully, it will be greatly improved in the future as the share of the existing and newly written code in C and C++ is quite significant.
[1] https://tree-sitter.github.io/tree-sitter/
[2] https://github.com/tree-sitter/tree-sitter-c/issues
[3] https://github.com/tree-sitter/tree-sitter-cpp/issues
[1] https://tree-sitter.github.io/tree-sitter/
[2] https://github.com/tree-sitter/tree-sitter-c/issues
[3] https://github.com/tree-sitter/tree-sitter-cpp/issues
As an Emacs user, the tree-sitter-based Elixir and HEEx modes are vastly superior to the built-in HTML+ and Elixir modes.
I'm happy that at last, my Stack Overflow question and answer have been fully solved with technology improvements!
https://stackoverflow.com/questions/43891605/search-partial-...
It's been almost 6 years, though... for a search scenario that would be trivial to implement with grep (at scale that's another thing...) Still, a nice example of perfect being the enemy of good, I guess.
https://stackoverflow.com/questions/43891605/search-partial-...
It's been almost 6 years, though... for a search scenario that would be trivial to implement with grep (at scale that's another thing...) Still, a nice example of perfect being the enemy of good, I guess.
Thanks for your patience! It has been a long road with some dead ends (we've wanted to add this since 2012 at least). We actually wrote about why we didn't just use grep in our last blog post: https://github.blog/2023-02-06-the-technology-behind-githubs...
I really enjoyed that blog post. Especially the comparisons that lay out how feasible it can actually be to do the stupid, simple thing of "grep the whole data set every time" up to a surprising point.
The time to pull request with a good code search, paired with GitHub.dev and Copilot is really a force multiplier. What a time to be alive.
I don't get the praise over GitHub code search. I find it very inaccurate and often missing references etc. Maybe it depends on the language you are using it with? (Go here)
Are you referring to what they just released? Github code search has always been notoriously bad. This is a new search product:
"today, our new code search and code view are generally available to all users on GitHub.com"
"today, our new code search and code view are generally available to all users on GitHub.com"
Have you tried the new search that was just launched? It seems to have significantly improved search accuracy. I agree the old version wasn't that great (though still was one of the better options for finding usage of things in the wild like rarely used OSS dependencies I needed to debug).
I have missed Google Code Search, which launched in 2006 and was discontinued in 2013. Similar to GitHub's code search, it supported searching by regex and filtering by language etc. - but obviously the amount of code to search through is orders of magnitude larger than it was 10-15 years ago. Still I wonder what took GitHub so long to build this - it's hardly a novel idea, and it seems like such an obvious power tool for programmers to have.
> Still I wonder what took GitHub so long to build this - it's hardly a novel idea, and it seems like such an obvious power tool for programmers to have.
Scale. You can read more about their engineering at https://github.blog/2023-02-06-the-technology-behind-githubs...
Scale. You can read more about their engineering at https://github.blog/2023-02-06-the-technology-behind-githubs...
You could try https://sourcegraph.com/search
Though GitHub will probably have all the same features eventually and cheaper too.
Though GitHub will probably have all the same features eventually and cheaper too.
It's good.
it's mind bogglingly insanely good compared to the garbage that was there before for years and (I think) still is in Azure DevOps.
it's mind bogglingly insanely good compared to the garbage that was there before for years and (I think) still is in Azure DevOps.
Don't get me wrong but how is this any better than say Scintilla searching locally? I feel like the previous one was very bad and this is baseline but please tell me where I am wrong. From a brief test this looks like something you'd have built with Sphinxsearch a decade plus ago?
No way to sort by recency makes the new code search completely unusable.
Having to hover the file title to see the updated date is horrible.
For some searches, I want to see how developers are solving problems nowadays, not 10 years ago.
No more number of matching files per language.
Not possible to filter by multiple languages anymore.
Such a downgrade.
This is a huge upgrade...but one major annoyance with their new interface is some keyboard shortcuts conflict with browser shortcuts. I'm looking at my repo and I ctrl+f to search the current file and start typing my search query assuming it's working...but it's actually focused on the file browser and started trying to edit the file instead?!?
>Without code search, you might have to clone a bunch of repositories and grep through them
Why is this an issue? I do have a clone of all my company repos, doesn't everyone? Memory is cheap and ripgrep/ag are fast.
Why is this an issue? I do have a clone of all my company repos, doesn't everyone? Memory is cheap and ripgrep/ag are fast.
No, every microservice or new project/ tool get its own repository. There's new ones being created quite frequently, so I have ~10 repositories cloned that most relate to the work I do, but nothing beyond that.
[deleted]
Bandwidth is not cheap for many of us.
how big are the repos at your company?
Bitbucket has a 2GB max size on each repo, I'd suspect other providers have similar restrictions. Would be a problem on mobile but repos that hit that limit are rare.
Bitbucket has a 2GB max size on each repo, I'd suspect other providers have similar restrictions. Would be a problem on mobile but repos that hit that limit are rare.
Seems like grep-aaS? Or am I missing something. It could not e.g. get the definition of a C++ function for me. Really useful still, but should have been there years ago.
My *first* test is not very successful.
> Whoa there!
> You have exceeded a secondary rate limit.
> Please wait a few minutes before you try again; in some cases this may take up to an hour.
> Whoa there!
> You have exceeded a secondary rate limit.
> Please wait a few minutes before you try again; in some cases this may take up to an hour.
It isn't generally available; it requires login.
How do one access the new search?
I think you have to visit https://github.com/search, type a query (use /.../ for a regex, and repo:... to filter by repo), then click "Code" in the left navigation bar (and sign in). The search box on project pages appears to use the old search.
Should be available to all users now, previously was on beta
I cannot detect any difference.
I initially thought “generally available” refers to removing the login wall. I remember when you didn’t have to sign in to search code on GitHub; I miss that.
Probably due to bot abuse. Can't have nice things on the internet.
Requiring an account makes rate limiting vs botnets easier.
Requiring an account makes rate limiting vs botnets easier.
Nice, regular expressions.
RIP Sourcegraph
In my experience Sourcegraph offers a better integration with Gitlab and Github both. And their code search is far superior.
hold on, I paid for copilot, what does github-code-search buy for me? not to mention I can kind of search its code already in the past.
credit to Microsoft for rehabbing their reputation with developers but it seems like a massive trojan horse