I'm all for a rewrite of existing systems, but I think that's going to be a long-term thing (5-10 years) that will happen in parallel to developments in treating code as a database.
I've found RDBMS is quite limited in terms of graph traversal. Maybe you've found a way around this? Would be happy to riff on that.
1. How do you do this without changing the way we program? The most amount of value I would get from this kind of tool is being able to spelunk in a huge codebase with lots of legacy gunk. The cost of entry for this cannot be "rewrite all the codes."
2. What does the query language look like and how do you expose this to the user? This is especially challenging because the query language needs to serve so many different roles.
I'm excited to see what comes of this as I see this problem as a blocker for the next generation of software development. We've more or less solved infrastructure, yet taming complexity still eludes us.
Shameless plug, with a bit of fortuitous timing:
I just released an alpha CLI for my project SourceScape.
It's a tool that indexes your Typescript and Javascript code (Ruby coming soon.) You can then query your code by code structure instead of just raw text. As a trivial example, you can search for all classes with a render method that returns a jsx element with name div.
Hey there HN,
I'm Jieren, and I'm building this app here.
I submitted this two months ago and I wanted to share again as there's been some significant progress. Instead of a fairly janky interface where you're clicking and dragging things around, now it's just a text editor where you type in code and you get search results that mirror this code.
All this is driving towards a vision of a "no-code" [0] builder for static analysis and making static analysis as easy as code search.
Looking forward to your feedback!
jieren at sourcescape.io @jierenchen
[0] Extreme air-quotes because you're literally searching with code blocks
I submitted this two months ago and I wanted to share again as there's been some significant progress. Instead of a fairly janky interface where you're clicking and dragging things around, now it's just a text editor where you type in code and you get search results that mirror this code.
All this is driving towards a vision of a "no-code" [0] builder for static analysis and making static analysis as easy as code search.
Looking forward to your feedback!
jieren at sourcescape.io @jierenchen
[0] Extreme air-quotes because you're literally searching with code blocks
I think it's in the nature of big companies to view everything as a process optimization problem and losing out on "innovators" is viewed as a gap in optimality that needs to be filled. An environment tuned to value extraction is inherently going to be hostile to exploration. Can't have your cake and eat it too, buddy.
This is very cool! It's really awesome to see this code as data concept gaining a lot of traction recently. Hope to see this project developed further.
I'm working on a similar project here: https://sourcescape.io/, but intended for use outside the IDE on larger collections of code (like the codebase of a large company.)
Agreed on the Prolog/Datalog approach of expressing a query as a collection of facts. CodeQL does the same. From one datastore nerd to another, I actually think this is a relatively unexplored area of querying knowledge graphs (code being a very complex, dense knowledge graph.)
Very excited to see where you go next with this "percolate search" functionality in the IDE.
I'm Jieren, creator of SourceScape. SourceScape is a query engine for source code that lets you build up constraints for the code you want to see, much like GraphQL in a GUI. It's a no-code builder for static analysis.
Static analysis is a powerful tool for understanding and verifying code. I want to make writing static code analysis as easy as code search so that you'll be using it all the time. Instead of creating migration spreadsheets, you'll just write queries. Instead of doing nitpicks for code reviews, you'll just write queries.
Happy to answer any questions you have! Also, any feedback would be much appreciated.
Figuring out how to fill the gap between code search and static analysis (code checks.)
Right now the tools we have for programmatically reading through code are:
1. Code search, which is fast, but inaccurate/heuristic.
2. Static analysis, which is slow to run and difficult to write, but very accurate.
I'm building a tool that is as fast and easy to use as code search, and is as accurate and expressive as static analysis.
Still just a landing page [0]. Looking to get a public playground people can mess around with this week.
I'm Jieren, creator of SourceScape. SourceScape is a query engine for source code that lets you build up constraints for the code you want to see, much like a SQL query. You can also think of it as a very fast no-code builder for static analysis checks.
Throughout my career, I've always felt that there was this gap in our ability to dig through source code quickly. Text search is good, but very unreliable. Static analysis takes too long to write. I wanted something that could, for example, find all `.create calls from any instance of UserServiceClient` with 100% accuracy, and quickly.
Earlier this year, I was working on a microservices logging migration that took way longer than it should have. The frustrations I encountered on that project around coordination and verification of the migration became the impetus for me to build SourceScape.
> The oldest: GC was originally invented for Lisp. It's common now.
LISP was so ahead of its time, its parents still haven't met yet. Not very FP, but another gem from SBCL: saving and restoring program state for later use. Now there's the CRIU [0] project for doing this with Linux and Docker containers.
> I'm looking forward to algebraic data types becoming really common.
I'm not too familiar with the full scope of algebraic data types. Wondering: does Typescript have this or is it still missing a few key components? Really like how it has Union types, which I wish Scala would have.
Or the visitor pattern, which is.... the map function.
One thing I will note from recent experience: programming for a long time with immutability cripples your ability to think about mutable things. I was writing up a streaming merge sort this week and it was a brutal nightmare because of all the state I had to deal with. Seems like a call to action to deal with a bit of mutable state every now and then. Everything is too pure these days. We're programmers, not mathematicians dammit.
We should draw a distinction between FP as religion and FP as tool kit.
FP as religion has failed to gain acceptance because it imposes too much cost on the user. I have to rethink my whole stack in terms of category theory AND deal with your terrible ecosystem? Hard pass.
FP as toolkit, on the other hand, has been a smashing success. Most of the core ideas of FP are mainstream now and some of the latest advances in non-FP ecosystems (React, for example) are based on FP ideas.
What I've found is that the real challenge for a beginner is making continuous and sustained progress. The key number in the (1.01)^365 equation is the ^365 part. Do something difficult every single day.
Beyond just sticking with it, you also need to continually challenge yourself. Avoid staying into the comfort zones that fear and ego will shunt you into.
These are very difficult to do as a beginner because you're going to encounter so many demoralizing roadblocks. I still remember when I first started writing code. It was incredibly painful. Things that now take me minutes would take days.
The real key here isn't some technique that will make the learning easier, it's in getting the emotional fortitude to push through these struggles. Your struggles will be unique and it's impossible for me to prepare you with solutions for every novel challenge you face. "Ask not for lighter burdens, ask for stronger backs" and all that jazz. There's a lot of resources online, esp. on Youtube, about increasing mental discipline. Cold showers, fasting, mental discipline, stoicism, etc. I recommend investing some time into trying these techniques for improving pain tolerance.
The funny thing about knowledge work is that it sits somewhere between additive and compounding.
Additive like: Let's say you shovel sand into a pit and you shovel 1% more than your peers every day for a year. By end of year, you will have shoveled 1% more.
Compounding like: You're a derivatives trader and make 1% more returns every day for a year than your peers. By end of year you will have made 3780% more.
Knowledge isn't perfectly compounding, but it must necessarily build upon previous knowledge. Abnormal levels of investment and dedication in learning definitely pay off in the long run and compound faster than you would think.
> They complement each other. There are certain things that I’d love to pull up a command line for (e.g. quickly aligning dozens of objects using a non trivial heuristic); but on the other hand there are plenty of things where the act of visually working through it, placing multiple options side by side on the canvas, rearranging things by hand etc., helps in solving the problem itself.
> What would be really nice is if we had a solid set of standards for inter operation, such that I could work on the same video editing project in After Effects, through the command line, through some open source tools someone else wrote... etc.
The big challenge I see here is a two-way mapping between code/representation and GUI. Ronin here has code >> graphics, but question is how to have a clean mapping from graphics >> code?
> I’m not holding my breath because this kind of computing, while more aligned in philosophy with the early composable UNIX roots, is IMO most entirely incompatible with the state of the (creative) software industry/economy (Adobe, like everyone else, would like to do as much as they can so that you do not give any of your time or money to creative software other than Adobe’s).
I do wonder if this is going to be a space that gets disrupted. We're already seeing this with smaller players like Sketch and Figma going into the image space. Still GUI tools, but proves that the barrier to entry isn't quite as high.
Is the technical moat for audio and video THAT insurmountable? I get that people won't be making the next hot music video on open source software right now, but I personally could use something dead simple to compose together a bunch of screen recordings into a demo video.
> I have a project right now where I have assets generated in Sketch, some other assets generated by Python scripts, all feeding into an AE project, the output of which gets fed back into some of the Python scripts... I can do great things, but it’s super kludgy/buggy to work with, and as a software engineer see so many trivial ways in which it could all be so much faster and more pleasant to work with (e.g. After Effects natively only supports very limited JS like scripting, so I have to have duplicated Python + AE code... and if the AE motion tracker could serialize tracking data for Python...).
This is exceptionally cool. Thanks for sharing and building!
When I'm using Sketch or Movie Maker or Logic or some other GUI-based >>x<< editing tool, the thing that keeps going through my head is: "Dammit I wish I had the abstraction tools that I have as a programmer. It would literally make this task 100x faster."
To me Ronin is some strong validation that all these editing tools would be much improved by a human-readable and programmable intermediate representation of what you're doing with the GUI. The GUI is just an editor for modifying this representation.
I've found RDBMS is quite limited in terms of graph traversal. Maybe you've found a way around this? Would be happy to riff on that.