Hi, I'm a developer on VS Code and wanted to clarify that there was actually a logical memory leak in there.
The problem is that I tried to reproduce it back in October while running from sources, and I could not reproduce any leak. I believed the issue is about the memory usage reported by the OS vs the memory usage reported by the logical JS Heap Snapshot, which is a common topic for garbage collected runtimes, such as JS / v8. That is why we believed the issue is about the v8 VM not freeing memory back to the OS when we freed memory logically.
But it turns out I tried the steps again just now, and there is a real memory leak in there! It only reproduces when running a built version of VS Code, such as VS Code Insiders or VS Code Stable. The built versions of VS Code offer file type extension recommendations, for example you open a `.php` file and a recommendation comes in to point you to the PHP extension. When running out of sources, those recommendations do not show up and the entire code responsible for those does not execute.
Thank you for bringing this up to my attention and I'm sorry for misunderstanding the initial issue.
Hovering over the activation time will show the reason the extension is active. e.g. "Activated because you opened a javascript file", "Activated on start-up", etc.
VS Code is architected in a way where extensions are not eagerly activated by default. Each extension can declare a list of activation events, such as e.g. opening a file of a certain language, invoking a specific command, starting debugging, etc. See for example the quite long list of activation events for our built-in TypeScript extension [1].
However, we also offer an activation event called ' * ', which means an extension can ask to be activated on startup. Some over-eager extensions might be using ' * ' to start up as soon as you open a VS Code window. You can find those extensions at any time using F1 > Developer: Show Running Extensions which will show the subset of extensions running at any time, and if they were activated on startup or not.
Moreover, that view can guide you into profiling the extension host and can help you easily figure out if any extension is consuming extensive CPU. This was added quite recently [2].
When we started the project, we did write tokenizers by hand. I mention that in the blog post. You can write some very fast tokenizers by hand, even in JavaScript. Of course they won't be as fast as hand written tokenizers in C, but you'd be surprised how well the code of a hand written tokenizer in JavaScript can be optimized by a JS engine, at least I was :). IR Hydra 2 is a great tool to visualise v8's IR representation of JS code [1]. It is a shame it is not built into the Chrome Dev Tools.
In the end, we simply could not write tokenizers for all languages by hand. And our users wanted to take their themes with them when switching to VS Code. That's why we added support for TM grammars and TM themes, and in hindsight I still consider it to be a very smart decision.
I hate slowness and inefficiency too, that's why I try to make the editor as fast as possible :), but at least in this case, it is not the dynamic nature of JS to blame, but rather the nature of TM grammars. TM grammars consist of rules that have regular expressions, which need to be constantly evaluated; and in order to implement a correct TM grammar interpreter, you must evaluate them.
I've looked in the past for optimization opportunities in the C land (mostly through better caching), which yielded quite nice results [1][2]. I would love if you'd want to take a look too.
At this point, in tokenization, 90% of the time is spent in C, matching regular expressions in oniguruma. More precisely, regular expressions are executed 3,933,859 times to tokenize checker.ts -- the 1.18MB file. That is with some very good caching in node-oniguruma and it just speaks to the inefficiency of the TM grammars regex based design, more than anything else.
It is definitely possible to write faster tokenizers, especially when writing them by hand (even in JS), see for example the Monaco Editor[3] where we use the TypeScript compiler's lexer as a tokenizer.
At least in this case, inefficiencies are not caused by our runtime.
Yes, those comparisons at the end show differences in rendering caused by the "approximations" used prior to VS Code 1.9. They were all caused by the difference between the ranking rules of CSS selectors and the ranking rules of TM scope selectors
- all the regular expressions in TM grammars are based on oniguruma, a regular expression library written in C.
- the only way to interpret the grammars and get anywhere near original fidelity is to use the exact same regular expression library (with its custom syntax constructs)
in VSCode, our runtime is node.js and we can use a node native module that exposes the library to JavaScript
- in the Monaco Editor, we are constrained to a browser environment where we cannot do anything similar
- we have experimented with Emscripten to compile the C library to asm.js, but performance was very poor even in Firefox (10x slower) and extremely poor in Chrome (100x slower).
- we can revisit this once WebAssembly gets traction in the major browsers, but we will still need to consider the browser matrix we support. i.e. if we support IE11 and only Edge will add WebAssembly support, what will the experience be in IE11, etc.
The problem is that I tried to reproduce it back in October while running from sources, and I could not reproduce any leak. I believed the issue is about the memory usage reported by the OS vs the memory usage reported by the logical JS Heap Snapshot, which is a common topic for garbage collected runtimes, such as JS / v8. That is why we believed the issue is about the v8 VM not freeing memory back to the OS when we freed memory logically.
But it turns out I tried the steps again just now, and there is a real memory leak in there! It only reproduces when running a built version of VS Code, such as VS Code Insiders or VS Code Stable. The built versions of VS Code offer file type extension recommendations, for example you open a `.php` file and a recommendation comes in to point you to the PHP extension. When running out of sources, those recommendations do not show up and the entire code responsible for those does not execute.
Thank you for bringing this up to my attention and I'm sorry for misunderstanding the initial issue.