Introducing the Windows Pseudo Console (ConPty)(blogs.msdn.microsoft.com)

297 pointsby mnkypete8 anni fa190 comments

blogs.msdn.microsoft.com

Introducing the Windows Pseudo Console (ConPty)

https://blogs.msdn.microsoft.com/commandline/2018/08/02/windows-command-line-introducing-the-windows-pseudo-console-conpty/

193 comments

ChuckMcM8 anni fa

This is pretty huge. For as long as I can remember the response to command line applications talking to command line applications was "Why would you want to do that? Use (RPC | shared memory | some other IPC mechanism)." And nobody at Microsoft seemed to understand how much simpler it was to use ptys. They seem to have completely capitulated to the notion ptys and are dropping them into the next release of W10. I wish this had happened 10 years ago but hey, I'll take it.

cryptonector8 anni fa

It's the reality of the market, which is why Windows is adding Linux compatibility (as is every *BSD, Illumos, ...).

But also it's the fact that three decades of not even life support has left the Windows console in pretty sad shape -- the folks tasked with getting it into better shape were bound to see the value of ptys.

Lastly, don't forget that Windows NT was meant to be a console OS, like VMS. There must still be people, even if very few, at MSFT who appreciate text-oriented apps.

For me, the tty/pty, shells, screen/tmux/..., ssh, and so on, are the things that make Unix so powerful. The fact is that Win32 is far superior in a number of areas (SIDs >> UIDs/GIDs, security descriptors >> {owner, group, mode, [ACL]}, access tokens >> struct cred), but far inferior in the things that really matter to a power user trying to get things done.

SideburnsOfDoom8 anni fa

> Lastly, don't forget that Windows NT was meant to be a console OS, like VMS. There must still be people, even if very few, at MSFT who appreciate text-oriented apps.

I expect that, like Linux compatibility, most of it is not about "apps" but about being better at running in the cloud, where a (virtual) machine or container needs to be as light as possible, and to be configured and a service launched in it as unattended/automated manner as possible. Stripping out the GUI and making command lines work better works towards these goals.

cryptonector8 anni fa

That's a really good point.

pjmlp8 anni fa

As power user that gets things done on Windows, it never bothered me that it hasn't an UNIX like console.

If fact it bothered me more that I couldn't get a Borland like devenv on Linux and had to keep myself happy with XEmacs.

acqq8 anni fa

I understand this as a "compatibility with * nix" existing software, not as an "amazing feature." Can anybody suggest why I should like it, except for the compatibility with the software written for * nix terminals? Even ssh being too interconnected with a terminal down to the many details was a shock for me... I expected the simple encryption mechanism, over which whatever communicates, even if that would implement the "sh" part of it only on top of that, but no, it's everything spaghettisized and recombined with everything unnecessary like emulating the devices that don't exist for many decades -- in something that should have a clear separation between the task of transporting encrypted data with authenticating from anything else. I can't see it as being positive, security-wise.

Isn't it strange that today everybody has very powerful GPUs and CPUs and the graphical displays with immense RAM and then using all that to emulate the terminals last existing decades ago appears to be so important, even for something that should be just a secure communication protocol?

Why do we still spend so much energy to decide which console of many decades ago we "support" when it seems that all are flawed, at least compared to what the modern OSes can provide, as soon as the "compatibility" is not needed?

Isn't all that "hardware console" compatibility stuff just a historical accident from the "bad old days" of 300 baud lines between the mainframe and the "terminal" which had a few bytes of RAM total? In the days when e.g. the Thunderbolt 3 can carry 5 GB/s, and the rest of the hardware matches? Why do people still so cling to it? I'd really like to know what I am missing.

pjc508 anni fa

> I expected the simple encryption mechanism, over which whatever communicates

In the UNIX world, that's what it gives you - a stream of bytes. Hence things like rsync-over-ssh or git-over-ssh. It also has a port forwarding mode which has special support for X11, which gives you remote windowing over a stream of bytes too.

The main, huge, benefit is that the abstraction is pretty simple, it's discoverable, and you can use the same interface as a human. You can also plug any stream-of-bytes into any other stream-of-bytes, whereas API or RPC based systems have to be designed to interoperate.

acqq8 anni fa

As I’ve tried to implement my minimal ssh client (just to connect, execute some command and get the result) I’ve had exactly opposite impression of the “just a stream of bytes” that you mention -- exactly the lack the abstraction. Can you point to any source that does ssh without having to care about a lot of weird terminal and console ancient stuff? I’d be really glad to see it! To me it looked as “everything and the kitchen sink” (that is, exactly the kind of things mentioned in the OP or the comments, like terminal signals and whatnot) has to be there.

SSL is straightforward compared that, at least, once the keys are set. But ssh... as seen in the OP even the console or the terminal or however that part it called has to be very special, and they are obviously proud they implemented that too. In 2018. Probably decades after the last single hardware terminal was sold.

pasabagi8 anni fa

I see your point - but don't you find it kinda charming? It's a bit like drinking tea in a china cup, or eating a scone with clotted cream.

I think there is probably a lot of room for improvement in the terminal world, and I agree that a lot of the really old stuff makes things a bit counter-intuitive, but for whatever reason, it seems that people who make really good software also tend to be the people who are pretty fanatical about backwards compatibility. Consider vim, for instance.

In the end, I think the basic model of interoperable, small programs that manipulate streams of text is really good - so people will put up with any number of weird rituals to live in that model. It's also very humble, and very unexciting, so it's the kind of thing that's hard to get people hyped about. So it probably filters by the people that like old things.

ThJ8 anni fa

You are conflating what SSH does with what the terminal you're running it in does. SSH barely knows what terminal emulation is. You can literally pipe binary data into it at one end and receive that data into a program at the other end. That's what programs like scp and rsync do. They launch ssh and use it as a data channel.

acqq8 anni fa

> You can literally pipe binary data into it at one end and receive that data into a program at the other end

That is the user perspective, not what has to be in a ssh program to work.

zvrba8 anni fa

> Why would you want to do that? Use (RPC | shared memory | some other IPC mechanism).

Yes, structured data exchange is the correct answer. When I have the opportunity to code something from scratch, this is the route I take.

pjc508 anni fa

> When I have the opportunity to code something from scratch

But how often does that happen, outside of toy systems and support utilities?

zvrba8 anni fa

It happens once in a while. Or you're lucky and stumble upon a team where people made the right choices in the beginning.

dboreham8 anni fa

26 years ago.

deepaksurti8 anni fa

Which is : https://en.wikipedia.org/wiki/Windows_3.1x

JdeBP8 anni fa

That is from a different family of operating systems to what is headlined here, and does not have the client-server system of the operating system family discussed.

* https://superuser.com/a/319187/38062

dboreham8 anni fa

No, NT is 26 years old (older if you were developing it).

Source : I have the t-shirt (polo shirt actually).

caf8 anni fa

Will there be a terminfo database entry for ConPty? What TERM string should we expect to see?

To elaborate: although an ordinary POSIX pty doesn't inherently have a terminal type - that's entirely down to whatever emulator is connected to the master side - the way the ConPty system translates Console API calls into terminal control codes means that it necessarily needs to pick a terminal emulation, which all actors in the ConPty system are expected to use.

A terminfo database entry would be useful both for applications running on *NIX hosts but displaying on a remote ConPty master somewhere, as well as for porting existing terminal applications to Windows where they will run on a ConPty slave.

As a follow-up question, presumably this means that the SSHD running on Windows as a ConPty master needs to translate between whatever terminal emulation the ssh client is connected to and the one expected by ConPty / ConPty apps (in the same way it must translate between the native ConPty UTF-8 and the remote charset)?

zadjii8 anni fa

So this is a confusing situation on Windows.

Commandline applications on linux rely on a TERM setting (with termcaps) to be able to know what VT sequences the terminal is able to support. On Windows, we only really have one terminal, conhost.exe, and our goal there is to be compatible with TERM=`xterm-256color`. That's why you'll see that WSL has that set as the default term setting.

Now even with ConPTY, when a client writes VT sequences, they still need to be interpreted by conhost. This is because technically, a console application could use both VT and the console API, and we need to make sure the buffer is consistent. So clients should still assume that they should write out `xterm-256color` compatible sequences.

Now on the other side of thngs, the "master"/terminal side of conpty, we're going to "render" the buffer changes to VT. Fortunately, we'd dont really need a deep VT vocabulary to make this possible, so the VT that's coming out of a conpty is actually pretty straightforward, probably even vt100 level (or I guess vt100-256colors, as insane a termcap that would be).

It's definitely a future feature that we'd like to add to make conpty support multiple different TERM settings, and change the sequences we emit based on what the terminal on the other side is going to expect.

We haven't really gotten into the nitty gritty of all of this quite yet, so if you find bugs or have feature requests, we're happy to take a look at them. You can file issues on [our github](https://github.com/microsoft/console) and we'll add them to our backlog

caf8 anni fa

Thanks. It makes sense to pick an existing terminal like xterm-256color to target - that way you don't have to worry about a new terminfo database entry getting distributed out.

The nitty-gritty can get quite nitty - things like bracketed paste and set window title.

zadjii8 anni fa

So this question is a little complicated currently, but I want you to know that I am planning on coming back to answer it, probably tomorrow morning

voltagex_8 anni fa

Add an issue: https://github.com/microsoft/console/issues

cryptonector8 anni fa

I second this question. Perhaps there will only be support for vt100/vt220?

lokedhs8 anni fa

That would be sad, as there is already far too much software out there that hard-codes escape sequences, completely ignoring the TERM environment variable.

Even worse, sometimes they won't even disable escape codes when they should not be displayed.

I've posted bug reports for very popular software packages whose commandline always output vt102, even when TERM is set to dumb or when run through pipes. That makes grepping for error messages somewhat annoying. In at least some cases these reports were ignored.

cryptonector8 anni fa

What next? Job control signals?? :) (EDIT: How about tmux?)

Anyways, this is fantastic. Finally, proper ssh functionality!

This will encourage development of console (text-oriented) apps for Windows, which I hope will be much simpler. Interfacing with the console can be really difficult if you're coming from *nix. Ideally all the WIN32-specific code in, e.g., jq[0], could be ripped out.

[0] https://github.com/stedolan/jq (look in src/main.c)

tom_8 anni fa

The lack of signals in Windows is the very opposite of a flaw! - Windows has just never pretended you can get away without a message loop.

quotemstr8 anni fa

Windows does have signals! It just splits them into a few facilities. POSIX "synchronous" signals correspond to SEH exceptions and can be handled roughly the same way --- except that signals have process global handlers and Windows has thread-local ones, because the glibc people are sticks in the mud and are hostile to any attempt to make signals suck less.

For asynchronous signals, like SIGINT, Windows create a new thread out of thin air to deliver your app a notification. That's not really all that much better than a signal from a concurrency perspective.

Windows even has APCs, which are like regular signals that are delivered only at explicit system call boundaries.

Every operating system needs some mechanism to tell a process to do something. Windows has evolved an approach that isn't all that different from Unix signal handling.

cryptonector8 anni fa

Spawning a new thread to handle a signal is much better than preemption: you then have no concerns about async-signal-safety that aren't plain old thread-safety concerns. I'd much rather have thread-safety constraints than async-signal-safety constraints.

quotemstr8 anni fa

Sort of. At least you can mask signals. I agree that threads are probably better overall, but I don't think it's a huge difference.

Vogtinator8 anni fa

Linux can do that as well, see SIGEV_THREAD.

quotemstr8 anni fa

Only for AIO, which few use.

andreyv8 anni fa

You can spawn a thread manually and use the POSIX sigwait() function.

quotemstr8 anni fa

That won't stop the process-wide registered signal handler working and it won't do anything about being able to reliably handle synchronous signals.

cryptonector8 anni fa

Yes, of course signals are easily the worst thing in Unix, but job control is nice.

zvrba8 anni fa

It's already there: https://docs.microsoft.com/en-us/windows/desktop/ProcThread/...

You can set various limits, though I haven't seen functions to stop/resume a job.

ygra8 anni fa

You can suspend and resume threads, so if you model your jobs with processes that could be used for that, I guess.

loa-in-backup8 anni fa

*with threads

ygra8 anni fa

Processes. A quick skim of the documentation linked by the person I answered suggests that processes can be associated with a job object, but threads cannot. But a process cannot really be suspended directly, except by suspending all its threads.

skrebbel8 anni fa

First, tmux works fine in WSL. Secondly, if you like both tmux and Windows, there's a fair chance you'll like ConEmu's split panel facility even better. It's basically tmux, but more "windowsy". ConEmu is spectacular.

amluto8 anni fa

For me, the most surprising thing is that the new PTY devices use UTF-8. Not UTF-16 or UCS-2 or weird little endian variants thereof, and not even wchar_t.

This is so un-Windows-like.

zadjii8 anni fa

It is! But this is a very un-windows like feature, isn't it? We want this to work on other platforms with as little modification as necessary, and frankly, jumping through the wchar_t<->char hoops is a _pain_. So we'll do it for you!

cryptonector8 anni fa

Hear hear! wchar_t is a disaster. UTF-16 is terrible. I'm not at all convinced that 2^21 codepoints will be enough, so someday it'd be nice to be able to get past UTF-16 and move to UTF-8, and Windows and ECMAScript are the biggest impediments to that. Your choice of UTF-8 will tend to place UTF-8 on a level playing field in Win32.

I guess, too, that this is the end of codepages -- I doubt they'd go away, but there should be no more need to struggle with them, just use UTF-8. You'll still need a semblance of locale, for localization purposes, naturally, but all-UTF-8-all-the-time is a great simplification.

mehrdadn8 anni fa

Confused, where does 2^21 code points come from and how is that related to the UTF-16 vs. UTF-8 distinction? Can't both of them encode all Unicode code points? Or are you thinking of code units perhaps, and UCS-2? Although even there I'm confused where the 2^21 came from.

yuubi8 anni fa

UTF-16 has a limit on the size of a code point because a code point is either a single normal code unit or a pair of surrogate code units, each encoding 10 bits of the code point (I think I used the right terminology). UTF-8 has a natural extension path to up to 7-byte encodings with all the usual UTF-8 properties (first code unit indicates how many remain, other code units are recognizable as not the first).

mehrdadn8 anni fa

Where are you getting this information though? I haven't worked out the bits myself yet but Wikipedia's first sentence itself says UTF-16 can encode all 1,112,064 valid code points of Unicode, which is already more than 2^(10+10) = 1,048,576.

JdeBP8 anni fa

Unicode code point space: Was 16-bit (0000 to FFFF), then 32-bit (00000000 to FFFFFFFF), and is now 21-bit (00000000 to 0010FFFF)

UTF-16: Encodes the entire 21-bit range, encoding most of the first 0000 to FFFF range as-is, and using surrogate pairs in that range to encode 00010000 to 0010FFFF. The latter range is shifted to 00000000 to 000FFFFF before encoding, which can be encoded in the 20 bits that surrogate pairs provide. This is a subtlety that one likely does not appreciate if one learns UTF-8 first and expects UTF-16 to be like it.

UTF-8: Could originally encode 00000000 to 7FFFFFFF, but since the limitation to just the first 17 planes a lot of UTF-8 codecs in the real world actually no longer contain the code for handling the longer sequences. Witness things like the UTF-8 codec in MySQL, whose 32-bit support conditional compilation switch is mentioned at https://news.ycombinator.com/item?id=17311048 .

amluto8 anni fa

> a lot of UTF-8 codecs in the real world actually no longer contain the code for handling the longer sequences.

Not exactly. A conforming decoder MUST reject them.

MySQL’s problem is that, by default, it can’t even handle all valid code points.

JdeBP8 anni fa

They reject them by not having a code path that successfully decodes them.

mehrdadn8 anni fa

I don't see anything wrong with what you're saying, but I still don't get how it explains the original comment I replied to [1]:

> I'm not at all convinced that 2^21 codepoints will be enough, so someday it'd be nice to be able to get past UTF-16 and move to UTF-8

UTF-16 currently uses up to 2 16-bit code units per code point, whereas UTF-8 uses up to 4 8-bit code units per code point, and the latter wastes more bits for continuation than the former. How is "getting past UTF-16 and moving to UTF-8" supposed to increase the number of code points we can represent, as claimed above? If anything, UTF-16 wastes fewer bits in the current maximum number of code units, so it should have more room for expansion without increasing the number of code units.

[1] https://news.ycombinator.com/item?id=17771351

JdeBP8 anni fa

If I had intended to explain that, I wouldn't be replying to your comment about your not working out the bits with an explanation that works out the bits for you, and shows that it really is capable of encoding all 17 planes even though surrogate pairs have only 20 bits.

And as you can see, if you do work out the bits, you find that cryptonector is wrong, since UTF-8 (as it has been standardized from almost the start of the 21st century, and as codecs in the real world have taken to implementing it since) encodes no more bits than UTF-16 does. It's 21 bits for both.

cryptonector8 anni fa

No, UTF-8 could, and used to, encode much more than 21 bits of codepoint space. It has been artificially limited to better match UTF-16 -- UTF-16's limits are not artificial but fundamental. If some day we need more bits, we'll simply obsolete UTF-16 and drop those limits on UTF-8. MSFT seems to be taking steps to put UTF-8 on a level or even higher playing field than UTF-16. We should welcome this.

mehrdadn8 anni fa

It seemed like you were trying to correct my comment, but everything you said seemed to support what I was saying, so I thought maybe you were trying to continue the initial discussion... I guess not.

With regards to the comment then: the range downshifting you mentioned is merely a step in the encoding process though -- the code point is still whatever it was. If you read parent comment, it had claimed that, in a surrogate pair, each of the 2 code units encodes 10 bits of the code point... but that would be missing 1 bit when the code points need 21 bits to be represented. That's all I was saying there. The extra bit indicating that it's in fact a surrogate pair isn't some kind of implicit dummy bit that you can pretend isn't encoding anything -- if it wasn't there then clearly it wouldn't be encoding the code point for a surrogate pair anymore.

jwilk8 anni fa

Yes, UTF-16 can encode all currently valid Unicode codepoints, which is more than 2²⁰ but less than 2²¹. But cryptonector doesn't believe it will be enough in the future.

OTOH, UTF-8, as originally defined, can encode 2³¹ codepoints.

speeder8 anni fa

Utf-16 is always 16 bit, utf-8 is variable, can go from 8 to 32 as needed...

This does make coding for utf-8 harder, but when it works is really wonderful stuff.

AnIdiotOnTheNet8 anni fa

UTF-16 is not always 16b/codepoint. UCS-2, its ancestor and more or less what is actually used by Windows when it talks about "Unicode", was 16b/codepoint. It turned out that 64k codepoints wasn't enough for everybody, so they increased the range and added "surrogate pairs" to UCS-2 to make UTF-16. Codepoints in UTF-16 are, therefore, either 16b/codepoint or 32b/codepoint. Many systems that adopted UCS-2 claim UTF-16 compatability but in fact allow unpaired surrogate pairs (an error in UTF-16). The necessity of encoding such invalid UTF-16 strings in UTF-8 under certain circumstances lead to the pseudo-standard known as WTF-8[0].

UTF-8 was designed (as legend has it, on the back of a napkin during a dinner break) after the increase in range and doesn't suffer from the same problem. Additionally, it's straight-forward "continuation" mechanism isn't any more difficult to deal with than surrogate pairs, and it doesn't have any endianess issues like UTF-16/UCS-2.

[0]https://simonsapin.github.io/wtf-8/

a3f8 anni fa

UCS-2 is always 2 octets, UTF-16 allows for so-called surrogate pairs which expands the bits available making it variable size.

mehrdadn8 anni fa

Look up UCS-2 that I mentioned in my comment.

[deleted]8 anni fa

cryptonector8 anni fa

UTF-16 is garbage. Windows is stuck with it because it was too early an adopter of Unicode. Oh the irony. This may set Windows on a path to deprecating UTF-16 -- godspeed!

MarkSweep8 anni fa

Another exciting development in moving beyond UTF-16: Microsoft is experimenting with adding a native UTF-8 string in .NET next to the existing UTF-16 string:

https://github.com/dotnet/coreclr/commits/feature/utf8string

GordonS8 anni fa

I've just spent some time reading through the proposals, it made for a fascinating read! It's really interesting to see the work and discussions that go into a seemingly simple feature like this.

swozey8 anni fa

Could you elaborate? I've been under the guise for most of my career that doubling a digit leads to huge benefits that I'm too comp-sci ignorant to understand.

cryptonector8 anni fa

Can I assume that's just subtle humor on your part?

If not:

UTF-16 is born of UCS-2 being a very poor codeset, as it was limited to the Unicode BMP, which means 2^16 codepoints, but Unicode has many more codepoints, so users couldn't have the pile-of-poo emoticon. Something had to be done, and that something was to create a variable-length (in terms of 16-bit code units) encoding using a few then-unassigned codepoints in the BMP. The result yields only a sad, pathetic, measly 2^21 codepoints, and that's just not that much. Moreover, while many codesets play well with ASCII, UTF-16 doesn't. Also, decomposed forms of Unicode glyphs necessarily involve multiple codepoints, thus multiple code units... Many programmers hate variable length text encoding because they can't do simple array indexing operations to find the nth character in a string, but with UTF-8, UTF-16, and just plain decomposition, that's a fact of life anyways. If you're going to have a variable-length codeset encoding, you might as well use UTF-8 and get all its plays-nice-with-ASCII benefits. For Latin-mostly text UTF-8 also is more efficient than UTF-16, so there is a slight benefit there.

Much of the rest of the non-Windows, non-ECMAScript world has settled on UTF-8, and that's a very very good thing.

swozey8 anni fa

No humor whatsoever. Thank you for the explanation! I'm an ops person who knows python/golang to a dangerous extent and have never gone out of my way to understand the UTF reasonings. Your post intrigued me and made me want to ask why you felt that way. This will make me sound horrendously ignorant to someone of the likes of someone such as yourself but I'm here to learn.

JonathonW8 anni fa

UTF-16 has a bit of a funky design (using four byte/two code unit surrogate pairs to encode code points outside the basic multilingual plane) that ultimately restricts Unicode (if compatibility is to be maintained with UTF-16, at least) to 17 planes, or 2^20 code points (about 1 million).

UTF-8 uses a variable length encoding that allows for more characters-- if restricted to four bytes, it allows for 2^21 total code points; it's designed to eventually allow for 2^31 code points, which works out to about 2 billion code points that can be expressed.

(Granted, this is all hypothetical-- Unicode isn't even close to filling all of the space that UTF-16 allows; there aren't enough known writing systems yet to be encoded to fill all of the remaining Unicode planes (3-13 of 17 are all still unassigned). But UTF-16's still nonstandard (most of the world's standardized on UTF-8) and kind of ugly, so the sooner it goes away, the better.)

swozey8 anni fa

Thank you, this was an incredibly understandable explanation.

JdeBP8 anni fa

That is a bit misleading to the point of error, on several points:

* Your timeline is backwards. UTF-8 was designed for a 31-bit code space. Far from that being its future, that is its past. In the 21st century it was explicitly reduced from 31-bit capable to 21 bits.

* UTF-16 is just as standard as UTF-8 is, it being standardized by the same people in the same places.

* 17 planes is 21 bits; it is 16 planes that is 20 bits.

peterkelly8 anni fa

Joel Spolsky's essay "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" is an excellent read:

https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

evanrelf8 anni fa

Wikipedia says: "UTF-8 requires 8, 16, 24 or 32 bits (one to four octets) to encode a Unicode character, UTF-16 requires either 16 or 32 bits to encode a character"

https://en.wikipedia.org/wiki/Comparison_of_Unicode_encoding...

[deleted]8 anni fa

TeMPOraL8 anni fa

> I've been under the guise for most of my career that doubling a digit leads to huge benefits that I'm too comp-sci ignorant to understand.

I was confused about this for years, too. But it turns out it's just a problem of bad naming. Happens more in this industry than we'd like to admit.

As other explained, it boils down to UTF-16 being 16-bit, and UTF-8 being anything from 8- to 32-bit. It should have been named UTF-V (from "variable") or something, but here we are.

cryptonector8 anni fa

UTF-16 is a variable-length encoding using up to two code units which each are 16-bits wide.

UTF-8 is a variable-length encoding using up to 4 code units (though it used to be up to 6, and could again be up to 6) each of which are 8-bits wide.

Both, UTF-16 and UTF-8 are variable-length encodings!

UTF-32 is not variable-length, but even so, the way Unicode works a character like ´ (á) can be written in two different ways, one of which requires one codepoint and one of which requires two (regardless of encoding), while ṻ (LATIN SMALL LETTER U WITH MACRON AND DIAERESIS) can be written in up to five different ways requiring from one to three different codepoints (regardless of encoding).

Not every character has a one-codepoint representation in Unicode, or at least not every character has a canonically-pre-composed one-codepoint representation in Unicode.

Therefore, many characters in Unicode can be expected to be written in multiple codepoints regardless of encoding. Therefore all programmers dealing with text need to be prepared for being unable to do an O(1) array index operation to get at the nth character of a string.

(In UTF-32 you can do an O(1) array index operation to get to the nth codepoint, not character, but one is usually only ever interested in getting the nth character.)

tremon8 anni fa

For completeness sake: the confusing part is that there is (used to be) a constant-length encoding that uses the exact same codepoints as UTF-16, but doesn't allow the variable-length extensions. That encoding is called UCS-2 and although deprecated, is the reason why many people think UTF-16 is constant-length.

cryptonector8 anni fa

Not the same codepoints as UTF-16, but a subset. But yes, UCS-2 is the source of all this evil.

TeMPOraL8 anni fa

Thanks. It turns out I was even more confused than I thought I was.

JdeBP8 anni fa

I've been waiting for two decades to revise this particular Frequently Given Answer.

* http://jdebp.info./FGA/capture-console-win32.html

red75prime8 anni fa

I suppose it's mostly TUI programs, which use low level console API. So did you try to capture output of something like Far Manager[0]? If so, will it be much simpler to parse escape sequences of VT100?

[0] https://farmanager.com/

zadjii8 anni fa

Hey I'm one of the Console devs who's been working on this feature for a while now. I'll be hanging around in the comments for a little while to try and answer any questions that people might have.

TL;DR of this announcement: We've added a new pseudoconsole feature to the Windows Console that will the people create "Terminal" applications on Windows very similarly to how they work on *nix. Terminals will be able to interact with the conpty using only a stream of characters, while commandline applications will be able to keep using the entire console API surface as they always have.

zokier8 anni fa

While it is nice that MS is focusing on console and command line now, it seems to me that you are mostly working on improving compatibility with legacy UNIXy stuff.

Do you have some vision or plans to go well beyond the classic UNIXy style of console and command line? I'm thinking in the lines of projects like DomTerm http://domterm.org/ which could have nice interactions with e.g. PowerShell.

joeyaiello8 anni fa

PM for PowerShell here!

I haven't seen DomTerm before, but it looks pretty awesome. At a glance, it's basically a GUI-fied tmux hosted in Electron? It would be awesome to have in Windows, but wouldn't that just require that DomTerm add support for these ConPty APIs?

In any case, I'm more interested in your proposed interactions. Did you have anything cool in mind? Given that we ship PowerShell on Linux, we could theoretically do some stuff there (including within PowerShell on WSL) before it's hooked up to ConPty

sime20098 anni fa

I'm not the person you were asking, but this should interest you.

I've been working on a terminal emulator ( Extraterm http://extraterm.org/ ) with some novel features which would dovetail nicely with how PowerShell works. The first is the ability to send files to the terminal where they can be displayed as text, images, sound, etc or as an opaque download. Extraterm also adds a command `from` which lets you use previous terminal output or files, as input in a command pipeline. See http://extraterm.org/features.html "Reusing Command Output" for a demo. This opens up other, more interactive and iterative workflows. For example, you could show some initial data and then in later commands filter and refine it while visually checking the intermediate results.

What I would like to do sometime is integrate this idea with PowerShell and its approach of processing objects instead of "streams of bytes". It should then be possible to display a PowerShell list of objects directly in the terminal, and then reuse that list in a different command while preserving the "objectness" of the data. For example, you could show a list of user objects in one tab and then in another tab (possibly a different machine) grab that list and filter it the same way as any normal list of objects in PowerShell. You could also directly show tabular data in the terminal, let the user edit it "in place" in the terminal, and then use that editted data in a new command. It allows for more hybrid and interactive workflows in the terminal while still remaining centered around the command line.

Extraterm does these features using extra (custom) vt escape codes. ConPty should allow me to extend these features to Windows too.

joeyaiello8 anni fa

Ooooh yeah, that sounds awesome! Going to share this with some people on our team (lots of folks love and use Hyper already, but we're always looking for new stuff to play with).

I would highly recommend you check out the excellent HistoryPx module[1]. Among (many) other things, it supports automatically saving the most recently emitted output to a magic `$__` variable. Theoretically, you could save a lot further back, but you may start to run into memory constraints (turns out .NET objects are a little heavier than text... ;) )

[1]: https://github.com/KirkMunro/HistoryPx

cryptonector8 anni fa

The ConPTY is not just about compatibility with nix. It's about proper remoting of consoles. Unix got that right / Windows got that wrong, and now Windows will finally get it right too -- that it helps nix compat seems like a happy accident (though obviously they want that too, so not so accidental.

quotemstr8 anni fa

Right. For a long time, the MS remoting philosophy was that applications should be remoted, not text streams. The stance goes all the way back to DCOM. That's why PowerShell remoting looks more like a local PowerShell executing commands on a remote machine than it looks like you just connecting to a PowerShell running elsewhere.

The difference is important, since in the traditional MS model, each program that wants to do the remote thing needs to essentially implement its own client-server setup, albeit with a massive amount of help from various runtimes. Named pipes and central authentication made this approach not quite as horrible as it sounds.

This new API is a departure from this model. It will make it possible to just remote via text streams. Perhaps that's uglier --- everyone knows in-band signaling is fragile. But long experience shows they just remoting the damn text streams is easily the more pragmatic option.

cryptonector8 anni fa

I mean, it could be a packet stream where some packets are control packets, and then the Console API could be used and remoted, but you'd need clients that understand it.

The fragility of in-band signaling in the TTY world is not *nix's fault or anything. VMS had that too, since VMS too had to deal with TTYs. The fact is that a) the system evolved from real, hardware TTYs of the 1970s, b) using a text stream with some in-band signaling doesn't even half suck -- mostly it rocks, and all you have to do for it to rock is get the right $TERM value and not output binary files to the tty.

quotemstr8 anni fa

https://m.facebook.com/notes/daniel-colascione/term-is-termi...

cryptonector8 anni fa

The Windows Console was even more terminally dead. Terminals have their limitations, but honestly, I'll take a terminal, tty/pty, and terminfo, any day over the Windows Console.

caf8 anni fa

You also have to sanitise untrusted data that you want to output.

cryptonector8 anni fa

And remember to use less -R or else pipe to col(1) or whatever.

quotemstr8 anni fa

I want a pipe2(2) flag or an fcntl or something that lets me signal "the other end of this pipe understands ANSI escapes"

JdeBP8 anni fa

First of all you have to define "ANSI escape".

Hints:

* You have the wrong country, on the wrong continent.

* It's not as simple in reality as your first answer will be. (-:

quotemstr8 anni fa

Yeah, I know

caf8 anni fa

I've considered before that you could allow each end of a pipe to set name/value attributes, which can be read by the other end.

cryptonector8 anni fa

And now you're reinventing the Windows Console API... :)

I mean, both approaches have their pluses, but the API approach is only ever going to work well for remoting if it is standardized and interoperable. And the installed base of Unix termcap/terminfo programs is huge, so plain old text-with-in-band-controls is not going away anytime soon.

caf8 anni fa

With things like SIGWINCH / ioctl(TIOCGWINSZ) we already know what kind of API it is, we are just haggling about the price.

loa-in-backup8 anni fa

It's a difference between making mv/move, cp/copy client-server aware and making the shell running these simple utilities client-server aware.

quotemstr8 anni fa

The Windows way would be to mount the remote filesystem and use local mv and cp. Which works fine, since Windows has excellent remote filesystem support. The Unix world only partially began to catch up to CIFS with the advent of NFSv4 and its lease infrastructure and it still isn't as seamless as the Windows stuff.

rogerbinns8 anni fa

Have you given thought on how to solve the "unwanted console" problem? For example if you run a .py file (Python) under Windows then you get a console. That is fine for command line stuff, but beyond annoying if the file displays as a gui. So there are now two Python binaries - python.exe and pythonw.exe. The only difference is the latter ensures no console appears. Also good luck if the script printed a help message since often the console disappears before you even know that happened.

I presume many tools deal with this issue, and do it in different ways. Perhaps it is as simple as making the console itself only appear once there is any output, or a blocking read of input.

zadjii8 anni fa

Technically, any executable that's compiled as a commandline application is going to get a console allocated for it, no matter what on Windows. I don't believe that's something we can fix retroactively unfortunately, that's just a part of how things have to be.

Now, I believe that python could have python.exe compiled as a win32 application, then call AllocateConsole as soon as the script called print() or something. If the app was already running in a console, I believe (don't quote me) that AllocateConsole won't allocate a new console for it, but if it doesn't yet have a console it'll spawn one.

rossy8 anni fa

I think the behaviour of cmd.exe is part of the problem here. When an interactive cmd.exe launches a console-subsystem app, it waits for the process to finish before showing the prompt again, but when it launches a GUI-subsystem app, cmd.exe writes the prompt again immediately, so even if the new process calls AttachConsole(ATTACH_PARENT_PROCESS) before it tries to write to the console, it will write over cmd.exe's prompt, which makes a poor user experience.

So, if someone wants to make a "dual-mode" app that works as a win32-subsystem app when launched from Explorer and a console-subsystem app when launched from a console, they have to choose between two bad options. They can make their app a console-subsystem app, which means a console will always briefly appear on screen when the app is started (no matter how quickly the app calls FreeConsole(),) or they can make their app a GUI-subsystem app (that opportunistically calls AttachConsole(),) which behaves sub-optimally in cmd.exe.

Maybe the solution is to add a flag (in the .manifest file?) that makes the console initially hidden for a console-subsystem app. That would prevent the brief appearance of a console window when launching a console-subsystem app from Explorer. Then there would be no need for pythonw.exe and python.exe could show the console window only after a message is printed.

exikyut8 anni fa

Hmmm.

MSDN doesn't really say much about AllocConsole(), if that's the right function: https://docs.microsoft.com/en-us/windows/console/allocconsol...

If AllocConsole does behave in the way you say (which I understand it may{, not}), then the documentation sorely needs updating, because right now that bit of functionality (if it is there) is rather implicit.

It would be really cool to effectively deprecate the current console functionality and make it relatively straightforward to use the PTY API going forward, adding the bits people need to support use cases like this (allocating a console when/as it's needed).

Perhaps Visual Studio could introduce a new template for commandline applications that targets the PTY, and put "(Recommended)" next to that one? :D

JdeBP8 anni fa

> It would be really cool to effectively deprecate the current console functionality and make it relatively straightforward to use the PTY API going forward,

Please no!

The console subsystem in Windows is what terminal I/O evolved into, in the 1980s. Going back to terminal I/O is a massively retrograde step.

In any case, for addressing the problem at hand removing the console API is not the answer. rossy has explained what the problems actually are, which relate to the command interpeter waiting for processes to terminate and whether Win32 program image files are marked as GUI or console.

Personally I have always regarded that behaviour of Microsoft's command interpreter as a flaw, not a feature. I've always turned it off in JP Software command interpreters, which make it configurable. I didn't implement it in my command interpreter. However, I do appreciate that Microsoft's strong commitment to backwards compatibility hampers what can be done.

* https://jpsoft.com./help/waiting.htm

exikyut8 anni fa

Hmmm. Good point. I can't say I think the PTY architecture is actually fun or anything, it's brain-numbing. Now I realize what I was asking I disagree too, hah!

IMO, the "real" answer - the viable one :) - is a redesigned console model that supersets/encapsulates what already exists in such a way that things remain backward compatible but incorporate features that allow for progressive enhancement. The question is in how to actually build that out; what to start with, what to do when and where, etc.

Probably the most important thing I'd start with is having each console be like an independent terminal server: make it so anything can watch all the stdio streams of everything attached to the PTY, make it possible to introspect the resulting terminal stream, etc. Then the TTY itself could be queried to get the character cell grid, query individual characters, etc. And also make it possible to change arbitrary PTY+TTY settings [out from under whatever's using a given console] as well.

By "anything can watch" I mean that there would be an actual "terminal server" somewhere, likely in a process that owned a bunch of ptys, and this would have an IPC API to do monitoring and so forth. Obviously security and permissions would need to be factored in.

But this would roughly take the best from the UNIX side (line disciplines are kind of cool, having the PTY architecture Just Work with RS232 is... I understand the history, but it makes for an interesting current-day status quo, IMO), and then combine this with the best of the Win32 side (reading what's on the screen!!!!! Yes please!!).

I'm not sure how to build something sane that could incorporate graphics though. ReGIS and Sixel are... no. 8-bit cleanness is an unfortunately-probable requirement for portability (at least UTF-8 can be shooed away in broken environments with a LC_ALL=C), but base64 encoding is also equally no. Referencing files (what Enlightenment's Terminology does) is a nononono. w3m-image's approach of taking over the X11 drawable associated with the terminal window is awesome in its hilarious terribleness. The best I can think of is a library that all image/UI operations would be delegated to, which would do some escape-sequence dances with the terminal (and whatever proxies were in the way?) to detect capabilities and either use out-of-channel communications (???) to send the image data over rapidly, or alternatively 8-bit or 7-bit encode the image data into the TTY stream (worst case scenario).

This is a bit hazy/sketchily laid out, but it's something I've been thinking about for several years. When I started out pondering all of this stuff circa 2006 I was most definitely all over the place :) I'm a bit better now but I still have a lot of unresolved ideas/things. I'm trying to build a from-scratch UX that provides a more flexible model to using terminals and browsing the web, but in a way that's backward-compatible and not "different to the point of being boring".

I've (very slowly...) come to understand that slow and progressive enhancement is the only viable path forward (that people will adopt), so I'm trying to understand the best way to do that.

JdeBP8 anni fa

Actually, extending it in a native Win32 way in line with the existing console model would have been quite a simple thing.

There are already CONIN and CONOUT objects. One simply needs to make the latter a synchronization object, the former already being waitable. The console screen buffer maintains a dirty rectangle of cells that have been dirtied by any output operation, be it high-level or low-level output. It becomes signalled when that dirty rectangle is not zero-sized, the cursor is moved, or the screen buffer is resized. And there's a new GetConsoleScreenBufferDirtyRect() call that atomically retrieves the current rectangle and resets it to zero.

With that, capturing console I/O is simply a matter of waiting for the console output buffer handle along with all of the other handles that one is waiting for, getting size/cursor info and clearing the dirty rectangle with GetConsoleCursorInfo()/GetConsoleScreenBufferInfo()/GetConsoleScreenBufferDirtyRect(), and reading the new cell values of the dirty rectangle with ReadConsoleOutput().

cryptonector8 anni fa

Why... can't you have I/O redirection to the null device and have it understand (and throw away) Console API messages? You might as well also add some pseudo-device to convert Console API messages into Unix-style text streams (with or without metadata converted to terminal control sequences), so that one could redirect console programs' output to files / pipes.

When the user's (programmer's) intent is to run a program with no console window, then that's what they should get: no console window.

int_19h8 anni fa

If I remember correctly, this approach breaks if stdin/out is redirected.

There's also some funky stuff about explicit AllocConsole-allocated consoles; for example, when you attach a native debugger, all output from such console is automatically redirected to that debugger (i.e. the VS Output window or similar). This is very annoying in practice.

theclaw8 anni fa

I'm unclear as to whether I'll be able to pipe binary data between a classic console application and a ConPTY application due to the VT translation and rendering components in ConHost.

So, for example if I was to pipe into 7z.exe, a classic console app, using something like "type mybinaryfile.bin | 7z.exe a -si c:\temp\myarchive.7z" from a ConPTY console, would the VT translation affect the piped stream?

zadjii8 anni fa

Nope! We're only rendering the effect of any attached processes to the VT on the conpty side of things. On the client side (where cmd, type, 7z.exe are all running), they're going to keep working just the same as they always have. They're all running on the "slave" side of conpty, while the emitted VT is coming from the "master" side of the conpty.

berbec8 anni fa

Will there be the ability to disown, background, nohup processes and close the console, leaving the commands running?

zadjii8 anni fa

Those sound like they'll be more like the responsibility of the terminal emulator, unfortunately.

Windows console applications aren't really able to live without being attached to a console. Now, a terminal might be able to implement those features...

actually now you've got me thinking. I'll play around with that idea. Definitely non-committal, but it might be possible in the future.

JdeBP8 anni fa

The sad thing is that this was all already implemented and done in Microsoft's second POSIX subsystem for Windows NT. It provided signals and process groups support for job control shells. It had a full control sequence interpreter for output and control sequence generator for extended keys. There were termcap/terminfo database records that people had added to other operating systems. It had a line discipline with "canonical" and "raw" modes. It had pseudo-terminals, with both BSD and System 5 access semantics.

* https://technet.microsoft.com/en-gb/library/bb497016.aspx

* https://technet.microsoft.com/en-gb/library/bb463219.aspx

* https://news.ycombinator.com/item?id=12866843

* http://jdebp.info./FGA/interix-terminal-type.html

And Microsoft owns it.

quotemstr8 anni fa

I wish people would stop pointing at the Linux subsystem and using it as an example of the stuff Windows in general could do. The Linux subsystem is a subsystem. In NT terminology, that makes it practically a container. It's disconnected from the rest of the system. It can interact with the win32 world in only a few constrained ways. You can't deliver SIGWINCH to a win32 process!

JdeBP8 anni fa

Actually, the second POSIX subsystem could deliver signals to Win32 processes. Who pointed at the Linux subsystem, by the way?

* https://news.ycombinator.com/item?id=11416392

cryptonector8 anni fa

I mean, the same is true on Unix. If an app is in the background and wants to write to (or read from) the terminal, it gets SIGTTOU (SIGTTIN) sent to it immediately. It might have to be impossible on Windows to ignore SIGTTOU/SIGTTIN/SIGTSTOP, but I think that's just fine.

Mind you, ptys + tmux/similar is certainly very good, and if that's all we'll get that's still way way better than the current state of affairs, but if that's all that will be possible it should at least be possible to pause the console's output (and flow-control the console application).

jamesgeck08 anni fa

This is awesome. Thank you for all ya'll's hard work!

quotemstr8 anni fa

Can conhost still do anything that users of the new API can't?

zadjii8 anni fa

Excellent question! There are a few limitations that we have to place on the ConPty to make it work quite right. Primarily, client apps running attached to the conpty will not be able to have separate viewport and buffer sizes. On *nix, the entire "console buffer" is just the size of the window, but on Windows, technically, the buffer is much larger than the window. (as an example, when you open up a command prompt, there's a giant empty space at the bottom if you scroll down). Fortunately, we haven't came across any apps that _need_ the buffer to be a different size than the viewport, and it's a technically valid console configuration, so apps should have been able to support it before.

Input is also tricky - VT doesn't let you express input with as much fidelity as a console app might be expecting, though this we're working on a solution for :)

quotemstr8 anni fa

Now Windows just needs to ship with a decent pager. :-)

Programmatic access to scroll back is useful for a few things. For example, back when I was on Windows Phone, I wrote a compiler wrapper that would scroll back to the first error message.

It'd be nice for the POSIX terminal world to standardize on similar scrollback access. I know the zsh people would love it.

caf8 anni fa

*nix consoles typically have two buffers - many full-screen temrinal applications switch to the "alternate screen" on start and back to the principal screen on exit. That's why when you exit vim(1), you see the terminal state back as it was before you started it. Will ConPty support this?

zadjii8 anni fa

Yep! It'll probably act a little different than you'd expect - the pseudoconsole itself will switch between the primary and alt buffers, leaving the terminal in the main buffer always. We'll "render" the contents of the alt buffer to the terminal, then when the client app switches back to the main buffer, conpty will re-render the contents of the main buffer to the terminal.

It's not the most elegant solution, but we're still very early in on this project. We still have lots of improvements to be made to the infrastructure and translation, and even as I type this up, I'm thinking there's probably a better way of handling alt buffers.

caf8 anni fa

That's pretty much how it works if you have a screen/tmux type program in your path as well, so that makes sense.

JdeBP8 anni fa

The Windows console subsystem itself has supported multiple output buffers from the start.

* https://docs.microsoft.com/en-gb/windows/console/console-scr...

cryptonector8 anni fa

I hate the alternate screen with a passion.

llampx8 anni fa

Will this replace Command Prompt or Powershell? Or is this more the back-end for these apps? I believe that we are heading for console overload (in a good way?!) with Command Prompt and Powershell installed on every computer and Debian/Ubuntu available on the MS store.

zadjii8 anni fa

First off: command prompt (cmd.exe) and powershell are commandline client applications. They are shells just the same as bash is.

All commandline clients run attached to a console server, and that server is conhost.exe. Conhost is responsible not only for being the console server, but drawing the actual terminal window these apps run in. So when you alunch cmd or powershell, what you're seeing is conhost.exe "hosting" these console applications.

What we're exposing here is the "master side" of conhost, which will the other applications act as Terminals, like how there is gnome-terminal, xterm, terminator, etc on linux.

paulie_a8 anni fa

Is there any chance cmd and powershell will improve from a user interface perspective? And perhaps become usable? Cmd has been garbage since it's inception.

zadjii8 anni fa

cmd is parked for pretty much everything except for major issues. It's a scary codebase that has a LOT of code that's dependent on it, and we can't really add any new features there without the possibility of breaking someone.

This feature is mostly focused on the other end of the communication, on being able to create new Terminal windows to run shells inside of them.

elFarto8 anni fa

Any chance that you could just fork it? Create cmd2.exe, hell you could even open-source it.

oblio8 anni fa

Is the pay-off big enough? Who knows what garbage code they might want to hide in there, for both technical reasons (including security) or even legal ones.

llampx8 anni fa

Isn't that basically Powershell?

paulie_a8 anni fa

Is there a chance this will be connected to a functional shell interface. I get your point that cmd cannot be upgraded because of legacy issues and that is understandable and unfortunate, but windows needs a proper shell. This is obviously a great start for one side of the equation. But until there is a decent terminal app, windows will continue to be a nonstarter.

zadjii8 anni fa

It sounds like you're asking for two different things here:

cmd.exe is a shell, and that's the guy that's parked.

conhost.exe is a terminal, and that's under active development, though it's slower than something like VsCode, because we can't just go adding features as we see fit, we have a LOT of back compat we still need to support.

Fortunately, conpty will allow for the creation of new terminal applications on Windows. If you're looking for a better shell experience on windows, I can point you to powershell or even [yori](http://www.malsmith.net/yori/), which looks pretty cool

cryptonector8 anni fa

I think u/paulie_a is just asking for a better shell. If cmd can't be made better, then make a new one.

cwyers8 anni fa

That's Powershell.

paulie_a8 anni fa

Exactly, I understand there are different underlying concepts and systems to the front end and what it interacts with. It just seem incredible that windows is basically stuck with a windows 95 interface for a shell.

nailer8 anni fa

Powershell is open source, the 6.1 preview 4 is nice and fast, and you get real objects with keys rather than scraping for regexs all the time like bash.

AnIdiotOnTheNet8 anni fa

Windows has a proper shell, it's called PowerShell and it is by far the most discoverable and consistent shell that exists.

wtallis8 anni fa

It's also the most absurdly verbose shell that exists, which means it sucks as a shell even if it's a half-decent scripting language.

AnIdiotOnTheNet8 anni fa

(1)

paulie_a8 anni fa

I've attempted to use powershell numerous times, it's a piece of shit compared to zsh.

tokyodude8 anni fa

some other HN thread recommended cmder. I haven't touched cmd since

http://cmder.net/

zadjii8 anni fa

Ah but see, now you're conflating two different things:

cmd.exe is a shell

conhost and cmder are terminals.

I believe cmder can come with git bash as well, which is also a shell.

The confusion comes from when you launch cmd, the window that appears by default is conhost, with cmd running attached to it. When you launch cmder, it's also running attached to cmd.

nailer8 anni fa

The back end. Basically people like ConEmu and Hyper and Terminus have been having to use various unreliable hacks for ages because there was no real console API. Now there is one.

nailer8 anni fa

Zadji I love your work. Which build is this going to land in? Do you know if any of the third party console apps using the new API yet?

zadjii8 anni fa

It's already available in current insider's builds, and will be landing officially in the next available Windows release some time later this year.

We're still working with ConEmu, VsCode, and OpenSSH to get them all over to the new API, with varying levels of adoption in the next few months likely.

Currently, WSL is also using the same functionality, if you open a WSL distro and run any Windows executables (eg `cmd.exe`), they'll run attached to a conpty. I use this as my daily driver.

quotemstr8 anni fa

You might want to talk to the Cygwin people to get their pty layer to use the new virtual console.

hoppelhase8 anni fa

Maybe the guys at alacritty are interested in this as well.

https://github.com/jwilm/alacritty

nailer8 anni fa

They are: https://github.com/jwilm/alacritty/issues/28

zadjii8 anni fa

That is an excellent suggestion, I'll get our PM to reach out :)

[deleted]8 anni fa

cryptonector8 anni fa

Out of curiosity, are you backporting onto Windows 10, or is Windows 10 the only release vehicle for everything in master? If the latter, how are you releasing piecemeal?

zadjii8 anni fa

Pretty much any new features we release are only available on new Windows 10 releases. Unless there's a gigantic demand for a feature, or business impact, we're not really capable on our team of backporting anything.

cryptonector8 anni fa

What I meant is: are you building a Windows 11, or is Windows 10 the only release vehicle? I'm trying to imagine you pushing to a master branch to integrate into what becomes the insider builds, and then... either those become a future release of Windows 10, or you backport onto a Windows 10 branch. Neither option would be very appealing to me if it was up to me.

zadjii8 anni fa

Windows 10 is the "last Windows ever". There's a pretty wide tree of branches under "winmain" that teams contribute to, and at each layer more branches are merged together, until they finally hit winmain, and then weekly they tag (more or less) a build from winmain as the Insider's build for the week.

It's actually a lot more elegant than you'd expect it to be for a project with as many developers as Windows has

cryptonector8 anni fa

Thanks. Got it. Having worked on Solaris, this sort of thing is very interesting to me.

asveikau8 anni fa

While we're talking Unixisms, Windows needs a dup2(2). That is, given a HANDLE, you should be able to swap out its backing kernel data structure with that of another HANDLE.

Without this, I/O redirection is slightly broken. Last I checked you can't change where stderr goes after the process starts, for example. [SetStdHandle doesn't do it at the right layer.]

hoppelhase8 anni fa

I always liked the Console API where you can set the color of the text without actually changing the text that is written to Stdout. No issues when piping the output somewhere else. No need to check whether the output is getting piped.

zadjii8 anni fa

That'll work just the same as it always has :) Existing commandline applications won't be affected by this feature, but it will open the doors for an entirely new class of applications.

hoppelhase8 anni fa

The existing Console API won't be extended with features of VT codes, or will it?

zadjii8 anni fa

Oh we added support for VT sequences in commandline apps years ago - case in point, WSL.

see [this docs page](https://docs.microsoft.com/en-us/windows/console/console-vir...) for a (surprisingly incomplete) list of VT sequences we support, and how to use them.

quotemstr8 anni fa

Finally! I've been waiting ten years or so for this API. It's about time that alternative terminal emulation becomes possible on Windows.

hoppelhase8 anni fa

If I use the System.Process API in .NET and redirect the Stdin/Stdout to a stream inside my application, does the framework spawn an invisible console and scrape the output? Or does this work differently? I always did it that way and thought the 3rd party terminal emulators also do that. Why do these emulators have to do it differently?

quotemstr8 anni fa

That API is using pipes.

exikyut8 anni fa

Wow. I remember the photo miniksa posted to GitHub when this was in process:

https://github.com/Microsoft/WSL/issues/111#issuecomment-238...

Awesome to see it's finally up and running! \o/

borekb8 anni fa

I currently use ConEmu + zsh via MSYS2 as my preferred shell on Windows. I need to run many interactive programs like `python`, `node` etc. via winpty, e.g.:

``` alias node='winpty node.cmd' ```

With the new ConPTY, will I be able to run native Windows programs directly? If so, that would be huge, winpty (while I'm really thankful it exists) is a PITA in practice, see e.g. https://github.com/Microsoft/vscode/issues/45693.

linuxlizard8 anni fa

This is very exciting. I'm looking forward to seeing where it goes.

mschuster918 anni fa

Is there any way to get this backported to Windows 7 - or run a W7 userland on top of a W10 kernel? I'm actually serious about this one, I can't stand this semi-"mobile-first", flat UI of newer Windows generations, and the privacy invasions and ads are other hard blockers for me - but that WSL layer or the new console subsystem seem to be pretty nice features.

red75prime8 anni fa

I hope control-S (XOFF) is disabled by default.

voltagex_8 anni fa

Wow, there might be able to be a proper ncurses port now!

zadjii8 anni fa

Functionally, we support all of the VT sequences you'd need to make ncurses work resonably well on windows for a few releases now (ever since WSL was introduced). If you could build an ncurses that assumed TERM=xterm-256color, then you might be able to get it to work on windows.

lambdas8 anni fa

No termcaps which ncurses depends on though so I don't think so

docode8 anni fa

Where can we try a .NET solution with this ConPty?

mobilehnuser8 anni fa

Thanks to WSL and this, I'm very hopeful that my next development laptop can be a windows device

OnlyRepliesToBS8 anni fa

218 anni fa

Does this mean that it will now be easy to port terminator to windows?

zadjii8 anni fa

I sure hope so! I can run cmd.exe inside gnome-terminal running on WSL right now - granted, WSL is doing some magic to make that all work, but it should still work for terminator to do it to.

nasoieu8 anni fa

Something looks very eerie in that Admiral Grace Hopper picture. Is it shopped?

kpil8 anni fa

-- Those who don't understand Unix are condemned to reinvent it, poorly. Henry Spencer

oblio8 anni fa

This is such a cliché.

Do you think that the people who implemented the Windows Console, especially the people working on Windows NT, did not know about Unix? People try different approaches, sometimes they don't work out.

And it's not like Unix is the Word of God, anyway, it has plenty of flaws.

(Yeah, after a long time on internet forums I get kind of touchy after someone copy-pastes the same old and tired line.)

gaius8 anni fa

Microsoft understood Unix very well, Xenix was a product of theirs in the 80’s

kpil8 anni fa

Since it took them 20 years to make a half-decent shell and 30 years how to figure out how stdout should work: no they had no clue.

Maybe they knew how a kernel should work though, but weren't the NT guys old VMS guys? That's a totally un-unixy OS actually.

oblio8 anni fa

1. They didn't care about command line tools, in many ways (usability) they're a huge regression. After all, it was in the name of the product: Windows.

2. Who says that in-band communication like Unix is doing is necessarily better? See pastejacking and other shenanigans.

c487bd628 anni fa

It's all about pushing Azure and dedicating resources to said goals

JdeBP8 anni fa

The problem is that in this case it's not understanding Microsoft's own prior software that condemns one to reinvent it. Microsoft's second POSIX subsystem for Windows NT, a.k.a. Interix, had all of this.

AnIdiotOnTheNet8 anni fa

Those who did understand unix reinvented it pretty well, called it Plan 9, and were more or less completely ignored by the unix twonks of the world.

Sometimes people just insist on using stuff that sucks.

badminton18 anni fa

    HRESULT WINAPI ResizePseudoConsole(_In_ HPCON hPC, _In_ COORD size);

If Microsoft is in the mood to fix old problems, right ^there you've got another old problem: its bizarre API that is different to everything else. Designed that way to lock everyone into their OS.

In 2018 nobody has the time to learn this. Just use a cross-platform API and if it doesn't run on Windows then just don't run Windows.

As a developer, using Windows for development is against your own best interest. If you like to be treated as a dog that is not allowed inside the house, use Windows.

Jasper_8 anni fa

What cross-platform API would that be? ioctl(tty_fd, TIOCSWINSZ, &size); ? How does the user get the TTY FD? open("/dev/tty0")? Or should they implement SYSV compatibility and use "/dev/vt0"? Or perhaps follow FreeBSD, which has "/dev/ttyv0"?

zadjii8 anni fa

Honestly, learning Windows is just like learning another programming language. This API is designed to be the Windows equivalent of a unix API - of course it's not going to be the exact same thing, but functionally it does the same thing.