When I moved to Japan in 1983, not much was computerized and I was able to use my preferred short version of my name--eight letters or five katakana, surname last in both cases--for nearly everything. But over the years more and more places started requiring that names be input into labeled fields, which would reverse the given-name surname order. And many places started to insist on the same name as on my passport--nineteen letters, eleven or twelve katakana, depending on how I wrote it. Many input forms don't allow so many characters, none that I have encountered distinguish first from middle names, and there's usually no way to link that long name to the short name that I have been using for work and in daily life for more than forty years. It's a constant annoyance, not just for me but for other people.
Now that Fable is back, I had it create two more browser-based music generators. For the first, I had Fable in chat write a full prompt based on the same metaprompt as above. Here is the prompt it wrote:
I can’t say that one is better than the other; that would take a lot more tests, and the judgments would be pretty subjective in any case.
Both were completely one-shot. Last year, when I was doing similar tests with various models, the synthesizers rarely worked right on the first shot, and I would have to do some back-and-forth to get them functioning. This time, Fable was able to open the files in Chrome itself, view and adjust the page layout, and monitor the browser events for errors. Each time it made some adjustments to the file. The only thing it wasn’t able to do was listen to and assess the sounds produced.
[1] “Write a browser-based synthesizer for me. The synthesizer should automatically create interesting polyphonic music in which the various voices play off against each other in both harmony and contrast. The controls will affect the tone, rhythmic patterns, number of voices, complexity and randomness of the melodies, and other features. The controlled features should be original—not just standard synthesizer functions—and encourage creative explorations even by naive users.”
That occurred to me, too, when I posted that metaprompt above. But that was only the second or third prompt I gave to Fable during the couple of days I had access to it, and I used the metaprompting strategy that had worked well with earlier models. If and when I get access to Fable again, I will try giving it a similar short prompt directly.
I should mention that Fable also did an impressive job on a couple of major project-redesign tasks I gave it. Those aren't things I can share here, though.
Just for reference, here is the metaprompt I first gave to Opus:
“I want to ask Claude Code to write a browser-based synthesizer for me. Please prepare a prompt for it that I can give to it for it to write the synthesizer. The synthesizer should automatically create interesting polyphonic music in which the various voices play off against each other in both harmony and contrast. The controls will affect the tone, rhythmic patterns, number of voices, complexity and randomness of the melodies, and other features. The controlled features should be original—not just standard synthesizer functions—and encourage creative explorations even by naive users. So write a prompt that I can give to Claude Code to create that synthesizer.”
I then gave the prompt produced by Opus to Fable in Claude Code.
My Fable example is not nearly as cool but still (to me) impressive.
Last year, I would occasionally test the latest models by vibe-coding in-browser music generators using only HTML, CSS, and JS. Here’s one made in July by Gemini:
As it happens, the current number-two article on HN is about a similar consequence of Chinese export controls--a car manufacturer developing electric motors that do not use rare earths:
As a former freelance translator (1986 to 2005, Japanese to English), I have much sympathy for the writer. But I wouldn’t be so confident that AI cannot do professional-level translation.
She writes: “I adapt, I localize, and I find the best way to convey the original message so it makes sense and feels natural. I research terminology. I make sure it’s consistent throughout.”
I’m sure she has other important insights into what enables her to do her job well. The problem is whether or not such insights can be incorporated into an AI-driven translation system, too.
Since early this year, I have been experimenting with a variety of agentic systems for language-related tasks, including dictionary-writing, research on topics in the philosophy of language, essay-writing, and translation. Other than the dictionary [1], I am keeping the results private, so they haven’t been evaluated by others. But my personal assessment is that agentic systems given suitable high-level guidance can be very good at such tasks now.
If I were still freelancing and I had a large translation job to do for a client, here is the outline of the prompt I would give to Claude to get it started:
“Use this private GitHub repository to build a system for translating [genre of text] from [Language1] to [Language2]. The directory samples/ contains examples of the type of document to be translated, high-quality human translations of those documents, and texts in [Language2] that are in writing styles that I believe to be appropriate for this genre of translation. The file guidelines.md contains my general instructions about the needs of my client and my preferences for how you should translate texts along various axes (natural vs. literal, informal vs. formal, preferred dialect in [Language2], consistency vs. variety in terminology translation, etc.). Begin building (1) a knowledge wiki for this project using Karpathy’s LLM-wiki framework and (2) a system inspired by Karpathy’s Autoresearch, AutoResearchClaw, etc. for testing and recursively improving both the functioning of the system and the quality of the translations. For the actual translation, editing, checking, etc., use not only your own ability and the knowledge assembled in (1) but also outsource such tasks to other frontier models through OpenRouter, and use adversarial evaluations among those models and yourself to check and recursively improve the system design, the prompt-writing for other models, and any translations created by the system. My OpenRouter API key is available in this environment. You may spend up to $xx per day in API calls until this project is ready to do real translations; before beginning a real job, give me an estimate for how much the API calls will cost for that job. The initial build-out of this project will take many sessions, so write a prompt called resume-prompt.md that I can point you to at the start of a scheduled Routine to have you work on this. Commit and squash-merge to main at the end of each session. I will be checking in occasionally to view your progress and to ask you to run translation tests, and I will offer guidance then on how to improve the pipeline further and make the translations closer to what my client needs. If you have any questions before you begin, please ask me.”
My first came in late 2016, when Google Translate switched from statistical machine translation to a neural-network-based system. I had worked as a Japanese-English translator and lexicographer for two decades, and I had been testing various machine-translation services over the years. For translation between Japanese and English, at least, they were uniformly terrible: the output for genuine texts was mostly incomprehensible and could not be used for any real-life applications. The neural Google Translate, while still far from perfect, was suddenly useful for some purposes.
But the neural models were still not translating meaning, which is the whole point of translation. I devised a variety of tests to see if GT could identify the meaning of ambiguous words from the context, and it couldn’t. One example I would show people was the sentences “I was born in 1998, and my sister was born in 1999” and “I was born in 1999, and my sister was born in 1998” translated into Japanese. Japanese uses different words for older and younger siblings, but GT translated “my sister” with the same word in both sentences. It was easy to come up with other examples where GT would fail, such as when the meaning of a word could only be determined based on context in a previous sentence; at that time, GT seemed to be translating sentence-by-sentence, with no consideration of what came before or after. I kept waiting to see whether computers would ever be able to handle meaning when translating, and for years thereafter there was little progress.
A minor shock came in mid-2022, when DALL-E 2 was released. Its ability to create images from natural-language prompts suggested that something deeper was going on than just statistical correlations. But I couldn’t see yet what the useful applications might be.
My biggest “oh shit” moment came with ChatGPT in late 2022. While the initial release didn’t translate Japanese well (I seem to recall that there were character-encoding issues), I ran various tests to see if it could, for example, identify the antecedents of pronouns and the meanings of polysemous words in English based on the context. It did really well. Last December, I gave a talk at a university in Tokyo in which I showed some examples done with the 2022-era GPT-3.5. They appear in slides 4 to 8 of the following:
The focus of the course wasn't writing criticism or particular authors. It was on how writing teachers can teach current and future researchers how to write research papers. A narrow field, perhaps, but at the research university where I worked there was demand for such writing courses, and my class looked at how such classes can be taught at a meta level. Many of the students themselves became writing teachers.
This was all before LLMs. The assumptions behind my class have since been upended, now that easily available chatbots can produce academic writing that, at least at the structural level, is perfectly fine.
> There is danger in evaluating for language patterns over its content
I agree, but it’s worth noting that that has been done since long before LLMs. Fifteen years ago, I used to teach a graduate course on academic writing pedagogy. The students and I would read research papers on the teaching of academic writing; we also analyzed textbooks and course syllabuses to get an idea about what was actually being done in classrooms. While phrases like “critical thinking” did come up, the overall focus was clearly on language patterns: sentence and paragraph structure, the use of transition words, vocabulary for hedging and boosting (i.e., making assertions seem weaker or stronger), etc.
In a university context, it can be very difficult to evaluate student writing based on its content. In humanities-focused and creative writing, what the student decides to say can be seen as an extension of the student’s personality, identity, and individual experience; if a teacher evaluates the content, including the reasoning, it can seem that the teacher is evaluating the student as a person. And if the students are in the sciences, especially at the graduate level, the writing teacher often won’t even understand what the students write because it is too technical. Teaching and evaluating language patterns, not content, is often the only option.
Since January, I’ve been having Claude build a static Japanese-English dictionary in which all of the kanji and jukugo can be displayed either with or without furigana:
I haven’t spotted any mistakes in the furigana myself, though there must be some. I have a scheduled routine running multiple times a day to have Claude check and polish existing entries; it should be correcting most of whatever furigana mistakes might be in the data. At some point, I will set up an agent to use a different LLM to run a similar set of checks to try to reduce the error rate even more.
As you note, the readings of Japanese words depend on the context, so producing accurate furigana cannot be done entirely programmatically. Sentences must be interpreted semantically.
I am releasing all of the dictionary data into the public domain, and anyone is free to fork it or adapt it however they like:
I was curious whether the distribution would vary from model to model. Here are the results for 1,000 queries each for smaller models in the Gemini, Mistral, Qwen, DeepSeek, and GLM series:
I really like working from home myself, but I am starting to suspect that the organizations that thrive in the years ahead will be those with lots of face-to-face interaction. If most people are working remotely and having AI agents communicate on their behalf, trust won’t form, consensus-building and decision-making will suffer, and employee salaries will start to seem like a waste of money.
His LLM-wiki framework has been very useful for me for some personal research and knowledge-building projects I've been working on recently. When I get an idea for a new project, I first give it to Claude together with LLM-wiki.md and have it spend a few sessions compiling knowledge in the wiki before beginning work on the project itself. I schedule further wiki-maintenance sessions for later, too. Over time, the wikis become especially valuable when planning major changes or additions to the projects, as they help to ground both me and Claude with knowledge specific to the project.
Here's an example wiki in a public repository for a dictionary I have been having Claude build for the past few months:
I’m teaching a class at a university in Japan (on AI-related issues, as it happens). I’ve been teaching for more than 40 years, but at 106 registered students this is by far the largest class I have ever taught. AI tools are very helpful for class management, such as keeping track of attendance and homework submissions.
I have to consciously avoid using AI for more cognitive tasks, though. It would be very tempting to have Claude, ChatGPT, or Gemini summarize, classify, and grade the students’ assignments, write individual feedback, prepare my lesson plans, etc. However, I know that my engagement with the material and with the students would suffer. I also want to show the students that they are learning together with me and with each other, not with bots.
I am semiretired and have a light teaching load that gives me plenty of time to prepare for class. I can see that full-time teachers might find it hard to resist the lure of offloading their thinking to AI.
The image produced by the program seemed unbalanced because Japan’s southernmost islands were included even though they are not part of the electrical grid. I used an image editing program to remove the outlines of those islands and shift the main part of the country toward the center.
Side comments:
Not indicated on the map is the fact that Japan’s electrical grid runs at 50 Hz in the eastern and northern parts of the country and 60 Hz in the west:
Presumably it will be possible to adjust that behavior with settings, the system prompt, etc. Not that most users will make such adjustments, though.
I'm currently teaching a class on AI-related issues at a university in Tokyo. Many of the students were surprised when I showed them that they can change the response behavior of chatbots to make them more or less verbose, sycophantic, etc. It shifted the direction of our discussions on the possible impacts of AI on the people who use it.
I did that once about twenty years ago. I was in Seoul for a few days for work, and I had the last day free before my plane out in the evening. Without checking a map or guidebook, I got on the subway, rode a few stops, went up to street level, and wandered around; I repeated this four or five times. Other than one nondescript office district, every area I emerged in was interesting: a wholesale textile market, an upscale residential neighborhood, a lively commercial district. Though I don’t know the names of the places I visited, I still remember them all these years later.
