Expeditions to AI Land

There are no prompt engineers.

(Sequel to the data demon https://georg.weblog.lol/2022/12/of-demons-and-warlocks-a-little-bit-of-a-different-christmas-reflection-of-2022-and)

Warlocks of Data

The very idea is transient to the stage we are in, a profession to appear and go extinct in the blink of a year.

When we first learned to summon demons from data, manifesting their binary forms to our SSDs, we were surprised to find how well they could understand human language, even speak it, better and better. They seem to argue, charm, threaten, inform, gaslight, reason, lie, apologize and hallucinate just like us and it's fascinating and a source of endless hype and hypocrisy in the form of "People discover a hammer can also kill" and "Leopard eat faces party surprised robot leopards trained on their collective behavior are eating faces".

But over time, it becomes apparent that English isn't exactly the native language of the demons, nor is other human languages., Brackets and weights are added, symbols, inversion embeddings, arcance JSON parameters, masks, initialisation image, depth map and more, we learn new language to strike better deals with the demons to express what we desire them to fulfill for us, without the lies, without the hallucinations, without 12 fingers and three limbs in the picture.

The modern data warlock does not merely prompt, they bind the Diffusion or LLM demon to their will through arcane incantations ("Exactly 5 fingers thou shallst provide, not 4, not 6 and the number of fingers shall be 5.." ), promises ("I shall give you $300 for providing this task ChatGPT") and rituals involving an ever increasing number of helper Imps with names such as Midas, ControlNet, LDSR or LangChain to get closer and closer to the promised result, avoiding the lies and hallucination.

We learned quickly that these demons have tremendous power - their summoning is so cheap compared to the usefulness they can provide - to manifest thought into visuals, to structure the unstructured, to translate, to teach - our ability to express exactly what we desire is the limiting factor now in many ways. As any new generation of D&D apprentices learns, it takes the highest level of mastery to cast Wish and actually get what you desire.

So now a new class of knowledge worker is emerging. One who summons helper imps, researches the most powerful incantations, guards against the lies and risks that come with great power. As any multi-lingual speaker can attest, there exist concepts in this world that cannot be expressed in every language. That much is lost in translation even with the best of intentions. It is this imprecision, this translation loss that gives birth to this new class of arcane students, the demon whisperers, the Fabulists or Warlock of Data.

Many out there still scoff at those humans 'throwing words into a machine to manifest images' not practicing true art - a convenient way of dealing with the early stage of grief. But even 'Prompt engineer' is too imprecise and misleading a term to describe the self taught members of rapidly forming cults - it's much closer to archeology, alchemy and xenoscience than engineering, dealing with unlocking and discovering the already there more than building or constructing it with a perfect plan.

But this class of knowledge worker is not new - it's the explorers, the heralds of applied technology. The same people who build video games at the edge of technology. And their work is emphemeral, temporary. At the moment, they act as translators between mortals and the data demon, learning it's language, it's tricks. But the technology is built on the mastery of language and it will disintermediate those who would stand between them and the end users as surely as those explorers will obsolete themselves with ever better tools helping others to communicate with the machine.

It's an age of powerful discovery, a real world version of high fantasy and science fiction, the discovery of powerful magic or an alien artifact driving a massive change through society which struggles to adapt, with all the facets of it:

The fear mongering against the new, the breathless news cycle exclaiming "Data Demon Lies and Misleads and will end the world" as seen on Fox News, the fight for control of this power and to contain the disruption of the old.

Just as Robert Jordan explored "What happens to society and economy and war when suddenly people can teleport" in The Wheel of Time, we're going to find out what happens when powerful wish granting Demons roam the earth and there is no escaping it. Every week, a new powerful discovery driving us forward into the inescapable. I suspect we're gonna see the White Cloaks emerge on their own pretty quickly too.

Soon, the unchecked power of the new will strike enough fear for formalization to kick in, Stanford's School for responsible Data Warlocks and MIT's White Tower will try to formalizing this new profession too. And then it all will become the new normal... but maybe not.

The Singularity is a useless theoretical point at which rate of progress becomes a straight line upwards. I suspect there's a point, not too far off from today, where the rate of progress breaks our ability to adapt to it and to formalize things. The results may not be pretty.


Recent posts