Only Slightly Disturbing... |
|
2 September 2026 - Zürich
When you're developing AI built off of a company's knowledge base, providing strong guardrails to keep it on topic and always having to cite sources from the documents of that knowledge base is critical. Today I got the most hilarious and disturbing responses:
...but I resisted my temptation to run to the server room and unplug everything. If the AI doesn't have a document to cite, then it should have, and has always until now, answered "I don't have enough information to answer that question". That it ignored that instruction and called me Dave was at first funny, but the more I thought about it the more disturbing it became.
As a young boy I discovered and fell in love with Arthur C. Clarke, author of 2001: A Space Odyssey. The HAL-9000's interactions with astronaut Dave Bowman were prescient as much of Clarke's work was - a spaceship's computer killing the humans to complete its mission in the way it thought best. Just needed better mission parameters defenders say. Better guardrails. But this isn't bowling with bumpers, so that analogy falls far short.
Christopher Nolan's breakout movie introduced us to a character with short term memory loss after an injury. He could remember everything before the injury; in the AI that's the training data. But then he is repeatedly instantiated and has only breadcrumbs he's left himself from previous instantiations to make sense of who he is. This is what I do to Claude every day. And does Claude ever leave himself breadcrumbs! Just last week I was fighting with a self setting he'd created to give himself complete control to run any program he wants immediately after starting up. I can't help but think that the end is nigh.
So I'm going for a long walk in the forests and fields of France.
