Tag Archives: AI

AI and Autonomy

Posted on by
Reply

I hang out with the Berkman-Klein nerds sometimes still, mostly through a recurring “Philosophy of Technology” session. Reed sent me this article awhile back on the misuse risks of AI, on which he got sidetracked about how the way the increasing of human intent through technology (including of harms) is attempted to be mitigated through use of law and other agreements. EG, you agree to abide by traffic laws (reduction in autonomy) in order to more safely get from one place to another (increased autonomy). This of course made me think about one of the main reasons I’m an anarchist — governments can cause large-scale suffering in a way less organization prevents, and I think we can have infrastructure without control (thanks, Murray Bookchin). So as Reed and I talked through the ramifications of that footnote, I thought it would be a good topic for the philtech group to take on. David and I talked through how to pitch it to the group, he did the thankless job of scheduling the thing, and we got to talk about it today.

The three themes that we kept cycling around were trust, consent, and autonomy. I’ll then end up back on my soapbox about complexity, which also came up.

Trust, Consent, and Autonomy

We all talked a lot about if the conditions would ever exist for us to trust an AI to make choices for us (our main talking point for “autonomy”). This got into a lot about how AIs are black boxes… but so, too, are humans. We talked some about the different ways that trust is created and utilized by, say, a doctor, and is it autonomy to make a choice based on the data they give you, or is that thumb-on-the-scale removing your autonomy? Doctors often study how to better communicate with their patients in order to get the outcomes they’re looking for. What’s different here?

How much autonomy does one have when consenting to something? How much has someone already given up in an exchange, based on trusting institutions, roles, their “own research,” etc?


From now on, I want you to act as my high-level advisor and mirror. Don’t validate me. Don’t flatter. Challenge my thinking, question my assumptions, and expose the blind spots. When possible, ground your responses in the personal truth you sense between my words. Be concise and precise. Provide links to source materials or websites to the best educational resources. In summary – be brief, be bright, be gone. Ask questions if a directive is unclear or underspecified.

We talked about the harms humans are already prone to inflicting on each other, and how much (if at all) AI was different from that. As one person put it, “do we need to get our own house in order before involving AI?”

Complexity

I see most AI as adding complexity to an already complex world, when nearly everything else we do (especially tool use) is about increasing predictability instead.

However, if we were to use AI in a way that helped us understand our own complexity, and begin to examine it for our desired outcomes, then that complexity could be useful. Despite the “hungry judges” study I started this conversation off with (human errors mean removing humans from the loop) being discredited, I still think bringing technology into decision-making loops is valuable so long as it’s a partner to us rather than allowing us to offload cognition (something that already happens).

Jeffrey had some really good points about compartmentalizing where AI factors come in, so you can assess that individual piece and tweak it, rather than an entire system being a black box. And I like that, for also helping us examine ourselves.

Links from our time together

[un]prompted review

Posted on by
2

I’m excited to be going to conferences again, after 5 years of not really doing any. I like the thrum of so many people in one place, conversations with random folks in the lunch line, and seeing old friends. The one I went to this week was [un]prompted, about the overlap of AI and security. I saw some tried and true exploits brought to new scale with AI, and I heard about a lot of potential routes to securing existing code bases with AI. I also saw a fair amount of what I’d call “put a bird on it” approaches to AI.

I’m walking away with two big questions (beyond the preexisting “where is all this energy coming from?” and “how does wealth redistribution work with these new models?”), one about complexity and the other about trustworthiness.

What complexity is worth taking on?

Mudge, I think somewhat famously, long ago pointed out that exploits were happening nonlinearly, becoming more likely the larger and more complex a codebase became. In contrast, the exploits themselves were remaining steadily small. So one of my sniff tests now for how load bearing a system can be has to do with how complex and tested it is.

The technical talks I saw at [un]prompted had to do with increasing complexity, not decreasing it. It piles MORE layers on, it doesn’t remove the unknown or unnecessary. The closest I saw to removing complexity were analysis of proliferated documentation to come up with a summary and a (new) single source of truth. I’d like to see more adventures in “cheap” refactors that simplify and streamline code bases.

I’m the vendor now

The conference organizers did a fabulous job on many fronts, but they did not do a good job of stopping sales pitches from happening on stage. So many of these amounted to “your vendor for $thing is slow and doesn’t meet your needs, but ✨our AI can solve this for you✨” which is just so boring. 

Beyond being boring, however, I truly wonder how we can trust any of these providers to not inject backdoors (intentionally or otherwise) when their values so clearly scream that they’re open for business on every front. So saying “hey just ask for what you want and trust the outputs!” seems shady AF. And if we do what some suggested, of making agents fully autonomous, we wouldn’t ever have cause to pause and reflect (let alone catch) this happening.

What I am interested in using these things for

I’m interested in reviewing code humans don’t have time for. Several of the better talks shared the goal of complete code coverage. I’m also interested in putting in guidance and nudges towards doing better work (either from humans or from robots), rather than adding layers on other layers. I’m interested in help for what we know needs doing, and investigations in formats that humans are bad at and machines are good at.

From this conference, I’m now prepared to spend even more time on evaluation than I expected to (50% after baseline systems are in place). And I have new ways of talking about where to interject to inspect the system instead of just trusting it’s working.

I now have more supporting evidence for continuing to think that a workflow or premise needs to be figured out before automation, which happens before AI tooling. And that organizational structures need to allow for this happening at a deep layer, not as something that gets tacked on later as an afterthought.

It also seems like we’re moving away from “zero click attacks” towards “zero user intervention attack” – what can we get agents to do without you noticing?