Preview

This post documents how you can use a lesser known authorized_keys config to give AI agents (or anything else) programmatically designed and limited access to any resources and tools on a different system, for example the system hosting the agent container.

See the ssh-restricted-dispatcher github repo for code to set this up yourself.

Background

I have been experimenting with different AI agent setups on a hobby Linux server at home.

My use-case is having an always-reachable AI agent that I can easily reach from my phone, for example via Telegram.

I have a small set of skills (visualize Liam Neeson reading this post…) that are actually useful to me, for example the one where the AI uses my Twitter web cookie to read the most recent 100 tweets from my personal “For you” feed and then summarizes the main themes that I am interested in. As an erstwhile Twitter scroller, this saves me a tonne of time.

After Hermes Agent (slightly too much functionality out of the box for me), and a longer stint with CC-Connect fronting for OpenCode, a lovely tool that I use mostly with OpenCode Zen credits for Zen and Go models (MiMo V2.5-Pro is my current favourite 1T-ish cheap and open model yum!), I recently tried the Pi coding agent along with TelePi for Telegram access and was delighted to discover that Pi fits my brain really well.

The problem

I usually run my personal agents inside docker containers. It’s not a perfect isolation system, but it helps to limit the blast damage (compare to folks runnig OpenClaw on Mac Minis logged in to all their account), should the agent for example ingest some poisonous prompting via innocent browsing of the web.

However, to make an agent more useful, you do need to connect it to tools, and so I was looking for a robust, low-friction solution to give my containerised agents access to a carefully specified subset of tools.

The solution

Enter SSH “command restriction” together with a user-customizable Python multi-subcommand CLI tool!

SSH “command restriction” is an authorized_keys setup whereby any remote command-execution by a client using a specific key can be forced into one specific command.

By doing exactly this, we can set it up so that the agent can only call our CLI with our defined sub-commands.

The ssh-restricted-dispatcher is an implementation of this idea, using the cyclopts CLI framework. I like Cyclopts more than click, typer and friends, because it enables one to write sub-commands with stock-standard type-annotated and documented Python functions, instead of framework-specific metadata.

The figure below illustrates the whole setup.

Agent uses dispatch_shim.sh which thunks through to the SSH server dispatcher.py

Agent uses dispatch_shim.sh which thunks through to the SSH server dispatcher.py

The agent can introspect the CLI and its sub-commands, so it auto-discovers what’s available to it.

In commands_example.py, you can see how one can for example give the agent access to sendmail on the host system, but tweaked so that the agent does not need any extra keys, and you can enforce aspects of the mail sending.

P.S. You could also use restricted-ssh-commands package. I personally like that I can configure the whole entrypoint with sub-commands via Python.

Conclusion

You can use SSH command restriction to limit an agent quite precisely to the tool and tool shapes you define.

You can use ssh-restricted-dispatcher as the basis for the Python part of your solution.