How to Secure a Self-Hosted AI Stack on Ubuntu

Security for a self-hosted AI stack does not require expert-level infrastructure knowledge. It requires consistent application of a small set of principles: reduce the attack surface, protect credentials, monitor what matters, and update regularly.

Start with SSH. Disable password authentication and require key-based access — the SSH hardening guide walks through the full implementation step by step. Move the SSH port if you want to reduce automated scanning noise, though that alone is not a security measure. Restrict SSH access to specific IP ranges where possible.

Configure UFW to deny inbound by default and allow only the ports you explicitly need. Port 80 and 443 for web traffic. Your SSH port. Nothing else that is not required. Review open ports regularly with ss -tulpn.

Docker introduces surface area. Adding Fail2ban for brute-force protection is a natural next hardening layer once SSH and firewall rules are in place. Containers that share the host network or mount sensitive host paths need careful review. Avoid running containers as root unless the image explicitly requires it. Use user-defined Docker networks instead of exposing services to the host unnecessarily.

Secrets should never appear in your repository, compose files, or environment logs in plain text. Use .env files excluded from Git, Docker secrets, or a secrets manager. Rotate credentials when you suspect exposure. Audit your compose files occasionally for accidentally committed keys.

Automatic security updates are worth enabling on Ubuntu for the operating system packages. Application updates, particularly for Docker images and the tools in your AI stack, require manual review because version bumps can introduce breaking changes. Check changelogs before updating inference engines, web UIs, or automation tools.

Log what matters without logging everything. For a small self-hosted setup, access logs, authentication failures, and application errors are the most useful signals. Store logs somewhere that persists container restarts. Do not log raw prompt content or output unless there is a specific operational need to do so.

Backups are part of security. If the system is ransomed or corrupted, a clean recent backup is often the only recovery path. Store at least one backup off the host itself.

A good baseline is to think in layers: access, network exposure, application configuration, secrets, and recovery. If you touch each layer intentionally, the stack becomes much safer even without enterprise tooling.

For access control, SSH keys are only the beginning. Use a non-root user for regular administration, keep sudo access limited, and review who can log in and from where. On small systems, unnecessary access tends to linger because nobody schedules cleanup.

For network exposure, list every open port and justify it. Many self-hosted AI setups accidentally expose model APIs, admin panels, or internal dashboards simply because a container published a port by default. If a service does not need to be public, bind it to localhost or keep it on an internal Docker network. Your reverse proxy should be the narrow public entry point, not one public endpoint among many.

For application configuration, avoid default passwords, review environment variables, and check container images before deploying updates. Official images and documented release notes are not guarantees, but they reduce risk compared with random community forks.

A small but effective hardening checklist includes:

• key-only SSH access
• UFW default deny with only required ports open
• regular package updates and planned image updates
• no secrets committed to Git
• container volumes reviewed for unnecessary host access
• off-host backups tested, not merely configured
• basic log review for auth failures and repeated errors

Testing recovery is especially important. A backup that has never been restored is only a hopeful theory. Periodically confirm that you can rebuild the service from configuration files and restore the content or database from backup. This is where many "secure" systems reveal that they are only hardened against small failures, not real ones.

Security is therefore less about one perfect configuration and more about eliminating easy wins for attackers while preserving your own ability to recover cleanly. On a small Ubuntu AI stack, that is the standard worth aiming for.

Readers working through adjacent decisions should also look at Why Your Self-Hosted AI Container Keeps Restarting and How to Deploy a Simple Blog Agent on Ubuntu with Docker.