What LiteLLM actually is​

LiteLLM is an open-source proxy and Python SDK that gives you a single OpenAI-compatible interface across 100+ providers, with routing, spend tracking, and caching. You self-host it, configure it via a proxy_config.yaml, and it does the provider-abstraction job extremely well — it's a dependable, widely-adopted piece of infrastructure, and if all you need is "one API, many providers, budget tracking," it's a completely reasonable place to stop.

Where the two diverge​

The gap shows up once "route to the right model" stops being the whole requirement.

• Policy enforcement. OpenProxyAI's pipeline runs PII redaction, secrets scanning, topic guarding, and prompt-injection detection on every request as a core stage — not a separate service you wire in. LiteLLM's core proxy is routing- and budget-focused; equivalent guardrail behavior generally means integrating something else in front of or alongside it.
• Compliance starting points. OpenProxyAI ships Healthcare, Finance, and Government policy templates you apply and customize. LiteLLM doesn't have an equivalent — you're building your policy configuration from scratch.
• Admin surface. LiteLLM is primarily config-file and CLI driven, with enterprise features like chargeback attribution gated behind a separate licensed tier. OpenProxyAI ships a full admin console — teams, budgets, policies, audit log — as the default way to operate it, not an add-on.
• Audit trail. OpenProxyAI's logging is immutable by design and built for handing to an auditor. LiteLLM's logging is flexible and integrates with a wide range of observability backends, but "audit-ready by default" isn't the framing it's built around.

Where LiteLLM is the better call​

If you're a single team that needs multi-provider routing and cost tracking, and you're comfortable owning the guardrail and compliance layer yourself (or you don't need one yet), LiteLLM's simplicity is a real advantage — it's less software to run, and its provider coverage is excellent. Plenty of OpenProxyAI's own architecture leans on the same library for exactly that reason.

The honest split: pick LiteLLM if the org chart approving your AI usage is small. Pick OpenProxyAI if there's a compliance or security team in that approval chain who needs redaction, an audit trail, and policy enforcement to already be there — not bolted on later.

Portkey's model​

Portkey is SaaS-first. The default way to run it is as a managed, multi-tenant service — your traffic goes through infrastructure Portkey operates. VPC hosting and fully air-gapped deployment exist, but as part of the Enterprise tier, not the starting point. For a lot of teams that's a perfectly reasonable trade: less infrastructure to run, a managed service handling scaling and uptime.

OpenProxyAI's model​

OpenProxyAI is self-hosted from the first tier up. It runs inside your VPC against your own Postgres and Redis from day one — there's no multi-tenant data plane to opt out of later, because it was never the default to begin with.

Why the ordering is the actual decision​

Both products can, in the end, run inside your infrastructure. But "self-hosted is available at the top tier" and "self-hosted is what you start with" are different conversations with your security team. The first one asks them to evaluate a SaaS vendor's data handling practices unless you pay for the option to avoid it. The second one means there's no SaaS data-handling question to evaluate in the first place — the software runs where you tell it to, from the free trial onward.

If your organization is comfortable with a managed multi-tenant service and wants the fastest path to a working gateway, Portkey's SaaS-first model is a legitimate, well-built option — the model catalog and guardrail feature set are genuinely strong. If "our prompts never touch infrastructure we don't control" is a requirement rather than a nice-to-have, that's the gap OpenProxyAI is built to close by default.

See Deploying with Docker Compose for what running it in your own VPC actually looks like.

Why the distinction matters​

It's common in this category to see a security page with a row of certification badges and fine print that, read carefully, describes a much narrower claim than the badge implies. We'd rather not do that. If you're evaluating OpenProxyAI for a regulated workload, the honest starting point is: we don't currently hold SOC 2, ISO 27001, or HIPAA certification. Full detail on where things actually stand is on the Trust Center.

What we do have is a real, useful thing that's easy to describe accurately: three starting configurations that encode the policy decisions a Healthcare, Finance, or Government deployment usually needs to make anyway.

What's actually in each one​

Healthcare sets PHI redaction defaults tuned for clinical data, restricts which models can receive requests tagged as containing clinical content, and extends audit retention beyond the platform default.

Finance blocks cardholder-data patterns before they reach a provider, captures transaction context in the audit log, and tightens default rate limits.

Government pairs with air-gapped deployment (AIRGAP_MODE=true) and enforces data residency with no fallback outside approved regions, plus 7-year audit retention.

Applying one doesn't lock you in​

A template merges into your organization's existing policy configuration — it doesn't overwrite what you've already customized, and nothing about applying one prevents you from adjusting individual policies afterward. Turn a policy from enforce to log-only while you evaluate its false-positive rate. Tighten a threshold. Swap the model allowlist. The template is a starting point calibrated to a framework's typical requirements, not a fixed configuration you're stuck with.

The honest pitch​

If your actual requirement is "we need a SOC 2 report to show our auditor," a policy template doesn't satisfy that — no software can, that's an organizational process, not a config file. What it does do: it gets you from a blank policy configuration to something aligned with what your framework typically requires, in one API call instead of reading the framework's requirements document and building each policy from scratch.

See Compliance Templates for how to apply one, and the Trust Center for where our own certification status actually stands today.

The pipeline​

Why the order isn't arbitrary​

Auth and rate limiting come first because there's no point doing any other work — redacting PII, checking a cache — for a request you're about to reject anyway. Reject cheap failures as early as possible.

Policy runs before cache, not after. This is the one that actually matters. If a request contains PII and policy redaction ran after the cache check, the cache key would be built from the raw, unredacted content — which means the cached entry itself could contain the PII that was supposed to be stripped. Running policy first means the cache only ever sees the already-redacted version. The cache can't leak something it never saw.

Cache comes before routing for the obvious reason: a cache hit means there's no routing decision to make at all. The request never reaches the provider-selection logic, because it never reaches a provider.

Logging is the only stage that happens after the response is sent, not before. Every other stage can reject or delay the request. Logging can't — it's fire-and-forget, written asynchronously once the response has already gone back to the caller. A slow or temporarily unavailable logging backend should never be the reason your application's request takes longer.

What this costs in practice​

Stages 1–3 and 5–6 typically add low-single-digit milliseconds combined — the actual overhead is dominated by whichever provider you're calling, not by OpenProxyAI. A cache hit at stage 4 doesn't add latency at all; it replaces the provider round-trip entirely, which is usually hundreds of milliseconds saved, not spent.

The four milliseconds aren't overhead you're paying for nothing. They're the four milliseconds where redaction, budget enforcement, and cache-key correctness all have to happen in the right order — before the request goes anywhere you can't take it back from.

Full detail on each stage in Request Pipeline.

We didn't want the answer to be discipline. We wanted it to be a database property that doesn't care whether the application code remembered.

Where the application-level approach breaks​

WHERE org_id = ? is a pattern, not a guarantee. It's only as strong as the least careful query in the codebase — one join that forgets the clause, one raw SQL escape hatch during a migration, one new engineer who doesn't know the convention yet. When it breaks, it doesn't throw an error. It just quietly returns rows that belong to someone else's organization. That's close to the worst possible failure mode for a security bug: silent, and only discovered when someone notices data that shouldn't be there.

What row-level security changes​

PostgreSQL's row-level security (RLS) moves the boundary into the database itself. Every table that holds organization-scoped data has a policy that filters rows based on a session-level variable — set once, at the start of a database session, before any query runs.

The critical property: a query that forgets to filter by organization doesn't leak the wrong rows. It returns no rows for anything outside its session's scope, because the database enforces the boundary independently of what the query asked for. The failure mode changes from "silently wrong" to "safely empty" — which is the difference between a data breach and a bug report.

The part that's easy to get wrong​

RLS only works if every code path that touches the database actually sets the session's org context before querying. That's straightforward in the request path — one middleware call at the start of every request. It's less obvious for background jobs: log archival, webhook retries, scheduled analytics refreshes all run outside the request-response cycle, on their own schedule, sometimes across every organization in a single sweep. Each of those has to open its own database session and explicitly scope it, rather than assuming a request-scoped session it can borrow — because there isn't one to borrow from.

Get that part right and the guarantee holds everywhere, not just in the code paths someone thought to test.

More detail on the actual policy mechanics in Row-Level Security.

That ordering isn't a licensing decision. It changes what you're actually agreeing to when you adopt the tool.

The question a SaaS gateway can't answer cleanly​

If your proxy is a multi-tenant SaaS product, every request your organization sends passes through infrastructure someone else operates. That's fine for a lot of tools. It's a much harder sell when the traffic is prompts containing customer PII, internal financial data, or anything else your compliance team has opinions about — because now the question isn't just "does this vendor have good security practices," it's "which of our regulatory obligations just became partially their responsibility."

A self-hosted-by-default proxy sidesteps the question entirely. The service runs on infrastructure you control — your VPC, your Postgres, your Redis. There's no data plane operated by us that your requests need to trust.

What this actually looks like​

The only thing that leaves your network is the outbound call to whichever model provider you've configured — the same call your application would make directly, just routed through policy enforcement and logging first. There's no OpenProxyAI-operated service in the request path at all.

The trade-off, honestly​

Self-hosted-first isn't free. You're running a service, which means you own its uptime, its scaling, and its patching cadence — that's real operational weight a managed SaaS product takes off your plate. If you'd rather not carry that, the hosted Starter and Growth plans exist for exactly that reason.

But for the buyer this actually matters to — a security team evaluating whether prompts containing regulated data are allowed to leave the building — self-hosted-first isn't a nice-to-have. It's the reason the tool clears procurement at all.

See Deploying with Docker Compose or Deploying with Helm to see what actually running it looks like.