Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions .github/scripts/provision-kvm-host.sh
Original file line number Diff line number Diff line change
Expand Up @@ -138,12 +138,16 @@ sudo chown "$(id -u):$(id -g)" /srv/hyper
mkdir -p /srv/hyper/layers

# firecracker.install writes into /opt/firecracker as the invoking user, then
# the helper requires both binaries root-owned and not group/world-writable.
# the helper requires both binaries root-owned and not group/world-writable,
# and every parent directory too: the jailer is handed firecracker by path.
# The hosted runner image leaves /opt itself writable beyond root.
sudo chown root:root /opt
sudo chmod go-w /opt
sudo mkdir -p /opt/firecracker
sudo chown "$(id -u)" /opt/firecracker
mix firecracker.install
sudo chown root:root /opt/firecracker/firecracker /opt/firecracker/jailer
sudo chmod 0755 /opt/firecracker/firecracker /opt/firecracker/jailer
sudo chown root:root /opt/firecracker /opt/firecracker/firecracker /opt/firecracker/jailer
sudo chmod 0755 /opt/firecracker /opt/firecracker/firecracker /opt/firecracker/jailer

# With passwordless sudo the task installs setuid-root itself, for the runner's group.
mix suidhelper.install --group "$(id -gn)"
Expand Down
2 changes: 2 additions & 0 deletions docs/cookbook/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -428,6 +428,8 @@ config :hyper, Hyper.Cfg.Cluster,
- Must be owned by `root`.
- Must be only writable by `root`.
- Must not be a symlink.
- For `firecracker` only: every directory above it must also be owned by
`root` and writable only by `root`, since the jailer is handed it by path.

#### Range

Expand Down
17 changes: 14 additions & 3 deletions docs/cookbook/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -368,9 +368,20 @@ mix firecracker.install # installs to /opt/firecracker
The task installs the binaries under their bare basenames (`firecracker`,
`jailer` – the setuid helper rejects version-stamped names), marks them
executable, and prints the `[tools]` snippet for `/etc/hyper/config.toml`.
After installing, make both binaries root-owned and not group- or
world-writable (the task prints the exact `chown` command); the helper refuses
them otherwise.
After installing, make both binaries **and their directory** root-owned and
not group- or world-writable (the task prints the exact commands); the helper
refuses them otherwise:

```sh
sudo chown root:root /opt/firecracker /opt/firecracker/firecracker /opt/firecracker/jailer
sudo chmod 0755 /opt/firecracker /opt/firecracker/firecracker /opt/firecracker/jailer
```

Every parent directory (`/opt` included) must be root-owned and not group- or
world-writable too.

Hosts installed before 0.2 left `/opt/firecracker` owned by the node user;
every VM launch there fails until the directory is chowned as above.

### SUID Helper

Expand Down
259 changes: 228 additions & 31 deletions lib/hyper/firecracker/api/transport.ex
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@ defmodule Hyper.Firecracker.Api.Transport do
@moduledoc """
`oapi_generator` client backend for the Firecracker API. Each generated
operation calls `request/1` with a plain map describing the call; this issues
it over the target daemon's HTTP-over-Unix-socket API via `Req` and decodes the
typed response.
it over the target daemon's HTTP-over-Unix-socket API on a raw `:gen_tcp`
socket and decodes the typed response. The daemon is untrusted: the status
line, headers and body are size-capped, and the whole call shares one 30 s
deadline.

The per-VM socket path arrives as `opts[:socket_path]` (injected by
`Hyper.Node.FireVMM.Client`). Request bodies are generated structs encoded by
Expand All @@ -12,8 +14,11 @@ defmodule Hyper.Firecracker.Api.Transport do
own generated `decode/1` (also from `Hyper.Firecracker.Api.Codec`).

Returns: `:ok` (204 / empty), `{:ok, decoded}` (2xx with a typed body),
`{:error, {:api, status, fault_message}}` (non-2xx), or
`{:error, {:transport, reason}}` (socket/connection failure).
`{:error, {:api, status, fault_message}}` (non-2xx),
`{:error, {:unexpected_body, status}}` (a typed reply that is not a JSON
object), `{:error, {:body_too_large, max_bytes}}`, or
`{:error, {:transport, reason}}` (socket failure, deadline, or an oversized
head).
"""

use OpenTelemetryDecorator
Expand All @@ -22,60 +27,252 @@ defmodule Hyper.Firecracker.Api.Transport do

@max_body_bytes Information.as_bytes(Information.mib(1))

# A hostile daemon controls every byte of the reply, so the head is bounded
# as tightly as the body: the length of each status, header, chunk-size and
# trailer line, and how many of them there may be.
@max_line_bytes Information.as_bytes(Information.kib(8))
@max_headers 100

# One deadline for the whole call, below `Hyper.Node.FireVMM.Client`'s 35 s
# call cap, so a daemon drip-feeding bytes cannot keep the client busy.
@call_budget_ms Unit.Time.as_ms(Unit.Time.s(30))

@type result ::
:ok
| {:ok, term()}
| {:error,
{:api, pos_integer(), String.t() | nil}
| {:transport, term()}
| {:body_too_large, pos_integer()}}
| {:body_too_large, pos_integer()}
| {:unexpected_body, pos_integer()}}

@spec request(map()) :: result()
@decorate with_span("Hyper.Firecracker.Api.Transport.request", include: [:method, :url])
def request(%{method: method, url: url, opts: opts} = info) do
socket_path = Keyword.fetch!(opts, :socket_path)
deadline = System.monotonic_time(:millisecond) + @call_budget_ms

req =
Req.new(
base_url: "http://localhost",
unix_socket: socket_path,
retry: false,
receive_timeout: 30_000
)
case call(Keyword.fetch!(opts, :socket_path), encode(method, url, info), deadline) do
{:ok, status, headers, body} ->
handle(status, decode(headers, body), Map.get(info, :response, []))

req_opts = [method: method, url: url, into: &collect_capped/2]
{:error, :body_too_large} ->
{:error, {:body_too_large, @max_body_bytes}}

req_opts =
if Map.has_key?(info, :body), do: Keyword.put(req_opts, :json, info.body), else: req_opts
{:error, reason} ->
{:error, {:transport, reason}}
end
end

req_opts = if q = info[:query], do: Keyword.put(req_opts, :params, q), else: req_opts
defp encode(method, url, info) do
target = if q = info[:query], do: url <> "?" <> URI.encode_query(q), else: url

case Req.request(req, req_opts) do
{:ok, %Req.Response{status: status, body: body} = response} ->
if Req.Response.get_private(response, :body_too_large),
do: {:error, {:body_too_large, @max_body_bytes}},
else: handle(status, body, Map.get(info, :response, []))
{headers, body} =
case Map.fetch(info, :body) do
{:ok, body} ->
json = Jason.encode_to_iodata!(body)

{:error, reason} ->
{:error, {:transport, reason}}
{[
"content-type: application/json\r\naccept: application/json\r\n",
"content-length: #{IO.iodata_length(json)}\r\n"
], json}

:error ->
{[], []}
end

[
method |> Atom.to_string() |> String.upcase(),
" ",
target,
" HTTP/1.1\r\nhost: localhost\r\n",
headers,
"\r\n",
body
]
end

defp call(socket_path, request, deadline) do
opts = [:binary, active: false, packet: :raw, send_timeout: @call_budget_ms]

with {:ok, sock} <- :gen_tcp.connect({:local, socket_path}, 0, opts, remaining(deadline)) do
try do
with :ok <- :gen_tcp.send(sock, request),
{:ok, status, headers} <- read_head(sock, deadline),
{:ok, body} <- read_body(sock, status, headers, deadline) do
{:ok, status, headers, body}
end
after
:gen_tcp.close(sock)
end
end
end

defp read_head(sock, deadline) do
:ok = :inet.setopts(sock, packet: :http_bin, packet_size: @max_line_bytes)

case recv(sock, 0, deadline) do
{:ok, {:http_response, _version, status, _reason}} ->
read_headers(sock, status, %{}, 0, deadline)

{:ok, _other} ->
{:error, :malformed_status_line}

{:error, :emsgsize} ->
{:error, :line_too_long}

{:error, _} = err ->
err
end
end

defp read_headers(_sock, _status, _headers, count, _deadline) when count > @max_headers,
do: {:error, :too_many_headers}

defp read_headers(sock, status, headers, count, deadline) do
case recv(sock, 0, deadline) do
{:ok, {:http_header, _, name, _, value}} ->
headers = Map.put(headers, header_name(name), value)
read_headers(sock, status, headers, count + 1, deadline)

{:ok, :http_eoh} ->
{:ok, status, headers}

{:ok, _other} ->
{:error, :malformed_header}

{:error, :emsgsize} ->
{:error, :line_too_long}

{:error, _} = err ->
err
end
end

defp header_name(name) when is_atom(name), do: name |> Atom.to_string() |> String.downcase()
defp header_name(name), do: String.downcase(name)

defp read_body(_sock, status, _headers, _deadline)
when status in 100..199 or status in [204, 304],
do: {:ok, ""}

defp read_body(sock, _status, headers, deadline) do
:ok = :inet.setopts(sock, packet: :raw)

cond do
chunked?(headers) -> read_chunks(sock, [], 0, deadline)
length = headers["content-length"] -> read_sized(sock, length, deadline)
true -> read_to_close(sock, [], 0, deadline)
end
end

@spec collect_capped({:data, binary()}, {Req.Request.t(), Req.Response.t()}) ::
{:cont | :halt, {Req.Request.t(), Req.Response.t()}}
defp collect_capped({:data, data}, {req, %Req.Response{body: body} = resp}) do
if byte_size(body) + byte_size(data) > @max_body_bytes do
{:halt, {req, %{Req.Response.put_private(resp, :body_too_large, true) | body: ""}}}
defp chunked?(%{"transfer-encoding" => coding}),
do: coding |> String.downcase() |> String.contains?("chunked")

defp chunked?(_headers), do: false

defp read_sized(sock, length, deadline) do
case Integer.parse(length) do
{0, ""} -> {:ok, ""}
{n, ""} when n > @max_body_bytes -> {:error, :body_too_large}
{n, ""} when n > 0 -> recv(sock, n, deadline)
_ -> {:error, :bad_content_length}
end
end

defp read_to_close(sock, acc, size, deadline) do
case recv(sock, 0, deadline) do
{:ok, data} when size + byte_size(data) > @max_body_bytes -> {:error, :body_too_large}
{:ok, data} -> read_to_close(sock, [acc, data], size + byte_size(data), deadline)
{:error, :closed} -> {:ok, IO.iodata_to_binary(acc)}
{:error, _} = err -> err
end
end

defp read_chunks(sock, acc, size, deadline) do
with {:ok, line} <- read_line(sock, deadline),
{:ok, chunk_size} <- chunk_size(line) do
cond do
chunk_size == 0 ->
with :ok <- skip_trailers(sock, 0, deadline), do: {:ok, IO.iodata_to_binary(acc)}

size + chunk_size > @max_body_bytes ->
{:error, :body_too_large}

true ->
with {:ok, chunk} <- recv(sock, chunk_size, deadline),
:ok <- expect_crlf(sock, deadline) do
read_chunks(sock, [acc, chunk], size + chunk_size, deadline)
end
end
end
end

defp chunk_size(line) do
[hex | _extensions] = String.split(line, ";", parts: 2)

case Integer.parse(String.trim(hex), 16) do
{n, ""} when n >= 0 -> {:ok, n}
_ -> {:error, :malformed_chunk}
end
end

defp expect_crlf(sock, deadline) do
case recv(sock, 2, deadline) do
{:ok, "\r\n"} -> :ok
{:ok, _other} -> {:error, :malformed_chunk}
{:error, _} = err -> err
end
end

defp skip_trailers(_sock, count, _deadline) when count > @max_headers,
do: {:error, :too_many_headers}

defp skip_trailers(sock, count, deadline) do
case read_line(sock, deadline) do
{:ok, "\r\n"} -> :ok
{:ok, _trailer} -> skip_trailers(sock, count + 1, deadline)
{:error, _} = err -> err
end
end

defp read_line(sock, deadline) do
:ok = :inet.setopts(sock, packet: :line, packet_size: @max_line_bytes)
line = recv(sock, 0, deadline)
:ok = :inet.setopts(sock, packet: :raw)

case line do
{:error, :emsgsize} -> {:error, :line_too_long}
other -> other
end
end

defp recv(sock, length, deadline), do: :gen_tcp.recv(sock, length, remaining(deadline))

defp remaining(deadline), do: max(deadline - System.monotonic_time(:millisecond), 0)

@spec decode(%{String.t() => binary()}, binary()) :: term()
defp decode(headers, body) do
with true <- body != "" and json?(headers),
{:ok, decoded} <- Jason.decode(body) do
decoded
else
{:cont, {req, %{resp | body: body <> data}}}
_ -> body
end
end

defp json?(%{"content-type" => type}) do
[media | _params] = String.split(type, ";", parts: 2)
String.downcase(String.trim(media)) == "application/json"
end

defp json?(_headers), do: false

@spec handle(pos_integer(), term(), [{pos_integer(), term()}]) :: result()
defp handle(status, body, responses) when status in 200..299 do
case List.keyfind(responses, status, 0) do
{^status, :null} -> :ok
{^status, {module, :t}} -> {:ok, module.decode(body)}
{^status, {module, :t}} when is_map(body) -> {:ok, module.decode(body)}
{^status, {_module, :t}} -> {:error, {:unexpected_body, status}}
_other -> if body in [nil, ""], do: :ok, else: {:ok, body}
end
end
Expand Down
17 changes: 16 additions & 1 deletion lib/hyper/node/vmlinux.ex
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,23 @@ defmodule Hyper.Node.Vmlinux do
defp present(path) do
cond do
not File.regular?(path) -> {:error, {:vmlinux_missing, path}}
not Provider.usable?(path) -> {:error, {:vmlinux_untrusted, path}}
not Provider.usable?(resolve(path)) -> {:error, {:vmlinux_untrusted, path}}
true -> :ok
end
end

# The helper canonicalizes the kernel path before vetting it, so a symlink is
# judged by its target here too. Past the hop bound the link itself is
# returned, which `Provider.usable?/1` refuses.
@max_link_hops 40

defp resolve(path, hops \\ @max_link_hops)
defp resolve(path, 0), do: path

defp resolve(path, hops) do
case :file.read_link_all(path) do
{:ok, target} -> resolve(Path.expand(to_string(target), Path.dirname(path)), hops - 1)
{:error, _} -> path
end
end
end
7 changes: 4 additions & 3 deletions lib/mix/tasks/firecracker.install.ex
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,11 @@ defmodule Mix.Tasks.Firecracker.Install do
Mix.shell().info("""

Almost done. Run these as root so the setuid helper will accept the binaries
(it refuses any jailer/firecracker not owned by root):
(it refuses any jailer/firecracker not owned by root, or in a directory
anyone but root can write):

sudo chown root:root #{fc} #{jailer}
sudo chmod 0755 #{fc} #{jailer}
sudo chown root:root #{prefix} #{fc} #{jailer}
sudo chmod 0755 #{prefix} #{fc} #{jailer}

Then add to /etc/hyper/config.toml (file: root-owned, mode 0644):

Expand Down
Loading
Loading