# `Plover.Protocol.Tokenizer`
[🔗](https://github.com/codonnell/plover/blob/main/lib/plover/protocol/tokenizer.ex#L1)

Tokenizes IMAP response lines from binary data into a token stream.

Uses NimbleParsec for compile-time parser generation. The generated code
is pure binary pattern matching with no runtime dependency on NimbleParsec.

RFC 9051 Section 9 - Formal Syntax

# `parse_line`

```elixir
@spec parse_line(binary(), keyword()) ::
  {:ok, [term()], rest, context, line, byte_offset}
  | {:error, reason, rest, context, line, byte_offset}
when line: {pos_integer(), byte_offset},
     byte_offset: non_neg_integer(),
     rest: binary(),
     reason: String.t(),
     context: map()
```

Parses the given `binary` as parse_line.

Returns `{:ok, [token], rest, context, position, byte_offset}` or
`{:error, reason, rest, context, line, byte_offset}` where `position`
describes the location of the parse_line (start position) as `{line, offset_to_start_of_line}`.

To column where the error occurred can be inferred from `byte_offset - offset_to_start_of_line`.

## Options

  * `:byte_offset` - the byte offset for the whole binary, defaults to 0
  * `:line` - the line and the byte offset into that line, defaults to `{1, byte_offset}`
  * `:context` - the initial context value. It will be converted to a map

# `tokenize`

```elixir
@spec tokenize(binary()) :: {:ok, list(), binary()} | {:error, term()}
```

Tokenize one IMAP response line from binary data.

Returns `{:ok, tokens, rest}` on success, or `{:error, reason}` on failure.
Parses one complete response line (up to and including CRLF).
Any data after the CRLF is returned as `rest`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*