What?
=====
This file describes the interface between rsyslog and external output plugins.

Basic Facts
===========
Rsyslog uses stdin and stdout to communicate with the external plugin. This is
an established mode of interprocess communication and well supported by all
decent languages. Parameters, if any, will be passed in via command line arguments,
which should also be easy to obtain in (almost) all languages. Where this is not
the case, it is suggested to either use an external config file or hardcode the
parameters inside the plugin code.

How the plugin receives messages
--------------------------------
Rsyslog pushes messages via stdin. Each message is terminated by a LF. So
a plugin can obtain a full message simply by reading a line.

This can cause problems with multi-line messages. There are some cures for
this. The recommended one is to use JSON message format (more on message
formats below). This will encode LF as "\n" (by JSON RFC requirements) and
thus will ensure there are no embedded LFs even in multiline messages.
An alternative is to use a message format which contains some other delimiter
and program the plugin to watch for this delimiter. With the near-universal
availability of JSON libraries in languages these days, we strongly think that
the JSON approach is superior.

The _message format_ is generated by standard rsyslog methods, that is via
a template. This gives full flexibility over what the plugin is fed. Unfortunately
this also means the necessary template definitions are needed. See the rsyslog
doc for how to do that (in the future, this file will also contain some
samples).

Providing Feedback to Rsyslog
=============================
The plugin may convey error information to rsyslog. To do this, set the
`confirmMessages` flag to `on` in the `omprog` action configuration (this flag
is disabled by default). When this flag is enabled, rsyslog will wait for a
confirmation from the plugin after sending each log message to it.

The plugin must confirm the message by writing a line with the word `OK` to
its standard output. That is, the plugin must write the characters `O`, `K` and
LF (line feed) to stdout.

If the plugin writes a line to stdout containing anything else (for example,
the string `Error: could not connect to the database` followed by a LF), rsyslog
will consider that the plugin has not been able to process the message. The
message will be retried later, according to the retry settings configured for
the `omprog` action (in particular, the `action.resumeInterval` setting). If the
`reportFailures` flag is set to `on`, rsyslog will also log the returned error
line.

If the plugin terminates, the message is also considered as non-processed.
The plugin will later be restarted, and the message retried, according to the
configured retry settings.

When starting the plugin, if `confirmMessages` is `on`, rsyslog will also wait
for the plugin to confirm its initialization. The plugin must write an `OK` line
to stdout just after starting. If it writes anything else or terminates, rsyslog
will consider the plugin initialization has failed, and will try to restart it
later. It is recommended to ensure that the program is really ready to start
receiving messages before emitting this first confirmation (for example, by
checking that a destination database is accessible).

Example of exchanged messages
-----------------------------
The following sequence illustrates the message exchanges between rsyslog and the
plugin. A right arrow (`=>`) indicates a message read by the plugin from its
stdin, and a left arrow (`<=`) indicates a message written by the plugin to its
stdout. Note that the arrows themselves are not read or written. Each line is
terminated by a LF (\n).

    <= OK
    => log message 1
    <= OK
    => log message 2
    <= OK
    ...
    => log message N
    <= OK

Note that the first `OK` does not confirm any message, but that the plugin has
correctly started and is ready to receive messages. When the plugin receives
an end-of-file (EOF), it must silently terminate.

Confirmation timeout
--------------------
The plugin must confirm each message before a timeout of 10 seconds. A different
timeout can be configured using the `confirmTimeout` setting. If the plugin does
not return anything to rsyslog before this timeout, rsyslog will restart the
program, and retry the message.

It is important that the `confirmTimeout` be adjusted to an appropriate value for
your plugin, since a too low timeout can cause duplicate processing of messages
(if rsyslog decides that the plugin has gone unresponsive when it really hasn't).
In case of doubt, consider either a) setting a rather high timeout (but note that
this can go against the _fail fast_ pattern), b) enabling the `signalOnClose`
and/or `killUnresponsive` flags in the configuration, to ensure that the plugin
will stop its processing if rsyslog decides to restart it, or c) using keep-alive
feedback as explained below.

Keep-alive feedback
-------------------
The plugin can also provide _keep-alive feedback_ to rsyslog. This allows the
plugin to exceed the `confirmTimeout` for certain messages requiring a long-
running processing (for example, attempting to reconnect to a database after a
temporary loss of connection), without having to increase too much the configured
timeout in anticipation of those peak latencies.

To provide keep-alive feedback, the plugin must periodically write a dot (`.`)
to stdout, without ending the line. Each dot must be written before the timeout
is reached (rsyslog will restart the timeout after receiving each dot). Once the
plugin completes the processing of the message, it must write the `OK` word (or
an error message) as usual, followed by a line feed. rsyslog will ignore all
leading dots when processing the received line.

The following sequence illustrates the use of the keep-alive feedback:

    <= OK
    => log message 1
    <= OK
    => log message 2
    <= .......OK
    => log message 3
    <= ..OK
    => log message 4
    <= ...OK

The plugin can also provide keep-alive feedback during its initialization, before
emitting the first `OK`. This can be useful to give enough time to the plugin to
complete its initialization checks.

Writing to stderr
-----------------
Aside from confirming messages via stdout, at any moment the plugin may write
anything it wants to stderr. The `output` setting of the `omprog` action allows
capturing the plugin's stderr to a file. This provides an easy way for the
plugin to record its own logs in case something fails (for example, logging the
details of a database connection error).

To prevent the output file from growing too much over time, it is recommended
to periodically rotate it using a tool like _logrotate_. After each rotation of
the file, a HUP signal must be sent to rsyslog. This will cause rsyslog to
reopen the file.

When multiple instances of the program are running concurrently (see [Threading
Model](#threading-model)), rsyslog guarantees that the lines written to stderr
by the various instances will not appear intermingled in the output file, as
long as: 1) the lines are short enough (the actual limit depends on the platform:
4KB on Linux, and at least 512 bytes on other systems), and 2) the program
writes each line at a time, without buffering multiple lines. (Commonly, this
can be accomplished by either flushing the stream after each line, writing to
the stream in line-buffered mode, or doing a single write in case the stream is
unbuffered. Which of these alternatives is the easiest or most natural to use
depends on the language and libraries the program is coded in.)

If the `output` setting is not specified, the plugin's stderr will be ignored
(it will be redirected to `/dev/null`).

When `confirmMessages` is set to `off`, the `output` setting will capture both
the stdout and stderr of the plugin to the specified file (and if omitted, both
stdout and stderr will be redirected to `/dev/null`). The same considerations
regarding the rotation of the file and the consistency of the lines apply in
this case.

Example implementation
----------------------
See [this Python plugin skeleton](skeletons/python/plugin-with-feedback.py) for
a featured example on how a plugin can provide feedback to rsyslog.


Batching of Messages (Transactions)
===================================
You can write a plugin that processes the messages in batches (also called
_transactions_), instead of individually. For a general explanation on how
rsyslog handles the batching of messages, see [dev-oplugins].

_**Warning:**
This feature is currently **experimental**. It could change in future releases
without keeping backwards compatibility with existing configurations or the
interface described below._

How to process the messages in batches (transactions)
-----------------------------------------------------
To enable transactions, set the `useTransactions` flag to `on` in the `omprog`
action configuration. When this flag is enabled, rsyslog will send a special
message line to the plugin's stdin to indicate that a batch of log messages is
going to be sent. This special message is `BEGIN TRANSACTION` by default, although
it can be customized using the `beginTransactionMark` setting of `omprog`.

After the `BEGIN TRANSACTION` line, rsyslog will send the log messages in the
batch, each one in its own line, and then another special message `COMMIT
TRANSACTION` to indicate the batch has ended. (The later can be customized via
the `commitTransactionMark` setting.)

That is:

    BEGIN TRANSACTION
    log message 1
    log message 2
    ...
    log message N
    COMMIT TRANSACTION
    BEGIN TRANSACTION
    ...
    COMMIT TRANSACTION
    BEGIN TRANSACTION
    ...
    COMMIT TRANSACTION
    ...

(with a LF at the end of each line)

When transactions are enabled, rsyslog will always send log messages within a
transaction block, never outside it, even when the batch consists of a single message.

How to provide feedback when using transactions
-----------------------------------------------
You can enable both the `useTransactions` and `confirmMessages` settings in
the `omprog` action, only one of them, or none of them. When both settings
are set to `on`, the plugin must confirm the `BEGIN TRANSACTION` and `COMMIT
TRANSACTION` messages, and the log messages within the transaction. The log
messages within the transaction can be confirmed with any of the following
status codes (which the plugin must write to stdout):
* `OK`
* `DEFER_COMMIT`
* `PREVIOUS_COMMITTED`

Refer to https://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html
for an explanation on the meaning of these status codes. You will typically
need to return the `DEFER_COMMIT` status code, since the other codes imply a
partial commit, and do not guarantee that the `COMMIT TRANSACTION` will be
received.

The following sequence illustrates the exchanges between rsyslog and the plugin
when transactions and message confirmations are enabled, and the plugin
confirms the log messages within each transaction with `DEFER_COMMIT`:

    <= OK
    => BEGIN TRANSACTION
    <= OK
    => log message 1
    <= DEFER_COMMIT
    => log message 2
    <= DEFER_COMMIT
    ...
    => log message 5
    <= DEFER_COMMIT
    => COMMIT TRANSACTION
    <= OK
    => BEGIN TRANSACTION
    <= OK
    => log message 6
    <= DEFER_COMMIT
    => log message 7
    <= DEFER_COMMIT
    ...
    => log message 10
    <= DEFER_COMMIT
    => COMMIT TRANSACTION
    <= OK


Threading Model
===============
Write your plugin as you would do in a single threaded environment. Rsyslog
automatically detects when it is time to spawn additional threads. If it
decides so, it will also spawn another instance of your script and feed it
concurrently. Note that rsyslog will also terminate instances that it knows
are no longer needed.

If your plugin for some reason cannot be run in multiple instances, there are ways
to tell rsyslog to work with a single instance. But it is strongly suggested to
not restrict rsyslog to do that. Multiple instances in almost all cases do NOT mean
any burden to the plugin developer. Just think of two (or more) independent
instances of your program running in different console windows. If that is no
problem, rsyslog running multiple instances of it is also no problem.

Future Enhancements
===================
Interfaces for external input, filter and message modification plugins are
planned. Most probably, they will become available in the order mentioned
in the last sentence.

External Message Modification Modules
-------------------------------------
The external plugin will use stdin to receive the message that it potentially
can modify. The message will be LF-terminated, and no LF must be present within
the message itself.  By default, the MSG part of the message is provided as input.
The "interface.input" parameter can be used to modify this. It may has the following
values:

* "msg" (the default)
* "rawmsg", which is the complete message (including header) as received by rsyslog
* "json", which is the complete message object (with all properties broken
  out) as a JSON object string. This is the "jsonmesg" dynamic message property.

**Note**: if multi-line messages are to be processed, JSON representation **must**
be used, otherwise errors will happen.

The ability to use non-JSON representations is primarily a performance
enhancement. Building the JSON representation causes some overhead, and
very often access to either msg or rawmsg is fully sufficient.

The plugin will emit a JSON representation of those properties
that **need to be modified** and their new values to stdout. Again, this
is delimited by LF, with no LF permitted inside the JSON representation.
Only properties that are to be changed must be included in the response.
Unchanged properties should NOT be included in the response, as this would
increase processing cost. If no property is to be modified, an empty
JSON representation is to be provided.

The plugin must emit one response line for each message (line) received, and
must do so in the same order in which the messages were put in stdin. Note that
like the output module interface, multiple instances of the plugin may be
activated. See above for more information.

Most message properties can be modified. Modifiable are:
* rawmsg
* msg
* syslogtag
* syslogfacility
* syslogseverity
* msgid
* procid
* structured-data
* hostname (aliased "source")
* fromhost
* fromhost-ip
* all message variable ("$!" tree)

If the message variable tree is modified, new variables may also be *added*. Deletion
of message variables is not directly supported. If this is desired, it is suggested
to set the variable in question to the empty string ("").

Implementation
--------------
The plugin interface is implemented via the "mmexternal" native plugin. See its
documentation on how to tie your plugin into rsyslog's processing flow.

[dev-oplugins]: https://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html
