Skip to content

LIVE #1 - Understand CLN Plugin mechanism with a Python example

In this live, we add the method myplugin to Core Lightning by writing a dynamic Python plugin called myplugin.py. Doing this, we try to understand: (1) how startup options are passed to the plugin, (2) how cli parameters are passed to the plugin and (3) how to communicate to the lightning node via JSON-RPC over unix sockets.

Transcript with corrections and improvements

Introduction

I am happy to see you and thank you to take the time to attend this live.

Before we begin, I would like to thank Blockstream for giving me the opportunity to do those lives.

Let's go.

Plugins are first class citizens in CLN implementation and one of the key part of CLN plugins is that they can be written in any languages.

This is really amazing.

Today we are going to write a plugin in Python without using pyln-client library in order to understand the ins and outs of its mechanism.

This way we should be able to transpose the script we write in our prefered languages.

To those who are interested in using pyln-client, please attend to the next live session where we will cover part of the library.

The plugin mechanism allows:

  1. to register new JSON-RPC methods that can be called via lightningd either with lightning-cli or directly using unix sockets,
  2. to add hooks and
  3. to add notifications.

Today we are going add the method myplugin to CLN by writing a dynamic Python plugin called myplugin.py.

When we start the plugin myplugin.py with the option foo_opt set to BAR like this

â—‰ tony@tony:~/clnlive:
$ l1-cli -k plugin subcommand=start plugin=$(pwd)/myplugin.py foo_opt=BAR

where l1-cli is an alias for lightning-cli --lightning-dir=/tmp/l1-regtest, we expect myplugin method called with the parameters foo1=bar1 and foo2=bar2 to gives us the following

â—‰ tony@tony:~/clnlive:
$ l1-cli -k myplugin foo1=bar1 foo2=bar2
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "BAR"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}

where the value of node_id is the ID of the node l1.

This should help us understand:

  1. how startup options are passed to the plugin,
  2. how cli parameters are passed to the plugin and
  3. how to communicate to the node l1 via JSON-RPC over unix sockets.

Setup

Here is my setup:

â—‰ tony@tony:~/clnlive:
$ lightningd --version | xargs printf "lightningd %s\n" && python --version && lsb_release -ds
lightningd v23.02.2
Python 3.10.6
Ubuntu 22.04.2 LTS

Start 2 Lightning nodes running on regtest

Let's start two Lightning nodes running on the Bitcoin regtest chain by sourcing the script lightning/contrib/startup_regtest.sh provided CLN repository and running the command start_ln:

â—‰ tony@tony:~/clnlive:
$ source lightning/contrib/startup_regtest.sh
lightning-cli is /usr/local/bin/lightning-cli
lightningd is /usr/local/bin/lightningd
Useful commands:
  start_ln 3: start three nodes, l1, l2, l3
  connect 1 2: connect l1 and l2
  fund_nodes: connect all nodes with channels, in a row
  stop_ln: shutdown
  destroy_ln: remove ln directories
â—‰ tony@tony:~/clnlive:
$ start_ln
Bitcoin Core starting
awaiting bitcoind...
Making "default" bitcoind wallet.
[1] 63973
[2] 64007
WARNING: eatmydata not found: instal it for faster testing
Commands:
        l1-cli, l1-log,
        l2-cli, l2-log,
        bt-cli, stop_ln, fund_nodes

We can check that l1-cli is just an alias for lightning-cli with the base directory being /tmp/l1-regtest:

â—‰ tony@tony:~/clnlive:
$ alias l1-cli
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'

To be sure that we have at least a lightning node running on regtest, we can call the subcommand getinfo like this:

â—‰ tony@tony:~/clnlive:
$ l1-cli getinfo
{
   "id": "030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542",
   "alias": "SILENTMONKEY",
   "color": "030120",
   "num_peers": 0,
   "num_pending_channels": 0,
   "num_active_channels": 0,
   "num_inactive_channels": 0,
   "address": [],
   "binding": [
      {
         "type": "ipv4",
         "address": "127.0.0.1",
         "port": 7171
      }
   ],
   "version": "v23.02.2",
   "blockheight": 1,
   "network": "regtest",
   "fees_collected_msat": 0,
   "lightning-dir": "/tmp/l1-regtest/regtest",
   "our_features": {
      "init": "08a000080269a2",
      "node": "88a000080269a2",
      "channel": "",
      "invoice": "02000000024100"
   }
}

What is a CLN plugin?

A plugin is a subprocess started by lightningd daemon which can interact with lightningd in several ways. JSON-RPC command passthrough is one of them, it allows a plugin to add its own commands to the JSON-RPC interface.

Once started a plugin communicates with lightningd through its stdin and stdout. Specifically,

  1. JSON-RPC requests to lightningd calling methods defined by the plugin are sent by lightningd to the plugin's stdin and
  2. the plugin answers to lightningd's requests by writing valid JSON-RPC responses to the plugin's stdout.

List of builtin CLN plugins

The plugin mechanism is not just a way to enhance CLN without modifying its implementation, it is how some of the features inside CLN are added and implemented, it is part of the design.

For instance when we run the command line lightning-cli pay ... to pay an invoice, under the hood we are using the plugin pay which is a subprocess of lightningd main process.

Let's see it.

As we have two nodes running, we can check for their processes by running the following:

â—‰ tony@tony:~/clnlive:
$ ps -ax | rg lightningd
  63975 pts/0    S      0:00 lightningd --lightning-dir=/tmp/l1-regtest
  64009 pts/0    S      0:00 lightningd --lightning-dir=/tmp/l2-regtest
  64158 pts/0    S+     0:00 rg lightningd

Then we can print the tree of the processes whose root is the lightningd process associated to the node l1:

â—‰ tony@tony:~/clnlive:
$ pstree 63975
lightningd─┬─autoclean
           ├─bcli
           ├─bookkeeper
           ├─chanbackup
           ├─commando
           ├─funder
           ├─keysend
           ├─lightning_conne
           ├─lightning_gossi
           ├─lightning_hsmd
           ├─offers
           ├─pay
           ├─spenderp
           ├─sql
           ├─topology
           └─txprepare

The subprocesses lightning_... (lightning_connectd, lightning_gossipd, lightning_hsmd) are subdaemons of lightningd (see subdaemons).

The other subprocesses (autoclean, …, txprepare) are builtin plugins implemented in CLN and spawned by lightningd (see plugins_set_builtin_plugins_dir).

We also can list plugins using the subcommand plugin (see json_plugin_control). Here is how we can read its man page documentation:

â—‰ tony@tony:~/clnlive:
$ l1-cli help plugin
LIGHTNING-PLUGIN(7)                                             LIGHTNING-PLUGIN(7)

NAME
       lightning-plugin -- Manage plugins with RPC

SYNOPSIS
       plugin subcommand [plugin|directory] [options] ...

DESCRIPTION
       The  plugin RPC command command can be used to control dynamic plugins, i.e.
       plugins that declared themself "dynamic" (in getmanifest).

       subcommand can be start, stop, startdir, rescan or list and determines  what
       action is taken

       plugin is the path or name of a plugin executable to start or stop

       directory is the path of a directory containing plugins

       options are optional keyword=value options passed to plugin, can be repeated

       subcommand  start takes a path to an executable as argument and starts it as
       plugin.  path may be an absolute path or a path relative to the plugins  di-
       rectory  (default  ~/.lightning/plugins).   If the plugin is already running
       and the  executable  (checksum)  has  changed,  the  plugin  is  killed  and
       restarted  except  if  its  an important (or builtin) plugin.  If the plugin
       doesn't complete the "getmanifest" and "init" handshakes within 60  seconds,
       the  command  will  timeout  and kill the plugin.  Additional options may be
       passed to the plugin, but requires all  parameters  to  be  passed  as  key-
       word=value  pairs,  for  example:  lightning-cli  -k plugin subcommand=start
       plugin=helloworld.py greeting='A crazy' (using the  -k|--keyword  option  is
       recommended)

and here is how we can list the running plugins (see plugin_dynamic_list_plugins):

â—‰ tony@tony:~/clnlive:
$ l1-cli plugin list
{
   "command": "list",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bcli",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/commando",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/funder",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/topology",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/keysend",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/offers",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/pay",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/txprepare",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/spenderp",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/sql",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
         "active": true,
         "dynamic": false
      }
   ]
}

Note that some are dynamic plugins like pay plugin and some are not like commando.

Implementation of myplugin.py dynamic plugin

Make the plugin executable

We are going to write our plugin in the file myplugin.py. So far this file only import the library we need:

#!/usr/bin/python
# -*- mode: python -*-

import sys
import os
import json
import socket

When we start a plugin dynamically, the path of the plugin can be:

  • relative (in this case it will be expanded against the plugin directory of the lightning node),
  • or absolute.

So using a relative path in our case won't work as we see:

â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start myplugin.py
{
   "code": -32602,
   "message": "/tmp/l1-regtest/plugins/myplugin.py is not executable: No such file or directory"
}

So using pwd, we can make the path to the plugin absolute, but now we get another error:

â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "code": -32602,
   "message": "/home/tony/clnlive/myplugin.py is not executable: Permission denied"
}

Plugin files must be executable to be started, so we do it using chmod and we try to start our plugin again which still doesn't work because our plugin does nothing so far and so return before respondind to the getmanifest request sent by lightningd:

â—‰ tony@tony:~/clnlive:
$ chmod a+x myplugin.py
â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to getmanifest"
}

This is what we expected. Let's continue.

How to print stuff in order to understand our system?

The communication that happens between lightningd and the plugin uses stdin and stdout of the plugin script. So using print function to print stuff out to understand or debug or program won't work as we can see by adding the line print("foo") in myplugin.py like this:

#!/usr/bin/python
# -*- mode: python -*-

import sys
import os
import json
import socket

printout("foo")

Indeed, if we now run the following

â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to getmanifest"
}

we get the same error as before with no string foo printed.

To be able to get some informations from our system, we add the helper function printout that print (append) strings to the file /tmp/myplugin_out.

#!/usr/bin/python
# -*- mode: python -*-

import sys
import os
import json
import socket

myplugin_out="/tmp/myplugin_out"
if os.path.isfile(myplugin_out):
    os.remove(myplugin_out)
def printout(s):
    with open(myplugin_out, "a") as output:
        output.write(s)

printout("foo")

Then by running the following

â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to getmanifest"
}

we still get the same error but we've also printed foo in the file /tmp/myplugin_out:

# /tmp/myplugin_out

foo

getmanifest request

When we start the plugin, at the beginning the plugin receives a getmanifest request in its input ended by two newline \n\n. We modify myplugin.py to get that request and we print out that request to see how it looks like:

#+BEGIN_SRC python
#!/usr/bin/python
# -*- mode: python -*-

import sys
import os
import json
import socket

myplugin_out="/tmp/myplugin_out"
if os.path.isfile(myplugin_out):
    os.remove(myplugin_out)
def printout(s):
    with open(myplugin_out, "a") as output:
        output.write(s)

# getmanifest

request = sys.stdin.readline()
sys.stdin.readline() # "\n"
printout(request)

We try to start the plugin

â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to getmanifest"
}

and here is the getmanifest request:

# /tmp/myplugin_out

{"jsonrpc": "2.0", "id": 86, "method": "getmanifest", "params": {"allow-deprecated-apis": false}}

If we prettify it we get:

{
  "jsonrpc": "2.0",
  "id": 86,
  "method": "getmanifest",
  "params": {
    "allow-deprecated-apis": false
  }
}

The params field indicate that the node l1 doesn't allow deprecated APIs which match with its config file:

# /tmp/l1-regtest/config

network=regtest
log-level=debug
log-file=/tmp/l1-regtest/log
addr=localhost:7171
allow-deprecated-apis=false

reload script and entr unix utility

In the live we use the following reload script to restart the plugin myplugin.py:

#!/bin/env bash

plugin_path=$(pwd)/myplugin.py
L1_CLI='lightning-cli --lightning-dir=/tmp/l1-regtest'

if [[ -n $($L1_CLI plugin list | rg $plugin_path) ]]; then
     $L1_CLI plugin stop $plugin_path
fi

$L1_CLI plugin start $plugin_path

We use it with entr unix utility like this:

â—‰ tony@tony:~/clnlive:
$ ls | entr -s './reload'
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to getmanifest"
}
bash returned exit code 1

This allows to restart the plugin each time a file change in the current directory automatically. This interesting for development.

response to getmanifest request

In myplugin.py, we store the getmanifest request id in the variable req_id. In the variable manifest, we construct our response to the getmanifest request. This is were we tell lightningd that:

  • we want our plugin to be dynamic,
  • our plugin has a startup option foo_opt with the default value bar and,
  • we declare a methode named myplugin.

We send that request to lightningd by writing our response to our stdout.

#!/usr/bin/python
# -*- mode: python -*-

import sys
import os
import json
import socket

myplugin_out="/tmp/myplugin_out"
if os.path.isfile(myplugin_out):
    os.remove(myplugin_out)
def printout(s):
    with open(myplugin_out, "a") as output:
        output.write(s)

# getmanifest

request = sys.stdin.readline()
sys.stdin.readline() # "\n"
# printout(request)

req_id = json.loads(request)["id"]

manifest = {
    "jsonrpc": "2.0",
    "id": req_id,
    "result": {
        "dynamic": True,
        "options": [{
            "name": "foo_opt",
            "type": "string",
            "default": "bar",
            "description": "description"
        }],
        "rpcmethods": [{
            "name": "myplugin",
            "usage": "",
            "description": "description"
        }]
    }
}

sys.stdout.write(json.dumps(manifest))
sys.stdout.flush()

Now when we try to start our plugin, we no longer have getmanifest error but the following init error:

â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to init"
}

This is what we expected because once we answer to the getmanifest request, lightningd send us back the init request to which we didn't answer.

init request

The init request that lightningd send us tells us that we can now communicate together. This request also contains information about the startup option of the plugin and the configuration of the lightning node.

Let's add a printout statement to see the content of the init request:

# getmanifest
...
# init
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
printout(request)

The file /tmp/myplugin_out now contains the init request

# /tmp/myplugin_out

{"jsonrpc": "2.0", "id": "cln:init#151", "method": "init", "params": {"options": {"foo_opt": "bar"}, "configuration": {"lightning-dir": "/tmp/l1-regtest/regtest", "rpc-file": "lightning-rpc", "startup": false, "network": "regtest", "feature_set": {"init": "08a000080269a2", "node": "88a000080269a2", "channel": "", "invoice": "02000000024100"}}}}

that we can prettify like this:

{
  "jsonrpc": "2.0",
  "id": "cln:init#151",
  "method": "init",
  "params": {
    "options": {
      "foo_opt": "bar"
    },
    "configuration": {
      "lightning-dir": "/tmp/l1-regtest/regtest",
      "rpc-file": "lightning-rpc",
      "startup": false,
      "network": "regtest",
      "feature_set": {
        "init": "08a000080269a2",
        "node": "88a000080269a2",
        "channel": "",
        "invoice": "02000000024100"
      }
    }
  }
}

response to init request

As in our example we do nothing special at that stage, we just answe to the init request with and empty result field. This is enough for lightningd to continue its communication with us. To send that response, we write it to our stdout as we did before:

# getmanifest
...
# init
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
# printout(request)

jsreq = json.loads(request)
req_id = jsreq["id"]

init = {
    "jsonrpc": "2.0",
    "id": req_id,
    "result": {}
}

sys.stdout.write(json.dumps(init))
sys.stdout.flush()

Now we no longer get errors when we try to start the plugin:

$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bcli",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/commando",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/funder",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/topology",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/keysend",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/offers",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/pay",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/txprepare",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/spenderp",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/sql",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
bash returned exit code 0

As myplugin.py is listed in the previous output, we might think that we've successfully started our plugin and defined myplugin method. Let's find out and run the following command:

â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
{
   "code": -32601,
   "message": "Unknown command 'myplugin'"
}

This error is normal, because after answering to the init request, myplugin.py does nothing and stop running. This is the moment where we have to add an IO loop that listen for incoming communication from lightningd.

IO loop

  • make the plugin works

    Let's modify myplugin.py to wait for incoming request in its stdin and add a printout statement to look at the type of request we receive from lightningd when we run l1-cli myplugin:

    # init
...
# io loop

for request in sys.stdin:
    sys.stdin.readline() # "\n"
    printout(request)

    Now, after restarting the plugin, when we run

    â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
^C

    we get

    # /tmp/myplugin_out

{"jsonrpc": "2.0", "method": "myplugin", "id": "cli:myplugin#65781/cln:myplugin#208", "params": []}

    which looks like this when prettified:

    {
  "jsonrpc": "2.0",
  "method": "myplugin",
  "id": "cli:myplugin#65781/cln:myplugin#208",
  "params": []
}

    Two things to notice:

    1. as we don't answer to that incoming request, the command line l1-cli myplugin doesn't return,
    2. the field params in the incoming request is empty because we passed no argument at the command line.

    Let's answer to that incoming request with a response whose result field is the string "foo". As we did before, to send that response to lightningd we write it to our stdout:

    # init
...
# io loop

for request in sys.stdin:
    sys.stdin.readline() # "\n"
    # printout(request)
    jsreq = json.loads(request)
    req_id = jsreq["id"]

    resp = {
        "jsonrpc": "2.0",
        "id": req_id,
        "result": "foo"
    }

    sys.stdout.write(json.dumps(resp))
    sys.stdout.flush()

    After restarting the plugin, when we run the following and see foo printed:

    â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
"foo"

    We are close to achieve our goal.

    Let's modify the response to get something that looks like what we want to achieve:

    # init
...
# io loop

for request in sys.stdin:
    sys.stdin.readline() # "\n"
    # printout(request)
    jsreq = json.loads(request)
    req_id = jsreq["id"]

    result = {
        "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
        "option": {
            "foo_opt": "BAR"
        },
        "cli_params": {
            "foo1": "bar1",
            "foo2": "bar2"
        }
    }

    resp = {
        "jsonrpc": "2.0",
        "id": req_id,
        "result": result
    }

    sys.stdout.write(json.dumps(resp))
    sys.stdout.flush()

    After restarting the plugin, here's what we get:

    â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "BAR"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}

  • startup options

    The information about startup option are in the init request. Let's defined the variable foo_opt containing the value of our startup option. Once defined, we can use it in the responses in the IO loop:

    ...
# init
...
sys.stdout.write(json.dumps(init))
sys.stdout.flush()

foo_opt = jsreq["params"]["options"]["foo_opt"]

# io loop

for request in sys.stdin:
    sys.stdin.readline() # "\n"
    # printout(request)
    jsreq = json.loads(request)
    req_id = jsreq["id"]

    result = {
        "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
        "option": {
            "foo_opt": foo_opt
        },
        "cli_params": {
            "foo1": "bar1",
            "foo2": "bar2"
        }
    }
    resp = {
        "jsonrpc": "2.0",
        "id": req_id,
        "result": result
    }

    sys.stdout.write(json.dumps(resp))
    sys.stdout.flush()

    Let's restart the plugin with the option foo_opt set to BAAAR like this:

    â—‰ tony@tony:~/clnlive:
$ l1-cli -k plugin subcommand=start plugin=$(pwd)/myplugin.py foo_opt=BAAAR
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      ...
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}

    Now when we call myplugin subcommand the option field is set taking the startup option into account:

    â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "BAAAR"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}

  • cli parameters

    The cli parameters are in the params field of the incoming request. We defined cli_params to store those parameters and use it in the result object:

    ...
# io loop

for request in sys.stdin:
    sys.stdin.readline() # "\n"
    # printout(request)
    jsreq = json.loads(request)
    req_id = jsreq["id"]
    cli_params = jsreq["params"]

    result = {
        "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
        "option": {
            "foo_opt": foo_opt
        },
        "cli_params": cli_params
    }
    resp = {
        "jsonrpc": "2.0",
        "id": req_id,
        "result": result
    }

    sys.stdout.write(json.dumps(resp))
    sys.stdout.flush()

    After restarting the plugin, we call the subcommand myplugin with different cli parameters and see in consequence cli_params field changing:

    â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": []
}
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin bar1 bar2
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": [
      "bar1",
      "bar2"
   ]
}
â—‰ tony@tony:~/clnlive:
$ l1-cli -k myplugin foo1=bar1 foo2=bar2
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}

  • get l1 node id via unix socket

    • using lightning-cli

      The last information we want is the node id. This is something that we can get at the command line like this using getinfo:

      â—‰ tony@tony:~/clnlive:
$ l1-cli getinfo
{
   "id": "030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542",
   "alias": "SILENTMONKEY",
   "color": "030120",
   "num_peers": 0,
   "num_pending_channels": 0,
   "num_active_channels": 0,
   "num_inactive_channels": 0,
   "address": [],
   "binding": [
      {
         "type": "ipv4",
         "address": "127.0.0.1",
         "port": 7171
      }
   ],
   "version": "v23.02.2",
   "blockheight": 1,
   "network": "regtest",
   "fees_collected_msat": 0,
   "lightning-dir": "/tmp/l1-regtest/regtest",
   "our_features": {
      "init": "08a000080269a2",
      "node": "88a000080269a2",
      "channel": "",
      "invoice": "02000000024100"
   }
}
â—‰ tony@tony:~/clnlive:
$ l1-cli getinfo | jq .id -r
030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542

    • using nc

      We don't have to use lightning-cli to communicate with lightningd. We can directly send to lightningd request via the node's unix socket file.

      Let's do it first at the terminal using nc utility:

      â—‰ tony@tony:~/clnlive:
$ file /tmp/l1-regtest/regtest/lightning-rpc
/tmp/l1-regtest/regtest/lightning-rpc: socket
â—‰ tony@tony:~/clnlive:
$ nc -U /tmp/l1-regtest/regtest/lightning-rpc
{"jsonrpc": "2.0", "id": "1", "method": "getinfo", "params": []}
{"jsonrpc":"2.0","id":"1","result":{"id":"030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542","alias":"SILENTMONKEY","color":"030120","num_peers":0,"num_pending_channels":0,"num_active_channels":0,"num_inactive_channels":0,"address":[],"binding":[{"type":"ipv4","address":"127.0.0.1","port":7171}],"version":"v23.02.2","blockheight":1,"network":"regtest","fees_collected_msat":0,"lightning-dir":"/tmp/l1-regtest/regtest","our_features":{"init":"08a000080269a2","node":"88a000080269a2","channel":"","invoice":"02000000024100"}}}

{"jsonrpc": "2.0", "id": "1", "method": "getinfo", "params": [], "filter": {"id": true}}
{"jsonrpc":"2.0","id":"1","result":{"id":"030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542"}}

^C

      Note that in the second request we used a filter and so received only the id in the result field.

    • in Python with socket library

      Here we add the python code needed to communicate via unix socket with lightningd in order to get the node id using the methode getinfo. We receive the information necessary to construct the socket file path in the init request.

      ...
# getmanifest
...
# init
...
foo_opt = jsreq["params"]["options"]["foo_opt"]
lightning_dir = jsreq["params"]["configuration"]["lightning-dir"]
rpc_file = jsreq["params"]["configuration"]["rpc-file"]
socket_file = os.path.join(lightning_dir,rpc_file)

# io loop

for request in sys.stdin:
    sys.stdin.readline() # "\n"
    # printout(request)
    jsreq = json.loads(request)
    req_id = jsreq["id"]
    cli_params = jsreq["params"]

    getinfo = {
        "jsonrpc": "2.0",
        "id": "1",
        "method": "getinfo",
        "params": [],
        "filter": {"id": True}
    }

    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
        s.connect(socket_file)
        s.sendall(bytes(json.dumps(getinfo), encoding="utf-8"))
        getinfo_resp = s.recv(4096)

    node_id = json.loads(getinfo_resp)["result"]["id"]
    result = {
        "node_id": node_id,
        "option": {
            "foo_opt": foo_opt
        },
        "cli_params": cli_params
    }
    resp = {
        "jsonrpc": "2.0",
        "id": req_id,
        "result": result
    }

    sys.stdout.write(json.dumps(resp))
    sys.stdout.flush()

      After restarting the plugin, we can check that we get the correct node id:

      â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
{
   "node_id": "030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": []
}
â—‰ tony@tony:~/clnlive:
$ l1-cli getinfo | jq .id -r
030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542
â—‰ tony@tony:~/clnlive:
$ l1-cli -k myplugin foo1=bar1 foo2=bar2
{
   "node_id": "030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}

      We are done!

Terminal sessions

Terminal 1

We ran the following commands in this order:

$ lightningd --version | xargs printf "lightningd %s\n" && python --version && lsb_release -ds
$ source lightning/contrib/startup_regtest.sh
$ start_ln
$ alias l1-cli
$ l1-cli getinfo
$ ps -ax | rg lightningd
$ pstree 63975
$ l1-cli help plugin
$ l1-cli plugin list
$ l1-cli plugin start myplugin.py
$ l1-cli plugin start $(pwd)/myplugin.py
$ chmod a+x myplugin.py
$ l1-cli plugin start $(pwd)/myplugin.py
$ ls | entr -s './reload'
$ l1-cli plugin start $(pwd)/myplugin.py
$ ls | entr -s './reload'
$ l1-cli plugin start $(pwd)/myplugin.py
$ l1-cli myplugin
$ ls | entr -s './reload'
$ alias l1-cli
$ ls | entr -s './reload'
$ l1-cli plugin stop $(pwd)/myplugin.py
$ l1-cli -k plugin subcommand=start plugin=$(pwd)/myplugin.py foo
$ l1-cli -k plugin subcommand=start plugin=$(pwd)/myplugin.py foo=bar
$ l1-cli -k plugin subcommand=start plugin=$(pwd)/myplugin.py foo_opt=BAAAR
$ ls | entr -s './reload'
$ ls | entr -s './reload'

And below you can read the terminal session (command lines and outputs):

â—‰ tony@tony:~/clnlive:
$ lightningd --version | xargs printf "lightningd %s\n" && python --version && lsb_release -ds
lightningd v23.02.2
Python 3.10.6
Ubuntu 22.04.2 LTS
â—‰ tony@tony:~/clnlive:
$ source lightning/contrib/startup_regtest.sh
lightning-cli is /usr/local/bin/lightning-cli
lightningd is /usr/local/bin/lightningd
Useful commands:
  start_ln 3: start three nodes, l1, l2, l3
  connect 1 2: connect l1 and l2
  fund_nodes: connect all nodes with channels, in a row
  stop_ln: shutdown
  destroy_ln: remove ln directories
â—‰ tony@tony:~/clnlive:
$ start_ln
Bitcoin Core starting
awaiting bitcoind...
Making "default" bitcoind wallet.
[1] 63973
[2] 64007
WARNING: eatmydata not found: instal it for faster testing
Commands:
        l1-cli, l1-log,
        l2-cli, l2-log,
        bt-cli, stop_ln, fund_nodes
â—‰ tony@tony:~/clnlive:
$ alias l1-cli
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'
â—‰ tony@tony:~/clnlive:
$ l1-cli getinfo
{
   "id": "030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542",
   "alias": "SILENTMONKEY",
   "color": "030120",
   "num_peers": 0,
   "num_pending_channels": 0,
   "num_active_channels": 0,
   "num_inactive_channels": 0,
   "address": [],
   "binding": [
      {
         "type": "ipv4",
         "address": "127.0.0.1",
         "port": 7171
      }
   ],
   "version": "v23.02.2",
   "blockheight": 1,
   "network": "regtest",
   "fees_collected_msat": 0,
   "lightning-dir": "/tmp/l1-regtest/regtest",
   "our_features": {
      "init": "08a000080269a2",
      "node": "88a000080269a2",
      "channel": "",
      "invoice": "02000000024100"
   }
}
â—‰ tony@tony:~/clnlive:
$ ps -ax | rg lightningd
  63975 pts/0    S      0:00 lightningd --lightning-dir=/tmp/l1-regtest
  64009 pts/0    S      0:00 lightningd --lightning-dir=/tmp/l2-regtest
  64158 pts/0    S+     0:00 rg lightningd
â—‰ tony@tony:~/clnlive:
$ pstree 63975
lightningd─┬─autoclean
           ├─bcli
           ├─bookkeeper
           ├─chanbackup
           ├─commando
           ├─funder
           ├─keysend
           ├─lightning_conne
           ├─lightning_gossi
           ├─lightning_hsmd
           ├─offers
           ├─pay
           ├─spenderp
           ├─sql
           ├─topology
           └─txprepare
â—‰ tony@tony:~/clnlive:
$ l1-cli help plugin
LIGHTNING-PLUGIN(7)                                             LIGHTNING-PLUGIN(7)

NAME
       lightning-plugin -- Manage plugins with RPC

SYNOPSIS
       plugin subcommand [plugin|directory] [options] ...

DESCRIPTION
       The  plugin RPC command command can be used to control dynamic plugins, i.e.
       plugins that declared themself "dynamic" (in getmanifest).

       subcommand can be start, stop, startdir, rescan or list and determines  what
       action is taken

       plugin is the path or name of a plugin executable to start or stop

       directory is the path of a directory containing plugins

       options are optional keyword=value options passed to plugin, can be repeated

       subcommand  start takes a path to an executable as argument and starts it as
       plugin.  path may be an absolute path or a path relative to the plugins  di-
       rectory  (default  ~/.lightning/plugins).   If the plugin is already running
       and the  executable  (checksum)  has  changed,  the  plugin  is  killed  and
       restarted  except  if  its  an important (or builtin) plugin.  If the plugin
       doesn't complete the "getmanifest" and "init" handshakes within 60  seconds,
       the  command  will  timeout  and kill the plugin.  Additional options may be
       passed to the plugin, but requires all  parameters  to  be  passed  as  key-
       word=value  pairs,  for  example:  lightning-cli  -k plugin subcommand=start
       plugin=helloworld.py greeting='A crazy' (using the  -k|--keyword  option  is
       recommended)

â—‰ tony@tony:~/clnlive:
$ l1-cli plugin list
{
   "command": "list",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bcli",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/commando",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/funder",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/topology",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/keysend",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/offers",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/pay",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/txprepare",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/spenderp",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/sql",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
         "active": true,
         "dynamic": false
      }
   ]
}
â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start myplugin.py
{
   "code": -32602,
   "message": "/tmp/l1-regtest/plugins/myplugin.py is not executable: No such file or directory"
}
â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "code": -32602,
   "message": "/home/tony/clnlive/myplugin.py is not executable: Permission denied"
}
â—‰ tony@tony:~/clnlive:
$ chmod a+x myplugin.py
â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to getmanifest"
}
â—‰ tony@tony:~/clnlive:
$ ls | entr -s './reload'
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to getmanifest"
}
bash returned exit code 1
...
bash returned exit code 1
Traceback (most recent call last):
  File "/home/tony/clnlive/myplugin.py", line 26, in <module>
    "id": req_id,
NameError: name 'req_id' is not defined
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to getmanifest"
}
bash returned exit code 1
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to getmanifest"
}
bash returned exit code 1
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to init"
}
bash returned exit code 1
â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to init"
}
â—‰ tony@tony:~/clnlive:
$ ls | entr -s './reload'
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: exited before replying to init"
}
bash returned exit code 1
...
bash returned exit code 1
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bcli",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/commando",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/funder",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/topology",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/keysend",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/offers",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/pay",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/txprepare",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/spenderp",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/sql",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
bash returned exit code 0
â—‰ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bcli",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/commando",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/funder",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/topology",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/keysend",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/offers",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/pay",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/txprepare",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/spenderp",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/sql",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
{
   "code": -32601,
   "message": "Unknown command 'myplugin'"
}
â—‰ tony@tony:~/clnlive:
$ ls | entr -s './reload'
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bcli",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/commando",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/funder",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/topology",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/keysend",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/offers",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/pay",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/txprepare",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/spenderp",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/sql",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
bash returned exit code 0
...
bash returned exit code 0
â—‰ tony@tony:~/clnlive:
$ alias l1-cli
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'
â—‰ tony@tony:~/clnlive:
$ ls | entr -s './reload'
{
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      ...
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
bash returned exit code 0
...
bash returned exit code 0
â—‰ tony@tony:~/clnlive:
$ l1-cli plugin stop $(pwd)/myplugin.py
{
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
â—‰ tony@tony:~/clnlive:
$ l1-cli -k plugin subcommand=start plugin=$(pwd)/myplugin.py foo
lightning-cli: Expected key=value in 'foo': Success
â—‰ tony@tony:~/clnlive:
$ l1-cli -k plugin subcommand=start plugin=$(pwd)/myplugin.py foo=bar
{
   "code": -3,
   "message": "/home/tony/clnlive/myplugin.py: unknown parameter \"foo\""
}
â—‰ tony@tony:~/clnlive:
$ l1-cli -k plugin subcommand=start plugin=$(pwd)/myplugin.py foo_opt=BAAAR
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      ...
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
â—‰ tony@tony:~/clnlive:
$ ls | entr -s './reload'
{
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      ...
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
bash returned exit code 0
...
bash returned exit code 0
â—‰ tony@tony:~/clnlive:
$ ls | entr -s './reload'
{
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      ...
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
bash returned exit code 0
...
bash returned exit code 0
Traceback (most recent call last):
  File "/home/tony/clnlive/myplugin.py", line 85, in <module>
    s.connect(...)
TypeError: a bytes-like object is required, not 'ellipsis'
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      ...
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
bash returned exit code 0
Traceback (most recent call last):
  File "/home/tony/clnlive/myplugin.py", line 86, in <module>
    s.sendall(...)
TypeError: a bytes-like object is required, not 'ellipsis'
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      ...
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
bash returned exit code 0
Traceback (most recent call last):
  File "/home/tony/clnlive/myplugin.py", line 86, in <module>
    s.sendall(json.dumps(getinfo))
TypeError: a bytes-like object is required, not 'str'
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      ...
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
bash returned exit code 0
{
   "command": "stop",
   "result": "Successfully stopped myplugin.py."
}
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      ...
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
bash returned exit code 0
...
bash returned exit code 0
â—‰ tony@tony:~/clnlive:
$

Terminal 2

We ran the following commands in this order:

$ alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'
$ l1-cli myplugin
$ while true; do sleep 1; l1-cli myplugin; done
$ l1-cli myplugin
$ l1-cli myplugin
$ while true; do sleep 1; l1-cli myplugin; done
$ l1-cli myplugin
$ l1-cli myplugin eeee
$ l1-cli myplugin
$ l1-cli myplugin bar1 bar2
$ l1-cli -k myplugin foo1=bar1 foo2=bar2
$ while true; do sleep 1; l1-cli myplugin; done
$ l1-cli myplugin
$ l1-cli myplugin bar1 bar2
$ l1-cli -k myplugin foo1=bar1 foo2=bar2
$ l1-cli getinfo
$ l1-cli getinfo | jq .id -r
$ file /tmp/l1-regtest/regtest/lightning-rpc
$ nc -U /tmp/l1-regtest/regtest/lightning-rpc
$ while true; do sleep 1; l1-cli myplugin; done
$ l1-cli myplugin
$ l1-cli getinfo | jq .id -r
$ l1-cli -k myplugin foo1=bar1 foo2=bar2

And below you can read the terminal session (command lines and outputs):

â—‰ tony@tony:~/clnlive:
$ alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
^C
â—‰ tony@tony:~/clnlive:
$ while true; do sleep 1; l1-cli myplugin; done
{
   "code": -4,
   "message": "Plugin terminated before replying to RPC call."
}
{
   "code": -4,
   "message": "Plugin terminated before replying to RPC call."
}
"foo"
"foo"
"foo"
...
"foo"
^C
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
"foo"
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
"foo"
â—‰ tony@tony:~/clnlive:
$ while true; do sleep 1; l1-cli myplugin; done
"foo"
...
"foo"
{
   "foo": "bar"
}
...
{
   "foo": "bar"
}
{
   "foo": "BAR"
}
...
{
   "foo": "BAR"
}
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "BAR"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
...
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "BAR"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
...
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
^C
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "BAAAR"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin eeee
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin bar1 bar2
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
â—‰ tony@tony:~/clnlive:
$ l1-cli -k myplugin foo1=bar1 foo2=bar2
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
â—‰ tony@tony:~/clnlive:
$ while true; do sleep 1; l1-cli myplugin; done
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
...
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": []
}
...
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": []
}
^C
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": []
}
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin bar1 bar2
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": [
      "bar1",
      "bar2"
   ]
}
â—‰ tony@tony:~/clnlive:
$ l1-cli -k myplugin foo1=bar1 foo2=bar2
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}
â—‰ tony@tony:~/clnlive:
$ l1-cli getinfo
{
   "id": "030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542",
   "alias": "SILENTMONKEY",
   "color": "030120",
   "num_peers": 0,
   "num_pending_channels": 0,
   "num_active_channels": 0,
   "num_inactive_channels": 0,
   "address": [],
   "binding": [
      {
         "type": "ipv4",
         "address": "127.0.0.1",
         "port": 7171
      }
   ],
   "version": "v23.02.2",
   "blockheight": 1,
   "network": "regtest",
   "fees_collected_msat": 0,
   "lightning-dir": "/tmp/l1-regtest/regtest",
   "our_features": {
      "init": "08a000080269a2",
      "node": "88a000080269a2",
      "channel": "",
      "invoice": "02000000024100"
   }
}
â—‰ tony@tony:~/clnlive:
$ l1-cli getinfo | jq .id -r
030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542
â—‰ tony@tony:~/clnlive:
$ file /tmp/l1-regtest/regtest/lightning-rpc
/tmp/l1-regtest/regtest/lightning-rpc: socket
â—‰ tony@tony:~/clnlive:
$ nc -U /tmp/l1-regtest/regtest/lightning-rpc
{"jsonrpc": "2.0", "id": "1", "method": "getinfo", "params": []}
{"jsonrpc":"2.0","id":"1","result":{"id":"030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542","alias":"SILENTMONKEY","color":"030120","num_peers":0,"num_pending_channels":0,"num_active_channels":0,"num_inactive_channels":0,"address":[],"binding":[{"type":"ipv4","address":"127.0.0.1","port":7171}],"version":"v23.02.2","blockheight":1,"network":"regtest","fees_collected_msat":0,"lightning-dir":"/tmp/l1-regtest/regtest","our_features":{"init":"08a000080269a2","node":"88a000080269a2","channel":"","invoice":"02000000024100"}}}

{"jsonrpc": "2.0", "id": "1", "method": "getinfo", "params": [], "filter": {"id": true}}
{"jsonrpc":"2.0","id":"1","result":{"id":"030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542"}}

^C
â—‰ tony@tony:~/clnlive:
$ while true; do sleep 1; l1-cli myplugin; done
{
   "code": -4,
   "message": "Plugin terminated before replying to RPC call."
}
{
   "code": -32601,
   "message": "Unknown command 'myplugin'"
}
...
{
   "code": -32601,
   "message": "Unknown command 'myplugin'"
}
{
   "code": -4,
   "message": "Plugin terminated before replying to RPC call."
}
{
   "code": -32601,
   "message": "Unknown command 'myplugin'"
}
...
{
   "code": -32601,
   "message": "Unknown command 'myplugin'"
}
{
   "code": -4,
   "message": "Plugin terminated before replying to RPC call."
}
{
   "code": -32601,
   "message": "Unknown command 'myplugin'"
}
...
{
   "code": -32601,
   "message": "Unknown command 'myplugin'"
}
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": []
}
...
{
   "node_id": "029999ce6a1b74cf077f503b81bb4d95af34114363bdac5922a9b353ce0449bac4",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": []
}
^C
â—‰ tony@tony:~/clnlive:
$ l1-cli myplugin
{
   "node_id": "030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": []
}
â—‰ tony@tony:~/clnlive:
$ l1-cli getinfo | jq .id -r
030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542
â—‰ tony@tony:~/clnlive:
$ l1-cli -k myplugin foo1=bar1 foo2=bar2
{
   "node_id": "030120a23ddec2d04dbd7d108300dee4ddcf915dd54f09b7e3db3565cd31e08542",
   "option": {
      "foo_opt": "bar"
   },
   "cli_params": {
      "foo1": "bar1",
      "foo2": "bar2"
   }
}

Source code

myplugin.py

#!/usr/bin/python
# -*- mode: python -*-

import sys
import os
import json
import socket

myplugin_out="/tmp/myplugin_out"
if os.path.isfile(myplugin_out):
    os.remove(myplugin_out)
def printout(s):
    with open(myplugin_out, "a") as output:
        output.write(s)

# getmanifest

request = sys.stdin.readline()
sys.stdin.readline() # "\n"
# printout(request)

req_id = json.loads(request)["id"]

manifest = {
    "jsonrpc": "2.0",
    "id": req_id,
    "result": {
        "dynamic": True,
        "options": [{
            "name": "foo_opt",
            "type": "string",
            "default": "bar",
            "description": "description"
        }],
        "rpcmethods": [{
            "name": "myplugin",
            "usage": "",
            "description": "description"
        }]
    }
}

sys.stdout.write(json.dumps(manifest))
sys.stdout.flush()

# init
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
# printout(request)

jsreq = json.loads(request)
req_id = jsreq["id"]

init = {
    "jsonrpc": "2.0",
    "id": req_id,
    "result": {}
}

sys.stdout.write(json.dumps(init))
sys.stdout.flush()

foo_opt = jsreq["params"]["options"]["foo_opt"]
lightning_dir = jsreq["params"]["configuration"]["lightning-dir"]
rpc_file = jsreq["params"]["configuration"]["rpc-file"]
socket_file = os.path.join(lightning_dir,rpc_file)

# io loop

for request in sys.stdin:
    sys.stdin.readline() # "\n"
    # printout(request)
    jsreq = json.loads(request)
    req_id = jsreq["id"]
    cli_params = jsreq["params"]

    getinfo = {
        "jsonrpc": "2.0",
        "id": "1",
        "method": "getinfo",
        "params": [],
        "filter": {"id": True}
    }

    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
        s.connect(socket_file)
        s.sendall(bytes(json.dumps(getinfo), encoding="utf-8"))
        getinfo_resp = s.recv(4096)

    node_id = json.loads(getinfo_resp)["result"]["id"]
    result = {
        "node_id": node_id,
        "option": {
            "foo_opt": foo_opt
        },
        "cli_params": cli_params
    }
    resp = {
        "jsonrpc": "2.0",
        "id": req_id,
        "result": result
    }

    sys.stdout.write(json.dumps(resp))
    sys.stdout.flush()

reload

#!/bin/env bash

plugin_path=$(pwd)/myplugin.py
L1_CLI='lightning-cli --lightning-dir=/tmp/l1-regtest'

if [[ -n $($L1_CLI plugin list | rg $plugin_path) ]]; then
     $L1_CLI plugin stop $plugin_path
fi

$L1_CLI plugin start $plugin_path

foo.json

{
  "jsonrpc": "2.0",
  "id": 86,
  "method": "getmanifest",
  "params": {
    "allow-deprecated-apis": false
  }
}

{
  "jsonrpc": "2.0",
  "id": "cln:init#151",
  "method": "init",
  "params": {
    "options": {
      "foo_opt": "bar"
    },
    "configuration": {
      "lightning-dir": "/tmp/l1-regtest/regtest",
      "rpc-file": "lightning-rpc",
      "startup": false,
      "network": "regtest",
      "feature_set": {
        "init": "08a000080269a2",
        "node": "88a000080269a2",
        "channel": "",
        "invoice": "02000000024100"
      }
    }
  }
}

{
  "jsonrpc": "2.0",
  "method": "myplugin",
  "id": "cli:myplugin#65781/cln:myplugin#208",
  "params": []
}

bar.json

{
  "jsonrpc": "2.0",
  "id": "1",
  "method": "getinfo",
  "params": [],
  "filter": {
    "id": true
  }
}

Resources