OT Node Support for Mainnet

Running an OT Node is not just install and forget about it. There are some tasks you should do on a regular basis to be sure everything is fine with your OT Node and that it's able to bid for offers and hopefully also win some of those.

This is basically a two step process, a technical and a financial check.

Technical Check

Check Port via OT Hub (discontinued) or Web Browser

Your OT Node has to be accessable by other nodes to work properly. OT Hub implemented an easy way to check this. By adding your node to My Nodes you get the option to check if your node is responding on the port other nodes want to communicate on.

OT Hub - My Nodes

Another way is by just typing a specific URL into your web browser. If the IP address of your server is 20.30.40.50 use this URL to check if your server is running. OT Node listens to port 5278 by default. If you used a different port during installation you have to change this accordingly:

https://20.30.40.50:5278

If your web browser shows an immediate answer with a message like this below, your OT Node most probably works fine. You can ignore this warning since OT Nodes use their own secure protocol which your web browser doesn't know. Page not secure

Check Docker Container

If you followed my guide to install OT Node then you installed it as a Docker container. To check if it's currently running type this:

docker ps
If you get something like this everything is fine:
CONTAINER ID        IMAGE                                                COMMAND                  CREATED             STATUS              PORTS                                                                    NAMES
8bfff3300796        quay.io/origintrail/otnode-mariner:release_mariner   "sh /ot-node/testnet…"   27 hours ago        Up 10 hours         0.0.0.0:3000->3000/tcp, 0.0.0.0:5278->5278/tcp, 0.0.0.0:8900->8900/tcp   otnode
If you don't see a process running something is wrong. You should definitely check the log (see below).

Starting, stopping and restarting OT Node can be done by

docker start otnode
docker stop otnode
docker restart otnode

Check OT Node Configuration

To be sure your node was configured correctly, runs the latest software and is on the right network, check it by

docker exec otnode curl -s 'http://127.0.0.1:8900/api/info'

to get this information:

{
  "blockchain": "Ethereum",
  "erc_725_identity": "0xe7250123456789abcdef0123456789abcdef0123",
  "is_bootstrap": false,
  "network": {
    "contact": {
      "agent": "1.0.0",
      "hostname": "your.server.ip.address",
      "index": 5535,    # generated during setup, specific for each node
      "network_id": "MarinerV1.0.1",
      "port": 5278,
      "protocol": "https:",
      "wallet": "0xeaddfedcba9876543210fedcba9876543210fedc",
      "xpub": "xpub3748ab572eaf3483d0f457aB23845..." # shortened
    },
    "identity": "abcdef0123456789abcdef0123456789abcdef01"
  },
  "node_wallet": "0xeaddfedcba9876543210fedcba9876543210fedc",
  "version": "2.0.50"
}

Make sure your node runs software version 2.0.50 or higher. Also check the other yellow highlighted data.

You can also crosscheck on Github, which version is the latest released by this command

curl -s 'https://raw.githubusercontent.com/OriginTrail/ot-node/release/mariner/package.json' | jq -r ".version"

If you get a higher version with this query then your node has installed, you can force the node to install it via docker restart otnode but OT Node checks periodically if there's a new version and will then update itself.

Check the Log

OT Node logs pretty much every step it does. Check the log by

docker logs otnode

This shows you the whole log file which may take some time depending on how long your node has been running already.

So there's a better way to look into your log. You can specify to skip older entries and just show the last two days (=48h). And to be able to skip through the log by Space bar, add | more (which 'pipes' the output to the program more)

docker logs --since 48h otnode | more

To find log entries of the last 5 hours with sentence "failed to process" you simply use this:

docker logs --since 5h otnode | grep -i "failed to process" | more
This command pipes the whole log of the last 5 hours to grep where it is filtered by our search string, -i means case-insensitive. The filtered output is passed to more which let you skip trough it.

If you just want to follow (-f) the log realtime and also get the last 90 minutes, use this command:

docker logs --since 90m -f otnode
Press CTRL-C to close the log view.

You can find out the file size of the OT Node log by

du -h $(docker inspect -f '{{.LogPath}}' otnode)

Check free Disk Space

Sometimes programs can go crazy and may write huge log files. To be sure your server has enough free disk space, check it from time to time:

df -h

Filesystem      Size  Used Avail Use% Mounted on
udev            463M     0  463M   0% /dev
tmpfs            99M  772K   98M   1% /run
/dev/vda1        25G  7.1G   18G  30% /
tmpfs           493M     0  493M   0% /dev/shm
...
If /dev/vda1 uses more than 80% you should definitely check your server.

If OT Node log file becomes huge (several GB) you need to handle this. There's probably errors which are responsible for these many entries which should be identified in the first place before get rid of the log to make free space.

If OT Node log file size is huge and you're sure you want to empty it completely, run:

truncate -s 0 $(docker inspect -f '{{.LogPath}}' otnode)

Inside Docker container OT Node also writes log files on a regular basis. But there every hour the current log file will be compressed and a new log file is created. To get rid of old - no longer needed - log files use the following but be careful, rm is a dangerous command, for example * (asterisk) at the wrong place can destroy your whole system. Good practice would be to delete files of just one specific month in the past at a time.

To delete the log files inside OT Node Docker container from May 2019 run:

docker exec otnode sh -c "rm /ot-node/current/logs/otnode-2019-05*"

Keep your Server up-to-date

From time to time there are updates for your server. Depending on type and version of the operating system you may see such comments when you log in. If you see something like this

12 packages can be updated.
 3 updates are security updates.
it's time to update your server.

In this case or if you use a system which doesn't notify you, use this command to check updates by

apt update
Before you do the actual upgrade check if Docker is part of the upgrade by

apt list --upgradable

If you see Docker (in any variation) in the list make sure our node is idle by checking the log since the upgrade will stop your node (which runs on Docker). You start the upgrade by

apt upgrade -y

After upgrade: Check if OT Node is running.

If Docker was included in the upgrade OT Node has to be started again manually by docker start otnode

Depending on the upgrade the system can remind you that it's necessary to reboot your server. In this case do the reboot also only when your node is idle.

Financial Check

Check OT Node Balance

OT Node offers an in-build command to query the balance of your profile:

docker exec otnode curl -s 'http://127.0.0.1:8900/api/balance?humanReadable=true'
{
  "profile": {
    "minimalStake": "1000",        # TRAC transferred during setup to be able to run a node
    "reserved": "50",              # Reserved amount of TRAC from currently running jobs
    "staked": "1800"               # Overall amount of TRAC in the profile
  },
  "wallet": {
    "address": "0xfc94463D3A0F323cE09B6dAa0f2a89830396003a",  # Operational (OP) Wallet address
    "ethBalance": "0.18578332",                               # ETH balance of OP wallet
    "tokenBalance": "100"                                     # Amount of TRAC token in OP wallet
  }
}

You need enough TRAC in your profile!

Make sure there is always enough TRAC staked in your profile since only staked - reserved - minimalStake (in this case 1800 - 50 - 1000 = 750) is available for jobs.

Also your OP Wallet needs enough ETH for payouts (and probably other tasks). Fees on Ethereum depend on the amount of gas a specific transaction needs and the gas price set by your node. Smart contract functions can need much more gas than regular transactions. You can estimate from previous transactions when it's time to refill your wallet.

Check Offers and Payouts

The main goal to run an OT Node is to bid for offers and win jobs to hold data for an agreed time span. To keep track of those events you can follow the log of your node. But there's a much better way, use OT Hub (discontinued) which offers a ton of features additional to offers and payouts. There's also a Discord Server where you can follow all offers and payouts but also register your own ERC725 identity if you want to get notified every time your node won an offer or a payout happened.

Additionally you can see each payout on etherscan.io if you query your OP wallet.

All Addresses at a Glance

With all those addresses and maybe more than one running OT node you can easily mix them up. With this command you get all addresses at a glance and it's a great example to build a script (next chapter) for:

echo -e "\nERC725 ID   : $(jq -r '.identity' ~/erc725_identity.json)\n
OP Wallet   : $(jq -r '.node_wallet' ~/.origintrail_noderc)\n
Mgmt Wallet : $(jq -r '.management_wallet' ~/.origintrail_noderc)\n"

All Checks at Once

All Checks described above can be done with the script otcheck which can be installed with one single command.

Some Simple Scripting

You probably want to use some commands on a regular basis. To type the whole command every time is not really efficient. So let's make things a bit easier.

Imagine, you want to check the log by just typing otl. For this you have to create a file with this name. To keep things organized let's do this in a separate folder called scripts.

mkdir -p ~/scripts    # Make directory scripts (if it doesn't already exist)

# This adds the scripts folder to the known PATH
grep -q "~/scripts" ~/.profile || echo 'export PATH=$PATH:~/scripts' >> ~/.profile
source ~/.profile

The following four steps you have to repeat for every script you want to create.

Step 1 - Let's open the editor nano with file otl in scripts folder

nano ~/scripts/otl
Step 2 - Copy/paste following lines into the editor

#!/bin/bash
if [ -n "$1" ]
then
    docker logs --since 48h otnode | grep -i "$1" | more
else
    docker logs -f --since 60m otnode
fi

Step 3 - Save the script by CTRL-O Enter and exit nano by CTRL-X.

Step 4 - Set Execute file permissions.

chmod +x ~/scripts/otl
Now you are able to call this new command - even with an additional parameter

otl                   # follow log in realtime - close with CTRL-C
otl important         # show all log lines which include 'important' within the last 2 days
otl "t been chosen"   # show all lines with 't been chosen' within the last 2 days

Learn more...

Have fun creating new scripts. If you want to learn more about scripting, google for bash shell script