Provider Rewards
💰 Earning Rewards
After staking on chain, providers serve consumers and get cryptographic proofs of relay service. These proofs are submitted on chain to accumulate CUs that lead to token rewards. The calculation for provider rewards is complex and contingent on the validity of submitted proofs. However, given valid proofs, rewards are mostly affected by 3 factors:
- ☑️ Passable Quality of Service - The individual score of a relay session. Passable QoS directly alters rewards in a given session.
- ⭐ Quality of Service Excellence - The reputational score of a provider. QoS Excellence affects provider selection both within a pairing and in the pairing probabilities.
- 🚨 Jail - The disabling mechanism for inferior/defunct providers. Jailing prevents a provider from giving service for a period of time.
📊 Passable QoS
Passable Quality of Service is scored separately in each relay session. Lower scores mean lower rewards. Up to half the accumulated CU can be reduced for bad service. Passable QoS metrics can be viewed both in the Lava Info explorer and Prometheus metrics.
Passable QoS is binary, either it's good or bad there is no in-between. Scores in the range 0-1 are a result of averaging that binary score across relays. You can learn more about Passable QoS from our 📄 RSCH-1000 research paper.
Metrics 📈
Passable Quality of Service divides into three metrics: Availability, Sync/Freshness of data, Latency.
🛎️ Availability
Score 0 or 1 per relay, averaged across relays in a session to give a range 0-1 for each session. 0 is given for each failed relay.
A lower availability score can be the result of failed relays by one of the following:
Low-Score Causes
❗time out
✅ Timeout can be checked for in the provider logs, requests that time out are also identified by the provider and terminated
❗ unsupported chain
❗ errors
✅ Error can be checked in the provider logs, and prometheus metrics for disabled chains
Improvements
Improving availability involves finding the cause of errors, and taking the necessary actions to resolve them.
⚡ add more endpoints
⏲️ Latency
Score 0 or 1 per relay, averaged across relays in a session to give a range 0-1 per session. 0 is given for each relay that took above half the timeout. A
lower latency score can be the result of slow responses, and can be identified by turning on debug logs in the provider to see the latency or checking Prometheus.
Improvements
⚡ activate lavap cache
⚡ set the provider service in proximity to the node
⚡ set the correct geolocation
⚡ reduce server load or improve the machine
⚡ add more endpoints
🌿 Sync/freshness
Score 0 or 1 per relay, averaged across relays in a session to give a range 0-1 per session. A 0 is given for each relay that has a latest block proof that is older than the spec allowed block lag for QoS sync.
How to Identify
The freshness proofs are updated by the provider service in a GET_BLOCKNUM request, and then returned together with consumer relay responses. It is possible to turn on debug logs and see the blocks advancing. The latest block is exported in prometheus, and also uploaded on chain. You can compare the latest block on your provider to other providers, a useful way to do this is filter provider_latest_block_report and compare your results to others:
Here and below ensure that you replace {PUBLIC_RPC} with the correct endpoint.
lavap test events 2000 --event lava_provider_latest_block_report --node {PUBLIC_RPC}
Low-Score Causes
❗ provider state is not updating fast enough
❗ latest block is too old
📊 QoS Excellence
QoS Excellence is calculated very similarly to Passable QoS. QoS Excellence provides a range of scores that are time-weighted to take the latest information all the actions mentioned here to improve passable QoS affect excellence
Metrics 📈
Excellence Quality of score divides into 3 metrics:
- Availability - score in the range
0-1 - Sync/ Freshness of data - how much time behind other providers are we, lower is better,
0means your sync is the best in the pairing - Latency - how many benchmark ticks passed during a relay in average (time taken / benchmark time). lower is better
🏛️ Jailing
How to Fix Getting Jailed 🔓
Lava Protocol removes providers that are providing inferior service. The mechanism is detached from QoS measurements. In order to avoid being jailed, a provider needs to avoid the following:
- have the staked endpoint not respond to connections
- have disabled chains in the staked endpoint
- have too many consecutive errors with a large group of consumers
- have a non TLS connection or an expired certificate
- block headers or origins
- miss on getting rewards
How Jailing Happens ❓
Once one or more of the aforementioned conditions are met, Lava's Blockchain jails a provider if:
- there are enough other providers in the spec
- the provider is not frozen (if you freeze for maintenance you will not get jailed)
- the provider is active for at least 8 epochs
- in the last 8 epochs the provider got less sum of rewards than reports sum in the last 2 epochs
Identify Getting Jailed 🩻
If your provider got jailed it will stop receiving requests upon the next epoch.
It is possible to monitor this event via info webpage or the following commands :
⌨️ command when being reported before getting jailed:
lavap test events 2000 --event lava_provider_reported --node {PUBLIC_RPC}
a group of reports can lead to being jailed.
⌨️ command when the blockchain's criteria for jail are met:
lavap test events 2000 --event lava_provider_jailed --node {PUBLIC_RPC}
Resuming Service ▶️
Since version 0.27.0, providers that are jailed can resume service by unfreezing. It is planned to have a cooldown period on unfreezing.
If repeated jails are activated in a short period:
lavad tx pairing unfreeze --help
Reasons ⚖️
Jail reports contain additional info on the report reason and they can be either due to:
- disconnections - a provider did not respond to connection attempts
- errors - a provider's responses were a sequence of consecutive errors.
In addition, the reports contain an exact time tag, so the provider can check the events:
lavap test events 2000 --event lava_provider_reported --node {PUBLIC_RPC}
Disconnections 💢
These mean the provider's endpoint did not respond and can be due to the following problems:
- TLS certificate outdated or not set
- misconfiguration proxying the requests to the provider service
- provider service not running
- wrong endpoint in the stake entry on chain: can be fixed by running
lavad tx pairing modify-provider ${CHAIN} --endpoints "${ENDPOINTS}" --geolocation ${GEOLOCATION} --from ${WALLET}
Make sure the url in the endpoint is the provider grpc listening address and not your node url
⏺️ Identifying a Disconnect
Disconnection problems can be identified by running the test command:
lavap test rpcprovider ${PUBLIC_ADDRESS}
Errors ❌
these mean the provider service connection was solid but all relays turned to be errors, might be caused by the following:
- disabled chain - the provider doesnt have access to the node, or a verification does not pass, and the chain is disabled
- unexpected errors
- timeouts
⏺️ Identifying an Error
Errors can be identified by looking at the provider service logs, it is recommended to run with debug if repeatedly getting jailed.