DB::flush_count overflow #185
Comments
|
I was surprised it took this long for a complaint. I did some work on this over a month ago but it is incomplete; I will pick it up again soon. |
|
Is that this problem? |
|
Yes. |
|
I'm not a programmer. Practically what is the problem? |
|
ElectrumX is not a wallet. It's a public blockchain index. Your question doesn't make a lot of sense. |
|
I'm sorry, I made a mistake! |
|
@erasmospunk - can you try the compact-hist script from 2f26e81. Once I have a couple of reports it works I'll have it run in the background of a running server. |
|
@kyuupichan I didn't keep the databases that had this issue but I run it on Dash that is near the limit: The db was 743M before compaction and 589M after that. I am also running Digibyte that will hit the limit in about a week. |
|
@kyuupichan update for Dash and Digibyte that exceeded the 65535 flush count, the compaction worked correctly. |
|
Thanks! I'll get something done so it isn't a separate script. |
|
@kyuupichan how do I run compact-hist ? |
|
@zilveer yes, but remember to set the same environment variables as when you run electrumx normally |
|
Hi kyuupichan, Running the compact_history script on a regular basis takes time and effort. Thanks, |
|
Same issue for Groestlcoin. @erasmospunk I heared from Jackie that you are adding Groestlcoin into Coinomi. Since that will need electrumx, it might affect Coinomi also. |
|
@kyuupichan is there any automated process for this now? |
|
@erasmospunk @kyuupichan how do I set same environment variables as I have for ElectrumX? any help is appreciated. |
|
@zilveer no, you need to run the script manually |
|
I just hit this on my Litecoin electrumx. Running compact_history.py resolved the issue. Is this safe to run while the server is operational, or does the service need to be stopped (which is what i did this time cause of the error anyways) |
|
It has to be stopped as only one process can open the DB at a time. |
|
Thanks for the quick reply. Now the question is how can we streamline this so an admin doesn't have to do maint on coins every month or so. |
|
I was seeing the |
|
The answer is to do history differently. I'm working on that, but no ETA. |
|
Also ran into this for the first time and was quite confused. Good to hear you are working on this! |
- the object also manages unflushed history - the history DB has its own version, starting at 0 This is the first step to making history management into a separate service and to larger block sizes. The next step is an improved history format, which I hope will both save space and be a solution to issue #185. The DB should be able to upgrade in-place without re-syncing the chain.
|
I get this problem when i run the script like this (Environment variables are exported from electrumx.conf): Error: In the script there are these both lines: What am I doing wrong? |
|
I'm now hit by this issue too (ElectrumX 1.14.0 running inside Docker). Is there now an easy way to fix this? Answering to myself: I ran |
|
@schildbach as stated before by other guys, you need to schedule this to run "often". By "often", I mean: it will depend on the network you'll be using (testnet, mainnet...) In an enterprise context, on testnet, we need to schedule a compact history every 2 weeks but we do this every week, to keep it consistent with mainnet, which needs to be compacted every week to be sure it's gonna keep on living without any crash. This number may vary in the future, as it already varied. The biggest the blockchain the more often you need to run it ? That's what has been said previously in this thread. Let's hope we won't have to do this every day :p Note that you can directly use |
|
Afaik you have to stop the server before compacting. It's not nice to have
to schedule downtime regularly. Compacting can also take quite a long time
too.
Imho this should really be fixed, as it is a hassle to deal with and many
people run into this by surprise.
…On Tue, Apr 28, 2020, 18:22 pandaatrail ***@***.***> wrote:
@schildbach <https://github.com/schildbach> as stated before by other
guys, you need to schedule this to run "often".
By "often", I mean: it will depend on the network you'll be using
(testnet, mainnet...)
In an enterprise context, on testnet, we need to schedule a compact
history every *2 weeks* but we do this every week, to keep it consistent
with mainnet, which needs to be compacted every week to be sure it's gonna
keep on living without any crash.
This number may vary in the future, as it already varied. The biggest the
blockchain the more often you need to run it ? That's what has been said
previously in this thread. Let's hope we won't have to do this every day :p
Note that you can directly use docker-compose exec electrumx
electrumx_compact_history without the need to enter the service manually.
This will help with scripting and cron.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#185 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJLEM6TULROFKFCZS5O5RDRO37F7ANCNFSM4DLNIPCA>
.
|
Because if you have to do anything inside the container while the process is stopped, you have to do it in a side car launched with the exact same parameters, env, and volumes. If you stop PID 1 in a docker container, the container dies. Yet another reason I really dislike docker. |
|
@nyetwurk just use the same |
For a complex container started with a deployment tool like ansible that becomes a total mess. Docker still inexpicably lacks a reliable way to convert |
|
If you want to get rid of regular database compaction tasks, use the electrs server. Electrs implements automatic database compaction. It has been a month since it started up, but it is operating stably. |
|
Hello,
I am mstill doing manual compaction .Is there a tutorial on how to implement the change? Or can you help?
Thanks
Adrian
From: FujiCoin [mailto:[email protected]]
Sent: June 13, 2020 7:14 AM
To: kyuupichan/electrumx <[email protected]>
Cc: OracolXor <[email protected]>; Comment <[email protected]>
Subject: Re: [kyuupichan/electrumx] DB::flush_count overflow (#185)
If you want to get rid of regular database compaction tasks, use the electrs server. Electrs implements automatic database compaction. It has been a month since it started up, but it is operating stably.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#185 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGYMHRM2VKXAOPQJIOYTVWLRWNNP5ANCNFSM4DLNIPCA> . <https://github.com/notifications/beacon/AGYMHRPQIKARWMDC6S3D3XDRWNNP5A5CNFSM4DLNIPCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEZOLCCA.gif>
|
|
Electrs github: If you use Docker-based installation, you can compile it without fail, but it's a little troublesome. Manual installation may not compile depending on the OS version. Ubuntu 18.04 will compile without problems. If that doesn't work, there is a way to get the binaries compiled with docker out of the container. |
|
Failed to do history compaction with same error that I got for electrumx daemon using 1.15.0 |
|
@vlddm I think that AssertionError suggests that your DB got corrupted. Unfortuntely you probably need to sync from scratch. |
|
Running electrumx in docker; and after upgrading to 1.15 I got bumped into this bug. What I did to workaround is these steps:
If we're gonna live with this compact_history stuff it'd be great to have such a pause for service merged into the electrumx_server script. |
|
I am considering making some changes to the db schema, and as part of that have been looking at this issue. Currently the history db schema is: I'm thinking it could be changed to: This would completely do away with flush_id and compaction; although I am not sure how it would affect performance. Re db size on disk, I guess it would increase it somewhat, hard to tell by how much though as it depends on address reuse and whether the db is compacted:
What do you think @kyuupichan ? To be clear, this is about the history db, so for current reference size (depends on when last compaction was though!), see: |
|
Seems worth trying, let me know how it goes. |
I also get this error, not sure what to do. Running in Debian w/ systemD, with LevelDb. Piping in environment with this command: export $(cat /etc/electrumx.conf | xargs) && python3.7 /var/lib/electrumx/elect_compact.py It seems like something is missing to define server.env, or possibly must I change server.env to something that matches my configuration? The electrumx server operates and executes normally otherwise, outside of database compaction issues. |
|
I was able to resolve this issue with the script mentioned in this post: It seems that the script in use above has since changed and is no longer applicable to our environment. |
|
I encountered this same struct 'H' format 65536 bug with a testnet BitcoinSegwit setup. The compact_history script completed, but I see the same error when restarting the service. EDIT: Re-running it a second time seems to have fixed all errors and the binary is running. |
|
May 15, 2017 |
|
One possible terrible hack (since it is pretty clear this isn't going to be fixed): run compact on startup if the db exists. That way, if it crashes, it will just run the compact script when it is run agian. |
|
Again, why not build compaction into electrumx itself? So at least it can be done regularly (and possibly even automatically on startup) so it can continue to be run in docker. |
|
Do you have a patch? |
I would if spesmilo/electrumx#88 (comment) worked And you wouldn't like it much, because it would likely be a hack to the docker container to just run it unconditionally on start up via a bash script |
and others
I got a case where the
DB::flush_countwent over 65535 and a full database resync was needed.Normally the
flush_countwould overflow after about 455 days in Bitcoin if we were indexing a fully synced node. However this is not the case if the node is not fully synced and ElectrumX manages to catch up the full node. In that scenario ElectrumX will quickly exhaust the 16bit space of theflush_countvariable as it flushes to disk after each block.Also several coins have shorter block intervals thus hitting the limit in as little as a couple of months.
The text was updated successfully, but these errors were encountered: