Planet import taking too long on placex data loading #3335
Replies: 5 comments 17 replies
-
|
During this stage of the import seeing all CPU cores at maximum, for several days, is expected. What are the last lines of the import log? Which rank does it process? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Thanks for replying. The last lines of the log are: And the I'm not sure about the best way to check for deadlocks. Please let me know if this is good enough: Apparently no deadlocks. |
Beta Was this translation helpful? Give feedback.
-
|
I see, so it's still in the loading phase, hasn't started indexing yet. Deadlocks wouldn't happen here. Actually taking days wouldn't happen either, I'd expect 1-2 hours. |
Beta Was this translation helpful? Give feedback.
-
|
Let me give that a check. |
Beta Was this translation helpful? Give feedback.
-
|
This sounds really slow I would point to the "2 system" scenario or a generic slowness either storage or network (is there traffic ?) What is Does syslog (both machines) or postgres log show something ? Can you test with import and DB on one VM to narrow this down ? |
Beta Was this translation helpful? Give feedback.
-
|
i didnt use threads and osm2pgsql-cache param postgres i used the params for a "64 gb ram machine" from https://nominatim.org/release-docs/latest/admin/Installation/ |
Beta Was this translation helpful? Give feedback.
-
|
I have both Thanks for the hints. Having a better understanding of the time scale helps already. I'll keep you posted! |
Beta Was this translation helpful? Give feedback.
-
|
Do not disable autovacuum. It does an important job of updating statistics in the background. |
Beta Was this translation helpful? Give feedback.
-
|
@lonvia i started an import and after ~20 hours it was at Rank 26 with ETA (of this rank) > 100.000 seconds Yes, its an NVMe Storage, 4 discs as Raid-10, this makes an impact compared to a VM but 100kb/s as he measured don't point directly to i/o overload |
Beta Was this translation helpful? Give feedback.
-
|
As @lonvia pointed out, my |
Beta Was this translation helpful? Give feedback.
-
|
The stage you are in is really just a copy from one (huuge) table to another. There is an INSERT trigger involved but it shouldn't be doing very complicated things. If it has been hanging for 5 days though, I suspect that something in the trigger is going wrong. This is really tricky to debug. First of all, can you check which triggers are installed on the |
Beta Was this translation helpful? Give feedback.
-
|
Hi, And checking the |
Beta Was this translation helpful? Give feedback.
-
|
Interesting, this section shouldn't be in there during import. Here is what the function should look like: If your function is longer and contains the highlighted section, you can cancel your import now. It will never finish. Have you done something unusual likes stopping and restarting the import? |
Beta Was this translation helpful? Give feedback.
-
|
To restart the process:
|
Beta Was this translation helpful? Give feedback.
-
I have not. My import script looks like this: Every time I attempt to do the import, a drop database always happens before so I'd expect to always start clean... |
Beta Was this translation helpful? Give feedback.
-
|
I believe I found the culprit. |
Beta Was this translation helpful? Give feedback.
-
|
Good morning folks, After a few hiccups (some infra related issues causing my VMs to reboot), everything went well.
For a next trial, I believe it will be a good idea to give more CPUs to the machine used for the loading (and reduce the database side as well). Everything was being done with a parallelism of 8. The database side was definitely underused all the time. Thank you for the help! :) |
Beta Was this translation helpful? Give feedback.
-
|
Sounds about right for the setup. I would expect that you don't even need to change the specification of the loader machine. Simply run the import with |
Beta Was this translation helpful? Give feedback.
-
|
Initially, as I pointed in the beginning of the discussion, I was setting the number of threads ( |
Beta Was this translation helpful? Give feedback.
-
|
Hi Team, Yesterday I have kicked off import process and found myself also into the same situation where I see it taking too long time, almost more than 6hr at "Load data into placex table...." (osm2pgsql took 48419s (13h 26m 59s) overall.)
But in my case, I have got a brand new cloud VM to start installing for the first time. However, I notice, based on entire conversation in this thread, that Im not using machine resources properly. It would be great if someone could review my machine details and what is the best way to fully utilize the resources to get he installation of nominatim done at the earliest. (especially postgres parameter, import command parameters also cleaning up database to start fresh import) Environment: I have installed both postgres database and nominatim on the same cloud vm, where postgres database data directory is pointed to another mounted drive having 5TB space. 370 GB RAM memory Postgres database parameters: Commands executed: /usr/local/bin/nominatim import -j 6 --osm-file /srv/nominatim/build/planet-latest.osm.pbf 2>&1 | tee setup.log -Prem |
Beta Was this translation helpful? Give feedback.
-
|
(Adding a new question as an answer to an older closed discussion will lead to confusion) I've seen "osm2pgsql took" with 7 hours on good non-VM hardware. The biggest factor is if you use a flatnode file, for a planet import you should (https://nominatim.org/release-docs/latest/admin/Import/#flatnode-files). Another factor is which data was selected, the so called style (https://nominatim.org/release-docs/latest/admin/Import/#filtering-imported-data) which can make comparing one import to another harder. osm2pgsql is single threaded at this step so what counts is raw CPU processor power plus how fast IO is. That's harder to determine with cloud providers, many don't mention processor clock speeds. If you have the option use the fastest IO, I think at AWS that's measured in IOPS (higher = faster) and ideally the IO should be on the machine not network mounted. The next steps "Starting rank 3" are multithreaded. I'd increase the |
Beta Was this translation helpful? Give feedback.
-
Hi folks,
I'm new to Nominatim and I'm currently trying to do a full Planet import. However, the import process is on the placex table loading for 5 days already. I can see I have a few cores completely maxed to while the actual disk writes are on the kB/s order.
At this point I'm out of ideas and given each attempt takes multiple days, randomly trying is very unproductive :(
Let me share with you my setup and where the process is currently at.
Do you have any ideas on how to debug or speed up this process? Any hints would be much appreciated :)
Thank you!
Environment
The whole process is running on GCP VMs: one for the postgres database, another for the Nominatim import process.
Postgres VM:
Postgres details:
Nominatim import VM:
Process
I've used the following Nominatim options:
In terms of extra data I'm including these in the project directory:
And, finally, I'm running this command:
Status
From the log of
nominatim importI can see the following:The VM running the
nominatim importis currently completely idle just waiting for postgres.On the postgres side I can see it is running 7 queries that look like:
7 CPUs of this VM are at 100%. In terms of IO the writes are at about 100 kB/s.
Currently the data disk has 412 GB occupied.
Beta Was this translation helpful? Give feedback.
All reactions