Is Crawling required again, for changing the folder structure path in Windows

[NivethikaM]

I have created a folder namely E:/index/test/ and i have added 10 files in it. I have indexed all the files in Elastic Search (fs.url: E:/index/test/) . Now i want to change the path of the folder structure from E:/Index/test/ to C:/index/test/. I have changed the fs.url to C:/index/test and copied all the files from E:/index/test/ to C:/index/test/.

In Elastic Search i have changed the path.real and file.url from E:/index/test/${filename} to C:/index/test/${filename} using update_by_query. Also made sure path.virutal is present as required.

Now when i am crawling for the same index from the different path C:/index/test/, all the files are added again and the number of files has been increased from 10 to 20. Though the content is same, path.real, path.virtual and file.url is same for the files, it is indexed again.

Is is possible to change the folder structure path for the same index in windows?

Fscrawler Version - 2.9
Elastic - 6.8.0

[dadoonet]

Hi @NivethikaM

There's no such a feature in FSCrawler.
That requires indeed to apply an update by query.

I think you were almost there. When you change the path to a file, you need to compute again:

• path.real
• file.url
• path.root

This last one represents the folder where the file is so it needs to be updated accordingly.
It's the encoded version of C:/index/test/.

Also, I think you need to update the folder index.
It contains documents like:

      {
        "_index": "fscrawler_fs_crawler_test_sub_dirs_i_t_test_subdirs_deep_tree_folder",
        "_type": "_doc",
        "_id": "5e6e88529a54132361ca34bc2c6b",
        "_score": 1,
        "_source": {
          "path": {
            "root": "68d15b87d1545c8425478dd9b7b59ac3",
            "virtual": "/",
            "real": "/var/folders/xn/47mdpxd12vq4zrjhkwbhd5_r0000gn/T/junit1606348347133366394/resources/test_subdirs_deep_tree"
          },
          "file": {
            "content_type": "text/directory",
            "filename": "test_subdirs_deep_tree"
          }
        }
      }

You need to update the document _id and path.root.

If you succeed in making that happen, I'd really appreciate if you could document exactly what you did.
This might be a nice feature to have in an FSCrawler admin tool.

[NivethikaM]

Thanks for your quick reply.

I have updated the path.root in the Index as you have mentioned.

Though i have mentioned index_folders as true in the '_settings.json', i am not able to update Path.root & _id in the Index_folder. I am having the mapping for the index_folder as mentioned above by you and the number of shards & replicas is mentioned as '1' in the Index_Folder/_settings. But i am not having any data when i queried the Index_folder/_search. I have almost 30 index and none of the index_folder is having any data in it (total is showing 0).

[dadoonet]

What is the output of

GET /_cat/indices?v

[NivethikaM]

Now i am getting the data in Index_Folder. The problem is when we are giving the path as "C:/index/test/" in fs.url, we need to have sub folders under it. If we are having only files in the path, we won't get the data in the Index_Folder. So, sub-directories needs to be present after the 'fs.url' path.

In the Index_folder, the value for the _id and the root is unique for each file location and this data is provided from fscrawler.

Test 1

1. For Example: The _id generated is '67ab' for the path.real 'C:/index/test/verify' & virtual 'verify'. I have deleted this record in the index_folder. Now i have re run with the same path, now also the _id generated is same i.e. '67ab' for the path.real 'C:/index/test/verify' & virtual 'verify'.
   Test 2
2. I have changed the path.real from 'D:/index/test/verify' to 'C:/index/test/verify' in the index_folder. Now the _id is changed to '67ab' in the Index_folder from another _id value.

According to my testing, _id and root value of index_folder is pushed by fscrawler for each path and this value is stored in the machine.

[dadoonet]

Could you run a test with some few documents?

One that you run from C:/index/test/ and the other from D:/index/test/?
In 2 different indices like test-c and test-d.

And then compare what are the values in both indices for the same folders?

[NivethikaM]

I have created 3 indices namely tmp_both, tmp_a & tmp_b.

The tmp_both_folder index has 2 records in it.

1. _id: '47cce77fc87eb153015eb17e1553579' and root: 'bc765182a9418d49299c6b31533edc96' for the path.real 'D:/index/test/verify' and path.virtual 'verify'. (fs.url: D:/index/test/)
2. _id: '6dd2bfd4ae85d09ffc4c6f3a44cad81' and root: '78486e9dba9752b4d5b8fcbec74d73c' for the path.real 'C:/index/test/verify' and path.virtual: 'verify'. (fs.url: C:/index/test/)
   During crawling the records of tmp_both are taking the _id of the tmp_both_folder as path.root based on the folder path.

The tmp_a_folder has 1 record in it.

1. _id: '47cce77fc87eb153015eb17e1553579' and root: 'bc765182a9418d49299c6b31533edc96' for the path.real 'D:/index/test/verify' and path.virtual 'verify'. (fs.url: D:/index/test/)
   The _id of the tmp_a_folder index is same as the _id of tmp_both_folder.

The tmp_b_folder has 1 record in it.

1. _id: '6dd2bfd4ae85d09ffc4c6f3a44cad81' and root: '78486e9dba9752b4d5b8fcbec74d73c' for the path.real 'C:/index/test/verify' and path.virtual: 'verify'. (fs.url: C:/index/test/)
   The _id of the tmp_b_folder index is same as the _id of tmp_both_folder.

So each path has some unique _id and root values. If we use the same path in multiple index then also the _id is same.

The values in the index are taking the _id of folder index as path.root based on the indexing path.

[NivethikaM]

@dadoonet Please let me know whether it will be possible to change the folder Path?

[dadoonet]

All what you said is correct.

So if you want to move a folder from D:/index/test/verify to C:/index/test/verify, you need to update tmp_a_folder so _id: 47cce77fc87eb153015eb17e1553579 and root: bc765182a9418d49299c6b31533edc96 are both replaced by _id: 6dd2bfd4ae85d09ffc4c6f3a44cad81 and root: 78486e9dba9752b4d5b8fcbec74d73c.

[NivethikaM]

For moving a folder from D:/index/test/verify to C:/index/test/verify, I have done the following:

Since it is not possible to update the _id in Elastic, I have deleted the _id: 47cce77fc87eb153015eb17e1553579 in tmp_a_folder. Now added a document with the _id : 6dd2bfd4ae85d09ffc4c6f3a44cad81 and root: 78486e9dba9752b4d5b8fcbec74d73c in the tmp_a_folder index. Also make sure that all the file.filename, path.root, path.virtual and path.real is correct.

Updated the path.root, path.real and file.url in the tmp_a index. Now also changed the url in the _settings.json file.

I added the document present in the tmp_a index to the new url path manually.
Now, when i crawled again the tmp_a index, the document already present is crawled again.

I have tested thrice in different ways.

Please guide me in solving this?

[NivethikaM]

@dadoonet Could you please clarify this?

1. As per my understanding, When we add a new file in the already indexed folder. The Fscrawler may initially check if the folder is indexed, if it is indexed it will pick the _id of _folder (tmp_a_folder) index. Now it may check the full path of the file in the index (tmp_a) along with the _id of the _folder index (stored as path.root in the main index). If it is not present the file will be indexed.

Can you please let me the is this is the logic used for indexing? Also kindly let me know in which file this part of logic is written?

[dadoonet]

Sorry for the delay. I think the code you are looking for is starting from: https://github.com/dadoonet/fscrawler/blob/master/core/src/main/java/fr/pilato/elasticsearch/crawler/fs/FsParserAbstract.java#L254-L314 and then https://github.com/dadoonet/fscrawler/blob/master/core/src/main/java/fr/pilato/elasticsearch/crawler/fs/FsParserAbstract.java#L337-L352

[NivethikaM]

Still we like to have this feature. Could you please provide more details.

[dadoonet]

What details do you need? I provided some links to the code. What would you need in addition to this?