Inconsistent results between Janusgraph 0.5.3 vs 0.6.1

[rkkris75]

We are observing inconsistent query results on Janus 0.6.1. outputs from both versions - Janus 0.6.1 and Janus 0.5.3 below.

JANUS 0.6.1 ---> sometimes it is showing 2 vids , sometimes 1.

`bin]$ ./gremlin.sh

,,,/
(o o)
-----oOOo-(3)-oOOo-----
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/janusgraph/janusgraph-full-0.6.1/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/janusgraph/janusgraph-full-0.6.1/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
plugin activated: tinkerpop.server
plugin activated: tinkerpop.tinkergraph
17:22:30 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17:22:30 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
plugin activated: tinkerpop.hadoop
plugin activated: tinkerpop.spark
plugin activated: tinkerpop.utilities
plugin activated: janusgraph.imports
gremlin> graph = JanusGraphFactory.open('/usr/janusgraph/janusgraph-full-0.6.1/conf/gremlin-server/janusgraph-hbase-solr.properties')
17:22:42 WARN org.janusgraph.diskstorage.configuration.builder.ReadConfigurationBuilder - Warning graph.allow-upgrade is currently set to true on graph null. Please set graph.allow-upgrade to false in your properties file.
17:22:42 WARN org.janusgraph.diskstorage.configuration.builder.ReadConfigurationBuilder - Local setting ids.authority.conflict-avoidance-mode=LOCAL_MANUAL (Type: GLOBAL_OFFLINE) is overridden by globally managed value (NONE). Use the ManagementSystem interface instead of the local configuration to control this setting.

gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
==>v[150658445456]
gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
==>v[150658445456]
gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
==>v[150658445456]
`

JANUS 0.5.3 - results below shows consistent :

`bin]$ ./gremlin.sh

,,,/
(o o)
-----oOOo-(3)-oOOo-----
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/janusgraph/janusgraph-full-0.5.3/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/janusgraph/janusgraph-full-0.5.3/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
plugin activated: tinkerpop.server
plugin activated: tinkerpop.tinkergraph
17:24:21 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17:24:21 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
plugin activated: tinkerpop.hadoop
plugin activated: tinkerpop.spark
plugin activated: tinkerpop.utilities
plugin activated: janusgraph.imports
gremlin> graph = JanusGraphFactory.open('/usr/janusgraph/janusgraph-full-0.5.3/conf/gremlin-server/janusgraph-hbase-solr.properties')
17:24:30 WARN org.janusgraph.diskstorage.configuration.builder.ReadConfigurationBuilder - Warning graph.allow-upgrade is currently set to true on graph null. Please set graph.allow-upgrade to false in your properties file.
17:24:30 WARN org.janusgraph.diskstorage.configuration.builder.ReadConfigurationBuilder - Local setting ids.authority.conflict-avoidance-mode=LOCAL_MANUAL (Type: GLOBAL_OFFLINE) is overridden by globally managed value (NONE). Use the ManagementSystem interface instead of the local configuration to control this setting.

gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
`

Any idea why this inconsistent results out of 0.6.1?

[hadoopmarc]

Thanks for reporting this. Can you please also show the schema, in particular how you defined the index on the vid property key? If it as a MixedIndex can you also show how you configured the indexing backend?

[ddecoursey]

it was not created as Mixed index. storage backend is configured pointing to hbase zookeper master. These are the two index created for vid with different names on different days.
BYCMP2_20180122_vid:ENABLED
CMP-20180109-vid:ENABLED

 

This was created way back, and couldnt find exact creation steps. but looking at scripts this what we have
String attr = "vid"
String IndexName = 'CMP-'+attr
mgmt.buildIndex(IndexName,Vertex.class).addKey(mgmt.getPropertyKey(attr)).buildCompositeIndex()

 

Index name was updated as CMP- and BYCMP2_ and created twice with same attribute in past.

[li-boxuan]

mgmt.printIndexes should show more information and tell you whether they are mixed indexes or composite indexes.

[douglasawh]

@hadoopmarc thanks for the response. Is there any additional information that would be useful?

I noticed there were some inconsistency fixes in 0.6.3, but they appear to be specific to MixedIndexes. Do you think it is worth testing on 0.6.3?

[li-boxuan]

gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]
==>v[150658445456]
gremlin> graph.traversal().V().has('vid',within('-202432825562704244'))
==>v[8611472625704]

Not sure what the root cause is, but I suspect v[150658445456] is a ghost vertex. What do you see when you do g.V(150658445456).valueMap()?

[ddecoursey]

We are getting responses for valueMap() below are the results. They doesn’t look like ghost vertex to me. If so how can ensure graph search doesn’t return ghost vertices.

gremlin> graph.traversal().V(150658445456).valueMap()
==>[Source_System_Member_Sequence_Number:[001],First_Name:[],Last_Name:[],Source_System_Name:[UIS],Source_Customer_ID:[0101350],Source_Member_ID:[],Smoker_Status_Code:[],Birth_Date:[Tue Nov 18 00:00:00 EST 1969],Gender_Code:[Female],Marital_Status_Code:[Married],Death_Date:[Sat Jan 01 00:00:00 EST 1],Student_Status_Code:[],Sponsor_Relationship_Code:[Employee],_DIRTY:[false],vid:[-202432825562704244],vlabel:[Member],Cross_Reference_Member_Identifier:[],Previous_Employee_Identifier:[],MetLife_Member_Identifier:[000000000],Cross_Reference_Member_Identifier_Code:[],Customer_Member_ID:[],Billing_Member_Identifier:[],Third_Party_Administrator_Member_Identifier:[],Customer_Member_Identifier:[],Member_Personnel_ID:[1000000000],Dependent_Sequence_Number:[1],Middle_1_Name:[],Product:[Disability],timestamp:[1569014 558144],CDF_Customer_Number:[0000000],Source_System_Member_Sequence_Number_TEMP:[001]]

gremlin> graph.traversal().V(8611472625704).hasLabel("__tringergraph_deleted").hasNext()
==>false
gremlin> graph.traversal().V(150658445456).hasLabel("__tringergraph_deleted").hasNext()
==>false

In my view we may have many such instances in graph that will be consuming lot of space. We can cleanup those if we know how to identify same VIDs ? I mean graph query to find such duplicates.

[mad]

From 0.5.3 result vertex with identifier 8611472625704 should contain vid:[-202432825562704244] but from your last answer vertex with 150658445456 has vid:[-202432825562704244] too

You can write VertexScanJob that scan entire graph for duplcated. This job may be run on YARN for effectively region scan

If this vid is primary key you could use this feature https://docs.janusgraph.org/schema/index-management/index-performance/#index-uniqueness

[hadoopmarc]

Some new questions that pop up:

• am I right in assuming that you created the graph with v0.5.3 and that you query the same hbase table and solr index with either v0.5.3 or v0.6.1 (which is OK)?
• would it be easy to recreate the same graph, separately with v0.5.3 and v0.6.1, and see whether the behavior for this vid value reproduces?
• is the within step essential to the behavior, in other words what happens with:
  graph.traversal().V().has('vid', '-202432825562704244')