.Net: Implementation of store using Azure SQL/SQL Server with vector search.

[cincuranet]

cc @yorek @SamMonoRT @roji @luisquintanilla

[roji]

LGTM, great to see this! See two comments below.

[SamMonoRT]

@markwallace-microsoft - please can we have a review from someone on your team.

[dmytrostruk]

Small comments, but in general looks good to me! Thank you!

[roji]

LGTM!

[Pookam90]

[heart] Pooja Kamath reacted to your message:

________________________________ From: Shay Rojansky ***@***.***> Sent: Thursday, May 16, 2024 9:49:54 AM To: microsoft/semantic-kernel ***@***.***> Cc: Pooja Kamath ***@***.***>; Comment ***@***.***> Subject: Re: [microsoft/semantic-kernel] .Net: Implementation of store using Azure SQL/SQL Server with vector search. (PR #6142) @roji approved this pull request. LGTM!

________________________________ In dotnet/src/Connectors/Connectors.Memory.SqlServer/SqlServerClient.cs<#6142 (comment)>:

+ }

+ await cmd.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false); + } + + /// <inheritdoc/> + public async IAsyncEnumerable<(SqlServerMemoryEntry, double)> GetNearestMatchesAsync(string tableName, ReadOnlyMemory<float> embedding, int limit, double minRelevanceScore = 0, bool withEmbeddings = false, [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + var queryColumns = withEmbeddings + ? "[key], [metadata], [timestamp], 1 - VECTOR_DISTANCE('cosine', [embedding], ***@***.***)) AS [cosine_similarity], VECTOR_TO_JSON_ARRAY([embedding]) AS [embedding]" + : "[key], [metadata], [timestamp], 1 - VECTOR_DISTANCE('cosine', [embedding], ***@***.***)) AS [cosine_similarity]"; + var fullTableName = this.GetFullTableName(tableName); + using var connection = new SqlConnection(this._connectionString); + await connection.OpenAsync(cancellationToken).ConfigureAwait(false); + using var cmd = connection.CreateCommand(); + cmd.CommandText = $""" + WITH data as ( Seems like there's no need for the CTE here, no? Can just append the queryColumns after the TOP, and the fullTableName after FROM?

________________________________ In dotnet/src/Connectors/Connectors.Memory.SqlServer/SqlServerMemoryStore.cs<#6142 (comment)>:

+

+ /// <summary> + /// Initializes a new instance of the <see cref="SqlServerMemoryStore"/> class. + /// </summary> + /// <param name="sqlServerClient">An instance of <see cref="ISqlServerClient"/>.</param> + public SqlServerMemoryStore(ISqlServerClient sqlServerClient) + { + this._sqlServerClient = sqlServerClient; + } + + /// <inheritdoc/> + public async Task CreateCollectionAsync(string collectionName, CancellationToken cancellationToken = default) + { + Verify.NotNull(collectionName); + + await this._sqlServerClient.CreateTableAsync(collectionName, cancellationToken).ConfigureAwait(false); FWIW not sure why we need the separation between SqlServerMemoryStore and SqlServerClient, i.e. why the code can't be directly in here... But I know other connectors are structured this way and it's not important. — Reply to this email directly, view it on GitHub<#6142 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOM3UAWL5CE3J3LNZMWW2D3ZCR6MFAVCNFSM6AAAAABHKPLDL6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDANJYG42TOMRQGU>. You are receiving this because you commented.Message ID: ***@***.***>

[cincuranet]

@dmytrostruk can you re-review the latest changes please (or merge). Thanks.

[yorek]

LGTM, thanks @cincuranet !