📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI #2355

DenisPalnitsky · 2024-04-08T11:39:24Z

Summary

Issue
Import conversation from Json (ChatGPT, Chatbot-ui and Librechat).

Important design points:

Agenda is used as a job scheduler to process imports in asynchronous manner
the type of file that is imported file is detected automatically so that the user does not have to make that decision. The detection algorithm is extremely naive but should work with current number of supported import files
poling should be used on a client to query for import job status

Diagram

graph TB
  linkStyle default fill:#ffffff

  subgraph diagram ["Import Service - Dynamic"]
    style diagram fill:#ffffff,stroke:#ffffff

    subgraph 2 [Import Service]
      style 2 fill:#ffffff,stroke:#9a9a9a,color:#9a9a9a

      10["<div style='font-weight: bold'>JobScheduler</div><div style='font-size: 70%; margin-top: 0px'>[Container]</div>"]
      style 10 fill:#dddddd,stroke:#9a9a9a,color:#000000
      12["<div style='font-weight: bold'>/api/convos/import indpoint</div><div style='font-size: 70%; margin-top: 0px'>[Container]</div>"]
      style 12 fill:#dddddd,stroke:#9a9a9a,color:#000000
      16["<div style='font-weight: bold'>/api/convos/import/jobs endpoint that returns status of import job</div><div style='font-size: 70%; margin-top: 0px'>[Container]</div>"]
      style 16 fill:#dddddd,stroke:#9a9a9a,color:#000000
      3["<div style='font-weight: bold'>MongoDB</div><div style='font-size: 70%; margin-top: 0px'>[Container]</div><div style='font-size: 80%; margin-top:10px'>Stores all conversations</div>"]
      style 3 fill:#dddddd,stroke:#9a9a9a,color:#000000
      4["<div style='font-weight: bold'>ImportBatchBuilder</div><div style='font-size: 70%; margin-top: 0px'>[Container]</div><div style='font-size: 80%; margin-top:10px'>Packs all conversations into<br />message and conversation<br />batches</div>"]
      style 4 fill:#dddddd,stroke:#9a9a9a,color:#000000
      6["<div style='font-weight: bold'>importerProviderConvo</div><div style='font-size: 70%; margin-top: 0px'>[Container]</div><div style='font-size: 80%; margin-top:10px'>Instance of importer<br />importChatGtpConvo,<br />importChatBotUiConvo or<br />importLibreChatConvo</div>"]
      style 6 fill:#dddddd,stroke:#9a9a9a,color:#000000
      8["<div style='font-weight: bold'>getImporter</div><div style='font-size: 70%; margin-top: 0px'>[Container]</div><div style='font-size: 80%; margin-top:10px'>Detects what type of import<br />file is use (chatGpt,<br />Chatbot-ui, Librechat)</div>"]
      style 8 fill:#dddddd,stroke:#9a9a9a,color:#000000
    end

    1["<div style='font-weight: bold'>Client</div><div style='font-size: 70%; margin-top: 0px'>[Person]</div>"]
    style 1 fill:#dddddd,stroke:#9a9a9a,color:#000000

    1-. "<div>1. Sends import file to</div><div style='font-size: 70%'></div>" .->12
    12-. "<div>2. Creates a job with file<br />content in</div><div style='font-size: 70%'></div>" .->10
    10-. "<div>3. Triggers function</div><div style='font-size: 70%'></div>" .->8
    8-. "<div>4. Creates importer instance<br />based on file content</div><div style='font-size: 70%'></div>" .->6
    6-. "<div>5. Creates message and<br />conversation batches using</div><div style='font-size: 70%'></div>" .->4
    4-. "<div>6. Stores conversations and<br />messages in</div><div style='font-size: 70%'></div>" .->3
    1-. "<div>7. Polls for jobs status<br />until the status is success<br />or failure</div><div style='font-size: 70%'></div>" .->16
    16-. "<div>8. Polls for job status</div><div style='font-size: 70%'></div>" .->10
  end

UI

Change Type

Please delete any irrelevant options.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update
Translation update
Documentation update

Testing

The examples of exported conversations could be found in /api/server/utils/import/__data__ directory.
Apart from unit-tests the functionality were tested on conversation files of ~5Mb size.

Checklist

Please delete any irrelevant options.

My code adheres to this project's style guidelines
I have performed a self-review of my own code
I have commented in any complex areas of my code
I have made pertinent documentation changes
My changes do not introduce new warnings
I have written tests demonstrating that my changes are effective or that my feature works
Local unit tests pass with my changes
Any changes dependent on mine have been merged and published in downstream modules.
New documents have been locally validated with mkdocs

danny-avila · 2024-04-08T12:29:27Z

Hey @DenisPalnitsky thanks for taking a look at this! This is a great feature I look forward to seeing myself. There is lots to do so I can't get to everything at once, so I appreciate you taking a look at this.

I took a quick glance and there are some issues with the approach, and I will use this opportunity to outline my ideas on how to optimize and design this based on long-term thinking for the project.

I don't know about you, but from before I made and started heavily using LibreChat (I never use ChatGPT or any other UI anymore), I have conversations in the thousands, especially from December 2022 - February 2023. I can only imagine from users only just discovering the project, so some things will prevent future headache now.

MongoDB batch/bulk operations should be used. Mongoose has a helpful SDK for these
- On this point, there should be database controllers designed to do this for messages, conversations, similar to how they exist now, having in mind that a complete refactor could happen one day where Mongo isn't our only database solution. This may or may not ever happen, but I would like to decouple from mongo-specific code as much as possible.
A job queue system would be ideal
- This helps fight against potential server congestion of many users importing. Speaking of npm libraries: based on our use of mongo, the agenda lib may help, Bree is also an option. bull is the most popular but I'm not keen on adding Redis as a required dependency for this quite yet.
- Implementation of this, similar to last point, should have dependency injection in mind, in case a library is replaced later on.
Rate limiting must be enforced on the import route
- This is primarily for use of agenda or Bree as bull may have rate limiting built in. This is probably the easiest considering there are already examples in the project, and you would need to follow a similar pattern.
Where possible, take advantage of asynchronous processes

DenisPalnitsky · 2024-04-09T12:16:07Z

Hi @danny-avila
Thanks for taking a look at the PR in advance. I understand you concerns and will try to take them into account however I'm not an expert in JS and do not know the ecosystem that well so my idea was to add this functionality without touching the existing code whenever possible.
That was the reason behind using existing interface for creating messages and conversations instead of batch updates. That way migration to other db or just changing the internal structure would have to be done only in one place which makes it easier to maintain. Performance wise it should not be a problem as it's one-time task per user. My 5mb conversations file takes 5 seconds on local machine to import.
That said I'll give 'agenda' a try and set the rate limiting for the route. I'll add those today-tomorrow and then we can see if there is a way forward.
Also if you have a specific suggestions I'm happy to take them into account. As I said, I have not been working with JS and you guidelines would be very helpful.
Overall, I think it's a useful feature and having it working somehow would be better then nothing 😄

danny-avila · 2024-04-09T12:59:34Z

That was the reason behind using existing interface for creating messages and conversations instead of batch updates. That way migration to other db or just changing the internal structure would have to be done only in one place which makes it easier to maintain.

I understand it's reasonable, but it would be very easy to add another method for each, for example saveBulkConvos and saveBulkMessages.

I've simply prompted GPT-4 using LibreChat showing the relevant parts of code, and it does a pretty good job (after some steering):

Create_SaveBulkMessages Method.md
Create_SaveBulkConvos Function.md

it includes validation using schemas that already exist. Please also use those existing zod schemas.

Performance wise it should not be a problem as it's one-time task per user. My 5mb conversations file takes 5 seconds on local machine to import.

This is not so much about the immediate performance, but the number of concurrent database transactions taking place, especially compounded by concurrent use of LibreChat.

Existing methods for each item works fine but when things start to grow, with more users, you really want to lean towards bulk operations, easing the load on your database, and helps keep costs down if you're using something like MongoDB Atlas.

DenisPalnitsky · 2024-04-09T16:34:07Z

@danny-avila In case we use job scheduler for the task, how do you see the implementation of the feedback to the user whether the import succeeded or not? Are there such examples in the code that I can refer to?

danny-avila · 2024-04-09T17:37:52Z

@danny-avila In case we use job scheduler for the task, how do you see the implementation of the feedback to the user whether the import succeeded or not? Are there such examples in the code that I can refer to?

thats a good question. no example exists as we don't have job scheduling yet.

For the sake of rolling this out sooner rather than later, you can tackle this PR with just points 1, and 2-4 from earlier, you can let me worry about the job queue handling as that could be an iterative improvement for this.

In my opinion, starting this off with bulk/batch operations is more important and easier to implement with what you built out already.

DenisPalnitsky · 2024-04-10T17:07:15Z

@danny-avila I added batch write, rate limiting and job scheduler. I also added an endpoint to poll for a jobs status. That would be easiest way to implement feedback for now.
Could you please check if overall structure looks ok. After confirmation I'll add a tests and submit PR for a review.

danny-avila · 2024-04-11T06:44:09Z

@danny-avila I added batch write, rate limiting and job scheduler. I also added an endpoint to poll for a jobs status. That would be easiest way to implement feedback for now. Could you please check if overall structure looks ok. After confirmation I'll add a tests and submit PR for a review.

I took a quick glance and it looks good to me (aside from pending conflict)! Thanks a lot for working on this.

One note about working in the /api workspace. Later in the project, I started making use of JSDocs and typedefs to provide "types" for the CommonJS code. Typing some of the new functions and classes you made will be extremely helpful for me.

For example, this case AI will excel in as well. Prompt it with the relevant code, and say "Please write JSDocs for this function/class, to provide intellisense in VS Code" and you can also give it examples of these definitions from the project.

DenisPalnitsky · 2024-04-12T16:22:10Z

@danny-avila this one is ready for review

danny-avila · 2024-04-17T14:58:58Z

@danny-avila this one is ready for review

Thank you, hope to review it this week

DenisPalnitsky · 2024-04-19T07:51:25Z

I have no idea why the test is failing. It passes locally. Is there a chance that MONGO_URI secret is not defined in Github action secrets?

walbercardoso · 2024-04-19T12:34:39Z

I have no idea why the test is failing. It passes locally. Is there a chance that MONGO_URI secret is not defined in Github action secrets?

Did you put the .env file inside api folder?

DenisPalnitsky · 2024-04-19T13:39:49Z

Did you put the .env file inside api folder?

yes, if I run tests locally they pass. However on ci they fail

walbercardoso · 2024-04-19T18:38:51Z

Seems like some PARAMS is missing at the .env file. can you share the result of npm run test:api from your local computer?

DenisPalnitsky · 2024-04-19T21:07:45Z

Seems like some PARAMS is missing at the .env file. can you share the result of npm run test:api from your local computer?

Sure

➜ npm run test:api 
(node:79658) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS utils/azureUtils.spec.js
(node:79657) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS app/clients/specs/BaseClient.test.js
(node:79656) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS server/services/Config/loadConfigModels.spec.js
(node:79655) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS app/clients/specs/AnthropicClient.test.js
PASS app/clients/tools/structured/specs/DALLE3.spec.js
(node:79654) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS app/clients/tools/util/handleTools.test.js
PASS server/services/ModelService.spec.js
PASS app/clients/tools/dynamic/OpenAPIPlugin.spec.js
PASS app/clients/document/tokenSplit.spec.js
PASS cache/banViolation.spec.js
PASS app/clients/tools/util/addOpenAPISpecs.spec.js
PASS server/services/AuthService.spec.js
PASS server/services/Config/loadCustomConfig.spec.js
PASS server/utils/handleText.spec.js
PASS utils/tokens.spec.js
PASS app/clients/tools/util/loadSpecs.spec.js
PASS models/tx.spec.js
PASS strategies/validators.spec.js
PASS utils/extractBaseURL.spec.js
PASS app/clients/prompts/formatMessages.spec.js
PASS server/services/AppService.spec.js
PASS app/clients/output_parsers/addImages.spec.js
PASS app/clients/prompts/formatGoogleInputs.spec.js
PASS server/utils/import/importers.spec.js
  ● Console

    console.log
      Conversation Conversation 1. Web Search imported

      at log (server/utils/import/importers.js:120:13)

(node:79653) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS server/services/Endpoints/assistants/initializeClient.spec.js
(node:79652) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS app/clients/specs/PluginsClient.test.js
(node:79651) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS server/services/Endpoints/google/initializeClient.spec.js
(node:79649) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS server/services/Endpoints/gptPlugins/initializeClient.spec.js
(node:79650) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS server/services/Endpoints/openAI/initializeClient.spec.js
(node:79648) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS app/clients/specs/OpenAIClient.test.js
A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --detectOpenHandles to find leaks. Active timers can also cause this, ensure that .unref() was called on them.

Test Suites: 1 skipped, 30 passed, 30 of 31 total
Tests:       1 skipped, 401 passed, 402 total
Snapshots:   0 total
Time:        4.006 s
Ran all test suites.
```➜ npm run test:api > test.results
(node:79658) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS utils/azureUtils.spec.js
(node:79657) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS app/clients/specs/BaseClient.test.js
(node:79656) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS server/services/Config/loadConfigModels.spec.js
(node:79655) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS app/clients/specs/AnthropicClient.test.js
PASS app/clients/tools/structured/specs/DALLE3.spec.js
(node:79654) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS app/clients/tools/util/handleTools.test.js
PASS server/services/ModelService.spec.js
PASS app/clients/tools/dynamic/OpenAPIPlugin.spec.js
PASS app/clients/document/tokenSplit.spec.js
PASS cache/banViolation.spec.js
PASS app/clients/tools/util/addOpenAPISpecs.spec.js
PASS server/services/AuthService.spec.js
PASS server/services/Config/loadCustomConfig.spec.js
PASS server/utils/handleText.spec.js
PASS utils/tokens.spec.js
PASS app/clients/tools/util/loadSpecs.spec.js
PASS models/tx.spec.js
PASS strategies/validators.spec.js
PASS utils/extractBaseURL.spec.js
PASS app/clients/prompts/formatMessages.spec.js
PASS server/services/AppService.spec.js
PASS app/clients/output_parsers/addImages.spec.js
PASS app/clients/prompts/formatGoogleInputs.spec.js
PASS server/utils/import/importers.spec.js
  ● Console

    console.log
      Conversation Conversation 1. Web Search imported

      at log (server/utils/import/importers.js:120:13)

(node:79653) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS server/services/Endpoints/assistants/initializeClient.spec.js
(node:79652) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS app/clients/specs/PluginsClient.test.js
(node:79651) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS server/services/Endpoints/google/initializeClient.spec.js
(node:79649) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS server/services/Endpoints/gptPlugins/initializeClient.spec.js
(node:79650) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS server/services/Endpoints/openAI/initializeClient.spec.js
(node:79648) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
PASS app/clients/specs/OpenAIClient.test.js
A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --detectOpenHandles to find leaks. Active timers can also cause this, ensure that .unref() was called on them.

Test Suites: 1 skipped, 30 passed, 30 of 31 total
Tests:       1 skipped, 401 passed, 402 total
Snapshots:   0 total
Time:        4.006 s
Ran all test suites.

walbercardoso · 2024-04-20T04:34:07Z

Seems like some PARAMS is missing at the .env file. can you share the result of npm run test:api from your local computer?

Sure


➜ npm run test:api 

(node:79658) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS utils/azureUtils.spec.js

(node:79657) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS app/clients/specs/BaseClient.test.js

(node:79656) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS server/services/Config/loadConfigModels.spec.js

(node:79655) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS app/clients/specs/AnthropicClient.test.js

PASS app/clients/tools/structured/specs/DALLE3.spec.js

(node:79654) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS app/clients/tools/util/handleTools.test.js

PASS server/services/ModelService.spec.js

PASS app/clients/tools/dynamic/OpenAPIPlugin.spec.js

PASS app/clients/document/tokenSplit.spec.js

PASS cache/banViolation.spec.js

PASS app/clients/tools/util/addOpenAPISpecs.spec.js

PASS server/services/AuthService.spec.js

PASS server/services/Config/loadCustomConfig.spec.js

PASS server/utils/handleText.spec.js

PASS utils/tokens.spec.js

PASS app/clients/tools/util/loadSpecs.spec.js

PASS models/tx.spec.js

PASS strategies/validators.spec.js

PASS utils/extractBaseURL.spec.js

PASS app/clients/prompts/formatMessages.spec.js

PASS server/services/AppService.spec.js

PASS app/clients/output_parsers/addImages.spec.js

PASS app/clients/prompts/formatGoogleInputs.spec.js

PASS server/utils/import/importers.spec.js

  ● Console



    console.log

      Conversation Conversation 1. Web Search imported



      at log (server/utils/import/importers.js:120:13)



(node:79653) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS server/services/Endpoints/assistants/initializeClient.spec.js

(node:79652) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS app/clients/specs/PluginsClient.test.js

(node:79651) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS server/services/Endpoints/google/initializeClient.spec.js

(node:79649) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS server/services/Endpoints/gptPlugins/initializeClient.spec.js

(node:79650) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS server/services/Endpoints/openAI/initializeClient.spec.js

(node:79648) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS app/clients/specs/OpenAIClient.test.js

A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --detectOpenHandles to find leaks. Active timers can also cause this, ensure that .unref() was called on them.



Test Suites: 1 skipped, 30 passed, 30 of 31 total

Tests:       1 skipped, 401 passed, 402 total

Snapshots:   0 total

Time:        4.006 s

Ran all test suites.

```➜ npm run test:api > test.results

(node:79658) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS utils/azureUtils.spec.js

(node:79657) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS app/clients/specs/BaseClient.test.js

(node:79656) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS server/services/Config/loadConfigModels.spec.js

(node:79655) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS app/clients/specs/AnthropicClient.test.js

PASS app/clients/tools/structured/specs/DALLE3.spec.js

(node:79654) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS app/clients/tools/util/handleTools.test.js

PASS server/services/ModelService.spec.js

PASS app/clients/tools/dynamic/OpenAPIPlugin.spec.js

PASS app/clients/document/tokenSplit.spec.js

PASS cache/banViolation.spec.js

PASS app/clients/tools/util/addOpenAPISpecs.spec.js

PASS server/services/AuthService.spec.js

PASS server/services/Config/loadCustomConfig.spec.js

PASS server/utils/handleText.spec.js

PASS utils/tokens.spec.js

PASS app/clients/tools/util/loadSpecs.spec.js

PASS models/tx.spec.js

PASS strategies/validators.spec.js

PASS utils/extractBaseURL.spec.js

PASS app/clients/prompts/formatMessages.spec.js

PASS server/services/AppService.spec.js

PASS app/clients/output_parsers/addImages.spec.js

PASS app/clients/prompts/formatGoogleInputs.spec.js

PASS server/utils/import/importers.spec.js

  ● Console



    console.log

      Conversation Conversation 1. Web Search imported



      at log (server/utils/import/importers.js:120:13)



(node:79653) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS server/services/Endpoints/assistants/initializeClient.spec.js

(node:79652) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS app/clients/specs/PluginsClient.test.js

(node:79651) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS server/services/Endpoints/google/initializeClient.spec.js

(node:79649) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS server/services/Endpoints/gptPlugins/initializeClient.spec.js

(node:79650) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS server/services/Endpoints/openAI/initializeClient.spec.js

(node:79648) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

PASS app/clients/specs/OpenAIClient.test.js

A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --detectOpenHandles to find leaks. Active timers can also cause this, ensure that .unref() was called on them.



Test Suites: 1 skipped, 30 passed, 30 of 31 total

Tests:       1 skipped, 401 passed, 402 total

Snapshots:   0 total

Time:        4.006 s

Ran all test suites.

Try to run npm jest [path/filename] where file name is the file that failed

danny-avila · 2024-04-22T22:00:40Z

I'll add a MONGO_URI so hopefully the tests can be run in CI env

DenisPalnitsky · 2024-04-23T07:34:17Z

I'll add a MONGO_URI so hopefully the tests can be run in CI env

I think I know what could be a problem. Let me try to fix it.

DenisPalnitsky · 2024-04-23T10:08:41Z

It fails here so adding a dummy MONGO_URI should do the job

danny-avila · 2024-04-23T12:30:57Z

Fixing the package issue, also seeing if I can handle the jest open handles issue, too.

danny-avila · 2024-04-23T12:41:00Z

I actually need to be added as a collaborator to your fork, but the tests are running fine for me locally

DenisPalnitsky · 2024-04-23T13:40:19Z

I actually need to be added as a collaborator to your fork, but the tests are running fine for me locally

I added you

danny-avila · 2024-05-01T17:52:44Z

I'm getting an error with a large import from ChatGPT (36 MB JSON file). Trying to see what might be the best course of action here:

Error processing file Error: Failed to schedule job 'import conversation': RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be >= 0 && <= 17825792. Received 17825794
at JobScheduler.now (/home/danny/denis-libre/api/server/utils/jobScheduler.js:42:13)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /home/danny/denis-libre/api/server/routes/convos.js:113:17

- organize imports - add additional jsdocs - use multer with diskStorage to avoid loading file into memory outside of job - use filepath instead of loading data string for imports - replace console logs and some logger.info() with logger.debug - only use multer for import route

…e case and replace ChatGtp with ChatGpt

danny-avila · 2024-05-01T19:11:04Z

my changes so far

organize imports
add additional jsdocs
use multer with diskStorage to avoid loading file into memory outside of job
use filepath instead of loading data string for imports
replace console logs and some logger.info() with logger.debug
only use multer for import route
fix edge cases found when importing over 2000 chatgpt convos
added typedefs for chatgpt data structures

and now im seeing that tree structure is not maintained in general. on second thought, I do want to see that in this PR as that is a big value add. going to see if i can easily implement it for now

… add userId to log message

…eadme

DenisPalnitsky · 2024-05-02T11:50:15Z

@danny-avila Thanks for fixing my bugs and merging this 👍

danny-avila · 2024-05-02T13:34:40Z

@danny-avila Thanks for fixing my bugs and merging this 👍

Thanks for your hard work and patience on this 🙏

DenisPalnitsky force-pushed the feature/chats-import branch from a469971 to 6f9ab88 Compare April 11, 2024 07:13

DenisPalnitsky marked this pull request as ready for review April 12, 2024 16:17

DenisPalnitsky force-pushed the feature/chats-import branch from 1e83bc0 to 75bec09 Compare April 12, 2024 16:20

DenisPalnitsky changed the title ~~[WIP] Import conversations from Json~~ Import conversations from Json Apr 12, 2024

DenisPalnitsky force-pushed the feature/chats-import branch from 75bec09 to 1b1f625 Compare April 15, 2024 08:55

DenisPalnitsky changed the title ~~Import conversations from Json~~ 📥 feat: Import conversations from Json Apr 17, 2024

DenisPalnitsky force-pushed the feature/chats-import branch from 8fdfb11 to 737d9f2 Compare April 19, 2024 21:10

danny-avila added this to the v0.7.2 milestone Apr 21, 2024

danny-avila force-pushed the feature/chats-import branch from fcc30f2 to 0adafd1 Compare April 24, 2024 14:54

fix: user role issue for librechat imports

84518da

danny-avila added 4 commits May 1, 2024 14:40

fix: undefined metadata edge case and replace ChatGtp -> ChatGpt

5ddfa53

Refactor importChatGptConvo function to handle undefined metadata edg…

9ff802b

…e case and replace ChatGtp with ChatGpt

fix: chatgpt importer

73edea9

danny-avila added 14 commits May 1, 2024 15:21

feat: maintain tree relationship for librechat messages

a66a4d1

chore: use enum

3a2e33d

refactor: saveMessage to use single object arg, replace console logs,…

63e1e0d

… add userId to log message

chore: additional comment

0fbe3d2

chore: multer edge case

327506c

feat: first pass, maintain tree relationship

cd9bd1e

chore: organize

7951b55

chore: remove log

f9e161d

ci: add heirarchy test for chatgpt

f406aaa

ci: test maintaining of heirarchy for librechat

dfc8895

wip: allow non-text content type messages

00c54e1

refactor: import content part object json string

8be339a

refactor: more content types to format

ade6971

chore: consolidate messageText formatting

8e7f458

danny-avila changed the title ~~📥 feat: Import conversations from Json~~ 📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI May 2, 2024

danny-avila changed the title ~~📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI~~ 📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI May 2, 2024

danny-avila added 4 commits May 2, 2024 02:21

docs: update on changes, bump data-provider/config versions, update r…

38c9db5

…eadme

refactor(indexSync): singleton pattern for MeiliSearchClient

edef308

refactor: debug log after batch is done

137c7b3

chore: add back indexSync error handling

2a486a6

danny-avila merged commit ab6fbe4 into danny-avila:main May 2, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI #2355

📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI #2355

DenisPalnitsky commented Apr 8, 2024 •

edited

danny-avila commented Apr 8, 2024 •

edited

DenisPalnitsky commented Apr 9, 2024

danny-avila commented Apr 9, 2024 •

edited

DenisPalnitsky commented Apr 9, 2024 •

edited

danny-avila commented Apr 9, 2024

DenisPalnitsky commented Apr 10, 2024

danny-avila commented Apr 11, 2024 •

edited

DenisPalnitsky commented Apr 12, 2024

danny-avila commented Apr 17, 2024

DenisPalnitsky commented Apr 19, 2024 •

edited

walbercardoso commented Apr 19, 2024

DenisPalnitsky commented Apr 19, 2024

walbercardoso commented Apr 19, 2024

DenisPalnitsky commented Apr 19, 2024

walbercardoso commented Apr 20, 2024

danny-avila commented Apr 22, 2024

DenisPalnitsky commented Apr 23, 2024

DenisPalnitsky commented Apr 23, 2024

danny-avila commented Apr 23, 2024

danny-avila commented Apr 23, 2024

DenisPalnitsky commented Apr 23, 2024

danny-avila commented May 1, 2024

danny-avila commented May 1, 2024 •

edited

DenisPalnitsky commented May 2, 2024

danny-avila commented May 2, 2024

📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI #2355

📥 feat: Import Conversations from LibreChat, ChatGPT, Chatbot UI #2355

Conversation

DenisPalnitsky commented Apr 8, 2024 • edited

Summary

Diagram

UI

Change Type

Testing

Checklist

danny-avila commented Apr 8, 2024 • edited

DenisPalnitsky commented Apr 9, 2024

danny-avila commented Apr 9, 2024 • edited

DenisPalnitsky commented Apr 9, 2024 • edited

danny-avila commented Apr 9, 2024

DenisPalnitsky commented Apr 10, 2024

danny-avila commented Apr 11, 2024 • edited

DenisPalnitsky commented Apr 12, 2024

danny-avila commented Apr 17, 2024

DenisPalnitsky commented Apr 19, 2024 • edited

walbercardoso commented Apr 19, 2024

DenisPalnitsky commented Apr 19, 2024

walbercardoso commented Apr 19, 2024

DenisPalnitsky commented Apr 19, 2024

walbercardoso commented Apr 20, 2024

danny-avila commented Apr 22, 2024

DenisPalnitsky commented Apr 23, 2024

DenisPalnitsky commented Apr 23, 2024

danny-avila commented Apr 23, 2024

danny-avila commented Apr 23, 2024

DenisPalnitsky commented Apr 23, 2024

danny-avila commented May 1, 2024

danny-avila commented May 1, 2024 • edited

DenisPalnitsky commented May 2, 2024

danny-avila commented May 2, 2024

DenisPalnitsky commented Apr 8, 2024 •

edited

danny-avila commented Apr 8, 2024 •

edited

danny-avila commented Apr 9, 2024 •

edited

DenisPalnitsky commented Apr 9, 2024 •

edited

danny-avila commented Apr 11, 2024 •

edited

DenisPalnitsky commented Apr 19, 2024 •

edited

danny-avila commented May 1, 2024 •

edited