AI-Enabled Search Engine using LLM Embeddings, Django, and pgvector

A lot of people have asked us for ideas of how they can leverage Large Language Models (LLMs) for their business applications. A common example is to use the native language comprehension capabilities of LLMs to find matching content. This makes LLMs an excellent tool for search!

This repo demonstrates a prototype application that enables searching for job descriptions using an unstructured, English-language description of a job seeker.

Watch the Demo on Youtube

Links & Resources

• https://www.sbert.net/ - Sentence Transformers package for Python
• https://github.com/pgvector/pgvector - Vector database plugin for PostgreSQL
• https://huggingface.co/spaces/mteb/leaderboard - The "Massive Text Embedding" leaderboard from HuggingFace. Look at this to find models you can use to generate embeddings.
• https://www.djangoproject.com/

Setup

Docker

If this is your first time...

1. Install Docker
2. Run pipenv lock to generate a Pipfile.lock
3. Run cd client && npm install so you have node_modules available outside of Docker
4. Back in the root directory, run make build
5. make run to start the app
6. If the DB is new, run make create-test-data
   1. SuperUser [email protected] with credentials from your .env
   2. User [email protected] with credentials from your .env is used by the Cypress tests
7. View other available scripts/commands with make commands
8. localhost:8080 to view the app.
9. localhost:8000/staff/ to log into the Django admin
10. localhost:8000/api/docs/ to view backend API endpoints available for frontend development

Backend

If not using Docker... See the backend README

Frontend

If not using Docker... See the frontend README

Testing & Linting Locally

1. pipenv install --dev
2. pipenv run pytest server
3. pipenv run black server
4. pipenv run isort server --diff (shows you what isort is expecting)
5. npm run cypress