Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reposilite 4.x wishlist #2005

Open
dzikoysk opened this issue Dec 31, 2023 · 26 comments
Open

Reposilite 4.x wishlist #2005

dzikoysk opened this issue Dec 31, 2023 · 26 comments
Assignees
Labels
breaking Issue breaks the compatibility with previous versions investigation Issue is currently investigeted, e.g. author is trying to reproduce problem next release Details about planned release

Comments

@dzikoysk
Copy link
Owner

dzikoysk commented Dec 31, 2023

About

Reposilite 3.x is a stable piece of software and for most of the time it serves you well. Of course, some decisions I made during 2.x -> 3.x rewrite were good, and some of them are a bit of a problem these days. I think that's a good time to start summing up the 3.x release line, so we can prepare for the next major version. Fortunately, this time we're more likely talking about an evolution, not a revolution - the backend module is much easier to maintain, Kotlin really paid off here. Sadly, I can't say it about frontend in JS + Vue 3.

Roadmap

Some high level estimation of a roadmap for Reposilite 4.x 💕

Date Description
Q1 2024 Get some feedback from the community about current state of the project and possible enhancements in the 4.x
Q2 2024 Refine ideas, prepare project & start development
Q2 & Q3 2024 Development phase
Q4 2024 Release 4.x 🎉

General goals

Goal Result
Include long overdue improvements that couldn't be addressed in 3.x Improve the offer
Remove tech debt and reduce the complexity of frontend implementation Easier to maintain
Prioritize simplicty and lightweightness Keep the core advantage over alternatives

RFCs

Priority Domain Description Status
P1 Tokens → Users Reposilite should introduce user system. Users should be able to generate personal tokens on their own. ✔️
P1 Authentication Thanks to the new user system in Reposilite, we should be able to support the following auth methods in the UI:
📢
P1 UI / Dashboard Maintaining current REST API + standalone frontend module requires a lot of work. With JTE-like template engine + some lightweight library like HTMX + Tailwind, we could probably do the same thing with a better result (and improve mobile UX)
P2 Security In-app alerting system to notify users in case of diagnosed security related issues (like CVEs) ✔️
P2 Licensing Reconsider license change to protect project from SaaS-like actors
P2 General Simplify sources: Let's try to simplify & reduce sources having some more experience ✔️
P2 Build-systems Prepare a better support on API level for other build systems / protocols
P2 Checksum Checksum plugin will be merged into the core module ✔️
P2 Prometheus Prometheus plugin will be merged into the core module
P3 Tests Optimize tests: We're launching a whole ecosystem for each test method. Let's try to reuse some resources there ✔️
P3 Database layer Exposed should be replaced with Sqiffy ✔️
P3 Built-in SSL Move built-in SSL to standalone plugin to reduce JAR size (this integration is rarely used anyway) ✔️
P3 Targets Split distribution across Linux, Windows and macOS to reduce JAR size ✔️
Priority: P1 - high, P2 - medium, P3 - low
Status: ✔️ - approved, 📢 - requires feedback/details, ❔ - unknown

Thanks to all sponsors that decided to support me this year! We wouldn't be here without you 💛

image

@dzikoysk dzikoysk added investigation Issue is currently investigeted, e.g. author is trying to reproduce problem next release Details about planned release breaking Issue breaks the compatibility with previous versions labels Dec 31, 2023
@dzikoysk dzikoysk self-assigned this Dec 31, 2023
@dzikoysk dzikoysk pinned this issue Dec 31, 2023
@solonovamax
Copy link
Contributor

Since it seems like the UI is going to be majorly reworked: thoughts on adding "looks at least somewhat decent on mobile" to the list? (especially the admin interface, so admin stuff can be done from a mobile device)

@solonovamax
Copy link
Contributor

also, one note: you mentioned using jte.
for me personally, jte is kinda okay-ish. I'd just like to add freemarker as a suggestion for a templating engine to consider.
there's also thymeleaf, which I've been using for my personal site. the experience has been okay-ish (solonovamax/solonovamax.gay if you wanna look at how I do stuff. kotlin + ktor.), but there have been a few kinda jank-ish parts, but other than that it's been nice.

@dzikoysk
Copy link
Owner Author

dzikoysk commented Jan 1, 2024

Since it seems like the UI is going to be majorly reworked: thoughts on adding "looks at least somewhat decent on mobile" to the list? (especially the admin interface, so admin stuff can be done from a mobile device)

Sure, the mobile support should be way better there. In current UI, it's mainly blocked by the tabs & jsonforms libraries, but it's a minor issue, so I didn't decide to invest my time into it for now.

I'd just like to add freemarker as a suggestion for a templating engine to consider. there's also thymeleaf, which I've been using for my personal site. [...]

When I'm thinking about a desired stack for a new frontend, I want to cover these 3 crucial requirements:

  1. Get rid of a translation layer between backend and frontend (optimized template engine)
  2. Avoid covering with JS basic CRUD operations (frontend framework that covers the most common boilerplate code)
  3. Interface should become a part of the plugin api, so it'll be customizable to some degree (some sort of a component system)

The main reason behind JTE choice is performance - it provides a compilation mode, where it generates classes for all template components:

obraz

This + a possibility to implement Node-like hot reloading makes it a good candidate. It has some issues tho - writing a JavaScript code inside a template might be a bit painful. Even if HTMX/Unpolly/something similar would cover most of the features, there are still some side cases, when we may need to cover with JS. Raw JS/state management is kind rough, so we could try to utilize something like Preact locally, for some specific features.

@freshpr
Copy link

freshpr commented Jan 4, 2024

The only thing I'm missing is an indexer+search built-in.
That's it :)

@solonovamax
Copy link
Contributor

The only thing I'm missing is an indexer+search built-in.
That's it :)

I was working on that for a bit as a plugin sometime ago, and may continue it one day™ (or I might just forgor again)

@SirEndii
Copy link
Sponsor

Maybe not too related to Reposilite. But what about an integrated docker registry?

@dzikoysk
Copy link
Owner Author

Yes, the Docker registry would be absolutely great. Unfortunately, it requires a lot of work, as the specification seems to be quite complex - there's an issue for that:

It'd be nice to document all things we need to do (from specification perspective), so later on we could try to estimate if we'll be able to do that or not.

@luolong
Copy link

luolong commented Jan 15, 2024

Maybe not too related to Reposilite. But what about an integrated docker registry?

There is already Harbor for that, so I don't think it is strictly necessary. I do appreciate Reposilite's focus on providing great light weight solution for hosting Maven repositories.

@SirEndii
Copy link
Sponsor

Yeah you're right. There is enough software which is already capable of providing a registry and reposilite should stay simple-ish

@Xanax-C-137
Copy link

I'd like to see support for byte ranges. I want to use reposilite as a simple arch mirror, and arch is complaining about failed downloads.

@dzikoysk
Copy link
Owner Author

dzikoysk commented Feb 5, 2024

Maybe not too related to Reposilite. But what about an integrated docker registry?

There is already Harbor for that, so I don't think it is strictly necessary. I do appreciate Reposilite's focus on providing great light weight solution for hosting Maven repositories.

Well, yes and no. These days, you're often distributing JVM apps as containers, so it'd be kinda nice to have it supported at some point - I could even distribute Reposilite itself there. Of course, it can't be considered as the new main target, because we don't have enough resources to compete with dedicated tools, like e.g. Harbor. Hopefully, something like that should be possible to implement purely as an optional plugin.

I'd like to see support for byte ranges. I want to use reposilite as a simple arch mirror, and arch is complaining about failed downloads.

It'd be nice if you could submit a new issue with a bit more detailed description. As long as we don't aim to support arch itself, supporting http range requests probably could be a thing.

@Desoroxxx
Copy link

One thing I would like to see is the ability to customize the appearance

@solonovamax
Copy link
Contributor

solonovamax commented Mar 6, 2024

yeah, the ability to provide your own css styles/themes would be quite nice

ie. use the default frontend, but just have some customized css (ie. to make the frontend a bit less "harsh")


also here's a few misc. things:

  1. smth I thought of a while back that would be nice is: the ability to store cached files from a proxied/mirrored repository in a separate location.

    so, for example, the contents of the releases (that have been uploaded directly to the repo) repo could be stored in repo/releases.
    but then, say you're mirroring foo.com and bar.net (and reposilite is configured to store those artifacts)
    then, the artifacts from each of those would be located in smth like mirrors/releases/foo.com and mirrors/releases/bar.net.

    with this, you'd also be able to provide the following things:

    • add a badge on remote artifacts/modules to indicate they're from a mirror (& maybe which mirror they're from?)
    • allow administrators to set storage limits on mirrored artifacts and use smth like an LRU policy to discard old cached artifacts.
  2. the ability to change the name of the repository provided in the generated "repository details" for maven, gradle kotlin, gradle groovy, and sbt. (eg. I'd like the "name" to be smth specific rather than what was generated, as well as using specific capitalization, etc.)

  3. since you're looking at making the frontend part of the plugin api, should a search plugin be added to the roadmap? if it is, I could start that one thing I was working on a while ago with maven-indexer back up.

  4. funny numbers: it would be cool if the statistics showed more information. for example:

    • a list of artifacts by popularity (eg. "here are the top 10 most downloaded artifacts in [time interval]: this artifact has been downloaded 123 times in the past [time interval]"
    • also maybe aggregate stats information, such as which modules have the most downloads, which versions of a certain module have the most downloads, etc.
    • showing shit like how many of the downloaded artifacts are from mirrored repos or not (ie. "123 downloads from the local repo, 123 downloads from cached mirrored artifacts, and 123 from uncached mirrored artifacts)"
    • which mirrors are the most "popular"? (eg. "123 downloads from mirror foo, 123 downloads from mirror bar")

    unsure if this might be better suited for just sending that data to prometheus and using grafana for this, though.

  5. minor change to the badges api: support for different styles (see: https://shields.io/badges), eg. flat, flat-square, plastic (current), for-the-badge, and social.

  6. increased visibility of metadata via a dashboard for a specific version. eg. when clicking on a specific version of an artifact, it will list (all of this is in the pom.xml)

    • the name (which may be different from the module name. see: the name field in the POM)
    • if it is a snapshot (some repos could be used to store versions & releases)
    • the description
    • the author/developers (+ contributors?)
    • the license(s)
    • the issues/repo url
    • the inception year
    • the dependencies
  7. the ability to have "repository overlays" or to "inherit repositories".
    eg. let's say I have snapshots and releases. when a user adds the snapshots repo and attempts to fetch a version from it, but said version does not exist in it (or any of its mirrors), it would be nice if it then went and checked if it existed in the releases repo. so, snapshots falls back on/inherits from/is an overlay on top of releases.


~ edit: merged both comments / @dzikoysk

Repository owner deleted a comment from solonovamax Mar 6, 2024
Repository owner deleted a comment from solonovamax Mar 6, 2024
@dzikoysk
Copy link
Owner Author

dzikoysk commented Mar 10, 2024

One thing I would like to see is the ability to customize the appearance

yeah, the ability to provide your own css styles/themes would be quite nice
ie. use the default frontend, but just have some customized css (ie. to make the frontend a bit less "harsh")

I'll most likely implement the new frontend with HTMX+Tailwindcss. These sources don't need a build process, so users should be able to pretty much do whatever they want there. I don't think I'll target the appearance that much tho, because it's quite niche feature, but the idea with a separate CSS file for some most common stuff like e.g. colors might be interesting. I think I'll be able to tell more once I'll start doing this, because I'm not completely sure how it should be done yet.

smth I thought of a while back that would be nice is: the ability to store cached files from a proxied/mirrored repository in a separate location.

This is already possible - just create a new repository per each mirrored repository and link these local mirrors via local loopback link:

You can set individual storage limits per each repo as it'd be a regular repo.

add a badge on remote artifacts/modules to indicate they're from a mirror (& maybe which mirror they're from?)
policy to discard old cached artifacts

Currently, we're only providing a built-in option to delete snapshot files, but if we'd have an option to determine if a given file was proxied or not, then date based cleanup for mirrors would be a great enhancement. This is not really related to 3.x/4.x, so I created a new issue for this:

the ability to change the name of the repository provided in the generated "repository details" for maven, gradle kotlin, gradle groovy, and sbt. (eg. I'd like the "name" to be smth specific rather than what was generated, as well as using specific capitalization, etc.)

It was requested a lot of times - not sure how it'll look like on the new frontend, but in general this is already planned ✔️

since you're looking at making the frontend part of the plugin api, should a search plugin be added to the roadmap? if it is, I could start that one thing I was working on a while ago with maven-indexer back up.

I still think that the search option in UI should be implemented internally in Reposilite, based on some metadata we'll store in the database. Speaking about the maven-indexer, we have a separate issue for this:

There were some issues with it, if you have some more data on this, it'd be nice to continue this thread there, just so we can keep this in one place.

funny numbers: it would be cool if the statistics showed more information, [...] unsure if this might be better suited for just sending that data to prometheus and using grafana for this, though.

The problem with detailed statistics is that it highly increases the database usage - that's why we're currently only counting unique requests per month by default. On the other hand, the prometheus integration will be merged into the core sources, so we should be fine with just pushing some extra events instead of storing this on our own. In this case, it's also not important to address this directly in 4.x - extra metrics can be added later as a feature requests.

minor change to the badges api: support for different styles (see: https://shields.io/badges), eg. flat, flat-square, plastic (current), for-the-badge, and social.

It wouldn't be a breaking change, so I think it could be moved to a standalone issue if needed. In general, I have a limited time for the project, so such issues usually have really low priority for me, but feel free to submit a PR - it's cool to have such options.

increased visibility of metadata via a dashboard for a specific version. eg. when clicking on a specific version of an artifact, it will list (all of this is in the pom.xml)

Probably could be addressed after 4.x, but I'll try to remember about this after rewriting the layout.

the ability to have "repository overlays" or to "inherit repositories".
eg. let's say I have snapshots and releases. when a user adds the snapshots repo and attempts to fetch a version from it, but said version does not exist in it (or any of its mirrors), it would be nice if it then went and checked if it existed in the releases repo. so, snapshots falls back on/inherits from/is an overlay on top of releases.

I think this should be also already possible with local links 🤔 nevermind, it'd require from us to generate a merged pom.xml file on the fly to contain all versions from different repositories. I think this is highly unlikely I'll do that in the core module, but such things should be possible to implement on plugin API level, and if not, we can work towards the enhancements in this area.

@andrm
Copy link
Sponsor

andrm commented Mar 15, 2024

Ability to hostdart/flutter packages: https://github.com/dart-lang/pub/blob/master/doc/repository-spec-v2.md

@pauliesnug
Copy link
Contributor

Copying this over here just in case: #1011 (comment)

@pauliesnug
Copy link
Contributor

Maintaining current REST API + standalone frontend module requires a lot of work. With JTE-like template engine + some lightweight library like HTMX + Tailwind, we could probably do the same thing with a better result (and improve mobile UX)

As for the UI of Reposilite, I think that using something like Tailwind or UnoCSS would be a good fit. I wonder if there could be a redesign of the way that data was fetched from the backend and the frontend that wouldn't require a templating engine serving the content. I wonder if there is some sort of RPC that existed for generating Web based code from Kotlin types and APIs? If not then a templating engine and a lightweight framework with styling sugar sounds like a good step towards there, as I don't think Kotlin has amazing web features.

@dzikoysk
Copy link
Owner Author

dzikoysk commented May 3, 2024

I wonder if there could be a redesign of the way that data was fetched from the backend and the frontend that wouldn't require a templating engine serving the content.

I'm a bit tired of introducing intermediate layers, just to display some sort of static content. Having a template engine, I could possibly get rid of the whole frontend module. There are some drawbacks as always, but as far as I see it could be an option to significantly reduce the size of the codebase, so it'd be easier to maintain it.

Another benefit of having the frontend generated by backend is extensibility - currently, there is no option to modify frontend via plugin API, because frontend is just a standalone app that is precompiled earlier. It's even hard to inject custom base path url, not speaking about some more advanced effects.

I wonder if there is some sort of RPC that existed for generating Web based code from Kotlin types and APIs

Even if we'd have some sort of RPC framework, it'd still need to make some calls over the network to fetch/update latest state for client-side logic. It means we're falling once again into the world with 2 independent states (client and server) that we need to synchronize at some point.

If not then a templating engine and a lightweight framework with styling sugar sounds like a good step towards there, as I don't think Kotlin has amazing web features.

I had some time to prototype a basic plugin for Javalin, and it's promising:

7GvUf0FLo5.1.mp4

So we can prototype frontend with hot-reloads. Of course, for backend changes we need to rebuild a project, but it's good enough I'd say. :shipit:

@solonovamax
Copy link
Contributor

Of course, for backend changes we need to rebuild a project, but it's good enough I'd say. :shipit:

if you don't introduce any methods/etc. you can hotswap code in intellij when running in debug mode. I personally have a macro set up to execute:

  • SaveAll action
  • CompileProject action
  • Hotswap action

when pressing ctrl + s (you can create a similar macro by starting macro recording, using double shift then searching for and executing each action, and then afterwards edit the macro to remove all the double shifts & other useless stuff)

it won't hotswap anything which introduces new methods or new classes, but if it's just a change to the method body, it hotswaps it. quite useful.

@pauliesnug
Copy link
Contributor

So we can prototype frontend with hot-reloads. Of course, for backend changes we need to rebuild a project, but it's good enough I'd say. :shipit:

That's perfect yeah! I understand that reasoning.

@pauliesnug
Copy link
Contributor

Even if we'd have some sort of RPC framework, it'd still need to make some calls over the network to fetch/update latest state for client-side logic. It means we're falling once again into the world with 2 independent states (client and server) that we need to synchronize at some point.

I do feel like that for this, if an RPC framework existed, there are good solutions for syncing up backend and client. Even just a simple SSE connection could go a long way for a simple dynamic frontend, and if there was an RPC, all the API infrastructure for the framework side code could be standardized and modified from the backend, without requiring a change to the frontend. Obviously this is easier said than done, but I'd be happy to look into if this is a possible solution, since generally the downsides of a templating engine can introduce limitations in the future, but I think in this case, either solution would work generally well.

@luolong
Copy link

luolong commented May 6, 2024

As long as we are talking about choices of frontend technology, what about going for a nice "clean" HTMX powered front-end?

Backend could use your own favorite templating engine to serve html smippets, so we get best of both worlds -- fast in-place reloads and templated/skinnable frontend.

@solonovamax
Copy link
Contributor

As long as we are talking about choices of frontend technology, what about going for a nice "clean" HTMX powered front-end?

Backend could use your own favorite templating engine to serve html smippets, so we get best of both worlds -- fast in-place reloads and templated/skinnable frontend.

afaik, htmx was/is already being considered. (and, from what I can tell, is currently the preferred option)

@dzikoysk
Copy link
Owner Author

dzikoysk commented May 30, 2024

We've got a decent conversation regarding the 4.x, so I'd say we're close to finalizing the RFCs list. I'll try to refine this in the upcoming month, so we'll have a batter overview of what needs be done.

Looking at my schedule, I think the roadmap is still quite up-to-date - the development phase will more likely start in Q3.

@freshpr
Copy link

freshpr commented Jun 5, 2024

So searching for an artifact didn't make it into the list? :(

@dzikoysk
Copy link
Owner Author

dzikoysk commented Jun 6, 2024

The RFCs list is not yet updated, but I can already say that some sort of artifact browser should be a part of 4.x :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking Issue breaks the compatibility with previous versions investigation Issue is currently investigeted, e.g. author is trying to reproduce problem next release Details about planned release
Projects
None yet
Development

No branches or pull requests

9 participants