Temporal Job Runner might help with the workflows / activities

[gedw99]

Is your feature request related to a problem? Please describe.

I noticed many issues related to Orchestration, and so think that perhaps a workflow runner is needed to track it all

Describe the solution you'd like

https://github.com/temporalio/temporal/ i think is pretty awesome. Its from the same team that build Uber Cadence, but much easier to use and looks quite solid.

https://docs.temporal.io/docs/server-quick-install

They have a golang SDK and samples.

If you want really decoupled you can even use a DSL like this sample: https://github.com/temporalio/samples-go/tree/master/dsl

Describe alternatives you've considered

Cadence, NATS, etc

Additional context

If you hate it its cool as i know dependencies are tough to decide on. I noticed that Temporal is very decoupled from whatever docker project you use it with.

[gedw99]

Here is a makefile that makes testing it easy.

# https://docs.temporal.io/docs/server-quick-install

LIB_NAME=docker-compose
LIB=github.com/temporalio/$(LIB_NAME)
CLI=docker exec temporal-admin-tools tctl

SAMPLE_LIB_NAME=samples-go
SAMPLE_LIB=github.com/temporalio/$(SAMPLE_LIB_NAME)


print:

	@echo
	@echo LIB: $(LIB)

	@echo SAMPLE_LIB $(SAMPLE_LIB)

dep:
	# docker 
	git clone https://$(LIB).git

	# go samples
	git clone https://$(SAMPLE_LIB)

docker-up:
	# works
	cd $(LIB_NAME) && docker-compose
	cd $(LIB_NAME) && docker-compose up

docker-tag:
	cd $(LIB_NAME) && $(MAKE) update-tag

ui-open:
	open http://localhost:8088/

cli-up:
	# works 
	$(CLI) namespace list
	$(CLI) workflow list

run-hello-worker:
	# works
	cd $(SAMPLE_LIB_NAME) && go run helloworld/worker/main.go
run-hello:
	# works
	cd $(SAMPLE_LIB_NAME) && go run helloworld/starter/main.go

[gedw99]

OMG: https://github.com/nirpadma/temporal-workflows/tree/main/media_processing_workflow :)

[BrenekH]

I suppose I am a bit confused as to what issues you are talking about and trying to solve with Temporal.

1. What exactly do you mean by "workflows/activities" in the title? Are they the FFmpeg jobs that the Runner completes, or is it more on the Controller side with scheduling jobs/discovering files?
2. What is "Orchestration"? I imagine it has to do with the Controller discovering files to do work on and dispatching that to Runners, but I'm not sure.

Overall, Temporal does look like an interesting project that could help Encodarr, but I don't think it should be the default. I'm definitely interested in making it an option for people to use, but I want to keep the required dependency count as low as possible so that Encodarr is dead simple to set up.

I will also note that I am working on redoing the entire architecture of the Controller code so that it has a more dependency injection style, similar to how the Runner is currently designed. I imagine it's going to take a bit of time to accomplish, but with that in place, Temporal as a configurable option would be much more feasible.

[BrenekH]

Closing because of inactivity

[gedw99]

sorry i missed this.

ok i read your comments and yes the issue is about managing the jobs.

I have been using Temporal now for a few weeks, and compared to others out here i feel its the best.
the webUI also makes it very easy to understand what its doing.

so thumbs up from me on this...

[gedw99]

also there is perhaps another option that Temporal.

Temporal needs a DB and Elastic Search under it.

https://www.benthos.dev/ needs nothing under it, and its pure golang

it also has exactly one semantics when it calls an API.

Still undecided to be honest...

[BrenekH]

I'm going to go ahead and reopen this because user choice is always good and this isn't a bad idea, but I won't be working on integrating an external job dispatch/management system myself. I'd much rather work on getting the base system as stable as possible all on its own.

That doesn't mean that someone else can't work on this and submit a PR.