Support executing multiple dependent long-running tasks in parallel

[robaca]

Describe the feature you'd like to request

In our monorepo, we have multiple services that to some degree depend on each other. For local development, we have to start a mock service first, then a second one, and then all others in any order.

For local development and testing, it would be great, to be able to start all these services or only one service and it's dependencies via turborepo in one step.

Describe the solution you'd like

It would be cool if turbo can be configured to not wait for a process to exit, but for some string or regexp to appear on stdout|stderr and put all processes into background until Turbo itself is terminated via Ctrl+C.

We could then just configure our server tasks like this:

    "mock-service#server": {
      "dependsOn": ["build"],
      "longRunning": {
        "waitFor": {
          "stdout": "Service started successfully"
          "timeout": "60s"
        },
      },
      "cache": false
    },
    "my-service#server": {
      "dependsOn": ["build", "mock-service#server"],
      "longRunning": true, 
      "cache": false
    },

For dependencies, it would be needed to be able to specify an explicit package, as runtime dependencies do not necessarily match build time dependencies. I'm not sure if this is already possble, but at least it's not documented.

A leaf task should not need to configure a waiting condition, but having them for all background tasks would make it possible to output some informative message that all background tasks had a successful startup and now everything is up and running.

If a dependency is detected on a task that has longRunning: true, turbo could fail to run because of misconfiguration.

Nice-to-have on top:

• Being able to configure what should happen with console output of these background services (showing them, hiding them, only showing stderr)
• having a repl to give runtime hints to turbo after all processes are in background, like /hide mock-service#server, /grep ERROR

Describe alternatives you've considered

Create a command line tool that starts a process, puts it in background and exits by itself when some message appears on stdout or stderr. I'm not sure if it's possible then to stop that background processes if the turbo command is interrupted via Ctrl+C.

[jlarmstrongiv]

Reminds me of https://www.npmjs.com/package/wait-on

[edwardwilson]

I have a similar requirement. In my mono repo, I have an app in which I need to have multiple long-running processors started at the beginning. The order doesn’t really matter to me. I just have three long-running that at all need to start.

I have a Vite build with the watch flag. Another process that needs to be run and a custom web server process. Each can start independently and in parallel.

This can be achieved with NPM, Scripps and packages to enable parallel processes, but turborepo does not seem to support this via the pipelines. Turborepo will wait for the first process to complete, which never happens due to the long runing nature.

[grabbou]

That would be so cool. Imagine having multiple packages that all need to spin up (run start-dev) in order for you to start working.

[weyert]

Yeah, this would be useful, I sometimes want to run some integration tests and want to make sure the backend services are running before they start.

[grabbou]

Not to spam anymore in the thread, but for anyone looking for a nice alternative on top of turborepo would be to configure Visual Studio Code to have a special "task" to spin all processes in a shared terminal session. Here's my gist that spins two processes, useful for development workflow: https://gist.github.com/grabbou/1e19049ebc6127b269f4230bfaed5170

[spigelli]

I've been looking for something like this for a while now. For me this addresses my docker compose problem.

I don't like to keep my docker compose services running in the background:

1. Ports conflict with my other projects when I forget to docker-compose down
2. Some things are process intensive

Additionally things start to get messy when you're integrating a few OSS tools that each depend on multiple services. Imagine for example you're building some sort of webrtc app,
you're integrating the following tools and they're docker services:

• Open Telem Services
  • prometheus
  • alertmanager
  • nodeexporter
  • cadvisor
  • grafana
  • pushgateway
  • caddy
• Supabase Services
  • studio
  • kong
  • auth
  • rest
  • realtime
  • storage
  • meta
  • db
• Livekit:
  • traefik
  • livekit-server
  • redis

So what I've considered in the past is making small npm wrappers as "apps" for different associated services, since that's where they would be if they were run from source.

[hymair]

In my case I want to run these in sequence and I can't find a proper way to do it currently as dev is a long-running task so dependsOn never resolves.

1. run backend#dev
2. run app#gen-types
3. run app#dev
4. run web#dev

[VanCoding]

Hi guys

I want to throw my protocol into the mix as a possible solution to this as well.

It's a very simple way how task-runners like (turborepo, Nx, you name it) could talk to non-terminating watch-processes and tell them when to rebuild, and get the results of those builds.

The idea is that we settle on one protocol, that then can be implemented by a lot of task runners and build tools. I'd really like to get your feedback on this! I hope to get the discussion started here

[sschneider-ihre-pvs]

Hi guys

I want to throw my protocol into the mix as a possible solution to this as well.

It's a very simple way how task-runners like (turborepo, Nx, you name it) could talk to non-terminating watch-processes and tell them when to rebuild, and get the results of those builds.

The idea is that we settle on one protocol, that then can be implemented by a lot of task runners and build tools. I'd really like to get your feedback on this! I hope to get the discussion started here

the protocol looks a xstate state machine

[eboody]

Hi guys

I want to throw my protocol into the mix as a possible solution to this as well.

It's a very simple way how task-runners like (turborepo, Nx, you name it) could talk to non-terminating watch-processes and tell them when to rebuild, and get the results of those builds.

The idea is that we settle on one protocol, that then can be implemented by a lot of task runners and build tools. I'd really like to get your feedback on this! I hope to get the discussion started here

could you provide an example of how I would get started using your protocol for managing tsup --watch or tsc -w to restart some file-watching server when some other package has a file change?

[kirill-konshin]

Usually this kind of management is required for (as bare minimum) TS lib + website, when TS lib in watch mode has to produce some output at least once, then website can pick up the output and keep watching it.

See my old article, section Starting/watching. The easiest approach so far would be to run regular build (not watch) task, which can successfully end, and then run watch task but without initial output.

Unfortunately, this brings overhead until webpack/webpack#4991 is fixed (previously TS was prone too: bug 12996.
UPDATE In TypeScript 3.4 new incremental option has been introduced: it will produce a build and a cache, so no matter how often you restart watchers it will be ready much faster. Unfortunately it’s not yet supported by TS Loader for Webpack It still does not eliminate the necessity to pre-build libraries.

In any case build+watch approach for library may cause web to be built twice, first after build then after first output of watch, if website rebuild is looking at timestamps, not contents.

[dobesv]

If you have interdependent processes you want to run, I think just putting a "wait" task in front of them makes sense. Can use wait-on to wait for the port to open.

This doesn't seem to require special support from turborepo to work.

[kirill-konshin]

True, but it’s an extra tool and it looks awkward to have all such scripts like this: “wait-on blabla && real-thing”. It would be much more convenient if turbo config can have “waitOn” property for such tasks. Property can either call an NPM script that has to exit when condition is met, or a list of files/globs/URLs.

It would be a nice advantage over NX which has same problem now.

I've implemented the wait approach in my Next Redux Wrapper repo, it works, but does not look as sleek as Turbo configs can look:

https://github.com/kirill-konshin/next-redux-wrapper/blob/3a14963a0c8a3ebf39baa331b6a467bbc6cb8ee5/packages/wrapper/package.json#L21
https://github.com/kirill-konshin/next-redux-wrapper/blob/3a14963a0c8a3ebf39baa331b6a467bbc6cb8ee5/packages/demo-redux-toolkit/package.json#L9

[dobesv]

Hmm true. Not sure how far down the road of being a dev job runner turbo wants to go but it could be handy to have all that stuff integrated, especially if it could watch files and rebuild and restart the dev processes.