[RFC] Decouple Turbo tasks from package.json scripts And allow more automation #7265

zaripych · 2022-07-07T13:51:11Z

zaripych
Jul 7, 2022

Describe the feature you'd like to request

I've been thinking about this for some time now. The idea keeps nagging me. Imagine - I have a dts task that I can run for every package in my monorepo, but that task doesn't have any value for the end user (read developer in a team using the monorepo) by itself - meaning user will never run that task manually via turbo run or pnpm run. The task is needed to run another dts-bundle task - essentially dts produces an intermediate artefact that allows dts-bundle that depend on it to succeed without errors. Now the problem is that dts-bundle needs to run dts for all dependencies.

This seem to work nicely:

  "pipeline": {
    "dts": {
      "inputs": [...],
      "outputs": [".tsc-out"]
    },
    "dts-bundle": {
      "dependsOn": ["^dts"],
      "inputs": [...],
      "outputs": ["dist/**/*.d.ts"]
    },
  }

But little issues creep up - mostly scalability:

Every package has to have that dts task declared in package.json scripts - it probably wouldn't make sense to have to copy-paste it everywhere since the task is basically single script that can work in every package sub-folder and produces output in that package sub-folder, but has no input other than source code and doesn't need any configuration on per package basis
The dts task must be run for every dependency of a package where we about to run dts-bundle, but a developer might forget to add a script to package.json for one of those dependencies - when adding that dts script they might also make a mistake and pass wrong parameters or things like that
When a developer decides that they want to start using dts-bundle for their package they have to go and check every dependency and whether that dependency has dts task declared. Hopefully it's very easy change and a quick PR away, but those packages could be owned by other teams.

In addition to that it's also possible that inputs of some other build task in one package is different to another package's inputs. But the fact that those differences should be specified in the root of the monorepo in a single file is kind of limiting. If we look at monorepos with huge number of packages I feel single turbo.json might become a bottleneck (with some merge issues maybe even?). Sometimes a script can determine those inputs/outputs programmatically and can operate in the entire monorepo for any sub-package. E.g. I imagine a script could determine all the dependent *.ts files imported and build a graph of files that it depends on (inputs) - and that would probably be more efficient than using globs like src/**/*.ts that could hit hundreds of files.

Another script could determine which dependencies need to have dts task run and which don't. There is generally a lot of things that can and should be automated and turbo doesn't provide too much here. The CLI could be considered as API and turbo.json can be generated - but it makes it harder to use hybrid approach - use ability of the turbo to cache and also allow devs to use turbo.json without having to deal with a script.

Describe the solution you'd like

Allow specifying commands right in the turbo.json, example:

  "pipeline": {
    "dts": {
      "command": "node ./monorepo-scripts/generateDts.js",
      // ^-- no need to copy-pasta in every package
      "inputs": [...],
      "outputs": [".tsc-out"]
    },
  }

turbo would assume the command exists in every package as if it was added to every package's scripts config.

Allow passing location of a custom turbo.dts.json file OR preferably - allow it to be passed as stdin. If location of the turbo.json implies root of the monorepo we could also add extra parameter or assume it's cwd. This will make turbo super powerful tool that would allow caching any task - whether it can be described by turbo.json manually or generated by a script.

Both of the above seem like a simple change on the surface. But there must be some caching implications. E.g. we would need to store hash of the stdin config somewhere which could serve as a cache key for the rest of the data associated with the run ...

Describe alternatives you've considered

Generating turbo.json is possible, but writing scripts to every package.json - seem like a less desired path as it's often modified manually.

jlarmstrongiv · 2022-07-09T20:18:07Z

jlarmstrongiv
Jul 9, 2022

@zaripych in the meantime, here’s a workaround for you.

In my projects, I have been duplicating package scripts just as you described. But, I try to abstract away all of the details. I have many package.json files that look like:

{
  "scripts": {
    "turbo:build": "../../../bin/nextjs/build",
    "turbo:clean": "../../../bin/clean/all",
    "turbo:dev": "../../../bin/nextjs/dev",
    "turbo:test:ci": "../../../bin/jest/ci",
  }
}

The only thing is, most packages have 40+ scripts, not 4.

I find that abstracting the scripts into a common bin folder helps a lot (I also like prefixing shared scripts with turbo:).

As my monorepo has grown, I have different types of packages. Some for nodejs, some for reactjs, some for apps, some for configs. There’s only a few scripts that are actually run in every single repo due to turbo disallowing circular dependencies. This abstraction allows me to group sub-packages and sub-apps like packages/reactjs/* or apps/nextjs/*. Plus, I can exclude config packages altogether by just not including the script. Lastly, it still lets me run scripts very easily in just the package or two that I’m working on via npm run <script>, without having to specify filter each time in the package root.

As for ensuring every package has a particular script, I find that using a template generator is awesome (see plop and hygen). Or, writing a short script in bash or the language of your choice is straightforward.

Anyway, the developer experience of turbo is interesting to explore. Reminds me of:

While I prefer convention over configuration too, it’s important not to become overspecialized and ensure exceptions can be specified.

0 replies

zaripych · 2022-07-10T13:08:30Z

zaripych
Jul 10, 2022
Author

@jlarmstrongiv Actually, already doing that. I have a dev package that is mounted as devDependencies to the root of monorepo and has bin's exposed and available to every sub-package. So no need to "../../thing" I just add it as "thing". It's all same shared scripts for every package that use same bespoke monorepo configs/conventions/tools.

The only thing is, most packages have 40+ scripts, not 4.

That's a lot.

Plus, I can exclude config packages altogether by just not including the script. Lastly, it still lets me run scripts very easily in just the package or two that I’m working on via npm run <script>, without having to specify filter each time in the package root.

Ok. I'm talking about different tasks that do not belong in scripts of a package but which I still want to cache and benefit from turbo. Developers won't ever care to run them manually.

As for ensuring every package has a particular script

Agreed. Code generation is ok as long as you do it within a monorepo and can regenerate it a little differently at any time 👍
Meaning results of the previous generation can be thrown away and changed at any time to something else. This becomes hard and dangerous for files which are also hand-edited or when you need to take backward compatibility into consideration. E.g. you can go and overwrite somebody else's script in package.json ... so - I'm doing code generation but making sure to avoid mixing hand-edited code and generated code.

#1412

FYI - I was annoyed and fixed that by wrapping standard turbo bin with my own. Now I can run it in sub-packages.

Anyway, the developer experience of turbo is interesting to explore.

It's still a little raw and a bit buggy for now. (E.g. at the moment running a canary because previous version was hitting the cache too much) I'm very enthusiastic though to see how this will evolve further and really considering using it in more projects at work.

it’s important not to become overspecialized

Agree. Customisation is needed at times, but those also might mean that turbo.json needs changing. If we want to over-blow this - imagine you need to customise every script - so multiply number of packages you have (I've seen people talking 200+ in some threads here) by the number of scripts you have (40+ ? :D ) and put it all into turbo.json pipeline ... will end-up containing 1k of task configs with entries like package1#build, ... package200#build ... fun 😅

0 replies

jlarmstrongiv · 2022-07-10T22:53:43Z

jlarmstrongiv
Jul 10, 2022

Actually, already doing that. I have a dev package that is mounted as devDependencies to the root of monorepo and has bin's exposed and available to every sub-package. So no need to "../../thing" I just add it as "thing". It's all same shared scripts for every package that use same bespoke monorepo configs/conventions/tools.

FYI - I was annoyed and fixed that by wrapping standard turbo bin with my own. Now I can run it in sub-packages.

Very cool! I like those workarounds.

I wonder if it should be a broader discussion on turbo UX. Discussion points include:

avoid defining scripts in the turbo.json (Zero-config #749 and Execute Arbitrary package.json Scripts #1254)
avoid defining duplicate scripts in the package.json (Decouple Turbo tasks from package.json scripts And allow more automation #1495)
ability to run common turbo tasks anywhere (Execute turbo from anywhere in a monorepo #1412)
ability to run a common turbo task in a package without defining it in scripts
configure the cache and exceptions to particular commands (https://turborepo.org/docs/core-concepts/caching and https://turborepo.org/docs/core-concepts/filtering and package#build)
- group common turbo tasks by subfolder (like apps/nextjs/* or packages/reactjs/*)
- group common turbo tasks by npm name prefix (@apps/*)
keep turbo fast by avoiding js/ts config files Request for having a turbo.js or/and turbo.ts to configure turbo #899
support scripts from other languages Add engines.turbo Validation #1361 (comment)

How would that look? Curious what your thoughts are on it. There’s a lot to think about and there are tradeoffs in choosing what to support too.

Ok. I'm talking about different tasks that do not belong in scripts of a package but which I still want to cache and benefit from turbo. Developers won't ever care to run them manually.

I think there are use cases where devs would want to run some common scripts manually in a package. Examples https://www.npmjs.com/package/sort-package-json and https://www.npmjs.com/package/typesync

0 replies

dsilvasc · 2023-02-01T12:42:14Z

dsilvasc
Feb 1, 2023

As a shorter term solution (until turbo supports additional config for this in turbo.json), when turbo doesn't find a task in a package's package.json scripts, can it then check if a command with that name exists in the context of that package?

If making it possible to declare this for all packages in turbo.json is far away in the turbo roadmap, then maybe a quicker change might be for turbo to do something similar to yarn bin $TASK_NAME within each package to discover commands provided by devDependencies for tasks when corresponding scripts are omitted.

For example, suppose your turbo.json defines a test task that depends on test-unit and test-ui, but you don't want to repeat this for every package.json:

"scripts": {
  "test": "true",
  "test-unit": "jest --passWithNoTests",
  "test-ui": "our-custom-ui-testing-script",
  ...
}

Today you can already define a package like @zaripych mentioned providing executables with the same names as those scripts/tasks, so your package.json files become:

"scripts": {
  "test": "test",
  "test-unit": "test-unit",
  "test-ui": "test-ui",
  ...
}

Running yarn bin test-ui yields $PWD/node_modules/.bin/test-ui, or $REPO_ROOT/node_modules.bin/test-ui (when hoisted), or a virtual path (when using yarn berry/v3 pnp), or some other place. Either way, turbo should be able to use that, since yarn already does: you can run yarn test-ui in the example above without the entry in package scripts.

0 replies

dobesv · 2023-02-09T01:13:35Z

dobesv
Feb 9, 2023

Another use case to throw in here is in our project in each package.json I have a script "build": "yarn build:node && yarn build:browser && yarn build:types".

It would be nice if I could tell turbo that build depends on build:node, build:browser, and build:types but that it shouldn't actually run build.

Currently I just use a different fake name for the build task for each package, _build.

0 replies

connorjs · 2023-03-22T18:44:12Z

connorjs
Mar 22, 2023

../../../bin/some-script

I wanted to share a new approach I am using for these cases: bin scripts.

Turbo already recommends having “config” packages (example: eslint-config). I then have the root package depend on the config as a dev dependency. Thus, instead of having a root bin directory and using ../../ to access it, I have the config package publish a bin script itself.

After some self-bike-shedding, I arrived at *x naming convention: something simple (one extra character), somewhat valid naming (x for eXtended), and still showcases the underlying script (which I found helpful, because I have multiple linter tools).

Then, in each workspace/sub-package I can replace eslint <args> or ../../bin/lint <args> with eslintx <args>.

Code example.

eslint-config’s package.json

{
	// normal package.json stuff...
	"directories": { "bin": "bin" }
}

eslint-config’s bin/eslintx (swap the command invocation with whatever you want)
```
#!/bin/sh
eslint -f pretty "$@"
```

some other package’s package.json

{
	// normal package.json stuff...
	"scripts": {
		// other scripts...
		// before
		"lint": "eslint -f pretty ."
		// after
		"lint": "eslintx ."
	}
}

0 replies

btucker · 2025-04-08T22:00:45Z

btucker
Apr 8, 2025

Curious how you all have evolved your solutions to this in the past couple years? We're in the process of getting a new turbo monorepo stood up & the duplication of scripts in each package.json is concerning me from a maintainability perspective.

0 replies

dobesv · 2025-04-09T00:33:34Z

dobesv
Apr 9, 2025

I've been trying Microsoft lage recently. I wish I could recommend it, unfortunately it has some bugs and gaps still.

It has the ability to define worker scripts that stay resident between packages, so you don't pay full startup time for those tools for every package. This speeds up my builds by at least 10x compared to turbo.

Also it is more flexible in the configuration of dependencies because the dependency file is a JS file so I can use code to define (or not define) the tasks and get very precise about what I want.

7 replies

dobesv Apr 15, 2025

I should probably note that I don't think babel is where I'm seeing the biggest gains here, although there are definitely significant gains. TypeScript and ESLint (with the TypeScript plugins) are the true hogs here, and TypeScript is a big tool that takes some time to start.

dobesv Apr 15, 2025

Oh and I just realized I started talking about babel above when we were supposed to be talking about eslint.

Here's the time it takes me to lint a package with 133 files:

$ time yarn lint

real    0m6.901s
user    0m10.880s
sys     0m1.029s

I think the TypeScript plugins are probably the biggest offender, it seems to take TypeScript a few seconds to get moving due to the size of the project and how many dependencies we have that it has to parse.

dobesv Apr 15, 2025

I thought I'd try eslint for depcheck as well to compare:

$ time yarn lint
  1 error and 0 warnings potentially fixable with the `--fix` option.


real    0m3.051s
user    0m4.925s
sys     0m0.335s

Seems to take 3 seconds, but actually I don't know how much is "startup" and how much is actual work 🤔

dobesv Apr 15, 2025

A bit of experimentation suggests yarn is adding a lot to the startup time somehow:

$ time yarn node -e 'require("@babel/core"); console.log("hello, world")'
hello, world

real    0m1.041s
user    0m1.219s
sys     0m0.237s
$ time yarn node -e 'console.log("hello, world")'
hello, world

real    0m0.733s
user    0m0.883s
sys     0m0.200s
$ time node -e 'console.log("hello, world")'
hello, world

real    0m0.016s
user    0m0.012s
sys     0m0.004s

Although babel is 300ms over plain console.log, yarn seems to be adding >600 ms. Interesting, and troubling.

anthonyshew Apr 15, 2025
Maintainer

Okay, cool, those numbers track with my expectations. I was wondering if we somehow had a bug that made starting up your tasks outrageously slow such that you ended up 10x faster. 😁

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] Decouple Turbo tasks from package.json scripts And allow more automation #7265

{{title}}

Replies: 8 comments 7 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[RFC] Decouple Turbo tasks from package.json scripts And allow more automation #7265

Describe the feature you'd like to request

Describe the solution you'd like

Describe alternatives you've considered

Replies: 8 comments · 7 replies

zaripych Jul 10, 2022 Author

anthonyshew Apr 15, 2025 Maintainer

Replies: 8 comments 7 replies

zaripych
Jul 10, 2022
Author

anthonyshew Apr 15, 2025
Maintainer