1  # jest-worker
  2  
  3  Module for executing heavy tasks under forked processes in parallel, by providing a `Promise` based interface, minimum overhead, and bound workers.
  4  
  5  The module works by providing an absolute path of the module to be loaded in all forked processes. Files relative to a node module are also accepted. All methods are exposed on the parent process as promises, so they can be `await`'ed. Child (worker) methods can either be synchronous or asynchronous.
  6  
  7  The module also implements support for bound workers. Binding a worker means that, based on certain parameters, the same task will always be executed by the same worker. The way bound workers work is by using the returned string of the `computeWorkerKey` method. If the string was used before for a task, the call will be queued to the related worker that processed the task earlier; if not, it will be executed by the first available worker, then sticked to the worker that executed it; so the next time it will be processed by the same worker. If you have no preference on the worker executing the task, but you have defined a `computeWorkerKey` method because you want _some_ of the tasks to be sticked, you can return `null` from it.
  8  
  9  The list of exposed methods can be explicitly provided via the `exposedMethods` option. If it is not provided, it will be obtained by requiring the child module into the main process, and analyzed via reflection. Check the "minimal example" section for a valid one.
 10  
 11  ## Install
 12  
 13  ```sh
 14  $ yarn add jest-worker
 15  ```
 16  
 17  ## Example
 18  
 19  This example covers the minimal usage:
 20  
 21  ### File `parent.js`
 22  
 23  ```javascript
 24  import JestWorker from 'jest-worker';
 25  
 26  async function main() {
 27    const worker = new JestWorker(require.resolve('./Worker'));
 28    const result = await worker.hello('Alice'); // "Hello, Alice"
 29  }
 30  
 31  main();
 32  ```
 33  
 34  ### File `worker.js`
 35  
 36  ```javascript
 37  export function hello(param) {
 38    return 'Hello, ' + param;
 39  }
 40  ```
 41  
 42  ## Experimental worker
 43  
 44  Node 10 shipped with [worker-threads](https://nodejs.org/api/worker_threads.html), a "threading API" that uses SharedArrayBuffers to communicate between the main process and its child threads. This experimental Node feature can significantly improve the communication time between parent and child processes in `jest-worker`.
 45  
 46  Since `worker_threads` are considered experimental in Node, you have to opt-in to this behavior by passing `enableWorkerThreads: true` when instantiating the worker. While the feature was unflagged in Node 11.7.0, you'll need to run the Node process with the `--experimental-worker` flag for Node 10.
 47  
 48  ## API
 49  
 50  The only exposed method is a constructor (`JestWorker`) that is initialized by passing the worker path, plus an options object.
 51  
 52  ### `workerPath: string` (required)
 53  
 54  Node module name or absolute path of the file to be loaded in the child processes. Use `require.resolve` to transform a relative path into an absolute one.
 55  
 56  ### `options: Object` (optional)
 57  
 58  #### `exposedMethods: $ReadOnlyArray<string>` (optional)
 59  
 60  List of method names that can be called on the child processes from the parent process. You cannot expose any method named like a public `Worker` method, or starting with `_`. If you use method auto-discovery, then these methods will not be exposed, even if they exist.
 61  
 62  #### `numWorkers: number` (optional)
 63  
 64  Amount of workers to spawn. Defaults to the number of CPUs minus 1.
 65  
 66  #### `maxRetries: number` (optional)
 67  
 68  Maximum amount of times that a dead child can be re-spawned, per call. Defaults to `3`, pass `Infinity` to allow endless retries.
 69  
 70  #### `forkOptions: Object` (optional)
 71  
 72  Allow customizing all options passed to `childProcess.fork`. By default, some values are set (`cwd`, `env` and `execArgv`), but you can override them and customize the rest. For a list of valid values, check [the Node documentation](https://nodejs.org/api/child_process.html#child_process_child_process_fork_modulepath_args_options).
 73  
 74  #### `computeWorkerKey: (method: string, ...args: Array<any>) => ?string` (optional)
 75  
 76  Every time a method exposed via the API is called, `computeWorkerKey` is also called in order to bound the call to a worker. This is useful for workers that are able to cache the result or part of it. You bound calls to a worker by making `computeWorkerKey` return the same identifier for all different calls. If you do not want to bind the call to any worker, return `null`.
 77  
 78  The callback you provide is called with the method name, plus all the rest of the arguments of the call. Thus, you have full control to decide what to return. Check a practical example on bound workers under the "bound worker usage" section.
 79  
 80  By default, no process is bound to any worker.
 81  
 82  #### `setupArgs: Array<mixed>` (optional)
 83  
 84  The arguments that will be passed to the `setup` method during initialization.
 85  
 86  #### `WorkerPool: (workerPath: string, options?: WorkerPoolOptions) => WorkerPoolInterface` (optional)
 87  
 88  Provide a custom worker pool to be used for spawning child processes. By default, Jest will use a node thread pool if available and fall back to child process threads.
 89  
 90  #### `enableWorkerThreads: boolean` (optional)
 91  
 92  `jest-worker` will automatically detect if `worker_threads` are available, but will not use them unless passed `enableWorkerThreads: true`.
 93  
 94  ## JestWorker
 95  
 96  ### Methods
 97  
 98  The returned `JestWorker` instance has all the exposed methods, plus some additional ones to interact with the workers itself:
 99  
100  #### `getStdout(): Readable`
101  
102  Returns a `ReadableStream` where the standard output of all workers is piped. Note that the `silent` option of the child workers must be set to `true` to make it work. This is the default set by `jest-worker`, but keep it in mind when overriding options through `forkOptions`.
103  
104  #### `getStderr(): Readable`
105  
106  Returns a `ReadableStream` where the standard error of all workers is piped. Note that the `silent` option of the child workers must be set to `true` to make it work. This is the default set by `jest-worker`, but keep it in mind when overriding options through `forkOptions`.
107  
108  #### `end()`
109  
110  Finishes the workers by killing all workers. No further calls can be done to the `Worker` instance.
111  
112  Returns a Promise that resolves with `{ forceExited: boolean }` once all workers are dead. If `forceExited` is `true`, at least one of the workers did not exit gracefully, which likely happened because it executed a leaky task that left handles open. This should be avoided, force exiting workers is a last resort to prevent creating lots of orphans.
113  
114  **Note:**
115  
116  `await`ing the `end()` Promise immediately after the workers are no longer needed before proceeding to do other useful things in your program may not be a good idea. If workers have to be force exited, `jest-worker` may go through multiple stages of force exiting (e.g. SIGTERM, later SIGKILL) and give the worker overall around 1 second time to exit on its own. During this time, your program will wait, even though it may not be necessary that all workers are dead before continuing execution.
117  
118  Consider deliberately leaving this Promise floating (unhandled resolution). After your program has done the rest of its work and is about to exit, the Node process will wait for the Promise to resolve after all workers are dead as the last event loop task. That way you parallelized computation time of your program and waiting time and you didn't delay the outputs of your program unnecessarily.
119  
120  ### Worker IDs
121  
122  Each worker has a unique id (index that starts with `1`), which is available inside the worker as `process.env.JEST_WORKER_ID`.
123  
124  ## Setting up and tearing down the child process
125  
126  The child process can define two special methods (both of them can be asynchronous):
127  
128  - `setup()`: If defined, it's executed before the first call to any method in the child.
129  - `teardown()`: If defined, it's executed when the farm ends.
130  
131  # More examples
132  
133  ## Standard usage
134  
135  This example covers the standard usage:
136  
137  ### File `parent.js`
138  
139  ```javascript
140  import JestWorker from 'jest-worker';
141  
142  async function main() {
143    const myWorker = new JestWorker(require.resolve('./Worker'), {
144      exposedMethods: ['foo', 'bar', 'getWorkerId'],
145      numWorkers: 4,
146    });
147  
148    console.log(await myWorker.foo('Alice')); // "Hello from foo: Alice"
149    console.log(await myWorker.bar('Bob')); // "Hello from bar: Bob"
150    console.log(await myWorker.getWorkerId()); // "3" -> this message has sent from the 3rd worker
151  
152    const {forceExited} = await myWorker.end();
153    if (forceExited) {
154      console.error('Workers failed to exit gracefully');
155    }
156  }
157  
158  main();
159  ```
160  
161  ### File `worker.js`
162  
163  ```javascript
164  export function foo(param) {
165    return 'Hello from foo: ' + param;
166  }
167  
168  export function bar(param) {
169    return 'Hello from bar: ' + param;
170  }
171  
172  export function getWorkerId() {
173    return process.env.JEST_WORKER_ID;
174  }
175  ```
176  
177  ## Bound worker usage:
178  
179  This example covers the usage with a `computeWorkerKey` method:
180  
181  ### File `parent.js`
182  
183  ```javascript
184  import JestWorker from 'jest-worker';
185  
186  async function main() {
187    const myWorker = new JestWorker(require.resolve('./Worker'), {
188      computeWorkerKey: (method, filename) => filename,
189    });
190  
191    // Transform the given file, within the first available worker.
192    console.log(await myWorker.transform('/tmp/foo.js'));
193  
194    // Wait a bit.
195    await sleep(10000);
196  
197    // Transform the same file again. Will immediately return because the
198    // transformed file is cached in the worker, and `computeWorkerKey` ensures
199    // the same worker that processed the file the first time will process it now.
200    console.log(await myWorker.transform('/tmp/foo.js'));
201  
202    const {forceExited} = await myWorker.end();
203    if (forceExited) {
204      console.error('Workers failed to exit gracefully');
205    }
206  }
207  
208  main();
209  ```
210  
211  ### File `worker.js`
212  
213  ```javascript
214  import babel from '@babel/core';
215  
216  const cache = Object.create(null);
217  
218  export function transform(filename) {
219    if (cache[filename]) {
220      return cache[filename];
221    }
222  
223    // jest-worker can handle both immediate results and thenables. If a
224    // thenable is returned, it will be await'ed until it resolves.
225    return babel.transformFileAsync(filename).then(result => {
226      cache[filename] = result;
227  
228      return result;
229    });
230  }
231  ```