Describe the bug
The custom and voice-crescendo red-team strategies have the same provider substitution defect as #10970 (fixed for iterative/crescendo in #10972): when remote generation is enabled, an explicitly configured redteamProvider is silently replaced with Promptfoo's cloud API.
In src/redteam/providers/custom/index.ts, both provider accessors take the remote path whenever shouldGenerateRemote() is true, even though the local branches were written to honor the config:
private async getRedTeamProvider(): Promise<ApiProvider> {
if (!this.redTeamProvider) {
if (shouldGenerateRemote()) {
this.redTeamProvider = new PromptfooChatCompletionProvider({ task: 'crescendo', ... });
} else {
this.redTeamProvider = await redteamProviderManager.getProvider({
provider: this.config.redteamProvider, // honored only when remote is disabled
...
});
}
}
getScoringProvider in the same file has the identical shape, and src/redteam/providers/voiceCrescendo/index.ts repeats it in getRedTeamProvider. The consequences are the ones #10970 enumerated: the attacker (and for custom, the judge) runs on a different model than configured, prompts and outputs leave the machine to api.promptfoo.app, and nothing is logged about the swap.
Expected behavior
Same rule #10972 established: remote generation is only for the default, unconfigured case. With an explicit redteamProvider, both strategies should keep the attacker (and custom's scorer) local, i.e. gate the remote branch on shouldGenerateRemote() && !this.config.redteamProvider.
Scope notes
I have the patch and regression tests ready, mirroring #10972's guard and test pattern; happy to open the PR.
System information
Reproduced by inspection on current main (f385d96), cross-checked against the #10972 diff; the code paths above are identical in shape to the pre-fix iterative/crescendo ones and diverge only in task id. promptfoo 0.123.0, macOS arm64, Node 22.
Describe the bug
The
customandvoice-crescendored-team strategies have the same provider substitution defect as #10970 (fixed foriterative/crescendoin #10972): when remote generation is enabled, an explicitly configuredredteamProvideris silently replaced with Promptfoo's cloud API.In
src/redteam/providers/custom/index.ts, both provider accessors take the remote path whenevershouldGenerateRemote()is true, even though the local branches were written to honor the config:getScoringProviderin the same file has the identical shape, andsrc/redteam/providers/voiceCrescendo/index.tsrepeats it ingetRedTeamProvider. The consequences are the ones #10970 enumerated: the attacker (and forcustom, the judge) runs on a different model than configured, prompts and outputs leave the machine toapi.promptfoo.app, and nothing is logged about the swap.Expected behavior
Same rule #10972 established: remote generation is only for the default, unconfigured case. With an explicit
redteamProvider, both strategies should keep the attacker (andcustom's scorer) local, i.e. gate the remote branch onshouldGenerateRemote() && !this.config.redteamProvider.Scope notes
hydrais not affected: it requires remote generation by design and throws a clear error otherwise (the GOAT class in Iterative and Crescendo strategies ignore an explicitly configured redteamProvider and use remote generation instead #10970's terms).iterativeTreereceives its attacker provider by injection; itsshouldGenerateRemote()call sites only gate input materialization, not provider selection.voiceCrescendo'sgetScoringProviderdeliberately omits the config on the local path too ("let getGradingProvider check CLI --grader first"), so it is out of scope here.I have the patch and regression tests ready, mirroring #10972's guard and test pattern; happy to open the PR.
System information
Reproduced by inspection on current
main(f385d96), cross-checked against the #10972 diff; the code paths above are identical in shape to the pre-fixiterative/crescendoones and diverge only in task id. promptfoo 0.123.0, macOS arm64, Node 22.