Skip to content

custom and voice-crescendo strategies ignore an explicitly configured redteamProvider and use remote generation instead #10977

Description

@he-yufeng

Describe the bug

The custom and voice-crescendo red-team strategies have the same provider substitution defect as #10970 (fixed for iterative/crescendo in #10972): when remote generation is enabled, an explicitly configured redteamProvider is silently replaced with Promptfoo's cloud API.

In src/redteam/providers/custom/index.ts, both provider accessors take the remote path whenever shouldGenerateRemote() is true, even though the local branches were written to honor the config:

private async getRedTeamProvider(): Promise<ApiProvider> {
    if (!this.redTeamProvider) {
      if (shouldGenerateRemote()) {
        this.redTeamProvider = new PromptfooChatCompletionProvider({ task: 'crescendo', ... });
      } else {
        this.redTeamProvider = await redteamProviderManager.getProvider({
          provider: this.config.redteamProvider,   // honored only when remote is disabled
          ...
        });
      }
    }

getScoringProvider in the same file has the identical shape, and src/redteam/providers/voiceCrescendo/index.ts repeats it in getRedTeamProvider. The consequences are the ones #10970 enumerated: the attacker (and for custom, the judge) runs on a different model than configured, prompts and outputs leave the machine to api.promptfoo.app, and nothing is logged about the swap.

Expected behavior

Same rule #10972 established: remote generation is only for the default, unconfigured case. With an explicit redteamProvider, both strategies should keep the attacker (and custom's scorer) local, i.e. gate the remote branch on shouldGenerateRemote() && !this.config.redteamProvider.

Scope notes

I have the patch and regression tests ready, mirroring #10972's guard and test pattern; happy to open the PR.

System information

Reproduced by inspection on current main (f385d96), cross-checked against the #10972 diff; the code paths above are identical in shape to the pre-fix iterative/crescendo ones and diverge only in task id. promptfoo 0.123.0, macOS arm64, Node 22.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions