registry / @vercel/agent-eval / 1.3.1

@vercel/agent-eval@1.3.1

⚠ Under review

Framework for testing AI coding agents in isolated sandboxes

Static Scan Results

scanned 3h ago · by rust-scanner

Static analysis flagged 13 finding(s) at 93.0% confidence. This version is warn-only unless an AI or security-team review confirms malicious behavior.

Static reason

One or more suspicious static signals were detected.; previous stored version diff introduced dangerous source

Decision evidence

public snapshot

Behavioral surface

Source

ChildProcessCryptoDynamicRequireEnvironmentVarsEvalFilesystemShell

Supply chain

HighEntropyStringsUrlStrings

ManifestNo manifest risk signals triggered.

scanned 40 file(s), 324 KB of source, external domains: ai-gateway.vercel.sh, api.openai.com, cursor.com, github.com, opencode.ai, vercel.com

Source & flagged code

6 flagged

dist/lib/agents/claude-code/run.mjsView file

Lines 1-41javascript

1/**

2 * Claude Code in-sandbox runner.

3 *

4 * This file is shipped INTO the sandbox by the orchestrator and executed there as

5 * `node __agent_eval__/run.mjs '<AgentRunInput JSON>'`. It is intentionally

6 * ZERO-DEPENDENCY (only `node:*` builtins) because the sandbox only has the

7 * fixture's own deps + the installed `claude` CLI — it cannot import anything from

8 * the @vercel/agent-eval package.

9 *

10 * Dual mode:

11 * - runnable: invoked directly → reads argv, runs the agent, writes the result

12 * file + prints a status line, exits 0.

13 * - importable: `import { runAgent } from './run.mjs'` → returns a RunnerResult

14 * (no file write, no exit). This is what a future in-sandbox judge reuses.

15 *

16 * `buildClaudeCodeCliArgs` and `extractObservedModelFromClaudeTranscript` are

17 * exported so the host unit tests can verify them directly (the test imports them

18 * from this very file, so there is no risk of the tested logic drifting from the

19 * logic the sandbox actually runs).

20 */

22import { spawnSync } from 'node:child_process';

23import { mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';

High

Child Process

Package source references child process execution.

dist/lib/agents/claude-code/run.mjsView on unpkg · L21

24import { homedir } from 'node:os';

25import { dirname, join } from 'node:path';

26import { fileURLToPath } from 'node:url';

28/**

29 * Build the Claude Code CLI argument list.

30 *

31 * Order is regression-sensitive (preserved verbatim from the old adapter):

32 * `--allowedTools` is variadic — it keeps consuming positional tokens until the

33 * next flag. So (1) the tools must be a single comma-separated value, and (2)

34 * `--allowedTools` must be followed by another flag, never directly by the prompt.

35 * `--dangerously-skip-permissions` is always emitted and terminates the variadic

36 * capture before the trailing positional prompt (the #141 regression).

37 *

38 * @param {{prompt:string, model?:string, webResearch?:boolean, agentOptions?:Record<string,unknown>}} input

39 * @returns {string[]}

40 */

41export function buildClaudeCodeCliArgs(input) {

dist/lib/agents/codex/run.mjsView file

Lines 176-216javascript

176 */

177export function buildCodexLoginArgs() {

178 return ['login', '--with-api-key'];

179}

180

181/**

182 * Build the `codex exec` arg list, reproducing the old shell command verbatim:

183 * codex exec --profile default[ --model <m>] --dangerously-bypass-approvals-and-sandbox

184 * --json --skip-git-repo-check[ -c model_reasoning_effort="<e>"]

dist/cli.jsView file

Lines 91-131javascript

91 if (evalNames.length === 0) {

92 console.error(chalk.red('No evals matched the filter'));

93 process.exit(1);

94 }

95 // Smoke mode: pick first eval alphabetically, override runs to 1

96 const smokeEvalNames

dist/lib/config.jsView file

Lines 110-150javascript

110 timeout: config.timeout ?? CONFIG_DEFAULTS.timeout,

111 setup: config.setup,

112 sandbox: config.sandbox ?? CONFIG_DEFAULTS.sandbox,

113 editPrompt: config.editPrompt,

114 copyFiles: config.copyFiles ?? CONFIG_DEFAULTS.copyFiles,

115

Source & flagged code

6 flagged · loading source

dist/lib/agents/claude-code/run.mjsView file

21L22: import { spawnSync } from 'node:child_process'; L23: import { mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';

High

Child Process

Package source references child process execution.

dist/lib/agents/claude-code/run.mjsView on unpkg · L21

dist/lib/agents/codex/run.mjsView file

196*/ L197: export function buildCodexExecArgs(input) { L198: const extra = input.extra ?? {};

High

Shell

Package source references shell execution.

dist/lib/agents/codex/run.mjsView on unpkg · L196

dist/cli.jsView file

•matchType = previous_version_dangerous_delta matchedPackage = @vercel/agent-eval@1.2.0 matchedIdentity = npm:QHZlcmNlbC9hZ2VudC1ldmFs:1.2.0 similarity = 0.622 summary = stored previous version shares package body but lacks this dangerous source file

Critical

Previous Version Dangerous Delta

This package version adds a dangerous source file absent from the previous stored version; route for source-aware review.

dist/cli.jsView on unpkg

235// Try to run the playground package directly, fall back to npx L236: const result = spawnSync('npx', ['@vercel/agent-eval-playground', ...playgroundArgs], { stdio: 'inherit', cwd: process.cwd() }); L237: process.exit(result.status ?? 1);

High

Runtime Package Install

Package source invokes a package manager install command at runtime.

dist/cli.jsView on unpkg · L235

111if (models.length > 1) { L112: console.log(chalk.blue(`\nRunning ${smokeEvalNames.length} eval(s) x ${smokeRuns} run(s) x ${models.length} model(s) = ${totalRuns} total runs`)); L113: console.log(chalk.blue(`Agent: ${config.agent}, Models: ${models.join(', ')}, Timeout: ${config.timeout}s, Early Exit: ${config.earlyExit}`));

Low

Eval

Package source references a known benign dynamic code generation pattern.

dist/cli.jsView on unpkg · L111

dist/lib/config.jsView file

130if (configPath.endsWith('.ts')) { L131: const { createJiti } = await import('jiti'); L132: const jiti = createJiti(import.meta.url, {

Medium

Dynamic Require

Package source references dynamic require/import behavior.

dist/lib/config.jsView on unpkg · L130

Findings

1 Critical3 High3 Medium6 Low

CriticalPrevious Version Dangerous Deltadist/cli.js

HighChild Processdist/lib/agents/claude-code/run.mjs

HighShelldist/lib/agents/codex/run.mjs

HighRuntime Package Installdist/cli.js

MediumDynamic Requiredist/lib/config.js

MediumEnvironment Vars

MediumStructural Risk Force Deep Review

LowNon Install Lifecycle Scripts

LowScripts Present

LowEvaldist/cli.js

LowFilesystem

LowHigh Entropy Strings

LowUrl Strings

Source & flagged code

6 flagged

dist/lib/agents/claude-code/run.mjsView file

Lines 1-41javascript

1/**

2 * Claude Code in-sandbox runner.

3 *

4 * This file is shipped INTO the sandbox by the orchestrator and executed there as

5 * `node __agent_eval__/run.mjs '<AgentRunInput JSON>'`. It is intentionally

6 * ZERO-DEPENDENCY (only `node:*` builtins) because the sandbox only has the

7 * fixture's own deps + the installed `claude` CLI — it cannot import anything from

8 * the @vercel/agent-eval package.

9 *

10 * Dual mode:

11 * - runnable: invoked directly → reads argv, runs the agent, writes the result

12 * file + prints a status line, exits 0.

13 * - importable: `import { runAgent } from './run.mjs'` → returns a RunnerResult

14 * (no file write, no exit). This is what a future in-sandbox judge reuses.

15 *

16 * `buildClaudeCodeCliArgs` and `extractObservedModelFromClaudeTranscript` are

17 * exported so the host unit tests can verify them directly (the test imports them

18 * from this very file, so there is no risk of the tested logic drifting from the

19 * logic the sandbox actually runs).

20 */

22import { spawnSync } from 'node:child_process';

23import { mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';

High

Child Process

Package source references child process execution.

dist/lib/agents/claude-code/run.mjsView on unpkg · L21

24import { homedir } from 'node:os';

25import { dirname, join } from 'node:path';

26import { fileURLToPath } from 'node:url';

28/**

29 * Build the Claude Code CLI argument list.

30 *

31 * Order is regression-sensitive (preserved verbatim from the old adapter):

32 * `--allowedTools` is variadic — it keeps consuming positional tokens until the

33 * next flag. So (1) the tools must be a single comma-separated value, and (2)

34 * `--allowedTools` must be followed by another flag, never directly by the prompt.

35 * `--dangerously-skip-permissions` is always emitted and terminates the variadic

36 * capture before the trailing positional prompt (the #141 regression).

37 *

38 * @param {{prompt:string, model?:string, webResearch?:boolean, agentOptions?:Record<string,unknown>}} input

39 * @returns {string[]}

40 */

41export function buildClaudeCodeCliArgs(input) {

dist/lib/agents/codex/run.mjsView file

Lines 176-216javascript

176 */

177export function buildCodexLoginArgs() {

178 return ['login', '--with-api-key'];

179}

180

181/**

182 * Build the `codex exec` arg list, reproducing the old shell command verbatim:

183 * codex exec --profile default[ --model <m>] --dangerously-bypass-approvals-and-sandbox

184 * --json --skip-git-repo-check[ -c model_reasoning_effort="<e>"]

dist/cli.jsView file

Lines 91-131javascript

91 if (evalNames.length === 0) {

92 console.error(chalk.red('No evals matched the filter'));

93 process.exit(1);

94 }

95 // Smoke mode: pick first eval alphabetically, override runs to 1

96 const smokeEvalNames

dist/lib/config.jsView file

Lines 110-150javascript

110 timeout: config.timeout ?? CONFIG_DEFAULTS.timeout,

111 setup: config.setup,

112 sandbox: config.sandbox ?? CONFIG_DEFAULTS.sandbox,

113 editPrompt: config.editPrompt,

114 copyFiles: config.copyFiles ?? CONFIG_DEFAULTS.copyFiles,

115

Lines 215-255javascript

215program

216 .command('playground')

217 .description('Launch the web-based playground for browsing experiment results')

218 .option('--port <port>', 'HTTP server port', '3000')

219 .option('--results-dir <dir>', 'Path to results directory', './results')

220 .option('--evals-dir <dir>', 'Path to evals directory', './evals')

221 .option('--watch', 'Enable live mode — watch results directory for changes')

222 .action(async (options) => {

223 const resultsDir = resolve(process.cwd(), options.resultsDir);

224 const evalsDir = resolve(process.cwd(), options.evalsDir);

225 console.log(chalk.blue('Starting Agent Eval Playground...'));

226 // Build args for the playground CLI

227 const playgroundArgs = [

228 '--results-dir', resultsDir,

229 '--evals-dir', evalsDir,

230 '--port', options.port,

231 ];

232 if (options.watch) {

233 playgroundArgs.push('--watch');

234 }

235 // Try to run the playground package directly, fall back to npx

236 const result = spawnSync('npx', ['@vercel/agent-eval-playground', ...playgroundArgs], { stdio: 'inherit', cwd: process.cwd() });

237 process.exit(result.status ?? 1);

High

Runtime Package Install

Package source invokes a package manager install command at runtime.

dist/cli.jsView on unpkg · L235

238});

239/**

240 * Run-all handler: discover and run all experiments with fingerprint reuse

241 * and classification. Used by both `run-all` subcommand and the default

242 * (no-args) invocation.

243 */

244async function runAllCommand(experimentArgs, options) {

245 try {

246 const projectDir = process.cwd();

247 const experimentsDir = resolve(projectDir, 'experiments');

248 const evalsDir = resolve(projectDir, 'evals');

249 const resultsDir = resolve(projectDir, 'results');

250 if (!existsSync(experimentsDir)) {

251 console.error(chalk.red('experiments/ directory not found'));

252 process.exit(1);

253 }

254 if (!existsSync(evalsDir)) {

255 console.error(chalk.red('evals/ directory not found'));