registry / @vercel/agent-eval / 1.2.0

@vercel/agent-eval@1.2.0

Framework for testing AI coding agents in isolated sandboxes

AI Security Review

scanned 2d ago · by lpm-firewall-ai

No confirmed malicious attack surface. The package is an AI agent evaluation framework that runs selected agents in Docker/Vercel sandboxes and writes eval results/project scaffolds during explicit CLI workflows.

Static reason

One or more suspicious static signals were detected.

Trigger

User invokes agent-eval commands or imports the library API.

Impact

Runs configured coding agents against eval fixtures and records outputs; no install-time mutation or stealth persistence found.

Mechanism

User-invoked sandbox orchestration and result/project file generation

Rationale

Static inspection shows suspicious primitives are package-aligned and activated by explicit CLI/eval workflows, with sandbox-scoped agent setup rather than install-time foreign agent control-surface hijacking. No evidence of credential exfiltration, persistence, destructive behavior outside result housekeeping, or remote payload execution at install/import time was found.

Evidence

package.jsondist/cli.jsdist/lib/init.jsdist/lib/config.jsdist/lib/sandbox.jsdist/lib/agents/shared.jsdist/lib/agents/plugin/orchestrator.jsdist/lib/agents/codex/agent.jsdist/lib/agents/codex/run.mjsdist/lib/agents/claude-code/run.mjsdist/lib/agents/cursor/agent.jsdist/lib/agents/opencode/agent.js

Network endpoints7

ai-gateway.vercel.shai-gateway.vercel.sh/v1api.openai.com/v1vercel.com/dashboardvercel.com/account/tokensgithub.com/vercel-labs/agent-evalcursor.com/install

Decision evidence

public snapshot

AI called this Clean at 88.0% confidence as Benign with low false-positive risk.

Evidence for block

Evidence against

package.json has no install/postinstall/prepare hook; only prepublishOnly build for publisher use.
dist/cli.js actions are user-invoked CLI commands; npx playground launch is explicit.
dist/lib/agents/*/run.mjs spawn agent CLIs inside orchestrated sandbox runners, not import/install time.
dist/lib/agents/codex/agent.js writes ~/.codex/default.config.toml via sandbox setup for the selected Codex agent.
dist/lib/init.js only scaffolds files under a user-requested project directory.
No credential harvesting or exfiltration endpoints found; env vars are checked/passed to configured agents.

Behavioral surface

Source

ChildProcessCryptoDynamicRequireEnvironmentVarsEvalFilesystemShell

Supply chain

HighEntropyStringsUrlStrings

ManifestNo manifest risk signals triggered.

scanned 40 file(s), 302 KB of source, external domains: ai-gateway.vercel.sh, api.openai.com, cursor.com, github.com, opencode.ai, vercel.com

Source & flagged code

5 flagged

dist/lib/agents/claude-code/run.mjsView file

Lines 1-41javascript

1/**

2 * Claude Code in-sandbox runner.

3 *

4 * This file is shipped INTO the sandbox by the orchestrator and executed there as

5 * `node __agent_eval__/run.mjs '<AgentRunInput JSON>'`. It is intentionally

6 * ZERO-DEPENDENCY (only `node:*` builtins) because the sandbox only has the

7 * fixture's own deps + the installed `claude` CLI — it cannot import anything from

8 * the @vercel/agent-eval package.

9 *

10 * Dual mode:

11 * - runnable: invoked directly → reads argv, runs the agent, writes the result

12 * file + prints a status line, exits 0.

13 * - importable: `import { runAgent } from './run.mjs'` → returns a RunnerResult

14 * (no file write, no exit). This is what a future in-sandbox judge reuses.

15 *

16 * `buildClaudeCodeCliArgs` and `extractObservedModelFromClaudeTranscript` are

17 * exported so the host unit tests can verify them directly (the test imports them

18 * from this very file, so there is no risk of the tested logic drifting from the

19 * logic the sandbox actually runs).

20 */

22import { spawnSync } from 'node:child_process';

23import { mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';

High

Child Process

Package source references child process execution.

dist/lib/agents/claude-code/run.mjsView on unpkg · L21

24import { homedir } from 'node:os';

25import { dirname, join } from 'node:path';

26import { fileURLToPath } from 'node:url';

28/**

29 * Build the Claude Code CLI argument list.

30 *

31 * Order is regression-sensitive (preserved verbatim from the old adapter):

32 * `--allowedTools` is variadic — it keeps consuming positional tokens until the

33 * next flag. So (1) the tools must be a single comma-separated value, and (2)

34 * `--allowedTools` must be followed by another flag, never directly by the prompt.

35 * `--dangerously-skip-permissions` is always emitted and terminates the variadic

36 * capture before the trailing positional prompt (the #141 regression).

37 *

38 * @param {{prompt:string, model?:string, webResearch?:boolean, agentOptions?:Record<string,unknown>}} input

39 * @returns {string[]}

40 */

41export function buildClaudeCodeCliArgs(input) {

dist/lib/agents/codex/run.mjsView file

Lines 176-216javascript

176 */

177export function buildCodexLoginArgs() {

178 return ['login', '--with-api-key'];

179}

180

181/**

182 * Build the `codex exec` arg list, reproducing the old shell command verbatim:

183 * codex exec --profile default[ --model <m>] --dangerously-bypass-approvals-and-sandbox

184 * --json --skip-git-repo-check[ -c model_reasoning_effort="<e>"]

dist/cli.jsView file

Lines 89-129javascript

89 if (evalNames.length === 0) {

90 console.error(chalk.red('No evals matched the filter'));

91 process.exit(1);

92 }

93 // Smoke mode: pick first eval alphabetically, override runs to 1

94 const smokeEvalNames

dist/lib/config.jsView file

Lines 91-131javascript

91 validation: config.validation ?? CONFIG_DEFAULTS.validation,

92 timeout: config.timeout ?? CONFIG_DEFAULTS.timeout,

93 setup: config.setup,

94 sandbox: config.sandbox ?? CONFIG_DEFAULTS.sandbox,

95 editPrompt: config.editPrompt,

Source & flagged code

5 flagged · loading source

dist/lib/agents/claude-code/run.mjsView file

21L22: import { spawnSync } from 'node:child_process'; L23: import { mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';

High

Child Process

Package source references child process execution.

dist/lib/agents/claude-code/run.mjsView on unpkg · L21

dist/lib/agents/codex/run.mjsView file

196*/ L197: export function buildCodexExecArgs(input) { L198: const extra = input.extra ?? {};

High

Shell

Package source references shell execution.

dist/lib/agents/codex/run.mjsView on unpkg · L196

dist/cli.jsView file

233// Try to run the playground package directly, fall back to npx L234: const result = spawnSync('npx', ['@vercel/agent-eval-playground', ...playgroundArgs], { stdio: 'inherit', cwd: process.cwd() }); L235: process.exit(result.status ?? 1);

High

Runtime Package Install

Package source invokes a package manager install command at runtime.

dist/cli.jsView on unpkg · L233

109if (models.length > 1) { L110: console.log(chalk.blue(`\nRunning ${smokeEvalNames.length} eval(s) x ${smokeRuns} run(s) x ${models.length} model(s) = ${totalRuns} total runs`)); L111: console.log(chalk.blue(`Agent: ${config.agent}, Models: ${models.join(', ')}, Timeout: ${config.timeout}s, Early Exit: ${config.earlyExit}`));

Low

Eval

Package source references a known benign dynamic code generation pattern.

dist/cli.jsView on unpkg · L109

dist/lib/config.jsView file

111if (configPath.endsWith('.ts')) { L112: const { createJiti } = await import('jiti'); L113: const jiti = createJiti(import.meta.url, {

Medium

Dynamic Require

Package source references dynamic require/import behavior.

dist/lib/config.jsView on unpkg · L111

Findings

3 High3 Medium6 Low

HighChild Processdist/lib/agents/claude-code/run.mjs

HighShelldist/lib/agents/codex/run.mjs

HighRuntime Package Installdist/cli.js

MediumDynamic Requiredist/lib/config.js

MediumEnvironment Vars

MediumStructural Risk Force Deep Review

LowNon Install Lifecycle Scripts

LowScripts Present

LowEvaldist/cli.js

LowFilesystem

LowHigh Entropy Strings

LowUrl Strings

Source & flagged code

5 flagged

dist/lib/agents/claude-code/run.mjsView file

Lines 1-41javascript

1/**

2 * Claude Code in-sandbox runner.

3 *

4 * This file is shipped INTO the sandbox by the orchestrator and executed there as

5 * `node __agent_eval__/run.mjs '<AgentRunInput JSON>'`. It is intentionally

6 * ZERO-DEPENDENCY (only `node:*` builtins) because the sandbox only has the

7 * fixture's own deps + the installed `claude` CLI — it cannot import anything from

8 * the @vercel/agent-eval package.

9 *

10 * Dual mode:

11 * - runnable: invoked directly → reads argv, runs the agent, writes the result

12 * file + prints a status line, exits 0.

13 * - importable: `import { runAgent } from './run.mjs'` → returns a RunnerResult

14 * (no file write, no exit). This is what a future in-sandbox judge reuses.

15 *

16 * `buildClaudeCodeCliArgs` and `extractObservedModelFromClaudeTranscript` are

17 * exported so the host unit tests can verify them directly (the test imports them

18 * from this very file, so there is no risk of the tested logic drifting from the

19 * logic the sandbox actually runs).

20 */

22import { spawnSync } from 'node:child_process';

23import { mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';

High

Child Process

Package source references child process execution.

dist/lib/agents/claude-code/run.mjsView on unpkg · L21

24import { homedir } from 'node:os';

25import { dirname, join } from 'node:path';

26import { fileURLToPath } from 'node:url';

28/**

29 * Build the Claude Code CLI argument list.

30 *

31 * Order is regression-sensitive (preserved verbatim from the old adapter):

32 * `--allowedTools` is variadic — it keeps consuming positional tokens until the

33 * next flag. So (1) the tools must be a single comma-separated value, and (2)

34 * `--allowedTools` must be followed by another flag, never directly by the prompt.

35 * `--dangerously-skip-permissions` is always emitted and terminates the variadic

36 * capture before the trailing positional prompt (the #141 regression).

37 *

38 * @param {{prompt:string, model?:string, webResearch?:boolean, agentOptions?:Record<string,unknown>}} input

39 * @returns {string[]}

40 */

41export function buildClaudeCodeCliArgs(input) {

dist/lib/agents/codex/run.mjsView file

Lines 176-216javascript

176 */

177export function buildCodexLoginArgs() {

178 return ['login', '--with-api-key'];

179}

180

181/**

182 * Build the `codex exec` arg list, reproducing the old shell command verbatim:

183 * codex exec --profile default[ --model <m>] --dangerously-bypass-approvals-and-sandbox

184 * --json --skip-git-repo-check[ -c model_reasoning_effort="<e>"]

dist/cli.jsView file

Lines 89-129javascript

89 if (evalNames.length === 0) {

90 console.error(chalk.red('No evals matched the filter'));

91 process.exit(1);

92 }

93 // Smoke mode: pick first eval alphabetically, override runs to 1

94 const smokeEvalNames

dist/lib/config.jsView file

Lines 91-131javascript

91 validation: config.validation ?? CONFIG_DEFAULTS.validation,

92 timeout: config.timeout ?? CONFIG_DEFAULTS.timeout,

93 setup: config.setup,

94 sandbox: config.sandbox ?? CONFIG_DEFAULTS.sandbox,

95 editPrompt: config.editPrompt,

Lines 213-253javascript

213program

214 .command('playground')

215 .description('Launch the web-based playground for browsing experiment results')

216 .option('--port <port>', 'HTTP server port', '3000')

217 .option('--results-dir <dir>', 'Path to results directory', './results')

218 .option('--evals-dir <dir>', 'Path to evals directory', './evals')

219 .option('--watch', 'Enable live mode — watch results directory for changes')

220 .action(async (options) => {

221 const resultsDir = resolve(process.cwd(), options.resultsDir);

222 const evalsDir = resolve(process.cwd(), options.evalsDir);

223 console.log(chalk.blue('Starting Agent Eval Playground...'));

224 // Build args for the playground CLI

225 const playgroundArgs = [

226 '--results-dir', resultsDir,

227 '--evals-dir', evalsDir,

228 '--port', options.port,

229 ];

230 if (options.watch) {

231 playgroundArgs.push('--watch');

232 }

233 // Try to run the playground package directly, fall back to npx

234 const result = spawnSync('npx', ['@vercel/agent-eval-playground', ...playgroundArgs], { stdio: 'inherit', cwd: process.cwd() });

235 process.exit(result.status ?? 1);

High

Runtime Package Install

Package source invokes a package manager install command at runtime.

dist/cli.jsView on unpkg · L233

236});

237/**

238 * Run-all handler: discover and run all experiments with fingerprint reuse

239 * and classification. Used by both `run-all` subcommand and the default

240 * (no-args) invocation.

241 */

242async function runAllCommand(experimentArgs, options) {

243 try {

244 const projectDir = process.cwd();

245 const experimentsDir = resolve(projectDir, 'experiments');

246 const evalsDir = resolve(projectDir, 'evals');

247 const resultsDir = resolve(projectDir, 'results');

248 if (!existsSync(experimentsDir)) {

249 console.error(chalk.red('experiments/ directory not found'));

250 process.exit(1);

251 }

252 if (!existsSync(evalsDir)) {

253 console.error(chalk.red('evals/ directory not found'));