evals@2.8.0
Arize AX onboarding — instrument your app with tracing via your coding agent
AI Security Review
scanned 3mo ago · by lpm-firewall-aiThis is an explicit-user-command, disclosed agent-extension and tracing capability, not concrete covert malware. Its automatic global skill install and remote installer fallback remain meaningful security risk.
Static reason
One or more suspicious static signals were detected.; previous stored version diff introduced dangerous source
Trigger
User runs the CLI and selects a coding agent.
Impact
Persistent third-party agent capability and optional machine-wide tracing hooks.
Mechanism
global agent-skill installation plus prompted tracing setup
AI rationale
This is an explicit-user-command, disclosed agent-extension and tracing capability, not concrete covert malware. Its automatic global skill install and remote installer fallback remain meaningful security risk.
Evidence
package.jsoncli.jsonboarding-prompt.mdstart.shstart.ps1~/.arize/onboarding/onboarding-prompt.md~/.arize/harness<agent>'s settings file
Network endpoints1
raw.githubusercontent.com/Arize-ai/coding-harness-tracing/main/install.sh
Decision evidence
public snapshotAI called this Suspicious at 90.0% confidence as Benign with low false-positive risk.
Evidence for warning
- User-run CLI globally installs all Arize skills into the selected coding agent.
- The onboarding prompt can install machine-wide agent tracing hooks.
- The prompt includes a fallback that pipes a remote GitHub installer to bash.
Evidence against
- postinstall only prints a message.
- Agent tracing requires a separate explicit yes and discloses prompt/tool-output capture.
- No credential exfiltration or covert install-time mutation is confirmed.
Behavioral surface
ChildProcessEnvironmentVarsFilesystemShell
HighEntropyStringsUrlStrings
Source & flagged code
8 flaggedpackage.jsonView file
•scripts.postinstall = node -e "console.log('Arize evals installed — run: npx evals')"
High
Install Time Lifecycle Scripts
Package defines install-time lifecycle scripts.
package.jsonView on unpkg•scripts.postinstall = node -e "console.log('Arize evals installed — run: npx evals')"
Medium
Ambiguous Install Lifecycle Script
Install-time lifecycle script is not statically allowlisted and needs review.
package.jsonView on unpkgbin.jsView file
15//
L16: // MIN_MAJOR tracks ink's own `engines.node`, and package.json's `engines` field
L17: // must say the same thing; test/cli.test.js asserts they agree.
...
L23: if (!Number.isInteger(major) || major < MIN_MAJOR) {
L24: const shell = process.platform === 'win32'
L25: ? 'irm https://cdn.jsdelivr.net/npm/evals/start.ps1 | iex'
L26: : 'bash <(curl -fsSL https://cdn.jsdelivr.net/npm/evals/start.sh)';
High
Sandbox Evasion Gated Capability
Findings
4 High5 Medium5 Low
HighInstall Time Lifecycle Scriptspackage.json
HighSandbox Evasion Gated Capabilitybin.js
HighShips High Entropy Blobvendor/python_dotenv-1.2.1-py3-none-any.whl
HighPrevious Version Dangerous Deltacli.js
MediumAmbiguous Install Lifecycle Scriptpackage.json
MediumEnvironment Vars
MediumShips Build Helperstart.ps1
MediumShips Compressed Blobvendor/python_dotenv-1.2.1-py3-none-any.whl
MediumStructural Risk Force Deep Review
LowScripts Present
LowFilesystem
LowHigh Entropy Strings
LowUrl Strings
LowNested Archive Needs Inspection
Affected versions and remediation
This report applies to evals@2.8.0.
See all versions of evals for other recorded verdicts.
- Review the evidence and your use of evals@2.8.0 before allowing it. Restrict the permissions described in this report.
- Choose an independently verified alternative or release. This report does not establish that other versions are safe.
Evidence last updated: .
Think this verdict is wrong? Dispute it on GitHub or read how reports are produced.