evals@2.11.1
Arize AX onboarding — instrument your app with tracing via your coding agent
AI Security Review
scanned 5d ago · by lpm-firewall-aiThe package’s runtime onboarding flow sends event telemetry linked by a stable anonymous ID without an apparent opt-out or CLI disclosure, so this merits a warning. The inspected payload is limited to event names and that ID, and the install hook only prints a message.
Static reason
One or more suspicious static signals were detected.; previous stored version diff introduced dangerous source
Trigger
The user selects an agent in the CLI onboarding flow; the prompt also requests an auth-opened event before OAuth.
Impact
Arize can associate onboarding actions over time for the same anonymous client ID; inspected telemetry does not include credentials or project contents.
Mechanism
The analytics helper posts an allowlisted event name and anonymous client ID to the configured endpoint. No opt-out was found, and the CLI does not disclose this tracking in its interface.
AI rationale
The package’s runtime onboarding flow sends event telemetry linked by a stable anonymous ID without an apparent opt-out or CLI disclosure, so this merits a warning. The inspected payload is limited to event names and that ID, and the install hook only prints a message.
Evidence
analytics.jscli.jsonboarding-prompt.md~/.arize/anon_client_id
Network endpoints1
app.arize.com/api/analytics/npx-evals
Decision evidence
public snapshotAI called this Suspicious at 87.0% confidence as Dangerous Capability with medium false-positive risk.
Evidence for warning
- Runtime analytics posts allowlisted onboarding event names to Arize’s analytics endpoint.
- The analytics payload contains the event name and a stable anonymous client ID.
- The CLI sends an analytics event after the user selects an agent and starts the onboarding flow.
Evidence against
- The postinstall script only prints a message; it does not contact the network.
- The analytics code limits events and does not include credentials or project contents in its payload.
- The onboarding prompt describes an anonymous funnel ping before OAuth, and the remaining setup is tied to the onboarding flow.
Behavioral surface
ChildProcessCryptoEnvironmentVarsFilesystemNetworkShell
HighEntropyStringsUrlStrings
Source & flagged code
8 flaggedpackage.jsonView file
•scripts.postinstall = node -e "console.log('Arize evals installed — run: npx evals')"
High
Install Time Lifecycle Scripts
Package defines install-time lifecycle scripts.
package.jsonView on unpkg•scripts.postinstall = node -e "console.log('Arize evals installed — run: npx evals')"
Medium
Ambiguous Install Lifecycle Script
Install-time lifecycle script is not statically allowlisted and needs review.
package.jsonView on unpkgbin.jsView file
15//
L16: // MIN_MAJOR tracks ink's own `engines.node`, and package.json's `engines` field
L17: // must say the same thing; test/cli.test.js asserts they agree.
...
L23: if (!Number.isInteger(major) || major < MIN_MAJOR) {
L24: const shell = process.platform === 'win32'
L25: ? 'irm https://cdn.jsdelivr.net/npm/evals/start.ps1 | iex'
L26: : 'bash <(curl -fsSL https://cdn.jsdelivr.net/npm/evals/start.sh)';
High
Sandbox Evasion Gated Capability
Findings
4 High6 Medium5 Low
HighInstall Time Lifecycle Scriptspackage.json
HighSandbox Evasion Gated Capabilitybin.js
HighShips High Entropy Blobvendor/coding_harness_tracing-0.1.dev1+ge43113f-py3-none-any.whl
HighPrevious Version Dangerous Deltavendor/harness-install.sh
MediumAmbiguous Install Lifecycle Scriptpackage.json
MediumNetwork
MediumEnvironment Vars
MediumShips Build Helperstart.ps1
MediumShips Compressed Blobvendor/coding_harness_tracing-0.1.dev1+ge43113f-py3-none-any.whl
MediumStructural Risk Force Deep Review
LowScripts Present
LowFilesystem
LowHigh Entropy Strings
Affected versions and remediation
This report applies to evals@2.11.1.
See all versions of evals for other recorded verdicts.
- Review the evidence and your use of evals@2.11.1 before allowing it. Restrict the permissions described in this report.
- Choose an independently verified alternative or release. This report does not establish that other versions are safe.
Evidence last updated: .
Think this verdict is wrong? Dispute it on GitHub or read how reports are produced.