feat: add GitHub workflows to sync agents/ to Devin Knowledge#26994
feat: add GitHub workflows to sync agents/ to Devin Knowledge#26994eunjae-lee merged 30 commits intomainfrom
Conversation
- Add parse-to-devin-knowledge.ts to convert agents/ markdown to Devin Knowledge JSON - Add validate-format.ts to validate rules have frontmatter and knowledge-base sections start with 'When...' - Add sync-to-devin.ts to sync knowledge entries to Devin API - Add export-devin-knowledge.sh to backup existing Devin knowledge - Add validate-agents-format.yml workflow to validate format on PRs - Add sync-agents-to-devin.yml workflow to sync on merge to main - Add devin-knowledge.json to .gitignore (generated file) Co-Authored-By: [email protected] <[email protected]>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
There was a problem hiding this comment.
2 issues found across 8 files
Prompt for AI agents (all issues)
Check if these issues are valid — if so, understand the root cause of each and fix them.
<file name="scripts/devin/sync-knowledge-to-devin.ts">
<violation number="1" location="scripts/devin/sync-knowledge-to-devin.ts:1">
P2: The shebang uses `env` with a multi-word command (`npx ts-node`), which fails when executing the script directly. Use a single executable (e.g., `ts-node`) or `env -S` to pass multiple args.</violation>
</file>
<file name="scripts/devin/validate-local-knowledge.ts">
<violation number="1" location="scripts/devin/validate-local-knowledge.ts:1">
P3: This shebang won’t run as an executable because `/usr/bin/env` can only take one argument; it will try to execute a command literally named `npx ts-node`. Use `env -S` (or a single command) so the script can be executed directly.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Devin AI is addressing Cubic AI's review feedbackNew feedback has been sent to the existing Devin session. ✅ No changes pushed |
There was a problem hiding this comment.
1 issue found across 3 files (changes from recent commits).
Prompt for AI agents (all issues)
Check if these issues are valid — if so, understand the root cause of each and fix them.
<file name="scripts/devin/sync-knowledge-to-devin.ts">
<violation number="1" location="scripts/devin/sync-knowledge-to-devin.ts:10">
P3: Update the usage comment to reference the correct script path so the documented command is runnable.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Devin AI is addressing Cubic AI's review feedbackNew feedback has been sent to the existing Devin session. ✅ Pushed commit |
Co-Authored-By: [email protected] <[email protected]>
There was a problem hiding this comment.
3 issues found across 8 files
Prompt for AI agents (all issues)
Check if these issues are valid — if so, understand the root cause of each and fix them.
<file name="scripts/devin/parse-local-knowledge.ts">
<violation number="1" location="scripts/devin/parse-local-knowledge.ts:1">
P2: The shebang won’t execute because `/usr/bin/env` can’t resolve `npx ts-node` as a single binary. Use `env -S` or call `ts-node` directly so the script runs when executed as a file.</violation>
</file>
<file name="scripts/devin/export-devin-knowledge.ts">
<violation number="1" location="scripts/devin/export-devin-knowledge.ts:1">
P2: The shebang uses multiple arguments without `-S`, so direct execution will fail because `/usr/bin/env` receives `"npx tsx"` as a single program name. Use `env -S` or a single executable in the shebang.</violation>
</file>
<file name="scripts/devin/validate-local-knowledge.ts">
<violation number="1" location="scripts/devin/validate-local-knowledge.ts:1">
P2: The shebang is invalid because `/usr/bin/env` cannot take multiple arguments without `-S`. Running the script directly will fail. Use `-S` or a single executable in the shebang.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Devin AI is addressing Cubic AI's review feedbackNew feedback has been sent to the existing Devin session. ✅ Pushed commit |
| name: Validate Agents Format | ||
|
|
||
| on: | ||
| pull_request: |
There was a problem hiding this comment.
@eunjae-lee for security reasons, this needs to be pull_request_target. Otherwise, a malicious user could update this workflow to override a command below and run malicious code. pull_request updates and runs automatically whereas pull_request_target waits for the GitHub workflow to make it into main
There was a problem hiding this comment.
Oh I didn't know about pull_request_target.
- validate-agents-format.yml: this file parses agents/ folder and check all the markdown files are in a good shape to be synced to Devin
- sync-agents-to-devin.yml: this actually create knowledges from agents/ folder and send it to Devin by using DEVIN_API_KEY.
So if I understand correctly,
- validate-agents-format.yml: pull_request is fine here
- sync-agents-to-devin.yml: needs pull_request_target
is it correct?
There was a problem hiding this comment.
Unfortunately, we can't really use pull_request for anything since malicious actors could make a PR with malicious code in this workflow file and it would automatically be run.
There was a problem hiding this comment.
Got it. Thanks for the confirmation !
…o modular rules" This reverts commit 8251b6b.
…nowledge - Slim down knowledge-base.md (356 → 96 lines) to domain knowledge only - Add Business rules section (managed events, orgs/teams, OAuth clients) - Delete coding-standards.md (content moved to rules) - Create 19 new rule files for coding guidelines: - quality-*: PR creation, error handling, imports, comments, code review - testing-*: playwright, mocking, timezone, incremental - ci-*: check failures, type-check-first, git workflow - data-prisma-*: migrations, feature flags - patterns-*: workflow triggers, app store - architecture-features-modules, reference-file-locations, reference-local-dev - Update agents/README.md as rules index (43 total rules) - Update _sections.md with CI/CD and Reference sections - Clean up parse-local-knowledge.ts Co-Authored-By: Claude Opus 4.5 <[email protected]>
Update section headers to pass validation rules: - "Business Rules" → "When working with managed events, organizations, or OAuth clients" - "Product & Codebase Knowledge" → "When you need product or codebase context" Co-Authored-By: Claude Opus 4.5 <[email protected]>
Split the combined "When working with managed events, organizations, or OAuth clients" section into three distinct ## sections for better Devin triggering: - When working with managed event types - When working with organizations and teams - When working with OAuth clients Co-Authored-By: Claude Opus 4.5 <[email protected]>
Since validate-local-knowledge.ts enforces that all section titles must start with "When...", we can remove the manual fallback logic and just use the title directly as the trigger description. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Simplify the validation to only check that section titles start with "When..." since we've standardized on that pattern. Remove the special cases for error, file naming, PR, and repo note sections. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add delete-all-devin-knowledge.ts script that: - Lists all knowledge entries before deletion - Requires interactive confirmation (Y) to proceed - Blocks execution in non-TTY environments (CI, piped input) - Shows progress while deleting entries Co-Authored-By: Claude Opus 4.5 <[email protected]>
Co-Authored-By: Claude Opus 4.5 <[email protected]>
There was a problem hiding this comment.
17 issues found across 33 files
Prompt for AI agents (all issues)
Check if these issues are valid — if so, understand the root cause of each and fix them.
<file name="agents/rules/quality-imports.md">
<violation number="1" location="agents/rules/quality-imports.md:10">
P2: This rule file doesn’t follow the required template (missing the Impact declaration and the explicit **Incorrect**/**Correct** sections), so automated tooling can’t parse it consistently.</violation>
</file>
<file name="scripts/devin/validate-local-knowledge.ts">
<violation number="1" location="scripts/devin/validate-local-knowledge.ts:1">
P2: The shebang invokes `/usr/bin/env` without `-S`, so executing this script directly fails because `env` receives the literal string `"npx ts-node"` as a single binary name. Use `-S` so the interpreter arguments are split correctly.</violation>
</file>
<file name="agents/rules/ci-check-failures.md">
<violation number="1" location="agents/rules/ci-check-failures.md:8">
P2: This new rule file doesn’t follow the required rule template (missing the standard "##" rule heading plus Impact/Incorrect/Correct/Reference sections). Please align it to the format described in agents/rules/README.md so rule parsing and consistency remain intact.</violation>
<violation number="2" location="agents/rules/ci-check-failures.md:29">
P2: This sentence advises doing nothing when E2E tests are skipped, but in this repo skipped required checks must be treated as failures and rerun (e.g., by adding the ready-for-e2e label). Update the guidance so engineers run E2E instead of ignoring it.
(Based on your team's feedback about skipped required CI jobs blocking merges.) [FEEDBACK_USED]</violation>
</file>
<file name="agents/rules/architecture-features-modules.md">
<violation number="1" location="agents/rules/architecture-features-modules.md:3">
P3: Architecture rules should use CRITICAL impact per agents/rules/README.md; update this frontmatter value to stay consistent with the documented rule format.</violation>
</file>
<file name="agents/rules/quality-code-review.md">
<violation number="1" location="agents/rules/quality-code-review.md:3">
P2: `quality-` rules are required to be marked as CRITICAL impact, but this rule’s frontmatter sets the impact to MEDIUM, which conflicts with the documented section standard.</violation>
</file>
<file name="scripts/devin/parse-local-knowledge.ts">
<violation number="1" location="scripts/devin/parse-local-knowledge.ts:1">
P2: Shebang uses multiple arguments without `-S`, so `/usr/bin/env` won’t execute `npx ts-node` correctly. Use `-S` to split arguments (or switch to a single-arg shebang) so the script can run in CI.</violation>
</file>
<file name="agents/rules/testing-timezone.md">
<violation number="1" location="agents/rules/testing-timezone.md:1">
P2: This new rule file doesn’t follow the required rule template (missing the "##" rule section, **Impact** line, and Incorrect/Correct examples). Please update it to match the standard rule format so the rules stay consistent and machine-parseable.</violation>
</file>
<file name="agents/rules/patterns-workflow-triggers.md">
<violation number="1" location="agents/rules/patterns-workflow-triggers.md:1">
P3: This new rule file doesn’t follow the required rule template (missing **Impact** and Incorrect/Correct example sections). Please align it with agents/rules/README.md so downstream tooling and readers get the expected structure.</violation>
<violation number="2" location="agents/rules/patterns-workflow-triggers.md:28">
P2: The translation path here points to `en/locale.json`, which doesn’t exist in this repo. Use the documented source locale file instead to avoid misdirecting contributors.</violation>
</file>
<file name="agents/rules/patterns-app-store.md">
<violation number="1" location="agents/rules/patterns-app-store.md:18">
P3: This guidance is contradictory: it says browser components should use static imports, but later notes browser component files may require dynamic imports. Please clarify the exception so readers know when dynamic imports are acceptable.</violation>
</file>
<file name="agents/rules/ci-git-workflow.md">
<violation number="1" location="agents/rules/ci-git-workflow.md:8">
P2: This rule file doesn’t follow the required template (missing the Impact callout and the Incorrect/Correct code example sections), so the rule can’t be parsed consistently with the rest of the ruleset.</violation>
</file>
<file name="agents/rules/data-prisma-feature-flags.md">
<violation number="1" location="agents/rules/data-prisma-feature-flags.md:8">
P2: This rule file doesn’t follow the required template—it’s missing the Impact statement, incorrect/correct examples, and a reference section, so automation can’t parse it consistently.</violation>
<violation number="2" location="agents/rules/data-prisma-feature-flags.md:34">
P1: Rule violated: **Prevent Direct NOW() Usage in Database Queries**
Replace NOW() in the SQL example with an approved timestamp expression to avoid direct NOW() usage without timezone specification.</violation>
</file>
<file name="scripts/devin/sync-knowledge-to-devin.ts">
<violation number="1" location="scripts/devin/sync-knowledge-to-devin.ts:124">
P3: The error message points to `parse-to-devin-knowledge.ts`, which doesn't exist. This will mislead users when the JSON file is missing; update it to reference the actual script name.</violation>
</file>
<file name="agents/rules/quality-pr-creation.md">
<violation number="1" location="agents/rules/quality-pr-creation.md:22">
P2: This wording conflicts with AGENTS.md’s hard limit on PR size. It should state that large PRs are not allowed, not merely discouraged.</violation>
<violation number="2" location="agents/rules/quality-pr-creation.md:29">
P2: This suggests skipping tests for most PRs, but AGENTS.md requires running relevant tests. Align this line with the repo standard.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
|
|
||
| # Import and Export Patterns | ||
|
|
||
| ## Named vs Default Exports |
There was a problem hiding this comment.
P2: This rule file doesn’t follow the required template (missing the Impact declaration and the explicit Incorrect/Correct sections), so automated tooling can’t parse it consistently.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agents/rules/quality-imports.md, line 10:
<comment>This rule file doesn’t follow the required template (missing the Impact declaration and the explicit **Incorrect**/**Correct** sections), so automated tooling can’t parse it consistently.</comment>
<file context>
@@ -0,0 +1,49 @@
+
+# Import and Export Patterns
+
+## Named vs Default Exports
+
+When working with imports in the Cal.com codebase, particularly in app-store integrations, pay attention to whether modules use named exports or default exports.
</file context>
| tags: ci, debugging, workflow | ||
| --- | ||
|
|
||
| # CI Check Failure Handling |
There was a problem hiding this comment.
P2: This new rule file doesn’t follow the required rule template (missing the standard "##" rule heading plus Impact/Incorrect/Correct/Reference sections). Please align it to the format described in agents/rules/README.md so rule parsing and consistency remain intact.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agents/rules/ci-check-failures.md, line 8:
<comment>This new rule file doesn’t follow the required rule template (missing the standard "##" rule heading plus Impact/Incorrect/Correct/Reference sections). Please align it to the format described in agents/rules/README.md so rule parsing and consistency remain intact.</comment>
<file context>
@@ -0,0 +1,33 @@
+tags: ci, debugging, workflow
+---
+
+# CI Check Failure Handling
+
+## What to Focus On
</file context>
| **E2E tests skipping is expected behavior:** | ||
| - When E2E tests are skipped, it's because the `ready-for-e2e` label has not been added to the PR | ||
| - The "required" check intentionally fails when E2E tests are skipped to prevent merging without E2E | ||
| - Do not try to fix anything related to skipped E2E tests - this is completely expected and normal |
There was a problem hiding this comment.
P2: This sentence advises doing nothing when E2E tests are skipped, but in this repo skipped required checks must be treated as failures and rerun (e.g., by adding the ready-for-e2e label). Update the guidance so engineers run E2E instead of ignoring it.
(Based on your team's feedback about skipped required CI jobs blocking merges.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agents/rules/ci-check-failures.md, line 29:
<comment>This sentence advises doing nothing when E2E tests are skipped, but in this repo skipped required checks must be treated as failures and rerun (e.g., by adding the ready-for-e2e label). Update the guidance so engineers run E2E instead of ignoring it.
(Based on your team's feedback about skipped required CI jobs blocking merges.) </comment>
<file context>
@@ -0,0 +1,33 @@
+**E2E tests skipping is expected behavior:**
+- When E2E tests are skipped, it's because the `ready-for-e2e` label has not been added to the PR
+- The "required" check intentionally fails when E2E tests are skipped to prevent merging without E2E
+- Do not try to fix anything related to skipped E2E tests - this is completely expected and normal
+
+## Before Blaming CI
</file context>
| ## PR Requirements | ||
|
|
||
| - PR title must follow Conventional Commits specification | ||
| - For most PRs, you only need to run linting and type checking |
There was a problem hiding this comment.
P2: This suggests skipping tests for most PRs, but AGENTS.md requires running relevant tests. Align this line with the repo standard.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agents/rules/quality-pr-creation.md, line 29:
<comment>This suggests skipping tests for most PRs, but AGENTS.md requires running relevant tests. Align this line with the repo standard.</comment>
<file context>
@@ -0,0 +1,36 @@
+## PR Requirements
+
+- PR title must follow Conventional Commits specification
+- For most PRs, you only need to run linting and type checking
+- E2E tests will only run if PR has "ready-for-e2e" label
+
</file context>
| @@ -0,0 +1,54 @@ | |||
| --- | |||
| title: packages/features vs apps/web/modules | |||
| impact: HIGH | |||
There was a problem hiding this comment.
P3: Architecture rules should use CRITICAL impact per agents/rules/README.md; update this frontmatter value to stay consistent with the documented rule format.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agents/rules/architecture-features-modules.md, line 3:
<comment>Architecture rules should use CRITICAL impact per agents/rules/README.md; update this frontmatter value to stay consistent with the documented rule format.</comment>
<file context>
@@ -0,0 +1,54 @@
+---
+title: packages/features vs apps/web/modules
+impact: HIGH
+impactDescription: Wrong placement causes tight coupling and import issues
+tags: architecture, features, modules, trpc
</file context>
| @@ -0,0 +1,39 @@ | |||
| --- | |||
There was a problem hiding this comment.
P3: This new rule file doesn’t follow the required rule template (missing Impact and Incorrect/Correct example sections). Please align it with agents/rules/README.md so downstream tooling and readers get the expected structure.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agents/rules/patterns-workflow-triggers.md:
<comment>This new rule file doesn’t follow the required rule template (missing **Impact** and Incorrect/Correct example sections). Please align it with agents/rules/README.md so downstream tooling and readers get the expected structure.</comment>
<file context>
@@ -0,0 +1,39 @@
+---
+title: Workflow Trigger Implementation
+impact: MEDIUM
+impactDescription: Consistent workflow patterns ensure reliable automation
+tags: workflows, triggers, automation
+---
+
+# Workflow Trigger Implementation
+
</file context>
|
|
||
| ## Import Patterns | ||
|
|
||
| Recent changes have moved from dynamic imports to static map-based imports for better performance. When working with browser components in the app-store, static imports should be used rather than dynamic imports. |
There was a problem hiding this comment.
P3: This guidance is contradictory: it says browser components should use static imports, but later notes browser component files may require dynamic imports. Please clarify the exception so readers know when dynamic imports are acceptable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agents/rules/patterns-app-store.md, line 18:
<comment>This guidance is contradictory: it says browser components should use static imports, but later notes browser component files may require dynamic imports. Please clarify the exception so readers know when dynamic imports are acceptable.</comment>
<file context>
@@ -0,0 +1,31 @@
+
+## Import Patterns
+
+Recent changes have moved from dynamic imports to static map-based imports for better performance. When working with browser components in the app-store, static imports should be used rather than dynamic imports.
+
+### Generated File Types
</file context>
| const jsonPath = path.join(agentsDir, "devin-knowledge.json"); | ||
|
|
||
| if (!fs.existsSync(jsonPath)) { | ||
| console.error("Error: devin-knowledge.json not found. Run parse-to-devin-knowledge.ts first."); |
There was a problem hiding this comment.
P3: The error message points to parse-to-devin-knowledge.ts, which doesn't exist. This will mislead users when the JSON file is missing; update it to reference the actual script name.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At scripts/devin/sync-knowledge-to-devin.ts, line 124:
<comment>The error message points to `parse-to-devin-knowledge.ts`, which doesn't exist. This will mislead users when the JSON file is missing; update it to reference the actual script name.</comment>
<file context>
@@ -0,0 +1,202 @@
+ const jsonPath = path.join(agentsDir, "devin-knowledge.json");
+
+ if (!fs.existsSync(jsonPath)) {
+ console.error("Error: devin-knowledge.json not found. Run parse-to-devin-knowledge.ts first.");
+ process.exit(1);
+ }
</file context>
Devin AI is addressing Cubic AI's review feedbackA Devin session has been created to address the issues identified by Cubic AI. |
…xample Co-Authored-By: unknown <>
- Remove duplicated Project Structure & Tech Stack from agents/README.md - Condense Commands section in AGENTS.md, link to agents/commands.md - Add AI-Assisted Development section to root README.md explaining the agents/ folder structure and symlink configuration Co-Authored-By: Claude Opus 4.5 <[email protected]>
There was a problem hiding this comment.
1 issue found across 3 files (changes from recent commits).
Prompt for AI agents (all issues)
Check if these issues are valid — if so, understand the root cause of each and fix them.
<file name="README.md">
<violation number="1" location="README.md:401">
P3: The README claims AGENTS.md is symlinked as CLAUDE.md, but the repo doesn’t contain CLAUDE.md. This is misleading for contributors—either add the symlink or update the docs to reflect current structure.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Devin AI is addressing Cubic AI's review feedbackA Devin session has been created to address the issues identified by Cubic AI. |
What does this PR do?
Adds automation to sync the
agents/folder documentation to Devin's Knowledge API. This enables automatic updates to Devin's knowledge base whenever agent documentation is modified.Components added:
scripts/devin/parse-local-knowledge.ts- Converts agents/ markdown files to Devin Knowledge JSON formatscripts/devin/validate-local-knowledge.ts- Validates rules have proper frontmatter (title, tags) and knowledge-base sections start with "When..."scripts/devin/sync-knowledge-to-devin.ts- Syncs knowledge entries to Devin API (create/update)scripts/devin/export-devin-knowledge.ts- Backup script to export existing Devin knowledgevalidate-agents-format.yml- Runs on PRs modifyingagents/**sync-agents-to-devin.yml- Runs on merge to main foragents/**changesMandatory Tasks (DO NOT REMOVE)
How should this be tested?
npx tsx scripts/devin/validate-local-knowledge.ts- should pass with "All files are valid!"npx tsx scripts/devin/parse-local-knowledge.ts- generatesagents/devin-knowledge.jsonwith ~55 entriesDEVIN_API_KEYenv var - not tested against live API yetBefore merging:
DEVIN_API_KEYas a GitHub secret for the sync workflow to workChecklist
Updates since last revision
scripts/devin/subfolder with clearer names (per user request):parse-local-knowledge.ts- parses local agents/ filesvalidate-local-knowledge.ts- validates local agents/ formatsync-knowledge-to-devin.ts- syncs to Devin APIexport-devin-knowledge.ts- exports from Devin API (converted from shell to TypeScript)../..since scripts are now two levels deepknowledge-base.md(consolidated to avoid bloat):Build[ServiceName]) in app-store sectionsync-knowledge-to-devin.tsto use-Sflag for proper env execution (addresses Cubic AI review with 9/10 confidence)data-prisma-feature-flags.mdto useCURRENT_TIMESTAMPinstead ofNOW()(addresses Cubic AI review with 9/10 confidence)Link to Devin run: https://app.devin.ai/sessions/66cd33c263024688adf7947cf0167f1e
Previous Devin runs: https://app.devin.ai/sessions/907872cc726c43db838918327716bd18, https://app.devin.ai/sessions/1007e75ecf9b4167a382d1ab36968972, https://app.devin.ai/sessions/b056870cc7fa42cbb2968159afa5929b
Requested by: @eunjae-lee