The problem: Production questions need tribal knowledge
Every production Rails app has questions that only 1-2 people can answer. “Why did sales drop for this product last week?” “Is this commission report correct?” “Why is data missing since yesterday for this client?”
These aren't hard questions technically. But answering them requires knowing which database scopes to apply, which internal transactions to exclude, how cumulative calculations cross tier boundaries, and which visibility settings filter what each tenant sees.
I work on a multi-tenant SaaS platform that tracks sales across multiple distribution channels, calculates tiered commissions, and predicts future demand using lifecycle-based forecasting. The codebase has 50+ models, 30+ services, and business logic scattered across calculators, fetchers, and JSONB columns.
When a client asks “is my commission report correct?”, someone has to trace through order visibility scopes, check that quantity tiers split correctly at boundaries, verify advance deductions, and confirm no manual adjustments are orphaned. That investigation takes 20-30 minutes even if you know the system well.
The approach: Encode domain knowledge as skill files
Claude Code has a feature called skills — markdown files that teach the AI how to handle specific types of tasks. When you ask a question that matches a skill's trigger, it loads the skill and follows its patterns, rules, and query templates.
I built 3 skills that cover the questions clients and operators ask most:
- Sales Inquiry — Sales by channel, top products, trend analysis, retailer breakdowns
- Commission Inquiry — Earnings verification, tier validation, payment status, advance deductions
- Operations Inquiry — Demand predictions, sync diagnostics, data freshness, forecast accuracy
Each skill teaches the AI how to generate the right ActiveRecord queries, which then run on your own server via kamal console. The AI writes the code; your infrastructure executes it. No direct database access — just generated queries using the app's actual models and scopes.
The process took about 2 days:
- Deep analysis of 3 subsystems (data sources, commissions, predictions)
- Extract actual business logic from services, controllers, and helpers
- Write ActiveRecord queries that mirror the app's code paths
- Verify every query against production data
- Add safety rules, disambiguation patterns, and error interpretation
Anatomy of a skill file
A skill file is structured markdown with a YAML header for trigger conditions. Here's the conceptual structure (sanitized):
---
name: app-sales-inquiry
description: |
Use when user asks about product sales, trends, channel
breakdown, or comparing performance.
- MANDATORY TRIGGERS: sales, top products, trending, channel
- PRIORITY: Invoke FIRST for any sales data question.
---
# Sales Inquiry
## MANDATORY: Read-Only Safety
Only run READ queries. Never delete, update, or trigger jobs
unless explicitly requested with double confirmation.
## MANDATORY: Visibility Scope
Every query touching orders MUST use the organization's
visibility scope. Without it, results include hidden orders
and won't match what the UI shows.
## Common Patterns
### Entity Lookup (handle ambiguity)
Search by partial name, present matches, confirm before querying.
### Date Range Interpretation
"this month", "Q4 2025", "since launch" → parse to date ranges.
### Result Interpretation
0 rows? Explain why. Spike? Check anomaly flags. Mismatch?
Explain the scope difference.
## Reference Queries
[ActiveRecord examples that follow all mandatory rules]The key insight: the queries are references, not rigid templates. The skill teaches the AI which scopes are mandatory, which columns to use, and which exclusions to apply — then the AI composes its own queries based on what the user actually asks.
Real queries, anonymized results
Here's what it looks like when the skill runs against production data. I've anonymized the specifics.
“Which products are selling best this month?”
The AI applies the visibility scope, excludes internal transfers, groups by product, and compares to the prior period:
Product A: 4,950 units (UP 1278%) [launch_peak]
Product B: 2,145 units (DOWN -17.9%) [long_tail]
Product C: 1,433 units (UP 42.4%) [decay]
Product D: 1,174 units (UP 60.4%) [long_tail]
Product E: 1,135 units (NEW) [launch_peak]“When will this product run out of stock?”
The AI reads the prediction model's output (lifecycle-based forecasting with seasonal adjustment) and finds the first date where projected inventory hits zero:
Current stock: 0
Method: lifecycle
Stockout: 2026-03-23 (4 days)
Sales forecast: 30d = 2,936 units
Lifecycle stage: launch_peak“Is this commission report correct?”
This is the most complex query. The AI runs a 5-step verification:
1. Tier rules: 0..3999 → 4.5%, 4000+ → 5.1% ✓ No gaps
2. Cumulative progression: monotonically increasing ✓
3. Tier boundary crossings: 0 splits found ✓
4. Orphaned manual adjustments: 0 ✓
5. Payment date overlaps: none for regular payments ✓“Why is data missing since yesterday?”
The AI checks data freshness per source, recent sync logs, credential state, and consecutive failure counts:
Source A: last = Mar 18 (1 day ago) ✓
Source B: last = Mar 18 (1 day ago) ✓
Recent syncs: all completed, 0 failures
Credentials: source_a = verified, source_b = verifiedThe guardrails that make this safe
Giving AI access to production sounds terrifying. It's not — if the guardrails are built into the skill itself:
Read-Only by Default
No delete, update, create, or job execution. Destructive actions require the user to explicitly request them AND confirm twice. The skill lists every blocked method by name.
Mandatory Visibility Scopes
Every query touching orders MUST use the app's visibility scope. This ensures results match what the UI shows — not raw unfiltered data.
ActiveRecord Only
No raw SQL. Queries use the app's actual models, scopes, and methods. If the business logic changes, the queries still follow it.
Disambiguate Before Querying
When someone says “check sales for the protein product”, the AI searches, presents matches, and asks which one — before running any data query.
The next step is pointing these queries at a read replica instead of the primary database. That's the production-grade setup — zero risk of accidental writes, and no load on the primary.
Privacy and compliance: the AI never touches your database
This is the part people get wrong. The AI does not access your database. Here's what actually happens:
The actual data flow
1. You ask a question in your terminal
2. AI reads the skill file (patterns, rules, scopes)
3. AI generates ActiveRecord code based on your question
4. Code runs on your server via kamal console
5. Results display in your terminal
6. AI reads the terminal output to interpret it for you
The AI is a code generator, not a database client. It writes the query; your infrastructure executes it. This is the same as asking a colleague to write you a Rails console command — they'd see the output too, but they never had direct database access.
What the AI sees vs. what it doesn't
- ✓Skill file — patterns, rules, scopes (no data, just instructions)
- ✓Terminal output — aggregated results like “Product A: 4,950 units”
- ✕Direct database access — AI cannot connect to your database
- ✕Raw data tables — AI only sees what the query prints to the console
Additional layers we built in
1. Queries return aggregates, not personal records
The generated code is designed to output sums, counts, and status checks. Even the terminal output the AI reads back contains business metrics — not names, emails, or payment details.
2. Local database clone for development
For building and testing skills, we use a local database snapshot. The AI generates and tests queries locally — production data never leaves the server during development.
3. Read replica isolation (recommended for production)
The production-grade setup runs AI-generated queries on a read replica with restricted column access. Sensitive columns can be excluded entirely, so personal data never appears in query output.
4. No data persistence
Claude Code processes conversations in-session. Terminal output is not stored, trained on, or retained after the session ends. The skill files contain only code patterns and rules — zero actual data.
The bottom line: the AI writes code, your server runs it. The AI never has a database connection. It's a code generation tool with domain knowledge, not a data pipeline.
What the AI couldn't figure out on its own
This is the part that matters. When I first asked the AI to write queries for this app, it guessed wrong on almost everything:
- ✕It read quantity from the wrong JSONB field. The app uses a method that validates per-source with regex — not a simple column read.
- ✕It didn't know about the visibility scope that filters which orders each tenant sees. Without it, totals were wrong.
- ✕It didn't exclude internal transfer orders that the dashboard always filters out.
- ✕It used raw SQL instead of ActiveRecord, which meant the queries would silently break when the business logic changed.
- ✕It got the prediction thresholds wrong (days required for each forecasting method) because it assumed instead of reading the actual constants from the code.
Every one of these mistakes was caught by a developer who knows the system. The skill file encodes those corrections so future AI sessions get it right from the start.
“The AI didn't figure out which scopes to use, how tier crossings split calculations, or why certain orders are excluded from dashboards. A developer who understands the system did. AI amplifies domain expertise — it doesn't replace it.”
What this means for engineering teams
The title of this post is deliberately provocative. I'm not saying AI replaces developers. I'm saying the type of work that creates value is shifting.
Before skill files:
- • Client asks “are the commissions correct?”
- • Support tickets it to a developer
- • Developer opens Rails console, runs 5-6 queries, traces the logic
- • 30 minutes later, replies with findings
- • This knowledge stays in that developer's head
After skill files:
- • Client asks “are the commissions correct?”
- • AI loads the commission skill, connects to production, runs the verification
- • 30 seconds later, presents the 5-step checklist with findings
- • The knowledge is encoded, versioned, and available 24/7
The developer who built the skill is more valuable, not less. They're the person who can:
- • Understand the system deeply enough to teach it to AI
- • Catch when the AI gets the logic wrong
- • Maintain the skills as the app evolves
- • Build the same layer for the next system
That's not a job that's going away. It's a job that's becoming the most important one on the team.
FAQ
What is a Claude Code skill file?
A markdown document that teaches Claude Code how to handle specific tasks. It defines trigger conditions, mandatory rules, query patterns, and interpretation guides. Loaded automatically when the AI detects a matching question.
Is it safe to give AI access to a production database?
The AI doesn't access your database directly. It generates code that runs on your server. It only sees the terminal output. Skills enforce read-only queries, mandatory scopes, and double confirmation for any write action. For extra safety, use a read replica.
Does this replace developers?
No. A developer who deeply understands the system builds and maintains the skills. The AI handles the routine queries that would otherwise interrupt deep work. Domain expertise becomes more valuable, not less.
How long did this take to build?
About 2 days: 1 day for deep codebase analysis and extracting business logic, 1 day for writing skills, verifying queries against production data, and iterating on edge cases and safety rules.
What stack does this require?
Claude Code (CLI), a Rails app deployed with Kamal (for kamal console access), and PostgreSQL. The concept works with any framework that has a production console and an ORM.
Want this for your Rails app?
I build AI operations layers for production Rails applications — custom skill files that encode your business logic, connect to your production console, and answer domain-specific questions with proper guardrails.
Get in touch