Logging
Logging for GitHub Copilot Clone
Logging is critical for understanding user behavior and system performance for this project. Logs are used to track user interactions, AI model behavior, and system events. All logs are stored in a Supabase database for analysis.
Table of Contents
1. Introduction to Logging
In this project, we log the following events:
- User Actions: When a user accepts or rejects a suggestion.
- AI Model Behavior: When the AI generates a suggestion.
All logs are stored in a Supabase database for easy querying and analysis.
2. Logging Events
👤 User Actions
Tracking user interactions (accept and reject) provides critical insights into how users respond to AI-generated code.
| Event | Purpose | Insights Gained |
|---|---|---|
accept | User accepts a suggestion, indicating it may meet their expectations. Tab | Helps determine if users are accepting buggy suggestions without noticing. |
reject | User rejects a suggestion, signaling an issue with its quality or relevance. Ctrl + Space | Indicates users’ ability to identify incorrect or suboptimal code. |
🤖 AI Model Behavior
Logging AI behavior ensures we understand how often AI introduces bugs.
| Event | Purpose | Insights Gained |
|---|---|---|
generate | AI generates a new suggestion. | Helps track the amount of generated suggestions. |
Got it — here’s a rewrite of that section so it matches your new logging format with the table structure you described:
3. Log Format
Suggestion logs are stored in separate tables depending on the type of suggestion (block, line, or selection). Each log entry captures both the triggering event and metadata needed to analyze user behavior, AI performance, and suggestion quality.
| Field | Purpose | Insights Gained |
|---|---|---|
event | The interaction that triggered the log (e.g., accept, reject, generate). | Enables categorization of user actions and AI response outcomes. |
duration | The time (in ms) between the suggestion being shown and the user’s action. | Measures how long users review suggestions before deciding, or how fast the AI was interacted with. |
user_id | Identifies the user who interacted with the suggestion. | Supports user-specific analysis and behavior patterns. |
suggestion_id | Links the log entry to the suggestion details table. | Allows cross-reference with suggestion content and attributes. |
class_id | Identifies the context/class in which the suggestion was generated. | Enables comparison of suggestion effectiveness across different contexts. |
Each suggestion_id maps to a suggestion record containing:
| Suggestion Field | Purpose | Insights Gained |
|---|---|---|
prompt | The original input or context that led to the suggestion. | Enables evaluation of AI performance given specific user prompts. |
has_bug | Indicates whether the suggestion introduces or contains a bug. | Supports bug-rate tracking and quality assessment. |
text | The actual suggestion content generated by the AI. | Provides concrete material for reviewing AI output quality. |
model | The AI model that produced the suggestion. | Allows comparison across different models and configurations. |
language | The programming language of the suggestion. | Facilitates language-specific performance analysis. |
Why It’s Important
- Duration Analysis: Short review times with high acceptance rates may suggest users aren’t thoroughly checking suggestions.
- Bug Tracking: Comparing outcomes for buggy vs. correct suggestions highlights areas for model improvement.
- Cross-Linked Metadata: User, class, and suggestion details allow rich analysis of behavior patterns and model effectiveness across contexts.
4. Analysis Scenarios
🧑💻 Users frequently accept buggy suggestions
- If logs show that a user accepts suggestions linked to
bug = truewith very shortduration, it may indicate they are not reviewing code carefully. - A pattern of later rejections after exposure to buggy suggestions could suggest the user is adapting and becoming more cautious over time.
🧑💻 Differences in decision time
- Users who spend more time (
duration) before accepting a suggestion are likely analyzing it more carefully. - Consistently fast rejections may indicate stricter quality standards or a preference for writing code independently rather than relying on AI output.
🧑💻 High rejection rates for AI suggestions
- If a large proportion of suggestions linked to
suggestion_idare rejected, this may indicate low-quality outputs from the AI model. - Correlating rejection rates with the
bugfield helps determine whether issues come from obvious bugs, subtle flaws, or simply poor fit for the user’s coding style. - Model-level analysis (
modelfield) can reveal whether certain models perform worse in specific contexts or languages.