Skip to content

fix json format use llm #1275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
iceljc merged 5 commits into from
Jan 15, 2026
Merged

fix json format use llm #1275

iceljc merged 5 commits into from
Jan 15, 2026

Conversation

@yileicn
Copy link
Member

@yileicn yileicn commented Jan 14, 2026 •
edited by qodo-code-review bot
Loading

PR Type

Enhancement

Description

  • Add JSON repair service using LLM for malformed JSON

  • Implement IJsonRepairService interface with repair methods

  • Create JsonRepairPlugin for dependency injection registration

  • Add Liquid template for JSON repair prompt generation

Diagram Walkthrough

flowchart LR
  A["Malformed JSON"] -->|RepairAndDeserialize| B["JsonRepairService"]
  B -->|Repair| C["LLM Agent"]
  C -->|json_repair.liquid| D["Repaired JSON"]
  D -->|Deserialize| E["Target Type T"]
  B -->|Direct Deserialization| F["Success/Default"]
Loading

File Walkthrough
Relevant files
Enhancement
IJsonRepairService.cs
JSON repair service interface definition                                 

src/Infrastructure/BotSharp.Abstraction/Utilities/IJsonRepairService.cs

  • Define IJsonRepairService interface with two methods
  • RepairAndDeserialize method for repair and deserialization
  • Repair method for JSON string repair only
 +23/-0   
JsonRepairPlugin.cs
Plugin registration for JSON repair service                           

src/Infrastructure/BotSharp.Core/JsonRepair/JsonRepairPlugin.cs

  • Create JsonRepairPlugin implementing IBotSharpPlugin
  • Register JsonRepairService in dependency injection
  • Provide plugin metadata with ID, name, and description
 +19/-0   
JsonRepairService.cs
JSON repair service implementation with LLM                           

src/Infrastructure/BotSharp.Core/JsonRepair/JsonRepairService.cs

  • Implement IJsonRepairService with LLM-based repair logic
  • RepairAndDeserialize attempts direct deserialization first
  • Repair method uses router agent and Liquid template
  • Handle exceptions and log repair results
 +99/-0   
json_repair.liquid
Liquid template for JSON repair prompt                                     

src/Infrastructure/BotSharp.Core/data/agents/01fcc3e5-9af7-49e6-ad7a-a760bd12dc4a/templates/json_repair.liquid

  • Create Liquid template for JSON repair prompt
  • Template instructs LLM to fix malformed JSON
  • Output valid JSON only from LLM response
 +3/-0     
Configuration changes
BotSharp.Core.csproj
Project file updates for template resources                           

src/Infrastructure/BotSharp.Core/BotSharp.Core.csproj

  • Add None Remove entry for json_repair.liquid template
  • Add ItemGroup with CopyToOutputDirectory for normalize_json.liquid
  • Update project file to include new template resources
 +7/-0     

@qodo-code-review
Copy link

qodo-code-review bot commented Jan 14, 2026 •
edited
Loading

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
🔴
Sensitive data logging

Description: The code logs the full LLM output (response.Content) for “JSON repair result”, which may
include sensitive user-provided data from malformedJson and therefore can cause sensitive
information exposure in application logs.
JsonRepairService.cs [95-96]

Referred Code 
_logger.LogInformation($"JSON repair result: {response.Content}");
return response.Content;
⚪
Untrusted LLM output

Description: Untrusted malformedJson is sent to an LLM and the unvalidated response (response.Content)
is returned and later deserialized, creating a realistic risk of
prompt-injection/manipulated output (e.g., returning unexpected structure or very large
payload) that could lead to downstream misuse or resource exhaustion unless strict
validation/size limits are enforced.
JsonRepairService.cs [68-97]

Referred Code 
var render = _services.GetRequiredService<ITemplateRender>();
var prompt = render.Render(template, new Dictionary<string, object>
{
    { "input", malformedJson }
});

var completion = CompletionProvider.GetChatCompletion(_services,
    provider: router?.LlmConfig?.Provider,
    model: router?.LlmConfig?.Model);

var agent = new Agent
{
    Id = Guid.Empty.ToString(),
    Name = "JsonRepair",
    Instruction = "You are a JSON repair expert."
};

var dialogs = new List<RoleDialogModel>
{
    new RoleDialogModel(AgentRole.User, prompt)
    {


 ... (clipped 9 lines)
Ticket Compliance
⚪
🎫 No ticket provided
  • Create ticket/issue
Codebase Duplication Compliance
⚪
Codebase context is not defined

Follow the guide to enable codebase context checks.

Custom Compliance
🟢
Generic: Meaningful Naming and Self-Documenting Code

Objective: Ensure all identifiers clearly express their purpose and intent, making code
self-documenting

Status: Passed

Learn more about managing compliance generic rules or creating your own custom rules

Generic: Secure Error Handling

Objective: To prevent the leakage of sensitive system information through error messages while
providing sufficient detail for internal debugging.

Status: Passed

Learn more about managing compliance generic rules or creating your own custom rules

🔴
Generic: Robust Error Handling and Edge Case Management

Objective: Ensure comprehensive error handling that provides meaningful context and graceful
degradation

Status:
Swallowed exception: The code catches and ignores the initial deserialization exception without logging/context
and performs no explicit null/empty validation on malformedJson, reducing debuggability
and edge-case safety.

Referred Code 
public async Task<T?> RepairAndDeserialize<T>(string malformedJson)
{

    try
    {
        // First try direct deserialization
        return malformedJson.JsonContent<T>();
    }
    catch
    {
        // Continue to repair
    }

Learn more about managing compliance generic rules or creating your own custom rules

Generic: Secure Logging Practices

Objective: To ensure logs are useful for debugging and auditing without exposing sensitive
information like PII, PHI, or cardholder data.

Status:
Sensitive data logged: The service logs the full LLM-produced repaired JSON (response.Content), which may contain
sensitive user-provided data, violating secure logging requirements.

Referred Code 
var response = await completion.GetChatCompletions(agent, dialogs);

_logger.LogInformation($"JSON repair result: {response.Content}");
return response.Content;

Learn more about managing compliance generic rules or creating your own custom rules

⚪
Generic: Comprehensive Audit Trails

Objective: To create a detailed and reliable record of critical system actions for security analysis
and compliance.

Status:
Missing audit context: The new JSON repair flow emits logs without user identity/correlation context, making it
hard to reconstruct who triggered the repair and its outcome if this action is considered
audit-critical.

Referred Code 
        _logger.LogError(ex, "Failed to repair and deserialize JSON");
        return default;
    }
}

public async Task<string> Repair(string malformedJson)
{
    var agentService = _services.GetRequiredService<IAgentService>();
    var router = await agentService.GetAgent(ROUTER_AGENT_ID);

    var template = router.Templates?.FirstOrDefault(x => x.Name == TEMPLATE_NAME)?.Content;
    if (string.IsNullOrEmpty(template))
    {
        _logger.LogWarning($"Template '{TEMPLATE_NAME}' not found in agent '{ROUTER_AGENT_ID}'");
        return malformedJson;
    }

    var render = _services.GetRequiredService<ITemplateRender>();
    var prompt = render.Render(template, new Dictionary<string, object>
    {
        { "input", malformedJson }


 ... (clipped 25 lines)

Learn more about managing compliance generic rules or creating your own custom rules

Generic: Security-First Input Validation and Data Handling

Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent
vulnerabilities

Status:
Unvalidated LLM input: The method forwards malformedJson directly into an LLM prompt without visible
validation/size limits/sanitization, which may allow prompt-injection or resource
exhaustion depending on how this input is sourced.

Referred Code 
public async Task<string> Repair(string malformedJson)
{
    var agentService = _services.GetRequiredService<IAgentService>();
    var router = await agentService.GetAgent(ROUTER_AGENT_ID);

    var template = router.Templates?.FirstOrDefault(x => x.Name == TEMPLATE_NAME)?.Content;
    if (string.IsNullOrEmpty(template))
    {
        _logger.LogWarning($"Template '{TEMPLATE_NAME}' not found in agent '{ROUTER_AGENT_ID}'");
        return malformedJson;
    }

    var render = _services.GetRequiredService<ITemplateRender>();
    var prompt = render.Render(template, new Dictionary<string, object>
    {
        { "input", malformedJson }
    });

Learn more about managing compliance generic rules or creating your own custom rules

  • Update
Compliance status legend 🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

@qodo-code-review
Copy link

qodo-code-review bot commented Jan 14, 2026 •
edited
Loading

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
High-level
Consider a simpler, non-LLM alternative

It is suggested to replace the LLM-based JSON repair mechanism with a dedicated,
deterministic JSON repair library or a rule-based parser. This would improve
performance, reduce cost, and increase reliability by avoiding the overhead and
non-determinism of an LLM call.

Examples:

src/Infrastructure/BotSharp.Core/JsonRepair/JsonRepairService.cs [56-97] 
    public async Task<string> Repair(string malformedJson)
    {
        var agentService = _services.GetRequiredService<IAgentService>();
        var router = await agentService.GetAgent(ROUTER_AGENT_ID);
        
        var template = router.Templates?.FirstOrDefault(x => x.Name == TEMPLATE_NAME)?.Content;
        if (string.IsNullOrEmpty(template))
        {
            _logger.LogWarning($"Template '{TEMPLATE_NAME}' not found in agent '{ROUTER_AGENT_ID}'");
            return malformedJson;

 ... (clipped 32 lines)

Solution Walkthrough:

Before:

public class JsonRepairService : IJsonRepairService
{
    public async Task<string> Repair(string malformedJson)
    {
        // 1. Get an agent
        var router = await agentService.GetAgent(ROUTER_AGENT_ID);
        
        // 2. Render a prompt for the LLM
        var prompt = render.Render(template, new { input = malformedJson });
        
        // 3. Get a chat completion provider
        var completion = CompletionProvider.GetChatCompletion(_services, ...);
        
        // 4. Call the LLM to get the repaired JSON
        var response = await completion.GetChatCompletions(agent, dialogs);
        return response.Content;
    }
}

After:

// Using a hypothetical deterministic JSON repair library
public class JsonRepairService : IJsonRepairService
{
    public Task<string> Repair(string malformedJson)
    {
        try
        {
            // Use a deterministic, local library to fix common JSON errors.
            // This is faster, cheaper, and more reliable than an LLM call.
            string repairedJson = SomeJsonRepairLibrary.Fix(malformedJson);
            return Task.FromResult(repairedJson);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to repair JSON deterministically.");
            return Task.FromResult(malformedJson); // Or re-throw
        }
    }
}
Suggestion importance[1-10]: 9

__

Why: This suggestion correctly identifies significant performance, cost, and reliability issues with using an LLM for JSON repair and proposes a more standard, efficient, and deterministic architectural alternative.

High
Possible issue
✅ Copy JSON repair template
Suggestion Impact:The commit updated the .csproj to copy json_repair.liquid to the output directory. However, instead of adding a new entry alongside normalize_json.liquid, it replaced the normalize_json.liquid copy entry with json_repair.liquid.

code diff:

   <ItemGroup>
-    <None Update="data\agents\01fcc3e5-9af7-49e6-ad7a-a760bd12dc4a\templates\normalize_json.liquid">
+    <None Update="data\agents\01fcc3e5-9af7-49e6-ad7a-a760bd12dc4a\templates\json_repair.liquid">
       <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
     </None>
   </ItemGroup>

In the .csproj file, add a entry for json_repair.liquid with to
ensure the template file is copied to the build output and available at runtime.

src/Infrastructure/BotSharp.Core/BotSharp.Core.csproj [295-299]

 <ItemGroup>
   <None Update="data\agents\01fcc3e5-9af7-49e6-ad7a-a760bd12dc4a\templates\normalize_json.liquid">
     <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
   </None>
+  <None Update="data\agents\01fcc3e5-9af7-49e6-ad7a-a760bd12dc4a\templates\json_repair.liquid">
+    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+  </None>
 </ItemGroup>

[Suggestion processed]

Suggestion importance[1-10]: 8

__

Why: This suggestion correctly identifies a critical missing configuration for the new json_repair.liquid template, which is essential for the JsonRepairService to function correctly at runtime.

Medium
Learned
best practice
✅ Add null checks and fallback
Suggestion Impact:The commit increased defensive behavior by wrapping the LLM repair path in a try/catch, using null-conditional access on router LLM config, and falling back to returning the original JSON on exceptions. However, it did not add the explicit router-not-found warning/early return or the explicit response.Content null/empty checks and warning logs as suggested.

code diff:

-    public async Task<string> Repair(string malformedJson)
+    private async Task<string> RepairByLLM(string malformedJson)
     {
         var agentService = _services.GetRequiredService<IAgentService>();
         var router = await agentService.GetAgent(ROUTER_AGENT_ID);
@@ -71,18 +77,20 @@
             { "input", malformedJson }
         });
 
-        var completion = CompletionProvider.GetChatCompletion(_services,
+        try
+        {
+            var completion = CompletionProvider.GetChatCompletion(_services,
             provider: router?.LlmConfig?.Provider,
             model: router?.LlmConfig?.Model);
 
-        var agent = new Agent
-        {
-            Id = Guid.Empty.ToString(),
-            Name = "JsonRepair",
-            Instruction = "You are a JSON repair expert."
-        };
+            var agent = new Agent
+            {
+                Id = Guid.Empty.ToString(),
+                Name = "JsonRepair",
+                Instruction = "You are a JSON repair expert."
+            };
 
-        var dialogs = new List<RoleDialogModel>
+            var dialogs = new List<RoleDialogModel>
         {
             new RoleDialogModel(AgentRole.User, prompt)
             {
@@ -90,9 +98,15 @@
             }
         };
 
-        var response = await completion.GetChatCompletions(agent, dialogs);
+            var response = await completion.GetChatCompletions(agent, dialogs);
 
-        _logger.LogInformation($"JSON repair result: {response.Content}");
-        return response.Content;
+            _logger.LogInformation($"JSON repair result: {response.Content}");
+            return response.Content.CleanJsonStr();
+        }
+        catch (Exception ex)
+        {
+            _logger.LogError(ex, "Failed to repair and deserialize JSON");
+            return malformedJson;
+        }

Guard against router or response.Content being null (e.g., agent not found,
provider errors) and fall back safely to the original JSON while logging the
reason.

src/Infrastructure/BotSharp.Core/JsonRepair/JsonRepairService.cs [58-96]

 var agentService = _services.GetRequiredService<IAgentService>();
 var router = await agentService.GetAgent(ROUTER_AGENT_ID);
+if (router == null)
+{
+    _logger.LogWarning("Agent '{AgentId}' not found; returning original JSON.", ROUTER_AGENT_ID);
+    return malformedJson;
+}
 
 var template = router.Templates?.FirstOrDefault(x => x.Name == TEMPLATE_NAME)?.Content;
 if (string.IsNullOrEmpty(template))
 {
-    _logger.LogWarning($"Template '{TEMPLATE_NAME}' not found in agent '{ROUTER_AGENT_ID}'");
+    _logger.LogWarning("Template '{TemplateName}' not found in agent '{AgentId}'; returning original JSON.", TEMPLATE_NAME, ROUTER_AGENT_ID);
     return malformedJson;
 }
 ...
 var response = await completion.GetChatCompletions(agent, dialogs);
 
-_logger.LogInformation($"JSON repair result: {response.Content}");
-return response.Content;
+var content = response?.Content;
+if (string.IsNullOrWhiteSpace(content))
+{
+    _logger.LogWarning("JSON repair returned empty content; returning original JSON.");
+    return malformedJson;
+}
 
+_logger.LogInformation("JSON repair result: {Json}", content);
+return content;
+

[To ensure code accuracy, apply this suggestion manually]

Suggestion importance[1-10]: 6

__

Why:
Relevant best practice - Improve defensive coding with precise null and state checks to prevent crashes and incorrect behavior.

Low
✅ Log exceptions instead of swallowing
Suggestion Impact:The commit refactored the flow so direct deserialization is no longer wrapped in a swallow-all catch; instead, it validates JSON separately and then deserializes without catching (so failures aren't silently swallowed). Additionally, it added a try/catch with LogError around the LLM-based repair call. However, it did not implement the specific suggested LogDebug in the "direct deserialization failed" catch (that catch was removed rather than updated).

code diff:

-        var completion = CompletionProvider.GetChatCompletion(_services,
+        try
+        {
+            var completion = CompletionProvider.GetChatCompletion(_services,
             provider: router?.LlmConfig?.Provider,
             model: router?.LlmConfig?.Model);
 
-        var agent = new Agent
-        {
-            Id = Guid.Empty.ToString(),
-            Name = "JsonRepair",
-            Instruction = "You are a JSON repair expert."
-        };
+            var agent = new Agent
+            {
+                Id = Guid.Empty.ToString(),
+                Name = "JsonRepair",
+                Instruction = "You are a JSON repair expert."
+            };
 
-        var dialogs = new List<RoleDialogModel>
+            var dialogs = new List<RoleDialogModel>
         {
             new RoleDialogModel(AgentRole.User, prompt)
             {
@@ -90,9 +98,15 @@
             }
         };
 
-        var response = await completion.GetChatCompletions(agent, dialogs);
+            var response = await completion.GetChatCompletions(agent, dialogs);
 
-        _logger.LogInformation($"JSON repair result: {response.Content}");
-        return response.Content;
+            _logger.LogInformation($"JSON repair result: {response.Content}");
+            return response.Content.CleanJsonStr();
+        }
+        catch (Exception ex)
+        {
+            _logger.LogError(ex, "Failed to repair and deserialize JSON");
+            return malformedJson;
+        }

Avoid swallowing exceptions; at minimum log the exception so callers/operators
can understand why parsing failed before repair is attempted.

src/Infrastructure/BotSharp.Core/JsonRepair/JsonRepairService.cs [33-41]

 try
 {
     // First try direct deserialization
     return malformedJson.JsonContent<T>();
 }
-catch
+catch (Exception ex)
 {
+    _logger.LogDebug(ex, "Direct JSON deserialization failed; attempting LLM repair.");
     // Continue to repair
 }

[Suggestion processed]

Suggestion importance[1-10]: 5

__

Why:
Relevant best practice - Preserve error details by capturing exception messages (and other error output) instead of swallowing exceptions and returning generic defaults.

Low
  • Update

iceljc
iceljc reviewed Jan 15, 2026
View reviewed changes
@yileicn
Co-authored-by: adenchen123 <[email protected]>
f5f0381 
Co-authored-by: Aden <[email protected]>
Co-authored-by: Jicheng Lu <[email protected]>
Co-authored-by: iceljc <[email protected]>
Co-authored-by: Jack <[email protected]>
Co-authored-by: jackjiang-sms <[email protected]>
optimize ResponseFormat
@iceljc
Copy link
Collaborator

iceljc commented Jan 15, 2026

The current changes cannot be compiled.

@yileicn
Copy link
Member Author

yileicn commented Jan 15, 2026

The current changes cannot be compiled.

Fixed

@iceljc iceljc merged commit 022a175 into SciSharp:master Jan 15, 2026
4 checks passed
@JackJiang1234
Copy link
Contributor

JackJiang1234 commented Jan 15, 2026

reviewed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iceljc iceljc iceljc left review comments

@Oceania2018 Oceania2018 Oceania2018 approved these changes

Assignees

@iceljc iceljc

Labels

Review effort 3/5

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yileicn @iceljc @JackJiang1234 @Oceania2018