Quand le prompt ne suffit plus
Quand le prompt ne suffit plus
| Position | Coefficient | SMC mensuel |
|---|---|---|
| 1.1 | 240 | 1 815,00 € |
| 1.2 | 250 | 1 845,00 € |
| 2.1 | 275 | 1 875,00 € |
| 2.2 | 310 | 1 905,00 € |
| 2.3 | 355 | 2 045,00 € |
| 3.1 | 400 | 2 185,00 € |
| 3.2 | 450 | 2 340,00 € |
| 3.3 | 500 | 2 490,00 € |
| Position | Coefficient | SMC mensuel |
|---|---|---|
| 1.1 | 95 | 2 135,00 € |
| 1.2 | 100 | 2 240,00 € |
| 2.1 | 105 | 2 315,00 € |
| 2.1 | 115 | 2 530,00 € |
| 2.2 | 130 | 2 850,00 € |
| 2.3 | 150 | 3 275,00 € |
| 3.1 | 170 | 3 650,00 € |
| 3.2 | 210 | 4 495,00 € |
| 3.3 | 270 | 5 755,00 € |
| Pos. | Coef. | SMC |
|---|---|---|
| 1.1 | 240 | 1 815 € |
| 1.2 | 250 | 1 845 € |
| 2.1 | 275 | 1 875 € |
| 2.2 | 310 | 1 905 € |
| 2.3 | 355 | 2 045 € |
| 3.1 | 400 | 2 185 € |
| 3.2 | 450 | 2 340 € |
| 3.3 | 500 | 2 490 € |
| Pos. | Coef. | SMC |
|---|---|---|
| 1.1 | 95 | 2 135 € |
| 1.2 | 100 | 2 240 € |
| 2.1 | 105 | 2 315 € |
| 2.1 | 115 | 2 530 € |
| 2.2 | 130 | 2 850 € |
| 2.3 | 150 | 3 275 € |
| 3.1 | 170 | 3 650 € |
| 3.2 | 210 | 4 495 € |
| 3.3 | 270 | 5 755 € |
[ { "classifications": ["ETAM", "1.1", "240"], "monthlyAmount": 1815 }, { "classifications": ["Cadres", "1.1", "95"], "monthlyAmount": 2135 }, ... { "classifications": ["Cadres", "3.3", "270"], "monthlyAmount": 5755 } ]
101 — Le prompt
Quand le prompt ne suffit plus
Non-déterministe ≠ Non-testable
101 — Le prompt
// appel au service LLM const result = await llmService.callLLM({ systemPrompt, userPrompt, });
<context> You're a French expert on labor and payroll law. You receive an excerpt of a law text (arrêté or avenant) updating the minimum wages (Salaire Minimum Conventionnel, SMC) of a convention collective nationale (CCN). You need to extract those updates as structured JSON. </context>
Your output MUST be a JSON array of MinimumWageUpdate operations matching the following type: type MinimumWageUpdate = { type: 'minimumWage'; classification: string[]; minimumWage: MinimumWage; applicabilityDate: { date?: string; firstDayOfMonthAfterPublication?: boolean; }; }; type MinimumWage = | MonthlyMinimumWage | HourlyMinimumWage; type MonthlyMinimumWage = { type: 'monthly'; monthlyAmount: number; }; type HourlyMinimumWage = { type: 'hourly'; hourlyRate: number; };
<classificationLevels> You MUST use the existing level values (between quotes) as provided in the user prompt. YOU MUST ALWAYS USE THE FULL PATH of the classification levels. <warning> When the text includes "emploi repères" or job titles, do NOT use these to restrict the categories to update. </warning> </classificationLevels> <minimumWageTypes> The minimum wage can be of different types: - Monthly amount (forfait mensuel) // preferred when both are present - Hourly rate <important> YOU MUST USE IN PRIORITY THE MONTHLY AMOUNT (FORFAIT MENSUEL) TYPE if the text mentions a monthly amount for that level anywhere. </important> </minimumWageTypes> <applicabilityDate> Extract the applicability date in YYYY-MM-DD format. If the text mentions "au plus tôt le 1er jour du mois civil suivant la date de publication...", set firstDayOfMonthAfterPublication to true. </applicabilityDate> <rules> - Use the EXACT CAPITALIZATION as provided in <classificationLevels> (e.g. "Ingénieurs et Cadres" with capital "C"). - If the text is not about SMC, return an empty array. </rules> <check> BEFORE finalizing your answer: 1. Verify the count matches the number of classification paths the text actually updates. 2. Verify each path has the SAME NUMBER OF LEVELS. 3. For each operation, confirm an EXACT MATCH in the existing classification levels. </check>
<examples> // Example: the text mentions coefficients only — we must // expand each one to its FULL classification path. // INPUT excerpt: "Les salaires minimums conventionnels sont fixés comme suit : Coefficient 240 : 1 820 €/mois Coefficient 250 : 1 850 €/mois Applicable au 1er jour du mois suivant publication." // EXPECTED OUTPUT: [ { type: 'minimumWage', classification: ["ETAM", "1.1", "240"], minimumWage: { type: 'monthly', monthlyAmount: 1820 }, applicabilityDate: { firstDayOfMonthAfterPublication: true } }, { type: 'minimumWage', classification: ["ETAM", "1.2", "250"], minimumWage: { type: 'monthly', monthlyAmount: 1850 }, applicabilityDate: { firstDayOfMonthAfterPublication: true } } ] </examples>
<format> ⚠️ CRITICAL: ANSWER STRICTLY IN JSON FORMAT ONLY ⚠️ - Do not include explanations - Do not include commentary - Do not include any text outside the JSON array - Your entire response must be valid JSON </format>
Current CCN is #${collectiveAgreementCode}: "${collectiveAgreementName}" <classificationLevels> Current classification level names are: Level 1: Catégorie Level 2: Position Level 3: Coefficient Current paths for the classification levels: "ETAM" > "1.1" > "240" "ETAM" > "1.2" > "250" [...] "Cadres" > "3.3" > "270" </classificationLevels>
<textContent> <title>${accord.title}</title> <content>${accord.content}</content> </textContent>
Pydantic / Zod)
const UpdateSchema = z.object({ type: z.literal("monthly"), classification: z.array(z.string()), monthlyAmount: z.number(), });
| OpenAI | Anthropic | Gemini | Mistral | |
|---|---|---|---|---|
| JSON valide | response_format:{ type: "json_object" } |
output_format(beta · schema requis) |
response_mime_type:"application/json" |
response_format:{ type: "json_object" }(idem OpenAI) |
| Schéma strict | response_format:{ type: "json_schema" }+ strict: true |
output_format+ JSON Schema · (beta) |
response_schema(OpenAPI) |
response_format:{ type: "json_schema" }+ strict: true · ou chat.parse() |
min/max, minLength, email... sont souvent ignorés"classification": ["Employé", "N2"], "monthlyAmount": 1829→
classification_levels| Coeff. | SMC |
|---|---|
| 210 | 1 820 € |
| 320 | 2 450 € |
[
{
"classification": ["Employés", "210"],
"monthlyAmount": 1820
},
{
"classification": ["Employés", "320"],
"monthlyAmount": 2450
}
]
→
classification_levels
Quand le prompt ne suffit plus
Validité structurelle du JSON
Conformité au schéma : types, champs requis, enums
Règles métier : bornes, formats, cohérences inter-champs
// 1. Appel initial — syntaxe + schéma gérés par promptAndValidateJSON let operations = await llmService.promptAndValidateJSON<MinimumWageUpdate[]>( systemPrompt, [{ role: 'user', content: initialPrompt }], z.array(minimumWageUpdateSchema), );
LLMService.promptAndValidateJSONasync promptAndValidateJSON<T>( systemPrompt: string, messages: LLMMessages, schema: z.ZodType<T>, maxRetries = 3, ): Promise<T> { for (let i = 0; i < maxRetries; i++) { const response = await this.callLLM(systemPrompt, messages); try { const json: unknown = JSON.parse(response); // ① syntaxe const parsed = schema.safeParse(json); // ② schéma if (parsed.success) return parsed.data; // ③ Échec Zod → feedback détaillé au LLM messages.push({ role: 'assistant', content: response }); messages.push({ role: 'user', content: `Zod error: ${z.prettifyError(parsed.error)}`, }); } catch (e) { if (!(e instanceof SyntaxError)) throw e; // ④ Échec JSON.parse → feedback brut + retry messages.push({ role: 'assistant', content: response }); messages.push({ role: 'user', content: 'Invalid JSON, please fix.' }); } } throw new MaxRetriesExceededError(); }
// 2. Historique conservé pour les itérations const messages: LLMMessages = [ { role: 'user', content: initialPrompt }, { role: 'assistant', content: JSON.stringify(operations) }, ]; // DB lookup const expectedClassificationLevel: Set<string> = await classificationRepo.listPaths(); // 3. Règles métier — boucle de retry jusqu'à maxRetries for (let i = 0; i < maxRetries; i++) { const uniqueClassificationPaths = new Set( operations.map(o => o.classification.join('>')), ); // ① chaque classification renvoyée est-elle connue ? const unknown = uniqueClassificationPaths.difference(expectedClassificationLevel); // ② une opération par classification attendue ? const missing = uniqueClassificationPaths.size < expectedClassificationLevel.size; if (unknown.size === 0 && !missing) break; if (unknown.size > 0) { messages.push({ /* "unknown classification paths: [...]" */ }); } if (missing) { messages.push({ /* feedback métier au LLM */ });
classification feedbackmessages.push({ role: 'user', content: `You returned ${uniqueClassificationPaths.size} operations but ${expectedClassificationLevel.size} classifications exist. Did you miss any level? [...] DON'T FORGET TO ANSWER IN JSON FORMAT`, });
} // Nouvelle passe — historique conservé operations = await llmService.promptAndValidateJSON(systemPrompt, messages, /* idem */); } return { operations, messages };
// du moins coûteux au plus coûteux ① JSON.parse(response) // sync, µs ② schema.safeParse(json) // sync, ms ③ await checkBusinessRules(parsed) // async, appel LLM / API / DB
const minimumWageUpdateSchema = z.object({ classification: z.array(z.string()).min(1).max(6), // 1 à 6 niveaux non vides minimumWage: z.discriminatedUnion('type', [ z.object({ type: z.literal('monthly'), monthlyAmount: z.number().positive(), // salaire mensuel > 0 }), // ... ]), applicabilityDate: z.object({ date: z.iso.date().optional(), // YYYY-MM-DD firstDayOfMonthAfterPublication: z.boolean().optional(), }), });
Non-déterministe ≠ non-testable
Ex. log produit par le scorer
[INFO] Processing CCN: 0018 (KALITEXT000051149719) [INFO] Extracted 31 parameter operations. [INFO] Found 31 existing parameters in the database. Starting comparison... [INFO] update: Ouvrier > 1 > 1 / 2025-03-05 / {"type":"monthly","monthlyAmount":1829} [INFO] update: Employé > 1 > 1 / 2025-03-05 / {"type":"monthly","monthlyAmount":1829} [...] [INFO] update: Cadre > IV > 1 / 2025-03-05 / {"type":"monthly","monthlyAmount":4848} [INFO] ✅ All operations are matching one parameter in the database 🎉
sur ~107 évaluations · ✓ succès · ⚠ à relire · ✗ échec
| Modèle | Résultats |
|---|---|
| Claude Sonnet 4.5reasoning 16k · sans validation |
73,4 %16,5 %10,1 %
|
| Claude Sonnet 4.5reasoning 16k |
82,2 %13,1 %4,7 %
|
| Claude Sonnet 4.6thinking: high |
77,6 %12,1 %10,3 %
|
| Claude Opus 4.7thinking: high · sans validation |
80,2 %13,2 %6,6 %
|
| Claude Opus 4.7thinking: high |
83,2 %13,1 %3,7 %
|
| GPT 5.4 |
74,1 %19,4 %6,5 %
|
| Pixtral Large2502 |
26,4 %10,9 %62,7 %
|
Des questions ?
villaren.fr/pubntalks/devoxx2026