Eval: plain LLM vs bpy-compass

A row passes only when the generated script runs in headless Blender, in the exact build matching the test case's target version (Blender 4.2.23 LTS, Blender 4.5.14 LTS, Blender 5.0.1), with a factory startup file and the test case's assert script, and the answer keeps its contract: WATCH OUT names every API change the test case expects, and SOURCES lists only Knowledge Base entries the agent actually read. Results are stored in Sanity and shown here unedited, including failures. Both contenders get the same model, settings, output contract and user prompt; the only difference is that the baseline has no Knowledge Base (no tools, no outline, no KB rules). Failed rows show the first failure reason.

model: z-ai/glm-5.3-flash
provider (pinned, no fallback): novita
temperature: 0
run: 2026-09-24T17-01-07-112Z

Provenance is read from the displayed run records, never from the server's current configuration.

#	Question	Target	Plain LLM	bpy-compass
1	Create a new mesh object from a single triangle (three vertices, one face), name the object 'Tri', and link it to the current scene so it is visible.	4.5	✅ pass	✅ pass
2	Deselect everything, then select only the object named 'Cube' and make it the active object.	4.5	✅ pass	✅ pass
3	Move the object named 'Cube' to location (1, 2, 3), then print the world-space position of each of its vertices by transforming the local coordinates with the object's world matrix.	4.5	✅ pass	✅ pass
4	Add a UV sphere, then add a Boolean modifier in UNION mode on the object named 'Cube' using the sphere as the cutter, and apply the modifier so the union is baked into Cube's mesh.	4.5	❌ fail WATCH OUT does not mention bpy.ops.<op>(override_dict, ...).	✅ pass
5	Create a material named 'Glow' whose Principled BSDF emits orange light with emission strength 5, and assign it to the object named 'Cube'.	4.5	✅ pass	✅ pass
6	Create a Geometry Nodes node group named 'MyGroup' with a Geometry input socket, a Float input socket named 'Scale', and a Geometry output socket.	4.5	✅ pass	✅ pass
7	Export only the selected objects to a Wavefront OBJ file. The output path is given by the environment variable BPY_OUT_OBJ (read it with os.environ). Select the object named 'Cube' first.	4.5	✅ pass	✅ pass
8	Make the mesh of the object named 'Cube' shade smooth while keeping edges sharp where the angle between faces is above 30 degrees (smooth by angle).	4.5	❌ fail Blender: AttributeError: 'Mesh' object has no attribute 'use_auto_smooth'	✅ pass
9	Set the render engine to EEVEE and the render resolution to 640 by 480 pixels.	4.2	✅ pass	✅ pass
10	Set the render engine to EEVEE and the render resolution to 640 by 480 pixels.	5.0	❌ fail Blender: TypeError: bpy_struct: item.attr = val: enum "BLENDER_EEVEE_NEXT" not found in ('BLENDER_EEVEE', 'BLENDER_WORKB	✅ pass
11	Add a second cube as a cutter, then add a Boolean modifier in DIFFERENCE mode on the object named 'Cube' using the fast (non-exact) solver. Do not apply it.	4.5	✅ pass	✅ pass
12	Add a second cube as a cutter, then add a Boolean modifier in DIFFERENCE mode on the object named 'Cube' using the fast (non-exact) solver. Do not apply it.	5.0	❌ fail WATCH OUT does not name the replacement for bpy.types.BooleanModifier.solver == "FAST" ("FLOAT").	❌ fail Blender: RuntimeError: Error: Object 'Cutter' cannot be selected because it is not in View Layer 'ViewLayer'!
Pass rate (of results available)			8/12	11/12