9 Top AI Coders (Claude, ChatGPT, Grok & Co.) Tackle Java LSP Server: 0% Success, 100% 'Magical Sleep' Fails & Silent Crashes

A recent study tested the capabilities of nine advanced language models, including Claude 4.5 Opus, GLM 4.7, and Gemini 3.0 Flash, to determine their ability to program and integrate with external systems. The models were given a prompt to create a Java class that interacts with a language server, jdt-ls, and navigates code in a Maven project. Despite their advanced capabilities, all nine models failed to generate functional code, with errors ranging from logical and integration issues to failures in execution. The study's author concluded that these models, while useful for tasks such as code completion and explanation, lack a true understanding of system states, protocols, and long-term consequences of their generations. The findings highlight the gap between promotional claims and real-world capabilities of language models, emphasizing the need for a more nuanced conversation about their limitations and potential applications in software engineering.