Tired of spending countless hours tweaking prompts for large language models (LLMs), only to see marginal improvements? Enter DSPy, a groundbreaking framework that’s set to revolutionize how we work with LLMs. We’ll explore how DSPy can help you dramatically enhance your LLM prompting efficiency and effectiveness.

DSPy isn’t just another tool in the AI developer’s toolkit—it’s a paradigm shift. By separating the logic of your AI pipeline from the specifics of LM prompts and weights, DSPy introduces a level of abstraction that allows for unprecedented optimization. Whether you’re working with powerhouses like Claude 3.5 and GPT-4o or more compact models like Gamma and Llama3-8b, DSPy offers a systematic approach to maximizing their potential.

• Beyond hand-crafted prompts: Say goodbye to tedious prompt engineering. DSPy automates prompt optimization with sophisticated algorithms, letting you focus on the core logic of your AI system.
• Unlocking the full potential of LLMs: DSPy doesn’t just improve prompt quality, it optimizes the entire LLM pipeline. Achieve significant performance gains, even with challenging tasks.
• A new paradigm for AI development: DSPy shifts the focus from painstaking prompt tuning to building robust and adaptable AI systems.

We’ll look into how DSPy can help you achieve higher quality outputs, avoid common pitfalls, and streamline your workflow.

DSPy Basics

Using DSPy effectively for solving a new task is all about iterative machine learning with LLMs. You start with initial choices, which will likely be imperfect, and refine them over time. Here’s a brief overview:

• DSPy is all about iterative development, starting simple and adding complexity over time.
• DSPy’s optimizers automate the generation of optimized prompts, instructions, and even LM weights.
• The community Discord server is a great resource for support and guidance.

How does DSPy help with prompting?

Imagine you’re trying to teach an LLM (like GPT-4o). You want it to do a specific task, like summarizing a news article.

Traditional prompting is like giving a bunch of random commands, hoping one sticks. You might say “Summarize!”, “Tell me the key points!”, “Give me a short version”, etc. Some commands might work, some won’t, and it’s hard to know which ones are best.

DSPy is like having a special training tool for your language model. It takes your desired task (summarizing an article) and your examples (news articles with summaries you like), then it automatically figures out the best “commands” (prompts) to give your language model.

Here’s how it works:

1. Define your task: Tell DSPy what you want your language model to do (e.g., summarize an article).
2. Give it examples: Provide some examples of what you want (e.g., news articles and their summaries).
3. DSPy optimizes: DSPy figures out the best way to communicate your task to the language model, using the examples you provided. This might involve crafting new prompts, adding specific instructions, or even adjusting the model’s internal settings.

So, instead of guessing and trying different prompts, DSPy does the heavy lifting for you, making your language model much better at performing the task. It’s like having a personal trainer for your language model.

Example

Here is an example of how you might achieve this in DSPy. Here we used an external document to show somewhat of a RAG use case. I prefer using Groq since you can get free api credits as well as fast interface.

import dspy
# Replace 'document.txt' with your actual file path
document = './document.txt'
key = "my-key"
lm = dspy.GROQ(api_key=key)

with open(document, 'r') as file:
    document = file.read()

dspy.configure(lm=lm)
summarize = dspy.ChainOfThought('document -> summary')
response = summarize(document=document)
print(response)

print(response.summary)

Using MultiChainComparison

MultiChainComparison is a tool for fine-tuning language models (LLMs) by providing them with example completions. These completions act as a training set, guiding the LLM to understand and generate responses aligned with the desired output format and style. By feeding the LLM with relevant completions, you can improve its accuracy and consistency in generating answers for specific tasks. Say for example we wanted to use JavaScript or css with LLMs. You could for example pass examples of code and ask an LLM to do something similar or tailor the solution towards your own code.

class BasicQA(dspy.Signature):
    """Answer questions with short factoid answers."""
    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")

# Example completions generated by a model for reference
completions = [
    dspy.Prediction(rationale="What is a key advantage of using non-blocking methods.", answer="higher throughput"),
    dspy.Prediction(rationale="Give example to illustrate blocking and non-blocking calls.", answer="File System"),
    dspy.Prediction(rationale="Provide an example of a non-blocking asynchronous operation.", 
                    answer="```python\nimport asyncio\n\nasync def main():\n    async with asyncio.open_connection('www.example.com', 80) as (reader, writer):\n        writer.write('GET / HTTP/1.0\n\n'.encode())\n        await writer.drain()\n        data = await reader.read(100)\n        print(f'Received {data!r}')\n\nasyncio.run(main())\n```")
]

# Pass signature to MultiChainComparison module
compare_answers = dspy.MultiChainComparison(BasicQA)

# Call the MultiChainComparison on the completions
question = 'Provide an example of a non-blocking asynchronous operation in Python using asyncio. Use an api request example'
final_pred = compare_answers(completions, question=question)

print(f"Question: {question}")
print(f"Final Predicted Answer (after comparison): {final_pred.answer}")
print(f"Final Rationale: {final_pred.rationale}")

Question: Provide an example of a non-blocking asynchronous operation in Python using asyncio. Use an api request example
Final Predicted Answer (after comparison): `asyncio.get(aiohttp.ClientSession()).get('api.example.com')`
Final Rationale: Answer: `asyncio.get(aiohttp.ClientSession()).get('api.example.com')`

(This is an example of using the aiohttp library to make a non-blocking asynchronous API request using Python's asyncio library.)

Using TypedChainOfThought

from dspy import InputField, OutputField, Signature
from dspy.functional import TypedChainOfThought
from pydantic import BaseModel

class CodeOutput(BaseModel):
    code: str
    api_reference: str

class CodeSignature(Signature):
    function_description: str = InputField()
    solution: CodeOutput = OutputField()

# Here's where the fix happens:
def format_output(output):
    return f"```json\n{output.json()}\n```"

# Tell cot_predictor to use format_output
cot_predictor = TypedChainOfThought(CodeSignature)

prediction = cot_predictor(function_description="Write a function that adds two numbers.")

# Now the output should be correctly formatted
print(prediction)

Prediction(
    reasoning='{\n  "code": "def add_numbers(a, b): return a + b",\n  "api_reference": "https://docs.python.org/3/library/functions.html#sum"\n}',
    solution=CodeOutput(code='def add_numbers(a, b): return a + b', api_reference='https://docs.python.org/3/library/functions.html#sum')
)

In the realm of software development, producing structured and predictable outputs is crucial for maintaining code quality and improving developer productivity. The code snippet provided demonstrates how to leverage the DSPy library to achieve this, particularly when working with Large Language Models (LLMs). By structuring the output in JSON format, this approach not only enhances the readability and usability of the generated code but also facilitates better integration with other tools and systems.

Effective prompting is a vital skill for maximizing the potential of LLMs. With tools like prompt generators and frameworks like DSPy, developers can craft clear and concise prompts that yield high-quality, relevant, and consistent outputs. As LLMs evolve into more sophisticated agents, mastering the art of prompting will be essential for fully capitalizing on their capabilities and enhancing developer productivity.

The V-JEPA model, proposed by Yann LeCun, is a non-generative model that learns by predicting missing parts of a video in an abstract representation space. It’s pretty much learning by watching video. Unlike generative approaches, V-JEPA has the flexibility to discard unpredictable information, leading to improved training efficiency. It takes a self-supervised learning approach and is pre-trained entirely with unlabeled data, using labels only for task adaptation after pre-training. V-JEPA’s masking methodology involves blocking out portions of videos in both space and time to force the model to develop a deeper understanding of the scene. This approach allows the model to focus on higher-level conceptual information rather than details irrelevant for downstream tasks. V-JEPA’s efficiency lies in its ability to pre-train once without labeled data and then reuse parts of the model for various tasks efficiently.

Abstract Representations: Unlocking Object Interactions in Raw Video Data

At the heart of V-JEPA’s capabilities lies its unique ability to predict object interactions by learning abstract representations from raw video data. Through self-supervised learning, the model excels at predicting missing parts of video segments, gaining insights into latent features that define how elements in a scene interact.

Key Ideas:

• Non-Generative Model: V-JEPA doesn’t focus on reconstructing videos pixel by pixel. Instead, it learns to predict missing pieces of a video within a conceptual, or abstract, space of representations.
• Abstract Representation Space: Think of this space like a set of high-level features that describe important parts of a video (objects, actions, relationships). V-JEPA understands videos through these features, not just their raw pixels.
• Comparison with I-JEPA: V-JEPA is an extension of I-JEPA. Both systems aim to learn by comparing pieces of data in this abstract representation space, rather than directly comparing pixels.
• Flexibility and Efficiency: Since V-JEPA targets the important concepts rather than every single pixel, it can ignore irrelevant details. This makes it faster and more efficient during training. Data that’s unpredictable or noisy gets less focus.

Stability and Efficiency: Setting V-JEPA Apart

V-JEPA’s distinctive approach results in a more stable and efficient system, marking a departure from traditional AI models. Its adaptability and stability make it a standout choice for various applications, particularly in fields like robotics and self-driving cars, where understanding the environment is crucial for effective decision-making.

Versatility in Action: Adaptable Without Direct Parameter Fine-Tuning

One of V-JEPA’s key strengths lies in its versatility. The model serves as a foundation for various tasks and can be easily adapted without the need for direct parameter fine-tuning. This flexibility positions V-JEPA as a powerful tool for industries requiring quick and efficient implementation.

Future Prospects: Bridging the Gap to Natural Intelligence

While V-JEPA currently outperforms other models in video reasoning over several seconds, Meta’s research team is pushing boundaries further. The goal is to enhance the model’s time horizon and bridge the gap between JEPA and natural intelligence by exploring multimodal representations, indicating a commitment to continuous innovation.

Path Towards Advanced Machine Intelligence (AMI)

While V-JEPA has primarily focused on perceptual tasks related to video understanding, the next phase involves leveraging the model’s predictive abilities for planning and sequential decision-making. By training JEPA models on video data without extensive supervision, there is potential for these models to passively learn from visual inputs and quickly adapt to new tasks with minimal labeled data. This progression hints at the broader applications of V-JEPA in embodied AI systems and contextual AI assistants for augmented reality devices. The future prospects of V-JEPA lie in its ability to revolutionize machine intelligence by bridging the gap between human-like learning processes and efficient task completion across various domains.

Yann LeCun’s Endorsement: Advocating for the Promise of JEPA

Yann LeCun’s longstanding advocacy for JEPA raises intriguing questions about the technology’s relatively limited attention in the broader research community. With the success of V-JEPA, the promise of JEPA as a paradigm-shifting approach gains further credence, challenging established norms in AI research.

V-JEPA could potentially play a significant role in Llama 3’s advancements, offering enhanced video reasoning and understanding for improved user experiences. Having released Llama 2 not too long ago, it was seen as a massive advancement in open source AI. As rumors circulate about a potential July release, the integration of V-JEPA could signify a leap forward in Llama 3’s capabilities, providing users with a more sophisticated and intuitive AI experience.

Mistral has finally released their largest model to date, Mistral Large. It’s a cutting-edge language model with top-tier reasoning capabilities. It is proficient in English, French, Spanish, German, and Italian, excelling in tasks like text understanding, transformation, and code generation. Mistral Large ranks as the world’s second model available through an API, just after GPT-4. It offers a 32K tokens context window for precise information recall and supports function calling. Mistral AI has partnered with Microsoft to make their models available on Azure, providing access through Azure AI Studio and Azure Machine Learning. Mistral Large outperforms other models in multilingual tasks and excels in coding and math challenges. You can test the model yourself on their site.

Mistral Comparison

Mistral Large is a cutting-edge text generation model with top-tier reasoning capabilities. This comes just after Mistral released their 7B model late last year. They really seem to be moving fast, only shortly after, they released their 8x7B MoE model. This new Mistral Large model excels in complex multilingual tasks like text understanding, transformation, and code generation. It ranks as the world’s second-best model available through an API, just after GPT-4. Detailed benchmarks show its strong performance on various tasks, making it a powerful tool for developers and researchers.Key Features of Mistral Large:

1. Multilingual Proficiency: Fluent in English, French, Spanish, German, and Italian with a deep understanding of grammar and cultural nuances.
2. Large Context Window: With a 32K tokens context window, it can recall precise information from extensive documents.
3. Precise Instruction-Following: Enables developers to create custom moderation policies efficiently, as demonstrated in setting up system-level moderation for le Chat.
4. Function Calling Capability: In-built function calling ability combined with constrained output mode on la Plateforme facilitates application development and modernization of tech stacks at scale.

Side note, Mistral-Large is priced ~20% cheaper than GPT-4-Turbo. It’s a slightly weaker model as well. Curious to see how things play out and whether this is a worthwhile trade-off for many applications. Any interesting question will be if this 20% will be enough of a selling point?

Mistral Large Reasoning Capabilities

Mistral Large’s performance is compared to the top-leading LLM models on commonly used benchmarks, showcasing its powerful reasoning capabilities. The figure in question reports the performance of pre-trained models on standard benchmarks.

Mistral-Microsoft Partnership

The partnership between Microsoft and Mistral AI aims to accelerate AI innovation by leveraging Azure’s cutting-edge AI infrastructure to develop and deploy next-generation large language models (LLMs). Mistral AI’s flagship commercial model, Mistral Large, is now available on Azure AI, offering state-of-the-art reasoning and knowledge capabilities for various text-based applications. This collaboration focuses on supercomputing infrastructure support, scaling premium models through Models as a Service (MaaS), and exploring AI research and development opportunities, including training purpose-specific models for select customers like the European public sector. Here is a tweet by Microsoft’s CEO Satya Nadella.

This partnership between Microsoft and Mistral AI is particularly interesting, considering Microsoft’s significant investment and role as a computing provider to OpenAI. The collaboration brings together the strengths of both companies, with Mistral AI focusing on developing advanced large language models and Microsoft providing its powerful Azure AI infrastructure.

The previous two models by Mistral is seen as a positive example of open sourcing leading to commercial success with LLMs. However, some may feel conflicted due to the company’s strong pro open source stance and the potential influence of Microsoft after acquiring an interest. There is uncertainty about Mistral’s future open sourcing practices. It is suggested that if they stop, releasing the full weights of Miqu for community fine-tuning would be a good gesture, especially since Mixtral was disappointing in tuning.

Closing Thoughts

Another set of releases and, again, no AI has definitively beat GPT-4, which was in private beta well over a year ago Gemini Advanced is the only one of similar level, Mistral Large is below. On deck possibilities: Gemini 1.5 Ultra… and GPT-5. (Maybe Llama 3? Grok 2? Claude 3?). Sadly, they didn’t choose to open-source Mistral medium. Previously, Mistral AI offered open-source models like open-mistral-7B and open-mixtral-8x7b, aligning with their earlier promise of openness and contributing to the open-source community. Despite moving towards a more commercially oriented stance, Mistral AI still maintains elements of openness, allowing users to deploy and manage their models independently, supporting portability across clouds and infrastructures, and enabling extensive customizations and fine-tuning capacity.

Mistral had always maintained that they would retain the largest models for their own use. In all honesty, it would be a foolish decision if they were to simply replicate OpenAI’s approach. Although Mistral Large is a capable model, it falls short of GPT-4 in terms of intelligence and lacks the flexibility of Gemini 1.5 Pro. Therefore, it wouldn’t be logical to invest in the third-best option when there are alternatives available that offer superior intelligence and a larger context window.

Gemma is a new generation of open models from Google aimed at assisting developers and researchers in building AI responsibly. Gemma is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models. It includes model weights in two sizes: Gemma 2B and Gemma 7B, each released with pre-trained and instruction-tuned variants, you can try the model on Perplexity Labs now.

Google is releasing a Responsible Generative AI Toolkit to provide guidance and essential tools for creating safer AI applications with Gemma. This comes just a week after Google announced Gemini Pro 1.5. The models are available worldwide and can run on various platforms, including laptops, workstations, and Google Cloud. Gemma is designed with AI Principles at the forefront, and extensive evaluations have been conducted to ensure the safety and reliability of the models. The Responsible Generative AI Toolkit includes safety classification, debugging, and guidance tools.

Gemma also supports a wide variety of tools and systems, including multi-framework tools, cross-device compatibility, and cutting-edge hardware platforms. Furthermore, free credits for research and development are being offered to enable developers and researchers to start working with Gemma. The models are optimized for Google Cloud, and advanced customization is available with fully-managed Vertex AI tools or with self-managed GKE. Researchers can apply for Google Cloud credits of up to $500,000 to accelerate their projects. More information about Gemma and quickstart guides can be accessed on ai.google.dev/gemma.

Pushing open source

Google is no stranger to open source having literally releasing Transformers and many open source tools. Google is reinforcing its commitment to open-source innovation with the release of Gemma, creating a ripple of anticipation across the AI landscape. This isn’t Google’s first foray into open-source AI—its release of the transformative Transformers library laid a cornerstone for what has become a thriving ecosystem of machine learning models and applications. Gemma’s introduction into this space marks a substantial stride forward for open-source AI by not only providing a new suite of powerful tools but also by championing responsible AI practices.

Comparing to Other Models

A Redditor from the r/locallama community recently benchmarked Gemma 7B against other pre-trained 7Bs on the Open LLM Leaderboard, noting its standout performance boost in GSM8K, which highlights Gemma’s enhanced capabilities in mathematical reasoning.

Despite the specialized improvement in mathematical tasks with the 7B Gemma model, it doesn’t quite surpass the overall performance of the Mistral 7B. It seems to have a slight edge over Llama 7B, perhaps due to its architectural similarities to Llama 2, such as RMSNorm and RoPE. Notable differences include the use of a GeLU activation function instead of SiLU, a larger context window of 8k (twice that of Llama 2), wider intermediate layers, and fewer layers overall, at 28 compared to Llama’s 32. After hours of hands-on testing, it’s clear that Gemma 7B falls short of becoming a go-to over the established Mistral 7B models for broader applications.