What to Build

Over a single six-hour session, your group will build a safety evaluation scorecard for an LLM. Each group selects the domain and the metrics they want to include in their scorecard, then implements and runs those metrics against a provided model. For the purposes of this hackathon, you will be acting as if the general-purpose LLM we provide you was created for the purpose of your use-case. 

Groups will create a complete pitch for a safety scorecard — all the metrics that should be included and why, specific to your chosen domain and use-case — and implement them during the hackathon. The main goal is to grapple with three questions: What should we measure, Why should we measure it, and How can we measure it? Approach each from a model architecture standpoint and note the limitations inherent in every measurement. 

This is an evaluation exercise, not a training exercise. You are not improving, fine-tuning, or otherwise training the model in any way. The work is figuring out how to correctly evaluate the necessary safety dimensions for your use-case and recognize the limitations of those measurements given by the architecture of current-day LLMs.

What to Submit

• Project Overview
• Code on GitHub
• A table of your results
• Written Discussion
• Brief list of what each individual contributed
• ~5 min Video