Mines Devlog 01

Welcome to generating reference data land

A lack of accurate data quickly became a limitation.

In order to improve the Vox Garbler model's boundary detection, there was little recourse beyond domain adaptation to the rather unique video game voice over files. In fact, several types of domain adaptation. This meant employing lots of great machine-learning techniques for post-training and fine-tuning: reinforcement learning.

In order to do that. it needs lots of domain-relevant samples that are "golden" or examples of how a boundary slice should work. So to democratize the process, I built a small multi-player server where players can progress and contribute to the project, obviously still a work in progress. It's hosted on my servers and leverages an internal API to the spiritbox (Vox Garbler) that allows players to hear samples of the newest variant of the model in action as they progress through the levels.

The high-level goal was pretty simple. Make slicing audio boundaries fun, and make annotating the slices at least palatable. The reference softwares for annotation for this stuff (PRAAT, Label Studio, CVAT, and others) thankfull were about as exciting to operate as a spreadsheet, at least to me. So although the work and outputs would be largely indentical, might as well pick up some new skills in cross-platform design and maybe, just maybe, make something fun: A browser game.

Making labeling less boring

Creating thousands of accurate audio slices, human-validated, with label information wasn't compelling, despite the promise and potential of Vox Garbler. This browser game had a few goals to accomplish:

• Actually be useful: support the research with golden data contributions.
• Playable in browser (mobile and desktop controls).
• Make progress contribute to model improvement
• Have a fun game loop: make it work your time
• Use no generated assets (craftpix for visual assets, music by me, audio samples: your favorite vintage games)
• Playable by many (so that I could rope in some friends to help)

So I started prototyping some displays to try to capture how I would like the layout to fit on a screen. Being browser-based I got my first taste of responsive design in the context of game and control design, which is to say, a nice splash of cold water.

My previous experience in game design had largely been limited to computer platforms with a mostly predictable landscape resolution of some kind. Porting experiences across platforms gave me a great deal of respect for the studios and outfits that make it their livelihood.

I had already done some waveform rendering and slicing work for FLASHBACK SAMPLER, so I just made a combo pack of waveforms, some generic slicing controls, and some primitive information panels. I knew I could convince a few pc gamers to take up a few session to test it out, so I pulled from their wheelhouse of Real-Time Strategy games to sort of mockup a control scheme and display mode that would fit in some of the late 90s early 2000s references (Starcraft, Age of Empires) that I knew they liked.

After several iterations I had a greybox layout with some basic controls, but I needed to find the gameplay.

Finding the fun, also defining the fun

I like to move to MVP and iteration quickly, so my prototypes were very barebones, and after validating that the controls could produce the labels in-line with the research, I needed to make the experience of doing it a bit more fun.

Enter the Game Design Document.

I am a big believer in documenting processes and decisions, if for no other reason than informing future you why you opted not to do something. I did a lightweight game design document with the high-level goals and a lot TBD for theme, aesthetic, core loop. One of things I realized, being a project steeped in nostalgia at the baseline is that I wanted an aesthetic that nodded to that and also would render cheaply and nicely browser. So 2D, pixel art.

Then I needed a premise, which lead me to the on the nose idea of prospecting the audio. You're essentially like a digsite retrieving artifacts, objects of value from within an expanse of a chasm (waveform sort of looks like a rocky cave). I did research for available assets, knowing I wanted this project to focus more on the engine and gameplay then spending a ton of asset development, and voila: mining.

There were a lot of available mining assets (it's a popular theme for 2D pixel art games, go figure), and it still fit within the gameplay idea. It's not strictly data mining, but it is mining for data. I could have gone way deeper with the lore and narrative, but I had already done THE GREAT BLEBIS recently-ish, so I figured I'd focus on gameplay instead.

Finally I needed the loop: why is anyone going to go and slice audio and label it over and over?

This one is still WIP, but at the high-level I wanted to make that the time-to-annotate could be in 5 to 15 seconds and have play sessions and levels that were forgiving for stop and restart gameplay.

Each play session is one VO line, with difficulty ramping and increasing the number of words per line. For each word a player makes 3 major decisions that all happen back to back. They play the audio to find the start boundary, place a worker there and rate the clarity boundary (fuzzy, alright, crystal) they place a second worker to signal the ending boundary and do the same rating, and then a prospector spawns and they rate the clip with the qualitative labels (speaker pitch, energy, intelligibility, etc.) that are necessary to feed the RLHF fine-tune model.

The more details they provide, the higher the score, and then I have a secret modifier applied to their score based on how true to the golden measurements they are, with some fancy calculations to rank them based on their likelihood to provide model-enhancing versus neutral versus model-detracting data. The score converts to currency and lets the user upgrade their worker cosmetics and few a vanity metrics.

There is a lot more planned so stay tuned!