dolphin-emu.org - Artigos da serie HLE Audiohttps://dolphin-emu.org/blog/series%23series-22018-08-10T02:23:22.850988+00:00Os últimos artigos na serie HLE AudioZinniaThe New Era of HLE Audio
2015-08-19T10:53:29+00:002018-08-10T02:23:11.170839+00:00MayImilaehttps://dolphin-emu.org/blog/authors/MayImilae/https://dolphin-emu.org/blog/2015/08/19/new-era-hle-audio/
<div style="max-width: 100%; margin: auto; margin-bottom: 1em;">
<img style="width: 100%" src="https://dolphin-emu.org/m/user/blog/newzeldahle/newzeldahleheader.jpg">
</div>
<p><br/></p>
<p>In early 2013, Dolphin had began its first steps in a new focus on accurate emulation. The 3.5 release represented a shift in the emulator's focus, and as such, saw great improvements in terms of compatibility and accuracy over the previous release. But one area that stuck out like a sore thumb during this era was the quality of High Level Emulation (HLE) audio. Hundreds of games suffered from crashes associated to audio, and thousands had significant problems, with missing effects, incorrect volume, and random bursts of noise. </p>
<p>The problems of HLE were systemic, deeply rooted problems within its design, and would require a complete rewrite in order to solve. Rewriting HLE audio was always a priority, but the daunting task to reverse engineer, implement, and test kept most developers away. So instead they pursued Low Level Emulation (LLE) to great success. LLE audio worked so well, the developers were able to avoid the mess of HLE and more or less just tell users to dump a GameCube/Wii DSP-ROM and use that instead. The problem with that option is performance: LLE audio is incredibly demanding, especially when the DSP is being strained by many sound effects.</p>
<p>This situation finally changed right after Dolphin 3.5 when delroth merged New-AX-HLE-GC, a rewrite of the most common <a href="https://en.wikipedia.org/wiki/Microcode">microcode (µcode)</a> for GameCube games, AX-GC. Thousands of bugs disappeared over night and stability increased greatly. While previously there was argument among developers that HLE audio bugs could be ignored because of the option for LLE, as tens of thousands of users finally experienced accurate audio for the first time it became apparent just how important HLE audio truly was. Later in the year, the AX-HLE rewrite was expanded to Wii games in a second cleanup. The ability for users to use HLE audio for most games instead of LLE audio resulted in one of the greatest performance increases in Dolphin's history!</p>
<p><br/></p>
<h3 id="the-non-ax-code-games">The Non-AX µcode Games<a class="headerlink" href="#the-non-ax-code-games" title="Permanent link">¶</a></h3>
<p>While over 99% of GameCube and Wii titles use the AX µcode, there are a small number of games that use a different µcode. The "Zelda µcode”, named after its exclusive use in Nintendo-created titles, represents only a tiny portion of the total games Dolphin can play; but those games are some of the most popular and interesting games on the GameCube and Wii.</p>
<p><br/></p>
<div style="width: 95%; margin: auto; margin-bottom: 1em;">
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=GameCube_Main_Menu">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/gamecubebios.jpg" title="GameCube Main Menu (BIOS)" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Luigi%27s_Mansion">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/LuigisMansionGC.jpg" title="Luigi's Mansion" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Animal_Crossing">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/AnimalCrossingGC.jpg" title="Animal Crossing" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Pikmin_(GC)">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/Pikmin.jpg" title="Pikmin" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Super_Mario_Sunshine">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/SuperMarioSunshineGC.jpg" title="Super Mario Sunshine" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=The_Legend_of_Zelda:_The_Wind_Waker">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/Windwaker.jpg" title="The Legend of Zelda: The Wind Waker" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=The_Legend_of_Zelda:_Collector%27s_Edition">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/collectorsedition.jpg" title="The Legend of Zelda: Collector's Edition" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Mario_Kart:_Double_Dash‼">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/doubledash.jpg" title="Mario Kart: Double Dash!!" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Pac-Man_Vs.">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/pacmanvs.jpg" title="Pac-Man Vs." />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Pikmin_2_(GC)">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/Pikmin2.jpg" title="Pikmin 2" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=The_Legend_of_Zelda:_Four_Swords_Adventures">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/fourswords.jpg" title="The Legend of Zelda: Four Swords Adventures" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Donkey_Kong_Jungle_Beat_%28GC%29">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/donkeykongjunglebeat.jpg" title="Donkey Kong: Jungle Beat" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=The_Legend_of_Zelda:_Twilight_Princess_(GC)">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/twilightprincessgc.jpg" title="The Legend of Zelda: Twilight Princess GC" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=The_Legend_of_Zelda:_Twilight_Princess_(Wii)">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/twilightprincesswii.jpg" title="The Legend of Zelda: Twilight Princess Wii" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Link%27s_Crossbow_Training">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/linkscrossbow.jpg" title="Link's Crossbow Training" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Super_Mario_Galaxy">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/supermariogalaxy.jpg" title="Super Mario Galaxy" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Pikmin_(Wii)">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/pikminwii.jpg" title="New Play Control! Pikmin" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Pikmin_2_(Wii)">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/pikmin2wii.jpg" title="New Play Control! Pikmin 2" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Donkey_Kong_Jungle_Beat_%28Wii%29">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/junglebeatwii.jpg" title="New Play Control! Donkey Kong Jungle Beat" />
</a>
</div>
<div style="max-width: 9.5%; min-width: 65px; float:left; text-align:center; padding: .5%;">
<a style="text-decoration: none;" href="https://wiki.dolphin-emu.org/index.php?title=Super_Mario_Galaxy_2">
<img style="max-width: 100%; border-style: none; margin-bottom: .5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/supermariogalaxy2.jpg" title="Super Mario Galaxy 2" />
</a>
</div>
</div>
<p><br clear="both"/>
<p style="max-width:90%; text-align:center; margin-top:.5em; margin-bottom:.5em;">The Zelda µcode games, in release order</p> </p>
<p><br/>
<a id="cuthere"></a></p>
<p>New-AX-HLE gave the greatest increase in audio accuracy and performance in the nearly twelve year history of the emulator, but all of the Zelda microcode games were completely unaffected. When Dolphin 4.0 released with New-AX-HLE as a core feature, it stuck out like a sore thumb that some of the most popular games on the consoles still had very lackluster HLE audio emulation. Just as before, users were forced to use LLE audio in order to get good audio emulation, at the expense of performance.</p>
<p>Obviously, the reason the AX-HLE rewrite was prioritized over Zelda-HLE was the sheer number of games. But there was a second reason: the old Zelda-HLE implementation was actually eons better than the old AX-HLE implementation. While AX-HLE was a mess of guesswork and wishful thinking, the Zelda-HLE microcode was more or less reverse-engineered from looking at the microcode binary. While there were some mistakes and implementation issues (it was designed around asynchronous audio, for instance), it at least approximated what the DSP was trying to do. AX-HLE, on the other hand, was one of the messiest pieces of patched together guesswork throughout the whole emulator.</p>
<p><br/></p>
<div class="highlight" style="background: #f8f8f8; width: 80%; margin: 0 auto;"><pre style="line-height: 125%; font-size: 75%;"><span style="color: #408080; font-style: italic">// TODO: WTF is going on here?!?</span>
<span style="color: #408080; font-style: italic">// Volume control (ramping)</span>
<span style="color: #008000; font-weight: bold">static</span> <span style="color: #008000; font-weight: bold">inline</span> u16 <span style="color: #0000FF">ADPCM_Vol</span>(u16 vol, u16 delta)
{
<span style="color: #B00040">int</span> x <span style="color: #666666">=</span> vol;
<span style="color: #008000; font-weight: bold">if</span> (delta <span style="color: #666666">&&</span> delta <span style="color: #666666"><</span> <span style="color: #666666">0x5000</span>)
x <span style="color: #666666">+=</span> delta <span style="color: #666666">*</span> <span style="color: #666666">20</span> <span style="color: #666666">*</span> <span style="color: #666666">8</span>; <span style="color: #408080; font-style: italic">// unsure what the right step is</span>
<span style="color: #408080; font-style: italic">//x += 1 * 20 * 8;</span>
<span style="color: #008000; font-weight: bold">else</span> <span style="color: #008000; font-weight: bold">if</span> (delta <span style="color: #666666">&&</span> delta <span style="color: #666666">></span> <span style="color: #666666">0x5000</span>)
<span style="color: #408080; font-style: italic">//x -= (0x10000 - delta); // this is to small, it's often 1</span>
x <span style="color: #666666">-=</span> (<span style="color: #666666">0x10000</span> <span style="color: #666666">-</span> delta) <span style="color: #666666">*</span> <span style="color: #666666">20</span> <span style="color: #666666">*</span> <span style="color: #666666">16</span>; <span style="color: #408080; font-style: italic">// if this was 20 * 8 the sounds in Fire Emblem and Paper Mario</span>
<span style="color: #408080; font-style: italic">// did not have time to go to zero before the were closed</span>
<span style="color: #408080; font-style: italic">//x -= 1 * 20 * 16;</span>
<span style="color: #408080; font-style: italic">// make lower limits</span>
<span style="color: #008000; font-weight: bold">if</span> (x <span style="color: #666666"><</span> <span style="color: #666666">0</span>) x <span style="color: #666666">=</span> <span style="color: #666666">0</span>;
<span style="color: #408080; font-style: italic">//if (pb.mixer_control < 1000 && x < pb.mixer_control) x = pb.mixer_control; // does this make</span>
<span style="color: #408080; font-style: italic">// any sense?</span>
<span style="color: #408080; font-style: italic">// make upper limits</span>
<span style="color: #408080; font-style: italic">//if (mixer_control > 1000 && x > mixer_control) x = mixer_control; // maybe mixer_control also</span>
<span style="color: #408080; font-style: italic">// has a volume target?</span>
<span style="color: #408080; font-style: italic">//if (x >= 0x7fff) x = 0x7fff; // this seems a little high</span>
<span style="color: #408080; font-style: italic">//if (x >= 0x4e20) x = 0x4e20; // add a definitive limit at 20 000</span>
<span style="color: #008000; font-weight: bold">if</span> (x <span style="color: #666666">>=</span> <span style="color: #666666">0x8000</span>) x <span style="color: #666666">=</span> <span style="color: #666666">0x8000</span>; <span style="color: #408080; font-style: italic">// clamp to 32768;</span>
<span style="color: #008000; font-weight: bold">return</span> x; <span style="color: #408080; font-style: italic">// update volume</span>
}
</pre></div>
<p><p style="max-width:90%; text-align:center; margin-top:.5em; margin-bottom:.5em;">Part of the old AX HLE implementation. The actual correct implementation of this function would be: <br/> <tt style="white-space: nowrap;">return vol + delta;</tt></p> </p>
<p><br/></p>
<p>By being more competent and used in far less games, Zelda-HLE ended up falling to delroth's backburner. But better it may have been, therewere still some incredibly frustrating issues throughout Zelda-HLE. In any of the games, volume balancing was almost always guaranteed to be wrong, effects wouldn't play, music would get desynced or stop altogether, and because of how the GameCube/Wii can use the DSP for timing events, sometimes games would outright hang when an audio event was missed or played at the wrong time. Many people probably don't remember the Wind Waker Item Hang cheatcode that was included with Dolphin for <em>years</em>, but it was made to work around this exact kind of issue!</p>
<p><br/></p>
<h3 id="rewriting-zelda-hle"><strong>Rewriting Zelda-HLE</strong><a class="headerlink" href="#rewriting-zelda-hle" title="Permanent link">¶</a></h3>
<p>Spurred on by Zelda-HLE being broken in Dolphin's updated audio dumping, magumagu started working on some <a href="https://github.com/dolphin-emu/dolphin/pull/214">Zelda-HLE cleanups</a> in March 2014. The main goal was to remove the asynchronous audio processing so that it could be dumped correctly along with the rest of HLE audio. On top of fixing that, magumagu's changes greatly increased the stability of the Zelda-HLE games. Without any further work, music looping bugs in games like Mario Kart: Double Dash!! and Super Mario Galaxy 1/2 disappeared.</p>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219434558%3Fsecret_token%3Ds-fX9h0"></iframe>
<p>Not only is the music too quiet, it outright disappears!</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219434560%3Fsecret_token%3Ds-MH6dk"></iframe>
<p>Music looping and playing properly in the finished implementation.</p>
</div>
<br clear="both"/>
</div>
<p>The effects were farther reaching in the long term. The Legend of Zelda: The Wind Waker and Twilight Princess were far less likely to hang on transitions while Super Mario Galaxy would no longer hang after the Grand Star cutscenes even during slowdown. Because this change fixed so much, it renewed interest in improving the quality of Zelda µcode HLE in Dolphin. More cleanups followed the first one, and while they didn't have the same end-effect, they did at least make things a little bit nicer in one of the darker spots of <a href="https://github.com/dolphin-emu/dolphin/pull/231">Dolphin's</a> <a href="https://github.com/dolphin-emu/dolphin/pull/233">codebase</a>.</p>
<p>Despite the improvements, it was very clear to everyone that just cleaning up the existing code would not be enough to bring Zelda-HLE up to par with the rest of audio emulation in Dolphin.</p>
<p><br/></p>
<h4 id="the-main-issues">The Main Issues<a class="headerlink" href="#the-main-issues" title="Permanent link">¶</a></h4>
<ul>
<li>There were readability issues that prevented it from being easily changed. When Zelda-HLE was originally written, the code was directly transcribed from its assembly code to C++. Little effort was put into understanding what the code was doing; it was akin to someone translating a book from one language to another without knowing anything about one of the languages. In fact, understanding the original assembly code was sometimes easier than looking at the pre-existing implementation!</li>
<li>Features were outright broken, such as audio generated on the DSP. It didn't seem to work at all in the old implementation. This was very apparent throughout several games across many µcode versions.</li>
</ul>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219464793%3Fsecret_token%3Ds-UY8Pn"></iframe>
<p>Super Mario Sunshine Generated Audio: Before rewrite.</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219464803%3Fsecret_token%3Ds-33pok"></iframe>
<p>Super Mario Sunshine Generated Audio: After Rewrite.</p>
</div>
<br clear="both"/>
</div>
<ul>
<li>There were a lot of things hard-coded to how Wind Waker, the game the implementation was based upon, worked. For example, games can tell the µcode what channel a sound should be mixed to. However, the old Zelda-HLE implementation would just ignore this completely and always assume that the first channel was the left audio channel, and the second the right audio channel, simply because this is how Wind Waker did it. As expected, this was a bad idea. Rather than fixing the issue, users were recommended to use the MONO audio output in these games.</li>
</ul>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219321741%3Fsecret_token%3Ds-ULdy0"></iframe>
<p>Four Swords Adventures in stereo on the old implementation.</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219321735%3Fsecret_token%3Ds-Jr8cK"></iframe>
<p>Once finished, the new implementation actually listens to the game, fixing the issue.</p>
</div>
<br clear="both"/>
</div>
<ul>
<li>Documentation was not adequate for a rewrite; meaning everything would have to be reverse engineered from the ground up.</li>
<li>Finally, a lot of features were just completely missing. However, the original developers realized this, and instead of properly implementing the features, they were worked around in completely wild ways. As an example, of the 15 audio related commands in the Zelda µcode, the old HLE implementation only implemented 2.</li>
</ul>
<p><br/></p>
<h3 id="tackling-the-behemoth">Tackling the Behemoth<a class="headerlink" href="#tackling-the-behemoth" title="Permanent link">¶</a></h3>
<p>On April 1st, 2014, delroth started his long project to rewrite Zelda-HLE from scratch. Almost all of the pre-existing code was torn down, and everything would be rebuilt from a clean slate, based on extensive analysis of the Zelda µcode behavior. The focus was put on a single game at first: The Legend of Zelda: The Wind Waker. The same game that served as the seed for the first implementation was the initial focus of the second one, nearly a decade later.</p>
<p>Progress was expectedly slow at first. Rather than jumping head first and attempting to make Wind Waker play sounds, delroth focused on reverse engineering and actually understanding what Wind Waker was attempting to do. By reusing the tools that were written for <a href="https://github.com/dolphin-emu/gcdsp-ida">delroth's AX HLE reverse engineering</a>, the Zelda TWW UCode was documented and re-discovered from scratch.</p>
<p><br/></p>
<div style="max-width: 80%; margin: 0 auto; text-align:center; margin-bottom: 1em;">
<a style="text-decoration: none;" href="https://dolphin-emu.org/m/user/blog/newzeldahle/reverseengineer.png">
<img style="width: 100%; margin-bottom:.5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/reverseengineer_thumb.png" alt="ReverseEngineer">
</a>
<p>This is what it looks like to reverse engineer a DSP</p>
</div>
<p><br/></p>
<p>For several months, not even The Legend of Zelda: The Wind Waker was booting in the New-Zelda-HLE branch. This was to be expected; rather than focusing on making the game play first and foremost, delroth was going through the DSP and implementing things as accurately as he could. While the end-goal was perfect audio, rushing toward that goal could lead to another case of the old implementation riddled with holes and flaws. As the months rolled on, Wind Waker finally took its first steps into playability. It didn't immediately crash due to the incomplete audio implementation! By December 2014, Wind Waker would even play some sounds! Barely. Some of the earliest examples of audio sound like something straight out of Dolphin 1.0.3.</p>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219873922%3Fsecret_token%3Ds-N0AlO"></iframe>
<p>Broken Resampling early on in the new implementation.</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219873938%3Fsecret_token%3Ds-eVH1q"></iframe>
<p>Broken Audio Loop early on in the new implementation. Outright creepy!</p>
</div>
<br clear="both"/>
</div>
<p>By this point, the implementation was actually fairly complete. Testers started going through The Wind Waker to try and find audio that sounded incorrect and samples that were not yet implemented. Several instruments in Wind Waker's miniboss theme triggered unknown samples which made it pretty much unplayable with all the popups! As things went on further and samples were implemented, the new implementation surpassed the old one. Even this early on, the fruit's of the reverse engineering labor were apparent.</p>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219292819%3Fsecret_token%3Ds-WDaap"></iframe>
<p>The beeping plaguing many Zelda-HLE games appeared in Wind Waker too!</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219292785%3Fsecret_token%3Ds-Glibz"></iframe>
<p>Even at an early state, it played correctly in the reimplementation.</p>
</div>
<br clear="both"/>
</div>
<p>Emulating new features that the old implementation ignored was a challenge as well. Reverb and echo in Wind Waker weren't particularly complicated, but they did put up some fight before they were properly emulated.</p>
<p><br/></p>
<iframe style="display:block; max-width: 600px; min-width:40%; margin: 0 auto;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219873949%3Fsecret_token%3Ds-4G91w"></iframe>
<p style="max-width:90%; text-align:center; margin-top:.5em; margin-bottom:.5em;">An early attempt at emulating echo and reverb</p>
<p><br/></p>
<p>Testers began digging for more and more subtle bugs as Wind Waker approached perfection. For some, <a href="https://code.google.com/p/dolphin-emu/issues/detail?id=7971">even LLE audio was doubted and console comparisons were brought in for comparison.</a> The fact that testers were arguing over very minor issues just showed how far Wind Waker had come in such a little amount of time. As such, delroth moved on to the next step of New-Zelda-HLE. As the <a href="https://github.com/delroth/dolphin/commit/6c61ee6278a7a2936a92087841a973bffa95a624">Zelda µcode has multiple versions</a>, focus was transferred toward implementing those iterations and figuring out what the differences were. The games released after Wind Waker had the µcode change a lot less than the earlier versions of the µcode, so there wasn't a huge delay between adding support and games playing. As a bonus, the reimplementation had fixed many of the bugs <em>before</em> support was even added to the other µcode!</p>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219431916%3Fsecret_token%3Ds-FLUeZ"></iframe>
<p>Reverb was non-existent in the old implementation.</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219431911%3Fsecret_token%3Ds-XOnhk"></iframe>
<p>in Four Swords Adventures, it was fixed before the µcode was implemented!</p>
</div>
<br clear="both"/>
</div>
<p>The differences between the µcodes actually revealed some interesting tidbits into how the old implementation was failing as well. For example, all µcode versions after The Wind Waker automatically double the positional audio volume to compensate for audio volume lost in the position computations. One of the most notable bugs throughout the old Zelda-HLE implementation was the fact that a lot of the audio was too quiet. Upon checking into things, this bug only happened in games released <em>after</em> Wind Waker!</p>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219450505%3Fsecret_token%3Ds-A4YBS"></iframe>
<p>Mario Kart: Double Dash!! - Position Audio Before.</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219450499%3Fsecret_token%3Ds-ROZlc"></iframe>
<p>Mario Kart: Double Dash!! - Positional Audio After.</p>
</div>
<br clear="both"/>
</div>
<p><br/></p>
<div style="max-width: 80%; margin: 0 auto; text-align:center; margin-bottom: 1em;">
<a style="text-decoration: none;" href="https://dolphin-emu.org/m/user/blog/newzeldahle/multiplyby2.png">
<img style="width: 100%; margin-bottom:.5em;" src="https://dolphin-emu.org/m/user/blog/newzeldahle/multiplyby2.png" alt="Multiplyby2">
</a>
<p>The later µcode used in Double Dash on the left automatically multiplies positional audio by two so it isn't too quiet. In the older µcode, the game had to handle this manually</p>
</div>
<p><br/></p>
<p>Adding support for each µcode after Wind Waker's didn't take very long. Even Wii support wasn't too big of a deal; The biggest difference of the Wii version of the µcode was that it used MRAM to emulate ARAM because the Wii doesn't have a dedicated ARAM chip like the GameCube. Beyond that, the implementation is very similar to the last implementation of the µcode for the GameCube. Interestingly enough, Wii Zelda µcode suffered from the same problems as the GameCube counterpart.</p>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219320410%3Fsecret_token%3Ds-DcSfS"></iframe>
<p>Positional Audio in Super Mario Galaxy - Before.</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219320427%3Fsecret_token%3Ds-LCKrb"></iframe>
<p>Positional Audio in Super Mario Galaxy - After.</p>
</div>
<br clear="both"/>
</div>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219464726%3Fsecret_token%3Ds-i2CE6"></iframe>
<p>Twilight Princess Wii suffered from the same bugs as its GameCube counterpart.</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219464752%3Fsecret_token%3Ds-7QAcV"></iframe>
<p>Fixing the bugs in one of them fixed the bugs in both them!</p>
</div>
<br clear="both"/>
</div>
<p>With all of the post Wind Waker µcodes taken care of, delroth's attention turned to implementing the older µcodes. Support for the older versions of the µcode was actually harder than the newer ones as they had a lot more meaningful differences. For instance, while Super Mario Sunshine shared a lot of the same bugs with Wind Waker, it used an entirely different way of syncing the audio. Implementing the per-frame synchronization used in Super Mario Sunshine and Pikmin 1 (PAL) was necessary to accurately emulate their µcode.</p>
<p>The µcode versions prior to Super Mario Sunshine are known as the Zelda µcode "Light". It has several differences between it and the full µcode; mostly that it has less features overall. Pikmin (NTSC), Luigi's Mansion and Animal Crossing all fell into this group. Coincidentally, they were among the most stable of the Zelda-HLE games on the old implementation. Another special thing in Pikmin and Animal Crossing is specialized crypto features for connecting to the GBA over a link cable. The old implementation didn't actually try to emulate these features, causing the NES games in Animal Crossing to be unplayable when transferred to a GBA emulator. Pikmin and Luigi's Mansion suffered from lots of strange issues even without that problem; and while none of the synchronization in these games was strenuous enough to cause hangs or crashes, sometimes the old implementation could do weird things.</p>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219291544%3Fsecret_token%3Ds-2mXST"></iframe>
<p>WARNING: VERY LOUD.</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219291546%3Fsecret_token%3Ds-Z5im3"></iframe>
<p>After being reimplemented, things sound much nicer.</p>
</div>
<br clear="both"/>
</div>
<p><br/></p>
<p>With all of the work put into New-Zelda-HLE, most of the games should output perfect or near perfect audio. There are a few minor problems that crop out in certain versions of the µcode, but none of them are going cause havoc like some of the old bugs. One of the biggest hurdles left is to fully implement support for the IPL (<a href="https://wiki.dolphin-emu.org/index.php?title=GameCube_Main_Menu">GameCube BIOS</a>), which uses the earliest version of the Zelda µcode. In the old implementation of Zelda-HLE implementation, the NTSC IPL would refuse to emit anything else than loud beeps. It turns out that the NTSC IPL's early version of the Zelda µcode version has smaller voice parameter blocks, so the parameters are not always where the code would expect them to be. While full support was not added, to make things more bearable, some basic support for the NTSC-IPL was implemented. Now testers won't have to worry about blowing out their ears, even if the sounds aren't entirely right yet.</p>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219288477%3Fsecret_token%3Ds-wKOez"></iframe>
<p>Beware: loud beep!</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F219288484%3Fsecret_token%3Ds-LsgPE"></iframe>
<p>The NTSC-IPL still has a ways to go before it's perfect.</p>
</div>
<br clear="both"/>
</div>
<p><br/></p>
<h3 id="future">Future<a class="headerlink" href="#future" title="Permanent link">¶</a></h3>
<p>With all of the systemic issues finally concluded, DSP-HLE has entered a new era. In the vast majority of games, Dolphin's HLE audio implementation is now at parity with the LLE audio. There are still some known bugs out there that are unique to HLE audio, but these bugs can be handled <em>as single bugs</em>, without the massive rewrites and labor that chases developers away! Now DSP-HLE is nearly perfect, and issues can be solved progressively over time, as an emulator should be.</p>
<ul>
<li><strong>Mario Kart: Double Dash!! Echo</strong>: Unlike the other Zelda µcode games, echo still doesn't seem to work right in Mario Kart: Double Dash!! It's hard to hear over the music, but there is definitely a difference between HLE and LLE audio.</li>
<li><strong>µcode Switching for GBA-Link</strong>: For some games, HLE Audio will cut out randomly when using GBA-Link. This has been recorded in at least one GBA<->GCN game: <a href="https://wiki.dolphin-emu.org/index.php?title=Mega_Man_X:_Command_Mission">Mega Man X: Command Mission</a>.</li>
<li><strong>Star Wars Rogue Squadron <a href="https://wiki.dolphin-emu.org/index.php?title=Star_Wars_Rogue_Squadron_II:_Rogue_Leader">II</a>/<a href="https://wiki.dolphin-emu.org/index.php?title=Star_Wars_Rogue_Squadron_III:_Rebel_Strike">III</a></strong>: These games have severe problems in HLE audio that need to be fixed in the future.</li>
<li><s><strong>48KHz Games</strong>: This is a bit of a confusing bug. Apparently some GameCube games could use 48000 Hz audio instead of the standard 32000 Hz, and HLE audio doesn't handle that properly.</s> -> <a href="https://dolphin-emu.org/download/dev/869c5d7a8c31e7cf7d191f1d0beda8b5c056dc9d/">Unexpectedly fixed hours after the article was posted.</a></li>
<li><s><strong>Wii FMV Audio</strong>: Another strange one that probably comes down to a timing issue. In HLE audio, certain FMVs seem to not play audio some of the time.</s> -> <a href="https://dolphin-emu.org/download/dev/869c5d7a8c31e7cf7d191f1d0beda8b5c056dc9d/">Unexpectedly fixed hours after the article was posted.</a></li>
<li><strong><a href="https://wiki.dolphin-emu.org/index.php?title=Pac-Man_World_Rally">Pac-Man World Rally</a></strong>: No one has any idea why the sounds play so loud in this weird namco racer.</li>
</ul>
<p><br/></p>
The Rise of HLE Audio
2014-11-12T04:51:52+00:002018-08-10T02:23:22.850988+00:00JMC47https://dolphin-emu.org/blog/authors/JMC47/https://dolphin-emu.org/blog/2014/11/12/the-rise-of-hle-audio/
<div style="max-width: 100%; margin: auto auto 1em;"><img style="width: 100%" src="https://dolphin-emu.org/m/user/blog/rise-of-hle-audio/hleaudioheader.jpg" alt="hleaudioheader.jpg"></div>
<p><br/></p>
<p>Like any artistic medium, games are emotional experiences filled with joy, sadness, frustration, and more. Special moments can bring tears, cause shouting, or even screams. But imagine during one of those emotional highs if the audio simply died, and the game continued onward in a deafening silence before eventually freezing. That kind of marred experience was commonplace under Dolphin's old way of handling HLE audio.</p>
<p>The users of today don't have to face those problems; modern High Level Emulation (HLE) audio is both fast and accurate, mostly matching the conventions and output of its high-accuracy counterpart, Low Level Emulation (LLE) audio. This change in behavior is thanks to the work by <a href="https://github.com/delroth">delroth</a> and <a href="https://github.com/magumagu">magumagu</a> that corrected the main fundamental flaw that afflicted old HLE audio. Fixing this defect and cleaning up the audio brought a multitude of features and fixes to the emulator that helped bring us into this modern era of speedy accuracy.</p>
<p><br/></p>
<div class="text-center"><iframe width="480" height="270" src="//www.youtube.com/embed/uHPy7ZUFleE?" frameborder="0" allowfullscreen></iframe></div>
<p><br/></p>
<p><a id="cuthere"></a></p>
<p>While the end result is great, the journey was anything but smooth, filled with uncovered bugs, regressions, controversy and more! Thanks to the help of contributors <a href="https://github.com/degasus">degasus</a>, <a href="https://github.com/booto">booto</a>, <a href="https://github.com/skidau">skidau</a>, <a href="https://github.com/konpie">konpie</a> and <a href="https://github.com/phire">phire</a> Dolphin has nearly perfect audio emulation.</p>
<h3 id="the-beginning-of-audio-in-dolphin"><strong>The Beginning of Audio in Dolphin</strong><a class="headerlink" href="#the-beginning-of-audio-in-dolphin" title="Permanent link">¶</a></h3>
<p>As the first GameCube emulator to boot commercial games, there was a time when having <strong>any</strong> audio emulation was an achievement! The GameCube's Digital Signal Processor (DSP) was effectively a black box to the early developers of Dolphin. As such, they did what any normal person would do and poked it with data and analyzed what it put out. While not the easiest or most efficient way to work, <a href="https://github.com/hrydgard/">ector (hrydgard)</a>, <a href="https://github.com/StapleButter">StapleButter</a> and many others brought the DSP-HLE plugin into Dolphin before the 1.0 releases. The original DSP-HLE mostly just played music in games that streamed audio, but was capable of <em>sometimes</em> playing sounds and other music.</p>
<p>This HLE audio core continued to evolve along with the rest of the emulator. By the time Dolphin hit open source, games could play music and sounds, even if things didn't sound exactly right. With fresh life breathed into Dolphin from the move to open source, DSP-HLE continued to evolve, providing the source of primary audio for the emulator for year after year. But that evolution would slow to a crawl in its final years, even as glaring issues such as crashing, hangs, and audio stopping plagued the emulator. The truth was that the original HLE audio implementation handled things in a broken way that could not be reconciled.</p>
<p><br/></p>
<iframe style="display:block; max-width: 300px; min-width:40%; margin: 0 auto;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/167463162%3Fsecret_token%3Ds-8LprK"></iframe>
<p><br/></p>
<h3 id="a-fundamental-flaw"><strong>A Fundamental Flaw</strong><a class="headerlink" href="#a-fundamental-flaw" title="Permanent link">¶</a></h3>
<p>To understand this flaw, one must first understand how the GameCube/Wii's DSP operates. The audio processing code runs on the DSP, which is a secondary processor engineered to be fast at auxiliary tasks like audio mixing. Every 5ms, the CPU sends the DSP a list of data blocks about sounds to process. Each of these blocks contain information like “location of the sound data in memory”, “volume”, “looping or oneshot”, as well as a list of commands for the DSP to run. When the DSP is done running the commands, it loads the mixed sound samples into RAM and sends an interrupt to the CPU to signal that it is done, then the CPU outputs the audio and goes to the next 5ms block.</p>
<p>The old HLE code was unable to do this properly, thanks to an asynchronous processing method. The 5ms interrupts weren't guaranteed to happen after 5ms* of CPU time. Old HLE audio could play these samples with up to 100ms of variance. In a worst case scenario, twenty updates could happen instantly, followed by 100ms of total silence or messed up audio. And if the emulator wasn't running full speed, things could get even more confusing as the DSP would plow ahead as if everything was fine. These behaviors resulted in many of the numerous problems that plagued DSP-HLE for years.</p>
<p>*Wii games and a small minority of GameCube games use 3ms interrupts.</p>
<p><br/></p>
<div style="max-width: 70%; margin: 0 auto 1em;">
<img style="width: 100%; margin-bottom: 0.5em;" src="https://dolphin-emu.org/m/user/blog/rise-of-hle-audio/asyncvssync.png" alt="asyncvssync.png">
<p style="text-align:center;">The left side shows how asynchronous audio handles DSP interrupts, while the right side shows how it would work on a typical game run on the GameCube.</p>
</div>
<p><br/></p>
<h4 id="aux-processing"><strong>AUX Processing</strong><a class="headerlink" href="#aux-processing" title="Permanent link">¶</a></h4>
<p>AUX processing is a feature of the DSP program that allows the DSP to apply audio effects on the sounds, such as echo, reverb, chorus, etc. AUX uses DSP commands to download and upload data to the CPU in order for the CPU to process the AUX effect while the DSP is working on something else.</p>
<p>This simply cannot be done with asynchronous audio processing. Not only is DSP<->CPU communication impossible (they don’t run in sync anymore), but all of the AUX code is designed to handle a fixed number of samples. This matches how many samples the DSP handles at a time (32×5, for 5ms at 32KHz). Without AUX processing almost all audio effects are lost, and games sound considerably inferior.</p>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F67619519"></iframe>
<p>Tales of Symphonia without AUX Processing</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F67619338"></iframe>
<p>Tales of Symphonia with AUX Processing</p>
</div>
<br clear="both"/>
</div>
<p><br/></p>
<h4 id="games-requiring-tight-timing"><strong>Games Requiring Tight Timing</strong><a class="headerlink" href="#games-requiring-tight-timing" title="Permanent link">¶</a></h4>
<p>Since the DSP always operates at set intervals, game developers used this to time when sounds should start and stop, especially when composing music from samples. Of course, 5ms accuracy is often not enough for music; so the DSP provides a feature called "sub-5ms updates" which can specify some slight changes on the sound data blocks for each millisecond. Failing to emulate this nifty feature can cause games to get exceedingly confused as the DSP doesn't make the changes it's expecting. Once the timings are off, anything can happen: audio garbling, instruments out of sync, popping/cracking, along with many, many more possible problems. It's not pretty.</p>
<p><br/></p>
<iframe style="display:block; max-width: 300px; min-width:40%; margin: 0 auto;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=http%3A%2F%2Fapi.soundcloud.com%2Ftracks%2F100518911"></iframe>
<p><br/></p>
<h4 id="games-using-the-sound-processing-status-to-trigger-events"><strong>Games Using the Sound Processing Status to Trigger Events</strong><a class="headerlink" href="#games-using-the-sound-processing-status-to-trigger-events" title="Permanent link">¶</a></h4>
<div class="container" style="margin-top:1em; margin-left:2em; margin-bottom:1em; max-width:38%; float:right;">
<a href="https://dolphin-emu.org/m/user/blog/rise-of-hle-audio/doubledash.jpg">
<img style="max-width: 100%; margin-bottom:.5em;" src="https://dolphin-emu.org/m/user/blog/rise-of-hle-audio/doubledash_thumb.jpg" />
<p style="text-align:center; font-size:12px;">Double Dash is notorious for music that just quits.</p>
</a>
</div>
<p>One of the more infamous problems in Dolphin emulation history was the fact that <a href="https://wiki.dolphin-emu.org/index.php?title=The_Legend_of_Zelda:_The_Wind_Waker">The Legend of Zelda: The Wind Waker</a> would hang a lot when collecting items. It was so common, cheat codes were added to the GameINI to work around the hang. Link picks up an item, the jingle plays, and then the game just sits there waiting, not frozen, not crashed, still emulating, just... waiting. What has happened in this case?</p>
<p>In a basic sense, some games wait for sounds to be completed before they will continue. When the sound processing is done asynchronously, the game may miss the moment when the sound has finished playing (because it went too fast compared to the emulated CPU speed) and just freeze. This tended to affect the Zelda µcode games more than anything. This separate µcode posed extra challenges for Dolphin due to tighter timings and tons of features unique to itself. In HLE audio, it's completely isolated from the handling of the AX µcode.</p>
<p>There were a lot of problems unique to these games because of this, such as <a href="http://code.google.com/p/dolphin-emu/issues/detail?id=5746">issues with certain sounds</a> and tons of music hangs. <a href="https://wiki.dolphin-emu.org/index.php?title=Mario_Kart:_Double_Dash%E2%80%BC">Mario Kart: Double Dash!!</a> and <a href="https://wiki.dolphin-emu.org/index.php?title=Super_Mario_Galaxy">Super Mario Galaxy 1</a>/<a href="https://wiki.dolphin-emu.org/index.php?title=Super_Mario_Galaxy_2">2</a> both suffered consistent music hangs, leading to the infamous Grand Star hang in the Super Mario Galaxy series.</p>
<p><br/></p>
<h3 id="the-solution"><strong>The Solution</strong><a class="headerlink" href="#the-solution" title="Permanent link">¶</a></h3>
<p>To work around the issues of DSP-HLE, <a href="https://code.google.com/u/nakeee@gmail.com/">nakeee</a>, <a href="http://code.google.com/u/103555087770768795143/">LordMark</a> and others created the DSP-LLE Interpreter. It represented an <em>extremely</em> accurate way of handling audio by perfectly reproducing what the DSP does, including synchronous audio processing. HLE audio could only approximate what the DSP does with C++ code, while LLE audio translated the binary code into x86, producing far more accurate results.</p>
<div class="container" style="margin-top:1em; margin-right:2em; margin-bottom:1em; max-width:30%; float:left;">
<img style="max-width: 100%; margin-bottom:.5em;" src="https://dolphin-emu.org/m/user/blog/rise-of-hle-audio/AudioSettings2.png" />
<p style="text-align:center; font-size:12px;">DSP-LLE and DSP-HLE still stand beside each other to this day.</p>
</div>
<p>Unfortunately, using an interpreter for the DSP is painfully slow and grinds the entire emulator to a halt. <a href="https://github.com/skidau/">skidau</a> and <a href="https://forums.dolphin-emu.org/User-kiesel-stein">kiesel</a> created DSP-LLE Recompiler to combat this slowdown the same way Dolphin uses a JIT for the main processor of the GameCube/Wii. Everything in DSP-LLE recompiler is the same as the interpreter, except it produces those very same results much faster.</p>
<p>However, DSP-LLE required DSP ROM dumps from a GameCube or Wii. While the procedure was fairly involved on a GameCube, later on the ease of softmodding a Wii would make retrieving DSP dumps simple - if you had one. Because of the dumps requirement and the chore required to get DSP ROM dumps, DSP-LLE could not be set as a default, limiting user exposure. That mixed with its slower way of handling audio made it difficult for users to swallow.</p>
<p><br/></p>
<h3 id="new-ax-hle-audio"><strong>New-AX-HLE Audio</strong><a class="headerlink" href="#new-ax-hle-audio" title="Permanent link">¶</a></h3>
<p>A change in audio hit in <a href="https://dolphin-emu.org/download/dev/ba348c29d7055314957d36d80fd515a5f4bb9a31/">3.5-78</a>, right after the release of Dolphin 3.5 late December 2012. Without having to worry about marring the release with a potentially buggy feature, delroth sprung one of the largest rewrites to Dolphin in years: New-AX-HLE. This not only rewrote and improved huge portions of the HLE audio system, but it also finally brought synchronous audio to DSP-HLE within the GameCube AX µcode.</p>
<p>The result was astounding; many games began outputting audio in DSP-HLE for the very first time, stability increased and the emulator was altogether faster because less titles needed DSP-LLE in order to run.</p>
<p>While this merge was sudden to Dolphin users, Delroth had already spent several months prior <a href="http://blog.lse.epita.fr/articles/38-emulating-the-gamecube-audio-processing-in-dolphin.html">rewriting most of the AX HLE code.</a> Of course, the first merge was only the beginning. His reverse engineering of the DSP resulted in him being able to write his own <a href="https://dolphin-emu.org/download/dev/9a404ca6d4020fa77c61d4cd210b924b06deaf94/">DSP-ROM that could be included with dolphin</a>! For the first time, users could get through problematic games with LLE audio without needing to go through the trouble of dumping one from their console!</p>
<p>Then DSP-LLE became mostly unneeded; delroth's second AX-HLE merger hit in <a href="https://dolphin-emu.org/download/dev/eb06c62a6e1f2c5b84a1a11cab269a5e81936f92/">3.5-1154</a>, fixing up the problems from the first one while also adding support for Wii AX games. Hundreds of games saw huge improvements in stability, HLE audio became usable in popular features such as netplay, and Dolphin found itself in a much stronger position for the future.</p>
<p>So what was the reaction to this increase in both speed and accuracy? Surprisingly mixed.</p>
<p><br/></p>
<h3 id="the-controversy-behind-synchronous-hle-audio"><strong>The controversy behind Synchronous HLE Audio</strong><a class="headerlink" href="#the-controversy-behind-synchronous-hle-audio" title="Permanent link">¶</a></h3>
<p>Sometimes <a href="http://xkcd.com/1172/">unexpected consequences completely alter how a fix is perceived</a>. The developers expected New-AX-HLE to be one of the most heralded features added to Dolphin in years; it improved speed and accuracy equally. We thought users would celebrate these changes! However, several factors mixed together to make New-AX-HLE a controversial feature.</p>
<p><img src="https://dolphin-emu.org/m/user/blog/rise-of-hle-audio/oldvsnewhle.svg" style="width:35%; margin: 1em 1em 2em 2em;float:right" /></p>
<ol>
<li>
<p>New-AX-HLE addition of synchronous audio brought over another behavior of DSP-LLE - when the game was not running full speed, audio wouldn't run full speed either.</p>
</li>
<li>
<p>New-AX-HLE made HLE audio behave akin to DSP-LLE, so it began to suffer a lot of the issues that went unaddressed in LLE audio due to its lack of use.</p>
</li>
<li>
<p>While not related to audio; Dolphin was going through a phase where it was getting slower due to hacks being removed and compatibility being raised. Games weren't running as fast as they were in older releases.</p>
</li>
<li>
<p>There was no official blog back then, so most users did not see the explanations as to why the behavior changed and the roadmap for the future.</p>
</li>
</ol>
<p>By far the biggest change in behavior was for users who couldn't run games fullspeed. The asynchronous behavior of old HLE allowed them to mostly ignore the slowdown because the game would sound reasonable unless audio outright crashed. The addition of synchronous audio processing took away that ability and made it very obvious that the game was lagging. Complaints mounted against the new way of handling HLE audio, and a small minority of users demanded the old system back.</p>
<p>The developers didn't just change audio overnight, though. The past months prior to merge were filled with discussions about what synchronous audio would bring to the emulator, and the best ways to transition. All that could be said is that the changes in behavior were intended; if the host machine could not run the emulator at 100%, sound should not be fullspeed. In order to guarantee stability in games, the audio <strong>must</strong> be synchronous with the action.</p>
<p>As you might expect, the latent explanation wasn't very convincing to the people that just wanted to play their games and didn't care how the emulator worked. What was thought to be one of the biggest features added to Dolphin ended up being one of its most controversial. Threads were started, emails sent, and it became very loud and agitated. To ease the process, delroth created an <a href="http://blog.lse.epita.fr/articles/38-emulating-the-gamecube-audio-processing-in-dolphin.html">article</a> to explain the changes in more detail, but this was long after the initial outrage.</p>
<p><br/></p>
<p>As with any outcry against a new feature, there were a lot of very valid complaints mixed in with the unreasonable ones. Long-time issues in Dolphin's audio interface made the jump from LLE to HLE thanks to the two systems working in similar manners. This included a massive 200ms latency on audio and a strange buffer bug that caused audio distortion after ninety minutes of play! By <a href="https://dolphin-emu.org/download/dev/cbe7656b2ffd669fbaa5e4b0c709e29250f98573/">4.0-879</a> those bugs and several others were squashed.</p>
<p><br/></p>
<h3 id="the-final-steps-toward-total-synchronous-audio"><strong>The Final Steps Toward Total Synchronous Audio</strong><a class="headerlink" href="#the-final-steps-toward-total-synchronous-audio" title="Permanent link">¶</a></h3>
<p><br/></p>
<p>New-AX-HLE covered roughly 96% of the games supported by Dolphin, but there were two small subsets of that did not fall under the scope of the New-AX-HLE merges. One set of games was extremely small, but included some of the most popular games in Dolphin: <a href="https://wiki.dolphin-emu.org/index.php?title=Zelda_ucode">Zelda µcode audio</a>.</p>
<p>The Zelda µcode, named after the fact it was primarily used in Zelda and Mario games (but surprisingly not Skyward Sword or New Super Mario Bros. Wii), is very difficult to emulate properly. It has more stringent timings and can use a lot of extra DSP features that the AX counterpart does not. That couldn't scare away magumagu. Without warning, he implemented synchronous HLE audio for Zelda µcode games in <a href="https://dolphin-emu.org/download/dev/784c636db8b885dc5b6e19ecf663a634e0f450bc/">4.0-1300</a>. This didn't handle all of the problems, instead it focused on fixing the critical problem of the asynchronous audio handling.</p>
<p>Then there was one more category of games that escaped the grasp of both rewrites: streaming audio, also known as Disc TracK (DTK) audio. DTK audio was mostly used for music in the background of levels, so there wasn't any real demand to make it synchronous except for the fact that it had a few issues with looping. Except, it wasn't <em>always</em> used for just background music. Some clever titles used streaming audio for voice acting and cutscenes, which absolutely demanded perfect timings.</p>
<p>After a lot of tweaking and testing, <a href="https://dolphin-emu.org/download/dev/11d304ae29b944b83e9962f2963f6d8556a0aa0f/">magumagu's DTK Rewrite hit Dolphin</a> finally bringing synchronous audio to the last holdouts.</p>
<p>The difference it makes can be easily seen in the following cutscene from <a href="https://wiki.dolphin-emu.org/index.php?title=Star_Fox_Adventures">Star Fox Adventures</a>.</p>
<p><br/></p>
<div class="text-center"><iframe width="360" height="240" src="//www.youtube.com/embed/691feBrEMLY?" frameborder="0" allowfullscreen></iframe></div>
<p><br/></p>
<h3 id="the-cleanup-crew"><strong>The Cleanup Crew</strong><a class="headerlink" href="#the-cleanup-crew" title="Permanent link">¶</a></h3>
<p>While delroth and magumagu earn top honors for the transition to synchronous audio in the AX-HLE, Zelda µcode HLE and DTK Audio, without the help of the rest of the development team, a lot of nasty bugs would still be lurking in the code!</p>
<ul>
<li>
<p><a href="https://github.com/degasus">degasus</a> cleaned up a lot of the bugs in the audio interface that old HLE skipped. This included the aforementioned <a href="https://dolphin-emu.org/download/dev/a1822a3acaf112dd37ed98818acdc0bc5014caba/">90 minutes bug</a> and <a href="https://dolphin-emu.org/download/dev/cbe7656b2ffd669fbaa5e4b0c709e29250f98573/">latency issues</a>.</p>
</li>
<li>
<p><a href="https://github.com/skidau">skidau</a> actually made a big change to Dolphin even before AX-HLE by adding the OpenAL audio backend and implementing time-stretching in it. Another fun-fact is that he applied that time-stretch to DTK audio, meaning although it was still buggy, it was at least synchronous when using OpenAL! More recently, he has made many contributions that include fixing many audio errors, missing audio, and weird behaviors that have emerged due to better timings. A lot of these changes have affected both LLE and HLE audio, including instrument loop-points and ARAM DMA timings. He also used the audio mixing in Dolphin to hookup play emulated Wiimote audio through the default audio device (usually system speakers).</p>
</li>
</ul>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/167098123%3Fsecret_token%3Ds-JcxfX"></iframe>
<p>At 20 seconds, notice the music slowly become more and more garbled.</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/167097389%3Fsecret_token%3Ds-VToYY"></iframe>
<p>skidau's fixed loop-points eliminate this issue in both HLE and LLE audio.</p>
</div>
<br clear="both"/>
</div>
<p><br/></p>
<ul>
<li>
<p><a href="https://github.com/Phire">Phire</a> handled a very strange bug that cropped up after the DTK Audio Rewrite. It turns out a missed static variable caused resampling issues between the DSP mixer and DVD mixer, resulting in the <a href="https://dolphin-emu.org/download/dev/ae65b3ba2c2ced70b3a04b392ebbd7ad2ccb0784/">audio <em>static</em></a> that was plaguing games. His relentless search for the bug is the reason it why it was fixed before it became a well-known problem.</p>
</li>
<li>
<p><a href="https://github.com/booto">booto</a> started an audio assault after being dragged in to fix other regressions in DTK audio. He used extensive hardware testing and reverse engineering to figure out how audio streaming worked and tuned up the emulator to get everything even remotely related to audio streaming working. The result is that not only were <a href="https://dolphin-emu.org/download/dev/30962ec36173eabfe3dc7ac21beeeaa31d459b77/">DTK audio looping bugs fixed</a>, but games like <a href="https://wiki.dolphin-emu.org/index.php?title=Pac-Man_Fever">Pac-Man Fever</a> began playing the right audio tracks for the very first time. He also fixed games with tight timing like <a href="https://wiki.dolphin-emu.org/index.php?title=Pokemon_snap">Pokemon Snap</a>, <a href="https://wiki.dolphin-emu.org/index.php?title=Harvest_Moon:_Magical_Melody">Harvest Moon: Magical Melody</a> and others with his <a href="https://dolphin-emu.org/blog/2014/08/31/dolphin-progress-report-august-2014/">audio assault</a>.</p>
</li>
<li>
<p><a href="https://github.com/konpie">konpie</a> has shown up to provide critical information and patches just when everyone else was out of ideas on several bugs. Two of the patches he provided fixed <a href="https://dolphin-emu.org/download/dev/305d2e18630ffb5660e2c4240fe9b9f344598c07/">severe crackling in several games</a>, and <a href="https://dolphin-emu.org/download/dev/b4e82a5b9bd36d3a7c8e2d0e7698e1ea47f30b13/">missing music</a> in <a href="https://wiki.dolphin-emu.org/index.php?title=GTJE5L">Tom and Jerry in War of the Whiskers</a>.</p>
</li>
</ul>
<p><br/></p>
<div class="container" style="display:block; max-width: 600px; min-width:50%; margin: auto auto 1em;">
<div style="max-width: 46%; float:left; text-align:center">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/172500692%3Fsecret_token%3Ds-jUvLk"></iframe>
<p>Crackling in games can be really annoying, especially when the only work-around is LLE.</p>
</div>
<div style="max-width: 46%; float:right; text-align:center;">
<iframe style="margin-bottom: .5em;" width="100%" height="166" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/172500642%3Fsecret_token%3Ds-9p2R4"></iframe>
<p>In this case, it turns out that the volume ramping was incorrect, causing crackling in various games.</p>
</div>
<br clear="both"/>
</div>
<p><br/></p>
<h3 id="conclusion"><strong>Conclusion</strong><a class="headerlink" href="#conclusion" title="Permanent link">¶</a></h3>
<p>While the benefits of Synchronous HLE took some time to take hold, Dolphin's audio emulation is in a far better place. With extremely high compatibility mixed with low requirement, DSP-HLE is now the default option for almost every title compatible with the emulator. This was something that was absolutely unthinkable just two years ago. There is still work to be done, but DSP-HLE has become a solid audio emulation option that will serve users well into the future.</p>
<p><br/></p>
<p style="width:70%; margin:0 auto; text-align:center;">Some information and examples used in this article were taken from delroth's original blog post, which covers the GameCube's audio technology in much more detail. It can be read <a href="http://blog.lse.epita.fr/articles/38-emulating-the-gamecube-audio-processing-in-dolphin.html">here</a>.</p>
<p><br/></p>