Page 8 of 11
Posted: 07-29-18 3:50 pm
by Nicholas
Progress Update!

Actually, 10.5 is moving along quite rapidly, especially in the last two weeks. The BASSMIDI-based replacement synth for Windows and Android has been a delight to implement. Their API is very nice and it does most of the work for you, which is the opposite of what I'm used to! :lol:

It's much faster and (hopefully once I finish working out licensing with the Voice Crystal folks) I'm hoping to get an upgraded version of the sound set we've been using on the iPad (since 2012) with a higher-quality grand piano. That's a huge upgrade for Windows and something approaching a quantum leap for Android. It's so nice, I'm even considering adding it to the Mac and iPad versions, too, to have a nice, even, high-quality sound across all four platforms.

The latency is so much better, I was hoping to actually quantify it in a repeatable way this time so that the "X% faster" bullet point in the feature list will be based on real data. As a little hobby project in the evenings, I've been cobbling together a Teensy 2.0-based latency tester that will send out a MIDI note (via USB or MIDI port) and start a timer to see how long it takes before its headphone jack picks up any sound. (Most of the work has already been done by Google, but it's still been some fun. I don't get to tinker with hardware nearly as much as I'd like.)

Choosing the software synth on the Settings screen now gives you a new "Reverb" slider which -- used with some restraint -- can add some nice extra depth to the sound. (Without restraint you can make it sound like your piano has been installed in a public restroom!) :lol: And, if the background song scanner (that populates the song list) happens to stumble across any SF2 SoundFont files in the same folders, they're shown in a list on the same settings screen and you can choose between and audition them in a single click. That means you can bring your own favorite/preferred SoundFont quite easily, even on Android. :D

Other work in the meantime: the constant tide of Android bugs continues to assault and be pushed back. I think I may have found and fixed the current #1 source of Android crashes, which also coincidentally resulted in a nice speed/responsiveness improvement on that platform.

Remaining work includes a final bug fix before Chromebook support is complete, automatic latency detection for the new synth on Android (I've still witnessed some drop-outs using BASS's built-in detection, so hopefully we'll be able to layer another detector on top of that or at least include a manual slider), some UI wrap-up for the new built-in synth settings screen, actually measuring some MIDI latency :D, and the last handful of tiny bugs that have been reported recently.

Then this eight month(?!) "emergency" bug-fix release will be out the door and we'll be able to get back to work on Synthesia 11.

Posted: 08-01-18 10:10 am
by Nicholas
tester.jpg (119.76 KiB) Viewed 23030 times

I built a little latency tester prototype and ran it against most of the test hardware I have around here.

There are 1000 milliseconds (ms) in one second. On a typical computer monitor or tablet (running at 60 frames per second), each frame lasts for 1000/60 = 16.7ms.


This testing device sends a "Note On: Middle C at max velocity" message through a MIDI cable, starts a timer immediately, and counts the time until the very first evidence of any sound at all coming in from the headphone jack. It can measure that result down to the nearest two microseconds (a thousandth of a millisecond), so we've got about 500x the resolution we need to accurately characterize things at the millisecond level. It continues to send Note On/Off pairs automatically to acquire 25 separate measurements.


As a sanity check (and because I was curious :D ), I tested a few digital pianos directly. Their whole job is to produce sound quickly after a key strike, so presumably they will also respond to MIDI messages quickly. Keyboards should be the fastest things around, so these tests establish a nice baseline for the fastest we should realistically ever hope to see a software synth behave.

Keyboard average time to respond to a MIDI "Note On" message:
  • 6.4ms (800μs std dev) - Casio LK-100
  • 9.9ms (260μs std dev) - Yamaha EZ-200
  • 12.4ms (780μs std dev) - Yamaha P-70
These are delightfully fast (as you'll see) with an amazingly tight distribution. Every one of them produces sound in less time (often substantially) than a single video frame.

More data points that may be interesting:
  • 57.1ms (2.3ms std dev) - Synthesia on iPad 4
  • 57.2ms (30μs std dev) - Synthesia on iPad 2


Computer/tablet average time to respond to a MIDI "Note On" using legacy synths (Windows MME on Windows and Sonivox on Android), with the latency tester connected through an E-MU Xmidi 1x1:
  • 241.2ms (10ms std dev) - Surface Pro 2 (Windows 8.1)
  • 245.5ms (8.8ms std dev) - Intel NUC (D54250WYK, Windows 10 MME driver)
  • 59.7ms (3.8ms std dev) - Intel NUC (D54250WYK, Windows 10 UWP driver)
  • 182.0ms (6.7ms std dev) - Nvidia SHIELD tablet (Android 7.0)
  • 119.9ms (10.5ms std dev) - Google Nexus 7 (2013, Android 6.0.1)
  • 195.2ms (11.8ms std dev) - Kindle Fire 7 (Android 5.1.1)
  • 254.2ms (10.7ms std dev) - ASUS Transformer Pad (TF103C, Android 4.4.2)
Right away we can see that the legacy synths take ~15-40x longer to produce a note than a keyboard. The Windows synth at a quarter second is especially shameful. The wider spread most-likely has to do with Synthesia's own MIDI polling once per frame. (2x either of those standard deviations is close to a 16.7ms frame.)

Computer/tablet average time to respond to a MIDI "Note On" using the new BASS based synth (WASAPI on Windows and OpenSL on Android), with the latency tester connected through an E-MU Xmidi 1x1:
  • 76.7ms (5.5ms std dev) - Surface Pro 2 (Windows 8.1)
  • 69.9ms (0.2ms std dev) - Surface Pro 2 (Windows 8.1) with V-sync off (~160 fps)
  • 69.4ms (6.5ms std dev) - Intel NUC (D54250WYK, Windows 10)
  • 74.8ms (6.1ms std dev) - Nvidia SHIELD tablet (Android 7.0)
  • (6.6ms std dev) - Google Nexus 7 (2013, Android 6.0.1)
  • 241.6ms (7.6ms std dev) - Kindle Fire 7 (Android 5.1.1)
  • 331.8ms (8.0ms std dev) - ASUS Transformer Pad (TF103C, Android 4.4.2)
chart.png (16.73 KiB) Viewed 21185 times
There is a lot to take in, here.

For computers and "newer" Android, BASS is consistently half to one-third the time it takes the old synths! Four video frames is still inside the range of "imperceptible delay" for me (and hopefully you!), and this synth can use arbitrary SoundFonts and add effects like reverb (without impacting that latency). This is very exciting. :D

Tripling the frame rate (on Surface Pro 2) appears to have soaked up the "average 8ms in either direction" slop and tightened up the standard deviation (more than should have been expected).

The older the Android device, the less rosy the picture. On devices that don't have an audio "Fast Path", it's neck-and-neck with Sonivox usually winning on speed but BASS easily still winning on sound quality. I've decided to leave it up to users: both synths will be available in the Android version of Synthesia 10.5 (defaulting to the new BASS synth, which is the easy answer on Android 6 and later).

(Checking against my manual measurements from six years ago, I'm happy to see these results are very consistent with those.)

  • The only software synths that can break into the high 50ms range are Apple's and the broken/unusable UWP synth in Windows 10 at the expense of pops and crackles. So, really just Apple's. :D
  • BASS is often just behind that at ~70ms response time.
  • BASS universally has tighter response timing (i.e., lower standard deviations)
  • BASS universally sounds (or can sound) better than the built-in alternatives.
  • BASS is always faster (often much faster) on modern hardware.
  • On older 4.x and 5.x Android hardware, your mileage may vary... but as the owner of one of those devices, you already knew that! :lol:

Posted: 08-01-18 1:10 pm
by jimhenry
The rule of thumb in the organ world, if I remember correctly, is that 20 mSec is the threshold of perceptible delay. Organists learn to cope with perceptible delay because the electro-pneumatic mechanism used to open a pipe valve takes some time to open from the time an organ key is depressed, a pipe takes some time to "speak" after air starts flowing into the pipe, and the pipes are usually far enough away from the organist that it takes a perceptible length of time for the sound of a pipe to reach the organist.Having a pipe 20 feet from the organist, which is close, is enough to introduce a perceptible delay. I haven't played real pipe organs enough to learn to cope with the delays. I'm told that good organists don't listen to the sound because it will throw you off.

Posted: 08-01-18 2:45 pm
by Nicholas
Yeah, playing with units a bit, it looks like sound travels at 1.1 ft/ms. So that pipe at twenty feet is already enough to eat the 20ms.

I suppose when my options are 75ms or 240ms after twelve years of 240ms, I'm pretty happy to take 75! :lol:

Certainly the advice (now backed up by real data) will continue to be "use your keyboard's synth if you've got one", but in those cases where a keyboard synth isn't available, things might have reached "passable" now instead of "what do you mean a quarter second?!"

Posted: 11-21-18 5:57 am
by Nicholas
Progress Update!

Whew! 10.5 is finally out the door. The final sound set ended up being 20 MB (instead of the 4 MB set used on the iPad). And all 16 MB of new samples are for the main Acoustic Grand Piano instrument. It sounds very nice now. :D

Download here (notably not via the pink box, above).

Posted: 11-21-18 6:30 am
by monkel
Congrats! Will check it out asap.

Posted: 11-24-18 4:00 pm
by spstarr
The new 10.5 build is generating a crash dump, but it seems to work fine, attached it here.

Aha, when OBS Streamlabs is open thats throwing some exception with the DirectX handling.. non-fatal though, though if I close it then reopen it didn't dump.. odd.

Posted: 11-24-18 8:32 pm
by Nicholas
Yeah, OBS is one of the apps whose overlays seem to crash Synthesia for whatever reason. I haven't been able to track down whether it's Synthesia doing something they don't expect or them doing something Synthesia doesn't expect, but the workaround for now is "don't run them at the same time". :D

Posted: 11-26-18 6:40 pm
by spstarr
Yeah or open it before OBS :)

Any update or preview on a Synthesia 11 build for us to try out? I'm especially eager to try using just the notation mode and see how that looks, to better learn notation.

Posted: 11-26-18 8:43 pm
by Nicholas
A couple of the things are already finished and released in past updates (see the first page in this topic).

That said, we're finally back on task now that 10.5 is out the door. There are a couple of fires to put out (the Android version always breaks in new and exciting ways) over the next few days and then I finally get to resume the "Multiple Lines of Notation" preview mentioned in the top post.

Posted: 11-28-18 12:56 am
by spstarr
yes, that preview I am absolutely looking forward to trying out, thank you so much!

Posted: 12-01-18 11:41 am
by Korados
I'm sorry to interrupt here but I have some problems with 10.5. I use the built-in Midi synth now but unfortunately it causes severe crackling when the volume gets too loud
or when there are many instruments. Furthermore, I use Synthesia in full screen mode but when I exit and restart, Synthesia is in window mode again. It doesn't seem to save the full screen.

Posted: 12-01-18 11:34 pm
by Nicholas
I think I noticed that "full screen isn't saved" thing just a few days ago. I'll investigate.

Otherwise, did you adjust the volume slider under Settings --> Built-in MIDI Synthesizer? If you reset that back to 100%, does the problem still happen?

Posted: 12-04-18 4:10 pm
by Korados
I tried now to set the volume slider to 50 % but instead of crackling, there's a dull "buzzing" in the background. When I set it back, it starts crackling again.

Posted: 12-04-18 6:07 pm
by Nicholas
The idea is that the volume slider on that particular screen is just for subtle adjustments if your particular SoundFont has its samples adjusted too loud/quiet. It's really easy to get clipping there, so the right answer is usually to leave it at 100% and adjust the system volume instead.

That said, what you're describing sounds like something different.

Try this: open the configuration window (hold Shift while launching the app), find the "Synth.BufferExtra" entry near the bottom of the list, and change it to something like 300.

That will add a lot of extra latency, but as a test does it get rid of the buzzing? If it does, you can try smaller numbers. (Even something like 20 or 50 might help stop the buzzing.) The real question will be: if it does solve the problem, why isn't BASS's built-in latency detection working? :?

Posted: 12-08-18 8:21 pm
by Korados
I'm using the General User Soundfont because I find the sounds very realistic. I set the Buffer to 250 and there indeed is a latency and I'm afraid to tell you
that the crackling got even worse. But I think that it might be the soundfonts I used. Do you have any recommendations for soundfonts?

Edit: I just remembered something. The CoolSoft Midi Synth could load the soundfont into the RAM before starting a piece. Maybe that would make it better?

Posted: 12-08-18 8:33 pm
by Nicholas
Ah ha, that is very interesting. (Especially since it got worse!) More interesting is that GeneralUser isn't particularly large. Maybe 20 MB or so, right?

When you switch back to the "Built-in sounds by Voice Crystal", do you still get crackling? Or is it only happening with outside SoundFonts?

And, one last thing to check: there is a much newer, unofficial version of BASS (24 revisions newer than the one Synthesia 10.5 shipped with). You can download it, here:

If you copy that bass.dll over the one in the Synthesia folder (with Synthesia closed), does that fix the crackling?

Posted: 12-12-18 12:28 pm
by Korados
I tried the other Bassmidi version now and there's much less crackling! It only crackles when it really gets loud, like fortissimo and louder.
I guess the problem is not caused by the soundfont, I had this issue with both the included soundfont as well as others.
There's just another little thing: Sometimes when an instrument starts to play, it leaves out the first note. So I guess Bassmidi
doesn't preload the soundfont to the RAM like Coolsoft's Midi Synth.

Posted: 12-13-18 1:42 am
by Nicholas
Korados wrote:Sometimes when an instrument starts to play, it leaves out the first note. So I guess Bassmidi doesn't preload the soundfont to the RAM like Coolsoft's Midi Synth.
If you're still getting clipping, you might even try reducing the volume on the Settings --> Built-in MIDI Synthesizer screen down below 100%. That may solve it.

So, a bit of nomenclature: both Synthesia and VirtualMIDISynth use BASSMIDI under the hood. In this case, "I guess Bassmidi doesn't preload..." is contradictory because "... like Coolsoft's Midi Synth" implies it does. ;)

What you mean to say is that Synthesia doesn't preload the fonts. And you would be correct. Until the next silent 10.5 update where it will preload fonts by default (but let you turn it off if you don't want to multiply Synthesia's RAM use by 10-200x when loading giant SoundFonts). :D

Posted: 12-21-18 4:04 am
by RobertM
Has Synthesia 11 been released yet, have just downloaded 10.5. Really looking forward to being able to switch off rolling note and just try and play with the sheet music.