configuration file path for song locations

Have questions? Just saying hello? This is the place.
No explicit, hateful, or hurtful language. Nothing illegal.
Post Reply
zeitverschwendung
Posts: 5

Post by zeitverschwendung »

Dear all,

I am new to Synthesia and just did the basic setup, I am refering to the Windows 10 version here.
I would like to setup locations which are searched for songs on network shares.
This obviously works very well since my Windows User Home directory is on a network share which was automatically added to those locations by Synthesia, so I can see and use it within Synthesia.
I just cannot add any new network share locations to the list by means of using the dialogue which appears when pressing the little '+' since it does not let me navigate to the network.
I searched for the location of the configuration files for the list of locations which are searched, but could not find it yet.
I found the configuration files within this folder C:\Users\username\AppData\Roaming\Synthesia, but could not find it here.
It is also not contained by the SQLite3 file settings.db within that folder, checked with a DB-tool.
I also could neither find it within the registry not within the program folder.

Please give me a hint where to find that part of the configuration which contains the list of those locations so that I can manually add my network share containing the MIDI-files, thanks in advance!

Best Regards,
Nicholas
Posts: 13135

Post by Nicholas »

You're looking for "shell:appdata\Synthesia\folders.xml". (If you type the shell:appdata part into a "Run..." dialog, it resolves to the AppData\Roaming folder you mentioned previously.)

The folders.xml file has the list of paths Synthesia searches. That said, I don't think I've ever tried a UNC network path before and I can't say how the file system code used by the app will respond. My guess is it will just say "that folder doesn't exist". Today the workaround is to map the network share as a drive (or local folder) and then add that drive to the list of scanned folders.
zeitverschwendung
Posts: 5

Post by zeitverschwendung »

Dear Nicholas,

thank you so much, that hint led to success, though I could swear that the folders.xml wasn't there when I checked last time... I guess it was added when I manually added a folder in the configuration screen.

It prefer using UNCs over mapping network shares for several reasons, and it works like a charm.
The entry with the UNC I added was:
<Folder version="1" path="\\nas\AUDIO\MIDI\" recursive="1" />
Of course the user needs to hold the necessary access rights.

It just seems to be quite slow, I can literally watch the number of songs in the folder rise, like 3 songs each second. This would be a problem if there were thousands of songs in that folder and I'd like to directly start practicing a song beginning with e.g. 'z'.

A possible approach to tackle this issue would be to add a path/name cache, being filled during the initial scan. This way, the files which the folder contained during the last startup would directly be available from the beginning when Synthesia is started the next time. A verification process could run on startup in the background.
Or the folders could be rescanned manually by means of a button, and entries could be removed automatically when trying to access an entry leading to a file which is not available anymore.

However, this is just what I'd do, besides adding the possibility to directly enter (and maybe verify) UNCs in the dialogue. :-P
But for now this is all I need, thanks again! :)
Nicholas
Posts: 13135

Post by Nicholas »

That's interesting that it's able to resolve them. I wonder why it goes so slowly? I'd imagine that whatever the OS is doing under the hood to make the connection, it shouldn't be too much different than a temporarily-mapped drive. If it's really going through the whole process of negotiation/authentication each time, that would be pretty silly. As far as I can tell, C++'s standard filesystem library (which Synthesia uses under the hood) is made to be as platform-agnostic as possible, so I don't know of any network-specific ways of holding onto any resources that might speed those accesses up. (Again, this feels like something the OS should be doing for apps automatically, but isn't in this case.) :?

Hmm.

As for library rebuild speed... it's one of the embarrassments that I'm hoping to fix (relatively) soon. Making the library populate at least 100x faster (and shooting for 1000x) is currently around line 200 of the list (although there is going to be a little shakeup in the order due to recent events).

I wonder how much those changes will improve this situation? If it's still negotiating the network connection for each song, it probably won't be much. (If you temporarily map the UNC path to a drive, does it speed up very much? Or is there some other bottleneck like network speed that is getting in the way?)
zeitverschwendung
Posts: 5

Post by zeitverschwendung »

I just mapped a network drive, there's no improvements in speed with this approach, sill quite slow.
I noticed that if I just add the whole drive (in my case M:\) to the list of folders by means of the dialogue, it will neither work nor be persisted, after the next start of Synthesia it's gone. I need to select a folder on drive M in order to make it work.

I guess your're right about the abstract approach making things that slow, no wonder if the library goes through the whole negotiation process for each file.
I use to develop in Java and C#, which people usually caim to be much slower than C++, but never had such issues.

My whole intranet is 1 GBit/s, and I use port aggregation for all potential bottlenecks, so it shouldn't be a network issue.

Of course midi-files are tiny compared to e.g. movies, so even with a 1 GBit network the speed, e.g. of copying those files, is around 100 KB/s rather than 100 MB/s, which it would be for large files.

I don't know if you only touch those files or also parse/analyze them during scanning. In case of latter, I think that building a persistent cache might still be a reasonalbe approach to tackle this issue, not only for the current performance issue with the potentially inperformant library that's used (and could be exchanged or re-implemented).
But also for reasons of scalability this would be very helpful, especially if there's someone who really suffers from a very slow network connection, which might even be routed via the internet... or connected to a cloud drive.
Nicholas
Posts: 13135

Post by Nicholas »

I just tried the same experiment: manually adding a UNC entry to folders.xml, from a Win10 machine to a Win10 share across a 1Gbps network. The folder of 2372 files (totaling 70MB) was able to fully index in the app in ~14 seconds in both Synthesia 10.8 and the latest 10.9 development code.

That's something around 60x faster than your report, which is another interesting data point. I don't have anything like port aggregation set up (it's about as vanilla a network as you can imagine: a handful of machines plugged into the same switch via Cat6e/RJ45 and a router providing DHCP) so I'm not sure what to test next. If you simply copy the folder of MIDI files (via your typical Windows Explorer drag-and-drop) does that also seem to go slower than usual? I just tried and it was actually a couple seconds slower than Synthesia's own "open and read the contents of every file" at around 17 seconds. (Lots of tiny files really kills those transfers. Theoretically that should take something like half a second!)

But seeing it perform at the same speed as Windows itself (or even a little faster) makes me worry less that it's something Synthesia is doing wrong.
zeitverschwendung
Posts: 5

Post by zeitverschwendung »

I just copied 500 midi files totaling 10MB within about 30 seconds. I'm using a Qnap NAS as file server, large files are copied with about 100 MB/s. Two switches inbetween.
I guess that the most relevant difference is that my NAS still has rotating HDDs, while both of your windows machines should have SSDs installed, which makes a huge difference when dealing with those tiny files, it's well possible that that's causing the factor 60.
Nicholas
Posts: 13135

Post by Nicholas »

I intentionally picked a network share that runs on a (WD Red NAS) 5400 RPM HDD. I should have mentioned it last time, sorry. It also gets the same "large files copy at ~100 MB/s" statistic which is roughly the speed capacity of both the drive and the 1Gbps connection.

So just the via-file-explorer "10MB in 30s" vs. "70MB in 17s" already accounts for about 4x, which is curious given that the hardware sounds about the same. I wonder if your NAS box just handles lots of small requests more slowly? My test was between two fairly high-end desktop machines. Hmm.
zeitverschwendung
Posts: 5

Post by zeitverschwendung »

My NAS is a Qnap TS-453Be, equipped with the same WD Red as yours, configured as RAID5.
It's a rather low end NAS, but I'd claim that it's still more than the average person has at hand, which is why it is that important for a software to be able to deal with such issues... or compensate for them, in my eyes.
Of course I am well aware that this has for sure not the highest priority on your list of things to be implemented ;)
Nicholas
Posts: 13135

Post by Nicholas »

zeitverschwendung wrote: 11-08-22 11:23 am... it's still more than the average person has at hand...
I would argue that the average person doesn't have a NAS setup in their home. :lol:

I'm still not sure why your library populates so slowly. But seeing the "file copy in Windows Explorer" roughly at parity with how long it takes Synthesia to index the same files (again, at least on my machine :? ) makes me think I'm not doing anything overtly wrong in the code for that part of it.

There is still a lot of room for improvement beyond that, though, like you suggested at the outset. I agree that the song database should be persisted between runs, seeing the filename on disk should be enough to get it back in the list, and that the contents can be confirmed to be the same at some later time. A scheme like that would mean you're only sitting through the long part once, which is way better than every time you start the app!

I have some work scheduled to improve the speed of scanning for songs and with any luck I'll be able to get this idea added at the same time.
Post Reply