Obsidian's limitations - number of files, slow search

Have I hit Obsidian file limit?

As I experiment with a vault dedicated to dictionaries and reference materials, I seem to have hit a wall. My desktop Obsidian hangs and wouldn't get past loading, when total number of notes is...

Meanwhile, the iPhone/iPad app loaded but hanged frequently at the 346k level. iOS performance doesn't matter that much. It's the desktop that must have decent performance.

At this point, I am declaring that this vault is damaged beyond repair, because previously I had a vault with 150K that eventually loaded and worked fine. 86K can't be the limit! My initial dump of 384K must have ruined it. Luckily, Obsidian vault management, being local file based, is a piece of cake. It's iCloud that's always the pain.

Alternative: the gradual approach

I created a new vault, and started adding folders/files gradually...

!250

Almost 300K. A new milestone!

Paring down...

Paring down...

!_attachments/Screen Shot 2023-07-12 at 15.57.20.png

But alas it blanked out in the end.

Don't disturb the indexing

Here's a tip: If a file is found, appears to be opened, but content isn't showing, consider waiting until the indexing is done. Don't wreck it.

!../_attachments/obsidian not showing file content - indexing in progress.mp4

While this problem occurred, iCloud is busy too.

!600

iCloud-caused re-indexing observed

Closing the app and restart: the famed re-indexing happened, as predicted by dev:

!_attachments/Screen Shot 2023-07-12 at 18.01.40.png

Conclusion: Obsidian on iCloud for 300~400K files = dead end

It's enough pain so far to decide to go against an Obsidian vault with at least 400K files on iCloud. To have a dictionary database with English words and phrases, and Chinese terms, the total number of entries naturally goes toward one million. If I include professional terminology from various fields, the number blows up to over two millions (think 金山詞霸). A 400K limit isn't going to cut it. I am hinging all my hope on the off-iCloud local vault.

Alt-alternative approach

I had an idea. If, as I learned, iCloud is the culprit in slow initializing and indexing, how about a pure local vault that does not require syncing? I set out to test it, dumping a huge load in one fell swoop:

By the way, this number happens to be a tad more than the limit reported by someone who actually tried it and experienced extremely slowness, at 280,986 files.

Yay! I know it's going to be okay, because the indexing (onetime) progress finally presented itself after some wait for the initial progress bar (loading vault, workspace, cache, etc.).

!500

And, I observed a much appreciated new behavior: while indexing, the CMD-O search and open found this entry and opened it right away:

!600

This is so uplifting! Next, attempting the one-third million milestone:

While I wait for good news, I found someone demanding 500K:

#Glasp

Indexing Large Number of Files - Still Not Working

Metadata

Page Comment

Highlights & Notes

The OP's use case is also linguistic. Great minds think alike? 😆 But can I up the ante? As my Glasp page comment says, my goal is at least 800K.

Paring down...

!../_attachments/ob-local-index-turns-blank-after-10hrs 2.mp4

Shut the app window. Restarted it. This time, I learned for the first time that it did make progress indexing despite the "hang." It now starts with 171K files remaining to index instead of 442K, meaning the previous attempt was at least 60% complete.

Conclusion

Update, only to confirm the dead end