You did your part in getting the movement going, and though it feels strange to back away, you do so anyways. You aren't a political person so you don't want to get too involved with Agudas Yisroel. Besides, your mission was to open a school for Jewish girls, which you've done. Mission accomplished, right?
At first the schools run fine without you, but eventually Bais Yaakov looses its momentum and spirit. Though you didn't realize it, you were the heart and soul of the whole thing and without you the movement crumbles.