Rubye was born on April 27, 1918 in Chipley, Florida. Her parents were Eldra and Rosa Fowler. Rubye enjoyed her childhood with her siblings—five brothers and two sisters. At a very early age, she showed a talent for music so she took piano lessons all through her school years. Later, she studied organ and was the organist for her home church. Active in high school, she was editor for the school paper and played for the glee club. In the community, she was asked to play for teas, coffees, weddings, and funerals. She also helped her mother and father in their business.