Background: Tertiary lymphoid structures (TLSs) mediate local antitumor immunity, and interest in them significantly increased since cancer immunotherapy was implemented. We examined TLS− tumor stromal blood vessel interplay for each breast cancer (BC) molecular subtype related to recurrence, lymphovascular invasion (LVI), and perineural invasion (PnI). Methods: TLSs were quantified on hematoxylin and eosin stain specimens followed by CD34/smooth muscle actin (SMA) double immunostaining for stromal blood vessel maturation assessment. Statistical analysis linked microscopy to recurrence, LVI, and PnI. Results: TLS negative (TLS−) subgroups in each BC molecular subtype (except to Luminal A) have higher LVI, PnI, and recurrence. A significant rise in LVI and PnI were observed for the HER2+/TLS− subgroup (p < 0.001). The triple negative breast cancer (TNBC)/TLS− subgroup had the highest recurrence and invasion risk which was also significantly related to tumor grade. PnI but not LVI significantly influenced recurrence in the TNBC/TLS+ subgroup (p < 0.001). TLS−stromal blood vessel interrelation was different amongst BC molecular subtypes. Conclusion: BC invasion and recurrence are strongly influenced by TLS presence and stromal blood vessels, especially for HER2 and TNBC BC molecular subtypes.