We propose a quantum information processing platform that utilizes the ultrafast time-bin encoding of photons. This approach offers a pathway to scalability by leveraging the inherent phase stability of collinear temporal interferometric networks at the femtosecond-to-picosecond timescale. The proposed architecture encodes information in ultrafast temporal bins processed using optically induced nonlinearities and birefringent materials while keeping photons in a single spatial mode. We demonstrate the potential for scalable photonic quantum information processing through two independent experiments that showcase the platform’s programmability and scalability, respectively. The scheme’s programmability is demonstrated in the first experiment, where we successfully program 362 different unitary transformations in up to eight dimensions in a temporal circuit. In the second experiment, we show the scalability of ultrafast time-bin encoding by building a passive optical network, with increasing circuit depth, of up to 36 optical modes. In each experiment, fidelities exceed 97%, while the interferometric phase remains passively stable for several days.