Traditional neural network architectures have largely evolved along two independent directions: improvements in learning mechanisms and improvements in representational capacity. Recurrent neural networks rely on backpropagation through time to learn temporal dependencies, while convolutional neural networks have been primarily confined to visual domains. This work explores an alternative perspective: neural networks can be extended at a higher level of abstraction, where learning mechanisms and representation strategies are treated as independent dimensions. Two complementary extensions are investigated. First, a persistent-memory sequence model replaces backpropagation through time with localized updates and lateral propagation, redefining temporal learning. Second, convolutional architectures are extended to multimodal structured data, demonstrating that convolution is not limited to visual domains but is a general method for learning localized structure. Experiments on sequence learning demonstrate that localized propagation produces stable learning behavior under constrained conditions, while multimodal convolution exhibits improved generalization with increased data. While both approaches remain incomplete, the results suggest that rethinking neural networks at the level of abstraction rather than architecture alone opens new directions for model design.