Multi-head self-attention seems to be the new trendy architectural primitive.
I don't know how feasible it would be - I guess you could take a set of base operations (matrix multiplication, softmax, etc.) and randomly generate feature transformations and check if any of them yield good features (stick a linear readout at the end of it and test the performance on some downstream tasks).
That would be an unguided search - I guess you could try something like GA or something. Also, it uses neural network training as an inner loop step, so it would probably be to expensive. Better would be if you could get the gradient w.r.t. to the tentative operation somehow.
Problem is that training NNs is nontrivial and you might need things like BatchNorm and residual connections to make things stable, so you'd somehow have to search for good architectures for each operation as well.
I don't know how feasible it would be - I guess you could take a set of base operations (matrix multiplication, softmax, etc.) and randomly generate feature transformations and check if any of them yield good features (stick a linear readout at the end of it and test the performance on some downstream tasks).
That would be an unguided search - I guess you could try something like GA or something. Also, it uses neural network training as an inner loop step, so it would probably be to expensive. Better would be if you could get the gradient w.r.t. to the tentative operation somehow.
Problem is that training NNs is nontrivial and you might need things like BatchNorm and residual connections to make things stable, so you'd somehow have to search for good architectures for each operation as well.