Convolution layers are useful when your data has positional invariance, such as visual data (image and video). Text, on the other hand, is almost never positionally invariant. See pic
The convolutional layer doesn't randomize the order does it?
1 month ago
Anonymous
That's right it doesn't. The convolutional layer loses a lot of positional information (, that's not the same as randomizing order,) which is why it is bad for NLP where meaning is extremely sensitive to positional information.
1 month ago
Anonymous
I just wanted my ot to "understand". Should I use LSTM?
1 month ago
Anonymous
Yeah that's a good alternative.
1 month ago
Anonymous
Doesn't it need to be iterative to work with an LSTM?
1 month ago
Anonymous
Instead of asking one small question at a time, how about you take some time to figure out what you want and then come back after you've thought about it for a while.
1 month ago
Anonymous
I've studied this for a short while, from what I understand LSTM has to be iterative. Conv would I presume find permutations that would still be valid and accept differently phrased text as the same.
you CAN do CNN for NLP (see https://arxiv.org/pdf/1610.10099.pdf) and it was pretty popular 2013-2017 but transformers just werk
language fundamentally has a tree structure so you need kernels at all distances if you want to do CNN
I thought it would be a bunch of if/else
You got lied to. It's actually a bunch of JNE
>JNE
What is that?
asm for goto
Oh, jump if not equal!
yeah
and a bunch of other's similar
BNE, JNZ, etc.
I'm just trying to make it so that my neural net outputs something readable.
benzos? jengo!? what the FRICK are you on about?
you guys have been playing jenga without me?
LGTM
I don't think a convolutional layer makes sense for NLP applications. Order of words matters a lot.
Please explain.
Convolution layers are useful when your data has positional invariance, such as visual data (image and video). Text, on the other hand, is almost never positionally invariant. See pic
But it's not random
what is this in response to
The convolutional layer doesn't randomize the order does it?
That's right it doesn't. The convolutional layer loses a lot of positional information (, that's not the same as randomizing order,) which is why it is bad for NLP where meaning is extremely sensitive to positional information.
I just wanted my ot to "understand". Should I use LSTM?
Yeah that's a good alternative.
Doesn't it need to be iterative to work with an LSTM?
Instead of asking one small question at a time, how about you take some time to figure out what you want and then come back after you've thought about it for a while.
I've studied this for a short while, from what I understand LSTM has to be iterative. Conv would I presume find permutations that would still be valid and accept differently phrased text as the same.
you CAN do CNN for NLP (see https://arxiv.org/pdf/1610.10099.pdf) and it was pretty popular 2013-2017 but transformers just werk
language fundamentally has a tree structure so you need kernels at all distances if you want to do CNN
I love how designing neural networks is a combination of empirical research, esoteric ancestral knowledge and dudebro "common sense".
I grok some of it.
>tf.keras.layers.Dropout
The Conv2D neural net had figured out that there are words and was trying to decide what to use for the space.