使用基于 ConvLSTM2D 的 Keras 模型从较低分辨率图像估计高分辨率图像

我正在尝试使用以下内容ConvLSTM2D从低分辨率图像序列估计高分辨率图像序列的架构：

import numpy as np, scipy.ndimage, matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, ConvLSTM2D, MaxPooling2D, UpSampling2D
from sklearn.metrics import accuracy_score, confusion_matrix, cohen_kappa_score
from sklearn.preprocessing import MinMaxScaler, StandardScaler
np.random.seed(123)

raw = np.arange(96).reshape(8,3,4)
data1 = scipy.ndimage.zoom(raw, zoom=(1,100,100), order=1, mode='nearest') #low res
print (data1.shape)
#(8, 300, 400)

data2 = scipy.ndimage.zoom(raw, zoom=(1,100,100), order=3, mode='nearest') #high res
print (data2.shape)
#(8, 300, 400)

X_train = data1.reshape(data1.shape[0], 1, data1.shape[1], data1.shape[2], 1)
Y_train = data2.reshape(data2.shape[0], 1, data2.shape[1], data2.shape[2], 1)
#(samples,time, rows, cols, channels)

model = Sequential()
input_shape = (data1.shape[0], data1.shape[1], data1.shape[2], 1)
#samples, time, rows, cols, channels
model.add(ConvLSTM2D(16, kernel_size=(3,3), activation='sigmoid',padding='same',input_shape=input_shape))     
model.add(ConvLSTM2D(8, kernel_size=(3,3), activation='sigmoid',padding='same'))

print (model.summary())

model.compile(loss='mean_squared_error',
              optimizer='adam',
              metrics=['accuracy'])

model.fit(X_train, Y_train, 
          batch_size=1, epochs=10, verbose=1)

x,y = model.evaluate(X_train, Y_train, verbose=0)
print (x,y)

该声明将导致以下结果Value error:

ValueError：输入 0 与层 conv_lst_m2d_2 不兼容：预期 ndim=5，发现 ndim=4

我该如何纠正这个问题ValueError？我认为问题出在输入形状上，但无法弄清楚到底出了什么问题。
请注意，输出也应该是图像序列，而不是分类结果。

发生这种情况是因为LSTMs需要时间数据，但你的第一个被声明为many-to-one模型，输出形状张量(batch_size, 300, 400, 16)。即批量图像：

model.add(ConvLSTM2D(16, kernel_size=(3,3), activation='sigmoid',padding='same',input_shape=input_shape))     
model.add(ConvLSTM2D(8, kernel_size=(3,3), activation='sigmoid',padding='same'))

您希望输出是形状张量(batch_size, 8, 300, 400, 16)（即图像序列），因此它们可以被第二个 LSTM 消耗。解决这个问题的方法是添加return_sequences在第一个 LSTM 定义中：

model.add(ConvLSTM2D(16, kernel_size=(3,3), activation='sigmoid',padding='same',input_shape=input_shape,
                     return_sequences=True))
model.add(ConvLSTM2D(8, kernel_size=(3,3), activation='sigmoid',padding='same'))

你提到了分类。如果你缩进的是对整个序列进行分类，那么你在最后需要一个分类器：

model.add(ConvLSTM2D(16, kernel_size=(3,3), activation='sigmoid',padding='same',input_shape=input_shape,
                     return_sequences=True))
model.add(ConvLSTM2D(8, kernel_size=(3,3), activation='sigmoid',padding='same'))
model.add(GlobalAveragePooling2D())
model.add(Dense(10, activation='softmax'))  # output shape: (None, 10)

但如果你想对每个图像进行分类within序列，然后你可以简单地使用分类器重新应用TimeDistributed:

x = Input(shape=(300, 400, 8))
y = GlobalAveragePooling2D()(x)
y = Dense(10, activation='softmax')(y)
classifier = Model(inputs=x, outputs=y)

x = Input(shape=(data1.shape[0], data1.shape[1], data1.shape[2], 1))
y = ConvLSTM2D(16, kernel_size=(3, 3),
               activation='sigmoid',
               padding='same',
               return_sequences=True)(x)
y = ConvLSTM2D(8, kernel_size=(3, 3),
               activation='sigmoid',
               padding='same',
               return_sequences=True)(y)
y = TimeDistributed(classifier)(y)  # output shape: (None, 8, 10)

model = Model(inputs=x, outputs=y)

最后，看一下 keras 存储库中的示例。有一个适合一个使用 ConvLSTM2D 的生成模型 https://github.com/keras-team/keras/blob/master/examples/conv_lstm.py.

编辑：从 data1 估计 data2...

如果这次我做对了X_train应该是 8 个 (300, 400, 1) 图像堆栈的 1 个样本，而不是 1 个形状 (300, 400, 1) 图像堆栈的 8 个样本。
如果这是真的，那么：

X_train = data1.reshape(data1.shape[0], 1, data1.shape[1], data1.shape[2], 1)
Y_train = data2.reshape(data2.shape[0], 1, data2.shape[1], data2.shape[2], 1)

应更新为：

X_train = data1.reshape(1, data1.shape[0], data1.shape[1], data1.shape[2], 1)
Y_train = data2.reshape(1, data2.shape[0], data2.shape[1], data2.shape[2], 1)

Also, accuracy当你的损失是 MSE 时，通常没有意义。您可以使用其他指标，例如mae.

现在您只需要更新模型以返回序列并在最后一层中有一个单元（因为您尝试估计的图像具有单个通道）：

model = Sequential()
input_shape = (data1.shape[0], data1.shape[1], data1.shape[2], 1)
model.add(ConvLSTM2D(16, kernel_size=(3, 3), activation='sigmoid', padding='same',
                     input_shape=input_shape,
                     return_sequences=True))
model.add(ConvLSTM2D(1, kernel_size=(3, 3), activation='sigmoid', padding='same',
                     return_sequences=True))

model.compile(loss='mse', optimizer='adam')

在那之后，model.fit(X_train, Y_train, ...)将正常开始训练：

Using TensorFlow backend.
(8, 300, 400)
(8, 300, 400)
Epoch 1/10

1/1 [==============================] - 5s 5s/step - loss: 2993.8701
Epoch 2/10

1/1 [==============================] - 5s 5s/step - loss: 2992.4492
Epoch 3/10

1/1 [==============================] - 5s 5s/step - loss: 2991.4536
Epoch 4/10

1/1 [==============================] - 5s 5s/step - loss: 2989.8523